{ "meta": { "machine": "Mac mini, Apple M2 Pro, 32 GB unified memory, plugged in, idle (verified via ps/top before timed runs)", "os": "macOS 26.5.2 (build 25F84), Darwin 25.5.0 arm64", "date": "2026-08-10/11 (local, America/Los_Angeles)", "versions": { "ollama": "0.32.7 (Homebrew)", "llama_cpp": "build 10330 (687e77892), Homebrew, AppleClang 21.0.0.21000101, arm64, Metal backend", "rapid_mlx": "0.12.9 (pip, fresh venv, Python 3.13.2)", "mlx_lm": "0.31.3", "mlx": "0.31.2", "mlx_vlm": "0.6.3 (installed only for the gemma raw-MLX run)", "lm_studio": "cask via Homebrew, CLI commit 71bd99c; GGUF engine llama.cpp-mac-arm64-apple-metal-advsimd@2.28.2, MLX engine mlx-llm-mac-arm64-apple-metal-advsimd@1.11.0" }, "protocol": { "server_path": "Client-side python script on localhost reading the stream: 1 warmup (short, discarded) then 3 timed SHORT + 3 timed LONG per engine/model. temperature 0, streaming on. TTFT = request sent to first generated chunk (content OR reasoning/thinking delta). decode_tps = (completion_tokens-1)/(t_last_chunk-t_first_chunk); completion_tokens from engine-reported usage/eval_count (never client-estimated).", "endpoints": "ollama /api/chat (NDJSON); llama-server and rapid-mlx and LM Studio OpenAI /v1/chat/completions (SSE, stream_options.include_usage)", "raw_framework": "llama-bench -p 512 -n 128 (5 reps, default); mlx_lm generate --max-tokens 128 --temp 0 with a fixed ~488-token real-text prompt, 3 reps", "context": "4096 where settable: llama-server -c 4096; ollama options.num_ctx=4096 per request; LM Studio --context-length 4096 at load; rapid-mlx serve does not take a context flag (model default window used)", "prefix_caching_note": "All engines keep the previous request's prompt KV by default, so LONG run 1 is a cold prefill and runs 2-3 are warm (near-zero prefill) for ollama/llama-server/rapid-mlx. LM Studio's MLX engine did NOT reuse the prompt cache across identical requests (all 3 long runs paid full prefill). Additional single-shot cold-prefill probes (prefix-broken prompt, model resident) are recorded under server.cold_prefill_probes." }, "flags": { "ollama": "default `ollama serve`; OLLAMA_MODELS pointed at a benchmark dir because ~/.ollama/models on this machine is a broken symlink to a missing external drive (never modified). No FLASH_ATTENTION/KV_CACHE_TYPE env set (brew caveat suggests them; we kept defaults: flash_attention=false).", "llama_server": "-c 4096 --port 8081 only; everything else default", "rapid_mlx": "--port 8000 only, defaults otherwise; gemma pair additionally --no-mllm (see pair note)", "lm_studio": "lms server start (port 1234), lms load --context-length 4096, defaults otherwise" }, "prompts": { "short": { "text": "Write a haiku about the ocean.", "max_tokens": 128 }, "long": { "text": "The history of personal computing is often told as a story of shrinking machines, but it is more accurately a story of shrinking distances. In the beginning, the distance between a person and a computer was institutional: computers lived in climate-controlled rooms inside universities, corporations, and government agencies, and ordinary people approached them, if at all, through intermediaries. A programmer wrote code on paper, a keypunch operator transferred it to cards, an operator fed the cards to the machine, and the results returned hours or days later. The computer was a civic monument, like a dam or a bridge, and computing was something done on your behalf rather than something you did.\n\nThe first great collapse of distance came with time-sharing in the 1960s. By slicing a mainframe's attention into small intervals and rotating it among dozens of terminals, time-sharing systems created the illusion that each user had the machine to themselves. The illusion mattered more than the engineering. For the first time, a person could type a command and watch the machine respond in seconds, could make an error and correct it immediately, could hold a conversation with a computer. Interactivity changed what people believed computers were for. They stopped being calculators that processed batches and started being environments that a person could inhabit.\n\nThe second collapse came with the microprocessor. When Intel condensed a computer's central logic onto a single chip in 1971, it made the economics of one-person-one-computer plausible for the first time. Hobbyists assembled machines like the Altair 8800 from kits, and within a few years companies like Apple, Commodore, and Tandy were selling assembled computers through retail stores. The distance between person and machine shrank from a terminal connected to a distant mainframe to a box on a desk at home. Ownership changed the psychology of computing again: an owned machine could be opened, modified, programmed for private ends, and used for purposes no institution would sanction, from games to genealogy.\n\nThe third collapse was graphical. Research at Xerox PARC in the 1970s, commercialized by Apple in 1984 and by Microsoft over the following decade, replaced the command line with windows, icons, and a pointing device. The significance was not aesthetic but cognitive: a graphical interface substituted recognition for recall. Instead of remembering an incantation, a user could survey a screen and recognize the thing they wanted to do. This lowered the literacy threshold for computing so dramatically that the machine could move from the den to the office desk of people who would never call themselves computer users.\n\nThe fourth collapse was connective. Networking, which had matured quietly inside research institutions since the ARPANET's first messages in 1969, reached ordinary households in the 1990s through dial-up modems and then broadband. The web gave the network a face, and suddenly the computer on the desk was a door rather than a box. Distance collapsed in a new dimension: not the distance between a person and their machine, but the distance between every person and every other machine. Commerce, publishing, correspondence, and community reorganized themselves around this fact with a speed that still has no historical parallel.\n\nThe fifth collapse was corporeal. The smartphone, crystallized by the iPhone in 2007, moved the computer from the desk to the pocket and the hand. Computing became continuous rather than sessional. A person no longer went to the computer; the computer accompanied the person, mediating navigation, photography, payment, and social contact. Touch interfaces completed the work the graphical interface began, making computing legible to toddlers and to people who had never used a keyboard. For most humans alive today, the first computer was a phone, and the desktop tradition is the exotic variant.\n\nEach collapse of distance carried a cost that was visible only in retrospect. Time-sharing introduced the administrator who could observe every user. Personal computers introduced fragmentation and the burden of maintenance. Graphical interfaces hid the machine's workings and produced generations of users with no mental model of what a file or a process is. The network introduced surveillance economics, in which attention became the commodity and the user became the product. The smartphone concentrated the industry into a duopoly of platform owners with veto power over software itself.\n\nThe pattern suggests a question for the present moment, as computing's next collapse of distance proceeds: the distance between intention and execution, closed by systems that act on a person's behalf from a description of what they want. If the historical pattern holds, the convenience will be real and immense, and the cost will be a new kind of dependence that becomes visible only after it is entrenched. The lesson of the first five collapses is not that progress is a trap, but that every interface is also an interest, and the terms of convenience are written by whoever owns the shortest distance.\n\nSummarize the essay above in a single paragraph.", "max_tokens": 256 }, "raw_prompt_for_mlx_lm": "The history of personal computing is often told as a story of shrinking machines, but it is more accurately a story of shrinking distances. In the beginning, the distance between a person and a computer was institutional: computers lived in climate-controlled rooms inside universities, corporations, and government agencies, and ordinary people approached them, if at all, through intermediaries. A programmer wrote code on paper, a keypunch operator transferred it to cards, an operator fed the cards to the machine, and the results returned hours or days later. The computer was a civic monument, like a dam or a bridge, and computing was something done on your behalf rather than something you did. The first great collapse of distance came with time-sharing in the 1960s. By slicing a mainframe's attention into small intervals and rotating it among dozens of terminals, time-sharing systems created the illusion that each user had the machine to themselves. The illusion mattered more than the engineering. For the first time, a person could type a command and watch the machine respond in seconds, could make an error and correct it immediately, could hold a conversation with a computer. Interactivity changed what people believed computers were for. They stopped being calculators that processed batches and started being environments that a person could inhabit. The second collapse came with the microprocessor. When Intel condensed a computer's central logic onto a single chip in 1971, it made the economics of one-person-one-computer plausible for the first time. Hobbyists assembled machines like the Altair 8800 from kits, and within a few years companies like Apple, Commodore, and Tandy were selling assembled computers through retail stores. The distance between person and machine shrank from a terminal connected to a distant mainframe to a box on a desk at home. Ownership changed the psychology of computing again: an owned machine could be opened, modified, programmed for private ends, and used for purposes no institution would sanction, from games to genealogy. The third collapse was graphical. Research at Xerox PARC in the 1970s, commercialized by Apple in 1984 and by Microsoft over the following decade, replaced the command line with windows, icons, and a pointing device. The significance was not aesthetic but cognitive: a graphical interface substituted recognition for recall. Instead of remembering an incantation," } }, "pairs": [ { "model": "gpt-oss-20b", "ollama_tag": "gpt-oss:20b", "ollama_quant": "MXFP4 (ollama show; arch gptoss, 20.9B)", "mlx_alias": "gpt-oss-20b", "mlx_repo": "mlx-community/gpt-oss-20b-MXFP4-Q8", "mlx_quant": "MXFP4 experts / Q8 rest (native mxfp4 checkpoint), ~11.3 GiB", "llama_cpp_gguf": "ggml-org/gpt-oss-20b-GGUF gpt-oss-20b-MXFP4.gguf (11.27 GiB) — ollama's blob could NOT be reused: ollama's conversion uses arch key 'gptoss' which upstream llama.cpp rejects ('unknown model architecture: gptoss'); official ggml-org mxfp4 GGUF used instead (same native MXFP4 quantization)", "lm_studio_artifacts": "GGUF: lmstudio-community/gpt-oss-20b-GGUF gpt-oss-20b-MXFP4.gguf (12.11 GB, variant @gguf-mxfp4); MLX: mlx-community/gpt-oss-20b-MXFP4-Q8 (identical repo to rapid-mlx, variant @mlx-mxfp4)", "bpw_note": "Both sides run the model's native MXFP4 weights - cleanest apples-to-apples in this set.", "workload_note": "gpt-oss is a reasoning model. ollama and llama-server stream reasoning tokens and hit the max_tokens cap (128/256). rapid-mlx by default injects enable_thinking=false for casual chat ('R12-T2F auto-disable' in server log), so its default completions are shorter (24/151 tokens); a supplementary suite with chat_template_kwargs.enable_thinking=true (rapid_mlx_thinking_enabled) hits the caps like the others. LM Studio completions were 49/183 (GGUF) and 41/209-241 (MLX) tokens. Per-token throughput (decode_tps) is comparable across all of these; total request time is not, because generated token counts differ.", "server": { "ollama": { "short": { "ttft_ms": [ 445.7, 461.7, 455.2 ], "median_ttft_ms": 455.2, "decode_tps": [ 33.37, 33.58, 32.1 ], "median_decode_tps": 33.37, "completion_tokens": [ 128, 128, 128 ], "token_source": "ollama_eval_count", "ollama_reported": [ { "eval_count": 128, "eval_duration": 3887534000, "prompt_eval_count": 75, "prompt_eval_duration": 97392000, "total_duration": 4250427958, "load_duration": 261684833 }, { "eval_count": 128, "eval_duration": 3864798000, "prompt_eval_count": 75, "prompt_eval_duration": 97688000, "total_duration": 4242260375, "load_duration": 276954000 }, { "eval_count": 128, "eval_duration": 4041008000, "prompt_eval_count": 75, "prompt_eval_duration": 91498000, "total_duration": 4411018333, "load_duration": 275779292 } ] }, "long": { "ttft_ms": [ 2662.3, 405.5, 421.4 ], "median_ttft_ms": 421.4, "decode_tps": [ 32.13, 32.59, 31.75 ], "median_decode_tps": 32.13, "completion_tokens": [ 256, 256, 256 ], "token_source": "ollama_eval_count", "ollama_reported": [ { "eval_count": 256, "eval_duration": 8018964000, "prompt_eval_count": 1044, "prompt_eval_duration": 2308722000, "total_duration": 10598611791, "load_duration": 253052125 }, { "eval_count": 256, "eval_duration": 7909786000, "prompt_eval_count": 1044, "prompt_eval_duration": 37869000, "total_duration": 8229618666, "load_duration": 267015916 }, { "eval_count": 256, "eval_duration": 8115834000, "prompt_eval_count": 1044, "prompt_eval_duration": 42488000, "total_duration": 8451376667, "load_duration": 277932625 } ] } }, "llama_server": { "short": { "ttft_ms": [ 165.5, 168.8, 159.6 ], "median_ttft_ms": 165.5, "decode_tps": [ 43.49, 43.98, 43.48 ], "median_decode_tps": 43.49, "completion_tokens": [ 128, 128, 128 ], "token_source": "usage.completion_tokens" }, "long": { "ttft_ms": [ 2019.3, 108.9, 115.2 ], "median_ttft_ms": 115.2, "decode_tps": [ 42.49, 42.62, 42.99 ], "median_decode_tps": 42.62, "completion_tokens": [ 256, 256, 256 ], "token_source": "usage.completion_tokens" } }, "rapid_mlx": { "short": { "ttft_ms": [ 195.7, 183.3, 166.4 ], "median_ttft_ms": 183.3, "decode_tps": [ 51.56, 50.99, 51.38 ], "median_decode_tps": 51.38, "completion_tokens": [ 24, 24, 24 ], "token_source": "usage.completion_tokens" }, "long": { "ttft_ms": [ 3663.4, 234.7, 237.4 ], "median_ttft_ms": 237.4, "decode_tps": [ 45.8, 45.55, 45.76 ], "median_decode_tps": 45.76, "completion_tokens": [ 151, 151, 151 ], "token_source": "usage.completion_tokens" } }, "rapid_mlx_thinking_enabled": { "short": { "ttft_ms": [ 249.0, 226.4, 216.4 ], "median_ttft_ms": 226.4, "decode_tps": [ 50.41, 49.84, 50.39 ], "median_decode_tps": 50.39, "completion_tokens": [ 128, 128, 128 ], "token_source": "usage.completion_tokens" }, "long": { "ttft_ms": [ 3705.1, 256.3, 260.6 ], "median_ttft_ms": 260.6, "decode_tps": [ 45.53, 45.21, 45.34 ], "median_decode_tps": 45.34, "completion_tokens": [ 256, 256, 256 ], "token_source": "usage.completion_tokens" } }, "lm_studio_gguf": { "short": { "ttft_ms": [ 144.2, 142.7, 149.4 ], "median_ttft_ms": 144.2, "decode_tps": [ 52.63, 52.78, 45.96 ], "median_decode_tps": 52.63, "completion_tokens": [ 49, 49, 49 ], "token_source": "usage.completion_tokens" }, "long": { "ttft_ms": [ 2338.9, 113.8, 113.1 ], "median_ttft_ms": 113.8, "decode_tps": [ 44.43, 42.24, 41.64 ], "median_decode_tps": 42.24, "completion_tokens": [ 183, 183, 183 ], "token_source": "usage.completion_tokens" } }, "lm_studio_mlx": { "short": { "ttft_ms": [ 452.2, 443.7, 448.4 ], "median_ttft_ms": 448.4, "decode_tps": [ 71.47, 72.1, 71.44 ], "median_decode_tps": 71.47, "completion_tokens": [ 41, 41, 41 ], "token_source": "usage.completion_tokens" }, "long": { "ttft_ms": [ 3839.9, 3996.2, 4001.7 ], "median_ttft_ms": 3996.2, "decode_tps": [ 48.21, 48.05, 47.98 ], "median_decode_tps": 48.05, "completion_tokens": [ 209, 241, 241 ], "token_source": "usage.completion_tokens" } } }, "raw": { "llama_bench": { "pp512_tps": 557.58, "pp512_stddev": 1.55, "tg128_tps": 52.03, "tg128_stddev": 0.02, "reps": 5 }, "mlx_lm": { "prompt_tps": [ 247.898, 249.759, 257.752 ], "median_prompt_tps": 249.759, "gen_tps": [ 49.64, 49.526, 49.536 ], "median_gen_tps": 49.536, "prompt_tokens": 534, "gen_tokens": 128, "peak_mem_gb": [ 12.527, 12.527, 12.527 ] } } }, { "model": "qwen3.6-35b-a3b (MoE, 3B active; hybrid linear-attention/Mamba)", "ollama_tag": "qwen3.6:35b-a3b-q4_K_M", "ollama_quant": "Q4_K_M (ollama show; arch qwen35moe, 36.0B, 23 GB on disk)", "mlx_alias": "qwen3.6-35b-4bit", "mlx_repo": "mlx-community/Qwen3.6-35B-A3B-4bit", "mlx_quant": "4-bit (~19.0 GiB)", "llama_cpp_gguf": "bartowski/Qwen_Qwen3.6-35B-A3B-GGUF Qwen_Qwen3.6-35B-A3B-Q4_K_M.gguf (20.74 GiB) — ollama's blob again NOT loadable by upstream llama.cpp (error: 'key qwen35moe.rope.dimension_sections has wrong array length; expected 4, got 3'); bartowski upstream-converted Q4_K_M used instead", "bpw_note": "GGUF Q4_K_M ~= 4.8 bpw effective vs MLX 4-bit group-64 ~= 4.5 bpw - MLX side carries slightly fewer bits per weight (19.0 vs 20.7 GiB). Reported, not hidden.", "storage_note": "23 GB ollama blob and GGUF lived on SMB volume (17 GB local free could not hold them); an extra full LONG generation was run before each timed suite to make weights RAM-resident so SMB page-in does not pollute timings. Extra warmup TTFTs (dominated by model load) recorded in anomalies.", "server": { "ollama": { "short": { "ttft_ms": [ 308.6, 272.0, 274.7 ], "median_ttft_ms": 274.7, "decode_tps": [ 39.25, 39.26, 39.25 ], "median_decode_tps": 39.25, "completion_tokens": [ 128, 128, 128 ], "token_source": "ollama_eval_count", "ollama_reported": [ { "eval_count": 128, "eval_duration": 3236329000, "prompt_eval_count": 18, "prompt_eval_duration": 73776000, "total_duration": 3543288458, "load_duration": 217791250 }, { "eval_count": 128, "eval_duration": 3235003000, "prompt_eval_count": 18, "prompt_eval_duration": 65879999, "total_duration": 3506137958, "load_duration": 203919958 }, { "eval_count": 128, "eval_duration": 3236347000, "prompt_eval_count": 18, "prompt_eval_duration": 63826000, "total_duration": 3510029792, "load_duration": 208671667 } ] }, "long": { "ttft_ms": [ 303.4, 277.0, 285.0 ], "median_ttft_ms": 285.0, "decode_tps": [ 39.09, 39.04, 39.03 ], "median_decode_tps": 39.04, "completion_tokens": [ 256, 256, 256 ], "token_source": "ollama_eval_count", "ollama_reported": [ { "eval_count": 256, "eval_duration": 6523232000, "prompt_eval_count": 1014, "prompt_eval_duration": 82624000, "total_duration": 6825547458, "load_duration": 210901167 }, { "eval_count": 256, "eval_duration": 6532092000, "prompt_eval_count": 1014, "prompt_eval_duration": 69060000, "total_duration": 6808431667, "load_duration": 200387625 }, { "eval_count": 256, "eval_duration": 6534569000, "prompt_eval_count": 1014, "prompt_eval_duration": 66879000, "total_duration": 6818526042, "load_duration": 209797250 } ] } }, "llama_server": { "short": { "ttft_ms": [ 106.9, 58.8, 59.2 ], "median_ttft_ms": 59.2, "decode_tps": [ 38.64, 38.72, 38.68 ], "median_decode_tps": 38.68, "completion_tokens": [ 128, 128, 128 ], "token_source": "usage.completion_tokens" }, "long": { "ttft_ms": [ 65.0, 62.3, 61.6 ], "median_ttft_ms": 62.3, "decode_tps": [ 38.34, 38.33, 38.31 ], "median_decode_tps": 38.33, "completion_tokens": [ 256, 256, 256 ], "token_source": "usage.completion_tokens" } }, "rapid_mlx": { "short": { "ttft_ms": [ 268.6, 258.1, 253.7 ], "median_ttft_ms": 258.1, "decode_tps": [ 65.26, 64.98, 65.35 ], "median_decode_tps": 65.26, "completion_tokens": [ 21, 21, 21 ], "token_source": "usage.completion_tokens" }, "long": { "ttft_ms": [ 375.1, 383.8, 320.3 ], "median_ttft_ms": 375.1, "decode_tps": [ 58.65, 58.97, 59.35 ], "median_decode_tps": 58.97, "completion_tokens": [ 187, 187, 187 ], "token_source": "usage.completion_tokens" } }, "rapid_mlx_thinking_enabled": { "short": { "ttft_ms": [ 228.7, 226.9, 227.6 ], "median_ttft_ms": 227.6, "decode_tps": [ 61.82, 61.78, 61.83 ], "median_decode_tps": 61.82, "completion_tokens": [ 128, 128, 128 ], "token_source": "usage.completion_tokens" }, "long": { "ttft_ms": [ 325.6, 312.5, 326.0 ], "median_ttft_ms": 325.6, "decode_tps": [ 59.02, 59.14, 57.57 ], "median_decode_tps": 59.02, "completion_tokens": [ 256, 256, 256 ], "token_source": "usage.completion_tokens" } }, "cold_prefill_probes": { "ollama": { "ttft_ms": 2642.1, "decode_tps": 39.06, "ollama_reported": { "eval_count": 256, "eval_duration": 6528805000, "prompt_eval_count": 1018, "prompt_eval_duration": 2305161000, "total_duration": 9154097541, "load_duration": 293643500 } }, "llama_server": { "ttft_ms": 2345.4, "decode_tps": 37.76, "usage": { "completion_tokens": 256, "prompt_tokens": 1018, "total_tokens": 1274, "prompt_tokens_details": { "cached_tokens": 0 } } }, "rapid_mlx": { "ttft_ms": 3436.4, "decode_tps": 58.81, "usage": { "prompt_tokens": 1020, "completion_tokens": 184, "total_tokens": 1204 } } } }, "raw": { "llama_bench": { "pp512_tps": 460.13, "pp512_stddev": 2.51, "tg128_tps": 39.13, "tg128_stddev": 0.02, "reps": 5 }, "mlx_lm": { "prompt_tps": [ 276.716, 276.847, 277.053 ], "median_prompt_tps": 276.847, "gen_tps": [ 61.815, 61.638, 61.725 ], "median_gen_tps": 61.725, "prompt_tokens": 490, "gen_tokens": 128, "peak_mem_gb": [ 20.261, 20.261, 20.261 ] } } }, { "model": "gemma4-12b-it (dense)", "ollama_tag": "gemma4:12b-it-q4_K_M", "ollama_quant": "Q4_K_M (ollama show; arch gemma4, 11.9B, 7.6 GB incl. vision projector)", "mlx_alias": "gemma-4-12b-4bit", "mlx_repo": "mlx-community/gemma-4-12B-it-4bit", "mlx_quant": "4-bit", "llama_cpp_gguf": "REUSED ollama blob directly (sha256-1278394b..., 6.86 GiB text model; vision projector blob not loaded) — loads fine in upstream llama.cpp", "lm_studio_artifacts": "GGUF: lmstudio-community/gemma-4-12B-it-GGUF gemma-4-12B-it-Q4_K_M.gguf (variant @q4_k_m; byte-size within 1 KB of the ollama blob); MLX: lmstudio-community/gemma-4-12B-it-MLX-4bit (variant @4bit, 6.31 GiB loaded)", "bpw_note": "GGUF Q4_K_M ~= 4.8 bpw vs MLX 4-bit g64 ~= 4.5 bpw.", "rapid_mlx_note": "gemma-4-12B-it-4bit is a multimodal (gemma4_unified) checkpoint. rapid-mlx serve requires the [vision] extra for it; benchmark used the documented text-only lane `--no-mllm` (fair vs llama.cpp which also ran the text model without the vision projector).", "raw_mlx_note": "mlx_lm 0.31.3 cannot load gemma4_unified ('Model type gemma4_unified not supported'), so the raw MLX number was collected with mlx_vlm 0.6.3 generate (same checkpoint, text-only prompt, same 3-rep protocol).", "server": { "ollama": { "short": { "ttft_ms": [ 630.6, 621.0, 632.9 ], "median_ttft_ms": 630.6, "decode_tps": [ 18.92, 18.92, 18.91 ], "median_decode_tps": 18.92, "completion_tokens": [ 128, 128, 128 ], "token_source": "ollama_eval_count", "ollama_reported": [ { "eval_count": 128, "eval_duration": 6876884000, "prompt_eval_count": 24, "prompt_eval_duration": 186968000, "total_duration": 7342137958, "load_duration": 272642583 }, { "eval_count": 128, "eval_duration": 6876829000, "prompt_eval_count": 24, "prompt_eval_duration": 184619000, "total_duration": 7332803375, "load_duration": 270029042 }, { "eval_count": 128, "eval_duration": 6880074000, "prompt_eval_count": 24, "prompt_eval_duration": 193996000, "total_duration": 7347198084, "load_duration": 271962917 } ] }, "long": { "ttft_ms": [ 6785.5, 668.6, 678.2 ], "median_ttft_ms": 678.2, "decode_tps": [ 17.99, 17.98, 17.92 ], "median_decode_tps": 17.98, "completion_tokens": [ 256, 256, 256 ], "token_source": "ollama_eval_count", "ollama_reported": [ { "eval_count": 256, "eval_duration": 14344770000, "prompt_eval_count": 1013, "prompt_eval_duration": 6332933000, "total_duration": 20958397167, "load_duration": 272439333 }, { "eval_count": 256, "eval_duration": 14355103000, "prompt_eval_count": 1013, "prompt_eval_duration": 217630000, "total_duration": 14852518625, "load_duration": 271502041 }, { "eval_count": 256, "eval_duration": 14405461000, "prompt_eval_count": 1013, "prompt_eval_duration": 225403000, "total_duration": 14911561208, "load_duration": 272274750 } ] } }, "llama_server": { "short": { "ttft_ms": [ 348.0, 343.3, 342.9 ], "median_ttft_ms": 343.3, "decode_tps": [ 19.42, 19.35, 19.3 ], "median_decode_tps": 19.35, "completion_tokens": [ 128, 128, 128 ], "token_source": "usage.completion_tokens" }, "long": { "ttft_ms": [ 6697.5, 389.8, 390.4 ], "median_ttft_ms": 390.4, "decode_tps": [ 18.31, 18.31, 18.31 ], "median_decode_tps": 18.31, "completion_tokens": [ 256, 256, 256 ], "token_source": "usage.completion_tokens" } }, "rapid_mlx": { "short": { "ttft_ms": [ 242.8, 241.9, 227.2 ], "median_ttft_ms": 241.9, "decode_tps": [ 23.64, 23.75, 23.67 ], "median_decode_tps": 23.67, "completion_tokens": [ 23, 23, 23 ], "token_source": "usage.completion_tokens" }, "long": { "ttft_ms": [ 9703.3, 639.8, 428.7 ], "median_ttft_ms": 639.8, "decode_tps": [ 18.58, 18.74, 18.72 ], "median_decode_tps": 18.72, "completion_tokens": [ 173, 165, 165 ], "token_source": "usage.completion_tokens" } }, "rapid_mlx_thinking_enabled": { "short": { "ttft_ms": [ 329.6, 335.0, 344.5 ], "median_ttft_ms": 335.0, "decode_tps": [ 22.41, 22.4, 22.41 ], "median_decode_tps": 22.41, "completion_tokens": [ 128, 128, 128 ], "token_source": "usage.completion_tokens" }, "long": { "ttft_ms": [ 9715.1, 719.7, 623.5 ], "median_ttft_ms": 719.7, "decode_tps": [ 18.68, 18.76, 18.66 ], "median_decode_tps": 18.68, "completion_tokens": [ 256, 256, 256 ], "token_source": "usage.completion_tokens" } }, "lm_studio_gguf": { "short": { "ttft_ms": [ 373.5, 385.8, 383.6 ], "median_ttft_ms": 383.6, "decode_tps": [ 17.83, 17.47, 17.53 ], "median_decode_tps": 17.53, "completion_tokens": [ 128, 128, 128 ], "token_source": "usage.completion_tokens" }, "long": { "ttft_ms": [ 7155.3, 397.8, 395.1 ], "median_ttft_ms": 397.8, "decode_tps": [ 18.01, 17.98, 18.0 ], "median_decode_tps": 18.0, "completion_tokens": [ 256, 256, 256 ], "token_source": "usage.completion_tokens" } }, "lm_studio_mlx": { "short": { "ttft_ms": [ 482.2, 477.3, 475.7 ], "median_ttft_ms": 477.3, "decode_tps": [ 24.09, 24.06, 24.08 ], "median_decode_tps": 24.08, "completion_tokens": [ 127, 127, 127 ], "token_source": "usage.completion_tokens" }, "long": { "ttft_ms": [ 10648.1, 3123.4, 3127.1 ], "median_ttft_ms": 3127.1, "decode_tps": [ 21.81, 22.02, 22.04 ], "median_decode_tps": 22.02, "completion_tokens": [ 255, 255, 255 ], "token_source": "usage.completion_tokens" } }, "cold_prefill_probes": { "rapid_mlx": { "ttft_ms": 9536.3, "decode_tps": 18.79, "usage": { "prompt_tokens": 1014, "completion_tokens": 161, "total_tokens": 1175, "prompt_tokens_details": { "cached_tokens": 4 } } } } }, "raw": { "llama_bench": { "pp512_tps": 172.34, "pp512_stddev": 0.01, "tg128_tps": 19.65, "tg128_stddev": 0.01, "reps": 5 }, "mlx_lm": { "prompt_tps": [ 96.001, 95.914, 95.934 ], "median_prompt_tps": 95.934, "gen_tps": [ 22.147, 22.057, 22.148 ], "median_gen_tps": 22.147, "prompt_tokens": 488, "gen_tokens": 128, "peak_mem_gb": [ 7.825, 7.825, 7.825 ], "tool_note": "mlx_vlm==0.6.3 (mlx_lm cannot load gemma4_unified)" } } } ], "anomalies": [ "gpt-oss/ollama LONG run 1 TTFT 2662 ms vs 405/421 ms runs 2-3: run 1 is the cold prefill; runs 2-3 hit ollama's prompt-prefix KV cache. Same pattern for llama-server (2019 vs 109/115 ms) and rapid-mlx (3663 vs 235/237 ms). Medians therefore reflect the warm-prefix case; cold numbers are the run-1 values and the cold_prefill_probes.", "qwen/ollama extra SMB warmup request had TTFT 180.6 s - that is model load from the SMB share (~23 GB at LAN speed), not inference; excluded from timed runs by design.", "qwen/llama-server extra warmup TTFT 2717 ms and decode 31.6 t/s (page-in still in progress); timed runs immediately after stabilized at 38.3-38.7 t/s.", "gemma/ollama LONG cold TTFT 6786 ms and llama-bench pp512 172 t/s confirm gemma4-12b prefill is simply slow on this GPU in llama.cpp-family engines; rapid-mlx gemma cold prefill was slower still (~9.5-9.7 s for ~1050 tokens, ~110 t/s).", "lm_studio_gguf gpt-oss SHORT decode rep 3 = 45.96 t/s vs 52.63/52.78 (-12.7%, just under the 15% re-run threshold); kept, noted.", "LM Studio MLX engine paid full prefill on every LONG run (TTFT 3840/3996/4002 ms) - no cross-request prompt cache observed in mlx-llm@1.11.0.", "rapid-mlx gemma LONG completions varied slightly across temp-0 runs (173/165/165 tokens run-to-run).", "LM Studio gemma MLX LONG: run 1 TTFT 10648 ms (cold), runs 2-3 ~3125 ms - unlike its gpt-oss MLX runs, some (partial) prompt-cache reuse appeared here; still far slower than the near-zero warm TTFTs of ollama/llama-server/rapid-mlx.", "gpt-oss completion token counts differ per engine at identical prompts because each engine applies different default reasoning behavior (see workload_note)." ], "not_measured": [ { "what": "ollama blob reuse for llama.cpp (gpt-oss and qwen3.6 pairs)", "reason": "ollama ships its own GGUF conversions that upstream llama.cpp rejects (gptoss arch key; qwen35moe rope.dimension_sections length mismatch). Equivalent-quant upstream GGUFs were downloaded instead and are documented per pair." }, { "what": "ollama and llama-server cold-prefill probes for the gpt-oss pair", "reason": "probe methodology was introduced during the qwen pair; for gpt-oss the LONG run-1 values (2662 ms ollama, 2019 ms llama-server, 3663 ms rapid-mlx) already capture the cold case." }, { "what": "mlx_lm raw numbers for gemma-4-12B-it-4bit via mlx_lm CLI", "reason": "mlx_lm 0.31.3 raises 'Model type gemma4_unified not supported'; measured with mlx_vlm 0.6.3 instead (recorded in raw.mlx_lm.tool_note)." }, { "what": "LM Studio qwen3.6-35b-a3b pair (both engines)", "reason": "Not measured. LM Studio setup itself succeeded, but adding this pair meant ~40 GB more downloads (separate GGUF + MLX artifacts in LM Studio's own store) and HF download throughput had collapsed to 1-4 MB/s at that point (the gemma MLX 5.3 GB shard alone took ~35 min). Exceeded the agreed timebox; gpt-oss-20b and gemma4-12b LM Studio pairs were completed instead." }, { "what": "LM Studio runtime cross-request prompt cache settings", "reason": "Engines were run at LM Studio defaults; no attempt was made to tune or explain the differing prompt-cache behavior between its llama.cpp and mlx engines." } ], "rerun_2026_08_12": { "note": "rapid-mlx column re-measured on the same machine with the same protocol, prompts and client after upgrading rapid-mlx 0.12.9 -> 0.12.11. All other engines were NOT re-run; their numbers remain the 2026-08-10/11 measurements at the versions in meta.versions. mlx/mlx-lm unchanged (0.31.2/0.31.3), so the delta is isolated to rapid-mlx itself.", "date": "2026-08-12", "versions": { "rapid_mlx": "0.12.11 (pip, same venv, Python 3.13.2)", "mlx": "0.31.2", "mlx_lm": "0.31.3" }, "machine_note": "same idle Mac mini; one unrelated interactive CLI session was resident during the run at ~25% of one CPU core, no GPU use; medians of 3 as before", "pairs": { "gpt-oss-20b": { "rapid_mlx": { "engine": "openai", "base": "http://127.0.0.1:8000", "model": "gpt-oss-20b", "warmup": { "ttft_ms": 930.9, "decode_tps": 38.99 }, "short": { "runs": [ { "ttft_ms": 143.0, "total_s": 0.61, "chunks": 20, "text_head": "Waves whisper secrets, \nSalt air kisses moonlit tides— \nOcean breathes, endless.", "text_tail": "Waves whisper secrets, \nSalt air kisses moonlit tides— \nOcean breathes, endless.", "usage": { "prompt_tokens": 84, "completion_tokens": 21, "total_tokens": 105, "prompt_tokens_details": { "cached_tokens": 84 } }, "completion_tokens": 21, "token_source": "usage.completion_tokens", "decode_tps": 42.81 }, { "ttft_ms": 130.7, "total_s": 0.679, "chunks": 20, "text_head": "Waves whisper secrets, \nSalt air kisses moonlit tides— \nOcean breathes, endless.", "text_tail": "Waves whisper secrets, \nSalt air kisses moonlit tides— \nOcean breathes, endless.", "usage": { "prompt_tokens": 84, "completion_tokens": 21, "total_tokens": 105, "prompt_tokens_details": { "cached_tokens": 84 } }, "completion_tokens": 21, "token_source": "usage.completion_tokens", "decode_tps": 36.45 }, { "ttft_ms": 178.4, "total_s": 0.602, "chunks": 20, "text_head": "Waves whisper secrets, \nSalt air kisses moonlit tides— \nOcean breathes, endless.", "text_tail": "Waves whisper secrets, \nSalt air kisses moonlit tides— \nOcean breathes, endless.", "usage": { "prompt_tokens": 84, "completion_tokens": 21, "total_tokens": 105, "prompt_tokens_details": { "cached_tokens": 84 } }, "completion_tokens": 21, "token_source": "usage.completion_tokens", "decode_tps": 47.21 } ], "ttft_ms": [ 143.0, 130.7, 178.4 ], "median_ttft_ms": 143.0, "decode_tps": [ 42.81, 36.45, 47.21 ], "median_decode_tps": 42.81 }, "long": { "runs": [ { "ttft_ms": 3980.7, "total_s": 7.923, "chunks": 156, "text_head": "The essay argues that the history of personal computing is better understood as a series of “collapses of distance” rather than mere shrinking machines. Each collapse—time‑sharing, the microprocessor,", "text_tail": "tion—will similarly reshape convenience and dependence, underscoring that every interface is also an interest and that the benefits of proximity are written by those who control the shortest distance.", "usage": { "prompt_tokens": 1053, "completion_tokens": 157, "total_tokens": 1210, "prompt_tokens_details": { "cached_tokens": 64 } }, "completion_tokens": 157, "token_source": "usage.completion_tokens", "decode_tps": 39.57 }, { "ttft_ms": 162.4, "total_s": 3.509, "chunks": 156, "text_head": "The essay argues that the history of personal computing is better understood as a series of “collapses of distance” rather than mere shrinking machines. Each collapse—time‑sharing, the microprocessor,", "text_tail": "tion—will similarly reshape convenience and dependence, underscoring that every interface is also an interest and that the benefits of proximity are written by those who control the shortest distance.", "usage": { "prompt_tokens": 1053, "completion_tokens": 157, "total_tokens": 1210, "prompt_tokens_details": { "cached_tokens": 1053 } }, "completion_tokens": 157, "token_source": "usage.completion_tokens", "decode_tps": 46.61 }, { "ttft_ms": 167.4, "total_s": 3.518, "chunks": 156, "text_head": "The essay argues that the history of personal computing is better understood as a series of “collapses of distance” rather than mere shrinking machines. Each collapse—time‑sharing, the microprocessor,", "text_tail": "tion—will similarly reshape convenience and dependence, underscoring that every interface is also an interest and that the benefits of proximity are written by those who control the shortest distance.", "usage": { "prompt_tokens": 1053, "completion_tokens": 157, "total_tokens": 1210, "prompt_tokens_details": { "cached_tokens": 1053 } }, "completion_tokens": 157, "token_source": "usage.completion_tokens", "decode_tps": 46.57 } ], "ttft_ms": [ 3980.7, 162.4, 167.4 ], "median_ttft_ms": 167.4, "decode_tps": [ 39.57, 46.61, 46.57 ], "median_decode_tps": 46.57 } }, "rapid_mlx_thinking_enabled": { "engine": "openai", "base": "http://127.0.0.1:8000", "model": "gpt-oss-20b", "warmup": { "ttft_ms": 563.5, "decode_tps": 50.08 }, "short": { "runs": [ { "ttft_ms": 202.7, "total_s": 2.743, "chunks": 125, "text_head": "The user wants a haiku about the ocean. Haiku is 5-7-5 syllable structure. Should be about ocean. Provide a haiku. Ensure syllable count. Let's craft: \"Waves whisper secrets, / moonlit tide cradles th", "text_tail": "rets, / moonlit tide cradles the night, / salt breath in silence.\" Count syllables: Waves (1) whis-per (2) sec-rets (3) = 3? Wait \"waves\" 1, \"whisper\" 2, \"secrets\" 2? Actually \"secrets\" 2. So 1+2+2=55", "usage": { "prompt_tokens": 75, "completion_tokens": 128, "total_tokens": 203, "completion_tokens_details": { "reasoning_tokens": 128 }, "prompt_tokens_details": { "cached_tokens": 75 } }, "completion_tokens": 128, "token_source": "usage.completion_tokens", "decode_tps": 50.0 }, { "ttft_ms": 190.6, "total_s": 2.739, "chunks": 125, "text_head": "The user wants a haiku about the ocean. Haiku is 5-7-5 syllable structure. Should be about ocean. Provide a haiku. Ensure syllable count. Let's craft: \"Waves whisper secrets, / moonlit tide cradles th", "text_tail": "rets, / moonlit tide cradles the night, / salt breath in silence.\" Count syllables: Waves (1) whis-per (2) sec-rets (3) = 3? Wait \"waves\" 1, \"whisper\" 2, \"secrets\" 2? Actually \"secrets\" 2. So 1+2+2=55", "usage": { "prompt_tokens": 75, "completion_tokens": 128, "total_tokens": 203, "completion_tokens_details": { "reasoning_tokens": 128 }, "prompt_tokens_details": { "cached_tokens": 75 } }, "completion_tokens": 128, "token_source": "usage.completion_tokens", "decode_tps": 49.84 }, { "ttft_ms": 206.4, "total_s": 2.754, "chunks": 125, "text_head": "The user wants a haiku about the ocean. Haiku is 5-7-5 syllable structure. Should be about ocean. Provide a haiku. Ensure syllable count. Let's craft: \"Waves whisper secrets, / moonlit tide cradles th", "text_tail": "rets, / moonlit tide cradles the night, / salt breath in silence.\" Count syllables: Waves (1) whis-per (2) sec-rets (3) = 3? Wait \"waves\" 1, \"whisper\" 2, \"secrets\" 2? Actually \"secrets\" 2. So 1+2+2=55", "usage": { "prompt_tokens": 75, "completion_tokens": 128, "total_tokens": 203, "completion_tokens_details": { "reasoning_tokens": 128 }, "prompt_tokens_details": { "cached_tokens": 75 } }, "completion_tokens": 128, "token_source": "usage.completion_tokens", "decode_tps": 49.86 } ], "ttft_ms": [ 202.7, 190.6, 206.4 ], "median_ttft_ms": 202.7, "decode_tps": [ 50.0, 49.84, 49.86 ], "median_decode_tps": 49.86 }, "long": { "runs": [ { "ttft_ms": 3602.4, "total_s": 9.117, "chunks": 247, "text_head": "We need to summarize the essay in a single paragraph. The essay discusses five collapses of distance in computing: institutional to time-sharing, microprocessor, graphical interface, networking, smart", "text_tail": ", opaque mental models, surveillance economies, and platform monopolies. The author argues that the next collapse will close the gap between intention and execution, automating actions from high‑level", "usage": { "prompt_tokens": 1044, "completion_tokens": 256, "total_tokens": 1300, "completion_tokens_details": { "reasoning_tokens": 97 }, "prompt_tokens_details": { "cached_tokens": 64 } }, "completion_tokens": 256, "token_source": "usage.completion_tokens", "decode_tps": 46.24 }, { "ttft_ms": 228.5, "total_s": 5.69, "chunks": 247, "text_head": "We need to summarize the essay in a single paragraph. The essay discusses five collapses of distance in computing: institutional to time-sharing, microprocessor, graphical interface, networking, smart", "text_tail": ", opaque mental models, surveillance economies, and platform monopolies. The author argues that the next collapse will close the gap between intention and execution, automating actions from high‑level", "usage": { "prompt_tokens": 1044, "completion_tokens": 256, "total_tokens": 1300, "completion_tokens_details": { "reasoning_tokens": 97 }, "prompt_tokens_details": { "cached_tokens": 1044 } }, "completion_tokens": 256, "token_source": "usage.completion_tokens", "decode_tps": 46.69 }, { "ttft_ms": 235.2, "total_s": 5.698, "chunks": 247, "text_head": "We need to summarize the essay in a single paragraph. The essay discusses five collapses of distance in computing: institutional to time-sharing, microprocessor, graphical interface, networking, smart", "text_tail": ", opaque mental models, surveillance economies, and platform monopolies. The author argues that the next collapse will close the gap between intention and execution, automating actions from high‑level", "usage": { "prompt_tokens": 1044, "completion_tokens": 256, "total_tokens": 1300, "completion_tokens_details": { "reasoning_tokens": 97 }, "prompt_tokens_details": { "cached_tokens": 1044 } }, "completion_tokens": 256, "token_source": "usage.completion_tokens", "decode_tps": 46.68 } ], "ttft_ms": [ 3602.4, 228.5, 235.2 ], "median_ttft_ms": 235.2, "decode_tps": [ 46.24, 46.69, 46.68 ], "median_decode_tps": 46.68 } }, "rapid_mlx_coldprefill_probe": { "ttft_ms": 3630.2, "total_s": 6.916, "chunks": 153, "text_head": "The essay argues that the history of personal computing is better seen as a series of “collapses of distance” rather than just shrinking machines. Each collapse—time‑sharing, the microprocessor, graph", "text_tail": " behalf from a simple description—will similarly trade convenience for a new, hidden dependence, underscoring that every interface is also an interest shaped by whoever controls the shortest distance.", "usage": { "prompt_tokens": 1064, "completion_tokens": 154, "total_tokens": 1218, "prompt_tokens_details": { "cached_tokens": 64 } }, "completion_tokens": 154, "token_source": "usage.completion_tokens", "decode_tps": 46.57 } }, "qwen3.6-35b-a3b": { "rapid_mlx": { "engine": "openai", "base": "http://127.0.0.1:8000", "model": "qwen3.6-35b-4bit", "warmup": { "ttft_ms": 302.6, "decode_tps": 63.9 }, "short": { "runs": [ { "ttft_ms": 264.8, "total_s": 0.578, "chunks": 20, "text_head": "Blue waves crash and roar,\nSalt spray kisses sandy shores,\nEndless deep calls out.", "text_tail": "Blue waves crash and roar,\nSalt spray kisses sandy shores,\nEndless deep calls out.", "usage": { "prompt_tokens": 20, "completion_tokens": 21, "total_tokens": 41, "prompt_tokens_details": { "cached_tokens": 5 } }, "completion_tokens": 21, "token_source": "usage.completion_tokens", "decode_tps": 63.81 }, { "ttft_ms": 239.9, "total_s": 0.552, "chunks": 20, "text_head": "Blue waves crash and roar,\nSalt spray kisses sandy shores,\nEndless deep calls out.", "text_tail": "Blue waves crash and roar,\nSalt spray kisses sandy shores,\nEndless deep calls out.", "usage": { "prompt_tokens": 20, "completion_tokens": 21, "total_tokens": 41, "prompt_tokens_details": { "cached_tokens": 5 } }, "completion_tokens": 21, "token_source": "usage.completion_tokens", "decode_tps": 64.0 }, { "ttft_ms": 246.0, "total_s": 0.556, "chunks": 20, "text_head": "Blue waves crash and roar,\nSalt spray kisses sandy shores,\nEndless deep calls out.", "text_tail": "Blue waves crash and roar,\nSalt spray kisses sandy shores,\nEndless deep calls out.", "usage": { "prompt_tokens": 20, "completion_tokens": 21, "total_tokens": 41, "prompt_tokens_details": { "cached_tokens": 5 } }, "completion_tokens": 21, "token_source": "usage.completion_tokens", "decode_tps": 64.52 } ], "ttft_ms": [ 264.8, 239.9, 246.0 ], "median_ttft_ms": 246.0, "decode_tps": [ 63.81, 64.0, 64.52 ], "median_decode_tps": 64.0 }, "long": { "runs": [ { "ttft_ms": 3542.1, "total_s": 6.997, "chunks": 194, "text_head": "The essay argues that the history of personal computing is defined not by the shrinking size of machines, but by a series of \"collapses of distance\" that have progressively reduced the gap between hum", "text_tail": "thor concludes that every interface serves the interests of its owner, meaning that the terms of convenience are dictated by whoever controls the shortest distance between the user and the technology.", "usage": { "prompt_tokens": 1016, "completion_tokens": 195, "total_tokens": 1211 }, "completion_tokens": 195, "token_source": "usage.completion_tokens", "decode_tps": 56.15 }, { "ttft_ms": 393.4, "total_s": 3.59, "chunks": 186, "text_head": "The essay argues that the history of personal computing is defined not by shrinking hardware, but by a series of \"collapses of distance\" that have progressively reduced the gap between humans and tech", "text_tail": "tention and automated execution—the pattern suggests that every interface serves specific interests, and the true cost of convenience is a dependence that only becomes apparent after it is entrenched.", "usage": { "prompt_tokens": 1016, "completion_tokens": 187, "total_tokens": 1203, "prompt_tokens_details": { "cached_tokens": 1001 } }, "completion_tokens": 187, "token_source": "usage.completion_tokens", "decode_tps": 58.19 }, { "ttft_ms": 260.8, "total_s": 3.42, "chunks": 186, "text_head": "The essay argues that the history of personal computing is defined not by shrinking hardware, but by a series of \"collapses of distance\" that have progressively reduced the gap between humans and tech", "text_tail": "tention and automated execution—the pattern suggests that every interface serves specific interests, and the true cost of convenience is a dependence that only becomes apparent after it is entrenched.", "usage": { "prompt_tokens": 1016, "completion_tokens": 187, "total_tokens": 1203, "prompt_tokens_details": { "cached_tokens": 1001 } }, "completion_tokens": 187, "token_source": "usage.completion_tokens", "decode_tps": 58.88 } ], "ttft_ms": [ 3542.1, 393.4, 260.8 ], "median_ttft_ms": 393.4, "decode_tps": [ 56.15, 58.19, 58.88 ], "median_decode_tps": 58.19 } }, "rapid_mlx_thinking_enabled": { "engine": "openai", "base": "http://127.0.0.1:8000", "model": "qwen3.6-35b-4bit", "warmup": { "ttft_ms": 230.6, "decode_tps": 60.38 }, "short": { "runs": [ { "ttft_ms": 233.1, "total_s": 2.329, "chunks": 128, "text_head": "Thinking Process:\n\n1. **Deconstruct the request:**\n * Topic: Ocean.\n * Form: Haiku (5-7-5 syllables).\n\n2. **Brainstorming imagery and themes:**\n * Waves, blue, deep, salt, sand, tides", "text_tail": "ss, crashing, calm, fish, horizon.\n * Keywords: Blue, waves, crash, deep, sea, salt, shore, endless.\n\n3. **Drafting lines (Syllable counting):**\n * *Attempt 1:*\n * Blue waves waves", "usage": { "prompt_tokens": 18, "completion_tokens": 128, "total_tokens": 146, "completion_tokens_details": { "reasoning_tokens": 64 }, "prompt_tokens_details": { "cached_tokens": 5 } }, "completion_tokens": 128, "token_source": "usage.completion_tokens", "decode_tps": 60.58 }, { "ttft_ms": 245.8, "total_s": 2.36, "chunks": 128, "text_head": "Thinking Process:\n\n1. **Deconstruct the request:**\n * Topic: Ocean.\n * Form: Haiku (5-7-5 syllables).\n\n2. **Brainstorming imagery and themes:**\n * Waves, blue, deep, salt, sand, tides", "text_tail": "ss, crashing, calm, fish, horizon.\n * Keywords: Blue, waves, crash, deep, sea, salt, shore, endless.\n\n3. **Drafting lines (Syllable counting):**\n * *Attempt 1:*\n * Blue waves waves", "usage": { "prompt_tokens": 18, "completion_tokens": 128, "total_tokens": 146, "completion_tokens_details": { "reasoning_tokens": 64 }, "prompt_tokens_details": { "cached_tokens": 5 } }, "completion_tokens": 128, "token_source": "usage.completion_tokens", "decode_tps": 60.06 }, { "ttft_ms": 253.1, "total_s": 2.331, "chunks": 128, "text_head": "Thinking Process:\n\n1. **Deconstruct the request:**\n * Topic: Ocean.\n * Form: Haiku (5-7-5 syllables).\n\n2. **Brainstorming imagery and themes:**\n * Waves, blue, deep, salt, sand, tides", "text_tail": "ss, crashing, calm, fish, horizon.\n * Keywords: Blue, waves, crash, deep, sea, salt, shore, endless.\n\n3. **Drafting lines (Syllable counting):**\n * *Attempt 1:*\n * Blue waves waves", "usage": { "prompt_tokens": 18, "completion_tokens": 128, "total_tokens": 146, "completion_tokens_details": { "reasoning_tokens": 64 }, "prompt_tokens_details": { "cached_tokens": 5 } }, "completion_tokens": 128, "token_source": "usage.completion_tokens", "decode_tps": 61.11 } ], "ttft_ms": [ 233.1, 245.8, 253.1 ], "median_ttft_ms": 245.8, "decode_tps": [ 60.58, 60.06, 61.11 ], "median_decode_tps": 60.58 }, "long": { "runs": [ { "ttft_ms": 260.2, "total_s": 4.606, "chunks": 256, "text_head": "Here's a thinking process:\n\n1. **Analyze User Input:**\n - **Input Text:** A detailed essay about the history of personal computing, framed as a series of \"collapses of distance\" (institutional, tim", "text_tail": "raphical UI (cognitive accessibility)\n 4. Isolated machine → Network/Internet (connectivity)\n 5. Desktop → Smartphone (corporeal/continuous computing)\n - Pattern: Each collapse brings brings", "usage": { "prompt_tokens": 1014, "completion_tokens": 256, "total_tokens": 1270, "completion_tokens_details": { "reasoning_tokens": 128 }, "prompt_tokens_details": { "cached_tokens": 1001 } }, "completion_tokens": 256, "token_source": "usage.completion_tokens", "decode_tps": 58.68 }, { "ttft_ms": 270.3, "total_s": 4.63, "chunks": 256, "text_head": "Here's a thinking process:\n\n1. **Analyze User Input:**\n - **Input Text:** A detailed essay about the history of personal computing, framed as a series of \"collapses of distance\" (institutional, tim", "text_tail": "raphical UI (cognitive accessibility)\n 4. Isolated machine → Network/Internet (connectivity)\n 5. Desktop → Smartphone (corporeal/continuous computing)\n - Pattern: Each collapse brings brings", "usage": { "prompt_tokens": 1014, "completion_tokens": 256, "total_tokens": 1270, "completion_tokens_details": { "reasoning_tokens": 128 }, "prompt_tokens_details": { "cached_tokens": 1001 } }, "completion_tokens": 256, "token_source": "usage.completion_tokens", "decode_tps": 58.49 }, { "ttft_ms": 257.5, "total_s": 4.646, "chunks": 256, "text_head": "Here's a thinking process:\n\n1. **Analyze User Input:**\n - **Input Text:** A detailed essay about the history of personal computing, framed as a series of \"collapses of distance\" (institutional, tim", "text_tail": "raphical UI (cognitive accessibility)\n 4. Isolated machine → Network/Internet (connectivity)\n 5. Desktop → Smartphone (corporeal/continuous computing)\n - Pattern: Each collapse brings brings", "usage": { "prompt_tokens": 1014, "completion_tokens": 256, "total_tokens": 1270, "completion_tokens_details": { "reasoning_tokens": 128 }, "prompt_tokens_details": { "cached_tokens": 1001 } }, "completion_tokens": 256, "token_source": "usage.completion_tokens", "decode_tps": 58.1 } ], "ttft_ms": [ 260.2, 270.3, 257.5 ], "median_ttft_ms": 260.2, "decode_tps": [ 58.68, 58.49, 58.1 ], "median_decode_tps": 58.49 } }, "rapid_mlx_coldprefill_probe": { "ttft_ms": 3371.8, "total_s": 6.902, "chunks": 205, "text_head": "The history of personal computing is characterized not by the shrinking size of machines, but by a series of five \"collapses of distance\" that progressively reduced the gap between human intention and", "text_tail": "ing on behalf of users—proceeds, we must recognize that convenience is never neutral; rather, every interface serves the interests of those who control the shortest distance between desire and action.", "usage": { "prompt_tokens": 1034, "completion_tokens": 206, "total_tokens": 1240 }, "completion_tokens": 206, "token_source": "usage.completion_tokens", "decode_tps": 58.07 } }, "gemma4-12b-it": { "rapid_mlx": { "engine": "openai", "base": "http://127.0.0.1:8000", "model": "gemma-4-12b-4bit", "warmup": { "ttft_ms": 604.3, "decode_tps": 24.09 }, "short": { "runs": [ { "ttft_ms": 208.2, "total_s": 1.133, "chunks": 22, "text_head": "Blue waves kiss the shore,\nDeep and vast and endless blue,\nTides pull at the sand.", "text_tail": "Blue waves kiss the shore,\nDeep and vast and endless blue,\nTides pull at the sand.", "usage": { "prompt_tokens": 21, "completion_tokens": 23, "total_tokens": 44, "prompt_tokens_details": { "cached_tokens": 21 } }, "completion_tokens": 23, "token_source": "usage.completion_tokens", "decode_tps": 23.79 }, { "ttft_ms": 199.8, "total_s": 1.129, "chunks": 22, "text_head": "Blue waves kiss the shore,\nDeep and vast and endless blue,\nTides pull at the sand.", "text_tail": "Blue waves kiss the shore,\nDeep and vast and endless blue,\nTides pull at the sand.", "usage": { "prompt_tokens": 21, "completion_tokens": 23, "total_tokens": 44, "prompt_tokens_details": { "cached_tokens": 21 } }, "completion_tokens": 23, "token_source": "usage.completion_tokens", "decode_tps": 23.68 }, { "ttft_ms": 205.9, "total_s": 1.135, "chunks": 22, "text_head": "Blue waves kiss the shore,\nDeep and vast and endless blue,\nTides pull at the sand.", "text_tail": "Blue waves kiss the shore,\nDeep and vast and endless blue,\nTides pull at the sand.", "usage": { "prompt_tokens": 21, "completion_tokens": 23, "total_tokens": 44, "prompt_tokens_details": { "cached_tokens": 21 } }, "completion_tokens": 23, "token_source": "usage.completion_tokens", "decode_tps": 23.67 } ], "ttft_ms": [ 208.2, 199.8, 205.9 ], "median_ttft_ms": 205.9, "decode_tps": [ 23.79, 23.68, 23.67 ], "median_decode_tps": 23.68 }, "long": { "runs": [ { "ttft_ms": 9379.0, "total_s": 16.329, "chunks": 147, "text_head": "The history of personal computing is defined by a series of \"collapses of distance\" that transformed the computer from a remote, institutional monument into an intimate, ubiquitous companion. Each sta", "text_tail": "chine, the essay argues that the current era of closing the gap between human intention and machine execution will likely offer immense convenience at the cost of a new, entrenched form of dependency.", "usage": { "prompt_tokens": 1010, "completion_tokens": 148, "total_tokens": 1158, "prompt_tokens_details": { "cached_tokens": 4 } }, "completion_tokens": 148, "token_source": "usage.completion_tokens", "decode_tps": 21.15 }, { "ttft_ms": 246.2, "total_s": 7.209, "chunks": 147, "text_head": "The history of personal computing is defined by a series of \"collapses of distance\" that transformed the computer from a remote, institutional monument into an intimate, ubiquitous companion. Each sta", "text_tail": "chine, the essay argues that the current era of closing the gap between human intention and machine execution will likely offer immense convenience at the cost of a new, entrenched form of dependency.", "usage": { "prompt_tokens": 1010, "completion_tokens": 148, "total_tokens": 1158, "prompt_tokens_details": { "cached_tokens": 1010 } }, "completion_tokens": 148, "token_source": "usage.completion_tokens", "decode_tps": 21.11 }, { "ttft_ms": 228.9, "total_s": 7.173, "chunks": 147, "text_head": "The history of personal computing is defined by a series of \"collapses of distance\" that transformed the computer from a remote, institutional monument into an intimate, ubiquitous companion. Each sta", "text_tail": "chine, the essay argues that the current era of closing the gap between human intention and machine execution will likely offer immense convenience at the cost of a new, entrenched form of dependency.", "usage": { "prompt_tokens": 1010, "completion_tokens": 148, "total_tokens": 1158, "prompt_tokens_details": { "cached_tokens": 1010 } }, "completion_tokens": 148, "token_source": "usage.completion_tokens", "decode_tps": 21.17 } ], "ttft_ms": [ 9379.0, 246.2, 228.9 ], "median_ttft_ms": 246.2, "decode_tps": [ 21.15, 21.11, 21.17 ], "median_decode_tps": 21.15 } }, "rapid_mlx_thinking_enabled": { "engine": "openai", "base": "http://127.0.0.1:8000", "model": "gemma-4-12b-4bit", "warmup": { "ttft_ms": 591.1, "decode_tps": 22.79 }, "short": { "runs": [ { "ttft_ms": 280.6, "total_s": 5.813, "chunks": 126, "text_head": "\n* Topic: The ocean.\n * Format: Haiku (5-7-5 syllables).\n\n * Waves, blue, deep, tides, salt, sand, endless, water, moon, shore, waves crashing, blue expanse.\n\n * Blue waves hit the sh", "text_tail": "waves hit the shore (5)\n * The water is very deep (7)\n * Salt is in the air (5)\n * *Critique:* A bit plain.\n\n * Endless blue expanse (5)\n * Waves dance on the sandy shore (7))", "usage": { "prompt_tokens": 24, "completion_tokens": 128, "total_tokens": 152, "completion_tokens_details": { "reasoning_tokens": 128 }, "prompt_tokens_details": { "cached_tokens": 24 } }, "completion_tokens": 128, "token_source": "usage.completion_tokens", "decode_tps": 22.96 }, { "ttft_ms": 300.9, "total_s": 5.906, "chunks": 126, "text_head": "\n* Topic: The ocean.\n * Format: Haiku (5-7-5 syllables).\n\n * Waves, blue, deep, tides, salt, sand, endless, water, moon, shore, waves crashing, blue expanse.\n\n * Blue waves hit the sh", "text_tail": "waves hit the shore (5)\n * The water is very deep (7)\n * Salt is in the air (5)\n * *Critique:* A bit plain.\n\n * Endless blue expanse (5)\n * Waves dance on the sandy shore (7))", "usage": { "prompt_tokens": 24, "completion_tokens": 128, "total_tokens": 152, "completion_tokens_details": { "reasoning_tokens": 128 }, "prompt_tokens_details": { "cached_tokens": 24 } }, "completion_tokens": 128, "token_source": "usage.completion_tokens", "decode_tps": 22.66 }, { "ttft_ms": 306.2, "total_s": 5.91, "chunks": 126, "text_head": "\n* Topic: The ocean.\n * Format: Haiku (5-7-5 syllables).\n\n * Waves, blue, deep, tides, salt, sand, endless, water, moon, shore, waves crashing, blue expanse.\n\n * Blue waves hit the sh", "text_tail": "waves hit the shore (5)\n * The water is very deep (7)\n * Salt is in the air (5)\n * *Critique:* A bit plain.\n\n * Endless blue expanse (5)\n * Waves dance on the sandy shore (7))", "usage": { "prompt_tokens": 24, "completion_tokens": 128, "total_tokens": 152, "completion_tokens_details": { "reasoning_tokens": 128 }, "prompt_tokens_details": { "cached_tokens": 24 } }, "completion_tokens": 128, "token_source": "usage.completion_tokens", "decode_tps": 22.66 } ], "ttft_ms": [ 280.6, 300.9, 306.2 ], "median_ttft_ms": 300.9, "decode_tps": [ 22.96, 22.66, 22.66 ], "median_decode_tps": 22.66 }, "long": { "runs": [ { "ttft_ms": 9447.2, "total_s": 21.494, "chunks": 254, "text_head": "\n\n* Topic: The history of personal computing as a series of \"shrinking distances.\"\n * Key Points:\n * Initial state: Institutional distance (computers as civic monuments).\n * 1", "text_tail": " computing is about shrinking distances between people and machines. It lists five stages: time-sharing (interactivity), microprocessors (ownership), graphical interfaces (ease of use), networking ( (", "usage": { "prompt_tokens": 1013, "completion_tokens": 256, "total_tokens": 1269, "completion_tokens_details": { "reasoning_tokens": 256 }, "prompt_tokens_details": { "cached_tokens": 11 } }, "completion_tokens": 256, "token_source": "usage.completion_tokens", "decode_tps": 21.17 }, { "ttft_ms": 329.9, "total_s": 12.296, "chunks": 254, "text_head": "\n\n* Topic: The history of personal computing as a series of \"shrinking distances.\"\n * Key Points:\n * Initial state: Institutional distance (computers as civic monuments).\n * 1", "text_tail": " computing is about shrinking distances between people and machines. It lists five stages: time-sharing (interactivity), microprocessors (ownership), graphical interfaces (ease of use), networking ( (", "usage": { "prompt_tokens": 1013, "completion_tokens": 256, "total_tokens": 1269, "completion_tokens_details": { "reasoning_tokens": 256 }, "prompt_tokens_details": { "cached_tokens": 1013 } }, "completion_tokens": 256, "token_source": "usage.completion_tokens", "decode_tps": 21.31 }, { "ttft_ms": 311.4, "total_s": 12.28, "chunks": 254, "text_head": "\n\n* Topic: The history of personal computing as a series of \"shrinking distances.\"\n * Key Points:\n * Initial state: Institutional distance (computers as civic monuments).\n * 1", "text_tail": " computing is about shrinking distances between people and machines. It lists five stages: time-sharing (interactivity), microprocessors (ownership), graphical interfaces (ease of use), networking ( (", "usage": { "prompt_tokens": 1013, "completion_tokens": 256, "total_tokens": 1269, "completion_tokens_details": { "reasoning_tokens": 256 }, "prompt_tokens_details": { "cached_tokens": 1013 } }, "completion_tokens": 256, "token_source": "usage.completion_tokens", "decode_tps": 21.31 } ], "ttft_ms": [ 9447.2, 329.9, 311.4 ], "median_ttft_ms": 329.9, "decode_tps": [ 21.17, 21.31, 21.31 ], "median_decode_tps": 21.31 } }, "rapid_mlx_coldprefill_probe": { "ttft_ms": 9340.1, "total_s": 16.131, "chunks": 145, "text_head": "The essay argues that the history of personal computing is defined by a series of \"collapses of distance\" that transformed the machine from a remote, institutional monument into an intimate, ubiquitou", "text_tail": "ntention and execution, warning that while this progress offers immense convenience, the terms of that convenience are ultimately dictated by those who control the infrastructure of the new proximity.", "usage": { "prompt_tokens": 1028, "completion_tokens": 146, "total_tokens": 1174, "prompt_tokens_details": { "cached_tokens": 4 } }, "completion_tokens": 146, "token_source": "usage.completion_tokens", "decode_tps": 21.35 } } } }, "concurrency_2026_08_12": { "note": "Concurrency lane, added 2026-08-12: 4 and 8 simultaneous streaming requests, each the ~1K-token LONG prompt with a unique salt prefix (prefix-cache busting), 256-token budget, temperature 0. One discarded single warmup + one discarded concurrent warmup batch, then 3 timed batches; medians reported. Parallelism pinned per lane and servers restarted between lanes: OLLAMA_NUM_PARALLEL=N; llama-server --parallel N -c N*4096; rapid-mlx default admission. rapid-mlx ran with enable_thinking=true to match the long-form decode table. ALL engines measured fresh this day: rapid-mlx 0.12.11, Ollama 0.32.7, llama.cpp build 10330 (same versions as the rest of this file). wall_agg_tps = sum(tokens)/batch wall incl. prefill; decode_agg_tps = sum(tokens-1)/global chunk window.", "date": "2026-08-12", "cells": { "gemma-llamaserver-conc4": { "engine": "openai", "base": "http://127.0.0.1:8081", "model": "default", "conc": 4, "enable_thinking": false, "batches": [ { "streams": [ { "idx": 0, "completion_tokens": 256, "token_source": "usage.completion_tokens", "error": null, "ttft_ms": 25683.9, "t_first_rel_s": 25.684, "t_last_rel_s": 61.789, "decode_tps": 7.06 }, { "idx": 1, "completion_tokens": 256, "token_source": "usage.completion_tokens", "error": null, "ttft_ms": 25684.4, "t_first_rel_s": 25.684, "t_last_rel_s": 61.79, "decode_tps": 7.06 }, { "idx": 2, "completion_tokens": 256, "token_source": "usage.completion_tokens", "error": null, "ttft_ms": 25684.1, "t_first_rel_s": 25.684, "t_last_rel_s": 61.789, "decode_tps": 7.06 }, { "idx": 3, "completion_tokens": 256, "token_source": "usage.completion_tokens", "error": null, "ttft_ms": 25684.7, "t_first_rel_s": 25.685, "t_last_rel_s": 61.79, "decode_tps": 7.06 } ], "n_ok": 4, "n_err": 0, "wall_s": 61.79, "total_tokens": 1024, "wall_agg_tps": 16.57, "decode_agg_tps": 28.25 }, { "streams": [ { "idx": 0, "completion_tokens": 256, "token_source": "usage.completion_tokens", "error": null, "ttft_ms": 26027.5, "t_first_rel_s": 26.028, "t_last_rel_s": 62.219, "decode_tps": 7.05 }, { "idx": 1, "completion_tokens": 256, "token_source": "usage.completion_tokens", "error": null, "ttft_ms": 26024.9, "t_first_rel_s": 26.025, "t_last_rel_s": 62.218, "decode_tps": 7.05 }, { "idx": 2, "completion_tokens": 256, "token_source": "usage.completion_tokens", "error": null, "ttft_ms": 26026.6, "t_first_rel_s": 26.027, "t_last_rel_s": 62.218, "decode_tps": 7.05 }, { "idx": 3, "completion_tokens": 256, "token_source": "usage.completion_tokens", "error": null, "ttft_ms": 26025.6, "t_first_rel_s": 26.026, "t_last_rel_s": 62.218, "decode_tps": 7.05 } ], "n_ok": 4, "n_err": 0, "wall_s": 62.219, "total_tokens": 1024, "wall_agg_tps": 16.46, "decode_agg_tps": 28.18 }, { "streams": [ { "idx": 0, "completion_tokens": 256, "token_source": "usage.completion_tokens", "error": null, "ttft_ms": 25515.7, "t_first_rel_s": 25.516, "t_last_rel_s": 61.451, "decode_tps": 7.1 }, { "idx": 1, "completion_tokens": 256, "token_source": "usage.completion_tokens", "error": null, "ttft_ms": 25515.4, "t_first_rel_s": 25.515, "t_last_rel_s": 61.45, "decode_tps": 7.1 }, { "idx": 2, "completion_tokens": 256, "token_source": "usage.completion_tokens", "error": null, "ttft_ms": 25514.9, "t_first_rel_s": 25.515, "t_last_rel_s": 61.448, "decode_tps": 7.1 }, { "idx": 3, "completion_tokens": 256, "token_source": "usage.completion_tokens", "error": null, "ttft_ms": 25515.1, "t_first_rel_s": 25.515, "t_last_rel_s": 61.449, "decode_tps": 7.1 } ], "n_ok": 4, "n_err": 0, "wall_s": 61.451, "total_tokens": 1024, "wall_agg_tps": 16.66, "decode_agg_tps": 28.38 } ], "median_wall_agg_tps": 16.57, "median_decode_agg_tps": 28.25, "all_streams_ok": true }, "gemma-llamaserver-conc8": { "engine": "openai", "base": "http://127.0.0.1:8081", "model": "default", "conc": 8, "enable_thinking": false, "batches": [ { "streams": [ { "idx": 0, "completion_tokens": 256, "token_source": "usage.completion_tokens", "error": null, "ttft_ms": 54786.4, "t_first_rel_s": 54.786, "t_last_rel_s": 120.319, "decode_tps": 3.89 }, { "idx": 1, "completion_tokens": 256, "token_source": "usage.completion_tokens", "error": null, "ttft_ms": 54262.3, "t_first_rel_s": 54.262, "t_last_rel_s": 120.028, "decode_tps": 3.88 }, { "idx": 2, "completion_tokens": 256, "token_source": "usage.completion_tokens", "error": null, "ttft_ms": 54262.0, "t_first_rel_s": 54.262, "t_last_rel_s": 120.027, "decode_tps": 3.88 }, { "idx": 3, "completion_tokens": 256, "token_source": "usage.completion_tokens", "error": null, "ttft_ms": 54786.0, "t_first_rel_s": 54.786, "t_last_rel_s": 120.319, "decode_tps": 3.89 }, { "idx": 4, "completion_tokens": 256, "token_source": "usage.completion_tokens", "error": null, "ttft_ms": 54786.7, "t_first_rel_s": 54.787, "t_last_rel_s": 120.319, "decode_tps": 3.89 }, { "idx": 5, "completion_tokens": 256, "token_source": "usage.completion_tokens", "error": null, "ttft_ms": 54262.6, "t_first_rel_s": 54.263, "t_last_rel_s": 120.028, "decode_tps": 3.88 }, { "idx": 6, "completion_tokens": 256, "token_source": "usage.completion_tokens", "error": null, "ttft_ms": 54261.8, "t_first_rel_s": 54.262, "t_last_rel_s": 120.026, "decode_tps": 3.88 }, { "idx": 7, "completion_tokens": 256, "token_source": "usage.completion_tokens", "error": null, "ttft_ms": 55047.5, "t_first_rel_s": 55.047, "t_last_rel_s": 120.378, "decode_tps": 3.9 } ], "n_ok": 8, "n_err": 0, "wall_s": 120.378, "total_tokens": 2048, "wall_agg_tps": 17.01, "decode_agg_tps": 30.85 }, { "streams": [ { "idx": 0, "completion_tokens": 256, "token_source": "usage.completion_tokens", "error": null, "ttft_ms": 56417.9, "t_first_rel_s": 56.418, "t_last_rel_s": 121.893, "decode_tps": 3.89 }, { "idx": 1, "completion_tokens": 256, "token_source": "usage.completion_tokens", "error": null, "ttft_ms": 56416.8, "t_first_rel_s": 56.417, "t_last_rel_s": 121.892, "decode_tps": 3.89 }, { "idx": 2, "completion_tokens": 256, "token_source": "usage.completion_tokens", "error": null, "ttft_ms": 55881.8, "t_first_rel_s": 55.882, "t_last_rel_s": 121.606, "decode_tps": 3.88 }, { "idx": 3, "completion_tokens": 256, "token_source": "usage.completion_tokens", "error": null, "ttft_ms": 55881.2, "t_first_rel_s": 55.881, "t_last_rel_s": 121.606, "decode_tps": 3.88 }, { "idx": 4, "completion_tokens": 256, "token_source": "usage.completion_tokens", "error": null, "ttft_ms": 56686.7, "t_first_rel_s": 56.687, "t_last_rel_s": 121.951, "decode_tps": 3.91 }, { "idx": 5, "completion_tokens": 256, "token_source": "usage.completion_tokens", "error": null, "ttft_ms": 55881.5, "t_first_rel_s": 55.882, "t_last_rel_s": 121.606, "decode_tps": 3.88 }, { "idx": 6, "completion_tokens": 256, "token_source": "usage.completion_tokens", "error": null, "ttft_ms": 55880.9, "t_first_rel_s": 55.881, "t_last_rel_s": 121.6, "decode_tps": 3.88 }, { "idx": 7, "completion_tokens": 256, "token_source": "usage.completion_tokens", "error": null, "ttft_ms": 56417.3, "t_first_rel_s": 56.417, "t_last_rel_s": 121.893, "decode_tps": 3.89 } ], "n_ok": 8, "n_err": 0, "wall_s": 121.951, "total_tokens": 2048, "wall_agg_tps": 16.79, "decode_agg_tps": 30.88 }, { "streams": [ { "idx": 0, "completion_tokens": 256, "token_source": "usage.completion_tokens", "error": null, "ttft_ms": 62301.0, "t_first_rel_s": 62.301, "t_last_rel_s": 128.093, "decode_tps": 3.88 }, { "idx": 1, "completion_tokens": 256, "token_source": "usage.completion_tokens", "error": null, "ttft_ms": 63084.2, "t_first_rel_s": 63.084, "t_last_rel_s": 128.438, "decode_tps": 3.9 }, { "idx": 2, "completion_tokens": 256, "token_source": "usage.completion_tokens", "error": null, "ttft_ms": 62823.6, "t_first_rel_s": 62.824, "t_last_rel_s": 128.38, "decode_tps": 3.89 }, { "idx": 3, "completion_tokens": 256, "token_source": "usage.completion_tokens", "error": null, "ttft_ms": 62300.6, "t_first_rel_s": 62.301, "t_last_rel_s": 128.093, "decode_tps": 3.88 }, { "idx": 4, "completion_tokens": 256, "token_source": "usage.completion_tokens", "error": null, "ttft_ms": 62823.2, "t_first_rel_s": 62.823, "t_last_rel_s": 128.379, "decode_tps": 3.89 }, { "idx": 5, "completion_tokens": 256, "token_source": "usage.completion_tokens", "error": null, "ttft_ms": 62300.3, "t_first_rel_s": 62.3, "t_last_rel_s": 128.092, "decode_tps": 3.88 }, { "idx": 6, "completion_tokens": 256, "token_source": "usage.completion_tokens", "error": null, "ttft_ms": 62822.8, "t_first_rel_s": 62.823, "t_last_rel_s": 128.379, "decode_tps": 3.89 }, { "idx": 7, "completion_tokens": 256, "token_source": "usage.completion_tokens", "error": null, "ttft_ms": 62300.2, "t_first_rel_s": 62.3, "t_last_rel_s": 128.091, "decode_tps": 3.88 } ], "n_ok": 8, "n_err": 0, "wall_s": 128.438, "total_tokens": 2048, "wall_agg_tps": 15.95, "decode_agg_tps": 30.84 } ], "median_wall_agg_tps": 16.79, "median_decode_agg_tps": 30.85, "all_streams_ok": true }, "gemma-ollama-conc4": { "engine": "ollama", "base": "http://127.0.0.1:11434", "model": "gemma4:12b-it-q4_K_M", "conc": 4, "enable_thinking": false, "batches": [ { "streams": [ { "idx": 0, "completion_tokens": 256, "token_source": "ollama_eval_count", "error": null, "ttft_ms": 27251.6, "t_first_rel_s": 27.252, "t_last_rel_s": 65.234, "decode_tps": 6.71 }, { "idx": 1, "completion_tokens": 256, "token_source": "ollama_eval_count", "error": null, "ttft_ms": 27398.8, "t_first_rel_s": 27.399, "t_last_rel_s": 65.293, "decode_tps": 6.73 }, { "idx": 2, "completion_tokens": 256, "token_source": "ollama_eval_count", "error": null, "ttft_ms": 26957.7, "t_first_rel_s": 26.958, "t_last_rel_s": 64.972, "decode_tps": 6.71 }, { "idx": 3, "completion_tokens": 256, "token_source": "ollama_eval_count", "error": null, "ttft_ms": 27104.2, "t_first_rel_s": 27.104, "t_last_rel_s": 65.132, "decode_tps": 6.71 } ], "n_ok": 4, "n_err": 0, "wall_s": 65.293, "total_tokens": 1024, "wall_agg_tps": 15.68, "decode_agg_tps": 26.61 }, { "streams": [ { "idx": 0, "completion_tokens": 256, "token_source": "ollama_eval_count", "error": null, "ttft_ms": 27346.5, "t_first_rel_s": 27.346, "t_last_rel_s": 64.737, "decode_tps": 6.82 }, { "idx": 1, "completion_tokens": 256, "token_source": "ollama_eval_count", "error": null, "ttft_ms": 27045.1, "t_first_rel_s": 27.045, "t_last_rel_s": 64.477, "decode_tps": 6.81 }, { "idx": 2, "completion_tokens": 256, "token_source": "ollama_eval_count", "error": null, "ttft_ms": 27493.9, "t_first_rel_s": 27.494, "t_last_rel_s": 64.795, "decode_tps": 6.84 }, { "idx": 3, "completion_tokens": 256, "token_source": "ollama_eval_count", "error": null, "ttft_ms": 27193.5, "t_first_rel_s": 27.194, "t_last_rel_s": 64.635, "decode_tps": 6.81 } ], "n_ok": 4, "n_err": 0, "wall_s": 64.795, "total_tokens": 1024, "wall_agg_tps": 15.8, "decode_agg_tps": 27.02 }, { "streams": [ { "idx": 0, "completion_tokens": 256, "token_source": "ollama_eval_count", "error": null, "ttft_ms": 28082.7, "t_first_rel_s": 28.083, "t_last_rel_s": 64.971, "decode_tps": 6.91 }, { "idx": 1, "completion_tokens": 256, "token_source": "ollama_eval_count", "error": null, "ttft_ms": 27933.9, "t_first_rel_s": 27.934, "t_last_rel_s": 64.869, "decode_tps": 6.9 }, { "idx": 2, "completion_tokens": 256, "token_source": "ollama_eval_count", "error": null, "ttft_ms": 27787.1, "t_first_rel_s": 27.787, "t_last_rel_s": 64.711, "decode_tps": 6.91 }, { "idx": 3, "completion_tokens": 256, "token_source": "ollama_eval_count", "error": null, "ttft_ms": 28229.8, "t_first_rel_s": 28.23, "t_last_rel_s": 65.029, "decode_tps": 6.93 } ], "n_ok": 4, "n_err": 0, "wall_s": 65.029, "total_tokens": 1024, "wall_agg_tps": 15.75, "decode_agg_tps": 27.39 } ], "median_wall_agg_tps": 15.75, "median_decode_agg_tps": 27.02, "all_streams_ok": true }, "gemma-ollama-conc8": { "engine": "ollama", "base": "http://127.0.0.1:11434", "model": "gemma4:12b-it-q4_K_M", "conc": 8, "enable_thinking": false, "batches": [ { "streams": [ { "idx": 0, "completion_tokens": 256, "token_source": "ollama_eval_count", "error": null, "ttft_ms": 36566.7, "t_first_rel_s": 36.567, "t_last_rel_s": 126.161, "decode_tps": 2.85 }, { "idx": 1, "completion_tokens": 256, "token_source": "ollama_eval_count", "error": null, "ttft_ms": 58096.0, "t_first_rel_s": 58.096, "t_last_rel_s": 127.326, "decode_tps": 3.68 }, { "idx": 2, "completion_tokens": 256, "token_source": "ollama_eval_count", "error": null, "ttft_ms": 58364.1, "t_first_rel_s": 58.364, "t_last_rel_s": 127.446, "decode_tps": 3.69 }, { "idx": 3, "completion_tokens": 256, "token_source": "ollama_eval_count", "error": null, "ttft_ms": 58632.3, "t_first_rel_s": 58.632, "t_last_rel_s": 127.521, "decode_tps": 3.7 }, { "idx": 4, "completion_tokens": 256, "token_source": "ollama_eval_count", "error": null, "ttft_ms": 43214.9, "t_first_rel_s": 43.215, "t_last_rel_s": 126.444, "decode_tps": 3.06 }, { "idx": 5, "completion_tokens": 256, "token_source": "ollama_eval_count", "error": null, "ttft_ms": 49886.8, "t_first_rel_s": 49.887, "t_last_rel_s": 126.674, "decode_tps": 3.32 }, { "idx": 6, "completion_tokens": 256, "token_source": "ollama_eval_count", "error": null, "ttft_ms": 56699.7, "t_first_rel_s": 56.7, "t_last_rel_s": 126.853, "decode_tps": 3.63 }, { "idx": 7, "completion_tokens": 256, "token_source": "ollama_eval_count", "error": null, "ttft_ms": 57393.5, "t_first_rel_s": 57.394, "t_last_rel_s": 127.003, "decode_tps": 3.66 } ], "n_ok": 8, "n_err": 0, "wall_s": 127.521, "total_tokens": 2048, "wall_agg_tps": 16.06, "decode_agg_tps": 22.43 }, { "streams": [ { "idx": 0, "completion_tokens": 256, "token_source": "ollama_eval_count", "error": null, "ttft_ms": 56907.8, "t_first_rel_s": 56.908, "t_last_rel_s": 125.935, "decode_tps": 3.69 }, { "idx": 1, "completion_tokens": 256, "token_source": "ollama_eval_count", "error": null, "ttft_ms": 57789.5, "t_first_rel_s": 57.79, "t_last_rel_s": 126.085, "decode_tps": 3.73 }, { "idx": 2, "completion_tokens": 256, "token_source": "ollama_eval_count", "error": null, "ttft_ms": 59108.3, "t_first_rel_s": 59.108, "t_last_rel_s": 126.552, "decode_tps": 3.78 }, { "idx": 3, "completion_tokens": 256, "token_source": "ollama_eval_count", "error": null, "ttft_ms": 50122.2, "t_first_rel_s": 50.122, "t_last_rel_s": 125.755, "decode_tps": 3.37 }, { "idx": 4, "completion_tokens": 256, "token_source": "ollama_eval_count", "error": null, "ttft_ms": 58821.1, "t_first_rel_s": 58.821, "t_last_rel_s": 126.493, "decode_tps": 3.77 }, { "idx": 5, "completion_tokens": 256, "token_source": "ollama_eval_count", "error": null, "ttft_ms": 43441.2, "t_first_rel_s": 43.441, "t_last_rel_s": 125.54, "decode_tps": 3.11 }, { "idx": 6, "completion_tokens": 256, "token_source": "ollama_eval_count", "error": null, "ttft_ms": 36843.8, "t_first_rel_s": 36.844, "t_last_rel_s": 125.27, "decode_tps": 2.88 }, { "idx": 7, "completion_tokens": 256, "token_source": "ollama_eval_count", "error": null, "ttft_ms": 58553.4, "t_first_rel_s": 58.553, "t_last_rel_s": 126.389, "decode_tps": 3.76 } ], "n_ok": 8, "n_err": 0, "wall_s": 126.552, "total_tokens": 2048, "wall_agg_tps": 16.18, "decode_agg_tps": 22.74 }, { "streams": [ { "idx": 0, "completion_tokens": 256, "token_source": "ollama_eval_count", "error": null, "ttft_ms": 59945.1, "t_first_rel_s": 59.945, "t_last_rel_s": 128.09, "decode_tps": 3.74 }, { "idx": 1, "completion_tokens": 256, "token_source": "ollama_eval_count", "error": null, "ttft_ms": 43475.1, "t_first_rel_s": 43.475, "t_last_rel_s": 127.247, "decode_tps": 3.04 }, { "idx": 2, "completion_tokens": 256, "token_source": "ollama_eval_count", "error": null, "ttft_ms": 59180.1, "t_first_rel_s": 59.18, "t_last_rel_s": 127.786, "decode_tps": 3.72 }, { "idx": 3, "completion_tokens": 256, "token_source": "ollama_eval_count", "error": null, "ttft_ms": 60212.7, "t_first_rel_s": 60.213, "t_last_rel_s": 128.194, "decode_tps": 3.75 }, { "idx": 4, "completion_tokens": 256, "token_source": "ollama_eval_count", "error": null, "ttft_ms": 50884.8, "t_first_rel_s": 50.885, "t_last_rel_s": 127.459, "decode_tps": 3.33 }, { "idx": 5, "completion_tokens": 256, "token_source": "ollama_eval_count", "error": null, "ttft_ms": 36333.8, "t_first_rel_s": 36.334, "t_last_rel_s": 126.974, "decode_tps": 2.81 }, { "idx": 6, "completion_tokens": 256, "token_source": "ollama_eval_count", "error": null, "ttft_ms": 58297.1, "t_first_rel_s": 58.297, "t_last_rel_s": 127.638, "decode_tps": 3.68 }, { "idx": 7, "completion_tokens": 256, "token_source": "ollama_eval_count", "error": null, "ttft_ms": 60480.5, "t_first_rel_s": 60.481, "t_last_rel_s": 128.253, "decode_tps": 3.76 } ], "n_ok": 8, "n_err": 0, "wall_s": 128.253, "total_tokens": 2048, "wall_agg_tps": 15.97, "decode_agg_tps": 22.19 } ], "median_wall_agg_tps": 16.06, "median_decode_agg_tps": 22.43, "all_streams_ok": true }, "gemma-rapidmlx-conc4": { "engine": "openai", "base": "http://127.0.0.1:8000", "model": "gemma-4-12b-4bit", "conc": 4, "enable_thinking": true, "batches": [ { "streams": [ { "idx": 0, "completion_tokens": 256, "token_source": "usage.completion_tokens", "error": null, "ttft_ms": 37536.2, "t_first_rel_s": 37.536, "t_last_rel_s": 79.277, "decode_tps": 6.11 }, { "idx": 1, "completion_tokens": 256, "token_source": "usage.completion_tokens", "error": null, "ttft_ms": 37536.4, "t_first_rel_s": 37.536, "t_last_rel_s": 79.277, "decode_tps": 6.11 }, { "idx": 2, "completion_tokens": 256, "token_source": "usage.completion_tokens", "error": null, "ttft_ms": 37536.3, "t_first_rel_s": 37.536, "t_last_rel_s": 79.277, "decode_tps": 6.11 }, { "idx": 3, "completion_tokens": 256, "token_source": "usage.completion_tokens", "error": null, "ttft_ms": 37536.5, "t_first_rel_s": 37.537, "t_last_rel_s": 79.277, "decode_tps": 6.11 } ], "n_ok": 4, "n_err": 0, "wall_s": 79.277, "total_tokens": 1024, "wall_agg_tps": 12.92, "decode_agg_tps": 24.44 }, { "streams": [ { "idx": 0, "completion_tokens": 256, "token_source": "usage.completion_tokens", "error": null, "ttft_ms": 39203.0, "t_first_rel_s": 39.203, "t_last_rel_s": 81.028, "decode_tps": 6.1 }, { "idx": 1, "completion_tokens": 256, "token_source": "usage.completion_tokens", "error": null, "ttft_ms": 39203.0, "t_first_rel_s": 39.203, "t_last_rel_s": 81.028, "decode_tps": 6.1 }, { "idx": 2, "completion_tokens": 256, "token_source": "usage.completion_tokens", "error": null, "ttft_ms": 39202.9, "t_first_rel_s": 39.203, "t_last_rel_s": 81.028, "decode_tps": 6.1 }, { "idx": 3, "completion_tokens": 256, "token_source": "usage.completion_tokens", "error": null, "ttft_ms": 39203.0, "t_first_rel_s": 39.203, "t_last_rel_s": 81.028, "decode_tps": 6.1 } ], "n_ok": 4, "n_err": 0, "wall_s": 81.028, "total_tokens": 1024, "wall_agg_tps": 12.64, "decode_agg_tps": 24.39 }, { "streams": [ { "idx": 0, "completion_tokens": 256, "token_source": "usage.completion_tokens", "error": null, "ttft_ms": 38141.5, "t_first_rel_s": 38.142, "t_last_rel_s": 80.351, "decode_tps": 6.04 }, { "idx": 1, "completion_tokens": 256, "token_source": "usage.completion_tokens", "error": null, "ttft_ms": 38141.6, "t_first_rel_s": 38.142, "t_last_rel_s": 80.351, "decode_tps": 6.04 }, { "idx": 2, "completion_tokens": 256, "token_source": "usage.completion_tokens", "error": null, "ttft_ms": 38141.6, "t_first_rel_s": 38.142, "t_last_rel_s": 80.351, "decode_tps": 6.04 }, { "idx": 3, "completion_tokens": 256, "token_source": "usage.completion_tokens", "error": null, "ttft_ms": 38141.7, "t_first_rel_s": 38.142, "t_last_rel_s": 80.351, "decode_tps": 6.04 } ], "n_ok": 4, "n_err": 0, "wall_s": 80.351, "total_tokens": 1024, "wall_agg_tps": 12.74, "decode_agg_tps": 24.17 } ], "median_wall_agg_tps": 12.74, "median_decode_agg_tps": 24.39, "all_streams_ok": true }, "gemma-rapidmlx-conc8": { "engine": "openai", "base": "http://127.0.0.1:8000", "model": "gemma-4-12b-4bit", "conc": 8, "enable_thinking": true, "batches": [ { "streams": [ { "idx": 0, "completion_tokens": 256, "token_source": "usage.completion_tokens", "error": null, "ttft_ms": 84994.3, "t_first_rel_s": 84.994, "t_last_rel_s": 174.462, "decode_tps": 2.85 }, { "idx": 1, "completion_tokens": 256, "token_source": "usage.completion_tokens", "error": null, "ttft_ms": 84994.8, "t_first_rel_s": 84.995, "t_last_rel_s": 174.462, "decode_tps": 2.85 }, { "idx": 2, "completion_tokens": 256, "token_source": "usage.completion_tokens", "error": null, "ttft_ms": 84994.8, "t_first_rel_s": 84.995, "t_last_rel_s": 174.462, "decode_tps": 2.85 }, { "idx": 3, "completion_tokens": 256, "token_source": "usage.completion_tokens", "error": null, "ttft_ms": 84994.9, "t_first_rel_s": 84.995, "t_last_rel_s": 174.462, "decode_tps": 2.85 }, { "idx": 4, "completion_tokens": 256, "token_source": "usage.completion_tokens", "error": null, "ttft_ms": 84994.7, "t_first_rel_s": 84.995, "t_last_rel_s": 174.462, "decode_tps": 2.85 }, { "idx": 5, "completion_tokens": 256, "token_source": "usage.completion_tokens", "error": null, "ttft_ms": 84994.9, "t_first_rel_s": 84.995, "t_last_rel_s": 174.462, "decode_tps": 2.85 }, { "idx": 6, "completion_tokens": 256, "token_source": "usage.completion_tokens", "error": null, "ttft_ms": 84994.8, "t_first_rel_s": 84.995, "t_last_rel_s": 174.462, "decode_tps": 2.85 }, { "idx": 7, "completion_tokens": 256, "token_source": "usage.completion_tokens", "error": null, "ttft_ms": 84994.8, "t_first_rel_s": 84.995, "t_last_rel_s": 174.462, "decode_tps": 2.85 } ], "n_ok": 8, "n_err": 0, "wall_s": 174.462, "total_tokens": 2048, "wall_agg_tps": 11.74, "decode_agg_tps": 22.8 }, { "streams": [ { "idx": 0, "completion_tokens": 256, "token_source": "usage.completion_tokens", "error": null, "ttft_ms": 74373.3, "t_first_rel_s": 74.373, "t_last_rel_s": 165.765, "decode_tps": 2.79 }, { "idx": 1, "completion_tokens": 256, "token_source": "usage.completion_tokens", "error": null, "ttft_ms": 74373.6, "t_first_rel_s": 74.374, "t_last_rel_s": 165.765, "decode_tps": 2.79 }, { "idx": 2, "completion_tokens": 256, "token_source": "usage.completion_tokens", "error": null, "ttft_ms": 74373.7, "t_first_rel_s": 74.374, "t_last_rel_s": 165.765, "decode_tps": 2.79 }, { "idx": 3, "completion_tokens": 256, "token_source": "usage.completion_tokens", "error": null, "ttft_ms": 74373.7, "t_first_rel_s": 74.374, "t_last_rel_s": 165.765, "decode_tps": 2.79 }, { "idx": 4, "completion_tokens": 256, "token_source": "usage.completion_tokens", "error": null, "ttft_ms": 74373.9, "t_first_rel_s": 74.374, "t_last_rel_s": 165.765, "decode_tps": 2.79 }, { "idx": 5, "completion_tokens": 256, "token_source": "usage.completion_tokens", "error": null, "ttft_ms": 74373.8, "t_first_rel_s": 74.374, "t_last_rel_s": 165.765, "decode_tps": 2.79 }, { "idx": 6, "completion_tokens": 256, "token_source": "usage.completion_tokens", "error": null, "ttft_ms": 74373.8, "t_first_rel_s": 74.374, "t_last_rel_s": 165.765, "decode_tps": 2.79 }, { "idx": 7, "completion_tokens": 256, "token_source": "usage.completion_tokens", "error": null, "ttft_ms": 74373.9, "t_first_rel_s": 74.374, "t_last_rel_s": 165.765, "decode_tps": 2.79 } ], "n_ok": 8, "n_err": 0, "wall_s": 165.765, "total_tokens": 2048, "wall_agg_tps": 12.35, "decode_agg_tps": 22.32 }, { "streams": [ { "idx": 0, "completion_tokens": 256, "token_source": "usage.completion_tokens", "error": null, "ttft_ms": 76129.8, "t_first_rel_s": 76.13, "t_last_rel_s": 166.48, "decode_tps": 2.82 }, { "idx": 1, "completion_tokens": 256, "token_source": "usage.completion_tokens", "error": null, "ttft_ms": 76129.8, "t_first_rel_s": 76.13, "t_last_rel_s": 166.48, "decode_tps": 2.82 }, { "idx": 2, "completion_tokens": 256, "token_source": "usage.completion_tokens", "error": null, "ttft_ms": 76129.9, "t_first_rel_s": 76.13, "t_last_rel_s": 166.48, "decode_tps": 2.82 }, { "idx": 3, "completion_tokens": 256, "token_source": "usage.completion_tokens", "error": null, "ttft_ms": 76129.8, "t_first_rel_s": 76.13, "t_last_rel_s": 166.48, "decode_tps": 2.82 }, { "idx": 4, "completion_tokens": 256, "token_source": "usage.completion_tokens", "error": null, "ttft_ms": 76130.0, "t_first_rel_s": 76.13, "t_last_rel_s": 166.48, "decode_tps": 2.82 }, { "idx": 5, "completion_tokens": 256, "token_source": "usage.completion_tokens", "error": null, "ttft_ms": 76129.9, "t_first_rel_s": 76.13, "t_last_rel_s": 166.48, "decode_tps": 2.82 }, { "idx": 6, "completion_tokens": 256, "token_source": "usage.completion_tokens", "error": null, "ttft_ms": 76129.9, "t_first_rel_s": 76.13, "t_last_rel_s": 166.48, "decode_tps": 2.82 }, { "idx": 7, "completion_tokens": 256, "token_source": "usage.completion_tokens", "error": null, "ttft_ms": 76129.9, "t_first_rel_s": 76.13, "t_last_rel_s": 166.48, "decode_tps": 2.82 } ], "n_ok": 8, "n_err": 0, "wall_s": 166.48, "total_tokens": 2048, "wall_agg_tps": 12.3, "decode_agg_tps": 22.58 } ], "median_wall_agg_tps": 12.3, "median_decode_agg_tps": 22.58, "all_streams_ok": true }, "gptoss-llamaserver-conc4": { "engine": "openai", "base": "http://127.0.0.1:8081", "model": "default", "conc": 4, "enable_thinking": false, "batches": [ { "streams": [ { "idx": 0, "completion_tokens": 256, "token_source": "usage.completion_tokens", "error": null, "ttft_ms": 7558.0, "t_first_rel_s": 7.558, "t_last_rel_s": 24.469, "decode_tps": 15.08 }, { "idx": 1, "completion_tokens": 256, "token_source": "usage.completion_tokens", "error": null, "ttft_ms": 7557.0, "t_first_rel_s": 7.557, "t_last_rel_s": 24.465, "decode_tps": 15.08 }, { "idx": 2, "completion_tokens": 256, "token_source": "usage.completion_tokens", "error": null, "ttft_ms": 7557.3, "t_first_rel_s": 7.557, "t_last_rel_s": 24.466, "decode_tps": 15.08 }, { "idx": 3, "completion_tokens": 256, "token_source": "usage.completion_tokens", "error": null, "ttft_ms": 7557.7, "t_first_rel_s": 7.558, "t_last_rel_s": 24.467, "decode_tps": 15.08 } ], "n_ok": 4, "n_err": 0, "wall_s": 24.469, "total_tokens": 1024, "wall_agg_tps": 41.85, "decode_agg_tps": 60.31 }, { "streams": [ { "idx": 0, "completion_tokens": 256, "token_source": "usage.completion_tokens", "error": null, "ttft_ms": 7569.0, "t_first_rel_s": 7.569, "t_last_rel_s": 24.451, "decode_tps": 15.1 }, { "idx": 1, "completion_tokens": 256, "token_source": "usage.completion_tokens", "error": null, "ttft_ms": 7568.5, "t_first_rel_s": 7.569, "t_last_rel_s": 24.449, "decode_tps": 15.11 }, { "idx": 2, "completion_tokens": 256, "token_source": "usage.completion_tokens", "error": null, "ttft_ms": 7568.4, "t_first_rel_s": 7.568, "t_last_rel_s": 24.448, "decode_tps": 15.11 }, { "idx": 3, "completion_tokens": 256, "token_source": "usage.completion_tokens", "error": null, "ttft_ms": 7568.8, "t_first_rel_s": 7.569, "t_last_rel_s": 24.45, "decode_tps": 15.11 } ], "n_ok": 4, "n_err": 0, "wall_s": 24.451, "total_tokens": 1024, "wall_agg_tps": 41.88, "decode_agg_tps": 60.42 }, { "streams": [ { "idx": 0, "completion_tokens": 256, "token_source": "usage.completion_tokens", "error": null, "ttft_ms": 7581.5, "t_first_rel_s": 7.582, "t_last_rel_s": 24.404, "decode_tps": 15.16 }, { "idx": 1, "completion_tokens": 256, "token_source": "usage.completion_tokens", "error": null, "ttft_ms": 7581.3, "t_first_rel_s": 7.581, "t_last_rel_s": 24.403, "decode_tps": 15.16 }, { "idx": 2, "completion_tokens": 256, "token_source": "usage.completion_tokens", "error": null, "ttft_ms": 7581.1, "t_first_rel_s": 7.581, "t_last_rel_s": 24.403, "decode_tps": 15.16 }, { "idx": 3, "completion_tokens": 256, "token_source": "usage.completion_tokens", "error": null, "ttft_ms": 7581.8, "t_first_rel_s": 7.582, "t_last_rel_s": 24.405, "decode_tps": 15.16 } ], "n_ok": 4, "n_err": 0, "wall_s": 24.405, "total_tokens": 1024, "wall_agg_tps": 41.96, "decode_agg_tps": 60.63 } ], "median_wall_agg_tps": 41.88, "median_decode_agg_tps": 60.42, "all_streams_ok": true }, "gptoss-llamaserver-conc8": { "engine": "openai", "base": "http://127.0.0.1:8081", "model": "default", "conc": 8, "enable_thinking": false, "batches": [ { "streams": [ { "idx": 0, "completion_tokens": 256, "token_source": "usage.completion_tokens", "error": null, "ttft_ms": 15559.4, "t_first_rel_s": 15.559, "t_last_rel_s": 47.355, "decode_tps": 8.02 }, { "idx": 1, "completion_tokens": 256, "token_source": "usage.completion_tokens", "error": null, "ttft_ms": 15297.5, "t_first_rel_s": 15.297, "t_last_rel_s": 47.224, "decode_tps": 7.99 }, { "idx": 2, "completion_tokens": 256, "token_source": "usage.completion_tokens", "error": null, "ttft_ms": 15560.0, "t_first_rel_s": 15.56, "t_last_rel_s": 47.356, "decode_tps": 8.02 }, { "idx": 3, "completion_tokens": 256, "token_source": "usage.completion_tokens", "error": null, "ttft_ms": 15295.0, "t_first_rel_s": 15.295, "t_last_rel_s": 47.223, "decode_tps": 7.99 }, { "idx": 4, "completion_tokens": 256, "token_source": "usage.completion_tokens", "error": null, "ttft_ms": 15296.6, "t_first_rel_s": 15.297, "t_last_rel_s": 47.224, "decode_tps": 7.99 }, { "idx": 5, "completion_tokens": 256, "token_source": "usage.completion_tokens", "error": null, "ttft_ms": 15295.7, "t_first_rel_s": 15.296, "t_last_rel_s": 47.224, "decode_tps": 7.99 }, { "idx": 6, "completion_tokens": 256, "token_source": "usage.completion_tokens", "error": null, "ttft_ms": 15560.8, "t_first_rel_s": 15.561, "t_last_rel_s": 47.356, "decode_tps": 8.02 }, { "idx": 7, "completion_tokens": 256, "token_source": "usage.completion_tokens", "error": null, "ttft_ms": 15689.4, "t_first_rel_s": 15.689, "t_last_rel_s": 47.378, "decode_tps": 8.05 } ], "n_ok": 8, "n_err": 0, "wall_s": 47.378, "total_tokens": 2048, "wall_agg_tps": 43.23, "decode_agg_tps": 63.59 }, { "streams": [ { "idx": 0, "completion_tokens": 256, "token_source": "usage.completion_tokens", "error": null, "ttft_ms": 15274.3, "t_first_rel_s": 15.274, "t_last_rel_s": 46.674, "decode_tps": 8.12 }, { "idx": 1, "completion_tokens": 256, "token_source": "usage.completion_tokens", "error": null, "ttft_ms": 15274.8, "t_first_rel_s": 15.275, "t_last_rel_s": 46.674, "decode_tps": 8.12 }, { "idx": 2, "completion_tokens": 256, "token_source": "usage.completion_tokens", "error": null, "ttft_ms": 15274.5, "t_first_rel_s": 15.275, "t_last_rel_s": 46.674, "decode_tps": 8.12 }, { "idx": 3, "completion_tokens": 256, "token_source": "usage.completion_tokens", "error": null, "ttft_ms": 15525.3, "t_first_rel_s": 15.525, "t_last_rel_s": 46.804, "decode_tps": 8.15 }, { "idx": 4, "completion_tokens": 256, "token_source": "usage.completion_tokens", "error": null, "ttft_ms": 15274.1, "t_first_rel_s": 15.274, "t_last_rel_s": 46.674, "decode_tps": 8.12 }, { "idx": 5, "completion_tokens": 256, "token_source": "usage.completion_tokens", "error": null, "ttft_ms": 15650.3, "t_first_rel_s": 15.65, "t_last_rel_s": 46.827, "decode_tps": 8.18 }, { "idx": 6, "completion_tokens": 256, "token_source": "usage.completion_tokens", "error": null, "ttft_ms": 15525.0, "t_first_rel_s": 15.525, "t_last_rel_s": 46.804, "decode_tps": 8.15 }, { "idx": 7, "completion_tokens": 256, "token_source": "usage.completion_tokens", "error": null, "ttft_ms": 15525.6, "t_first_rel_s": 15.526, "t_last_rel_s": 46.805, "decode_tps": 8.15 } ], "n_ok": 8, "n_err": 0, "wall_s": 46.827, "total_tokens": 2048, "wall_agg_tps": 43.74, "decode_agg_tps": 64.65 }, { "streams": [ { "idx": 0, "completion_tokens": 256, "token_source": "usage.completion_tokens", "error": null, "ttft_ms": 15287.9, "t_first_rel_s": 15.288, "t_last_rel_s": 48.45, "decode_tps": 7.69 }, { "idx": 1, "completion_tokens": 256, "token_source": "usage.completion_tokens", "error": null, "ttft_ms": 15287.7, "t_first_rel_s": 15.288, "t_last_rel_s": 48.45, "decode_tps": 7.69 }, { "idx": 2, "completion_tokens": 256, "token_source": "usage.completion_tokens", "error": null, "ttft_ms": 15537.0, "t_first_rel_s": 15.537, "t_last_rel_s": 48.58, "decode_tps": 7.72 }, { "idx": 3, "completion_tokens": 256, "token_source": "usage.completion_tokens", "error": null, "ttft_ms": 15287.3, "t_first_rel_s": 15.287, "t_last_rel_s": 48.449, "decode_tps": 7.69 }, { "idx": 4, "completion_tokens": 256, "token_source": "usage.completion_tokens", "error": null, "ttft_ms": 15537.2, "t_first_rel_s": 15.537, "t_last_rel_s": 48.581, "decode_tps": 7.72 }, { "idx": 5, "completion_tokens": 256, "token_source": "usage.completion_tokens", "error": null, "ttft_ms": 15661.5, "t_first_rel_s": 15.662, "t_last_rel_s": 48.603, "decode_tps": 7.74 }, { "idx": 6, "completion_tokens": 256, "token_source": "usage.completion_tokens", "error": null, "ttft_ms": 15287.4, "t_first_rel_s": 15.287, "t_last_rel_s": 48.449, "decode_tps": 7.69 }, { "idx": 7, "completion_tokens": 256, "token_source": "usage.completion_tokens", "error": null, "ttft_ms": 15537.5, "t_first_rel_s": 15.538, "t_last_rel_s": 48.581, "decode_tps": 7.72 } ], "n_ok": 8, "n_err": 0, "wall_s": 48.603, "total_tokens": 2048, "wall_agg_tps": 42.14, "decode_agg_tps": 61.23 } ], "median_wall_agg_tps": 43.23, "median_decode_agg_tps": 63.59, "all_streams_ok": true }, "gptoss-ollama-conc4": { "engine": "ollama", "base": "http://127.0.0.1:11434", "model": "gpt-oss:20b", "conc": 4, "enable_thinking": false, "batches": [ { "streams": [ { "idx": 0, "completion_tokens": 256, "token_source": "ollama_eval_count", "error": null, "ttft_ms": 8434.3, "t_first_rel_s": 8.434, "t_last_rel_s": 27.987, "decode_tps": 13.04 }, { "idx": 1, "completion_tokens": 256, "token_source": "ollama_eval_count", "error": null, "ttft_ms": 8505.4, "t_first_rel_s": 8.505, "t_last_rel_s": 28.031, "decode_tps": 13.06 }, { "idx": 2, "completion_tokens": 256, "token_source": "ollama_eval_count", "error": null, "ttft_ms": 8361.8, "t_first_rel_s": 8.362, "t_last_rel_s": 27.929, "decode_tps": 13.03 }, { "idx": 3, "completion_tokens": 256, "token_source": "ollama_eval_count", "error": null, "ttft_ms": 8577.0, "t_first_rel_s": 8.577, "t_last_rel_s": 28.032, "decode_tps": 13.11 } ], "n_ok": 4, "n_err": 0, "wall_s": 28.032, "total_tokens": 1024, "wall_agg_tps": 36.53, "decode_agg_tps": 51.86 }, { "streams": [ { "idx": 0, "completion_tokens": 256, "token_source": "ollama_eval_count", "error": null, "ttft_ms": 8533.3, "t_first_rel_s": 8.533, "t_last_rel_s": 27.378, "decode_tps": 13.53 }, { "idx": 1, "completion_tokens": 256, "token_source": "ollama_eval_count", "error": null, "ttft_ms": 8315.3, "t_first_rel_s": 8.315, "t_last_rel_s": 27.091, "decode_tps": 13.58 }, { "idx": 2, "completion_tokens": 256, "token_source": "ollama_eval_count", "error": null, "ttft_ms": 8389.4, "t_first_rel_s": 8.389, "t_last_rel_s": 27.305, "decode_tps": 13.48 }, { "idx": 3, "completion_tokens": 256, "token_source": "ollama_eval_count", "error": null, "ttft_ms": 8461.9, "t_first_rel_s": 8.462, "t_last_rel_s": 27.348, "decode_tps": 13.5 } ], "n_ok": 4, "n_err": 0, "wall_s": 27.378, "total_tokens": 1024, "wall_agg_tps": 37.4, "decode_agg_tps": 53.51 }, { "streams": [ { "idx": 0, "completion_tokens": 256, "token_source": "ollama_eval_count", "error": null, "ttft_ms": 8266.5, "t_first_rel_s": 8.267, "t_last_rel_s": 27.571, "decode_tps": 13.21 }, { "idx": 1, "completion_tokens": 256, "token_source": "ollama_eval_count", "error": null, "ttft_ms": 8410.6, "t_first_rel_s": 8.411, "t_last_rel_s": 27.628, "decode_tps": 13.27 }, { "idx": 2, "completion_tokens": 254, "token_source": "ollama_eval_count", "error": null, "ttft_ms": 8339.2, "t_first_rel_s": 8.339, "t_last_rel_s": 27.443, "decode_tps": 13.24 }, { "idx": 3, "completion_tokens": 256, "token_source": "ollama_eval_count", "error": null, "ttft_ms": 8196.2, "t_first_rel_s": 8.196, "t_last_rel_s": 27.513, "decode_tps": 13.2 } ], "n_ok": 4, "n_err": 0, "wall_s": 27.628, "total_tokens": 1022, "wall_agg_tps": 36.99, "decode_agg_tps": 52.39 } ], "median_wall_agg_tps": 36.99, "median_decode_agg_tps": 52.39, "all_streams_ok": true }, "gptoss-ollama-conc8": { "engine": "ollama", "base": "http://127.0.0.1:11434", "model": "gpt-oss:20b", "conc": 8, "enable_thinking": false, "batches": [ { "streams": [ { "idx": 0, "completion_tokens": 256, "token_source": "ollama_eval_count", "error": null, "ttft_ms": 12647.5, "t_first_rel_s": 12.647, "t_last_rel_s": 51.09, "decode_tps": 6.63 }, { "idx": 1, "completion_tokens": 256, "token_source": "ollama_eval_count", "error": null, "ttft_ms": 16808.3, "t_first_rel_s": 16.808, "t_last_rel_s": 51.401, "decode_tps": 7.37 }, { "idx": 2, "completion_tokens": 256, "token_source": "ollama_eval_count", "error": null, "ttft_ms": 10617.1, "t_first_rel_s": 10.617, "t_last_rel_s": 50.948, "decode_tps": 6.32 }, { "idx": 3, "completion_tokens": 256, "token_source": "ollama_eval_count", "error": null, "ttft_ms": 17395.0, "t_first_rel_s": 17.395, "t_last_rel_s": 51.576, "decode_tps": 7.46 }, { "idx": 4, "completion_tokens": 256, "token_source": "ollama_eval_count", "error": null, "ttft_ms": 16993.0, "t_first_rel_s": 16.993, "t_last_rel_s": 51.465, "decode_tps": 7.4 }, { "idx": 5, "completion_tokens": 256, "token_source": "ollama_eval_count", "error": null, "ttft_ms": 14622.1, "t_first_rel_s": 14.622, "t_last_rel_s": 51.214, "decode_tps": 6.97 }, { "idx": 6, "completion_tokens": 256, "token_source": "ollama_eval_count", "error": null, "ttft_ms": 16610.0, "t_first_rel_s": 16.61, "t_last_rel_s": 51.317, "decode_tps": 7.35 }, { "idx": 7, "completion_tokens": 256, "token_source": "ollama_eval_count", "error": null, "ttft_ms": 17129.9, "t_first_rel_s": 17.13, "t_last_rel_s": 51.51, "decode_tps": 7.42 } ], "n_ok": 8, "n_err": 0, "wall_s": 51.576, "total_tokens": 2048, "wall_agg_tps": 39.71, "decode_agg_tps": 49.81 }, { "streams": [ { "idx": 0, "completion_tokens": 256, "token_source": "ollama_eval_count", "error": null, "ttft_ms": 14790.7, "t_first_rel_s": 14.791, "t_last_rel_s": 51.696, "decode_tps": 6.91 }, { "idx": 1, "completion_tokens": 256, "token_source": "ollama_eval_count", "error": null, "ttft_ms": 16788.4, "t_first_rel_s": 16.788, "t_last_rel_s": 51.799, "decode_tps": 7.28 }, { "idx": 2, "completion_tokens": 256, "token_source": "ollama_eval_count", "error": null, "ttft_ms": 17301.1, "t_first_rel_s": 17.301, "t_last_rel_s": 52.019, "decode_tps": 7.34 }, { "idx": 3, "completion_tokens": 256, "token_source": "ollama_eval_count", "error": null, "ttft_ms": 12870.5, "t_first_rel_s": 12.871, "t_last_rel_s": 51.579, "decode_tps": 6.59 }, { "idx": 4, "completion_tokens": 256, "token_source": "ollama_eval_count", "error": null, "ttft_ms": 17566.7, "t_first_rel_s": 17.567, "t_last_rel_s": 52.088, "decode_tps": 7.39 }, { "idx": 5, "completion_tokens": 256, "token_source": "ollama_eval_count", "error": null, "ttft_ms": 10706.1, "t_first_rel_s": 10.706, "t_last_rel_s": 51.448, "decode_tps": 6.26 }, { "idx": 6, "completion_tokens": 256, "token_source": "ollama_eval_count", "error": null, "ttft_ms": 17167.1, "t_first_rel_s": 17.167, "t_last_rel_s": 51.963, "decode_tps": 7.33 }, { "idx": 7, "completion_tokens": 256, "token_source": "ollama_eval_count", "error": null, "ttft_ms": 16986.8, "t_first_rel_s": 16.987, "t_last_rel_s": 51.89, "decode_tps": 7.31 } ], "n_ok": 8, "n_err": 0, "wall_s": 52.088, "total_tokens": 2048, "wall_agg_tps": 39.32, "decode_agg_tps": 49.3 }, { "streams": [ { "idx": 0, "completion_tokens": 256, "token_source": "ollama_eval_count", "error": null, "ttft_ms": 17268.6, "t_first_rel_s": 17.269, "t_last_rel_s": 52.503, "decode_tps": 7.24 }, { "idx": 1, "completion_tokens": 256, "token_source": "ollama_eval_count", "error": null, "ttft_ms": 14739.2, "t_first_rel_s": 14.739, "t_last_rel_s": 52.229, "decode_tps": 6.8 }, { "idx": 2, "completion_tokens": 254, "token_source": "ollama_eval_count", "error": null, "ttft_ms": 10707.3, "t_first_rel_s": 10.707, "t_last_rel_s": 51.595, "decode_tps": 6.19 }, { "idx": 3, "completion_tokens": 256, "token_source": "ollama_eval_count", "error": null, "ttft_ms": 16954.3, "t_first_rel_s": 16.954, "t_last_rel_s": 52.401, "decode_tps": 7.19 }, { "idx": 4, "completion_tokens": 256, "token_source": "ollama_eval_count", "error": null, "ttft_ms": 12861.3, "t_first_rel_s": 12.861, "t_last_rel_s": 52.121, "decode_tps": 6.5 }, { "idx": 5, "completion_tokens": 256, "token_source": "ollama_eval_count", "error": null, "ttft_ms": 16745.4, "t_first_rel_s": 16.745, "t_last_rel_s": 52.326, "decode_tps": 7.17 }, { "idx": 6, "completion_tokens": 256, "token_source": "ollama_eval_count", "error": null, "ttft_ms": 17134.4, "t_first_rel_s": 17.134, "t_last_rel_s": 52.459, "decode_tps": 7.22 }, { "idx": 7, "completion_tokens": 256, "token_source": "ollama_eval_count", "error": null, "ttft_ms": 17543.8, "t_first_rel_s": 17.544, "t_last_rel_s": 52.24, "decode_tps": 7.35 } ], "n_ok": 8, "n_err": 0, "wall_s": 52.503, "total_tokens": 2046, "wall_agg_tps": 38.97, "decode_agg_tps": 48.76 } ], "median_wall_agg_tps": 39.32, "median_decode_agg_tps": 49.3, "all_streams_ok": true }, "gptoss-rapidmlx-conc4": { "engine": "openai", "base": "http://127.0.0.1:8000", "model": "gpt-oss-20b", "conc": 4, "enable_thinking": true, "batches": [ { "streams": [ { "idx": 0, "completion_tokens": 256, "token_source": "usage.completion_tokens", "error": null, "ttft_ms": 13398.2, "t_first_rel_s": 13.398, "t_last_rel_s": 30.005, "decode_tps": 15.36 }, { "idx": 1, "completion_tokens": 256, "token_source": "usage.completion_tokens", "error": null, "ttft_ms": 13398.3, "t_first_rel_s": 13.398, "t_last_rel_s": 30.005, "decode_tps": 15.36 }, { "idx": 2, "completion_tokens": 256, "token_source": "usage.completion_tokens", "error": null, "ttft_ms": 13398.0, "t_first_rel_s": 13.398, "t_last_rel_s": 30.005, "decode_tps": 15.36 }, { "idx": 3, "completion_tokens": 256, "token_source": "usage.completion_tokens", "error": null, "ttft_ms": 13398.2, "t_first_rel_s": 13.398, "t_last_rel_s": 30.005, "decode_tps": 15.36 } ], "n_ok": 4, "n_err": 0, "wall_s": 30.005, "total_tokens": 1024, "wall_agg_tps": 34.13, "decode_agg_tps": 61.42 }, { "streams": [ { "idx": 0, "completion_tokens": 256, "token_source": "usage.completion_tokens", "error": null, "ttft_ms": 13402.9, "t_first_rel_s": 13.403, "t_last_rel_s": 30.025, "decode_tps": 15.34 }, { "idx": 1, "completion_tokens": 256, "token_source": "usage.completion_tokens", "error": null, "ttft_ms": 13403.0, "t_first_rel_s": 13.403, "t_last_rel_s": 30.025, "decode_tps": 15.34 }, { "idx": 2, "completion_tokens": 256, "token_source": "usage.completion_tokens", "error": null, "ttft_ms": 13403.1, "t_first_rel_s": 13.403, "t_last_rel_s": 30.025, "decode_tps": 15.34 }, { "idx": 3, "completion_tokens": 256, "token_source": "usage.completion_tokens", "error": null, "ttft_ms": 13403.1, "t_first_rel_s": 13.403, "t_last_rel_s": 30.025, "decode_tps": 15.34 } ], "n_ok": 4, "n_err": 0, "wall_s": 30.025, "total_tokens": 1024, "wall_agg_tps": 34.1, "decode_agg_tps": 61.36 }, { "streams": [ { "idx": 0, "completion_tokens": 256, "token_source": "usage.completion_tokens", "error": null, "ttft_ms": 13390.5, "t_first_rel_s": 13.39, "t_last_rel_s": 30.061, "decode_tps": 15.3 }, { "idx": 1, "completion_tokens": 256, "token_source": "usage.completion_tokens", "error": null, "ttft_ms": 13390.4, "t_first_rel_s": 13.39, "t_last_rel_s": 30.061, "decode_tps": 15.3 }, { "idx": 2, "completion_tokens": 256, "token_source": "usage.completion_tokens", "error": null, "ttft_ms": 13390.6, "t_first_rel_s": 13.391, "t_last_rel_s": 30.062, "decode_tps": 15.3 }, { "idx": 3, "completion_tokens": 256, "token_source": "usage.completion_tokens", "error": null, "ttft_ms": 13390.5, "t_first_rel_s": 13.391, "t_last_rel_s": 30.062, "decode_tps": 15.3 } ], "n_ok": 4, "n_err": 0, "wall_s": 30.062, "total_tokens": 1024, "wall_agg_tps": 34.06, "decode_agg_tps": 61.18 } ], "median_wall_agg_tps": 34.1, "median_decode_agg_tps": 61.36, "all_streams_ok": true }, "gptoss-rapidmlx-conc8": { "engine": "openai", "base": "http://127.0.0.1:8000", "model": "gpt-oss-20b", "conc": 8, "enable_thinking": true, "batches": [ { "streams": [ { "idx": 0, "completion_tokens": 256, "token_source": "usage.completion_tokens", "error": null, "ttft_ms": 26234.6, "t_first_rel_s": 26.235, "t_last_rel_s": 59.037, "decode_tps": 7.77 }, { "idx": 1, "completion_tokens": 256, "token_source": "usage.completion_tokens", "error": null, "ttft_ms": 26234.8, "t_first_rel_s": 26.235, "t_last_rel_s": 59.037, "decode_tps": 7.77 }, { "idx": 2, "completion_tokens": 256, "token_source": "usage.completion_tokens", "error": null, "ttft_ms": 26234.8, "t_first_rel_s": 26.235, "t_last_rel_s": 59.037, "decode_tps": 7.77 }, { "idx": 3, "completion_tokens": 256, "token_source": "usage.completion_tokens", "error": null, "ttft_ms": 26234.7, "t_first_rel_s": 26.235, "t_last_rel_s": 59.037, "decode_tps": 7.77 }, { "idx": 4, "completion_tokens": 256, "token_source": "usage.completion_tokens", "error": null, "ttft_ms": 26234.6, "t_first_rel_s": 26.235, "t_last_rel_s": 59.037, "decode_tps": 7.77 }, { "idx": 5, "completion_tokens": 256, "token_source": "usage.completion_tokens", "error": null, "ttft_ms": 26234.9, "t_first_rel_s": 26.235, "t_last_rel_s": 59.037, "decode_tps": 7.77 }, { "idx": 6, "completion_tokens": 256, "token_source": "usage.completion_tokens", "error": null, "ttft_ms": 26234.9, "t_first_rel_s": 26.235, "t_last_rel_s": 59.037, "decode_tps": 7.77 }, { "idx": 7, "completion_tokens": 256, "token_source": "usage.completion_tokens", "error": null, "ttft_ms": 26235.0, "t_first_rel_s": 26.235, "t_last_rel_s": 59.037, "decode_tps": 7.77 } ], "n_ok": 8, "n_err": 0, "wall_s": 59.037, "total_tokens": 2048, "wall_agg_tps": 34.69, "decode_agg_tps": 62.19 }, { "streams": [ { "idx": 0, "completion_tokens": 256, "token_source": "usage.completion_tokens", "error": null, "ttft_ms": 26608.7, "t_first_rel_s": 26.609, "t_last_rel_s": 59.047, "decode_tps": 7.86 }, { "idx": 1, "completion_tokens": 256, "token_source": "usage.completion_tokens", "error": null, "ttft_ms": 26608.9, "t_first_rel_s": 26.609, "t_last_rel_s": 59.047, "decode_tps": 7.86 }, { "idx": 2, "completion_tokens": 256, "token_source": "usage.completion_tokens", "error": null, "ttft_ms": 26608.8, "t_first_rel_s": 26.609, "t_last_rel_s": 59.047, "decode_tps": 7.86 }, { "idx": 3, "completion_tokens": 256, "token_source": "usage.completion_tokens", "error": null, "ttft_ms": 26608.9, "t_first_rel_s": 26.609, "t_last_rel_s": 59.047, "decode_tps": 7.86 }, { "idx": 4, "completion_tokens": 256, "token_source": "usage.completion_tokens", "error": null, "ttft_ms": 26609.0, "t_first_rel_s": 26.609, "t_last_rel_s": 59.047, "decode_tps": 7.86 }, { "idx": 5, "completion_tokens": 256, "token_source": "usage.completion_tokens", "error": null, "ttft_ms": 26609.1, "t_first_rel_s": 26.609, "t_last_rel_s": 59.047, "decode_tps": 7.86 }, { "idx": 6, "completion_tokens": 256, "token_source": "usage.completion_tokens", "error": null, "ttft_ms": 26609.1, "t_first_rel_s": 26.609, "t_last_rel_s": 59.047, "decode_tps": 7.86 }, { "idx": 7, "completion_tokens": 256, "token_source": "usage.completion_tokens", "error": null, "ttft_ms": 26609.2, "t_first_rel_s": 26.609, "t_last_rel_s": 59.047, "decode_tps": 7.86 } ], "n_ok": 8, "n_err": 0, "wall_s": 59.047, "total_tokens": 2048, "wall_agg_tps": 34.68, "decode_agg_tps": 62.89 }, { "streams": [ { "idx": 0, "completion_tokens": 256, "token_source": "usage.completion_tokens", "error": null, "ttft_ms": 26678.1, "t_first_rel_s": 26.678, "t_last_rel_s": 59.094, "decode_tps": 7.87 }, { "idx": 1, "completion_tokens": 256, "token_source": "usage.completion_tokens", "error": null, "ttft_ms": 26678.0, "t_first_rel_s": 26.678, "t_last_rel_s": 59.094, "decode_tps": 7.87 }, { "idx": 2, "completion_tokens": 256, "token_source": "usage.completion_tokens", "error": null, "ttft_ms": 26677.9, "t_first_rel_s": 26.678, "t_last_rel_s": 59.094, "decode_tps": 7.87 }, { "idx": 3, "completion_tokens": 256, "token_source": "usage.completion_tokens", "error": null, "ttft_ms": 26678.2, "t_first_rel_s": 26.678, "t_last_rel_s": 59.094, "decode_tps": 7.87 }, { "idx": 4, "completion_tokens": 256, "token_source": "usage.completion_tokens", "error": null, "ttft_ms": 26678.0, "t_first_rel_s": 26.678, "t_last_rel_s": 59.094, "decode_tps": 7.87 }, { "idx": 5, "completion_tokens": 256, "token_source": "usage.completion_tokens", "error": null, "ttft_ms": 26678.1, "t_first_rel_s": 26.678, "t_last_rel_s": 59.094, "decode_tps": 7.87 }, { "idx": 6, "completion_tokens": 256, "token_source": "usage.completion_tokens", "error": null, "ttft_ms": 26678.1, "t_first_rel_s": 26.678, "t_last_rel_s": 59.094, "decode_tps": 7.87 }, { "idx": 7, "completion_tokens": 256, "token_source": "usage.completion_tokens", "error": null, "ttft_ms": 26678.2, "t_first_rel_s": 26.678, "t_last_rel_s": 59.094, "decode_tps": 7.87 } ], "n_ok": 8, "n_err": 0, "wall_s": 59.094, "total_tokens": 2048, "wall_agg_tps": 34.66, "decode_agg_tps": 62.93 } ], "median_wall_agg_tps": 34.68, "median_decode_agg_tps": 62.89, "all_streams_ok": true }, "qwen-llamaserver-conc4": { "engine": "openai", "base": "http://127.0.0.1:8081", "model": "default", "conc": 4, "enable_thinking": false, "batches": [ { "streams": [ { "idx": 0, "completion_tokens": 256, "token_source": "usage.completion_tokens", "error": null, "ttft_ms": 9393.4, "t_first_rel_s": 9.393, "t_last_rel_s": 28.589, "decode_tps": 13.28 }, { "idx": 1, "completion_tokens": 256, "token_source": "usage.completion_tokens", "error": null, "ttft_ms": 9394.4, "t_first_rel_s": 9.394, "t_last_rel_s": 28.59, "decode_tps": 13.28 }, { "idx": 2, "completion_tokens": 256, "token_source": "usage.completion_tokens", "error": null, "ttft_ms": 9524.2, "t_first_rel_s": 9.524, "t_last_rel_s": 28.619, "decode_tps": 13.35 }, { "idx": 3, "completion_tokens": 256, "token_source": "usage.completion_tokens", "error": null, "ttft_ms": 9393.8, "t_first_rel_s": 9.394, "t_last_rel_s": 28.59, "decode_tps": 13.28 } ], "n_ok": 4, "n_err": 0, "wall_s": 28.619, "total_tokens": 1024, "wall_agg_tps": 35.78, "decode_agg_tps": 53.05 }, { "streams": [ { "idx": 0, "completion_tokens": 256, "token_source": "usage.completion_tokens", "error": null, "ttft_ms": 9449.0, "t_first_rel_s": 9.449, "t_last_rel_s": 28.609, "decode_tps": 13.31 }, { "idx": 1, "completion_tokens": 256, "token_source": "usage.completion_tokens", "error": null, "ttft_ms": 9451.2, "t_first_rel_s": 9.451, "t_last_rel_s": 28.612, "decode_tps": 13.31 }, { "idx": 2, "completion_tokens": 256, "token_source": "usage.completion_tokens", "error": null, "ttft_ms": 9449.8, "t_first_rel_s": 9.45, "t_last_rel_s": 28.611, "decode_tps": 13.31 }, { "idx": 3, "completion_tokens": 256, "token_source": "usage.completion_tokens", "error": null, "ttft_ms": 9593.7, "t_first_rel_s": 9.594, "t_last_rel_s": 28.641, "decode_tps": 13.39 } ], "n_ok": 4, "n_err": 0, "wall_s": 28.641, "total_tokens": 1024, "wall_agg_tps": 35.75, "decode_agg_tps": 53.15 }, { "streams": [ { "idx": 0, "completion_tokens": 256, "token_source": "usage.completion_tokens", "error": null, "ttft_ms": 9719.1, "t_first_rel_s": 9.719, "t_last_rel_s": 28.755, "decode_tps": 13.4 }, { "idx": 1, "completion_tokens": 256, "token_source": "usage.completion_tokens", "error": null, "ttft_ms": 9719.3, "t_first_rel_s": 9.719, "t_last_rel_s": 28.758, "decode_tps": 13.39 }, { "idx": 2, "completion_tokens": 256, "token_source": "usage.completion_tokens", "error": null, "ttft_ms": 9719.3, "t_first_rel_s": 9.719, "t_last_rel_s": 28.758, "decode_tps": 13.39 }, { "idx": 3, "completion_tokens": 256, "token_source": "usage.completion_tokens", "error": null, "ttft_ms": 9867.1, "t_first_rel_s": 9.867, "t_last_rel_s": 28.8, "decode_tps": 13.47 } ], "n_ok": 4, "n_err": 0, "wall_s": 28.8, "total_tokens": 1024, "wall_agg_tps": 35.56, "decode_agg_tps": 53.46 } ], "median_wall_agg_tps": 35.75, "median_decode_agg_tps": 53.15, "all_streams_ok": true }, "qwen-llamaserver-conc8": { "engine": "openai", "base": "http://127.0.0.1:8081", "model": "default", "conc": 8, "enable_thinking": false, "batches": [ { "streams": [ { "idx": 0, "completion_tokens": 256, "token_source": "usage.completion_tokens", "error": null, "ttft_ms": 14553.8, "t_first_rel_s": 14.554, "t_last_rel_s": 56.084, "decode_tps": 6.14 }, { "idx": 1, "completion_tokens": 256, "token_source": "usage.completion_tokens", "error": null, "ttft_ms": 19806.8, "t_first_rel_s": 19.807, "t_last_rel_s": 56.26, "decode_tps": 7.0 }, { "idx": 2, "completion_tokens": 256, "token_source": "usage.completion_tokens", "error": null, "ttft_ms": 14554.5, "t_first_rel_s": 14.554, "t_last_rel_s": 56.085, "decode_tps": 6.14 }, { "idx": 3, "completion_tokens": 256, "token_source": "usage.completion_tokens", "error": null, "ttft_ms": 19808.2, "t_first_rel_s": 19.808, "t_last_rel_s": 56.261, "decode_tps": 7.0 }, { "idx": 4, "completion_tokens": 256, "token_source": "usage.completion_tokens", "error": null, "ttft_ms": 20012.9, "t_first_rel_s": 20.013, "t_last_rel_s": 56.292, "decode_tps": 7.03 }, { "idx": 5, "completion_tokens": 256, "token_source": "usage.completion_tokens", "error": null, "ttft_ms": 14555.4, "t_first_rel_s": 14.555, "t_last_rel_s": 56.085, "decode_tps": 6.14 }, { "idx": 6, "completion_tokens": 256, "token_source": "usage.completion_tokens", "error": null, "ttft_ms": 18345.6, "t_first_rel_s": 18.346, "t_last_rel_s": 56.182, "decode_tps": 6.74 }, { "idx": 7, "completion_tokens": 256, "token_source": "usage.completion_tokens", "error": null, "ttft_ms": 19809.1, "t_first_rel_s": 19.809, "t_last_rel_s": 56.261, "decode_tps": 7.0 } ], "n_ok": 8, "n_err": 0, "wall_s": 56.292, "total_tokens": 2048, "wall_agg_tps": 36.38, "decode_agg_tps": 48.88 }, { "streams": [ { "idx": 0, "completion_tokens": 256, "token_source": "usage.completion_tokens", "error": null, "ttft_ms": 14753.1, "t_first_rel_s": 14.753, "t_last_rel_s": 56.333, "decode_tps": 6.13 }, { "idx": 1, "completion_tokens": 256, "token_source": "usage.completion_tokens", "error": null, "ttft_ms": 20250.7, "t_first_rel_s": 20.251, "t_last_rel_s": 56.539, "decode_tps": 7.03 }, { "idx": 2, "completion_tokens": 256, "token_source": "usage.completion_tokens", "error": null, "ttft_ms": 20048.5, "t_first_rel_s": 20.049, "t_last_rel_s": 56.509, "decode_tps": 6.99 }, { "idx": 3, "completion_tokens": 256, "token_source": "usage.completion_tokens", "error": null, "ttft_ms": 20047.9, "t_first_rel_s": 20.048, "t_last_rel_s": 56.509, "decode_tps": 6.99 }, { "idx": 4, "completion_tokens": 256, "token_source": "usage.completion_tokens", "error": null, "ttft_ms": 14753.1, "t_first_rel_s": 14.753, "t_last_rel_s": 56.334, "decode_tps": 6.13 }, { "idx": 5, "completion_tokens": 256, "token_source": "usage.completion_tokens", "error": null, "ttft_ms": 18547.3, "t_first_rel_s": 18.547, "t_last_rel_s": 56.43, "decode_tps": 6.73 }, { "idx": 6, "completion_tokens": 256, "token_source": "usage.completion_tokens", "error": null, "ttft_ms": 14752.7, "t_first_rel_s": 14.753, "t_last_rel_s": 56.331, "decode_tps": 6.13 }, { "idx": 7, "completion_tokens": 256, "token_source": "usage.completion_tokens", "error": null, "ttft_ms": 20049.2, "t_first_rel_s": 20.049, "t_last_rel_s": 56.509, "decode_tps": 6.99 } ], "n_ok": 8, "n_err": 0, "wall_s": 56.539, "total_tokens": 2048, "wall_agg_tps": 36.22, "decode_agg_tps": 48.82 }, { "streams": [ { "idx": 0, "completion_tokens": 256, "token_source": "usage.completion_tokens", "error": null, "ttft_ms": 14970.6, "t_first_rel_s": 14.971, "t_last_rel_s": 56.565, "decode_tps": 6.13 }, { "idx": 1, "completion_tokens": 256, "token_source": "usage.completion_tokens", "error": null, "ttft_ms": 18793.7, "t_first_rel_s": 18.794, "t_last_rel_s": 56.662, "decode_tps": 6.73 }, { "idx": 2, "completion_tokens": 256, "token_source": "usage.completion_tokens", "error": null, "ttft_ms": 20263.1, "t_first_rel_s": 20.263, "t_last_rel_s": 56.742, "decode_tps": 6.99 }, { "idx": 3, "completion_tokens": 256, "token_source": "usage.completion_tokens", "error": null, "ttft_ms": 20476.7, "t_first_rel_s": 20.477, "t_last_rel_s": 56.773, "decode_tps": 7.03 }, { "idx": 4, "completion_tokens": 256, "token_source": "usage.completion_tokens", "error": null, "ttft_ms": 20262.1, "t_first_rel_s": 20.262, "t_last_rel_s": 56.742, "decode_tps": 6.99 }, { "idx": 5, "completion_tokens": 256, "token_source": "usage.completion_tokens", "error": null, "ttft_ms": 14969.3, "t_first_rel_s": 14.969, "t_last_rel_s": 56.564, "decode_tps": 6.13 }, { "idx": 6, "completion_tokens": 256, "token_source": "usage.completion_tokens", "error": null, "ttft_ms": 20264.0, "t_first_rel_s": 20.264, "t_last_rel_s": 56.742, "decode_tps": 6.99 }, { "idx": 7, "completion_tokens": 256, "token_source": "usage.completion_tokens", "error": null, "ttft_ms": 14972.5, "t_first_rel_s": 14.973, "t_last_rel_s": 56.565, "decode_tps": 6.13 } ], "n_ok": 8, "n_err": 0, "wall_s": 56.773, "total_tokens": 2048, "wall_agg_tps": 36.07, "decode_agg_tps": 48.8 } ], "median_wall_agg_tps": 36.22, "median_decode_agg_tps": 48.82, "all_streams_ok": true }, "qwen-ollama-conc4": { "engine": "ollama", "base": "http://127.0.0.1:11434", "model": "qwen3.6:35b-a3b-q4_K_M", "conc": 4, "enable_thinking": false, "batches": [ { "streams": [ { "idx": 0, "completion_tokens": 256, "token_source": "ollama_eval_count", "error": null, "ttft_ms": 31325.0, "t_first_rel_s": 31.325, "t_last_rel_s": 37.721, "decode_tps": 39.87 }, { "idx": 1, "completion_tokens": 256, "token_source": "ollama_eval_count", "error": null, "ttft_ms": 21594.4, "t_first_rel_s": 21.594, "t_last_rel_s": 28.752, "decode_tps": 35.62 }, { "idx": 2, "completion_tokens": 256, "token_source": "ollama_eval_count", "error": null, "ttft_ms": 12085.5, "t_first_rel_s": 12.085, "t_last_rel_s": 19.007, "decode_tps": 36.84 }, { "idx": 3, "completion_tokens": 256, "token_source": "ollama_eval_count", "error": null, "ttft_ms": 2896.5, "t_first_rel_s": 2.897, "t_last_rel_s": 9.592, "decode_tps": 38.09 } ], "n_ok": 4, "n_err": 0, "wall_s": 37.721, "total_tokens": 1024, "wall_agg_tps": 27.15, "decode_agg_tps": 29.29 }, { "streams": [ { "idx": 0, "completion_tokens": 256, "token_source": "ollama_eval_count", "error": null, "ttft_ms": 11681.1, "t_first_rel_s": 11.681, "t_last_rel_s": 18.658, "decode_tps": 36.55 }, { "idx": 1, "completion_tokens": 256, "token_source": "ollama_eval_count", "error": null, "ttft_ms": 21063.9, "t_first_rel_s": 21.064, "t_last_rel_s": 27.444, "decode_tps": 39.97 }, { "idx": 2, "completion_tokens": 256, "token_source": "ollama_eval_count", "error": null, "ttft_ms": 2742.0, "t_first_rel_s": 2.742, "t_last_rel_s": 9.173, "decode_tps": 39.65 }, { "idx": 3, "completion_tokens": 256, "token_source": "ollama_eval_count", "error": null, "ttft_ms": 29865.2, "t_first_rel_s": 29.865, "t_last_rel_s": 36.265, "decode_tps": 39.84 } ], "n_ok": 4, "n_err": 0, "wall_s": 36.265, "total_tokens": 1024, "wall_agg_tps": 28.24, "decode_agg_tps": 30.43 }, { "streams": [ { "idx": 0, "completion_tokens": 256, "token_source": "ollama_eval_count", "error": null, "ttft_ms": 2678.4, "t_first_rel_s": 2.678, "t_last_rel_s": 9.24, "decode_tps": 38.86 }, { "idx": 1, "completion_tokens": 256, "token_source": "ollama_eval_count", "error": null, "ttft_ms": 29592.0, "t_first_rel_s": 29.592, "t_last_rel_s": 35.994, "decode_tps": 39.83 }, { "idx": 2, "completion_tokens": 256, "token_source": "ollama_eval_count", "error": null, "ttft_ms": 11750.8, "t_first_rel_s": 11.751, "t_last_rel_s": 18.231, "decode_tps": 39.35 }, { "idx": 3, "completion_tokens": 256, "token_source": "ollama_eval_count", "error": null, "ttft_ms": 20687.9, "t_first_rel_s": 20.688, "t_last_rel_s": 27.154, "decode_tps": 39.44 } ], "n_ok": 4, "n_err": 0, "wall_s": 35.994, "total_tokens": 1024, "wall_agg_tps": 28.45, "decode_agg_tps": 30.62 } ], "median_wall_agg_tps": 28.24, "median_decode_agg_tps": 30.43, "all_streams_ok": true }, "qwen-ollama-conc8": { "engine": "ollama", "base": "http://127.0.0.1:11434", "model": "qwen3.6:35b-a3b-q4_K_M", "conc": 8, "enable_thinking": false, "batches": [ { "streams": [ { "idx": 0, "completion_tokens": 256, "token_source": "ollama_eval_count", "error": null, "ttft_ms": 50286.4, "t_first_rel_s": 50.286, "t_last_rel_s": 56.755, "decode_tps": 39.42 }, { "idx": 1, "completion_tokens": 256, "token_source": "ollama_eval_count", "error": null, "ttft_ms": 2801.0, "t_first_rel_s": 2.801, "t_last_rel_s": 9.157, "decode_tps": 40.12 }, { "idx": 2, "completion_tokens": 256, "token_source": "ollama_eval_count", "error": null, "ttft_ms": 30287.9, "t_first_rel_s": 30.288, "t_last_rel_s": 38.276, "decode_tps": 31.92 }, { "idx": 3, "completion_tokens": 256, "token_source": "ollama_eval_count", "error": null, "ttft_ms": 11585.7, "t_first_rel_s": 11.586, "t_last_rel_s": 18.184, "decode_tps": 38.65 }, { "idx": 4, "completion_tokens": 256, "token_source": "ollama_eval_count", "error": null, "ttft_ms": 40855.2, "t_first_rel_s": 40.855, "t_last_rel_s": 47.78, "decode_tps": 36.83 }, { "idx": 5, "completion_tokens": 256, "token_source": "ollama_eval_count", "error": null, "ttft_ms": 68368.1, "t_first_rel_s": 68.368, "t_last_rel_s": 75.314, "decode_tps": 36.71 }, { "idx": 6, "completion_tokens": 256, "token_source": "ollama_eval_count", "error": null, "ttft_ms": 20620.1, "t_first_rel_s": 20.62, "t_last_rel_s": 27.331, "decode_tps": 38.0 }, { "idx": 7, "completion_tokens": 256, "token_source": "ollama_eval_count", "error": null, "ttft_ms": 59250.4, "t_first_rel_s": 59.25, "t_last_rel_s": 65.894, "decode_tps": 38.38 } ], "n_ok": 8, "n_err": 0, "wall_s": 75.314, "total_tokens": 2048, "wall_agg_tps": 27.19, "decode_agg_tps": 28.13 }, { "streams": [ { "idx": 0, "completion_tokens": 256, "token_source": "ollama_eval_count", "error": null, "ttft_ms": 59621.8, "t_first_rel_s": 59.622, "t_last_rel_s": 67.465, "decode_tps": 32.51 }, { "idx": 1, "completion_tokens": 256, "token_source": "ollama_eval_count", "error": null, "ttft_ms": 50656.4, "t_first_rel_s": 50.656, "t_last_rel_s": 57.123, "decode_tps": 39.43 }, { "idx": 2, "completion_tokens": 256, "token_source": "ollama_eval_count", "error": null, "ttft_ms": 41336.1, "t_first_rel_s": 41.336, "t_last_rel_s": 48.183, "decode_tps": 37.24 }, { "idx": 3, "completion_tokens": 256, "token_source": "ollama_eval_count", "error": null, "ttft_ms": 2843.2, "t_first_rel_s": 2.843, "t_last_rel_s": 10.146, "decode_tps": 34.92 }, { "idx": 4, "completion_tokens": 256, "token_source": "ollama_eval_count", "error": null, "ttft_ms": 69895.9, "t_first_rel_s": 69.896, "t_last_rel_s": 77.749, "decode_tps": 32.47 }, { "idx": 5, "completion_tokens": 256, "token_source": "ollama_eval_count", "error": null, "ttft_ms": 12701.6, "t_first_rel_s": 12.702, "t_last_rel_s": 19.634, "decode_tps": 36.78 }, { "idx": 6, "completion_tokens": 256, "token_source": "ollama_eval_count", "error": null, "ttft_ms": 32050.3, "t_first_rel_s": 32.05, "t_last_rel_s": 38.874, "decode_tps": 37.37 }, { "idx": 7, "completion_tokens": 256, "token_source": "ollama_eval_count", "error": null, "ttft_ms": 22119.8, "t_first_rel_s": 22.12, "t_last_rel_s": 28.959, "decode_tps": 37.29 } ], "n_ok": 8, "n_err": 0, "wall_s": 77.749, "total_tokens": 2048, "wall_agg_tps": 26.34, "decode_agg_tps": 27.23 }, { "streams": [ { "idx": 0, "completion_tokens": 256, "token_source": "ollama_eval_count", "error": null, "ttft_ms": 72551.4, "t_first_rel_s": 72.551, "t_last_rel_s": 78.936, "decode_tps": 39.94 }, { "idx": 1, "completion_tokens": 256, "token_source": "ollama_eval_count", "error": null, "ttft_ms": 53674.5, "t_first_rel_s": 53.675, "t_last_rel_s": 60.305, "decode_tps": 38.46 }, { "idx": 2, "completion_tokens": 256, "token_source": "ollama_eval_count", "error": null, "ttft_ms": 23152.4, "t_first_rel_s": 23.152, "t_last_rel_s": 31.089, "decode_tps": 32.13 }, { "idx": 3, "completion_tokens": 256, "token_source": "ollama_eval_count", "error": null, "ttft_ms": 2767.6, "t_first_rel_s": 2.768, "t_last_rel_s": 10.416, "decode_tps": 33.34 }, { "idx": 4, "completion_tokens": 256, "token_source": "ollama_eval_count", "error": null, "ttft_ms": 62725.2, "t_first_rel_s": 62.725, "t_last_rel_s": 70.1, "decode_tps": 34.58 }, { "idx": 5, "completion_tokens": 256, "token_source": "ollama_eval_count", "error": null, "ttft_ms": 43442.9, "t_first_rel_s": 43.443, "t_last_rel_s": 51.233, "decode_tps": 32.73 }, { "idx": 6, "completion_tokens": 256, "token_source": "ollama_eval_count", "error": null, "ttft_ms": 33585.7, "t_first_rel_s": 33.586, "t_last_rel_s": 40.919, "decode_tps": 34.77 }, { "idx": 7, "completion_tokens": 256, "token_source": "ollama_eval_count", "error": null, "ttft_ms": 12875.6, "t_first_rel_s": 12.876, "t_last_rel_s": 20.664, "decode_tps": 32.74 } ], "n_ok": 8, "n_err": 0, "wall_s": 78.936, "total_tokens": 2048, "wall_agg_tps": 25.95, "decode_agg_tps": 26.78 } ], "median_wall_agg_tps": 26.34, "median_decode_agg_tps": 27.23, "all_streams_ok": true }, "qwen-rapidmlx-conc4": { "engine": "openai", "base": "http://127.0.0.1:8000", "model": "qwen3.6-35b-4bit", "conc": 4, "enable_thinking": true, "batches": [ { "streams": [ { "idx": 0, "completion_tokens": 256, "token_source": "usage.completion_tokens", "error": null, "ttft_ms": 12037.4, "t_first_rel_s": 12.037, "t_last_rel_s": 25.683, "decode_tps": 18.69 }, { "idx": 1, "completion_tokens": 256, "token_source": "usage.completion_tokens", "error": null, "ttft_ms": 12037.3, "t_first_rel_s": 12.037, "t_last_rel_s": 25.683, "decode_tps": 18.69 }, { "idx": 2, "completion_tokens": 256, "token_source": "usage.completion_tokens", "error": null, "ttft_ms": 12037.4, "t_first_rel_s": 12.037, "t_last_rel_s": 25.683, "decode_tps": 18.69 }, { "idx": 3, "completion_tokens": 256, "token_source": "usage.completion_tokens", "error": null, "ttft_ms": 12037.0, "t_first_rel_s": 12.037, "t_last_rel_s": 25.683, "decode_tps": 18.69 } ], "n_ok": 4, "n_err": 0, "wall_s": 25.683, "total_tokens": 1024, "wall_agg_tps": 39.87, "decode_agg_tps": 74.75 }, { "streams": [ { "idx": 0, "completion_tokens": 256, "token_source": "usage.completion_tokens", "error": null, "ttft_ms": 11917.7, "t_first_rel_s": 11.918, "t_last_rel_s": 25.368, "decode_tps": 18.96 }, { "idx": 1, "completion_tokens": 256, "token_source": "usage.completion_tokens", "error": null, "ttft_ms": 11917.8, "t_first_rel_s": 11.918, "t_last_rel_s": 25.368, "decode_tps": 18.96 }, { "idx": 2, "completion_tokens": 256, "token_source": "usage.completion_tokens", "error": null, "ttft_ms": 11917.7, "t_first_rel_s": 11.918, "t_last_rel_s": 25.368, "decode_tps": 18.96 }, { "idx": 3, "completion_tokens": 256, "token_source": "usage.completion_tokens", "error": null, "ttft_ms": 11917.4, "t_first_rel_s": 11.917, "t_last_rel_s": 25.368, "decode_tps": 18.96 } ], "n_ok": 4, "n_err": 0, "wall_s": 25.368, "total_tokens": 1024, "wall_agg_tps": 40.37, "decode_agg_tps": 75.83 }, { "streams": [ { "idx": 0, "completion_tokens": 256, "token_source": "usage.completion_tokens", "error": null, "ttft_ms": 13247.9, "t_first_rel_s": 13.248, "t_last_rel_s": 26.722, "decode_tps": 18.92 }, { "idx": 1, "completion_tokens": 256, "token_source": "usage.completion_tokens", "error": null, "ttft_ms": 13248.2, "t_first_rel_s": 13.248, "t_last_rel_s": 26.722, "decode_tps": 18.93 }, { "idx": 2, "completion_tokens": 256, "token_source": "usage.completion_tokens", "error": null, "ttft_ms": 13248.2, "t_first_rel_s": 13.248, "t_last_rel_s": 26.722, "decode_tps": 18.93 }, { "idx": 3, "completion_tokens": 256, "token_source": "usage.completion_tokens", "error": null, "ttft_ms": 13248.3, "t_first_rel_s": 13.248, "t_last_rel_s": 26.722, "decode_tps": 18.93 } ], "n_ok": 4, "n_err": 0, "wall_s": 26.722, "total_tokens": 1024, "wall_agg_tps": 38.32, "decode_agg_tps": 75.7 } ], "median_wall_agg_tps": 39.87, "median_decode_agg_tps": 75.7, "all_streams_ok": true }, "qwen-rapidmlx-conc8": { "engine": "openai", "base": "http://127.0.0.1:8000", "model": "qwen3.6-35b-4bit", "conc": 8, "enable_thinking": true, "batches": [ { "streams": [ { "idx": 0, "completion_tokens": 256, "token_source": "usage.completion_tokens", "error": null, "ttft_ms": 24402.0, "t_first_rel_s": 24.402, "t_last_rel_s": 48.98, "decode_tps": 10.37 }, { "idx": 1, "completion_tokens": 256, "token_source": "usage.completion_tokens", "error": null, "ttft_ms": 24403.2, "t_first_rel_s": 24.403, "t_last_rel_s": 48.981, "decode_tps": 10.38 }, { "idx": 2, "completion_tokens": 256, "token_source": "usage.completion_tokens", "error": null, "ttft_ms": 24403.3, "t_first_rel_s": 24.403, "t_last_rel_s": 48.981, "decode_tps": 10.38 }, { "idx": 3, "completion_tokens": 256, "token_source": "usage.completion_tokens", "error": null, "ttft_ms": 24403.4, "t_first_rel_s": 24.403, "t_last_rel_s": 48.981, "decode_tps": 10.38 }, { "idx": 4, "completion_tokens": 256, "token_source": "usage.completion_tokens", "error": null, "ttft_ms": 24403.2, "t_first_rel_s": 24.403, "t_last_rel_s": 48.981, "decode_tps": 10.38 }, { "idx": 5, "completion_tokens": 256, "token_source": "usage.completion_tokens", "error": null, "ttft_ms": 24403.2, "t_first_rel_s": 24.403, "t_last_rel_s": 48.981, "decode_tps": 10.38 }, { "idx": 6, "completion_tokens": 256, "token_source": "usage.completion_tokens", "error": null, "ttft_ms": 24403.3, "t_first_rel_s": 24.403, "t_last_rel_s": 48.981, "decode_tps": 10.38 }, { "idx": 7, "completion_tokens": 256, "token_source": "usage.completion_tokens", "error": null, "ttft_ms": 24403.4, "t_first_rel_s": 24.403, "t_last_rel_s": 48.981, "decode_tps": 10.38 } ], "n_ok": 8, "n_err": 0, "wall_s": 48.981, "total_tokens": 2048, "wall_agg_tps": 41.81, "decode_agg_tps": 83.0 }, { "streams": [ { "idx": 0, "completion_tokens": 256, "token_source": "usage.completion_tokens", "error": null, "ttft_ms": 24061.6, "t_first_rel_s": 24.062, "t_last_rel_s": 48.661, "decode_tps": 10.37 }, { "idx": 1, "completion_tokens": 256, "token_source": "usage.completion_tokens", "error": null, "ttft_ms": 24061.8, "t_first_rel_s": 24.062, "t_last_rel_s": 48.661, "decode_tps": 10.37 }, { "idx": 2, "completion_tokens": 256, "token_source": "usage.completion_tokens", "error": null, "ttft_ms": 24061.2, "t_first_rel_s": 24.061, "t_last_rel_s": 48.661, "decode_tps": 10.37 }, { "idx": 3, "completion_tokens": 256, "token_source": "usage.completion_tokens", "error": null, "ttft_ms": 24061.6, "t_first_rel_s": 24.062, "t_last_rel_s": 48.661, "decode_tps": 10.37 }, { "idx": 4, "completion_tokens": 256, "token_source": "usage.completion_tokens", "error": null, "ttft_ms": 24061.7, "t_first_rel_s": 24.062, "t_last_rel_s": 48.661, "decode_tps": 10.37 }, { "idx": 5, "completion_tokens": 256, "token_source": "usage.completion_tokens", "error": null, "ttft_ms": 24061.8, "t_first_rel_s": 24.062, "t_last_rel_s": 48.661, "decode_tps": 10.37 }, { "idx": 6, "completion_tokens": 256, "token_source": "usage.completion_tokens", "error": null, "ttft_ms": 24061.7, "t_first_rel_s": 24.062, "t_last_rel_s": 48.661, "decode_tps": 10.37 }, { "idx": 7, "completion_tokens": 256, "token_source": "usage.completion_tokens", "error": null, "ttft_ms": 24061.8, "t_first_rel_s": 24.062, "t_last_rel_s": 48.661, "decode_tps": 10.37 } ], "n_ok": 8, "n_err": 0, "wall_s": 48.661, "total_tokens": 2048, "wall_agg_tps": 42.09, "decode_agg_tps": 82.93 }, { "streams": [ { "idx": 0, "completion_tokens": 256, "token_source": "usage.completion_tokens", "error": null, "ttft_ms": 24249.5, "t_first_rel_s": 24.25, "t_last_rel_s": 48.979, "decode_tps": 10.31 }, { "idx": 1, "completion_tokens": 256, "token_source": "usage.completion_tokens", "error": null, "ttft_ms": 24249.2, "t_first_rel_s": 24.249, "t_last_rel_s": 48.979, "decode_tps": 10.31 }, { "idx": 2, "completion_tokens": 256, "token_source": "usage.completion_tokens", "error": null, "ttft_ms": 24249.4, "t_first_rel_s": 24.249, "t_last_rel_s": 48.979, "decode_tps": 10.31 }, { "idx": 3, "completion_tokens": 256, "token_source": "usage.completion_tokens", "error": null, "ttft_ms": 24249.5, "t_first_rel_s": 24.249, "t_last_rel_s": 48.979, "decode_tps": 10.31 }, { "idx": 4, "completion_tokens": 256, "token_source": "usage.completion_tokens", "error": null, "ttft_ms": 24249.5, "t_first_rel_s": 24.249, "t_last_rel_s": 48.979, "decode_tps": 10.31 }, { "idx": 5, "completion_tokens": 256, "token_source": "usage.completion_tokens", "error": null, "ttft_ms": 24249.5, "t_first_rel_s": 24.25, "t_last_rel_s": 48.979, "decode_tps": 10.31 }, { "idx": 6, "completion_tokens": 256, "token_source": "usage.completion_tokens", "error": null, "ttft_ms": 24249.5, "t_first_rel_s": 24.25, "t_last_rel_s": 48.979, "decode_tps": 10.31 }, { "idx": 7, "completion_tokens": 256, "token_source": "usage.completion_tokens", "error": null, "ttft_ms": 24249.5, "t_first_rel_s": 24.25, "t_last_rel_s": 48.979, "decode_tps": 10.31 } ], "n_ok": 8, "n_err": 0, "wall_s": 48.979, "total_tokens": 2048, "wall_agg_tps": 41.81, "decode_agg_tps": 82.49 } ], "median_wall_agg_tps": 41.81, "median_decode_agg_tps": 82.93, "all_streams_ok": true } } } }