Performance flags reference
Speculative decoding, long-prompt compression, and KV-cache
compression — the four rapid-mlx-native performance techniques and
the flags that turn them on. Every row below is verifiable against
the rapid-mlx serve --help output of the shipped
0.10.3 build.
rapid-mlx info <alias> for the per-alias
eligibility grid — it tells you which of MTP / DFlash / DDTree /
SuffixDecoding actually apply to that model.
MTP · native Multi-Token Prediction heads
For models with built-in MTP heads (Qwen 3.5 / 3.6 native checkpoints per the 0.9.13 blueprint, and DeepSeek V3 / V4). Uses cache snapshot / restore for speculative generation.
| Flag | Default | Purpose |
|---|---|---|
--enable-mtp |
off | Turn on MTP speculative generation for MTP-eligible aliases. |
--mtp-num-draft-tokens N |
1 |
Number of draft tokens per MTP step. |
--mtp-optimistic |
off | Skip MTP acceptance check for maximum speed. ~5–10% wrong tokens. Best for chat, not for code. |
--speculative-config '{"method":"mtp",…}' |
— | vLLM-style JSON. Fields: num_speculative_tokens, disable_auto_k. Preferred for new usage. |
$ rapid-mlx serve qwen3.6-35b-8bit --enable-mtp --mtp-num-draft-tokens 3 # or vLLM-style JSON: $ rapid-mlx serve qwen3.6-35b-8bit \ --speculative-config '{"method":"mtp","num_speculative_tokens":3,"disable_auto_k":false}'
Empirical: 0.9.13 gemma-4-12b-it MTP shipped +14.4% structured
throughput with 5/5 determinism gates PASS. K histogram was
balanced across the tested prompt classes — see
specdecoding.md
for the latest MTP validation memo (2026-07-06: Gemma 4
assistant-sidecar MTP is not supported after greedy-lossless
correctness failed on 3 of 4 prompts).
DFlash · block-diffusion drafter
Block-diffusion speculative drafter from arXiv 2410.04097.
Single-user serial mode. Requires a DFlash-eligible alias — check
with rapid-mlx info <alias>.
| Flag | Default | Purpose |
|---|---|---|
--enable-dflash |
off | Enable DFlash. Requires pip install 'rapid-mlx[dflash]'. |
--spec-decode {none,dflash} |
none |
Compatibility selector; prefer --speculative-config for new usage. |
--dflash-drafter-path <HF-repo> |
alias-default | Compatibility override for the per-alias drafter HF path; empty defers to vllm_mlx.spec_decode.dflash.drafter_registry. |
--speculative-config '{"method":"dflash"}' |
— | vLLM-style equivalent. |
Eligibility gates surfaced by rapid-mlx info:
declared support, non-MoE, precision ≥ 8-bit, drafter declared,
mlx-vlm 0.5.0+ present. The 0.10.3 rapid-mlx info
qwen3.5-27b-8bit shows DFlash-eligible; the same
output for gemma-4-12b-4bit shows ineligible
(4-bit precision + no declared drafter).
DDTree · draft tree verification (experimental)
DFlash draft-tree verification, single-user serial mode. Requires
a DDTree-eligible alias and the external dtree-mlx
runtime.
| Flag | Default | Purpose |
|---|---|---|
--enable-ddtree |
off | Enable experimental DDTree. Fails loud if the alias isn't declared eligible or if dtree-mlx isn't importable. |
--speculative-config '{"method":"ddtree"}' |
— | vLLM-style equivalent. |
SuffixDecoding · drafter-free statistical drafts
Model-free speculative decoding using suffix trees built from past outputs (NeurIPS 2025 Spotlight, arXiv 2411.04975). Explicit opt-in — useful for long high-overlap workloads (prompt-copy, code editing, repeated tool XML) on validated aliases; local benches showed regressions on gpt-oss / Qwen families under mixed chat traffic. Auto-disabled on hybrid models (Qwen 3.5 / 3.6 A3B / A10B, Granite 4, Mamba / Jamba / RWKV).
| Flag | Default | Purpose |
|---|---|---|
--suffix-decoding |
off | Enable SuffixDecoding drafter-free spec decode. Explicit opt-in. |
--suffix-max-draft N |
8 |
Max draft tokens per verify step. Verify forward cost grows linearly with this. |
--suffix-max-suffix-len N |
4 |
Max k-gram length indexed for suffix matching. |
--suffix-min-confidence F |
0.3 |
Vote confidence floor for draft truncation. Lower → more optimistic drafts; higher → fewer but more reliable. |
--suffix-min-draft-len N |
2 |
Skip the verify forward when drafter returns fewer than this many tokens. Set to 1 to verify every draft (more aggressive; can regress chat). |
--speculative-config '{"method":"suffix","num_speculative_tokens":8}' |
— | vLLM-style equivalent. |
$ rapid-mlx serve gemma-4-12b-4bit \
--suffix-decoding --suffix-max-draft 8 --suffix-min-confidence 0.3
Per-alias suffix tier is surfaced by rapid-mlx info
<alias> (values: validated,
rejected, unknown). "Unknown" means the
bench scripts/bench_suffix_decoding.py hasn't been
run for the alias yet — it's safe to try but not guaranteed.
PFlash · long-prompt prefill compression
Scores 32K+ prompts and only prefills the sink + recent tail + query-relevant middle blocks. Default-on for verified Qwen 3.5 / 3.6 aliases (per PR #287), off elsewhere. Bench-validated profile (PR #649): TTFT 3.87×–8.5× faster, needle-in-haystack recall 5/5 across tested cells.
| Flag | Default | Purpose |
|---|---|---|
--pflash {off,auto,always} |
always on verified aliases, off elsewhere |
Master switch. auto respects the threshold; always compresses every prompt over the threshold. |
--pflash-threshold N |
32768 |
Minimum prompt tokens before auto compresses. |
--pflash-keep-ratio F |
0.20 |
Fraction of prompt tokens to keep when compressing. Matches the bench-validated profile. |
--pflash-min-keep-tokens N |
2048 |
Minimum tokens to keep when compressing. |
--pflash-sink-tokens N |
256 |
Leading prompt tokens always kept. |
--pflash-tail-tokens N |
2048 |
Trailing prompt tokens always kept. |
--pflash-block-size N |
128 |
Middle-token scoring block size. |
--pflash-query-window N |
512 |
Trailing query window used to score middle blocks. |
--pflash-stride-blocks N |
8 |
Keep every Nth middle block as an anchor during scoring (0 disables anchors). |
--pflash-include-tools |
off | Allow PFlash compression on prompts with tool definitions. By default tool prompts are skipped for tool-call reliability. |
TurboQuant · KV-cache compression
Post-quantization KV-cache compression. v4 is
V-only 3–4 bit Lloyd-Max with K in FP16; k8v4 is the
R15 Phase 4 mix — K at 8-bit Walsh-Hadamard + V at 4-bit Lloyd-Max
(~4.6× KV compression on dense models, ~1/2.4 = ~58% savings on
the shipped MoE cells). k8v4 is default-on for 9
verified Qwen 3.5 / 3.6 MoE aliases per the 0.9 changelog.
| Flag | Default | Purpose |
|---|---|---|
--kv-cache-turboquant {v4,k8v4,none} |
alias-driven (k8v4 on 9 verified Qwen 3.5 / 3.6 MoE) |
Master switch. none is the explicit off-switch. |
--kv-cache-turboquant-bits {3,4} |
auto (3 for head_dim≥96, 4 for head_dim=64) | V-side bit width. Ignored on k8v4 (V pinned to 4-bit). |
--kv-cache-turboquant-group-size N |
32 |
Group size for V-side quantization. |
Eligibility check
rapid-mlx info <alias> is the source of truth
for what applies to a given model. Example output for
gemma-4-12b-4bit in the shipped 0.10.3 build:
$ rapid-mlx info gemma-4-12b-4bit
Alias: gemma-4-12b-4bit → mlx-community/gemma-4-12B-it-4bit
┌──────────────────────────────────────────────────────────────┐
│ Model: mlx-community/gemma-4-12B-it-4bit │
│ Tool format : gemma4 │
│ Reasoning parser : gemma4 │
│ Architecture : pure attention │
│ Spec decode : ✓ supported │
│ Suffix tier : unknown — run scripts/bench_suffix_decod… │
└──────────────────────────────────────────────────────────────┘
┌──────────────────────────────────────────────────────────────┐
│ DFlash eligibility: ✗ ineligible │
│ Declared support : ✗ no │
│ Not MoE : ✓ yes (dense) │
│ Precision ≥8-bit : ✗ no (4-bit/mxfp4/nvfp4) │
│ Drafter declared : ✗ no (dflash_draft_model unset) │
└──────────────────────────────────────────────────────────────┘