Performance flags reference

Speculative decoding, long-prompt compression, and KV-cache compression — the rapid-mlx-native performance techniques and the flags that turn them on. Every row below is verifiable against the rapid-mlx serve --help output of the shipped 0.11.3 build.

MTP · native Multi-Token Prediction heads

For models with built-in MTP heads (Qwen 3.5 / 3.6 native checkpoints per the 0.9.13 blueprint, and DeepSeek V3 / V4). Uses cache snapshot / restore for speculative generation.

$ rapid-mlx serve qwen3.6-35b-8bit \
    --speculative-config '{"method":"mtp","num_speculative_tokens":3,"disable_auto_k":false}' --force-spec-decode

Empirical: 0.9.13 gemma-4-12b-it MTP shipped +14.4% structured throughput with 5/5 determinism gates PASS. K histogram was balanced across the tested prompt classes — see specdecoding.md for the latest MTP validation memo (2026-07-06: Gemma 4 assistant-sidecar MTP is not supported after greedy-lossless correctness failed on 3 of 4 prompts).

DSpark · DeepSeek V4's checkpoint-native drafter

DeepSeek ships the drafter inside the V4 Flash 0731 checkpoint — three MTP stages plus low-rank Markov heads. There is no sidecar to download and no separate draft model to configure: if you have the checkpoint, you have the drafter. Added in 0.11.6.

# Point serve at the local checkpoint directory, not the alias — see below.
$ rapid-mlx serve /path/to/DeepSeek-V4-Flash-0731-MXFP4-MLX \
    --speculative-config '{"method":"dspark","num_speculative_tokens":5}' --port 8000

Detection is fail-closed. The geometry (n_mtp_layers, dspark_block_size, dspark_target_layer_ids, dspark_markov_rank) is read from the checkpoint, and every required tensor is verified against model.safetensors.index.json — so a conversion that stripped the MTP weights refuses to start rather than quietly decoding without a drafter. Observed acceptance on real Codex /v1/responses tool rounds during development: 2.6–2.9 accepted tokens per round.

DFlash · block-diffusion drafter

Block-diffusion speculative drafter, built on the block-diffusion language-modelling line of work — Arriola et al., "Block Diffusion: Interpolating Between Autoregressive and Diffusion Language Models" (arXiv 2503.09573, ICLR 2025). Single-user serial mode. Requires a DFlash-eligible alias — check with rapid-mlx info <alias>.

Measured effect: 1.4× pooled on qwen3.5-27b-8bit, verified lossless. An earlier 3.5× figure came from a broken bench and was retracted. You may also find a 2.6–3.2× community result on qwen3.6-35b-a3b-8bit (issue #472) — that was a single run, and it is not reachable today: the alias is MoE, and MoE models do not pass the eligibility gates below.

Eligibility gates surfaced by rapid-mlx info: declared support, non-MoE, precision ≥ 8-bit, drafter declared, mlx-vlm 0.5.0+ present. The 0.11.3 rapid-mlx info qwen3.5-27b-8bit shows DFlash-eligible; the same output for gemma-4-12b-4bit shows ineligible (4-bit precision + no declared drafter).

DDTree · draft tree verification (experimental)

DFlash draft-tree verification, single-user serial mode. Requires a DDTree-eligible alias and the external dtree-mlx runtime.

SuffixDecoding · drafter-free statistical drafts

Model-free speculative decoding using suffix trees built from past outputs (NeurIPS 2025 Spotlight, arXiv 2411.04975). Explicit opt-in — useful for long high-overlap workloads (prompt-copy, code editing, repeated tool XML) on validated aliases; local benches showed regressions on gpt-oss / Qwen families under mixed chat traffic. Auto-disabled on hybrid models (Qwen 3.5 / 3.6 A3B / A10B, Granite 4, Mamba / Jamba / RWKV).

$ rapid-mlx serve gemma-4-12b-4bit \
    --speculative-config '{"method":"suffix","num_speculative_tokens":8}' --force-spec-decode

Per-alias suffix tier is surfaced by rapid-mlx info <alias> (values: validated, rejected, unknown). "Unknown" means the bench scripts/bench_suffix_decoding.py hasn't been run for the alias yet — it's safe to try but not guaranteed.

PFlash · long-prompt prefill compression

Scores 32K+ prompts and only prefills the sink + recent tail + query-relevant middle blocks. Default-on for verified Qwen 3.5 / 3.6 aliases (per PR #287), off elsewhere. Bench-validated profile (PR #649): TTFT 3.87×–8.5× faster, needle-in-haystack recall 5/5 across tested cells. Read that recall figure for what it is — five trials with zero failures, which bounds the failure rate only loosely (95% upper bound ≈ 45%). It is evidence that PFlash is not grossly lossy, not evidence that retrieval is safe on your workload. PFlash discards ~80% of middle-prompt tokens, so if exact mid-document recall matters to you, benchmark it before relying on it — or set --pflash off.

TurboQuant · KV-cache compression

Post-quantization KV-cache compression. v4 is V-only 3–4 bit Lloyd-Max with K in FP16; k8v4 is the R15 Phase 4 mix — K at 8-bit Walsh-Hadamard + V at 4-bit Lloyd-Max (~4.6× KV compression on dense models, ~1/2.4 = ~58% savings on the shipped MoE cells). k8v4 is default-on for 9 verified Qwen 3.5 / 3.6 MoE aliases per the 0.9 changelog.

Prompt & response caching

Two opt-in caches added in 0.10.12, both default-OFF and byte-for-byte inert until you pass their flag. The response cache short-circuits a completely repeated greedy request and decodes nothing; trim-free prefix reuse keeps a stable prefix warm for hybrid / sliding-window models so a new suffix each turn skips re-prefilling the prefix. These two rows are verified against the rapid-mlx serve --help of the 0.10.12 build.

Separately, the prefix cache is on by default and is what makes a repeated prompt cheap. Its measured effect scales with prompt length, so there is no single multiplier: a short prompt saves little, while an exact-repeat 8,131-token DeepSeek V4 prompt went from ~24.6 s to 0.318 s (81.2× vs cold) with byte-identical output in 0.11.6. That number is one prompt on one machine — see the changelog for the conditions. Cross-family cached-TTFT gains against upstream mlx-lm are much smaller, typically 1.2×–2.1×.

Eligibility check

rapid-mlx info <alias> is the source of truth for what applies to a given model. Example output for gemma-4-12b-4bit in the shipped 0.11.3 build:

$ rapid-mlx info gemma-4-12b-4bit
  Alias: gemma-4-12b-4bit → mlx-community/gemma-4-12B-it-4bit

┌──────────────────────────────────────────────────────────────┐
│ Model: mlx-community/gemma-4-12B-it-4bit                     │
│ Tool format      : gemma4                                    │
│ Reasoning parser : gemma4                                    │
│ Architecture     : pure attention                            │
│ Spec decode      : ✓ supported                               │
│ Suffix tier      : unknown — run scripts/bench_suffix_decod… │
└──────────────────────────────────────────────────────────────┘

┌──────────────────────────────────────────────────────────────┐
│ DFlash eligibility: ✗ ineligible                             │
│ Declared support  : ✗ no                                     │
│ Not MoE           : ✓ yes (dense)                            │
│ Precision ≥8-bit  : ✗ no (4-bit/mxfp4/nvfp4)                 │
│ Drafter declared  : ✗ no (dflash_draft_model unset)          │
└──────────────────────────────────────────────────────────────┘

See also