Roadmap
The 0.9.x phase plan, sourced from a six-scout source survey at the 0.8.19 → 0.9.0 boundary. Each phase aims to land within a clean release rather than getting absorbed into a long-running branch. Most of Phase 0 / 1 / 2 / 4 shipped in 0.9.9 — see the changelog; items still open are marked below.
Differentiation thesis
MLX serving is bifurcating, not converging with vLLM. The
mlx-lm core stays lean (the 0.31.x line is correctness
and sampling work, not serving features). Serving features have
migrated out to downstream servers — Ollama-MLX,
vllm-mlx, LM Studio's MLX engine, oMLX, and Rapid-MLX. Apple has
shipped hardware (M5 NAX) and low-level APIs
(Metal 4 TensorOps, MX dtypes in macOS 27 beta) but no MLX-side
surface yet to drive them.
Rapid-MLX's differentiation is the middle path: deeper than Ollama-MLX's preview, lighter than vllm-mlx's full port, willing to vendor unmerged kernels (TurboQuant KV, disk-KV-checkpoint, MTP speculative decode) before they land upstream. We follow a "rural surrounds the city" model — we will not out-ship the first-party documentation for Qwen / Gemma / GLM hero models; instead we ship the standout niche models first and shore up the serving substrate underneath them.
Phase 0 — Almost-free wins shipped in 0.9
| task | expected impact | status |
|---|---|---|
Default --kv-cache-dtype int4 via mlx_lm.QuantizedKVCache |
+1.1% tok/s & 4× context room in the memory-bandwidth-bound regime | Superseded by K8V4 default-on (Phase 4 shipped early). |
Floor-only bump mlx 0.31.2 + mlx-lm 0.31.3 (keep the cross-thread default_stream shim) |
5–15% decode + JIT-compiled qmm kernels + Qwen3.5 pipelining | Shipped. |
| MXFP4 + MoE + multi-device defensive guardrail (mlx#3402, mlx#2962) | Prevent 0.27 tok/s regressions on GLM / DeepSeek / Kimi MoE | Shipped (load-time detect + warn). |
Phase 1 — Infrastructure unlocks shipped in 0.9
| task | expected impact | status |
|---|---|---|
KV cache supports arbitrary position_ids (not just monotonic offset) |
Unlocks every tree-based speculative decode (EAGLE-3, DFlash, DDTree); also DSA per-token gather. | Shipped. |
| Radix-tree prefix-cache index + system-prompt dedup | 5× multi-tenant throughput (SGLang HiRadix), 30–80% pflash footprint reduction. | Shipped & default-on. Measured 13× aggregate throughput at 10 concurrent clients. |
| Disk-backed KV checkpointing at 256-tok boundaries (port LM Studio MLX-engine #326, MIT) | 2.2× parallel chat throughput, 82% peak RAM reduction at long context; fixes Gemma 4 sliding-window + Qwen3.5 hybrid attention non-rewindable cache. | Shipped (opt-in via --enable-disk-kv). |
Phase 2 — Speculative-decode trio MTP + DFlash shipped in 0.9
| task | expected impact | status |
|---|---|---|
| Vendor mlx-lm PR #990 MTP speculative decode for Qwen3.5 / 3.6 | 1.57× decode tok/s (M4 Pro Qwen3.5-27B w4), 85% accept, no draft model needed. | Shipped & verified lossless. Enable with the qwen3.5-9b-mtp-4bit sidecar alias. |
| DFlash-MLX (block-diffusion drafter, 16 tok/pass, Apache-2) | 4.37× lossless (M5 Max Qwen3.5-9B w4, 30.95 → 135.34 tok/s). | Shipped & verified lossless. Measured 1.4× pooled on Qwen3.5-27B-8bit — the 3.5× number was from a broken bench and has been retracted. Flag-gated: pip install 'rapid-mlx[dflash]' + --enable-dflash. |
| DDTree-MLX for Tmax architecture (needs a ~1B Tmax drafter trained first) | +10–15% on top of DFlash on code / structured output. | Not started — needs a Tmax drafter checkpoint first. |
Phase 3 — Model alias integrations (parallel)
| model | 256 GB Mac Studio fit | status |
|---|---|---|
| Mistral-Small-4-119B-2603 (Apache-2, 4bit ~60 GB / 8bit ~120 GB) | Easy fit. | Shipped in 0.9.9 as mistral-small-4-119b (4bit / 8bit). |
| Holo3.1-35B-A3B | Fits. | Shipped in 0.9.9 as holo3.1-35b-a3b (4bit / 8bit). |
Phase 4 — TurboQuant KV shipped in 0.9
- 4.6× KV compression at K8V4 mixed (vLLM 2026-05-11 verified upstream).
- Vendored mlx-lm PR #1067 + arozanov's fused Metal kernel (turboquant-mlx).
- Decode 0.98× with fused kernel; 0.5× without — only the fused path ships.
- Stacks with the radix-tree prefix cache — measured 13× aggregate throughput at 10 concurrent clients on the K8V4 hero aliases.
- Default-on for 9 hero aliases: Qwen3.5-9B / 27B and the Qwen3.6-35B-A3B family (4 / 6 / 8-bit + DWQ). Opt out with
--kv-cache-turboquant none. - Skipped on sliding-window models (Gemma 3, GPT-OSS) — explicitly unsupported.
- Skipped on MLA models (DeepSeek V3+, Kimi-K2.5 FlashMLA) — K projection is already compressed.
Phase 5 — Stretch & nice-to-have
- PonyExl3 EXL3 trellis quant on Metal (Apache-2, v0.3.0 2026-06-20) — opt-in for users who want 2.5–4 bpw; 4.15 bpw Qwen3.6-27B at KLD 0.144 vs bf16.
- M5 NAX dtype-fallback audit — telemetry counter for matmul shapes that miss NAX, surfaced in
/metrics. - ResComp GPTQ calibration swap — once paper numbers are independently verified, re-quantize the hero MLX models.
Open invitations
Anything on this list could land sooner with the right help — especially the Phase 1 KV / radix work and the Phase 2 speculative decode trio. Issue tracker and contribution guide are on GitHub.