0.12.4 — Caches that were quietly not caching

Released 2026-08-04 · full changelog · GitHub releases

Four of these are the same shape: a cache that reported success while storing nothing, or storing the wrong thing. None of them raised an error — they just made every turn pay full price. Plus an 8 GB Mac finally gets an answer instead of a rejection.

• An 8 GB Mac gets a recommendation instead of a rejection. Every previous pick was refused by auto-start on the smallest Macs, so the tier existed only to say no. It now recommends lfm2.5-2.6b-4bit — about 2.0 GB resident, fast, and explicitly not a coding model. Serving it required teaching the alias resolver about Hugging Face repos that ship every quantization in a subdirectory of one repo: LiquidAI/LFM2.5-2.6B-MLX holds eight, so a naive pull fetched ~20 GB to use 1.6 GB. The curl | bash banner now recommends exactly what the desktop app recommends, with one test that fails if the two ever drift apart (PRs #1442, #1443, #1447).
• --enable-prefix-cache was a silent no-op on the dense Qwen 3.5 / 3.6 aliases and Ternary Bonsai. These models carry non-trimmable recurrent layers whose state is reusable only through the bounded snapshot path — but the auto-default that enables that path keyed on a hybrid flag these models are deliberately pinned off, so it skipped exactly the models that needed it. Every store was dropped and a multi-turn tool agent re-prefilled its whole accumulated context every turn: turn-2 TTFT 22.3 s on qwen3.5-9b-4bit. Routing is unchanged; only the reuse path is switched on (PR #1445).
• A three-token drift at a message boundary could discard a 40K+ reusable prefix. Tokenization is not compositional: appending the next assistant or tool segment can change the previous prompt's final few byte-BPE tokens. A non-trimmable cache cannot trim to absorb that, so the whole snapshot was thrown away. An 8-token replay window covers the drift observed on Codex turns at negligible re-prefill cost (PR #1449).
• Shutdown persisted the shallowest prefix first. The save has a short deadline and may commit only one entry; LRU order could start with a one-token bootstrap entry, spend the single guaranteed slot on it, and skip the real 100K+ frontier. Deadline-aware saves now go longest-first (PR #1451).
• The repetition guard answered a runaway loop with 503 and discarded the partial output. Worse, 503 is the same code as a genuine Metal-runtime abort — where “retry smaller” is correct advice — so agent frameworks read it as a transient outage and re-sent the identical prompt straight back into the loop. The guard stop is now distinguishable from a runtime abort and returns the tokens already generated. Measured on qwen3.5-4b-4bit, a 30-turn tool-agent loop hit this twice; larger models, not at all (PR #1450).
• DeepSeek V4 long decodes could exhaust Metal's resource count while memory looked healthy. The V4 cache updates functionally, so without realizing each forward's leaf arrays the decode retained one live lazy-graph chain per layer per token — hitting the fixed 499000 resource limit with plenty of bytes free (PR #1448).
• DeepSeek V4 could reopen thinking after it had been turned off, re-entering the reasoning parser mid-turn and leaking reasoning into content. Prompt shaping cannot enforce this; it is now suppressed at decode time. V4 also now accepts the r:-prefixed DSML tool aliases it actually samples, instead of forwarding them as malformed calls (PRs #1452, #1446).
• The desktop app's memory column was two different metrics wearing one hat. Some rows were a bare-mlx_lm allocation high-water mark, others were what rapid-mlx serve actually uses — and serve quantizes the KV cache to int4 by default, worth roughly 2 GB on a 27B. Re-measured through serve on one machine, the tier table, the hardware-tiers doc and the blog now agree (PR #1453).
• The first text-to-speech request could 500 when Kokoro's spaCy G2P model had not been resolved yet; it is now pre-resolved at the route gate (PR #1254).
• Benchmark artifacts carry a schema version and a methodology hash, so results produced under different methodologies can no longer be silently aggregated together (PR #1455).