0.12.11 — Faster under load, faster alone

Released 2026-08-12 · full changelog · GitHub releases

Upgrade: pip install -U rapid-mlx · brew upgrade rapid-mlx · or grab the desktop app.

A bench-driven performance release: concurrent throughput climbs 8–41% depending on the model, a lone request gets its own KV-cache fast path, and forced tool calls always arrive as valid JSON objects.

Concurrent requests prefill as one wave. The hybrid admission throttle is retired — requests that arrive together are admitted together instead of being staggered 200 ms apart, which kept batched attention off its fast path for the batch’s whole lifetime (#1866). Measured on an M3 Ultra: Qwen3.6-35B-A3B-8bit at 8 concurrent streams goes 190 → 268 aggregate tok/s (+41%); Qwen3.5-4B +20%, 9B +13%, gpt-oss-20b +12%, 27B +8%.
Deep batches coalesce scheduler steps. With four or more running requests the engine runs up to 4 scheduler steps per dispatch, cutting per-step executor round-trips (~10 ms/step at high aggregate rates). Finishes, pending admissions, and the memory-pressure check cadence are preserved (#1878).
A lone request keeps a singleton KV cache. Single-stream decode skips batched-form bookkeeping — plain causal mask straight onto the native SDPA fast path, and the dense-sampler fast path now engages at batch size 1, promoting to batched caches only when a second request joins (#1874). Qwen3.5-4B single-stream: 174.8 tok/s at short context, 147.5 at 16k.
Forced tool calls always carry object arguments. Under tool_choice, a weak continuation could produce a bare scalar ("arguments": "1") — shipped as-is on the non-stream path and silently dropped as an empty turn when streaming. Both paths now repair to the OpenAI wire contract, with schema-required violations still surfacing explicitly, and tool logits processors are no longer built for plain-chat requests (#1880, #1878).
Reliability. Prefix-cache entries can no longer be corrupted by the new singleton lane (loaned state is copied on admission); coalesced-step errors deliver already-produced tokens before surfacing; admission-wave timing survives malformed tuning env vars (#1874, #1866, #1878).
Catalog. New alias nemotron-3.5-lightning-30b-4bit joins the Nemotron line (170 text aliases, 219 total).