Rapid-MLX
|
Issue #2 |
|
|
Your local models on Mac
got faster.
You don't have to do anything.
Four qualified Qwen builds now select their validated speculative-decoding preset and the continuous scheduler on their own, whenever you don't pass a speculative option yourself. Same command, same model, more throughput under concurrent work.
|
|
Aggregate throughput change across mixed four-request cohorts. No ordinary-pass case became a continuous-mode failure. Prefer the old behaviour? --no-spec-decode — and Desktop has the same switch.
|
|
Download for Mac →
Free · macOS 14+ · Apple Silicon · updates itself
|
|
already installed? one line
rapid-mlx upgrade
|
|
01 / LONG CONTEXT
The longer the conversation, the bigger the win
Speculative decoding usually helps least where you need it most. Here it's the other way round — the gain grows with the conversation.
That's the shape you want for agent work, where the context only grows.
|
|
02 / VIDEO
Video generation, on your Mac
Two shots from one 9-second clip · 1920×1080 · Desktop Video workspace
Turn on Video Generation under Experimental Features and you get a real workspace — queued jobs, live progress, cancellation, and completed results that survive a restart. The signed app already contains the audited runtime and encoder, so there's nothing to install afterwards.
LTX 2.5 q8
Text or image to video.
CogVideoX-Fun 5B q4
Text to video.
Wan 2.1 1.3B bf16
Text to video. Needs a Mac with at least 40 GB of unified memory.
It's experimental and labelled as such. Tell us what breaks.
|
|
03 / BENCHMARK
Benchmark your Mac. Share nothing by default.
rapid-mlx benchmark and the new Community Benchmark page plan, run, validate and archive text, image and video measurements entirely on your machine.
Results stay local unless you explicitly choose to share them. If you do, the consent screen shows you the exact destination and the payload digest first — and a durable receipt stops you submitting the same run twice by accident.
|
|
04 / YOUR REPORTS
You reported these. They're fixed.
The best bug reports this cycle came with a stack trace, a hardware line and a reproduction. Thank you — genuinely.
Tool calls no longer 503 when speculative decoding is on.
Any request carrying a tools array could crash cleanup and fail persistently. Reported by @korellas and independently confirmed by @TomSchuman on a 512 GB Mac Studio, who had been running production with speculative decoding switched off as a workaround since September 1. You can switch it back on.
Two concurrent requests no longer take each other down.
When one request qualified for speculative decoding and another didn't, both used to abort with a 503. The scheduler now keeps ownership until the first request departs, and the second one simply waits.
Bad cache configurations fail before Ready, not mid-flight.
Explicit quantized-KV and paged-cache options on unsupported model layouts used to start healthy and then fail every request. They're now rejected up front with an error that tells you what to change.
Gemma 4 reuses your conversation prefix again.
A growing chat on the default vision-capable lane reported zero reuse. In the maintainer's reproduction it now reports 10,695 cached tokens on turn two — 1.526s down to 0.209s for the same answer.
And a thank-you to Pierre Lamy
whose community contribution helped move this speculative-decoding and performance work forward. The speed at the top of this email is partly theirs.
|
|
While we're here — the same app makes these
Not new in 0.13.4, just easy to forget: image generation runs on the same install, offline, with no key and no per-image cost.
FLUX.2 klein 4B · 1024×1024 · 9 seconds each on a Mac Studio · nothing left the machine
|
|
05 / SMALL THINGS
And the smaller things you'll actually notice
It tells you why a model won't fit.
Memory limits are tuned from your Mac's real available memory, and a rejected launch returns fit guidance instead of failing deep inside model loading.
Always-on, headless.
A documented LaunchDaemon deployment for Macs that serve rather than sit on a desk — same user-owned cache, same service boundaries.
Memory across conversations, if you want it.
Opt-in, and editable or removable in Settings at any time.
Mermaid diagrams render inline.
In an isolated preview with the network blocked.
Dictation feels quicker on first use.
The speech model loads while you're still talking.
Faster still, if you like living on the edge.
An opt-in fused kernel for Qwen3.8 Flash-Next improved end-to-end decode from 25.43 to 27.04 tok/s on the qualified M3 Ultra run, with bit-identical output. Off by default until more hardware reports back.
|
|
|
That's 0.13.4. If something on this list is broken on your Mac,
open an issue — that's how most of section 04 got written.
|