rapid-mlx blog

Rapid-MLX 0.11.0 — 25.6× faster to first token, meet your Mac's new agent

2026-07-24T12:00:00Z

Rapid-MLX 0.11.0 is out today. The headline numbers are about speed; the headline story is about what that speed unlocks: a built-in chat agent that chains real tools, five new model families supported on day one, and a reliability pass that makes the whole thing boring in the best way.

Install it in one line:

brew install rapid-mlx        # now in homebrew-core
# or: pip install -U rapid-mlx / uv tool install rapid-mlx

⚡️ Performance: stop paying prefill twice

All numbers below are single-request wall-clock measurements on an M3 Ultra from this week's dogfooding — not synthetic peak-decode figures.

Prefix-cache reuse: 13.1s → 0.51s TTFT (25.6×). Repeat a 6k-token prefix — a shared system prompt, a long agent context — and the second turn reuses the KV instead of recomputing it. One test session saved 38,032 tokens of prefill. For multi-turn chat and agent loops, this is the single biggest latency win we've shipped.
Response cache: 656ms → 2ms (284×). An identical greedy request comes back byte-identical in 2 milliseconds with zero GPU decode. Opt-in, and ideal for eval loops and CI.
Throughput: 152 tok/s on Qwen3.5-4B-4bit, 104 tok/s on gpt-oss-20b, ~100 tok/s on Gemma-4-26B-4bit — wall-clock, TTFT included.
Cold starts in seconds: ~2s for a 4B model, ~3s for gpt-oss-20b.
Bounded memory on long prompts: chunked text-lane prefill keeps a 19k-token prompt at a 17.1 GB peak.
KV cache quantization: int4 KV cache is the M-series default; int8 override verified.

🤖 rapid-mlx chat grew into a real agent

chat started as a REPL. This cycle it became an agent we're genuinely happy with — simple, but neat:

A real MCP agent loop. Point it at any MCP server and it chains tools on its own. In testing against the standard filesystem server — 14 tools exposed — it autonomously ran list_directory → read_text_file and answered from the file contents, with a live tool-activity UI and per-call timing. 2.8 seconds, clean exit.
Thinking budgets. reasoning_max_tokens force-closes the <think> block at decode time: set a 24-token budget and a model that would reason for 3,396 characters wraps up in ~90 — and still answers. Verified across all four reasoning families we tested.
Zero ceremony. It spawns the server for you, gives you /model /save /reset /help, and gets out of the way. --base-url mode attaches to an already-running server as a pure MCP client.

🧠 Five new model families, day one

165 aliases, 16 tool-call wires, ~18 model families — dense, MoE, hybrid-linear, ternary, vision, and audio:

HY3 (Tencent Hunyuan 3) 295B MoE — our fifth Tier-1 family, with native MTP self-speculative decoding (~1.5–1.7× decode)
Qwen3-Coder-Next 80B — day-one 4bit alias
MiniCPM5 — including a brand-new tool-call wire
LiquidAI LFM2.x
Ternary Bonsai 1.7B-2bit — yes, 2-bit, and it runs
Audio round-trip: Kokoro TTS → Parakeet STT, verified word-perfect, both services up in under 4 seconds
Vision: Qwen3-VL reads controlled test images accurately — all 5 ground-truth features in 2.1s
Also new: KV cache export/import API, and structured output migrated to llguidance (−200 MB install, native Metal token bitmask)

🛠️ Tool calls, schema-valid by construction

One paragraph, because it just works now: agent loops used to die on almost-JSON tool arguments, so we constrain decoding at the grammar level — the model cannot emit a malformed tool call. It's verified across all four major wire formats (harmony/gpt-oss, qwen3_coder_xml, gemma4, deepseek_v3) plus hermes, for auto, forced, and required modes, on by default with zero flags. The 0.11.0 keystone: the guarantee now holds for reasoning models too — models that <think> first used to break forced calls because the grammar engaged at the wrong offset. Not anymore. (Engineering-minded readers: this was the hard part. Everyone else: you'll never notice it, which is the point.)

🛡️ Reliability and honest gaps

Every release goes through an automated dogfood battery — 5 models × 11 checks covering streaming, reasoning, forced tool calls, error paths, and performance — on real hardware. The 0.10.16 round caught three P1 regressions (a Gemma-4 required loop, a flagship-serve crash, a silent schema fallback); all fixed and re-verified within 48 hours. Error paths are now honest across the board: invalid schemas → 400, unknown models → 404 with the list of what is available. No silent fallbacks, ever.

Honest gaps, because you'd find them anyway: Qwen3.6 native MTP is parked (its SSM-hybrid architecture clamps speculative K to 1 — the engine refuses loudly rather than silently degrading), the MCP config key is servers rather than the ecosystem-standard mcpServers (fix queued), and we haven't run a same-machine head-to-head against llama.cpp/Ollama this cycle. That's next.

Get it

brew install rapid-mlx
rapid-mlx chat qwen3.5-4b-4bit

Rapid-MLX is open source (Apache-2.0), MLX-native, and built for people who want their agents fast, local, and correct. 3,400+ of you have starred it — thank you. The best thing you can do for the project is break it and tell us: issues and PRs are answered fast.

All performance figures are single-request wall-clock measurements on an Apple M3 Ultra (256 GB), macOS 26.5.2, from 2026-07 evaluation runs. tok/s includes TTFT and is not comparable across quantization levels or architectures.

Moving from Ollama to rapid-mlx on Apple Silicon

2026-07-21T12:00:00Z

If you run local models on a Mac with Ollama and you're curious about more speed on Apple Silicon, a drop-in Anthropic API (not just OpenAI), or an Apache-2.0 engine — moving to rapid-mlx is mostly muscle memory. The commands you already type map almost one-to-one, and because the two servers listen on different ports, you can run both side by side and migrate at your own pace. Nothing to rip out.

Should you switch?

Straight, no spin:

What rapid-mlx gives you here: faster inference on Apple Silicon, a drop-in **OpenAI and Anthropic** API (so Claude Code, Cursor, Codex, and OpenAI clients all work), Apache-2.0, and newer model generations out of the box (Qwen 3.6, Gemma 4).
Where Ollama still wins: it's cross-platform (Windows/Linux too), and its model library is larger. If you need a non-Mac box, or a specific model rapid-mlx doesn't carry, stay on Ollama — or run both.

That last point is the theme: this isn't a burn-the-boats migration.

The commands are almost the same

rapid-mlx deliberately mirrors Ollama's verbs, so most of your muscle memory just works:

You typed (Ollama)	Now type (rapid-mlx)	What it does
`ollama run <model>`	`rapid-mlx run <model>`	interactive chat REPL
`ollama pull <model>`	`rapid-mlx pull <model>`	download a model
`ollama rm <model>`	`rapid-mlx rm <model>`	delete a cached model
`ollama ps`	`rapid-mlx ps`	show running models
`ollama list`	`rapid-mlx models`	list available models
`ollama serve`	`rapid-mlx serve <model>`	start the local server

One real difference: ollama serve starts a model-agnostic daemon, while rapid-mlx serve <model> boots a server for a specific model at http://localhost:8000. (run still auto-loads on demand, Ollama-style.)

Model-name cheat sheet

Model names don't carry over verbatim — and that's mostly good news, because rapid-mlx ships newer generations than what a lot of Ollama setups are pinned to. Reach for the equivalent:

On Ollama	On rapid-mlx (recommended)
`qwen2.5:3b`	`qwen3.5-4b-4bit`
`qwen2.5:7b`	`qwen3-8b-4bit` / `qwen3.6-35b-4bit`
`gemma2:9b`	`gemma-4-12b-4bit`
`gemma2:27b`	`gemma-4-26b-4bit`
`llama3.1:8b`	`qwen3.6-35b-4bit` or `gemma-4-12b-4bit` (Llama 3 ships as `llama3-3b-4bit`)
`deepseek-r1:8b`	`deepseek-r1-8b-4bit`
`deepseek-r1:32b`	`deepseek-r1-32b-4bit`
`mistral`	`mistral-24b-4bit`
`phi3`	`phi-3.5-mini-4bit`
`gpt-oss:20b`	`gpt-oss-20b`
_tiny, punches up_	`bonsai-27b-2bit` — a 27B in 7.9 GB, runs on a 16 GB Mac

Run rapid-mlx models for the full list, or browse the model mirror with a RAM filter.

Point your tools at the new port

Anything you have wired to Ollama's OpenAI-compatible endpoint just needs a new base URL:

Ollama:     http://localhost:11434/v1
rapid-mlx:  http://localhost:8000/v1

That covers most modern setups — Cursor, Continue, LangChain, and any OpenAI client. (Claude Code is even easier — it speaks the Anthropic wire; see the Claude Code recipe.)

The one real gotcha: rapid-mlx serves the OpenAI (/v1/chat/completions) and Anthropic (/v1/messages) APIs — not Ollama's native /api/generate · /api/chat · /api/tags. If a tool is hardcoded to Ollama's own API, point it at the OpenAI path instead. Most tools already default to OpenAI-compatible, so this is usually a non-issue.

Run both side by side while you migrate

Because the ports don't collide, keep Ollama on 11434 and bring rapid-mlx up on 8000:

# terminal 1 — your existing Ollama, untouched
ollama serve            # :11434

# terminal 2 — rapid-mlx alongside it
rapid-mlx serve qwen3.6-35b-4bit    # :8000

Move one tool over, use it for a day, and if anything's missing you haven't lost your old setup. When you're confident, stop Ollama — or don't. Nothing forces the choice.

Honest expectations

A local model is a local model on either engine — neither is a frontier cloud model. What you're actually buying by switching on Apple Silicon is speed, the Anthropic API surface (which unlocks Claude Code locally), Apache-2.0, and current-gen weights. What you'd give up is cross-platform support and the breadth of Ollama's library. For most Mac-based, OpenAI/Anthropic-tool workflows, the move is a few minutes and reversible — which is the whole point of doing it side by side.

Where to go next

Run Claude Code on a local model · Cursor · Codex · Aider
Browse the model mirror — RAM filter to pick the right replacement model.
Install rapid-mlx — Homebrew or one curl. Apache 2.0.

Run Aider on a Local Model — Pair-Program Offline on Your Mac

2026-07-21T12:00:00Z

Aider is a terminal-first pair-programmer: it reads your file, sends it to the model with a plain-English instruction, and applies the returned edit blocks locally. It drives models through LiteLLM, so any OpenAI-compatible endpoint works — including a rapid-mlx server on your own Mac.

Fourth in the series on running your favorite agent locally (after Claude Code, Cursor, and Codex). About five minutes.

Your code stays local. Aider stops sending your files to the cloud.
No meter. Edit all day for the cost of electricity.
Forgiving with small models. Aider uses SEARCH/REPLACE edit blocks, not function calling — so even reasoning-only models pass. The signal is "did it rewrite the file correctly," and that works broadly.

The whole thing

pip install aider-chat
brew install rapid-mlx → rapid-mlx serve qwen3.6-35b-4bit
export OPENAI_API_BASE=http://localhost:8000/v1

1 · Install Aider + rapid-mlx

pip install aider-chat
brew install rapid-mlx    # in Homebrew core — no tap, no trust

2 · Serve a model that fits your Mac

rapid-mlx serve qwen3.6-35b-4bit

Your Mac	Model	Good for
16 GB	`qwen3.5-4b`	Quick single-file edits
32 GB	`gpt-oss-20b`	All-round
48 GB+	`qwen3.6-35b-4bit`	The coding sweet spot

3 · Point Aider at it

Set two env vars, then run Aider on a file:

export OPENAI_API_BASE=http://localhost:8000/v1
export OPENAI_API_KEY=not-needed

aider --model openai/qwen3.6-35b-4bit main.py

Prefer per-invocation flags instead of env vars? That works too:

aider \
  --model openai/qwen3.6-35b-4bit \
  --openai-api-base http://localhost:8000/v1 \
  --openai-api-key not-needed \
  main.py

Keep the openai/ prefix on the model name — Aider drives LiteLLM, which uses it to route to the OpenAI-shape provider. And the base URL ends in /v1. Both are easy to drop and both cause confusing errors.

4 · Pair on it

> add input validation to parse_config() and a test for the bad-path

Aider reads main.py, proposes the edit as a diff, and applies it in place — locally.

Honest expectations

A model on your Mac isn't a frontier cloud model, but Aider's plain-text edit format is unusually tolerant — it leans on the model rewriting code, not on brittle tool-call JSON. Great for private/offline edits and the everyday 80%; point --model back at a cloud provider for the hard 20%.

Where to go next

Full Aider guide — the LiteLLM openai/ prefix, recommended aliases, and edit-format notes.
Claude Code → local · Cursor → local · Codex → local
Install rapid-mlx — Homebrew or one curl. Apache 2.0.

Run Claude Code on a Fully Local Model — in 5 Minutes

2026-07-21T12:00:00Z

Claude Code is a great agent. But every keystroke, every file it reads, every tool call goes to the cloud — and gets billed per token. For a lot of work — refactors, codebase Q&A, editing files you'd rather not upload — you don't need a frontier model in a datacenter. You need a capable model on your desk.

Here's why the swap is worth it:

Your code stays on the Mac. Nothing is uploaded — works on a plane, behind an air-gap, or on a repo you're not allowed to send anywhere.
No meter running. Once the model is on disk, an all-day agent session costs exactly nothing.
It actually works — not a hack. rapid-mlx speaks Claude Code's native wire (/v1/messages) and ships guaranteed-parseable tool calling, so the agent's edits and tool calls land instead of flailing. More on that below.

The whole setup is about five minutes, most of which is a model download. You'll need an Apple Silicon Mac (M1 or newer).

Before you start

You need Claude Code itself. If you don't have the CLI yet:

npm install -g @anthropic-ai/claude-code

1 · Install rapid-mlx

rapid-mlx is a local, OpenAI- and Anthropic-compatible inference server for Apple Silicon. It's in Homebrew core:

brew install rapid-mlx

Prefer not to use Homebrew? curl -fsSL https://rapidmlx.com/install.sh | bash does the same — no sudo, self-contained.

2 · Serve a model that fits your Mac

Pick a model sized to your RAM and start the server. A solid default for coding is Qwen 3.6:

rapid-mlx serve qwen3.6-35b-4bit

First run downloads the weights (cached after that). The server comes up at http://localhost:8000 with both OpenAI and Anthropic routes.

Your Mac	Model	Good for
16 GB	`qwen3.5-4b`	Fast edits, Q&A
32 GB	`gpt-oss-20b`	All-round + strong tools
48 GB+	`qwen3.6-35b-4bit`	The coding sweet spot

No tool flags needed. Since 0.10.14, serve auto-configures the tool-call parser from the model's profile and turns on grammar-constrained tool calling by default — so tool calls come back guaranteed-parseable, with nothing to configure. (On builds older than 0.10.14, add --tool-call-parser qwen3_coder_xml --enable-auto-tool-choice.)

3 · Point Claude Code at it

Claude Code honors the ANTHROPIC_BASE_URL environment variable. Set it to your local server and launch — that's the whole integration:

export ANTHROPIC_BASE_URL=http://localhost:8000
export ANTHROPIC_API_KEY=not-needed
claude

The base URL is the root, not …/v1 — the Anthropic SDK appends /v1/messages itself. http://localhost:8000/v1 would POST to /v1/v1/messages and 404.

4 · Do something real

You're now running a fully local agent. Point it at a repo and give it a task:

# inside any project
claude
> add a --dry-run flag to scripts/deploy.py and update the README

It reads your files, plans, and edits — all on your machine. Pull the network cable; it keeps working.

Why it just works (and where it doesn't)

Two things make this a real integration rather than a duct-tape proxy:

Native Anthropic wire. Claude Code speaks /v1/messages, not OpenAI's format. rapid-mlx serves that as a first-class route, so it's an env-var flip — no shim, no translation layer to break.
Guaranteed-parseable tool calls. Tool-call output is constrained by a grammar at generation time, so the model can't emit a malformed call. That's the difference between an agent that edits your files and one that stalls on a broken tool block. (Verified on rapid-mlx 0.10.15: a bare serve plus an Anthropic /v1/messages request with a tool returns a proper tool_use block.)

Honest expectations. A 35B model on your Mac is not Claude Opus, and pretending otherwise helps no one. Where local shines: private and offline work, codebase Q&A, straightforward edits and refactors, and never watching a bill. Where the cloud still wins: the hardest multi-step reasoning and the largest context.

The good news — you don't have to choose. Flip the two environment variables off and Claude Code is back on the cloud model. Local for the 80%, cloud for the hard 20%.

Where to go next

Full Claude Code guide — SDK snippet, tool schemas, and every gotcha.
Integration matrix — 11 agents across the four Tier-1 families, tested.
Install rapid-mlx — Homebrew or one curl. Apache 2.0.
Browse models — pick one that fits your RAM.

Using Cursor, Codex CLI, or Aider instead? Same idea — each has a one-page guide in the docs.

Run Codex CLI on a Local Model — OpenAI's Agent, on Your Mac

2026-07-21T12:00:00Z

Codex CLI is OpenAI's official terminal agent — a fast, Rust-based coding loop. It talks to api.openai.com over the Responses API. The nice part: rapid-mlx exposes that exact wire (/v1/responses) as a first-class route, so you can point Codex at a model on your own Mac and it behaves the same — just local.

Third in the series on running your favorite agent locally (after Claude Code and Cursor). About five minutes.

Your code stays local. Nothing leaves the Mac.
No meter. Once the model's on disk, an all-day session is free.
Same wire, not a shim. rapid-mlx speaks /v1/responses natively — you're changing one base_url.

The whole thing

brew install codex (if you don't have it)
brew install rapid-mlx → rapid-mlx serve qwen3.6-35b-4bit
rapid-mlx agents codex --setup — writes the Codex config for you

1 · Install Codex + rapid-mlx

brew install codex        # or: npm install -g @openai/codex
brew install rapid-mlx    # in Homebrew core — no tap, no trust

2 · Serve a model that fits your Mac

rapid-mlx serve qwen3.6-35b-4bit

Your Mac	Model	Good for
16 GB	`qwen3.5-4b`	Fast edits, Q&A
32 GB	`gpt-oss-20b`	All-round + strong tools
48 GB+	`qwen3.6-35b-4bit`	The coding sweet spot

3 · Point Codex at it

The one-shot writes ~/.codex/config.toml for you:

rapid-mlx agents codex --setup

Or drop it in by hand:

# ~/.codex/config.toml
model = "default"
model_provider = "rapid-mlx"

[model_providers.rapid-mlx]
name = "Rapid-MLX (local)"
base_url = "http://localhost:8000/v1"

base_url must include the trailing /v1 — Codex speaks the Responses API and appends /responses itself. rapid-mlx is unauthed by default, so there's no api_key line (Codex's --strict-config rejects an inline literal). If you started the server with --api-key, add env_key = "RAPID_MLX_API_KEY" and export it.

4 · Run

codex
> refactor the auth middleware in server.py and add a test

Codex plans, edits, and runs — all against the model on your Mac.

Honest expectations

A local model isn't a frontier cloud model. Great for private/offline work and the everyday 80% — refactors, tests, codebase Q&A — with no bill. For the hardest reasoning, switch model_provider back to OpenAI. Two configs, your choice per task.

Where to go next

Full Codex CLI guide — the config block, auth, and the /v1/responses route in detail.
Claude Code → local · Cursor → local
Integration matrix — every agent × the four Tier-1 families.
Install rapid-mlx — Homebrew or one curl. Apache 2.0.

Run Cursor on a Local Model — Point the IDE at Your Own Mac

2026-07-21T12:00:00Z

Cursor is where a lot of us live now, and its AI is genuinely good. But every completion, every Ctrl-K, every composer run goes to the cloud and gets billed. For a lot of day-to-day editing — refactors, explain-this, small features — you don't need a frontier model in a datacenter. You need a capable one on your Mac, wired into the editor you already use.

This is the second in a short series on pointing your favorite agent at a local model. (First up was Claude Code.) Cursor takes about five minutes.

Your code stays local. Cursor's chat and composer stop shipping your files to the cloud.
No meter. Once the model's on disk, editing all day costs nothing.
It just works. rapid-mlx serves the OpenAI wire Cursor expects — you're changing one URL, not installing a bridge.

The whole thing

brew install rapid-mlx
rapid-mlx serve qwen3.6-35b-4bit
Cursor → Settings → Models → set the OpenAI base URL to http://localhost:8000/v1

1 · Install rapid-mlx

Local, OpenAI-compatible inference for Apple Silicon. It's in Homebrew core:

brew install rapid-mlx

2 · Serve a model that fits your Mac

rapid-mlx serve qwen3.6-35b-4bit

The server comes up at http://localhost:8000. First run downloads the weights; after that it's seconds.

Your Mac	Model	Good for
16 GB	`qwen3.5-4b`	Fast edits, Q&A
32 GB	`gpt-oss-20b`	All-round + strong tools
48 GB+	`qwen3.6-35b-4bit`	The coding sweet spot

No tool flags needed. Since 0.10.14, serve auto-configures the tool-call parser and turns on grammar-constrained tool calling — Cursor's composer, which leans on tool calls, gets clean results with nothing to set.

3 · Point Cursor at it

Open Settings → Models → Add Model, then set:

OpenAI API Base:  http://localhost:8000/v1
API Key:          not-needed
Model name:       default        # or any served alias, e.g. qwen3.6-35b-4bit

The base URL ends in /v1 here — that's the OpenAI shape Cursor speaks. (Claude Code is the opposite: it wants the bare root. Different wires.) And if Cursor's dropdown hides an alias it doesn't recognize, pick default — it always resolves to whatever the server booted.

4 · Use it

Pick default from the model dropdown. Now chat, Ctrl-K, and composer route to http://localhost:8000/v1 — running on your Mac. Ask it to explain a function, refactor a file, or scaffold a component; it reads and edits in the editor as usual.

What this covers — and what it doesn't

Cursor's interactive surfaces — chat, Ctrl-K, and composer — honor the custom endpoint. Cursor's fully-managed background agent routes through Cursor's own backend and can't be pointed at a local server. So this covers the editing you do by hand all day; the autonomous cloud agent stays cloud. For a fully local agent loop, Claude Code or Codex is the better fit.

Honest expectations. A model on your Mac isn't a frontier cloud model. Where local shines: private and offline work, quick edits, explain-and-refactor, and never watching a bill. Where the cloud still wins: the hardest reasoning and the largest context. You don't have to pick one — switch Cursor's model back anytime.

Where to go next

Full Cursor guide — settings, model dropdown notes, and the agent-path caveat in detail.
Claude Code → local — the fully-local CLI agent.
Integration matrix — every agent × the four Tier-1 families.
Install rapid-mlx — Homebrew or one curl. Apache 2.0.

The 5 Best AI Models You Can Actually Run on a MacBook Pro (2026)

2026-07-15T12:00:00Z

There's a model in this lineup that has no business working.

It's 27 billion parameters — the size of a model you'd expect to need a serious desktop for — but someone crushed every weight down to two bits, essentially three possible values each. On paper that should lobotomize it. It fits in 8 GB. It runs on the cheapest MacBook Pro Apple sells. And when I put it through the same gauntlet as four "normal" models, it landed second place overall.

I'll get to which one it is. First, the setup: your MacBook Pro is a genuinely capable private AI workstation. No account, no monthly bill, no network round-trip, nothing you type ever leaving the machine. The only real question is which model to run — and most of the advice out there is either pure vibes or benchmarked on a $6,000 Mac Studio that looks nothing like your laptop.

So I took five local models that actually fit a MacBook Pro and ran every one through identical coding, reasoning, general-knowledge, and tool-calling tests, tracking exactly how much memory each one really eats. Here's the leaderboard, then the story of each model — counted down from the one the internet oversold me to the one that changed what I think a laptop can do.

The leaderboard

Sorted by memory footprint — the one that fits the most Macs is at the top:

Model	RAM (peak)	Decode*	Tools	Code	Reason	General	Avg
Bonsai-27B `ternary`	8.0 GB	48 t/s	93%	80%	70%	80%	81%
GPT-OSS-20B `mxfp4`	12.2 GB	125 t/s	80%	50%	80%	50%	65%
Gemma-4-26B `4-bit`	14.7 GB	110 t/s	87%	90%	100%	60%	84%
Qwen3.6-27B `4-bit`	15.5 GB	37 t/s	93%	100%	50%	70%	78%
Nemotron-Nano-30B `4-bit`	18.0 GB	124 t/s	83%	50%	70%	40%	61%

One honest warning before you read too much into this, because this community checks: each capability score is a 10-question probe (30 for tools), not the full 500-question academic gauntlet. That's plenty to sort these five into tiers and catch a model that face-plants — it is not enough to litigate a three-point gap. Treat ±10% as noise, treat the tiers as real. Every model ran with enable_thinking: false and temperature: 0; the whole harness and all five raw result files are in the repo if you want to check my work or run it on your own Mac.

Now the cast, worst to first.

5. Nemotron-Nano-30B — the one the internet oversold

I went in expecting NVIDIA's Nemotron to win. On the public leaderboards it beats Qwen3 and GPT-OSS on LiveCodeBench; it's the shiny new Mamba-Transformer hybrid everyone's posting about. On my bench it came dead last at 61%, with a brutal 40% on general knowledge.

Before you @ me: yes, I know it's a reasoning model, and yes, I ran it with thinking disabled like everyone else — which hurts a think-first model more than the others. It could also be an answer-extraction quirk on my end. Both are on the retest list with thinking turned on. But here's the honest takeaway for today, on a laptop, out of the box: the model the benchmarks told me to love got quietly outclassed by models half its hype. It's fast (124 t/s) and architecturally the most interesting thing here — it's just not the one I'd actually run yet.

4. GPT-OSS-20B — the reliable commuter car

OpenAI's little open model is the Toyota Corolla of this list, and I mean that as a compliment. It's the fastest genuinely-useful model here (125 t/s), sips just 12 GB, and never embarrasses itself — 80% on tool calling, 80% on reasoning. It's not going to win a drag race on any single skill (50% on coding and general knowledge), but it starts every morning and gets you there. This is why it became the default "just give me something that works" model on 16 GB Macs, and nothing here dethrones it from that job.

3. Qwen3.6-27B — the savant

Every list has one genius with a blind spot. Qwen3.6 posted a perfect 100% on coding and 93% on tool calling — the sharpest pure-code model in the room, tailor-made for driving Aider or Claude Code. Then it turned around and scored 50% on math reasoning, the kind of split that makes you double-check the logs. It's also the slowest thing here by a mile (37 t/s), because it's the only old-school dense model in the group and pays full price for all 27 billion parameters on every single token. A specialist worth hiring — for exactly one job, on a 32 GB Mac.

2. Gemma-4-26B — the quiet valedictorian

While everyone else has a spiky profile, Google's Gemma just quietly does everything well. Highest average on the board at 84%. A perfect 100% on reasoning, 90% on coding, 87% tool calling — the only model here without a real weakness (general knowledge at 60% is its softest, and it's still fine). At 14.7 GB and 110 t/s it's fast and fits a 24 GB Mac. If you want the top raw score and one model that never lets you down, technically this is your winner.

Technically. Because there's one model I'd hand a lot of people instead — even though it scored three points lower.

1. Bonsai-27B — the one that shouldn't exist

This is the 2-bit model from the intro. A 27B brain, ternary-quantized into 8.0 GB — less memory than models with a third of its parameters — and it posted an 81% average, second only to Gemma, while tying for the best tool-calling score in the entire lineup (93%).

I didn't believe it either. A 2-bit 27B keeping pace with 4-bit models nearly twice its size sounds like a broken test, and if you're skeptical, good — go run it yourself, the harness is right there. But the number held up across all four suites, and it reframes the whole question. Every other model on this list makes you choose between "smart" and "fits a normal MacBook." Bonsai is the first one that just... doesn't. It's the only model here a base 16 GB MacBook Pro can run with room to spare — and it's nearly the smartest one too.

It's not perfect: at 48 t/s it's middling on speed (ternary weights don't get the acceleration that MoE models do). But "a genuinely smart 27B that runs on the cheapest MacBook Pro made" is a category that basically didn't exist a year ago. For most people, on the most common Macs, this is the one to run.

So which one fits your MacBook Pro?

The scores are hardware-independent, but the memory is the part that decides whether a model runs on your machine at all. Match the peak-RAM column to your Mac and leave ~20% headroom for macOS and your context window:

Your MacBook Pro	The move
16 GB	Bonsai-27B (8 GB). It's the whole reason 16 GB is enough now.
18–24 GB	Gemma-4-26B (15 GB) for the top score, or Bonsai for headroom.
32 GB	Anything here. Gemma to do everything, Qwen3.6 to write code.
36 GB+	All five, with room for a long context window on top.

Reality check from the machine I wrote this on — a MacBook Pro with an M3 Pro and 18 GB, one of the most common configs Apple ships: it comfortably holds only the two smallest models here, Bonsai and GPT-OSS. Everything from Gemma up is too tight once macOS and a working context are in memory. Which is exactly why that weird little ternary model matters more than its spec sheet suggests.

About that speed column

Two caveats, because they're the honest ones. First, the decode speeds (\*) were measured on a Mac Studio that was busy serving other models at the time — GPT-OSS clocked 125 t/s here versus 124 on a clean run, so they're trustworthy as a floor, but a lab-clean laptop pass is still coming. Second, a Mac Studio (M3 Ultra, ~819 GB/s of memory bandwidth) decodes a lot faster than any laptop. Bandwidth is the lever, so to estimate your MacBook Pro, derate roughly:

Your Mac	Bandwidth	Rough decode vs Studio
M4 Max	~546 GB/s	~0.6–0.7×
M4 Pro	~273 GB/s	~0.4–0.5×
M4 (base)	~120 GB/s	~0.3–0.4×

First community data point (thanks, Lucid). A reader ran the exact harness on an M5 MacBook Air, 16 GB — qwen3.5-4b-4bit decoded at ~47 tok/s, against 158 t/s for the same model on the M3 Ultra. That's the laptop derate above, made real. They also kept every round, and the spread is the actual lesson: short-prompt decode bounced between 38.5 and 48.0 tok/s across five runs, while long-prompt decode held steady at 45.4–46.2. A single timing isn't a machine constant — which is exactly why the scorecard stores raw rounds. Run it on your own Mac and your numbers land here too.

The capability, memory, and tool-calling numbers don't move — only speed does.

Run any of these in two minutes

Everything above ran on rapid-mlx, an OpenAI-compatible local server, which means your existing tools (Claude Code, Cursor, Aider, anything that speaks the OpenAI API) point at it with a one-line change — same API as the cloud, just localhost:

pip install rapid-mlx
rapid-mlx serve bonsai-27b-2bit
# → OpenAI-compatible server on http://localhost:8000/v1

If a model won't fit your Mac, it tells you before loading instead of thrashing your machine — drop to a smaller alias and go again. For the full memory-to-model map across every model size, not just these five, see Which Local LLM Fits Your Mac's RAM? or the live picker at models.rapidmlx.com.

The fine print

Suites: 30 tool-calling scenarios (9 categories) plus 10-question probes

each of HumanEval+ (coding), MATH-500 (reasoning), and MMLU-Pro (general) — the same harness as our 17-model M3 Ultra benchmark, so the two posts are directly comparable. Small suites: good for tiers, not for splitting hairs.

Settings: enable_thinking: false and temperature: 0 for every model, so

each gets its full token budget for the answer and the run reproduces. This under-reads think-first models like Nemotron — noted above.

Hardware: Mac Studio, M3 Ultra, 256 GB. Capability and memory are

hardware-independent; decode speed is Studio-measured and derated for laptops as noted.

Reproduce it: the eval harness and all five raw result files ship in the

repo. Run rapid-mlx serve <model>, point the harness at localhost, and you'll get your own numbers — on your own Mac. Corrections welcome; that's how the last one got better.

Which MLX Models Are Actually Smart and Fast? 17 Benchmarked on M3 Ultra

2026-03-05T12:00:00Z

Everyone has an opinion about which local model is "best" on a Mac. Almost nobody backs it with numbers. So we ran standardized evals — not vibes — on 17 MLX models on a single 256 GB M3 Ultra, scoring each on coding, reasoning, general knowledge, and tool calling, and measuring decode speed and peak RAM alongside. The goal was a practical question: for agent and coding work on Apple Silicon, which models are worth actually running?

This started as an 11-model run; after a lot of sharp feedback from the local LLM community we expanded it to 17, fixed a parser bug that was unfairly sinking one model, and added the RAM and average columns people asked for. The community's own findings are folded in throughout — that back-and-forth is where a lot of the real signal is.

Here's the short version before the full table:

Best overall: Qwen3.5-122B-A10B 8-bit — 89% average, smart across every

suite. Needs a big Mac.

Best value: the same model at MXFP4 — near-identical scores at 65 GB, so

it fits a 96 GB Mac.

Best "fits anywhere": Qwen3.5-27B 4-bit — 76% at just 15 GB RAM.
Fastest smart model: GPT-OSS-20B — 124 t/s at 12 GB, once we fixed a

parser bug (more on that below).

The scoreboard

Every score is the same suite run identically per model. Decode is pure generation speed (tok/s); RAM is peak resident memory; the four percentage columns are HumanEval+ (Code), MATH-500 (Reason), MMLU-Pro (General), and 30 tool-calling scenarios (Tools). All runs used enable_thinking: false for a fair cross-model comparison — see the methodology note on why.

Model	Quant	RAM	Decode	Tools	Code	Reason	General	Avg
Qwen3.5-122B-A10B	`8bit`	129.8 GB	43 t/s	87%	90%	90%	90%	89%
Qwen3.5-122B-A10B	`mxfp4`	65.0 GB	57 t/s	90%	90%	80%	90%	88%
Qwen3.5-35B-A3B	`8bit`	36.9 GB	80 t/s	90%	90%	80%	80%	85%
Qwen3-Coder-Next	`6bit`	64.8 GB	66 t/s	87%	90%	80%	70%	82%
Qwen3-Coder-Next	`4bit`	44.9 GB	74 t/s	90%	90%	70%	70%	80%
GLM-4.5-Air	`4bit`	60.3 GB	54 t/s	73%	90%	70%	80%	78%
GLM-4.7-Flash	`8bit`	31.9 GB	57 t/s	73%	100%	90%	50%	78%
Qwen3.5-27B	`4bit`	15.3 GB	38 t/s	83%	90%	50%	80%	76%
Qwen3.5-35B-A3B	`4bit`	19.6 GB	95 t/s	87%	90%	50%	70%	74%
Qwen3.5-9B	`4bit`	5.1 GB	106 t/s	83%	70%	60%	70%	71%
MiniMax-M2.5	`4bit`	128.9 GB	50 t/s	87%	10%	80%	90%	67%
GPT-OSS-20B	`mxfp4-q8`	12.1 GB	124 t/s	80%	20%	60%	90%	62%
Devstral-Small-2	`4bit`	13.4 GB	47 t/s	17%	90%	70%	70%	62%
Qwen3.5-4B	`4bit`	2.4 GB	158 t/s	73%	50%	50%	50%	56%
Mistral-Small-3.2	`4bit`	13.4 GB	47 t/s	17%	80%	60%	60%	54%
Hermes-3-Llama-8B	`4bit`	4.6 GB	123 t/s	17%	20%	30%	40%	27%
Qwen3-0.6B	`4bit`	0.4 GB	365 t/s	30%	20%	20%	30%	25%

The full scorecard — with TTFT (time-to-first-token, from 94 ms on Qwen3-0.6B to ~1.3 s on the 122B and MiniMax) and per-question breakdowns — lives in the repo: evals/SCORECARD.md.

What the numbers say

Qwen owns the top of the board. Alibaba's Qwen family takes the top five slots. Qwen3.5-122B-A10B at 8-bit is the only model that clears 90% on all four suites — and because it's a Mixture-of-Experts model with just 10B active parameters, it still decodes at 43 t/s despite the "122B" label. If you have a 256 GB Mac, this is the one to run.

The best value is the same model, quantized. Qwen3.5-122B-A10B at MXFP4 scores 88% — statistically the same — while dropping to 65 GB RAM, which fits a 96 GB Mac and decodes faster (57 t/s) than the 8-bit version. Quantization paid for itself here.

Qwen3.5-27B is the "fits anywhere" pick. 76% average at only 15 GB of RAM means it runs comfortably on a 32 GB Mac with room for a long context window. For most people this is the sweet spot — see why RAM, not model size, is the real ceiling.

For coding specifically, Qwen3-Coder-Next is the speed king — 90% coding at 74 t/s in 4-bit. If you're driving Aider, Cursor, or Claude Code and want fast responses, it's the pick. GLM-4.7-Flash is the sleeper: a perfect 100% on coding and 90% reasoning, but only 50% on MMLU-Pro — brilliant at code tasks, weak on general knowledge.

Two models are coding-only traps for agents. Devstral-Small-2 scores 90% on code but 17% on tool calling — its chat template has no tool support, so it's a great code model and a terrible agent. Mistral-Small-3.2 has the same problem. If you're building agents, tool-calling accuracy matters more than raw coding score.

MiniMax-M2.5 can't code — but reasons well. 10% on HumanEval+ despite 87% tool calling and 80% reasoning. Something is off with its code-generation format (possibly a quant artifact at 4-bit — the community had thoughts on this, below); it's a strong reasoning model, not a coder.

Small models are fast and useless for agents. Hermes-3-Llama-8B and Qwen3-0.6B scream along at 120–365 t/s but score under 30% on tool calling and under 20% on coding. Fine for autocomplete or simple chat; not viable for agent work.

The GPT-OSS bug that a benchmark caught

The most useful thing about running real evals is that they surface engine bugs, not just model quality. In the first run, GPT-OSS-20B scored a dismal 17% on tool calling — wildly out of line with its reputation. It turned out not to be the model at all: our harmony parser was serializing multi-turn tool history as plain text instead of native harmony tokens. One flag — SUPPORTS_NATIVE_TOOL_FORMAT=True — and tool calling jumped to 80%.

That's the difference between a leaderboard and a benchmark you can act on: GPT-OSS-20B is now, at 12 GB RAM and 124 t/s, the fastest genuinely smart model in the set — the default lightweight agent model we'd put on a 16 GB MacBook.

What the community found

The best corrections to this table came from other people running the same hardware. A few threads worth surfacing:

The benchmark ceiling isn't the real-world ceiling. Several people on 512 GB M3 Ultras reported that Qwen3.5-397B beats the 122B in day-to-day use — specifically manipulating large amounts of text without drifting into errors — even though 122B matches it on these benchmark questions. The lesson: at the top of the board, a 10-question suite stops discriminating, and real-world robustness pulls ahead. 397B is on the retest list.

Watch your KV cache, not just your weights. Running the 122B at 4-bit lands around 200 GB, which leaves very little on a 256 GB Mac — and a full 128K-token context can add tens of gigabytes of KV cache on top. Multiple readers pointed out that "fits in RAM" and "fits in RAM with your working context" are different questions. If you run long contexts, size down a tier. (This is the whole premise of the companion memory-to-model guide.)

Quantization is not free, especially for coding. The sharpest note was that heavily quantized weights aren't the same model — coding is usually the first capability to fall off, and newer models seem more quant-sensitive than older ones. Dynamic Q4 tends to track normal Q6/Q8 far better than a plain Q4. That makes MiniMax-M2.5's 10% coding score suspect: it may be a 4-bit artifact rather than the real model, which is exactly why 6-bit+ is on the retest list.

Turning off thinking is a real trade-off. The fairest objection to the methodology: disabling enable_thinking penalizes models designed to reason step-by-step, so this table under-reads their true ceiling. That's correct — see the methodology note for why we still made it the default, and why a thinking-enabled eval is warranted as a separate board.

Decode vs prompt-processing. A few readers rightly wanted prompt-processing speed broken out rather than folded in. To be clear: the Decode column here is pure generation speed; TTFT (which captures prompt processing) is a separate column in the full scorecard, ranging from 94 ms to ~1.3 s.

Vision works, quietly. For the multimodal question: the engine auto-detects vision models (Gemma 3, Qwen2.5-VL, etc.) and routes them through the MLLM pipeline, or you can force it with --force-mllm. Qwen3.5 itself is a text-only MoE, so don't expect image input from it.

Methodology

Suites: HumanEval+ (coding), MATH-500 (competition math / reasoning),

MMLU-Pro (general knowledge), plus 30 tool-calling scenarios across 9 categories.

Server: rapid-mlx — an OpenAI-compatible MLX inference server on

localhost, with the eval framework included in the repo so anyone can reproduce or submit their own numbers. Cache cleared between suites.

enable_thinking: false for everyone. This is the one deliberate choice

worth defending. Reasoning models like Qwen3.5 emit thousands of thinking tokens; under a fixed max_tokens budget the actual answer gets truncated mid-thought and scores collapse to 30–40% instead of 90%. Disabling thinking across the board gives every model its full token budget for the answer. It is not a perfect apples-to-apples for models designed to think — a thinking-enabled eval is on the roadmap — but it's the fairest single knob for a cross-model table.

Quantization caveat. Heavily quantized weights are not identical to the full model, and coding is usually the first capability to degrade. A model that underperforms at 4-bit (MiniMax-M2.5, here) may look very different at 6-bit or higher. Read the table as "this model at this quant," not "this model, period."

The hardware

Every number above is from one machine: a Mac Studio M3 Ultra — 28-core CPU, 60-core GPU, 32-core Neural Engine, 256 GB unified memory. The M3 Ultra's memory bandwidth is the main reason decode speeds hold up on the large models; a 64–128 GB M4 Max will run most of these but decode the big MoEs slower — bandwidth, not core count, is the lever.

What's next

Still on the bench, much of it straight from the community's request list: Qwen3.5-397B, GPT-OSS-120B, Step 3.5 Flash, Nemotron-Nano-30B, LFM-2-24B, 3-bit variants for 32 GB machines, and MiniMax-M2.5 at 6-bit+ to test the quantization theory above.

The scorecard is a living document — it's built to take community submissions, so if you run the eval on your own Mac you can add your numbers. Run rapid-mlx serve <alias>, point the eval harness at localhost:8000/v1, and submit. Not sure which model fits your machine first? Start with the memory-to-model guide or the live picker at models.rapidmlx.com.