Blog

Which MLX Models Are Actually Smart and Fast? 17 Benchmarked on M3 Ultra

Not vibes — actual evals. 17 local models run through the same coding, reasoning, general-knowledge, and tool-calling suites on a 256 GB M3 Ultra, ranked by what's both smart and fast — with the community's own findings folded in.

By raullenchai · · 10 min read
benchmarksmlxapple-siliconm3-ultra

Everyone has an opinion about which local model is "best" on a Mac. Almost nobody backs it with numbers. So we ran standardized evals — not vibes — on 17 MLX models on a single 256 GB M3 Ultra, scoring each on coding, reasoning, general knowledge, and tool calling, and measuring decode speed and peak RAM alongside. The goal was a practical question: for agent and coding work on Apple Silicon, which models are worth actually running?

This started as an 11-model run; after a lot of sharp feedback from the local LLM community we expanded it to 17, fixed a parser bug that was unfairly sinking one model, and added the RAM and average columns people asked for. The community's own findings are folded in throughout — that back-and-forth is where a lot of the real signal is.

Here's the short version before the full table:

suite. Needs a big Mac.

it fits a 96 GB Mac.

parser bug (more on that below).

The scoreboard

Every score is the same suite run identically per model. Decode is pure generation speed (tok/s); RAM is peak resident memory; the four percentage columns are HumanEval+ (Code), MATH-500 (Reason), MMLU-Pro (General), and 30 tool-calling scenarios (Tools). All runs used enable_thinking: false for a fair cross-model comparison — see the methodology note on why.

Model Quant RAM Decode Tools Code Reason General Avg
Qwen3.5-122B-A10B 8bit 129.8 GB 43 t/s 87% 90% 90% 90% 89%
Qwen3.5-122B-A10B mxfp4 65.0 GB 57 t/s 90% 90% 80% 90% 88%
Qwen3.5-35B-A3B 8bit 36.9 GB 80 t/s 90% 90% 80% 80% 85%
Qwen3-Coder-Next 6bit 64.8 GB 66 t/s 87% 90% 80% 70% 82%
Qwen3-Coder-Next 4bit 44.9 GB 74 t/s 90% 90% 70% 70% 80%
GLM-4.5-Air 4bit 60.3 GB 54 t/s 73% 90% 70% 80% 78%
GLM-4.7-Flash 8bit 31.9 GB 57 t/s 73% 100% 90% 50% 78%
Qwen3.5-27B 4bit 15.3 GB 38 t/s 83% 90% 50% 80% 76%
Qwen3.5-35B-A3B 4bit 19.6 GB 95 t/s 87% 90% 50% 70% 74%
Qwen3.5-9B 4bit 5.1 GB 106 t/s 83% 70% 60% 70% 71%
MiniMax-M2.5 4bit 128.9 GB 50 t/s 87% 10% 80% 90% 67%
GPT-OSS-20B mxfp4-q8 12.1 GB 124 t/s 80% 20% 60% 90% 62%
Devstral-Small-2 4bit 13.4 GB 47 t/s 17% 90% 70% 70% 62%
Qwen3.5-4B 4bit 2.4 GB 158 t/s 73% 50% 50% 50% 56%
Mistral-Small-3.2 4bit 13.4 GB 47 t/s 17% 80% 60% 60% 54%
Hermes-3-Llama-8B 4bit 4.6 GB 123 t/s 17% 20% 30% 40% 27%
Qwen3-0.6B 4bit 0.4 GB 365 t/s 30% 20% 20% 30% 25%

The full scorecard — with TTFT (time-to-first-token, from 94 ms on Qwen3-0.6B to ~1.3 s on the 122B and MiniMax) and per-question breakdowns — lives in the repo: evals/SCORECARD.md.

What the numbers say

Qwen owns the top of the board. Alibaba's Qwen family takes the top five slots. Qwen3.5-122B-A10B at 8-bit is the only model that clears 90% on all four suites — and because it's a Mixture-of-Experts model with just 10B active parameters, it still decodes at 43 t/s despite the "122B" label. If you have a 256 GB Mac, this is the one to run.

The best value is the same model, quantized. Qwen3.5-122B-A10B at MXFP4 scores 88% — statistically the same — while dropping to 65 GB RAM, which fits a 96 GB Mac and decodes faster (57 t/s) than the 8-bit version. Quantization paid for itself here.

Qwen3.5-27B is the "fits anywhere" pick. 76% average at only 15 GB of RAM means it runs comfortably on a 32 GB Mac with room for a long context window. For most people this is the sweet spot — see why RAM, not model size, is the real ceiling.

For coding specifically, Qwen3-Coder-Next is the speed king — 90% coding at 74 t/s in 4-bit. If you're driving Aider, Cursor, or Claude Code and want fast responses, it's the pick. GLM-4.7-Flash is the sleeper: a perfect 100% on coding and 90% reasoning, but only 50% on MMLU-Pro — brilliant at code tasks, weak on general knowledge.

Two models are coding-only traps for agents. Devstral-Small-2 scores 90% on code but 17% on tool calling — its chat template has no tool support, so it's a great code model and a terrible agent. Mistral-Small-3.2 has the same problem. If you're building agents, tool-calling accuracy matters more than raw coding score.

MiniMax-M2.5 can't code — but reasons well. 10% on HumanEval+ despite 87% tool calling and 80% reasoning. Something is off with its code-generation format (possibly a quant artifact at 4-bit — the community had thoughts on this, below); it's a strong reasoning model, not a coder.

Small models are fast and useless for agents. Hermes-3-Llama-8B and Qwen3-0.6B scream along at 120–365 t/s but score under 30% on tool calling and under 20% on coding. Fine for autocomplete or simple chat; not viable for agent work.

The GPT-OSS bug that a benchmark caught

The most useful thing about running real evals is that they surface engine bugs, not just model quality. In the first run, GPT-OSS-20B scored a dismal 17% on tool calling — wildly out of line with its reputation. It turned out not to be the model at all: our harmony parser was serializing multi-turn tool history as plain text instead of native harmony tokens. One flag — SUPPORTS_NATIVE_TOOL_FORMAT=True — and tool calling jumped to 80%.

That's the difference between a leaderboard and a benchmark you can act on: GPT-OSS-20B is now, at 12 GB RAM and 124 t/s, the fastest genuinely smart model in the set — the default lightweight agent model we'd put on a 16 GB MacBook.

What the community found

The best corrections to this table came from other people running the same hardware. A few threads worth surfacing:

The benchmark ceiling isn't the real-world ceiling. Several people on 512 GB M3 Ultras reported that Qwen3.5-397B beats the 122B in day-to-day use — specifically manipulating large amounts of text without drifting into errors — even though 122B matches it on these benchmark questions. The lesson: at the top of the board, a 10-question suite stops discriminating, and real-world robustness pulls ahead. 397B is on the retest list.

Watch your KV cache, not just your weights. Running the 122B at 4-bit lands around 200 GB, which leaves very little on a 256 GB Mac — and a full 128K-token context can add tens of gigabytes of KV cache on top. Multiple readers pointed out that "fits in RAM" and "fits in RAM with your working context" are different questions. If you run long contexts, size down a tier. (This is the whole premise of the companion memory-to-model guide.)

Quantization is not free, especially for coding. The sharpest note was that heavily quantized weights aren't the same model — coding is usually the first capability to fall off, and newer models seem more quant-sensitive than older ones. Dynamic Q4 tends to track normal Q6/Q8 far better than a plain Q4. That makes MiniMax-M2.5's 10% coding score suspect: it may be a 4-bit artifact rather than the real model, which is exactly why 6-bit+ is on the retest list.

Turning off thinking is a real trade-off. The fairest objection to the methodology: disabling enable_thinking penalizes models designed to reason step-by-step, so this table under-reads their true ceiling. That's correct — see the methodology note for why we still made it the default, and why a thinking-enabled eval is warranted as a separate board.

Decode vs prompt-processing. A few readers rightly wanted prompt-processing speed broken out rather than folded in. To be clear: the Decode column here is pure generation speed; TTFT (which captures prompt processing) is a separate column in the full scorecard, ranging from 94 ms to ~1.3 s.

Vision works, quietly. For the multimodal question: the engine auto-detects vision models (Gemma 3, Qwen2.5-VL, etc.) and routes them through the MLLM pipeline, or you can force it with --force-mllm. Qwen3.5 itself is a text-only MoE, so don't expect image input from it.

Methodology

MMLU-Pro (general knowledge), plus 30 tool-calling scenarios across 9 categories.

localhost, with the eval framework included in the repo so anyone can reproduce or submit their own numbers. Cache cleared between suites.

worth defending. Reasoning models like Qwen3.5 emit thousands of thinking tokens; under a fixed max_tokens budget the actual answer gets truncated mid-thought and scores collapse to 30–40% instead of 90%. Disabling thinking across the board gives every model its full token budget for the answer. It is not a perfect apples-to-apples for models designed to think — a thinking-enabled eval is on the roadmap — but it's the fairest single knob for a cross-model table.

Quantization caveat. Heavily quantized weights are not identical to the full model, and coding is usually the first capability to degrade. A model that underperforms at 4-bit (MiniMax-M2.5, here) may look very different at 6-bit or higher. Read the table as "this model at this quant," not "this model, period."

The hardware

Every number above is from one machine: a Mac Studio M3 Ultra — 28-core CPU, 60-core GPU, 32-core Neural Engine, 256 GB unified memory. The M3 Ultra's memory bandwidth is the main reason decode speeds hold up on the large models; a 64–128 GB M4 Max will run most of these but decode the big MoEs slower — bandwidth, not core count, is the lever.

What's next

Still on the bench, much of it straight from the community's request list: Qwen3.5-397B, GPT-OSS-120B, Step 3.5 Flash, Nemotron-Nano-30B, LFM-2-24B, 3-bit variants for 32 GB machines, and MiniMax-M2.5 at 6-bit+ to test the quantization theory above.

The scorecard is a living document — it's built to take community submissions, so if you run the eval on your own Mac you can add your numbers. Run rapid-mlx serve <alias>, point the eval harness at localhost:8000/v1, and submit. Not sure which model fits your machine first? Start with the memory-to-model guide or the live picker at models.rapidmlx.com.

Run this yourself. rapid-mlx is an open-source, OpenAI-compatible inference server for Apple Silicon. pip install rapid-mlx, then rapid-mlx serve <alias>. Browse every supported model on the family docs or pick one by your Mac's RAM at models.rapidmlx.com.