OpenAI + Anthropic API · for coding agents

Up to 4× faster than Apple's MLX.

1.5× faster on a typical task — same model, same weights, on your Mac. How we measured →

Run LLMs locally on your Mac with an OpenAI- and Anthropic-compatible inference server for Apple Silicon. One Mac can answer Cursor, Claude Code, your agents and your own apps at the same time — they share the machine instead of queueing behind each other.

$ curl -fsSL https://rapidmlx.com/install.sh | bash

Prefer Homebrew? brew install rapid-mlx · Then rapid-mlx serve <model>

Download for Mac Free · macOS 14+ · Apple Silicon

Free and open source (Apache 2.0), the Desktop app and the engine alike. No account, no cloud inference: prompts stay on your Mac.

All install options → About the Desktop app → Coming from Ollama? → Read the docs → View on GitHub → 中文指南 →
100% local·Apple Silicon·27 tool-call parsers·20,105 tests

Qwen3.5-9B 4-bit, Mac mini M4 Pro, 48 GB, greedy decoding, whole-file code edits: decode 46.4 → 198.4 tok/s. Median 1.5× across the 18 everyday tasks that stream text (decode speed; 1.37× end to end over all 20 tasks). mlx-lm 0.32.0 (mlx_lm.server) vs Rapid-MLX 0.15.6, both at defaults, measured 2026-10-04.

rapid-mlx · local inference
http://localhost:8000/v1
Qwen3.6 35B-A3B · 4bit· Mac mini M4 Pro · 48 GB
104.5tok/s · single stream
rapid-mlx 0.15.4 · measured 2026-10-02 · local only

Hot right now

The models the local-AI world is running this month — measured, not marketed. Raw data for the October run is published; each card links its measured page.

Qwen3.8 27B

27B dense · 4bit · vision · MTP

The open model of the season — 6.9M Hugging Face downloads since its 2026-08-05 release; 16.3 GB of weights, the smart pick for every Mac from 32 GB up.

24.1 tok/s decode · 24.0 aggregate on 4 streams — Mac mini M4 Pro 48 GB, 0.15.4, 2026-10-02

$ rapid-mlx serve qwen3.8-27b-4bit
Measured page →

GLM-5.3 Flash

320B / 18B active MoE · 4bit · MTP

The frontier MoE flagship on a 256 GB Mac Studio — 320B parameters, 18B active per token; 181.7 GB of weights, 192 GB catalog floor.

27.8 tok/s decode at an 8K prompt — Mac Studio M3 Ultra 256 GB, 0.13.4, 2026-09-02 (engine benchmark doc)

$ rapid-mlx serve glm5.3-flash-4bit
Measured page →

Ternary Bonsai 2 27B

27B ternary · 2bit · vision

A full-size model in 8.6 GB that takes images — #1 trending text model on Hugging Face (2026-10-02); 32 GB is the practical floor.

17.7 tok/s decode — M4 Pro 48 GB, 0.15.4, 2026-10-02

$ rapid-mlx serve bonsai2-27b-2bit
Measured page →

Qwen3.6 35B-A3B

35B / 3B active MoE · 4bit

The fast MoE sweet spot — and the fastest measured decode in the October run; 17.5 GB of weights.

104.5 tok/s decode · 106.5 aggregate on 4 streams — M4 Pro 48 GB, 0.15.4, 2026-10-02

$ rapid-mlx serve qwen3.6-35b-4bit
Measured page →

DeepSeek V4.1 Flash (REAP)

MoE · REAP 2bit + K4 sidecar

Frontier DeepSeek on one Mac — deliberately constrained (greedy only, serial requests); 212.9 GB of weights, 224 GB measured floor.

≈19.4 tok/s with the K4 sidecar — M3 Ultra 256 GB, 0.15.0 qualification, 2026-09

$ rapid-mlx pull deepseek-v41-flash-reap-2bit
Measured page →

MiMo-V2.6 Flash

309B / 15B active MoE · 4bit · MTP

Xiaomi's frontier reasoner, new 2026-09-21 — 15B active per token; 171.7 GB of weights, 192 GB floor.

59 → 49 tok/s across 4K→30K context — M3 Ultra 256 GB, 0.15.0 release qualification, 2026-09

$ rapid-mlx serve mimo-v2.6-flash-4bit
Measured page →

Found a model on Hugging Face?

Rapid checks whether your Mac can run it before downloading, then gives you a guided import and quantization path.

1 · Check

Before any download

Weight format, architecture, chat template and fit in your Mac's memory, read from the repo's metadata. A model it can prove won't run is refused, with the reason.

2 · Suggest or import

A path that runs

When it finds one, it suggests an MLX build of the same model or a catalog model that fits. It never switches for you. Unquantized checkpoints convert with one explicit, cancel-safe command.

3 · Serve

Same endpoints

Your model or import is served on the OpenAI- and Anthropic-compatible API, ready for coding agents and chat apps.

$ rapid-mlx serve <org/repo>
$ rapid-mlx import <repo> --quantize 4
Bring your own model → Connect an agent
182.8

tok/s aggregate on 4 streams — Qwen3.5 4B, M4 Pro 48 GB (0.15.4, 2026-10-02)

104.5

tok/s decode, fastest measured — Qwen3.6 35B-A3B, same run

250

MLX-native models, text · vision · audio · video

3.5k★

stars on GitHub

Included in 0.16.2

Features available with the current release.

Multimodal setup · vision, image and video runtimes in the base install Experimental accelerated profiles · guarded TensorFold aliases for Bonsai 2, Gemma 4 and DeepSeek V4 Flash Safer local serving · foreign browser origins and unexpected Host headers rejected by default More local models · Nemotron 3.5 and small Bonsai 2 aliases Agent-session prefix cache · shared prompt prefixes retained more reliably

How it compares

Same model and same Mac for every engine.

Rapid-MLX 0.16.1oMLX 0.7.0LM Studio
Decode, 1 stream115.5 tok/s77.9 tok/s81.0 tok/s
4 streams, aggregate248.2 tok/s224.6 tok/s230.0 tok/s
Agent session (23k-token prefix), turns 2–10, median0.23 s0.24 s0.61 s
Same session, turn 1 after a restart0.20 s0.27 s32.7 s

Measured 2026-10-12 on Mac Studio, M5 Max, 36 GB with Qwen3.5-9B 4-bit. Single run per engine. Rapid-MLX 0.16.1 default used speculative decoding on a text-only lane. Full comparison, conditions and raw data →

Performance

What a 48 GB Mac mini serves today

HardwareMac mini M4 Pro · 48 GB
Concurrency1 & 4 streams
Methodmedian of 3 · temp 0
Metricdecode tok/s · 0.15.4 · 2026-10-02
ModelDecode, 1 stream4 streams, aggregateFirst token (2K prompt)Peak memory (Metal)
lfm2.5-1b-4bit314.0 tok/s618.2 tok/s0.9 s1.8 GB
qwen3.6-35b-4bit104.5 tok/s106.5 tok/s2.4 s23.2 GB (18.7 RSS)
qwen3.5-4b-4bit85.8 tok/s182.8 tok/s2.9 s4.3 GB
gemma-4-26b-4bit78.5 tok/s121.6 tok/s3.1 s20.9 GB
gpt-oss-20b-mxfp4-q877.2 tok/s126.7 tok/s2.8 s15.7 GB
qwen3.5-9b-4bit63.0 tok/s62.4 tok/s5.3 s8.0 GB
qwen3.8-27b-4bit24.1 tok/s24.0 tok/s18.8 s25.3 GB
bonsai2-27b-2bit17.7 tok/s16.8 tok/s23.4 s24.4 GB

One machine, one engine version, every number from the same October run — decode is a single stream; the aggregate column is four concurrent streams sharing the server. Gemma 4 26B was measured with --no-mllm (vision disabled; the default load peaks at 26.4 GB with a 16K prompt). Peak memory is the server's reported Metal peak. The 192 GB+ flags (GLM-5.3 Flash, DeepSeek V4.1 Flash, MiMo-V2.6 Flash) don't fit this Mac — their measured pages carry the engine's published 256 GB numbers. Raw data and method →

Multi-turn stays instant: prompt caching with KV trimming gives sub-100ms time to first token on transformers, and RNN state snapshots bring the same to hybrid architectures — a first on MLX. Full methodology and reproduction scripts →

Got a different Mac? See — and contribute — community-measured numbers for your Mac →

Fast is only half of it. We scored 17 MLX models on coding, reasoning, and tool calling on an M3 Ultra — see which ones are actually smart →

Which model fits your Mac?

The engine's own RAM-tier picks — the same table the installer and rapid-mlx recipe use, from an 8 GB MacBook Air to 320B-MoE flagships on a Mac Studio.

GLM-5.3 Flash 320B · measured on M3 Ultra 256 GB Qwen3.8 27B · measured on M4 Pro 48 GB DeepSeek V4 Flash 158B · up to 1M context 192–256 GB Mac Studio class
Your MacSmart pickFast pick
8 GBLFM2.5-2.6B 4bit
3.0 GB · 93.5 tok/s† · not for coding
LFM2.5-1B 4bit
1.9 GB · 208.4 tok/s† · basic chat
16 GBQwen3.5-4B 4bit
6.0 GB · 60.7 tok/s†
LFM2.5-1B 4bit
1.9 GB · 208.4 tok/s† · basic chat
18 GBQwen3.5-9B 4bit
8.7 GB · 35.7 tok/s†
Qwen3.5-4B 4bit
6.0 GB · 60.7 tok/s†
24 GBBonsai 27B 2bit
13 GB · 17.5 tok/s†
Qwen3.5-4B 4bit
6.0 GB · 60.7 tok/s†
32 GBQwen3.8-27B 4bit
20 GB · 40.7 tok/s‡
Qwen3.5-4B 4bit
6.0 GB · 60.7 tok/s†
48 GB+Qwen3.8-27B 4bit
20 GB · 40.7 tok/s‡
Qwen3.6-35B-A3B 4bit
20 GB · ≈60 tok/s · estimate, unmeasured

Rows are the engine's recommendation policy (rapid_mlx/model_recommendations.json at v0.15.4) — the same tiers the installer and rapid-mlx recipe print; sizes are 8K-prompt peaks. † measured via rapid-mlx serve on a Mac mini M2 Pro 32 GB (Rapid-MLX 0.12.7, 2026-08-07, short prompt). ‡ measured on a Mac Studio M3 Ultra 256 GB (0.12.15, 2026-08-18) — expect less on smaller chips; the same 27B measured 24.1 tok/s decode on an M4 Pro 48 GB in the October run. The Qwen3.6-35B fast pick is the engine's reviewed estimate, not a measurement (its only 32 GB-host run aborted with swap) — the M4 Pro run measured it at 104.5 tok/s on 48 GB. Above 48 GB the policy stays with these two picks; the big-MoE aliases (GLM-5.3 Flash, DeepSeek V4.1 Flash, MiMo-V2.6 Flash) are also served, with measured numbers on their model pages. Each of 23 models has a per-model page, and all 250 aliases are browseable in the model catalog →

Have a specific chip? Decode speed, first-token time and peak memory from real runs: best local LLMs by Mac chip →

Quickstart

01
$ curl -fsSL https://rapidmlx.com/install.sh | bash
02
zsh
# serve a model — auto-downloads on first run
$ rapid-mlx serve qwen3.5-4b-4bit
⚡ serving on http://localhost:8000/v1

# or just chat
$ rapid-mlx chat
03
example.pyserving · localhost:8000/v1
# point any OpenAI client at it — no key needed
from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:8000/v1",
    api_key="not-needed",
)
r = client.chat.completions.create(
    model="default",
    messages=[
        {"role": "user", "content": "Say hello"},
    ],
)
print(r.choices[0].message.content)

Built for real work

tool_call→ recovered

Tool calling that holds up

27 tool-call parser modules with automatic recovery when quantized models degrade.

<think>answer

Reasoning models

Chain-of-thought separation for Qwen3.x, GLM-5.x, gpt-oss, Gemma 4 and DeepSeek — thinking stays out of the answer stream.

Vision, audio & video

Vision via rapid-mlx[vision]. 44 audio aliases — TTS, speech-to-text, zero-shot voice cloning and forced alignment. 9 video aliases — Wan 2.2, CogVideoX-Fun and LTX-2.3/2.5 generate video locally behind an async /v1/videos API.

20105 testsrapid-mlx doctor ✓

Production-minded

20,105 test functions at v0.15.4 and a rapid-mlx doctor self-check.

PFlash prefill (opt-in) · 3.87–8.5× cold TTFT Prompt cache · 81.2× on an exact repeat DFlash speculative decoding · 1.4× pooled Audio TTS/STT · 44 aliases Video generation · Wan 2.2 · LTX-2.3 Continuous batching KV-cache quantization Structured JSON output

Works with

Anything that speaks OpenAI. Tested with:

CursorCursor
Claude CodeClaude Code
AiderAider
Continue.devContinue.dev
Open WebUIOpen WebUI
localhost:8000/v1
OpenAI-compatible
LibreChatLibreChat
PydanticAIPydanticAI
LangChainLangChain
smolagentssmolagents
GooseGoose

Also verified end-to-end: Codex CLI · Hermes · DeepSeek Harness · OpenCode · Qwen Code · OpenHands · Kilo Code · GitHub Copilot — full integration matrix →

Star on GitHub ↗ Read the docs →

Built to chat and serve

Leave one Mac running and everything else can ask it questions — the apps on your laptop, the agents in your editor, a script on another machine. Rapid-MLX is a server first: it is built to answer many callers, not one person typing.

Answers several callers at once

When requests land together, Rapid-MLX works on them in the same pass instead of finishing one before it starts the next — the technique is called continuous batching. It is why the third app to ask a question doesn't wait for the first two to finish.

Runs with nobody watching

No window open, nobody logged in — a Mac mini in a corner becomes the private AI endpoint everything else on your network calls. For the terminal: rapid-mlx serve --host 0.0.0.0 runs it headless. Set --trusted-hosts to the Mac IP or host name clients use, and --api-key for authentication.

Your bill stops at the hardware

No per-token charges, no rate limits, no quota to plan around. Prompts and documents are processed on the machine you own and never leave it — which is often the reason the workload has to be local in the first place.

One reported workload · from the community

A community member runs a 16 GB M4 Mac mini as a dedicated, always-on classifier: gemma-4-12b-it-4bit over the LAN, strict JSON, temperature 0, thousands of calls a day. Sending one request at a time, each call took 1.42 s. With four in flight, the batch came back at 1.03 s per call — the same machine got through roughly 1.4× the work per second, because it overlaps requests instead of running them end to end.

That is time per call measured across the batch, not a promise that any single request returns faster: under load an individual call still waits its share. It is throughput you can feel as a queue that stops growing.

Their machine, their workload, their measurement — one model on one Mac, reported to us and not reproduced here. It is not a Rapid-MLX benchmark and does not predict your numbers. Our own published figures are the M3 Ultra ones above, with the method stated next to them.

Serve your first model → OpenAI-compatible API surface →

From the community

People are building on top of it and running it in production. These are their own posts, not ours.

Production setup

MiniMax M2.5 as a local brain for OpenClaw

A 229B MoE at 4-bit on a Mac Studio, with prefix caching and streaming tool calls — the setup they landed on after iterating through several models and runtimes.

littlemac51 · GitHub Discussions
Community project

rapidmlx-menubar

Menu-bar control and a memory monitor for a 16 GB Mac mini daily driver — because Activity Monitor can't show Metal active memory.

CortonKwok-GGD · open source
Independent benchmark

mlx-Chronos

A free CLI benchmark suite and community leaderboard for local MLX inference engines on Apple Silicon. Rapid-MLX is one of the engines it measures — run it yourself.

igurss · open source
Built by 48 people
raullenchai waybarrios osdodo xiaoxiunique janhilgard loriz-art Jevin-F guo masonjames pierre427 angusgastle ersintarhan kol22 selimrecep 66Ton99 Oxygen56 YUHAO-corn michaelasper ryo1sato unsaltedbutter-ai 2005rishabh Chida82 warwickmei MIt9 jedisct1 ImL1s jpcarranza94 karmawastakenalready dashitongzhi krestenkrab ShiroKSH dearliory MaXoS-Agent MetricorTechnologies marsmike MishkinBerteig parth-951 romanbsd samuelfaj Shreyas-Gowda26 Sumu004 Thump604 aftersnow camerhann chen-yijiang kumosan2 wuwangzhang1216 wparuch
3,567GitHub stars
402forks
48contributors
Apache 2.0free, forever

Built something with Rapid-MLX? Show it in Discussions →

Frequently asked questions

What is Rapid-MLX?

Rapid-MLX is an open-source (Apache 2.0) OpenAI- and Anthropic-compatible LLM inference server for Apple Silicon, built on MLX, focused on reliable tool calling for coding agents. It runs open models such as Qwen 3.6, Gemma 4, DeepSeek and GPT-OSS on your Mac and serves them at localhost:8000/v1.

Is Rapid-MLX faster than Apple's MLX?

On the models we measured, yes. Rapid-MLX is built on MLX, so the comparison is against mlx-lm, Apple's MLX library and server (mlx_lm.server 0.32.0), with the same weights. On a Mac mini M4 Pro (48 GB) with Qwen3.5-9B 4-bit and greedy decoding, Rapid-MLX 0.15.6 decoded whole-file code edits up to 4.27× faster (46.4 → 198.4 tok/s) and 1.50× faster at the median of the 18 everyday tasks that stream text; Qwen3.6-35B-A3B 4-bit (MoE) was up to 3.16× faster, 1.57× at the median. The gain comes from speculative decoding (MTP and prompt lookup), on by default for these models. Long-prompt agent turns and the first token of a fresh prompt are close to a tie. Per-task results and method →

How fast is Rapid-MLX?

On a Mac mini M4 Pro (48 GB, rapid-mlx 0.15.4, measured 2026-10-02), Qwen3.8 27B decodes at 24.1 tokens per second single-stream and 24.0 tok/s aggregate across 4 concurrent streams; the fast MoE Qwen3.6 35B-A3B reaches 104.5 tok/s on the same run. The number that usually matters more in practice is what happens when requests overlap: continuous batching keeps latency per call from stacking up as callers are added, instead of queueing them behind one another.

Can I run Rapid-MLX as an always-on server for other machines?

Yes — that is what it is built for. Plenty of people leave a Mac mini running in a corner and let every other device in the house or office use it. Start it with rapid-mlx serve --host 0.0.0.0 and set --trusted-hosts to the Mac IP or host name those clients use; then other machines can call it through the OpenAI-compatible API at port 8000, with no window open and no user logged into a GUI session. Because continuous batching overlaps requests that arrive together, one Mac can back several applications, agents or scripts at once rather than serving them one after another. Authentication is opt-in: start the server with --api-key once it is reachable beyond your own machine.

Does Rapid-MLX run on Intel Macs?

No. Rapid-MLX is built on Apple's MLX framework and requires an Apple Silicon Mac (M1 or newer) running macOS 14 or later. Intel Macs are not supported.

Is Rapid-MLX free and open source?

Yes. Rapid-MLX is free and open source under the Apache 2.0 license. The source is on GitHub.

Can Rapid-MLX generate video and clone voices locally?

Yes. Rapid-MLX runs Wan 2.1/2.2, CogVideoX-Fun and LTX-2.5 / 2.3 for local video generation on Apple Silicon behind an asynchronous OpenAI-compatible /v1/videos API — install rapid-mlx[video], which needs ffmpeg and Python 3.11+. It also ships four zero-shot voice cloning models — IndexTTS, Qwen3-TTS Base, F5-TTS and Chatterbox — plus Qwen3-TTS VoiceDesign, where you describe a voice in natural language instead of picking one. Everything runs on the Mac; nothing is sent to a cloud service. See the video and audio family pages.

Which models does Rapid-MLX support?

Rapid-MLX supports 261 models across 15 families, including our Tier-1 families Qwen 3.6, Gemma 4, DeepSeek, and GPT-OSS — covering text, vision, audio, and video. Audio ships 44 aliases — 24 TTS (Kokoro, Qwen3-TTS, IndexTTS, Chatterbox, VibeVoice, VoxCPM, F5-TTS, Dia) and 20 STT (Whisper, Parakeet, SenseVoice, Qwen3-ASR, Qwen3-ForcedAligner) — via /v1/audio/speech and /v1/audio/transcriptions. Install with pip install 'rapid-mlx[audio]'.

Is Rapid-MLX OpenAI-compatible?

Yes. Rapid-MLX exposes a drop-in OpenAI-compatible API, so tools like Aider, Continue, LangChain, and any OpenAI client work unchanged by pointing them at localhost:8000/v1. Claude Code uses the Anthropic API at localhost:8000. Cursor calls models from its own servers, so it needs an HTTPS tunnel to your Mac.

How do I install Rapid-MLX?

Install from the terminal with one command: curl -fsSL https://rapidmlx.com/install.sh | bash. The script probes for Python 3.10+ (auto-installs python-build-standalone if missing), creates a venv at ~/.rapid-mlx, and symlinks rapid-mlx into ~/.local/bin. No sudo required. Prefer Homebrew? It's in homebrew/core — just brew install rapid-mlx (no tap, no trust). Then run rapid-mlx serve <model> to start a local OpenAI- and Anthropic-compatible server. Prefer a window to a terminal? The Rapid-MLX Desktop app runs the same engine behind a one-click Mac interface.

Product updates

New models and speedups, in your inbox.

A short note whenever Rapid-MLX gets faster or adds models worth running on your Mac. No spam — unsubscribe anytime.

We'll only email you about Rapid-MLX.

Run your first local model today.

Free and open source. Your prompts stay on your Mac. Anonymous usage stats are on by default; turn them off anytime.

$ curl -fsSL https://rapidmlx.com/install.sh | bash

Prefer Homebrew? brew install rapid-mlx · All install options →

Download for Mac Free · macOS 14+ · Apple Silicon