OpenAI-compatible · built on MLX · Apple Silicon

Run LLMs locally
on your Mac. Fast.

Q: Can Rapid-MLX generate video and clone voices locally?

Yes. Since 0.11.9, Rapid-MLX runs Wan 2.1/2.2, CogVideoX-Fun and LTX-2.3 for local video generation on Apple Silicon behind an asynchronous OpenAI-compatible /v1/videos API (install 'rapid-mlx[video]', which needs ffmpeg and Python 3.11+). It also ships four zero-shot voice cloning models — IndexTTS, Qwen3-TTS Base, F5-TTS and Chatterbox — plus Qwen3-TTS VoiceDesign, where you describe a voice in natural language instead of picking one. Everything runs on the Mac; nothing is sent to a cloud service.

Q: How do I install Rapid-MLX?

The fastest path is the desktop app — download the .dmg from rapidmlx.com/download, open it, and you're chatting with a local model in a couple of clicks. Prefer the terminal? Install with one curl command: curl -fsSL https://rapidmlx.com/install.sh | bash. The script probes for Python 3.10+ (auto-installs python-build-standalone if missing), creates a venv at ~/.rapid-mlx, and symlinks rapid-mlx into ~/.local/bin. No sudo required. Prefer Homebrew? It's in homebrew/core — just brew install rapid-mlx (no tap, no trust). Then run rapid-mlx serve to start a local OpenAI-compatible server.

An OpenAI-compatible inference server for Apple Silicon. One Mac can answer Cursor, Claude Code, your agents and your own apps at the same time — they share the machine instead of queueing behind each other.

Download for Mac Free · macOS 14+ · Apple Silicon

Prefer the terminal? One-line install

$ curl -fsSL https://rapidmlx.com/install.sh | bash

All install options → Read the docs → View on GitHub → 中文指南 →

100% local·Apple Silicon·233 models·6222 tests

rapid-mlx · local inference

http://localhost:8000/v1

Qwen3.5-4B· Apple Silicon

261tok/s · 4 streams

local only · no cloud

261

tok/s aggregate throughput

0.08s

time to first token, cached

231

models across 15 families

3.5k★

stars on GitHub

Performance

HardwareMac Studio M3 Ultra

Concurrency4 concurrent streams

Methodmedian of 3 rounds

Metricaggregate throughput

Qwen3.5-4B261 tok/s

GPT-OSS 20B221 tok/s

Qwen3.5-9B180 tok/s

Qwen3.6-35B-A3B176 tok/s

Qwen3.5-35B-A3B 8bit151 tok/s

The flagships, one request at a time

HardwareMac Studio M3 Ultra · 256 GB

Concurrencysingle request

Methodmedian of 3 after warmup

Metricdecode tokens per second

Model	Shape	Workload	First token	Decode	Memory in use
`glm5.3-flash-4bit`	320B total · 18B active	47 → 512	not measured	29.2 tok/s	165 GB
`qwen3.8-flash-next-4bit`	180B total · 6B active	8,156 → 256	9.2 s	23.4 tok/s	~103 GB
`qwen3.8-flash-next-4bit` + MTP	same checkpoint, opt-in	8,156 → 256	14.6 s	32.2 tok/s	~107 GB
`qwen3.8-27b-4bit`	27B dense	8,156 → 256	24.2 s	37.4 tok/s	17.6 GB

A 320B model answering on a desktop is the headline; the rest is why it is usable. The 180B Flash-Next reaches its first token 2.6× sooner than the 27B on the same 8K prompt — 883 against 336 prompt tokens per second — because only 6B parameters are active per token. Multi-token prediction trades about 4 GB and a slower first token for 38% faster decode, and is off by default. Rows are not a like-for-like race: the GLM qualification used a short prompt and did not capture time to first token, so that cell is blank rather than derived. Memory is MLX active memory, not process RSS — the two big MoEs want a 192 GB Mac, the 27B is comfortable on 32 GB.

Multi-turn stays instant: prompt caching with KV trimming gives sub-100ms time to first token on transformers, and RNN state snapshots bring the same to hybrid architectures — a first on MLX. Full methodology and reproduction scripts →

Got a different Mac? See — and contribute — community-submitted numbers across Apple Silicon →

Fast is only half of it. We scored 17 MLX models on coding, reasoning, and tool calling on an M3 Ultra — see which ones are actually smart →

Which model fits your Mac?

From a 16 GB MacBook Air to frontier 158B-MoE models on a Mac Studio — there's a model sized for your machine.

Qwen3.5-122B DeepSeek V4 Flash 158B · day-0 GPT-OSS 120B up to 1M context all on one Mac Studio

Your Mac	Best model	Speed · M3 Ultra ref	What you get
16 GB	Qwen3.5-4B	147 tok/s	Chat, coding, tool calling
24 GB	Qwen3.5-9B	101 tok/s	Great all-rounder
32 GB	GPT-OSS 20B	119 tok/s	Harmony-native · 100% tool calling
48 GB	Qwen3.5-35B-A3B 8bit	80 tok/s	Sweet spot — smart + fast
96 GB	Qwen3.5-122B	43 tok/s	Frontier-level intelligence
128 GB	DeepSeek V4 Flash 158B	31–56 tok/s	Day-0 frontier MoE · 1M context

Speeds are single-stream rapid-mlx throughput measured on an M3 Ultra reference machine — smaller Macs run the same model slower. These are the fastest-for-your-RAM picks; the installer's more conservative defaults are in the hardware tiers table, and the 18 most-run models each have a measured per-model page. Browse all 233 models — including day-0 support for our Tier-1 families — Qwen 3.6, Gemma 4, DeepSeek, and gpt-oss 20B/120B — with a live RAM picker in the model catalog →

Quickstart

$ curl -fsSL https://rapidmlx.com/install.sh | bash

zsh

# serve a model — auto-downloads on first run
$ rapid-mlx serve qwen3.5-4b-4bit
⚡ serving on http://localhost:8000/v1

# or just chat
$ rapid-mlx chat

example.pyserving · localhost:8000/v1

# point any OpenAI client at it — no key needed
from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:8000/v1",
    api_key="not-needed",
)
r = client.chat.completions.create(
    model="default",
    messages=[
        {"role": "user", "content": "Say hello"},
    ],
)
print(r.choices[0].message.content)

Built for real work

tool_call→ recovered

Tool calling that holds up

17 parsers with automatic recovery when quantized models degrade.

<think>answer

Reasoning models

Chain-of-thought separation for DeepSeek-R1, Qwen3, and friends.

Vision, audio & video

Vision via rapid-mlx[vision]. 44 audio aliases — TTS, speech-to-text, zero-shot voice cloning and forced alignment. 9 video aliases — Wan 2.2, CogVideoX-Fun and LTX-2.3/2.5 generate video locally behind an async /v1/videos API.

6222 passingrapid-mlx doctor ✓

Production-minded

6222 unit tests and a rapid-mlx doctor self-check.

PFlash prefill · 3.87–8.5× cold TTFT Prompt cache · 81.2× on an exact repeat DFlash speculative decoding · 1.4× pooled Audio TTS/STT · 44 aliases Video generation · Wan 2.2 · LTX-2.3 Continuous batching Smart cloud routing KV-cache quantization Structured JSON output

Works with

Anything that speaks OpenAI. Tested with:

Open WebUI

localhost:8000/v1

OpenAI-compatible

LibreChat

PydanticAI

LangChain

smolagents

Goose

Also verified end-to-end: Codex CLI · Hermes · DeepSeek Harness · OpenCode · Qwen Code · OpenHands · Kilo Code · GitHub Copilot — full integration matrix →

Star on GitHub ↗ Read the docs →

Built to chat and serve

Leave one Mac running and everything else can ask it questions — the apps on your laptop, the agents in your editor, a script on another machine. Rapid-MLX is a server first: it is built to answer many callers, not one person typing.

Answers several callers at once

When requests land together, Rapid-MLX works on them in the same pass instead of finishing one before it starts the next — the technique is called continuous batching. It is why the third app to ask a question doesn't wait for the first two to finish.

Runs with nobody watching

No window open, nobody logged in — a Mac mini in a corner becomes the private AI endpoint everything else on your network calls. For the terminal: rapid-mlx serve --host 0.0.0.0 runs it headless, and --api-key turns on authentication once it reaches past your own machine.

Your bill stops at the hardware

No per-token charges, no rate limits, no quota to plan around. Prompts and documents are processed on the machine you own and never leave it — which is often the reason the workload has to be local in the first place.

One reported workload · from the community

A community member runs a 16 GB M4 Mac mini as a dedicated, always-on classifier: gemma-4-12b-it-4bit over the LAN, strict JSON, temperature 0, thousands of calls a day. Sending one request at a time, each call took 1.42 s. With four in flight, the batch came back at 1.03 s per call — the same machine got through roughly 1.4× the work per second, because it overlaps requests instead of running them end to end.

That is time per call measured across the batch, not a promise that any single request returns faster: under load an individual call still waits its share. It is throughput you can feel as a queue that stops growing.

Their machine, their workload, their measurement — one model on one Mac, reported to us and not reproduced here. It is not a Rapid-MLX benchmark and does not predict your numbers. Our own published figures are the M3 Ultra ones above, with the method stated next to them.

Serve your first model → OpenAI-compatible API surface →

From the community

People are building on top of it and running it in production. These are their own posts, not ours.

Production setup

MiniMax M2.5 as a local brain for OpenClaw

A 229B MoE at 4-bit on a Mac Studio, with prefix caching and streaming tool calls — the setup they landed on after iterating through several models and runtimes.

littlemac51 · GitHub Discussions

Community project

rapidmlx-menubar

Menu-bar control and a memory monitor for a 16 GB Mac mini daily driver — because Activity Monitor can't show Metal active memory.

CortonKwok-GGD · open source

Independent benchmark

mlx-Chronos

A free CLI benchmark suite and community leaderboard for local MLX inference engines on Apple Silicon. Rapid-MLX is one of the engines it measures — run it yourself.

igurss · open source

Built by 48 people

3,567GitHub stars

402forks

48contributors

Apache 2.0free, forever

Built something with Rapid-MLX? Show it in Discussions →

Frequently asked questions

What is Rapid-MLX?

Rapid-MLX is a high-performance, OpenAI-compatible LLM server for Apple Silicon Macs, built on Apple's MLX framework. It runs the latest open models — Qwen 3.6, Gemma 4, DeepSeek, GPT-OSS, and more — locally and exposes a drop-in OpenAI API at localhost:8000/v1.

How fast is Rapid-MLX?

On an M3 Ultra it reaches 261 tokens per second of aggregate throughput across 4 concurrent streams, with a 0.08-second time to first token on a cached prompt. The number that usually matters more in practice is what happens when requests overlap: continuous batching keeps latency per call from stacking up as callers are added, instead of queueing them behind one another.

Can I run Rapid-MLX as an always-on server for other machines?

Yes — that is what it is built for. Plenty of people leave a Mac mini running in a corner and let every other device in the house or office use it. Start it with rapid-mlx serve --host 0.0.0.0 and any machine on your network can call it through the OpenAI-compatible API at port 8000, with no window open and no user logged into a GUI session. Because continuous batching overlaps requests that arrive together, one Mac can back several applications, agents or scripts at once rather than serving them one after another. Authentication is opt-in: start the server with --api-key once it is reachable beyond your own machine.

Does Rapid-MLX run on Intel Macs?

No. Rapid-MLX is built on Apple's MLX framework and requires an Apple Silicon Mac (M1 or newer) running macOS 14 or later. Intel Macs are not supported.

Is Rapid-MLX free and open source?

Yes. Rapid-MLX is free and open source under the Apache 2.0 license. The source is on GitHub.

Can Rapid-MLX generate video and clone voices locally?

Yes. Since 0.11.9, Rapid-MLX runs Wan 2.1/2.2, CogVideoX-Fun and LTX-2.3 for local video generation on Apple Silicon behind an asynchronous OpenAI-compatible /v1/videos API — install rapid-mlx[video], which needs ffmpeg and Python 3.11+. It also ships four zero-shot voice cloning models — IndexTTS, Qwen3-TTS Base, F5-TTS and Chatterbox — plus Qwen3-TTS VoiceDesign, where you describe a voice in natural language instead of picking one. Everything runs on the Mac; nothing is sent to a cloud service. See the video and audio family pages.

Which models does Rapid-MLX support?

Rapid-MLX supports 233 models across 15 families, including our Tier-1 families Qwen 3.6, Gemma 4, DeepSeek, and GPT-OSS — covering text, vision, audio, and video. Audio ships 44 aliases — 24 TTS (Kokoro, Qwen3-TTS, IndexTTS, Chatterbox, VibeVoice, VoxCPM, F5-TTS, Dia) and 20 STT (Whisper, Parakeet, SenseVoice, Qwen3-ASR, Qwen3-ForcedAligner) — via /v1/audio/speech and /v1/audio/transcriptions. Install with pip install 'rapid-mlx[audio]'.

Is Rapid-MLX OpenAI-compatible?

Yes. Rapid-MLX exposes a drop-in OpenAI-compatible API, so tools like Cursor, Claude Code, Aider, Continue, LangChain, and any OpenAI client work unchanged by pointing them at localhost:8000/v1.

How do I install Rapid-MLX?

The fastest path is the desktop app — download the .dmg, open it, and you're chatting with a local model in a couple of clicks. Prefer the terminal? One curl command: curl -fsSL https://rapidmlx.com/install.sh | bash. The script probes for Python 3.10+ (auto-installs python-build-standalone if missing), creates a venv at ~/.rapid-mlx, and symlinks rapid-mlx into ~/.local/bin. No sudo required. Prefer Homebrew? It's in homebrew/core — just brew install rapid-mlx (no tap, no trust). Then run rapid-mlx serve <model> to start a local OpenAI-compatible server.

Run LLMs locally
on your Mac. Fast.