Run a Local OpenAI-Compatible MLX Server on Mac
Run rapid-mlx serve qwen3.5-4b-4bit on your Mac. Set your OpenAI client base URL to http://localhost:8000/v1 and send the served alias as the model.
$ brew install rapid-mlx # or: curl -fsSL https://rapidmlx.com/install.sh | bash $ rapid-mlx serve qwen3.5-4b-4bit $ curl http://localhost:8000/v1/chat/completions -H "Content-Type: application/json" \ -d '{"model": "qwen3.5-4b-4bit", "messages": [{"role": "user", "content": "Hello"}]}'
| Setting | Value |
|---|---|
| OpenAI base URL | http://localhost:8000/v1 |
| Anthropic base URL | http://localhost:8000 (no /v1; the SDK adds /v1/messages) |
| Port | 8000. If it's taken, the next free port up to 8009, and the startup banner says which. --port N pins it. |
| Host | 127.0.0.1, this Mac only. --host 0.0.0.0 listens on your network; allow the names clients use with --trusted-hosts. |
| API key | Not checked by default; SDKs need any non-empty string. Set --api-key or RAPID_MLX_API_KEY to require one. |
| Model name | The alias you served (qwen3.5-4b-4bit) or its Hugging Face repo id. GET /v1/models lists both. |
Start the server
Install with Homebrew (brew install rapid-mlx) or the
one-line installer, then serve a model by alias or by Hugging Face
repo id. The first run downloads the weights. When the server is up it
prints the URLs to use:
$ rapid-mlx serve qwen3.5-4b-4bit … Ready: http://127.0.0.1:8000 OpenAI: http://127.0.0.1:8000/v1 Anthropic: http://127.0.0.1:8000 Model: qwen3.5-4b-4bit
localhost and 127.0.0.1 are the same address
here. Not sure which model fits your Mac? rapid-mlx recipe
prints the pick for your memory, and
Pick a model for my Mac explains each
tier. In another terminal, rapid-mlx connect prints the same
URLs for a server that is already running.
Port and base URL
- Default port 8000. Without
--port,servetakes the first free port from 8000 to 8009 and says so:Port 8000 is in use; using 8001 instead (pass --port to choose).The base URL is thenhttp://localhost:8001/v1. - A fixed port.
rapid-mlx serve qwen3.5-4b-4bit --port 8080. An explicit port never falls back: if it's busy,serveexits with an error. - Include
/v1in the base URL for OpenAI clients (they append/chat/completions,/responsesand so on). Leave it off for Anthropic clients such as Claude Code. - Other machines on your network.
--host 0.0.0.0listens on every interface. Set an API key and--trusted-hoststo the Host names or IPs clients use; include loopback names if local clients also connect.
Browser and Host access
A default server accepts native clients using localhost, 127.0.0.1, [::1], or *.localhost. It rejects foreign browser Origins, including simple POST requests, and unexpected Host headers. A same-origin browser client can connect.
For a frontend on another origin, pass its exact origin with --cors-origins http://localhost:3000 (or set RAPID_MLX_CORS_ALLOW_ORIGINS). For LAN or reverse-proxy clients, use --host 0.0.0.0 when needed and pass the Host names or IPs clients send with --trusted-hosts (or RAPID_MLX_TRUSTED_HOSTS). An explicit trusted-host list replaces the loopback defaults, so include them if local clients also connect. Use an API key on shared servers. Wildcard origins or hosts explicitly disable their respective checks.
In 1.0.5, an unexpected Host returns HTTP 400 with a sanitized hostname and guidance to configure --trusted-hosts <name>. Add only the intended hostname. Browser extension clients can opt in with --cors-origins 'chrome-extension://*', 'moz-extension://*' or 'safari-web-extension://*'. Each pattern allows all valid extension origins for that scheme; cross-origin access remains closed by default.
Endpoints
Paths are relative to http://localhost:8000.
| Endpoint | API | Notes |
|---|---|---|
| POST /v1/chat/completions | OpenAI Chat Completions | Streaming, tool calling, vision inputs, reasoning_content. |
| POST /v1/responses | OpenAI Responses | What Codex CLI speaks; streaming events, reasoning and tool-call items. |
| POST /v1/messages | Anthropic Messages | What Claude Code speaks; tools, streaming, thinking blocks. Plus /v1/messages/count_tokens. |
| POST /v1/completions | OpenAI Completions | Raw text completion, no chat template. |
| GET /v1/models | OpenAI Models | The served model, with its context window and capabilities. |
| POST /v1/embeddings | OpenAI Embeddings | Set --embedding-model with a chat model or alone for dedicated embedding serving (EmbeddingGemma 2 uses the bundled vision runtime); without one it returns an error that says so. See EmbeddingGemma 2. |
| POST /v1/audio/speech, /v1/audio/transcriptions | OpenAI Audio | Text-to-speech and speech-to-text with an audio model; needs the [audio] extra. |
| POST /v1/images/generations | OpenAI Images | Text-to-image with an image model; uses the bundled image runtime. |
| GET /health | Probe | Health check; also /healthz, /readyz, /livez. |
| GET /docs, /openapi.json | Schema | Swagger UI and the OpenAPI schema of the running server. |
Video, audio translation and music, image edits, metrics and model management are in the full API reference, with every place the server differs from OpenAI.
Call it from curl, Python and Node
curl
$ curl http://localhost:8000/v1/models $ curl http://localhost:8000/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{"model": "qwen3.5-4b-4bit", "messages": [{"role": "user", "content": "Hello"}]}'
Python (openai)
from openai import OpenAI client = OpenAI(base_url="http://localhost:8000/v1", api_key="not-needed") r = client.chat.completions.create( model="qwen3.5-4b-4bit", messages=[{"role": "user", "content": "Hello"}], ) print(r.choices[0].message.content) # the Responses API works on the same client print(client.responses.create(model="qwen3.5-4b-4bit", input="Hello").output_text)
Node / TypeScript (openai)
import OpenAI from "openai"; const client = new OpenAI({ baseURL: "http://localhost:8000/v1", apiKey: "not-needed" }); const r = await client.chat.completions.create({ model: "qwen3.5-4b-4bit", messages: [{ role: "user", content: "Hello" }], }); console.log(r.choices[0].message.content);
Without changing code
The OpenAI SDKs read the base URL and key from the environment:
$ export OPENAI_BASE_URL=http://localhost:8000/v1 $ export OPENAI_API_KEY=not-needed $ python your_script.py
qwen3.5-4b-4bit on a Mac mini (M2 Pro, 32 GB): the
startup banner and port fallback above, the curl calls, the Python
snippet (openai 3.26.0) and the Node snippet (openai 7.30.0) all
returned replies.
API key and network access
A plain rapid-mlx serve doesn't check keys, and it only
listens on 127.0.0.1. To require a key, set it when you
start the server:
$ export RAPID_MLX_API_KEY="$(openssl rand -hex 24)" $ rapid-mlx serve qwen3.5-4b-4bit # or: --api-key <key>
Clients then send Authorization: Bearer <key>
(what the OpenAI SDKs do with api_key); on
/v1/messages, Anthropic's x-api-key header
works too. A missing or wrong key gets 401; the health
probes stay open. Set a key before you use --host 0.0.0.0
or put the server behind a tunnel.
New in 1.0.3: share Desktop's API on your network
In Desktop, open Settings → Tools to enable LAN access. The server restarts with a persistent bearer key saved in Keychain and shows addresses and copyable connection details for other devices on your network. Network clients can use inference; model management, administrative routes and Local Search, including grounded chat, stay on this Mac.
Supervised serve in 1.1.0
Supervised rapid-mlx serve runs a parent process that restarts a crashed runner on the same port. Ctrl-C stops the runner and frees the port. Use rapid-mlx status to inspect it, rapid-mlx log for recent API calls, and rapid-mlx jobs to list agent runs. rapid-mlx jobs cancel <id> cancels one. See CLI monitoring and the loopback control API.
The restarting parent and status, log and jobs controls apply only to catalog text and vision models on the supervised plain path; models that default to MTP, DFlash or Tensorfold (including qwen3.5-9b-4bit, qwen3.8-27b-4bit and qwen3.6-35b-4bit), audio/video and non-catalog repositories use the single-process path, with --no-spec-decode enabling supervision only for models whose exclusion is a default MTP sidecar, while native MTP and Tensorfold remain single-process.
For batch inference, X-Rapid-Priority: background lets waiting interactive requests go first, with 30-second aging and no preemption. Chat completions can continue a partial assistant message with continue_final_message.
Build on Rapid-MLX
Build a menu bar app, editor plugin, script or model picker around one local server. Run rapid-mlx serve as a child process without sudo, or connect to an existing listener. Probe the known port before starting another process; a server returning HTTP 503 may still be starting. Only stop a child your controller owns. For a Mac that serves without login, use the system daemon guide.
rapid-mlx serve qwen3.5-4b-4bit --host 127.0.0.1 --port 8000 rapid-mlx ps --json rapid-mlx top --port 8000 --once
GET /health/ready returns HTTP 503 while blocked and HTTP 200 when ready; lazy standby can be ready without resident weights. GET /healthz is liveness, not model readiness. GET /v1/status describes the primary engine, while GET /v1/models/residency lists model records, states and the memory ceiling. A registered record does not always have weights loaded.
POST /v1/models/load accepts a model alias or repo ID. To switch the assistant group while rejecting active work, send {"model":"qwen3.5-9b-4bit","replace_group":"assistant","replace_mode":"reject"}. A successful load returns HTTP 200; conflicts return 409 and insufficient capacity returns 507. Re-read residency after failures because eviction may have happened before loading.
DELETE /v1/models/{id} evicts an idle, unpinned model without deleting cached weights. Use the returned residency ID; pinned or busy models return 409, unknown IDs return 404, and success returns 204 with an empty body. Model controls and status require your bearer key when configured. Health probes and /metrics are public; keep the listener on loopback or behind access controls. The documented rapid_mlx_model_loaded gauge tracks the primary, not the number of resident models.
Accept additional response fields. The engine’s Build on Rapid-MLX guide distinguishes stable integration contracts from experimental metrics and shows controller lifecycle examples. See CLI monitoring and JSON discovery and prefix-cache retention.
Frequently asked questions
What port does the MLX OpenAI-compatible server use?
Rapid-MLX listens on port 8000 by default, so the OpenAI base URL is http://localhost:8000/v1. If 8000 is taken it uses the next free port up to 8009 and prints it; --port sets it explicitly.
What is the base URL for a local OpenAI-compatible server on Mac?
http://localhost:8000/v1 for OpenAI clients (Python, Node, LangChain, Codex CLI) and http://localhost:8000 for Anthropic clients such as Claude Code. Use the alias you served as the model name.
Does it need an OpenAI API key?
No. The server doesn't check keys unless you start it with --api-key or RAPID_MLX_API_KEY. The OpenAI SDKs refuse an empty key, so pass any placeholder string such as not-needed.
Which OpenAI endpoints are supported?
Chat Completions, Completions, Models, Responses, Embeddings (with an embedding model attached), Audio speech and transcription, and Images, plus Anthropic Messages. The API reference lists every route and where it differs from OpenAI.
Can I use it from another computer on my network?
Yes. Start it with --host 0.0.0.0, an API key, and --trusted-hosts naming the address clients use, then use http://<your-mac's-ip>:8000/v1 from the other machine. By default it only accepts connections from the Mac itself.
Next
- Use Goose with a local model: create a custom provider for this server.
- Connect a coding agent: Claude Code, Codex CLI, OpenCode and more, one setup command each.
- OpenAI SDK guide: the base URL switch, streaming and tool calls.
- API reference: every route and each place it differs from OpenAI.
- Run it as a service: start the server at boot.