API surface

Rapid-MLX exposes an OpenAI-shaped HTTP API at http://localhost:8000/v1 by default. If a tool speaks OpenAI, it speaks Rapid-MLX. Point the base URL at your local server and pass any non-empty API key.

Endpoints

What's the same as OpenAI

• Request and response shapes for /chat/completions, /completions, /embeddings, /responses.
• SSE streaming via stream: true with data: [DONE] terminator.
• Function / tool calling via tools and tool_choice.
• Vision inputs as image_url content parts on multimodal models (Gemma 4, Qwen3-VL).
• Authorization header: when the server is started without --api-key it is parsed but not enforced (pass any string); start with --api-key and a missing or wrong key returns 401.

What is on by default in 0.9

Two features flipped from opt-in to on-by-default in 0.9 because they help every multi-user workload and cost nothing on the single-user path.

• Compressed KV cache (K8V4) on 9 hero MoE aliases. Qwen3.5-9B / 27B and the Qwen3.6-35B-A3B family (4 / 6 / 8-bit + DWQ) automatically use the K8V4 compressed KV cache. This roughly doubles the concurrent-user count at the same RAM budget, and the output is byte-identical to the uncompressed baseline. If you'd rather run without it — for a bench or an A/B — start the server with rapid-mlx serve <alias> --kv-cache-turboquant none.
• Radix-tree prefix cache. When several requests share a common prefix (system prompt, tool schema, RAG boilerplate) the server reuses the already-computed KV. Turns on automatically — no flag to flip, no per-request opt-in. Measured 13× aggregate throughput at 10 concurrent clients on the K8V4 hero aliases.
• Speculative decoding is opt-in. MTP and DFlash are both verified lossless (byte-identical output) but you have to enable them explicitly: MTP via --speculative-config '{"method":"mtp","num_speculative_tokens":3}', DFlash via pip install 'rapid-mlx[dflash]' + --speculative-config '{"method":"dflash"}'.

Where it diverges

• reasoning_content is surfaced as a sibling to content for reasoning models — same field name as DeepSeek-R1's API, not OpenAI's reasoning blocks.
• reasoning_max_tokens caps the thinking portion of a request (tokens inside <think>…</think>); the model then answers. It does not bound the answer — pair it with max_tokens for total length.
• /v1/models includes Rapid-MLX-specific fields: tool_call_parser, reasoning_parser, is_hybrid, is_moe, supports_spec_decode.
• Tool-call output respects the effective parser for the served model rather than always emitting OpenAI-flavoured JSON — most aliases use one of Hermes / Qwen3-XML / GLM4.7 / Harmony / UI-TARS / Llama / DeepSeek / Minimax. The wire shape the client sees is normalised back to OpenAI's tool_calls array.
• Streaming SSE for /v1/responses emits response.output_item.added, response.output_item.done, reasoning items, and tool-call items in the order OpenAI specifies — but the per-token deltas are batched per model: an AR model emits one token per delta, a diffusion model emits a block.

Examples

List models

$ curl http://localhost:8000/v1/models | jq .data[0]
{
  "id": "qwen3.5-4b-4bit",
  "object": "model",
  "tool_call_parser": "hermes",
  "reasoning_parser": "qwen3",
  "is_hybrid": false,
  "is_moe": false,
  "supports_spec_decode": false
}

Streaming chat completion

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="x")
stream = client.chat.completions.create(
    model="qwen3.5-4b-4bit",
    messages=[{"role": "user", "content": "Hello"}],
    stream=True,
)
for chunk in stream:
    print(chunk.choices[0].delta.content or "", end="")

Embeddings

# first install: pip install 'rapid-mlx[embeddings]'
$ rapid-mlx serve embeddinggemma-300m-6bit --port 8001 &
$ curl http://localhost:8001/v1/embeddings \
    -H "Content-Type: application/json" \
    -d '{"model": "embeddinggemma-300m-6bit", "input": "ping"}'

Long inputs. Before 0.11.8 the tokenizer was pinned at 512 tokens, so anything longer came back HTTP 200, correctly shaped, and quietly missing its tail — which degrades a vector index with no signal. The limit is now derived from the model (config.max_position_embeddings, else tokenizer.model_max_length), so a 32K-context embedder like Qwen3-Embedding-4B uses its real ceiling. Overflow is never silent: it either logs and increments rapid_mlx_embedding_truncations_total on /metrics, or — with --embedding-overflow-policy error — returns a structured 400:

$ rapid-mlx serve embeddinggemma-300m-6bit --port 8001 \
    --embedding-max-length 2048 --embedding-overflow-policy error

# a 3,000-token input then returns, instead of a silently short vector:
{"error": {"code": "input_too_long", "message": "…observed vs allowed tokens…"}}

Audio (TTS)

# first install: pip install 'rapid-mlx[audio]'
$ rapid-mlx serve kokoro-tts --port 8002 &
$ curl http://localhost:8002/v1/audio/speech \
    -H "Content-Type: application/json" \
    -d '{"model": "kokoro-tts", "voice": "af_heart", "input": "hello"}' \
    -o speech.wav

Where to next

For the full request/response field reference, hit the live Swagger UI at http://localhost:8000/docs on your own server — it stays in sync with the running build, including any flags wired up by recent releases. The engine README covers CLI flags, environment variables, and the rapid-mlx launch <client> bootstrap targets.