Bring your own model

Found a model on Hugging Face? Hand rapid-mlx serve its repo id. For a model outside the catalog that isn't downloaded yet, it first checks that your Mac can run it, then downloads and serves it.

Run it

$ rapid-mlx serve Qwen/Qwen3-0.6B       # any org/repo, or a local folder

  ✓ Checked before download
    Format        safetensors · bf16 · 1.4 GB
    Architecture  qwen3 (supported)
    Chat template found
    Fits your Mac yes · ~2.7 GB of 18 GB
  …

The check reads only the repo's metadata. Then the download runs, and once the model has loaded the server prints Ready: with its address (http://127.0.0.1:8000 by default). Talk to it from another terminal:

$ curl -s http://localhost:8000/v1/chat/completions -H 'Content-Type: application/json' \
    -d '{"model":"Qwen/Qwen3-0.6B","messages":[{"role":"user","content":"Say hello in five words."}]}' \
    | jq -r '.choices[0].message.content'
Hello!

Popular community fine-tunes

Dolphin, Hermes and "abliterated" builds are community fine-tunes of open models, tuned to follow your instructions and system prompt with fewer built-in refusals. They run like any other Hugging Face model. Each command below was checked, served and answered a chat request on rapid-mlx 0.15.6.

These are third-party weights that Rapid-MLX hasn't reviewed. The model's own license and model card govern how you may use it, so read the card first.

Dolphin 3.0 · Llama 3.2 3B

8 GB Mac and up · 1.7 GB download · MLX 4-bit · Llama 3.2 license

$ rapid-mlx serve mlx-community/dolphin3.0-llama3.2-3B-4Bit

Dolphin is steered through the system prompt: you set its tone and rules there. Its chat template has no tool definitions, so use it for chat rather than agents.

$ curl -s http://localhost:8000/v1/chat/completions -H 'Content-Type: application/json' \
    -d '{"model":"mlx-community/dolphin3.0-llama3.2-3B-4Bit","messages":[
      {"role":"system","content":"You are a concise assistant."},
      {"role":"user","content":"In two sentences, what is Apple Silicon unified memory?"}]}' \
    | jq -r '.choices[0].message.content'

Dolphin 3.0 · Llama 3.1 8B

16 GB Mac and up · 4.2 GB download · MLX 4-bit · Llama 3.1 license

$ rapid-mlx serve mlx-community/Dolphin3.0-Llama3.1-8B-4bit

The same Dolphin 3.0 tuning on the larger Llama 3.1 8B base. Chat only, like the 3B.

Hermes 3 · Llama 3.2 3B

8 GB Mac and up · 1.7 GB download · MLX 4-bit · Llama 3 license

$ rapid-mlx serve mlx-community/Hermes-3-Llama-3.2-3B-4bit

Nous Research's Hermes 3, steerable through the system prompt. Rapid-MLX picks the hermes tool-call parser by itself, but in our tool-call test the 3B answered in text instead of calling the tool. For agents, try Hermes 4 14B.

Hermes 4 · 14B

16 GB Mac and up · 7.7 GB download · MLX 4-bit · Apache 2.0

$ rapid-mlx serve mlx-community/Hermes-4-14B-4bit

Hermes 4 on a Qwen3 14B base. Rapid-MLX selects the hermes tool-call parser by itself, and in our test a request with a tools list came back as a structured tool_calls entry, the shape an agent expects.

Abliterated Qwen3 4B, quantized with import

8 GB Mac and up after import · 7.5 GB bf16 download, 2.1 GB on disk after import · Apache 2.0

An abliterated build edits a model's weights to give fewer built-in refusals (here, Qwen3-4B-Instruct-2507, per its model card). Many ship only unquantized bf16 safetensors, so quantize once with rapid-mlx import, then serve the name it prints:

$ rapid-mlx import huihui-ai/Huihui-Qwen3-4B-Instruct-2507-abliterated --quantize 4

  Source   bf16 · 7.5 GB · qwen3 (supported)
  Needs    ~11 GB free disk  (you have 85 GB)
  Output   ~2.1 GB, fits in 18 GB ✓
  ✓ Smoke test passed
  rapid-mlx serve huihui-qwen3-4b-instruct-2507-abliterated-4bit
  …

$ rapid-mlx serve huihui-qwen3-4b-instruct-2507-abliterated-4bit

The hermes tool-call parser is picked automatically; in our test a tools request came back as a structured tool_calls entry.

Hermes 3 8B and Hermes 4 70B are already in the catalog (hermes3-8b-4bit, hermes4-70b-4bit), see Hermes. To find more, search Hugging Face for the model name plus mlx. An mlx-community/…-4bit build serves directly, and a bf16 repo goes through import first.

If it doesn't run

A refused model exits with status 1 and nothing is downloaded. Find the message you got:

Import and quantize: rapid-mlx import

serve and pull never convert anything. Converting an unquantized (bf16 / fp16) safetensors checkpoint to quantized MLX is a separate, explicit command:

$ rapid-mlx import Qwen/Qwen3-0.6B --quantize 4

  Source   bf16 · 1.4 GB · qwen3 (supported)
  Needs    ~2.0 GB free disk  (you have 99 GB)
  Output   ~0.4 GB, fits in 18 GB ✓

  ✓ Smoke test passed
  rapid-mlx serve qwen3-0.6b-4bit-local
  (stored at ~/.rapid-mlx/imports/qwen3-0.6b-4bit-local)

$ rapid-mlx serve qwen3-0.6b-4bit-local

What it checks first, from metadata, before any download or conversion:

• The source has model*.safetensors weights at its root. GGUF / .bin-only repos are refused.
• The source isn't already quantized. Otherwise: is already quantized; serve it directly: rapid-mlx serve <repo>.
• Its model_type has an mlx-lm converter in this install (text models). Repos that ship their own model code (model_file / auto_map) are refused, because import never runs repository code.
• Disk: the output plus 10%, plus the source download plus 10% when the Hugging Face cache is on the same disk and the source isn't cached yet.
• Memory: the quantized output plus max(4 GB, 20% of RAM) must fit this Mac. Otherwise it's refused with Try fewer bits (--quantize 3 or 2).

Cancel-safe and cached.

• Conversion and a one-token smoke test run in child processes, inside a temporary directory next to the import cache. Only a converted, smoke-tested model is renamed into place.
• Ctrl-C, a failed conversion or a failed smoke test leaves the cache untouched: Cancelled. The import cache is unchanged. If a --force replacement is killed between its two renames, the next import of that name restores the previous copy.
• A per-name lock keeps two imports of the same name apart.
• Hub sources download into the normal Hugging Face cache, so an interrupted download resumes. The source is never deleted.
• An import is keyed by source, revision, mlx-lm version and recipe. Re-running the same import is a no-op (✓ Already imported). The same name with a different key is refused unless you pass --name or --force.

Where it lives. ~/.rapid-mlx/imports/<name> (or $RAPID_MLX_HOME/imports).

$ rapid-mlx models --cached      # or: rapid-mlx ls
  …
  ── Imported with `rapid-mlx import` (serve or rm by name) ──
  • qwen3-0.6b-4bit-local  330.9 MiB  from Qwen/Qwen3-0.6B · 4-bit

$ rapid-mlx serve qwen3-0.6b-4bit-local
$ rapid-mlx rm qwen3-0.6b-4bit-local       # asks first; -y skips the prompt

Reference

What the check looks at

rapid-mlx serve <org/repo> and rapid-mlx pull <org/repo> run the check for models outside the Rapid-MLX catalog. It reads metadata only: one Hugging Face model_info call, plus config.json fetched into a temporary directory when an architecture verdict needs confirming. No weights are fetched.

The check is conservative. Anything it can't establish gives no verdict, and serve / pull continue without a verdict. That covers network trouble, a Hub answer slower than 10 seconds, a gated repo you can't read yet, and an unusual repo layout. It doesn't run at all for catalog aliases, repos already in your Hugging Face cache, offline mode (HF_HUB_OFFLINE), or pull --bits / --format.

--no-preflight skips the check for one run, on both serve and pull.

What refusals look like

$ rapid-mlx pull Qwen/Qwen3-0.6B-GGUF

  ! Qwen/Qwen3-0.6B-GGUF only has GGUF files (Q8_0).
    Rapid-MLX runs MLX and safetensors weights, not GGUF. Nothing was downloaded.
    MLX builds of the same base model (third-party, not reviewed by Rapid-MLX;
    matched by base model, architecture and size):
      mlx-community/Qwen3-0.6B-4bit · 4-bit · ~0.3 GB · apache-2.0
      lmstudio-community/Qwen3-0.6B-MLX-4bit · 4-bit · ~0.3 GB · apache-2.0
    To try one, check its model card first, then: rapid-mlx pull mlx-community/Qwen3-0.6B-4bit
    (Think this is wrong? Re-run with --no-preflight to skip this check.)

rapid-mlx pull <repo> --format gguf is refused outright: Rapid-MLX cannot run GGUF files, so `--format gguf` is not available.

$ rapid-mlx serve Qwen/Qwen3-235B-A22B

  ✗ Qwen/Qwen3-235B-A22B needs ~526 GB of memory; this Mac has 18 GB.
    Its weights alone are 438 GB. Nothing was downloaded.
    Pick a smaller or more quantized build: rapid-mlx models
    A catalog model that runs well on your Mac:
      rapid-mlx serve qwen3.5-9b-4bit
    (Think this is wrong? Re-run with --no-preflight to skip this check.)

Suggested alternatives (never switched automatically)

A refusal can name up to two things that will run. Neither is ever substituted for the model you asked for. To use one, type the command it prints.

• An MLX build of the same model (GGUF-only Hub repos). The refused repo must declare itself a quantization of exactly one base repo (finetunes and merges count as different models). A candidate is shown only when all of these hold:
  • its model_type matches the base
  • its parameter count is within 5%
  • it uses an MLX quantization (not AWQ / GPTQ)
  • this install supports its architecture
  • it fits this Mac comfortably
  Candidates are labelled as third-party builds that Rapid-MLX has not reviewed, with their publisher, size and license. Read the model card before you try one.
• A catalog model of similar size, picked by the same recommendation policy the installer and Desktop use, limited to what this Mac's memory supports. When the refusal was "too big", it suggests this Mac's own recommendation instead of the nearest size.

All Hub lookups for suggestions share one short time budget. A slow Hub means fewer suggestions, never a slower refusal.

“Ask us to support it?” (opt-in)

When the check refuses a public Hugging Face model for its architecture or its weight format (GGUF / .bin only), an interactive terminal asks:

  Ask us to support it? This sends the repo id, architecture,
  format and your Rapid-MLX version. Nothing else. [y/N]

The request also carries the refusal's failure class, as the payload below shows. Nothing is sent unless you answer y. In scripts and other non-interactive shells there's no prompt. Re-run with --request (on serve or pull) to send it without asking. When you do, exactly five fields go to rapidmlx.com/api/model-request:

{
  "repo": "Qwen/Qwen3-0.6B-GGUF",
  "model_type": null,
  "format": "gguf",
  "failure": "unsupported_format",
  "version": "0.15.5"
}

• Never offered for gated or private repos, local paths, or "too big for this Mac" refusals.
• The site groups requests by failure class + architecture/format into one GitHub issue per key. Yours opens it or adds a vote, and the CLI prints the issue URL so you can follow it.
• If rapidmlx.com can't be reached, the CLI prints a prefilled GitHub issue link so you can file the request yourself.
• The HTTP User-Agent is the fixed string rapid-mlx-cli. Separately, if anonymous usage telemetry is on, the refusal is recorded like any failed start: its failure class, plus a model identity that is a repo id only when the repo is proven public, and otherwise <custom> or <local> (telemetry details).

Gated and private repos

Until you have access, the check can't read a gated repo's metadata, so it gives no verdict and the download stops with:

  Error: access to '<org/repo>' is gated on Hugging Face.
  Accept the licence or request access at https://huggingface.co/<org/repo>
  Then run `huggingface-cli login` or set `HF_TOKEN`, and try again.

Once you're signed in, the check runs normally. Support requests are never offered for gated or private repos. rapid-mlx import of a gated repo needs the same sign-in.

Local directories

The format, architecture and memory checks run on the folder's own files and config.json. There's nothing to download, and support requests aren't offered for local paths.

$ rapid-mlx serve ~/models/my-finetune
$ rapid-mlx import ~/models/my-finetune --quantize 4 --name my-ft-4bit
$ rapid-mlx serve my-ft-4bit

A local import is keyed by a fingerprint of the folder (file names, sizes and modification times, plus the contents of small files like configs and tokenizers). Re-running on an unchanged folder is a no-op. After you edit the folder, the stale import isn't reused: re-run with --force to replace it, or pick a new --name.

Frequently asked questions

Can Rapid-MLX run a GGUF model?

No. Rapid-MLX runs MLX and safetensors weights. A GGUF-only repo is refused before anything is downloaded and, when the check can verify one, it names an MLX build of the same base model that fits your Mac.

Does the check download the model?

No. It reads Hugging Face metadata only: one model_info call, plus config.json into a temporary directory when it needs to confirm an architecture verdict. If it can't read the metadata, it gives no verdict and the normal download proceeds.

What does a support request send?

It's sent only after you answer y or pass --request, and it contains the repo id, the architecture (model_type), the weight format, the failure class and your Rapid-MLX version. It's never offered for gated, private or local models, or for models that are simply too big for your Mac.

Can I run Dolphin or Hermes fine-tunes?

Yes. Serve an MLX build by its Hugging Face repo id, for example rapid-mlx serve mlx-community/dolphin3.0-llama3.2-3B-4Bit (8 GB Mac and up) or rapid-mlx serve mlx-community/Hermes-4-14B-4bit (16 GB Mac and up, with tool calling). A fine-tune that ships only bf16 weights, such as an abliterated build, is quantized once with rapid-mlx import <org/repo> --quantize 4. These are third-party weights; each model's own license and model card govern its use.

How do I quantize a bf16 model for my Mac?

rapid-mlx import <org/repo> --quantize 4 (2, 3, 4, 6 or 8 bits). It checks disk and memory first, converts in a temporary directory, runs a one-token smoke test, and only then publishes the model under a name you can serve, list and remove.

Next: connect an agent or a chat app to your model.