Hero model · V4 sparse attention · day-0 MLX

DeepSeek V4-Flash on Apple Silicon

DeepSeek's V4 family — Channel-Sparse Attention (CSA) plus Hierarchical-Cluster Attention (HCA) — at 2-bit, makes the previously server-only V-family architecture run on a single 256 GB M3 Ultra workstation. rapid-mlx shipped day-0 support (PR #168) and the dedicated deepseek tool parser.

TL;DR — one command. pip install rapid-mlx && rapid-mlx serve deepseek-v4-flash-2bit runs V4-Flash at 2-bit on a 256 GB M3 Ultra Mac Studio. 4-bit and 8-bit variants are wired for larger memory tiers. Headline tok/s: TODO — bench pending on M3 Ultra.

Why V4-Flash matters

The V4 family is DeepSeek's first generation built around CSA + HCA sparse attention — instead of full O(n²) attention, each query head attends only to a learned subset of channels (CSA) grouped into hierarchical clusters (HCA). The result is a model that scales context length without the usual quadratic KV blow-up and serves at meaningfully higher throughput than a dense V3 of comparable parameter count. V4-Flash is the smaller member of the family, sized to fit the unified-memory envelope of a workstation Mac rather than a datacenter rack.

Engineering notes — what we shipped

Upstream status

Upstream mlx-lm support landed in mlx-lm #1192 but you'd still need to wire the tokenizer + tool parser yourself to serve V4 over an OpenAI-compatible API. rapid-mlx ships all of that in the alias config.

MoE + MXFP4 + multi-device caveat (mlx#3402). If you go for an MXFP4 quant on a multi-Mac setup, rapid-mlx will print a warning at boot. V4-Flash at 2/4/8-bit is not affected because those quants don't trigger the MXFP4 path.

Pick your quant

Alias Bits Approx. disk Unified-memory floor HF repo
deepseek-v4-flash-2bit 2 (DQ) ~110 GB 192 GB (256 GB comfortable) DeepSeek-V4-Flash-2bit-DQ
deepseek-v4-flash-4bit 4 ~210 GB 256 GB Mac Studio DeepSeek-V4-Flash-4bit
deepseek-v4-flash-8bit 8 ~390 GB Cluster — exceeds single-Mac envelope DeepSeek-V4-Flash-8bit

The 2-bit DQ build is the headline result — it's the smallest quant the team is comfortable putting next to the DeepSeek name without quality regressions on the published evals. If you have a 256 GB Mac Studio, start here.

Recommended settings

V4-Flash inherits the DeepSeek-R1 reasoning lineage. The alias ships with the deepseek tool parser and the deepseek_r1 reasoning parser, so /v1/responses exposes a structured reasoning_content field. Sampling defaults to temperature=0.6, top_p=0.95 per the model card. 

rapid-mlx serve deepseek-v4-flash-2bit \
  --port 8000 \
  --host 127.0.0.1

Tutorial — chat + tool call

curl http://127.0.0.1:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "deepseek-v4-flash-2bit",
    "messages": [{"role": "user", "content": "What is the weather in Tokyo?"}],
    "tools": [
      {"type": "function", "function": {
        "name": "get_weather",
        "description": "Look up current weather",
        "parameters": {
          "type": "object",
          "properties": {"city": {"type": "string"}},
          "required": ["city"]
        }
      }}
    ]
  }'

Python via the OpenAI client, using the reasoning surface:

from openai import OpenAI

client = OpenAI(base_url="http://127.0.0.1:8000/v1", api_key="none")

resp = client.responses.create(
    model="deepseek-v4-flash-2bit",
    input="Plan a three-step approach to optimizing a slow SQL query.",
    reasoning={"effort": "high"},
)

print(resp.output_text)
for item in resp.output:
    if item.type == "reasoning":
        for chunk in item.summary:
            print("[reasoning]", chunk.text)

Performance

TODO — M3 Ultra 256 GB headline decode tok/s and TTFT pending. We'll publish 2-bit-DQ, 4-bit, and a 64K-context curve with the next release.

Known limitations

Related