Hero model · vision-grounded computer-use agent

UI-TARS on Apple Silicon

ByteDance's vision-grounded UI automation agent. UI-TARS reads screen pixels and emits click / type / scroll actions — the operator side of a computer-use loop. rapid-mlx ships nine MLX quants across the 1.5-7B, 7B-SFT, 7B-DPO, and 72B-DPO variants, with the custom ui_tars tool parser wired into the alias config.

TL;DR — one command. pip install rapid-mlx && rapid-mlx serve ui-tars-1.5-7b-4bit runs the latest 1.5 SFT checkpoint at 4-bit on any 16 GB+ Apple Silicon Mac. The ui_tars parser is auto-selected, so the OpenAI /v1/chat/completions endpoint emits clean tool_calls for every screen action. Headline tok/s: TODO — bench pending on M3 Ultra.

Why UI-TARS matters

Computer-use agents are the niche where a small open VLM serving on a laptop is dramatically more useful than a cloud API: latency, privacy of the screen, and the long tail of "this app, on this OS, with these accessibility settings" all bias toward local execution. UI-TARS is the strongest open competitor in that niche — better grounding than generic VLMs because it was post-trained on a large screen-action dataset. The 1.5-7B is small enough to live next to whatever it is operating; the 72B-DPO is for the rare case where a single workstation drives a fleet of remote desktops.

Engineering notes — what we shipped

UI-TARS was the case study for our streaming-parser hardening epic — twelve commits between r5-B and r10-F landed across PRs #818, #823, #829, #833, #834, and #850. The thread of bugs and fixes:

These fixes are why rapid-mlx serve ui-tars-1.5-7b-4bit works end-to-end on day one. The parser source lives at vllm_mlx/tool_parsers/ui_tars_tool_parser.py and the streaming reasoning parser at vllm_mlx/reasoning/ui_tars_parser.py.

Upstream status

Upstream mlx-vlm can load the UI-TARS weights but does not understand the action grammar — it emits the raw Thought: … Action: click(x,y) string as chat content. rapid-mlx converts that into tool_calls with a strongly-typed JSON arguments payload, which is what downstream operator frameworks (browser controllers, OSWorld harnesses) actually consume.

Pick your quant

All nine quants serve through rapid-mlx serve <alias> with no extra flags.

Alias Bits Approx. disk Recommended for HF repo
ui-tars-1.5-7b-4bit4~4.0 GBDefault — latest 1.5 SFT, fits 16 GB MacUI-TARS-1.5-7B-4bit
ui-tars-1.5-7b-6bit6~5.5 GBQuality bump on 24 GB+UI-TARS-1.5-7B-6bit
ui-tars-1.5-7b-8bit8~7.5 GBLossless tier for eval runsUI-TARS-1.5-7B-8bit
ui-tars-7b-sft-4bit4~4.0 GBOriginal 7B SFT — broadest task coverageUI-TARS-7B-SFT-4bit
ui-tars-7b-sft-8bit8~7.5 GBOriginal 7B SFT, lossless tierUI-TARS-7B-SFT-8bit
ui-tars-7b-dpo-4bit4~4.0 GBDPO-aligned 7B — better instruction followingUI-TARS-7B-DPO-4bit
ui-tars-7b-dpo-6bit6~5.5 GBDPO 7B quality bumpUI-TARS-7B-DPO-6bit
ui-tars-7b-dpo-8bit8~7.5 GBDPO 7B lossless tierUI-TARS-7B-DPO-8bit
ui-tars-72b-dpo-4bit4~40 GBM3 Ultra workstation — best accuracyUI-TARS-72B-DPO-4bit

Recommended settings

UI-TARS is sensitive to sampling. The defaults baked into the alias config are temperature=0.0, top_p=1.0 — deterministic, because operator agents that take physical-world actions on the user's machine must be reproducible. Override only if you're running an exploration / data-collection workload. 

rapid-mlx serve ui-tars-1.5-7b-4bit \
  --port 8000 \
  --host 127.0.0.1

The system prompt is auto-prepended when the request includes tools=. If you supply your own sysprompt, the parser will still produce tool_calls as long as the Thought: / Action: grammar is present in the model's output.

Tutorial — cURL with a screen-action tool

curl http://127.0.0.1:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "ui-tars-1.5-7b-4bit",
    "messages": [
      {"role": "user", "content": [
        {"type": "text", "text": "Click the Submit button."},
        {"type": "image_url", "image_url": {"url": "data:image/png;base64,<screenshot>"}}
      ]}
    ],
    "tools": [
      {"type": "function", "function": {
        "name": "click",
        "description": "Click at screen coordinates",
        "parameters": {
          "type": "object",
          "properties": {"x": {"type": "number"}, "y": {"type": "number"}},
          "required": ["x", "y"]
        }
      }}
    ]
  }'

Python via the OpenAI client:

from openai import OpenAI

client = OpenAI(base_url="http://127.0.0.1:8000/v1", api_key="none")

resp = client.chat.completions.create(
    model="ui-tars-1.5-7b-4bit",
    messages=[
        {"role": "user", "content": [
            {"type": "text", "text": "Click the Submit button."},
            {"type": "image_url", "image_url": {"url": f"data:image/png;base64,{screenshot_b64}"}},
        ]},
    ],
    tools=[{
        "type": "function",
        "function": {
            "name": "click",
            "description": "Click at screen coordinates",
            "parameters": {
                "type": "object",
                "properties": {"x": {"type": "number"}, "y": {"type": "number"}},
                "required": ["x", "y"],
            },
        },
    }],
)
for call in resp.choices[0].message.tool_calls:
    print(call.function.name, call.function.arguments)

Performance

TODO — M-series headline tok/s pending. We'll publish 1.5-7B-4bit and 72B-DPO-4bit numbers on M3 Ultra alongside the next release.

Known limitations

Related