Integration matrix

Real end-to-end pass / xfail state for 11 first-class agents + 3 frameworks × 4 Tier-1 model families = 56 cells. Transcribed from tests/integrations/README.md as of 0.11.3 (post-#1047 Aider + post-#1048 OpenHands + post-#1051 harmony stop-scope fix).

Agent tiers

Two orthogonal tier axes run in parallel: the 4 Tier-1 model families (Qwen 3.6 / Gemma 4 / DeepSeek / gpt-oss) and 5 Tier-1 agents. A Tier-1 agent gets real end-to-end re-verification against the current client binary every release, first-class launch/--setup, and a dedicated maintained guide with a freshness stamp. Tier-2 agents stay in the matrix but are wire-smoke / on-demand and carry a "last verified on …" caveat rather than a per-release guarantee.

TIER-1 claude-code (/v1/messages) · codex-cli (/v1/responses) · hermes · aider · deepseek-harness (all three /v1/chat/completions) — one guardian per wire; the crowded chat-completions wire is covered for depth (Hermes' 60+ tool tool-call torture), reach (Aider's install base), and DeepSeek's own plugin-composed harness protocol.

TIER-2 cursor (IDE-chat only; agent path is locked to Cursor's backend) · opencode · qwen-code · openhands · kilo-code · copilot · droid · kimi-code · gpt-oss (raw SDK) · and the three frameworks.

Totals across all 4 families

60 cells run → 49 PASS · 11 XFAIL · 0 FAIL. All 11 XFAILs are documented root-caused failures (10 architectural, 1 server-side), not test-harness flakes.

Independent re-run on 0.11.5

The table above is the Studio pilot. On 2026-08-01 the same 56 cells were re-run independently against rapid-mlx 0.11.5 on a different machine — an M2 Pro Mac mini, 32 GB — to check the integrations still hold on current code rather than trusting the earlier run. Result: 43 PASS · 12 skipped · 1 XFAIL · 0 FAIL.

Every skip has a named cause, and none of them is an integration defect:

• 4 × OpenHands — its harness needs a Docker daemon, which was not running on the mini.
• 8 × DeepSeek — DeepSeek-R1-Distill-Qwen-32B-4bit returns no tool_calls. A reasoning-distill model declining to emit tool calls is a model-capability limit, not a broken wire; the harness classifies it as a skip rather than a failure for exactly that reason.

The single XFAIL (PydanticAI × DeepSeek-R1-Distill) is the same registered one as the pilot. Model tier differs from the Studio run — this pass used the 4-bit and 8-bit aliases a 32 GB machine can hold (qwen3.5-9b-4bit, gemma-4-12b-4bit, gpt-oss-20b-mxfp4-q8, deepseek-r1-32b-4bit), so the two runs are complementary rather than identical.

Cheap-alias policy

The matrix boots the smallest viable alias per family so the per-process resident footprint stays under the M3 Ultra OOM budget (operator services baseline + matrix + Metal overhead ≤ 150 GB). Boot aliases used for the 2026-07-07 pilot:

Agent × Family matrix

11 agents × 4 families = 44 cells.

Framework × Family matrix

3 frameworks × 4 families = 12 cells.

XFAIL reasons (root-caused)

XFAIL (arch) — DeepSeek R1-Distill tool-emission gap

9 DeepSeek tool-call cells (7 agents + LangChain + PydanticAI) strict-XFAIL on the deepseek-r1-32b-4bit boot alias. R1's post-training was reasoning-only per DeepSeek's own paper (arXiv 2501.12948 §2.3.3), and distillation into Qwen 32B lost the base model's tool-emission behavior. The refusal "I cannot provide the current weather in Tokyo as I cannot access the get_weather tool." reproduces deterministically at both 4bit (16 GB) and 8bit (34.8 GB) quant levels — not a rapid-mlx parser bug, not a quant artifact.

Text-only cells (CodexCLI + ClaudeCode) and smolagents' code-execution routing PASS on the same booted server, proving the wire is healthy. Aider and OpenHands PASS too because they parse text-action tags out of plain-text LLM output (SEARCH/REPLACE + <execute_ipython>), not via OpenAI tool_calls. Full tool-trained coverage needs V4-Chat / V4-Coder / V4-Flash weights, all > 96 GB — tracked in follow-up #1041.

XFAIL (format) — gpt-oss + OpenHands

One cell — gpt-oss on OpenHands — XFAILs because gpt-oss's native harmony output never emits the <execute_bash> / <execute_ipython> text-action tags that OpenHands' CodeActAgent parses out of plain text. This is an upstream OpenHands parser gap, tracked at All-Hands-AI/OpenHands #15167, not a rapid-mlx bug.

A separate rapid-mlx wire-level bug — user-supplied stop=['</execute_ipython>', ...] was applied against the raw token stream across both harmony channels, so the model's analysis-channel CoT triggered a premature stop — was root-caused inside the OpenHands container (conftest.py::_GPTOSS_OPENHANDS_XFAIL_REASON) and fixed in PR #1051. That fix does not flip the cell to PASS: the harness still strict-xfails it, because the text-action format gap above is upstream and unresolved.

Reproduce locally

Boot the server for a family shard first, then run one matrix:

$ rapid-mlx serve qwen3.6-35b-8bit \
    --tool-call-parser hermes --enable-auto-tool-choice

# All 44 agent cells; only the family matching the running server passes,
# the other three skip (non-strict) or fail (strict).
$ pytest tests/integrations/test_agents_matrix.py -v

# Strict CI — per-family shard (this is the intended workflow).
$ RAPID_MLX_MATRIX_STRICT=1 RAPID_MLX_AGENT_MATRIX_FAMILY=qwen36 \
    pytest tests/integrations/test_agents_matrix.py

Environment overrides

Next steps