Integration matrix

Real end-to-end pass / xfail state for 11 first-class agents + 3 frameworks × 4 Tier-1 model families = 56 cells. Transcribed from tests/integrations/README.md as of 0.10.3 (post-#1047 Aider + post-#1048 OpenHands + post-#1051 harmony stop-scope fix).

Totals across all 4 families

56 cells run → 46 PASS · 10 XFAIL · 0 FAIL. All 10 XFAILs are documented root-caused failures (9 architectural, 1 server-side), not test-harness flakes.

Cheap-alias policy

The matrix boots the smallest viable alias per family so the per-process resident footprint stays under the M3 Ultra OOM budget (operator services baseline + matrix + Metal overhead ≤ 150 GB). Boot aliases used for the 2026-07-07 pilot:

Agent × Family matrix

11 agents × 4 families = 44 cells.

Framework × Family matrix

3 frameworks × 4 families = 12 cells.

XFAIL reasons (root-caused)

XFAIL (arch) — DeepSeek R1-Distill tool-emission gap

9 DeepSeek tool-call cells (7 agents + LangChain + PydanticAI) strict-XFAIL on the deepseek-r1-32b-4bit boot alias. R1's post-training was reasoning-only per DeepSeek's own paper (arXiv 2501.12948 §2.3.3), and distillation into Qwen 32B lost the base model's tool-emission behavior. The refusal "I cannot provide the current weather in Tokyo as I cannot access the get_weather tool." reproduces deterministically at both 4bit (16 GB) and 8bit (34.8 GB) quant levels — not a rapid-mlx parser bug, not a quant artifact.

Text-only cells (CodexCLI + ClaudeCode) and smolagents' code-execution routing PASS on the same booted server, proving the wire is healthy. Aider and OpenHands PASS too because they parse text-action tags out of plain-text LLM output (SEARCH/REPLACE + <execute_ipython>), not via OpenAI tool_calls. Full tool-trained coverage needs V4-Chat / V4-Coder / V4-Flash weights, all > 96 GB — tracked in follow-up #1041.

XFAIL (server) — gpt-oss + OpenHands, resolved in-flight

One cell — gpt-oss on OpenHands — XFAILs on the historical scoping bug in the harmony parser: user-supplied stop=['', ...] applied against the raw token stream across both channels, so the model's analysis-channel CoT (which mentions </execute_ipython> while reasoning about the action) triggered a premature stop before the final channel emitted visible content.

Root-caused empirically inside the OpenHands container (conftest.py::_GPTOSS_OPENHANDS_XFAIL_REASON block). Server-side fix landed in PR #1051 (harmony parser now restricts user-supplied stops to final-channel content). The matrix's XFAIL cell will flip PASS once the harness re-runs against a rapid-mlx build that includes #1051 — the source README hasn't been re-transcribed yet.

Reproduce locally

Boot the server for a family shard first, then run one matrix:

$ rapid-mlx serve qwen3.6-35b-8bit \
    --tool-call-parser hermes --enable-auto-tool-choice

# All 44 agent cells; only the family matching the running server passes,
# the other three skip (non-strict) or fail (strict).
$ pytest tests/integrations/test_agents_matrix.py -v

# Strict CI — per-family shard (this is the intended workflow).
$ RAPID_MLX_MATRIX_STRICT=1 RAPID_MLX_AGENT_MATRIX_FAMILY=qwen36 \
    pytest tests/integrations/test_agents_matrix.py

Environment overrides

Next steps