0.11.9 — Requests that hang, and requests that lie

Released 2026-08-02 · full changelog · GitHub releases

Every fix here is a case where the server failed to give an honest answer: it either never came back, or it reported success on output it had already judged bad. Nothing new is added; seven ways to be misled are removed.

• A streaming tool call could wedge a connection indefinitely. Once the tool parser sees an opener (<tool_call>, <function=), the post-processor withholds every delta until the block closes — and that suppression had no bound. A block that never closed held all content forever, so the SSE generator emitted nothing but : keepalive comments and no timeout at any layer could fire: the reported request generated 2,207 tokens, put 1 chunk on the wire, and ran 518 s — outliving its own 300 s client timeout. From the client it was indistinguishable from a server that never answered. Withheld bytes are now capped at 64 KB, after which the text is released as content and tool suppression latches off for the rest of the turn. The cap only applies before any tool call has reached the wire, so genuine incrementally-streamed tool calls are never touched (issue #1359, PR #1391).
• A forced tool call no longer fabricates arguments it knows are invalid. Under tool_choice: "required" or a named function, when the parser surfaced no call the route synthesised one to honour the tool-call-guaranteed contract — with arguments: "{}". If the target tool's schema declared required properties, that was a call the server already knew was schema-invalid, handed to the client under finish_reason: "tool_calls" for it to parse and execute. Unmet required properties now return 422 on chat non-stream and on both Responses sites; streaming cannot 422 after headers are sent, so it finishes stop rather than fabricating. A tool with no required fields still synthesises {} exactly as before (issue #1256, PR #1394).
• The repetition guard reports a failure instead of a success. Running Codex against a local DeepSeek V4 build, the model emitted ambigu over and over. rapid-mlx correctly stopped it at 304 completion tokens — then logged finished normally, so Codex emitted turn.completed and exited successfully without changing a single file. Guard termination is now an aborted generation carrying an explicit error (PR #1396).
• Short degenerate loops are caught before they exhaust Metal. A 46,475-token Responses request fell into a repeated short CJK loop, generated 10,844 completion tokens, and hit [metal::malloc] Resource limit (499000) exceeded around scheduler step 12,032 — and the recovery path emitted a terminal output with no error, so the request was reported as a success. Sustained 1–5 token loops are now detected, with a conservative 256-repeat floor, reading only the trailing 768 tokens regardless of context length, and still restricted to tool-bearing requests. Scheduler-local Metal failures now propagate through RequestOutput.error (PR #1393).
• Embeddings no longer leak Metal buffers. /v1/embeddings never called mx.clear_cache(), unlike every LLM path in the engine — and because padding=True makes each batch a different sequence length, MLX's size-keyed allocator pool almost never had a reusable block. It only grew: roughly 70 MB retained per input text, 2.3 GB → 24 GB over 320 texts, and about 50 GB in a week-old production process. Buffers are now released after each batch on both the string and pre-tokenized paths (issue #1380, PR #1390).
• A warm start is no longer an outage. The persisted prefix cache was loaded synchronously between engine start and readiness, so /health/ready and /v1/models returned 503 for the entire multi-second read — an orchestrator gating traffic on readiness saw a warm start as downtime. Readiness now flips first and the cache warms in the background; shutdown cancels a load still in flight. This is safe because each entry installs as a single atomic swap under its own lock, so an early request either misses and recomputes (always correct) or hits a fully-installed cache — never a half-populated one (issue #1350, PR #1392).
• ffmpeg is found when it is not on PATH. Video remux and crop now resolve the binary from FFMPEG_BINARY, then the process PATH, then the usual Homebrew and system locations — so a GUI-launched process with a minimal environment completes instead of failing at the last step (issue #1352, PR #1397).