0.11.0 — Quantized live KV cache — int4/int8 on the continuous-batching cache

Released 2026-07-24 · full changelog · GitHub releases

Community contributor: @66Ton99 — Codex Responses long-context handling (#1141).

• The live continuous-batching KV cache is now quantized. --kv-cache-dtype int4 (the default) and int8 previously only shrank the retained prefix cache — the live decode cache stayed bf16, so a long-context or high-concurrency batch still grew its KV memory at full precision (and --disable-prefix-cache saved nothing). The continuous-batching cache is now quantized with dequant-on-read, cutting steady-state KV memory on long-context and multi-request serving. Hybrid, sliding-window and MLA-latent caches stay bf16 where no supported group size fits, so the path is safe to leave on by default (PR #1197, #1199).
• Forced tool-call arguments are now grammar-constrained on reasoning models too. A forced or named tool call (tool_choice="required" or a specific function) on a reasoning model previously opted out of the decode-time grammar while it was inside its thinking budget, leaving the arguments unconstrained. They are now hard-constrained like every other tool call — closing the last gap in the #558 tool-call-integrity work: forced or free, thinking or not, a tool call is guaranteed parseable (PR #1192, #558 line①).