LangChain (+ LangGraph)

LangChain — and the graph runtime LangGraph built on the same clients — point at rapid-mlx via ChatOpenAI(base_url=…). Structured output, tool calls, streaming, and multi-tool selection all work against the shipped 0.10.3 server.

Install

$ pip install langchain-openai

Config

Constructor kwargs (verified by tests/integrations/test_langchain.py):

from langchain_openai import ChatOpenAI

llm = ChatOpenAI(
    model="default",
    base_url="http://localhost:8000/v1",
    api_key="not-needed",
    temperature=0.0,
)

Or use env vars (per rapid-mlx agents langchain):

$ export OPENAI_BASE_URL=http://localhost:8000/v1
$ export OPENAI_API_KEY=not-needed

Run

$ rapid-mlx serve qwen3.5-9b-4bit \
    --tool-call-parser hermes --enable-auto-tool-choice
$ python -c "from langchain_openai import ChatOpenAI; \
  llm = ChatOpenAI(model='default', base_url='http://localhost:8000/v1', api_key='not-needed'); \
  print(llm.invoke('Say hello').content)"

Recommended aliases (per rapid-mlx agents langchain): qwen3.5-9b-4bit, qwen3.6-35b-4bit, qwen3.5-4b-4bit.

Tool calling

LangChain's bind_tools converts Python callables into the OpenAI tools schema. rapid-mlx returns tool_calls in the standard OpenAI shape. From test_langchain.py:

from langchain_core.tools import tool
from langchain_core.messages import HumanMessage

@tool
def get_weather(city: str) -> str:
    """Get weather for a city."""
    return f"sunny, 22C in {city}"

llm_with_tools = llm.bind_tools([get_weather])
r = llm_with_tools.invoke([HumanMessage(content="What's the weather in Paris?")])
# r.tool_calls[0] == {"name": "get_weather", "args": {"city": "Paris"}, ...}

Structured output

with_structured_output(BaseModel) routes through rapid-mlx's guided-JSON path — which requires the [guided] extra. Without it, the server returns guided_extra_required and the test cleanly SKIPs (not FAILs). Install:

$ pip install "rapid-mlx[guided]"

from pydantic import BaseModel, Field

class Person(BaseModel):
    name: str = Field(description="The person's name")
    age: int = Field(description="The person's age in years")

structured_llm = llm.with_structured_output(Person)
r = structured_llm.invoke([HumanMessage(content="Extract: 'Bob is 42 years old'")])
# r == Person(name="Bob", age=42)

Gotchas

• with_structured_output() overrides max_tokens to None. Thinking-family models burn budget in the reasoning block before JSON output — set max_tokens explicitly in the constructor for reasoning models.
• Guided JSON needs the [guided] extra. pip install "rapid-mlx[guided]". If not installed, the server returns {"error": {"code": "guided_extra_required"}}.
• LangGraph reuses the same client. No extra config — from langgraph.prebuilt import create_react_agent works against the same ChatOpenAI instance.
• DeepSeek R1-Distill XFAILs tool calls. Same architectural gap as every other function-calling client — see XFAIL arch.

Empirical

The LangChain row of the integration matrix is ✅ on Qwen 3.6, Gemma 4, and gpt-oss; DeepSeek R1-Distill XFAIL(arch). Deep flow assertions in test_langchain.py cover plain invoke, system prompt, streaming, single tool call, multi-tool selection, and structured output — see test_langchain.py.

See also