Chat model for services that expose an OpenAI-compatible Chat Completions endpoint: Cloudflare Workers AI, Groq, OpenRouter, Together, DeepInfra, Fireworks, vLLM, SGLang, Ollama, LM Studio and others.

When to use this rather than ChatOpenAI

LangChain.ChatModels.ChatOpenAI is the client for OpenAI's own API and follows OpenAI's changes, such as sending the token limit as max_completion_tokens and sending system messages under the developer role for reasoning models. Compatible services often do not adopt those changes, and many of them ignore or silently drop what they do not understand. A developer message sent to an SGLang or vLLM server, for example, reaches a model whose chat template has no developer branch and is rendered as nothing, so the model never sees its instructions.

This module targets the part of the API that compatible services share:

• It sends only what you set. Every optional field defaults to nil and is left out of the request when nil. A model configured with only endpoint and model sends exactly model, stream and messages.
• System messages are always sent as system. Every compatible server and chat template handles that role.
• The token limit is sent as max_tokens.
• reasoning_effort is sent whenever it is set, independent of any other option. Accepted values differ between services, so it is not validated here.
• Credentials are explicit. There is no fallback to the global OpenAI API key, and no OpenAI organization or project headers are sent. With api_key: nil, no Authorization header is sent at all, which suits local servers that need no key.

Usage

ChatOpenAICompatible.new!(%{
  endpoint: "https://api.groq.com/openai/v1/chat/completions",
  api_key: System.fetch_env!("GROQ_API_KEY"),
  model: "llama-3.3-70b-versatile"
})

endpoint is the full URL of the chat completions route, not a base URL.

Provider-specific parameters

Anything a service supports that has no field here goes in extra_body. It is merged into the request body last. A nil value removes a key. See LangChain.Utils.merge_extra_body/2 for the merge rules.

ChatOpenAICompatible.new!(%{
  endpoint: "http://localhost:8000/v1/chat/completions",
  model: "Qwen/Qwen3-8B",
  extra_body: %{"top_k" => 20, "repetition_penalty" => 1.05}
})

Extra headers go in req_config:

req_config: %{headers: [{"x-custom-header", "value"}]}

Overriding stream, messages, tools or model through extra_body can break response handling.

Thinking

Compatible services that serve reasoning models return the model's thinking in a reasoning_content field beside content, on choices[].message for a single response and on choices[].delta for each streamed chunk. That thinking becomes a LangChain.Message.ContentPart of type :thinking, held separately from the answer text and ordered ahead of it:

[
  %ContentPart{type: :thinking, content: "Comparing the tenths place..."},
  %ContentPart{type: :text, content: "9.9 is larger."}
]

Thinking parts are never sent back to the service. A conversation presents the same prompt prefix turn after turn, which keeps prompt caching working.

Provider recipes

Cloudflare Workers AI

ChatOpenAICompatible.new!(%{
  endpoint:
    "https://api.cloudflare.com/client/v4/accounts/#{account_id}/ai/v1/chat/completions",
  api_key: api_token,
  model: "@cf/zai-org/glm-5.3-flash",
  reasoning_effort: "low",
  req_config: %{
    headers: [
      {"cf-aig-gateway-id", gateway_id},
      {"x-session-affinity", session_id}
    ]
  }
})

• cf-aig-gateway-id routes the call through a named AI Gateway. On the Workers Free plan some models, GLM among them, are reachable only this way.
• x-session-affinity sends requests that share a session id to the same replica, so a repeated prompt prefix is served from the prompt cache. Use one id for requests that share a system prompt, such as one per conversation. Without it, repeated prefixes are not cached.
• reasoning_effort controls how long the model thinks, which affects latency and output tokens.
• AI Gateway can answer a byte-identical request from its response cache.

Ollama and LM Studio

Local servers need no key:

ChatOpenAICompatible.new!(%{
  endpoint: "http://localhost:11434/v1/chat/completions",
  model: "llama3.2"
})

LM Studio listens on http://localhost:1234/v1/chat/completions by default.

vLLM and SGLang

ChatOpenAICompatible.new!(%{
  endpoint: "http://localhost:8000/v1/chat/completions",
  model: "Qwen/Qwen3-8B"
})

Add api_key when the server was started with one.

OpenRouter and Groq

ChatOpenAICompatible.new!(%{
  endpoint: "https://openrouter.ai/api/v1/chat/completions",
  api_key: System.fetch_env!("OPENROUTER_API_KEY"),
  model: "meta-llama/llama-3.3-70b-instruct"
})

Groq's endpoint is https://api.groq.com/openai/v1/chat/completions.

Token usage

Usage is reported as a LangChain.TokenUsage in the :usage key of the returned message's metadata. When streaming, most services report usage only when stream_options: %{include_usage: true} is set.

Connection Retry Behavior

retry_count controls how many times a request is retried when a pooled HTTP connection turns out to be stale (the server closed it between requests). Only closed-connection errors are retried. Timeouts, rate limits, authentication errors and invalid requests are returned immediately.