# `LangChain.ChatModels.ChatOpenAICompatible`
[🔗](https://github.com/brainlid/langchain/blob/v0.15.3/lib/chat_models/chat_open_ai_compatible.ex#L1)

Chat model for services that expose an OpenAI-compatible Chat Completions
endpoint: Cloudflare Workers AI, Groq, OpenRouter, Together, DeepInfra,
Fireworks, vLLM, SGLang, Ollama, LM Studio and others.

## When to use this rather than `ChatOpenAI`

`LangChain.ChatModels.ChatOpenAI` is the client for OpenAI's own API and
follows OpenAI's changes, such as sending the token limit as
`max_completion_tokens` and sending system messages under the `developer`
role for reasoning models. Compatible services often do not adopt those
changes, and many of them ignore or silently drop what they do not
understand. A `developer` message sent to an SGLang or vLLM server, for
example, reaches a model whose chat template has no `developer` branch and is
rendered as nothing, so the model never sees its instructions.

This module targets the part of the API that compatible services share:

- **It sends only what you set.** Every optional field defaults to `nil` and
  is left out of the request when `nil`. A model configured with only
  `endpoint` and `model` sends exactly `model`, `stream` and `messages`.
- **System messages are always sent as `system`.** Every compatible server
  and chat template handles that role.
- **The token limit is sent as `max_tokens`.**
- **`reasoning_effort` is sent whenever it is set**, independent of any other
  option. Accepted values differ between services, so it is not validated
  here.
- **Credentials are explicit.** There is no fallback to the global OpenAI API
  key, and no OpenAI organization or project headers are sent. With
  `api_key: nil`, no `Authorization` header is sent at all, which suits local
  servers that need no key.

## Usage

    ChatOpenAICompatible.new!(%{
      endpoint: "https://api.groq.com/openai/v1/chat/completions",
      api_key: System.fetch_env!("GROQ_API_KEY"),
      model: "llama-3.3-70b-versatile"
    })

`endpoint` is the full URL of the chat completions route, not a base URL.

## Provider-specific parameters

Anything a service supports that has no field here goes in `extra_body`. It
is merged into the request body last. A `nil` value removes a key. See
`LangChain.Utils.merge_extra_body/2` for the merge rules.

    ChatOpenAICompatible.new!(%{
      endpoint: "http://localhost:8000/v1/chat/completions",
      model: "Qwen/Qwen3-8B",
      extra_body: %{"top_k" => 20, "repetition_penalty" => 1.05}
    })

Extra headers go in `req_config`:

    req_config: %{headers: [{"x-custom-header", "value"}]}

Overriding `stream`, `messages`, `tools` or `model` through `extra_body` can
break response handling.

## Thinking

Compatible services that serve reasoning models return the model's thinking
in a `reasoning_content` field beside `content`, on `choices[].message` for a
single response and on `choices[].delta` for each streamed chunk. That
thinking becomes a `LangChain.Message.ContentPart` of type `:thinking`, held
separately from the answer text and ordered ahead of it:

    [
      %ContentPart{type: :thinking, content: "Comparing the tenths place..."},
      %ContentPart{type: :text, content: "9.9 is larger."}
    ]

Thinking parts are never sent back to the service. A conversation presents
the same prompt prefix turn after turn, which keeps prompt caching working.

## Provider recipes

### Cloudflare Workers AI

    ChatOpenAICompatible.new!(%{
      endpoint:
        "https://api.cloudflare.com/client/v4/accounts/#{account_id}/ai/v1/chat/completions",
      api_key: api_token,
      model: "@cf/zai-org/glm-5.3-flash",
      reasoning_effort: "low",
      req_config: %{
        headers: [
          {"cf-aig-gateway-id", gateway_id},
          {"x-session-affinity", session_id}
        ]
      }
    })

- `cf-aig-gateway-id` routes the call through a named AI Gateway. On the
  Workers Free plan some models, GLM among them, are reachable only this way.
- `x-session-affinity` sends requests that share a session id to the same
  replica, so a repeated prompt prefix is served from the prompt cache. Use
  one id for requests that share a system prompt, such as one per
  conversation. Without it, repeated prefixes are not cached.
- `reasoning_effort` controls how long the model thinks, which affects
  latency and output tokens.
- AI Gateway can answer a byte-identical request from its response cache.

### Ollama and LM Studio

Local servers need no key:

    ChatOpenAICompatible.new!(%{
      endpoint: "http://localhost:11434/v1/chat/completions",
      model: "llama3.2"
    })

LM Studio listens on `http://localhost:1234/v1/chat/completions` by default.

### vLLM and SGLang

    ChatOpenAICompatible.new!(%{
      endpoint: "http://localhost:8000/v1/chat/completions",
      model: "Qwen/Qwen3-8B"
    })

Add `api_key` when the server was started with one.

### OpenRouter and Groq

    ChatOpenAICompatible.new!(%{
      endpoint: "https://openrouter.ai/api/v1/chat/completions",
      api_key: System.fetch_env!("OPENROUTER_API_KEY"),
      model: "meta-llama/llama-3.3-70b-instruct"
    })

Groq's endpoint is `https://api.groq.com/openai/v1/chat/completions`.

## Token usage

Usage is reported as a `LangChain.TokenUsage` in the `:usage` key of the
returned message's `metadata`. When streaming, most services report usage
only when `stream_options: %{include_usage: true}` is set.

## Connection Retry Behavior

`retry_count` controls how many times a request is retried when a pooled
HTTP connection turns out to be stale (the server closed it between
requests). Only closed-connection errors are retried. Timeouts, rate limits,
authentication errors and invalid requests are returned immediately.

# `t`
[🔗](https://github.com/brainlid/langchain/blob/v0.15.3/lib/chat_models/chat_open_ai_compatible.ex#L229)

```elixir
@type t() :: %LangChain.ChatModels.ChatOpenAICompatible{
  api_key: term(),
  callbacks: term(),
  endpoint: term(),
  extra_body: term(),
  frequency_penalty: term(),
  json_response: term(),
  json_schema: term(),
  max_tokens: term(),
  model: term(),
  parallel_tool_calls: term(),
  presence_penalty: term(),
  reasoning_effort: term(),
  receive_timeout: term(),
  req_config: term(),
  retry_count: term(),
  seed: term(),
  stop: term(),
  stream: term(),
  stream_options: term(),
  temperature: term(),
  tool_choice: term(),
  top_p: term(),
  verbose_api: term()
}
```

# `call`
[🔗](https://github.com/brainlid/langchain/blob/v0.15.3/lib/chat_models/chat_open_ai_compatible.ex#L351)

Calls the service passing the ChatOpenAICompatible struct with
configuration, plus either a simple message or the list of messages to act
as the prompt.

Optionally pass in a list of tools available to the LLM for requesting
execution in response.

**NOTE:** This function *can* be used directly, but the primary interface
should be through `LangChain.Chains.LLMChain`.

# `for_api`
[🔗](https://github.com/brainlid/langchain/blob/v0.15.3/lib/chat_models/chat_open_ai_compatible.ex#L291)

```elixir
@spec for_api(t(), [LangChain.Message.t()], LangChain.ChatModels.ChatModel.tools()) ::
  %{
    required(atom()) => any()
  }
```

Return the params formatted for an API request.

# `new`
[🔗](https://github.com/brainlid/langchain/blob/v0.15.3/lib/chat_models/chat_open_ai_compatible.ex#L262)

```elixir
@spec new(attrs :: map()) :: {:ok, t()} | {:error, Ecto.Changeset.t()}
```

Setup a ChatOpenAICompatible client configuration.

# `new!`
[🔗](https://github.com/brainlid/langchain/blob/v0.15.3/lib/chat_models/chat_open_ai_compatible.ex#L277)

```elixir
@spec new!(attrs :: map()) :: t() | no_return()
```

Setup a ChatOpenAICompatible client configuration and return it or raise an
error if invalid.

# `restore_from_map`
[🔗](https://github.com/brainlid/langchain/blob/v0.15.3/lib/chat_models/chat_open_ai_compatible.ex#L612)

Restores the model from the config.

# `retry_on_fallback?`
[🔗](https://github.com/brainlid/langchain/blob/v0.15.3/lib/chat_models/chat_open_ai_compatible.ex#L566)

```elixir
@spec retry_on_fallback?(LangChain.LangChainError.t()) :: boolean()
```

Determine if an error should be retried. If `true`, a fallback LLM may be
used. If `false`, the error is understood to be more fundamental with the
request rather than a service issue and it should not be retried or fallback
to another service.

# `serialize_config`
[🔗](https://github.com/brainlid/langchain/blob/v0.15.3/lib/chat_models/chat_open_ai_compatible.ex#L580)

```elixir
@spec serialize_config(t()) :: %{required(String.t()) => any()}
```

Generate a config map that can later restore the model's configuration.

`api_key`, `callbacks` and `req_config` are left out. `req_config` can carry
credentials in its headers.

---

*Consult [api-reference.md](api-reference.md) for complete listing*