LangChain.ChatModels.ChatReqLLM (LangChain v0.15.9)

Copy Markdown View Source

ChatModel adapter using the req_llm library as the HTTP/LLM backend.

Provides access to any provider supported by req_llm (Anthropic, OpenAI, Google Gemini, Groq, Ollama, AWS Bedrock, etc.) through the unified LangChain framework.

Model Specification

The model field takes a req_llm-format specifier string: "provider:model_id".

Usage

alias LangChain.ChatModels.ChatReqLLM
alias LangChain.Chains.LLMChain
alias LangChain.Message

# Anthropic via req_llm
llm = ChatReqLLM.new!(%{model: "anthropic:claude-haiku-4-5"})

# OpenAI
llm = ChatReqLLM.new!(%{model: "openai:gpt-4o"})

# Ollama local model
llm = ChatReqLLM.new!(%{model: "ollama:llama3", base_url: "http://localhost:11434"})

# Groq with streaming
llm = ChatReqLLM.new!(%{model: "groq:llama-3.3-70b-versatile", stream: true})

{:ok, chain} =
  %{llm: llm}
  |> LLMChain.new!()
  |> LLMChain.add_message(Message.new_user!("Hello!"))
  |> LLMChain.run()

Tool Use

Tools are translated to req_llm format automatically. The callback field in the req_llm Tool struct is set to a stub — tool execution remains the LLMChain's responsibility, as with all other ChatModel adapters.

Provider Options

Provider-specific options (e.g. thinking, tool_choice, seed) can be passed via provider_opts:

ChatReqLLM.new!(%{
  model: "anthropic:claude-haiku-4-5",
  provider_opts: %{thinking: %{type: "enabled", budget_tokens: 2000}}
})

Functions marked defer_loading: true (see "Tool search options" in LangChain.Function) are declared on every request but kept out of the model's context until it searches for them. Add the search tool to the chain's tools:

%{llm: ChatReqLLM.new!(%{model: "anthropic:claude-sonnet-5-5"})}
|> LLMChain.new!()
|> LLMChain.add_tools([NativeTool.tool_search(), deferred_functions, other_tools])

What reaches the provider depends on the model:

  • Anthropic (anthropic: models, and Anthropic models on amazon_bedrock:): the search tool is req_llm's :tool_search provider option, BM25 or regex as LangChain.NativeTool.tool_search/1 sets it. At least one tool must not be deferred; the search tool counts. A request where every function is deferred and there is no search tool is an error before it is sent.
  • OpenAI (openai: models): the hosted tool_search tool, and defer_loading on each deferred function. A deferred function without the search tool is an error before the request is sent, because the API rejects it. Functions that share a :namespace option go out as one namespace tool, described by :namespace_description from any of them, or "Tools for <namespace>." without one.
  • Any other provider: the search tool is left out and deferred functions are sent as ordinary tools.

Any other LangChain.NativeTool is an error.

Search blocks in the conversation

On Anthropic models a search adds server_tool_use and tool_search_tool_result blocks to the assistant message. On OpenAI models it adds tool_search_call and tool_search_output items. Each becomes a LangChain.Message.ContentPart of type :provider_item in the position it arrived, and goes back unchanged, in place, on every later request. That is how a loaded tool stays loaded, so keep these parts wherever the conversation is stored.

On OpenAI, a call to a function that a search loaded carries a namespace: the namespace's name, or the function's own name when it has none. It is kept in the LangChain.Message.ToolCall's metadata under "namespace", as LangChain.ChatModels.ChatOpenAIResponses keeps it, and sent back with the call. The API rejects a replayed search whose call comes back without its namespace.

When streaming, thinking that follows a search block arrives twice: as thinking text for display, marked stream_only: true, and as a :provider_item holding the signed block. Only the provider item goes back.

A response paused by a server tool is identified by Anthropic's pause_turn stop reason, which req_llm reports in provider_meta["stop_reason"]. It is :complete with metadata[:end_turn] set to false, so LangChain.Chains.LLMChain sends it back and the model continues. Any other :incomplete response is :length.

Loading tools from your own tool

A function can load tools by returning :tool_reference parts. On an Anthropic model each becomes a tool_reference block, and the API loads that tool. Every referenced tool must be declared, normally deferred. A tool result that holds references cannot hold anything else, so the result's other content is sent after the tool results as a user message. On other providers a reference reads as text naming the tool.

Narration (Assistant Phase)

The OpenAI Responses API labels each assistant message item with a phase: "commentary" for the model narrating what it is about to do, often just before a tool call, and "final_answer" for its reply. Each label becomes a marker on the text LangChain.Message.ContentPart it belongs to, "narration" or "answer" (see LangChain.Message.ContentPart.utterance/1). Text with no label is left unmarked, which is every other provider.

A message whose text is entirely narration and has no tool calls is not an answer (LangChain.Message.narration?/1), so LangChain.Chains.LLMChain calls the model again to finish its turn.

The marker goes back with assistant history, because the model degrades when its own commentary is replayed unlabelled. Keep it wherever the conversation is stored.

req_llm labels each content part, so two items stay two parts, and a streaming consumer knows a part is narration from its first token. Going back, each text part carries its label, and req_llm groups consecutive parts that share one into a single message item.

Reasoning Continuity

A reasoning model's own reasoning goes back to it with the assistant message that produced it. OpenAI's Responses API wants its encrypted reasoning items alongside the function call outputs of a tool loop, and Anthropic needs its signed thinking blocks. req_llm reports both as the message's reasoning details and rebuilds the provider's form from them.

The details are kept on the assistant message as one :unsupported LangChain.Message.ContentPart whose :reasoning_details option holds them as plain string-keyed maps, for streamed and non-streamed responses alike. Because the part holds only JSON values, it survives a conversation being stored and reloaded, and a reloaded message sends its reasoning back exactly as the original did. Other chat models skip the part.

Stream Completion

A streamed turn is finished when a chunk marks the response terminal, which req_llm derives from a provider's terminal event. A stream whose body simply ends carries no such chunk, and the accumulated text would then never become a message.

The finish reason req_llm records alongside the stream says which case this is. When it reports a finished response (:stop, :tool_calls, :length, :content_filter), only the marker went missing and the turn is closed here with that status. When it reports anything else, including the :incomplete of a body that ended without a recognized terminal event, the response is left unfinished so LangChain.Chains.LLMChain reports it as an "incomplete_stream" error and retries rather than admitting half a response into the conversation. Both paths log a warning.

Connection Retry Behavior

The retry_count option controls how many times a request is retried when a pooled HTTP connection turns out to be stale (server closed it between requests). This is a transport-level issue where retrying with a fresh connection is the correct response.

Only closed-connection errors are retried. Timeouts, rate limits (429), overloaded (529), authentication errors, and invalid requests all return immediately -- they are not problems that a simple retry will fix.

retry_countTotal HTTP requests
01 (no retries)
12 (1 initial + 1 retry)
2 (default)3 (1 initial + 2 retries)

Req's built-in HTTP retry is disabled to prevent the two retry layers from compounding. See GitHub issue #503.

When running LLM calls from a background job queue (e.g., Oban) that has its own retry logic, set retry_count: 0 so there are no hidden retries:

ChatReqLLM.new!(%{model: "...", retry_count: 0})

Summary

Functions

Call the LLM via req_llm with a prompt or list of messages.

Convert a LangChain ContentPart to a ReqLLM.Message.ContentPart.

Convert a single LangChain.Function to a ReqLLM.Tool with a stub callback.

Convert a list of LangChain Function structs to ReqLLM.Tool structs.

Convert a single LangChain Message to a list of ReqLLM.Message structs.

Convert a list of LangChain messages to a ReqLLM.Context.

Create a ChatReqLLM configuration.

Create a ChatReqLLM configuration, raising on error if invalid.

Determine if an error should be retried via a fallback LLM.

Translate a req_llm finish_reason atom to a LangChain.Message status atom.

Translate a req_llm usage map to a LangChain.TokenUsage struct.

Types

t()

@type t() :: %LangChain.ChatModels.ChatReqLLM{
  api_key: term(),
  base_url: term(),
  callbacks: term(),
  max_tokens: term(),
  model: term(),
  provider_opts: term(),
  receive_timeout: term(),
  req_opts: term(),
  retry_count: term(),
  stream: term(),
  temperature: term(),
  verbose_api: term()
}

Functions

call(model, prompt, functions \\ [])

Call the LLM via req_llm with a prompt or list of messages.

content_part_to_req_llm(part)

@spec content_part_to_req_llm(LangChain.Message.ContentPart.t()) ::
  ReqLLM.Message.ContentPart.t() | nil

Convert a LangChain ContentPart to a ReqLLM.Message.ContentPart.

Returns nil for unsupported types (they are filtered out of the content list).

do_process_response(model, response)

@spec do_process_response(t(), ReqLLM.Response.t()) ::
  LangChain.Message.t() | {:error, LangChain.LangChainError.t()}

Convert a ReqLLM.Response to a LangChain.Message.

function_to_req_llm_tool(fun)

@spec function_to_req_llm_tool(LangChain.Function.t()) :: ReqLLM.Tool.t()

Convert a single LangChain.Function to a ReqLLM.Tool with a stub callback.

The stub callback is never invoked in normal LangChain operation — the tool definition is only used for schema generation (telling the LLM what tools exist).

The :strict flag is passed through so providers that support it (e.g. OpenAI structured outputs) can enforce the parameter schema. A function deferred for tool search (LangChain.Function.deferred?/1) is marked defer_loading for Anthropic and OpenAI. Both the :parameters (list of LangChain.FunctionParam) and :parameters_schema (raw JSONSchema map) forms are supported, mirroring LangChain.ChatModels.ChatOpenAI.

functions_to_req_llm_tools(functions)

@spec functions_to_req_llm_tools([LangChain.Function.t()] | nil) :: [ReqLLM.Tool.t()]

Convert a list of LangChain Function structs to ReqLLM.Tool structs.

Each tool gets a stub callback — tool execution remains the LLMChain's responsibility.

message_to_req_llm_messages(message, references? \\ false)

@spec message_to_req_llm_messages(LangChain.Message.t(), boolean()) :: [
  ReqLLM.Message.t()
]

Convert a single LangChain Message to a list of ReqLLM.Message structs.

Most roles map 1-to-1. The :tool role expands to one message per ToolResult.

With references? true (an Anthropic model), a result's :tool_reference parts become tool_reference blocks, which make the API load those tools. A tool result holding references cannot hold anything else, so the result's other parts follow all the tool results as a user message.

messages_to_req_llm_context(messages, model \\ nil)

@spec messages_to_req_llm_context([LangChain.Message.t()], t() | nil) ::
  ReqLLM.Context.t()

Convert a list of LangChain messages to a ReqLLM.Context.

Tool messages are expanded: a single LangChain :tool message (which may carry multiple ToolResult structs) becomes one ReqLLM.Message per result, matching the one-result-per-message convention expected by OpenAI-compatible providers.

Pass the model so tool references are encoded for its provider. On an Anthropic model they become tool_reference blocks; see message_to_req_llm_messages/2. Without a model they read as text.

new(attrs \\ %{})

@spec new(attrs :: map()) :: {:ok, t()} | {:error, Ecto.Changeset.t()}

Create a ChatReqLLM configuration.

new!(attrs \\ %{})

@spec new!(attrs :: map()) :: t() | no_return()

Create a ChatReqLLM configuration, raising on error if invalid.

retry_on_fallback?(arg1)

@spec retry_on_fallback?(LangChain.LangChainError.t()) :: boolean()

Determine if an error should be retried via a fallback LLM.

translate_finish_reason(other)

@spec translate_finish_reason(atom() | nil) :: atom()

Translate a req_llm finish_reason atom to a LangChain.Message status atom.

translate_stream_chunk(arg1)

@spec translate_stream_chunk(ReqLLM.StreamChunk.t()) :: [LangChain.MessageDelta.t()]

Translate a single ReqLLM.StreamChunk to a list of LangChain.MessageDelta structs.

Returns an empty list for chunks that produce no LangChain deltas (e.g. empty content, non-terminal metadata).

translate_usage(usage)

@spec translate_usage(map() | nil) :: LangChain.TokenUsage.t() | nil

Translate a req_llm usage map to a LangChain.TokenUsage struct.

req_llm hands every provider's usage over as a complete snapshot of the message so far, and a provider may report more than one over a stream. Anthropic reports two, one opening the message and one closing it, which TokenUsage.add/2 combines by keeping the larger count per class.

An unreported class arrives normalized to zero rather than omitted, so the derived :total_tokens a snapshot carries describes only that snapshot. Read a combined total from TokenUsage.total/1.