ChatModel adapter using the req_llm library as the HTTP/LLM backend.
Provides access to any provider supported by req_llm (Anthropic, OpenAI, Google Gemini, Groq, Ollama, AWS Bedrock, etc.) through the unified LangChain framework.
Model Specification
The model field takes a req_llm-format specifier string: "provider:model_id".
Usage
alias LangChain.ChatModels.ChatReqLLM
alias LangChain.Chains.LLMChain
alias LangChain.Message
# Anthropic via req_llm
llm = ChatReqLLM.new!(%{model: "anthropic:claude-haiku-4-5"})
# OpenAI
llm = ChatReqLLM.new!(%{model: "openai:gpt-4o"})
# Ollama local model
llm = ChatReqLLM.new!(%{model: "ollama:llama3", base_url: "http://localhost:11434"})
# Groq with streaming
llm = ChatReqLLM.new!(%{model: "groq:llama-3.3-70b-versatile", stream: true})
{:ok, chain} =
%{llm: llm}
|> LLMChain.new!()
|> LLMChain.add_message(Message.new_user!("Hello!"))
|> LLMChain.run()Tool Use
Tools are translated to req_llm format automatically. The callback field in the
req_llm Tool struct is set to a stub — tool execution remains the LLMChain's
responsibility, as with all other ChatModel adapters.
Provider Options
Provider-specific options (e.g. thinking, tool_choice, seed) can be passed
via provider_opts:
ChatReqLLM.new!(%{
model: "anthropic:claude-haiku-4-5",
provider_opts: %{thinking: %{type: "enabled", budget_tokens: 2000}}
})Tool Search
Functions marked defer_loading: true (see "Tool search options" in
LangChain.Function) are declared on every request but kept out of the
model's context until it searches for them. Add the search tool to the
chain's tools:
%{llm: ChatReqLLM.new!(%{model: "anthropic:claude-sonnet-5-5"})}
|> LLMChain.new!()
|> LLMChain.add_tools([NativeTool.tool_search(), deferred_functions, other_tools])What reaches the provider depends on the model:
- Anthropic (
anthropic:models, and Anthropic models onamazon_bedrock:): the search tool is req_llm's:tool_searchprovider option, BM25 or regex asLangChain.NativeTool.tool_search/1sets it. At least one tool must not be deferred; the search tool counts. A request where every function is deferred and there is no search tool is an error before it is sent. - OpenAI (
openai:models): the hostedtool_searchtool, anddefer_loadingon each deferred function. A deferred function without the search tool is an error before the request is sent, because the API rejects it. Functions that share a:namespaceoption go out as onenamespacetool, described by:namespace_descriptionfrom any of them, or "Tools for <namespace>." without one. - Any other provider: the search tool is left out and deferred functions are sent as ordinary tools.
Any other LangChain.NativeTool is an error.
Search blocks in the conversation
On Anthropic models a search adds server_tool_use and
tool_search_tool_result blocks to the assistant message. On OpenAI
models it adds tool_search_call and tool_search_output items. Each
becomes a LangChain.Message.ContentPart of type :provider_item in the
position it arrived, and goes back unchanged, in place, on every later
request. That is how a loaded tool stays loaded, so keep these parts
wherever the conversation is stored.
On OpenAI, a call to a function that a search loaded carries a
namespace: the namespace's name, or the function's own name when it has
none. It is kept in the LangChain.Message.ToolCall's metadata under
"namespace", as LangChain.ChatModels.ChatOpenAIResponses keeps it, and
sent back with the call. The API rejects a replayed search whose call
comes back without its namespace.
When streaming, thinking that follows a search block arrives twice: as
thinking text for display, marked stream_only: true, and as a
:provider_item holding the signed block. Only the provider item goes
back.
A response paused by a server tool is identified by Anthropic's
pause_turn stop reason, which req_llm reports in
provider_meta["stop_reason"]. It is :complete with
metadata[:end_turn] set to false, so LangChain.Chains.LLMChain sends
it back and the model continues. Any other :incomplete response is
:length.
Loading tools from your own tool
A function can load tools by returning :tool_reference parts. On an
Anthropic model each becomes a tool_reference block, and the API loads
that tool. Every referenced tool must be declared, normally deferred. A
tool result that holds references cannot hold anything else, so the
result's other content is sent after the tool results as a user message.
On other providers a reference reads as text naming the tool.
Narration (Assistant Phase)
The OpenAI Responses API labels each assistant message item with a
phase: "commentary" for the model narrating what it is about to do,
often just before a tool call, and "final_answer" for its reply. Each
label becomes a marker on the text
LangChain.Message.ContentPart it belongs to, "narration" or
"answer" (see LangChain.Message.ContentPart.utterance/1). Text with no
label is left unmarked, which is every other provider.
A message whose text is entirely narration and has no tool calls is not an
answer (LangChain.Message.narration?/1), so
LangChain.Chains.LLMChain calls the model again to finish its turn.
The marker goes back with assistant history, because the model degrades when its own commentary is replayed unlabelled. Keep it wherever the conversation is stored.
req_llm labels each content part, so two items stay two parts, and a streaming consumer knows a part is narration from its first token. Going back, each text part carries its label, and req_llm groups consecutive parts that share one into a single message item.
Reasoning Continuity
A reasoning model's own reasoning goes back to it with the assistant message that produced it. OpenAI's Responses API wants its encrypted reasoning items alongside the function call outputs of a tool loop, and Anthropic needs its signed thinking blocks. req_llm reports both as the message's reasoning details and rebuilds the provider's form from them.
The details are kept on the assistant message as one :unsupported
LangChain.Message.ContentPart whose :reasoning_details option holds
them as plain string-keyed maps, for streamed and non-streamed responses
alike. Because the part holds only JSON values, it survives a conversation
being stored and reloaded, and a reloaded message sends its reasoning back
exactly as the original did. Other chat models skip the part.
Stream Completion
A streamed turn is finished when a chunk marks the response terminal, which req_llm derives from a provider's terminal event. A stream whose body simply ends carries no such chunk, and the accumulated text would then never become a message.
The finish reason req_llm records alongside the stream says which case this
is. When it reports a finished response (:stop, :tool_calls, :length,
:content_filter), only the marker went missing and the turn is closed here
with that status. When it reports anything else, including the :incomplete
of a body that ended without a recognized terminal event, the response is
left unfinished so LangChain.Chains.LLMChain reports it as an
"incomplete_stream" error and retries rather than admitting half a
response into the conversation. Both paths log a warning.
Connection Retry Behavior
The retry_count option controls how many times a request is retried when
a pooled HTTP connection turns out to be stale (server closed it between
requests). This is a transport-level issue where retrying with a fresh
connection is the correct response.
Only closed-connection errors are retried. Timeouts, rate limits (429), overloaded (529), authentication errors, and invalid requests all return immediately -- they are not problems that a simple retry will fix.
retry_count | Total HTTP requests |
|---|---|
0 | 1 (no retries) |
1 | 2 (1 initial + 1 retry) |
2 (default) | 3 (1 initial + 2 retries) |
Req's built-in HTTP retry is disabled to prevent the two retry layers from compounding. See GitHub issue #503.
When running LLM calls from a background job queue (e.g., Oban) that has its
own retry logic, set retry_count: 0 so there are no hidden retries:
ChatReqLLM.new!(%{model: "...", retry_count: 0})
Summary
Functions
Call the LLM via req_llm with a prompt or list of messages.
Convert a LangChain ContentPart to a ReqLLM.Message.ContentPart.
Convert a ReqLLM.Response to a LangChain.Message.
Convert a single LangChain.Function to a ReqLLM.Tool with a stub callback.
Convert a list of LangChain Function structs to ReqLLM.Tool structs.
Convert a single LangChain Message to a list of ReqLLM.Message structs.
Convert a list of LangChain messages to a ReqLLM.Context.
Create a ChatReqLLM configuration.
Create a ChatReqLLM configuration, raising on error if invalid.
Determine if an error should be retried via a fallback LLM.
Translate a req_llm finish_reason atom to a LangChain.Message status atom.
Translate a single ReqLLM.StreamChunk to a list of LangChain.MessageDelta structs.
Translate a req_llm usage map to a LangChain.TokenUsage struct.
Types
Functions
Call the LLM via req_llm with a prompt or list of messages.
@spec content_part_to_req_llm(LangChain.Message.ContentPart.t()) :: ReqLLM.Message.ContentPart.t() | nil
Convert a LangChain ContentPart to a ReqLLM.Message.ContentPart.
Returns nil for unsupported types (they are filtered out of the content list).
@spec do_process_response(t(), ReqLLM.Response.t()) :: LangChain.Message.t() | {:error, LangChain.LangChainError.t()}
Convert a ReqLLM.Response to a LangChain.Message.
@spec function_to_req_llm_tool(LangChain.Function.t()) :: ReqLLM.Tool.t()
Convert a single LangChain.Function to a ReqLLM.Tool with a stub callback.
The stub callback is never invoked in normal LangChain operation — the tool definition is only used for schema generation (telling the LLM what tools exist).
The :strict flag is passed through so providers that support it (e.g. OpenAI
structured outputs) can enforce the parameter schema. A function deferred
for tool search (LangChain.Function.deferred?/1) is marked
defer_loading for Anthropic and OpenAI. Both the :parameters
(list of LangChain.FunctionParam) and :parameters_schema (raw JSONSchema
map) forms are supported, mirroring LangChain.ChatModels.ChatOpenAI.
@spec functions_to_req_llm_tools([LangChain.Function.t()] | nil) :: [ReqLLM.Tool.t()]
Convert a list of LangChain Function structs to ReqLLM.Tool structs.
Each tool gets a stub callback — tool execution remains the LLMChain's responsibility.
@spec message_to_req_llm_messages(LangChain.Message.t(), boolean()) :: [ ReqLLM.Message.t() ]
Convert a single LangChain Message to a list of ReqLLM.Message structs.
Most roles map 1-to-1. The :tool role expands to one message per ToolResult.
With references? true (an Anthropic model), a result's :tool_reference
parts become tool_reference blocks, which make the API load those tools.
A tool result holding references cannot hold anything else, so the result's
other parts follow all the tool results as a user message.
@spec messages_to_req_llm_context([LangChain.Message.t()], t() | nil) :: ReqLLM.Context.t()
Convert a list of LangChain messages to a ReqLLM.Context.
Tool messages are expanded: a single LangChain :tool message (which may carry
multiple ToolResult structs) becomes one ReqLLM.Message per result, matching
the one-result-per-message convention expected by OpenAI-compatible providers.
Pass the model so tool references are encoded for its provider. On an
Anthropic model they become tool_reference blocks; see
message_to_req_llm_messages/2. Without a model they read as text.
@spec new(attrs :: map()) :: {:ok, t()} | {:error, Ecto.Changeset.t()}
Create a ChatReqLLM configuration.
Create a ChatReqLLM configuration, raising on error if invalid.
@spec retry_on_fallback?(LangChain.LangChainError.t()) :: boolean()
Determine if an error should be retried via a fallback LLM.
Translate a req_llm finish_reason atom to a LangChain.Message status atom.
@spec translate_stream_chunk(ReqLLM.StreamChunk.t()) :: [LangChain.MessageDelta.t()]
Translate a single ReqLLM.StreamChunk to a list of LangChain.MessageDelta structs.
Returns an empty list for chunks that produce no LangChain deltas (e.g. empty content, non-terminal metadata).
@spec translate_usage(map() | nil) :: LangChain.TokenUsage.t() | nil
Translate a req_llm usage map to a LangChain.TokenUsage struct.
req_llm hands every provider's usage over as a complete snapshot of the
message so far, and a provider may report more than one over a stream.
Anthropic reports two, one opening the message and one closing it, which
TokenUsage.add/2 combines by keeping the larger count per class.
An unreported class arrives normalized to zero rather than omitted, so the
derived :total_tokens a snapshot carries describes only that snapshot.
Read a combined total from TokenUsage.total/1.