Custom LLM

By default your agent generates replies with a Fish Audio platform model. With a custom LLM, every reply comes from an endpoint you host instead, whether that is your own fine-tuned model, your own memory and conversation logic, or a proxy inside your VPC. Fish Audio keeps running the rest of the call, including speech recognition, voice synthesis, interruption handling, tool execution, recordings, and billing.

How it works

Your endpoint speaks the standard OpenAI chat completions protocol. The platform sends POST {base_url}/chat/completions with stream: true and reads the reply as server-sent events, so if your prototype already runs against another voice-agent platform through a custom LLM, the same server works here unchanged. Each conversation turn, your endpoint receives the same fully assembled context a platform model would see. The messages array carries the system prompt with dynamic variables and session overrides applied, the complete conversation history, and any retrieved knowledge, and the agent’s tools are included in OpenAI function format. When your model returns tool_calls, the platform executes the tool and calls you again with the result. A few extra top-level fields ride each request body so your server can look up its own state:

Field	Content
`session_id`	The Fish Audio session id.
`user_id`	The `end_user_id` you passed when creating the session. Omitted when the session has none.
`fishaudio_extra_body`	The `llm_extra_body` object you passed when creating the session, forwarded verbatim on every request. Use it to carry your own identifiers, such as which chat or thread the user is in. Omitted when the session has none.

If your server proxies requests to an upstream provider such as OpenAI, consider removing session_id, user_id, and fishaudio_extra_body from the body first. Some upstream APIs reject parameters they don’t recognize.

Example request

Suppose the session was created from your backend with end_user_id: "user-42" and llm_extra_body: {"chat_id": "chat-9"}, the agent has one webhook tool lookup_order, and the user has just asked about an order. Your endpoint receives:

POST /v1/chat/completions HTTP/1.1
Host: llm.example.com
Authorization: Bearer sk-your-endpoint-key
Content-Type: application/json

{
  "model": "persona-70b",
  "stream": true,
  "stream_options": { "include_usage": true },
  "messages": [
    {
      "role": "system",
      "content": "Today is Wednesday, August 26, 2026. Session timezone: America/New_York (UTC-4).\n\n…\n\nYou are Aria, the support assistant for Acme Shoes. Help customers with orders and returns. …"
    },
    { "role": "assistant", "content": "Hi, this is Aria from Acme Shoes. How can I help you today?" },
    { "role": "user", "content": "I want to check on my order." },
    { "role": "assistant", "content": "Sure, what's the order number?" },
    { "role": "system", "content": "Current date and time: Wednesday, August 26, 2026, 14:07 (America/New_York)." },
    { "role": "user", "content": "It's A one two three four five." }
  ],
  "tools": [
    {
      "type": "function",
      "function": {
        "name": "lookup_order",
        "description": "Look up an order by its number.",
        "parameters": {
          "type": "object",
          "properties": {
            "order_number": { "type": "string", "description": "The order number, e.g. A12345." }
          },
          "required": ["order_number"]
        }
      }
    }
  ],
  "session_id": "sess_01j9x4k2m8v3q7n5p6r8t9w0y1",
  "user_id": "user-42",
  "fishaudio_extra_body": { "chat_id": "chat-9" }
}

A few things to note:

The first system message is the assembled prompt: your agent’s system prompt with dynamic variables and overrides applied, plus the platform’s own context lines. A short system line carrying the current time is inserted before the latest user turn on every request.
messages carries the full transcript so far. Speech recognition output arrives as plain user text; your earlier replies come back as assistant messages.
tools is present only when the agent has tools configured. The model, stream, and stream_options fields are fixed; no temperature or max_tokens is sent, so apply your own defaults.
A tool whose parameters are validated arrives with "strict": true on its function, and its parameters in the shape OpenAI’s strict function calling requires: every property listed in required, the optional ones made nullable, additionalProperties: false, and keywords strict mode does not accept (minLength, maxLength, default) removed. Honor the flag if your model supports it, ignore it otherwise. The platform checks each call against the tool’s own schema before running it either way.

When your model returns a lookup_order tool call, the platform executes it and immediately sends the next request with the call and its result appended to messages, everything else unchanged:

{
  "role": "assistant",
  "tool_calls": [
    { "id": "call_1", "type": "function", "function": { "name": "lookup_order", "arguments": "{\"order_number\":\"A12345\"}" } }
  ]
},
{ "role": "tool", "tool_call_id": "call_1", "content": "{\"status\":\"shipped\",\"eta\":\"2026-08-28\"}" }

Reply to that request with the spoken answer as ordinary delta.content chunks ending in finish_reason: "stop". Requests authenticate with Authorization: Bearer <your API key>. If your endpoint needs other headers, add them with headers. The endpoint must use https on a publicly reachable host, and must support function calling if the agent has tools configured.

Point the agent at your endpoint

To set it up in the console:

Open the agent’s Configuration and click the Model selector.
Choose + Custom LLM at the bottom of the menu.
Enter the base URL, the model name and your API key.
Click Run test. It sends one streaming turn with the agent’s prompt and tools, just like a live call.
Click Save and use. The agent now uses your endpoint.

Run test reports:

Whether the endpoint was reached.
Whether it streamed delta.content.
Whether it accepted the tools.
The time to first token.

The saved endpoint appears under Custom in the selector. You can edit or remove it there. If you pick a platform model again, the endpoint, the stored key and any headers are removed. Through the API, set the llm.custom section on the agent’s configuration:

curl --request PATCH "https://api.fish.audio/v1/agent/agents/$AGENT_ID/config" \
  --header "Authorization: Bearer $FISH_API_KEY" \
  --header "Content-Type: application/json" \
  --data '{
    "llm": {
      "custom": {
        "base_url": "https://llm.example.com/v1",
        "model": "persona-70b",
        "api_key": "sk-your-endpoint-key",
        "headers": {
          "your-header-key": "your-header-value"
        }
      }
    }
  }'

Field	Rules
`base_url`	`https` only, publicly reachable host (no localhost, private addresses, single-label names, or names under `.local`, `.internal`, `.svc` or `.arpa`), no query string or fragment. A trailing `/chat/completions` is stripped.
`model`	Sent to your endpoint verbatim as the request’s `model` field.
`api_key`	Sent to your endpoint as the `Authorization` bearer token. The field is write-only, so config and version reads return it as `null`. Omit it (or send `null`) on later patches to keep the stored key.
`headers`	Optional. Extra headers sent on every request to your endpoint. Names are case-insensitive, and values are printable ASCII. You can’t set headers the platform sends itself, such as `Authorization`, `Content-Type` or `Host`. Write-only like `api_key`: reads return `null`. Sending it replaces all stored headers, so include every header you want to keep. Omit it (or send `null`) to keep them; send `{}` to clear them.

The console doesn’t show or edit headers, so set them through the API. Saving the endpoint in the console keeps them, and Run test sends them while the base URL matches the saved one. Once published, every session of the agent generates on your endpoint, whether it starts from the web, a phone call, or a preview call in the Builder. Rotating the key or headers works the same way as any other config change. Patch the new values into the draft, then publish; sessions started after the publish use them. Send { "llm": { "custom": null } } to switch back to platform models.

Failure behavior

By default (llm.fallback at platform), a custom LLM never falls back to another model. A platform model answering with the wrong persona and no memory would be worse than a failed turn. To opt in, set llm.fallback.mode to manual and list the models to try when your endpoint fails. Platform models and other endpoints both work. Your endpoint can also be a fallback entry for another agent. See LLM Fallback.

Each request gets one retry and a 10 second response cap.
When a generation still fails, the agent speaks a brief hold line and stays on the call.
After three consecutive failed generations the agent apologizes, hangs up, and the session ends with end_reason: llm_endpoint_failure (in the sessions API, the dashboard, and the call.ended webhook’s ended_reason).

See what your endpoint returned

Every failed generation is recorded, whether or not the call ended because of it. You can find them in three places:

Conversations in the dashboard shows each failure in the transcript, where it happened.
The sessions API returns them as llm_endpoint_errors.
The call.ended webhook carries the same list.

Each failure has a type:

`type`	Meaning
`http_error`	Your endpoint answered with a non-2xx status
`timeout`	No response within 10 seconds
`connection_error`	The request got no answer, for example a DNS, TLS or refused connection
`invalid_response`	A 2xx response that was not a usable reply, such as a broken stream

For http_error, response_body holds the start of your response body, up to 300 characters. Your API key is masked. connection_error and invalid_response also carry a reason, such as dns_failed or not_event_stream. The lines the agent spoke in place of a reply have fallback_reason: "llm_error".

Voice conversations are latency sensitive:

Aim for a time-to-first-token under 800 ms. This is a recommendation, not a limit.
Reasoning models (GPT-6, o-series and similar) are often slow. For voice, turn reasoning off or use a non-reasoning model.
Run test shows the time to first token of one turn.
Conversation history shows turn latency per session. It separates endpoint time from platform time.

Limitations

Single Turn and Tool agent tests are not supported. A scripted test run refuses to execute rather than substitute a platform model for yours. Simulation tests run on your endpoint.
The prompt-level safety guardrails still ride the assembled context, but your model decides whether to honor them.

Get Started

Build

Test

Deploy

Telephony

Monitor

How it works

Example request

Point the agent at your endpoint

Failure behavior

See what your endpoint returned

Limitations

​How it works

​Example request

​Point the agent at your endpoint

​Failure behavior

​See what your endpoint returned

​Limitations

How it works

Example request

Point the agent at your endpoint

Failure behavior

See what your endpoint returned

Limitations