Custom LLM

By default your agent generates replies with a Fish Audio platform model. With a custom LLM, every reply comes from an endpoint you host instead, whether that is your own fine-tuned model, your own memory and conversation logic, or a proxy inside your VPC. Fish Audio keeps running the rest of the call, including speech recognition, voice synthesis, interruption handling, tool execution, recordings, and billing.

How it works

Your endpoint speaks the standard OpenAI chat completions protocol. The platform sends POST {base_url}/chat/completions with stream: true and reads the reply as server-sent events, so if your prototype already runs against another voice-agent platform through a custom LLM, the same server works here unchanged. Each conversation turn, your endpoint receives the same fully assembled context a platform model would see. The messages array carries the system prompt with dynamic variables and session overrides applied, the complete conversation history, and any retrieved knowledge, and the agent’s tools are included in OpenAI function format. When your model returns tool_calls, the platform executes the tool and calls you again with the result. Two extra fields ride each request body so your server can look up its own state:

Field	Content
`session_id`	The Fish Audio session id.
`user_id`	The `end_user_id` you passed when creating the session. Omitted when the session has none.

Requests authenticate with Authorization: Bearer <your API key>. The endpoint must use https on a publicly reachable host, and must support function calling if the agent has tools configured.

Point the agent at your endpoint

Set the llm.custom section on the agent’s configuration:

curl --request PATCH "https://api.fish.audio/v1/agent/agents/$AGENT_ID/config" \
  --header "Authorization: Bearer $FISH_API_KEY" \
  --header "Content-Type: application/json" \
  --data '{
    "llm": {
      "custom": {
        "base_url": "https://llm.example.com/v1",
        "model": "persona-70b",
        "api_key": "sk-your-endpoint-key"
      }
    }
  }'

Field	Rules
`base_url`	`https` only, publicly reachable host, no query string or fragment. A trailing `/chat/completions` is stripped.
`model`	Sent to your endpoint verbatim as the request’s `model` field.
`api_key`	Sent to your endpoint as the `Authorization` bearer token. The field is write-only, so config and version reads return it as `null`. Omit it (or send `null`) on later patches to keep the stored key.

Once published, every session of the agent generates on your endpoint, whether it starts from the web, a phone call, or a preview call in the Builder. Rotating the key works the same way as any other config change. Patch a new api_key into the draft, then publish; sessions started after the publish use the new key. Send { "llm": { "custom": null } } to switch back to platform models.

Failure behavior

A custom LLM never silently falls back to a platform model, because a platform model answering with the wrong persona and no memory would be worse than a failed turn.

Each request gets one retry and a 10 second response cap.
When a generation still fails, the agent speaks a brief hold line and stays on the call.
After three consecutive failed generations the agent apologizes, hangs up, and the session records ended_reason: llm_endpoint_failure. Each failure also emits an llm.endpoint_error event on the session timeline.

Voice conversations are latency sensitive, so aim for a time-to-first-token under 800 ms. Turn latency is attributed per session in conversation history, which lets you tell endpoint time from platform time.

Limitations

Agent tests are not supported. A scripted test run refuses to execute rather than substitute a platform model for yours.
Configuration is API-only for now. A console UI comes later.
The prompt-level safety guardrails still ride the assembled context, but your model decides whether to honor them.

Get Started

Build

Test

Deploy

Telephony

Monitor

How it works

Point the agent at your endpoint

Failure behavior

Limitations

​How it works

​Point the agent at your endpoint

​Failure behavior

​Limitations

How it works

Point the agent at your endpoint

Failure behavior

Limitations