# Client Tools
Source: https://docs.fish.audio/agents/build/client-tools

Let the agent trigger actions in your own app: declared on the agent, executed by your code through the SDK

Client tools run inside your application, not on Fish Audio's servers. You declare the tool on the agent (name, description, parameters) and register a handler in your app with the [Web SDK](/agents/deploy/web-sdk). When the agent decides to call the tool mid-conversation, the SDK invokes your handler and returns its result to the agent.

Use client tools for anything only your frontend can do: navigate to a page, highlight a product, open a modal, read app state, or hand off to a human.

<Note>
  Client tools need code on your side. For tools that call an HTTP endpoint from Fish Audio's side, use [webhook tools](/agents/build/webhook-tools). For built-in capabilities like hanging up, see [system tools](/agents/build/system-tools).
</Note>

## How it works

<Steps>
  <Step title="Declare the tool on the agent">
    Add a tool of type `client` to the agent's tool list, alongside any webhook tools. The name, description, and arguments tell the model what the tool does and when to call it.
  </Step>

  <Step title="Register a handler in your app">
    Pass a handler for the same tool name when starting a session with `@fishaudio/agent-client`.
  </Step>

  <Step title="The agent calls the tool">
    During the conversation, the agent invokes the tool with parameter values. The SDK dispatches the call to your handler.
  </Step>

  <Step title="Your result goes back to the agent">
    The handler's return value is JSON-serialized and sent back, and the agent continues with the result, unless the tool is fire-and-forget.
  </Step>
</Steps>

## Declare a client tool

Client tools sit alongside webhook tools in the agent's tool list. A declaration looks like this:

```json Tool definition theme={null}
{
  "tool_type": "client",
  "name": "highlight_product",
  "description": "Visually highlight a product on the page the user is viewing.",
  "arguments": [
    { "name": "product_id", "description": "ID of the product to highlight." }
  ],
  "expects_response": true
}
```

| Field | Description |
| - | - |
| `name` | How the model refers to the tool. Must match the name you register in the SDK exactly |
| `description` | Tells the model what the tool does and when to use it |
| `arguments` | Named inputs the model fills in; delivered to your handler as one JSON object |
| `expects_response` | Whether the agent waits for your handler's return value before continuing |

### Naming rules

Tool names must match `^[a-zA-Z][a-zA-Z0-9_-]{0,63}$`: start with a letter, then letters, digits, underscores, or hyphens, up to 64 characters total.

* Built-in tools live under a leading underscore (like `_transfer_call`), which the pattern makes unreachable.
* Names must be unique per agent. Declaring a duplicate fails with a `400`.

## Handle the call in your app

Register handlers with the `clientTools` option when starting a session, or later with `registerClientTool`:

<CodeGroup>
  ```javascript At session start theme={null}
  import { AgentSession } from "@fishaudio/agent-client";

  const session = await AgentSession.start({
    agentId: "YOUR_AGENT_ID",
    clientTools: {
      highlight_product: async (params) => {
        const el = document.getElementById(String(params.product_id));
        el?.scrollIntoView({ behavior: "smooth" });
        el?.classList.add("highlighted");
        return { highlighted: true };
      },
    },
  });
  ```

  ```javascript After session start theme={null}
  session.registerClientTool("open_page", (params, { callId, toolName }) => {
    window.location.assign(`/products/${String(params.product_id)}`);
  });
  ```
</CodeGroup>

A handler receives the parameter values as a single object, plus a context object with `callId` and `toolName`. It can be synchronous or async:

```typescript Handler signature theme={null}
type ClientToolHandler = (
  params: Record<string, unknown>,
  ctx: { callId: string; toolName: string },
) => unknown | Promise<unknown>;
```

<Tip>
  Parameter values are produced by the model. Validate them in your handler before acting on them, just as you would any external input.
</Tip>

### What the agent receives

| Handler outcome | Result |
| - | - |
| Returns a value | JSON-serialized and sent back to the agent |
| Throws | An error result is sent back; the SDK emits an `error` event with code `tool_failed` |
| Exceeds the timeout (default 15 s) | An error result is sent back automatically; code `tool_timeout` |
| Tool called but no handler registered | An error result is sent back automatically |
| Returns a value that cannot be sent: larger than about 60 KB serialized (`MAX_CLIENT_TOOL_RESULT_BYTES`) or not JSON-serializable | An error result explaining why is sent back instead (SDK 0.2.1 and later); code `tool_failed` |

The agent is never left hanging: while `expects_response` is `true` the agent suspends that tool call until your result arrives or the timeout fires, then continues either way. Errors and timeouts return to the model as tool errors, so the agent can recover in conversation ("I couldn't open that page. Let me try something else"). Keep results small: they travel over the realtime data channel, which caps a message at 64 KB, and the model reads the whole result on every turn. Return a summary or a reference (an id, a URL) rather than a large payload. To change the handler timeout, pass `clientToolTimeoutMs` (in milliseconds) when starting the session. Note there is a second, server-side deadline: the agent stops waiting after the tool's `timeout_seconds` (default 30 s, settable 1–120 on the tool), and a handler result arriving after that is ignored, so `clientToolTimeoutMs` can only tighten the client-side deadline, not extend the agent's wait.

## Fire-and-forget tools

Set `expects_response` to `false` for tools that are pure side effects: the agent triggers your handler and keeps talking without waiting. Any return value is ignored.

This suits UI actions where confirmation adds nothing: scrolling, opening a panel, firing an analytics event. If the agent should react to the outcome ("the page is open, now walk the user through it"), keep `expects_response: true`.

## Observe tool activity

Every tool call (client and webhook alike) emits `toolCallStarted`, `toolCallCompleted`, and `toolCallFailed` events on the session, with the call ID, tool name, and payloads. The events are on by default; sessions created with `tool_events: false` don't emit them. Use them to render tool activity in your UI. See the [Web SDK](/agents/deploy/web-sdk) for the event reference.

## Going further

<CardGroup>
  <Card title="Web SDK" icon="js" href="/agents/deploy/web-sdk">
    Full `@fishaudio/agent-client` API: sessions, events, audio.
  </Card>

  <Card title="Webhook tools" icon="webhook" href="/agents/build/webhook-tools">
    Tools that call your HTTP endpoints server-side.
  </Card>

  <Card title="System tools" icon="gears" href="/agents/build/system-tools">
    Built-in capabilities you toggle per agent.
  </Card>

  <Card title="Tools overview" icon="screwdriver-wrench" href="/agents/build/tools">
    How the agent chooses and combines tools.
  </Card>
</CardGroup>


# Configuration
Source: https://docs.fish.audio/agents/build/configuration

Set your agent's system prompt, first message, and conversation behavior, in the Builder or through the API

Configuration is the Builder's home tab: define how your agent behaves, how it opens each conversation, and how it paces its turns. Every edit is saved to the agent's **draft** automatically. Callers only hear your changes after you [publish](/agents/deploy/versions-publishing). New to Agents? Start with the [Quickstart](/agents/quickstart).

## System prompt

The system prompt sets the personality, goals, and guardrails that steer every reply. It is capped at **32,000 tokens**. The API rejects longer prompts with `422 Unprocessable Entity`. Keeping it under **2,000 tokens** is recommended: shorter prompts cut latency and cost, and hold the model's attention better. It can also be replaced for a single session by sending `overrides.system_prompt` when [creating the session](/agents/deploy/authenticated-sessions#overrides).

To personalize the prompt per session without replacing it, write `{{placeholders}}`: [custom variables](/agents/build/dynamic-variables#custom-variables) such as `{{customer_name}}` take the values you pass when creating the session, and [system variables](/agents/build/dynamic-variables#system-variables) such as `{{system.caller_number}}` or `{{system.today}}` are filled in by the platform.

<Tip>
  Keep the prompt focused:

  * State who the agent is and what it should accomplish, in a few sentences each.
  * Spell out guardrails explicitly ("If asked about pricing, direct the caller to sales").
  * Write for the ear: replies are spoken aloud, so ask for short, conversational answers.
</Tip>

## First message

Choose how the agent opens each conversation:

| Mode | Behavior | Field on the wire |
| - | - | - |
| **Off** | The agent stays silent and waits for the caller to speak first. | — |
| **Fixed message** | The agent opens with the exact text you provide, every time. | `first_message` |
| **Prompt** | The agent generates its opening line from instructions you provide. | `first_message_prompt` |

`first_message` and `first_message_prompt` can each hold up to 10,000 characters. The active mode (`first_message_mode`: `off`, `fixed`, or `prompt`) decides what callers hear. Both fields accept the same `{{placeholders}}` as the system prompt (see [Dynamic variables](/agents/build/dynamic-variables)), and both can be replaced for a single session by sending `overrides.first_message` or `overrides.first_message_prompt` when [creating the session](/agents/deploy/authenticated-sessions#overrides).

## Voice

The Voice panel selects the voice your agent speaks with (`voice_id`) and its speaking language (`speaking_language`, one of the [52 supported languages](/agents/build/voice-language#speaking-language)). It also holds expressive delivery (`expressive`) and the speech recognition settings, which the API keeps in their own `asr` section: the recognition model (`asr.model`) and the multilingual recognition switch (`asr.multilingual`). See [Voice & language](/agents/build/voice-language) for all of them.

## Conversation settings

### Turn-taking

`conversation.response_wait_ms` (300–2000, default `550`) is how long the agent waits after the caller pauses before it starts talking. In the Builder this is the **Response wait** slider, with Fast (0.45 s), Balanced (0.55 s), and Patient (0.9 s) shortcuts.

When the agent thinks the caller is mid-sentence it waits longer, up to `conversation.response_max_wait_ms` (600–4000). Leave it `null` (the default) to follow the response wait plus 0.5 s; send `0` to clear an explicit value back to automatic. A max wait below the response wait is rejected with `422`.

`conversation.eagerness` is the same setting as a preset and still works: writing `relaxed`, `balanced`, or `eager` sets the response wait to 900, 550, or 450 ms, and reads report the preset that matches the current wait.

<Note>
  A short answer such as "yes" normally lands on the fast path. When the agent asks for digits or a spelled-out name, the wait is widened automatically so the caller can pause between groups.
</Note>

`conversation.speculative_response` (default `true`) lets the agent start preparing its reply while the caller may still be speaking; the draft is discarded if they continue, and tools never run before the caller's turn is confirmed. Turn it off if your [custom LLM](/agents/build/custom-llm) bills per token and you would rather not pay for discarded drafts. With it off, the response wait is added directly to every reply.

### Interruptions

* `conversation.interruptible` (default `true`): Whether the caller can barge in while the agent is speaking.
* `conversation.interruption_sensitivity` sets how much caller speech counts as an interruption:
  * `low` (**Hard to interrupt** in the Builder): The agent stops only for a firm, worded interjection, talking through background noise and short acknowledgements.
  * `balanced` (default): Interrupts on normal speech.
  * `high` (**Easy to interrupt**): The agent stops on the caller's first word.
* `conversation.interruption_ignore_phrases` (up to 50 entries of 40 characters): Backchannels such as `"uh-huh"` or `"okay"` that never interrupt the agent when the caller says only them. Matching ignores case and punctuation. Send `[]` to clear the list.

### Call duration

`conversation.max_duration_seconds` (60–3600, default `1800`) ends the session automatically when the limit is reached.

<Note>
  [Preview calls](/agents/test/preview-calls) in the Builder are capped at 10 minutes or this setting, whichever is lower.
</Note>

### Recording

`conversation.record_audio` (default `true`) stores per-speaker audio for [playback and download](/agents/monitor/conversation-history#what-gets-stored), with a per-session override on the [session request](/agents/deploy/authenticated-sessions). In the console this lives under your agent's **Settings**. Make sure callers are informed and consent where your jurisdiction requires it.

### Timezone

`conversation.timezone` is the default IANA timezone (like `Asia/Tokyo`) the agent uses for dates and times in conversation. Leave it empty for **automatic**: each session follows the caller's device or phone number, falling back to UTC. Set one when your agent serves a single region regardless of who calls. A per-session `timezone` on the [session request](/agents/build/time-timezone) overrides this. See [Time & timezone](/agents/build/time-timezone) for the full resolution order.

## Autosave and publishing

There is no Save button. Each change is written to the agent's draft moments after you stop editing, and the **Saving… / Saved** indicator at the bottom-left of the page shows the current state. If a save fails, the Builder tells you and keeps your pending edits so nothing is lost.

Drafts never affect live traffic: active integrations keep using the last published version until you press **Publish**, which snapshots the draft as a new immutable version. The Publish button lights up whenever the draft differs from what is published. See [Versions & publishing](/agents/deploy/versions-publishing).

## Configure through the API

The same draft is readable and writable over REST, which is useful for provisioning agents from your own systems.

### Read the draft

```bash Request theme={null}
curl --request GET "https://api.fish.audio/v1/agent/agents/$AGENT_ID/config" \
  --header "Authorization: Bearer $FISH_API_KEY"
```

```json Response (abridged) theme={null}
{
  "config_hash": "sha256:5f2c1a9e…",
  "prompt": {
    "system_prompt": "You are Aria, a friendly support agent for Acme…",
    "first_message_mode": "fixed",
    "first_message": "Hi, thanks for calling Acme. How can I help?"
  },
  "voice": {
    "voice_id": "802e3bc2b27e49c2995d23ef70e6ac89",
    "speaking_language": "en"
  },
  "asr": {
    "model": "deepgram:nova-3",
    "multilingual": false,
    "strict_language": false,
    "keyterms": []
  },
  "conversation": {
    "max_duration_seconds": 1800,
    "response_wait_ms": 550,
    "response_max_wait_ms": null,
    "eagerness": "balanced",
    "interruptible": true,
    "interruption_sensitivity": "balanced",
    "interruption_ignore_phrases": [],
    "speculative_response": true,
    "timezone": ""
  }
}
```

The response also includes the `tools`, `knowledge_base`, `analysis`, and `webhooks` sections, each covered on its own page. `config_hash` identifies this draft revision; a mismatch with the published version is what marks an agent as having unpublished changes.

### Update sections

`PATCH` takes a partial body and deep-merges it into the draft: send only the sections and fields you want to change, and everything else keeps its value. Fields that hold a list (`webhooks.post_call` among them) are replaced as a whole, so send the complete list whenever you change one.

```bash theme={null}
curl --request PATCH "https://api.fish.audio/v1/agent/agents/$AGENT_ID/config" \
  --header "Authorization: Bearer $FISH_API_KEY" \
  --header "Content-Type: application/json" \
  --data '{
    "prompt": { "system_prompt": "You are Aria, a concise support agent for Acme." },
    "conversation": { "response_wait_ms": 900 }
  }'
```

The response includes the draft's new `config_hash`. Values outside the documented limits (a system prompt over 32,000 tokens, `max_duration_seconds` outside 60–3600) are rejected with `422 Unprocessable Entity` and the draft is left unchanged.

### Config sections

| Section | What it holds | Documented in |
| - | - | - |
| `prompt` | System prompt and first-message settings | This page |
| `voice` | Voice profile, speaking language, and expressive mode | [Voice & language](/agents/build/voice-language) |
| `asr` | Speech recognition model, multilingual recognition, strict language, and recognition keyterms | [Voice & language](/agents/build/voice-language#speech-recognition-model) |
| `conversation` | Call duration, turn-taking, interruption behavior, timezone, and storage | This page; storage in [Conversation history](/agents/monitor/conversation-history#what-gets-stored) |
| `tools` | Attached webhook tools and system tool switches | [Tools](/agents/build/tools) |
| `knowledge_base` | Attached knowledge sources | [Knowledge base](/agents/build/knowledge-base) |
| `analysis` | Post-call summary, data fields, and success criteria | [Post-call analysis](/agents/monitor/post-call-analysis) |
| `webhooks` | Post-call webhook delivery and the inbound call webhook | [Webhooks](/agents/monitor/webhooks) |
| `llm` | Conversation model settings, including a custom LLM endpoint | [Custom LLM](/agents/build/custom-llm) |

## Going further

<CardGroup>
  <Card title="Voice & language" icon="waveform-lines" href="/agents/build/voice-language">
    Choose a voice profile and control the speaking language.
  </Card>

  <Card title="Versions & publishing" icon="code-branch" href="/agents/deploy/versions-publishing">
    Turn the draft into an immutable live version.
  </Card>

  <Card title="Preview calls" icon="phone" href="/agents/test/preview-calls">
    Talk to your draft in the Builder before publishing.
  </Card>

  <Card title="Tools" icon="wrench" href="/agents/build/tools">
    Let the agent call your backend mid-conversation.
  </Card>

  <Card title="Dynamic variables" icon="brackets-curly" href="/agents/build/dynamic-variables">
    Personalize the prompt and first message per session.
  </Card>
</CardGroup>


# Custom LLM
Source: https://docs.fish.audio/agents/build/custom-llm

Serve your agent's replies from your own OpenAI-compatible model endpoint

By default your agent generates replies with a Fish Audio platform model. With a custom LLM, every reply comes from an endpoint you host instead, whether that is your own fine-tuned model, your own memory and conversation logic, or a proxy inside your VPC. Fish Audio keeps running the rest of the call, including speech recognition, voice synthesis, interruption handling, [tool](/agents/build/tools) execution, recordings, and billing.

## How it works

Your endpoint speaks the standard OpenAI [chat completions](https://platform.openai.com/docs/api-reference/chat) protocol. The platform sends `POST {base_url}/chat/completions` with `stream: true` and reads the reply as server-sent events, so if your prototype already runs against another voice-agent platform through a custom LLM, the same server works here unchanged.

Each conversation turn, your endpoint receives the same fully assembled context a platform model would see. The `messages` array carries the system prompt with [dynamic variables](/agents/build/dynamic-variables) and [session overrides](/agents/deploy/authenticated-sessions#overrides) applied, the complete conversation history, and any retrieved [knowledge](/agents/build/knowledge-base), and the agent's tools are included in OpenAI function format. When your model returns `tool_calls`, the platform executes the tool and calls you again with the result. A few extra top-level fields ride each request body so your server can look up its own state:

| Field | Content |
| - | - |
| `session_id` | The Fish Audio session id. |
| `user_id` | The `end_user_id` you passed when [creating the session](/agents/deploy/authenticated-sessions). Omitted when the session has none. |
| `fishaudio_extra_body` | The `llm_extra_body` object you passed when creating the session, forwarded verbatim on every request. Use it to carry your own identifiers, such as which chat or thread the user is in. Omitted when the session has none. |

<Tip>
  If your server proxies requests to an upstream provider such as OpenAI, consider removing `session_id`, `user_id`, and `fishaudio_extra_body` from the body first. Some upstream APIs reject parameters they don't recognize.
</Tip>

### Example request

Suppose the session was created from your backend with `end_user_id: "user-42"` and `llm_extra_body: {"chat_id": "chat-9"}`, the agent has one webhook tool `lookup_order`, and the user has just asked about an order. Your endpoint receives:

```http theme={null}
POST /v1/chat/completions HTTP/1.1
Host: llm.example.com
Authorization: Bearer sk-your-endpoint-key
Content-Type: application/json
```

```json theme={null}
{
  "model": "persona-70b",
  "stream": true,
  "stream_options": { "include_usage": true },
  "messages": [
    {
      "role": "system",
      "content": "Today is Wednesday, August 26, 2026. Session timezone: America/New_York (UTC-4).\n\n…\n\nYou are Aria, the support assistant for Acme Shoes. Help customers with orders and returns. …"
    },
    { "role": "assistant", "content": "Hi, this is Aria from Acme Shoes. How can I help you today?" },
    { "role": "user", "content": "I want to check on my order." },
    { "role": "assistant", "content": "Sure, what's the order number?" },
    { "role": "system", "content": "Current date and time: Wednesday, August 26, 2026, 14:07 (America/New_York)." },
    { "role": "user", "content": "It's A one two three four five." }
  ],
  "tools": [
    {
      "type": "function",
      "function": {
        "name": "lookup_order",
        "description": "Look up an order by its number.",
        "parameters": {
          "type": "object",
          "properties": {
            "order_number": { "type": "string", "description": "The order number, e.g. A12345." }
          },
          "required": ["order_number"]
        }
      }
    }
  ],
  "session_id": "sess_01j9x4k2m8v3q7n5p6r8t9w0y1",
  "user_id": "user-42",
  "fishaudio_extra_body": { "chat_id": "chat-9" }
}
```

A few things to note:

* The first `system` message is the assembled prompt: your agent's [system prompt](/agents/build/configuration#system-prompt) with [dynamic variables](/agents/build/dynamic-variables) and [overrides](/agents/deploy/authenticated-sessions#overrides) applied, plus the platform's own context lines. A short `system` line carrying the current time is inserted before the latest user turn on every request.
* `messages` carries the full transcript so far. Speech recognition output arrives as plain `user` text; your earlier replies come back as `assistant` messages.
* `tools` is present only when the agent has tools configured. The `model`, `stream`, and `stream_options` fields are fixed; no `temperature` or `max_tokens` is sent, so apply your own defaults.

When your model returns a `lookup_order` tool call, the platform executes it and immediately sends the next request with the call and its result appended to `messages`, everything else unchanged:

```json theme={null}
{
  "role": "assistant",
  "tool_calls": [
    { "id": "call_1", "type": "function", "function": { "name": "lookup_order", "arguments": "{\"order_number\":\"A12345\"}" } }
  ]
},
{ "role": "tool", "tool_call_id": "call_1", "content": "{\"status\":\"shipped\",\"eta\":\"2026-08-28\"}" }
```

Reply to that request with the spoken answer as ordinary `delta.content` chunks ending in `finish_reason: "stop"`.

Requests authenticate with `Authorization: Bearer <your API key>`. The endpoint must use `https` on a publicly reachable host, and must support function calling if the agent has tools configured.

## Point the agent at your endpoint

Set the `llm.custom` section on the agent's [configuration](/agents/build/configuration#configure-through-the-api):

```bash theme={null}
curl --request PATCH "https://api.fish.audio/v1/agent/agents/$AGENT_ID/config" \
  --header "Authorization: Bearer $FISH_API_KEY" \
  --header "Content-Type: application/json" \
  --data '{
    "llm": {
      "custom": {
        "base_url": "https://llm.example.com/v1",
        "model": "persona-70b",
        "api_key": "sk-your-endpoint-key"
      }
    }
  }'
```

| Field | Rules |
| - | - |
| `base_url` | `https` only, publicly reachable host, no query string or fragment. A trailing `/chat/completions` is stripped. |
| `model` | Sent to your endpoint verbatim as the request's `model` field. |
| `api_key` | Sent to your endpoint as the `Authorization` bearer token. The field is write-only, so config and version reads return it as `null`. Omit it (or send `null`) on later patches to keep the stored key. |

Once [published](/agents/deploy/versions-publishing), every session of the agent generates on your endpoint, whether it starts from the web, a phone call, or a [preview call](/agents/test/preview-calls) in the Builder. Rotating the key works the same way as any other config change. Patch a new `api_key` into the draft, then publish; sessions started after the publish use the new key. Send `{ "llm": { "custom": null } }` to switch back to platform models.

## Failure behavior

A custom LLM never silently falls back to a platform model, because a platform model answering with the wrong persona and no memory would be worse than a failed turn.

* Each request gets one retry and a 10 second response cap.
* When a generation still fails, the agent speaks a brief hold line and stays on the call.
* After three consecutive failed generations the agent apologizes, hangs up, and the session ends with `end_reason: llm_endpoint_failure` (in the [sessions API](/agents/monitor/conversation-history#status-and-end-reason), the dashboard, and the `call.ended` webhook's `ended_reason`). Each failure also emits an `llm.endpoint_error` event on the session timeline.

<Tip>
  Voice conversations are latency sensitive, so aim for a time-to-first-token under 800 ms. Turn latency is attributed per session in [conversation history](/agents/monitor/conversation-history), which lets you tell endpoint time from platform time.
</Tip>

## Limitations

* Single Turn and Tool [agent tests](/agents/test/agent-tests) are not supported. A scripted test run refuses to execute rather than substitute a platform model for yours. Simulation tests run on your endpoint.
* Configuration is API-only for now. A console UI comes later.
* The prompt-level safety guardrails still ride the assembled context, but your model decides whether to honor them.


# Dynamic Variables
Source: https://docs.fish.audio/agents/build/dynamic-variables

Personalize each session with template variables: your own values, and facts the platform fills in

An agent's published configuration is shared by every caller. Dynamic variables personalize it per session: write `{{name}}` placeholders in the configuration and they are substituted once, when the published configuration is assembled for the session. There are two kinds:

* **[Custom variables](#custom-variables)**: values you supply when you create the session, such as the caller's name or plan.
* **[System variables](#system-variables)**: facts about the session itself, such as the caller's phone number or the date, filled in by the platform under the reserved `system.*` names.

## Where variables render

Placeholders are substituted in the agent's configured text and in webhook tool requests. Everything else (knowledge base content, tool names and descriptions, post-call analysis prompts) is left as written.

| Where | Custom variables | System variables |
| - | - | - |
| System prompt (`prompt.system_prompt`) | Yes | Yes |
| First message, fixed text or prompt (`prompt.first_message`, `prompt.first_message_prompt`) | Yes | Yes |
| Session [overrides](/agents/deploy/authenticated-sessions#overrides) of those fields (`overrides.system_prompt`, `overrides.first_message`, `overrides.first_message_prompt`) | Yes | Yes |
| [Warm transfer briefing](/agents/telephony/transfers#customizing-the-briefing) (`conversation.transfer_destinations[].warm_briefing_instructions`) | Yes | Yes |
| [Fixed transfer announcement](/agents/telephony/transfers#the-transfer-announcement) (`conversation.transfer_destinations[].announcement_message`) | Yes | Yes |
| [Webhook tool](/agents/build/webhook-tools#argument-templating) URL, header values, and body template | No | Yes |

Custom values travel with the request that starts the session: `dynamic_variables` on [`POST /v1/agent/sessions`](/agents/deploy/authenticated-sessions), `dynamicVariables` in the SDK's `AgentSession.start()` for [public agents](/agents/deploy/public-agents), `dynamic_variables` on [`POST /v1/agent/phone-calls`](/agents/telephony/outbound-calls#request-fields) for outbound calls, and the variables panel of a [preview call](/agents/test/preview-calls) in the Builder. Inbound phone calls have no such request: values come from the [inbound call webhook](/agents/monitor/webhooks#inbound-call-webhook) when one is configured; the caller's number and the other session facts are always available as system variables.

## Custom variables

Write `{{variable_name}}` placeholders in your agent's system prompt or first message and supply values when you create the session:

```text System prompt theme={null}
You are a support agent for {{company}}. The caller's name is {{customer_name}}
and they are on the {{plan}} plan. Greet them by name.
```

Pass values as a flat object of strings, numbers, or booleans. Variable names must match `[A-Za-z][A-Za-z0-9_]*` (no hyphens or dots; the dotted `system.*` names are reserved for [system variables](#system-variables)), string values are capped at 1,000 characters, and a request can carry at most 50 variables. Violations reject session creation with `422`:

<CodeGroup>
  ```bash API (curl) theme={null}
  curl --request POST https://api.fish.audio/v1/agent/sessions \
    --header "Authorization: Bearer $FISH_API_KEY" \
    --header "Content-Type: application/json" \
    --data '{
      "agent_id": "YOUR_AGENT_ID",
      "dynamic_variables": {
        "company": "Acme",
        "customer_name": "Ada",
        "plan": "Pro"
      }
    }'
  ```

  ```javascript JavaScript SDK theme={null}
  import { AgentSession } from "@fishaudio/agent-client";

  // Public agent: the SDK creates the session and sends the variables for you.
  const session = await AgentSession.start({
    agentId: "YOUR_AGENT_ID",
    dynamicVariables: { company: "Acme", customer_name: "Ada", plan: "Pro" },
  });
  ```
</CodeGroup>

<Warning>
  Placeholders with no matching variable are **not** removed: the literal `{{variable_name}}` text stays in the prompt, visible to the model. Make sure every placeholder in your configuration has a value at session creation.
</Warning>

Variables fill placeholders in the configured text. To replace whole configuration fields for a session (the prompt itself, the opener, voice, language), use [overrides](/agents/deploy/authenticated-sessions#overrides) on the same request; `{{placeholders}}` render inside overridden text too.

### Who supplies the values

Where custom variable values come from depends on the session's [access mode](/agents/deploy/overview#who-may-start-sessions):

| Mode | Who creates the session | Who supplies the values |
| - | - | - |
| `agentId` ([public agents](/agents/deploy/public-agents)) | The SDK, directly from the browser | The browser, via `AgentSession.start()` options |
| `sessionToken` (private agents) | Your backend, with your API key | Your backend, in its `POST /v1/agent/sessions` call |

In `sessionToken` mode the SDK's `dynamicVariables` option has no effect. The session already exists by the time the token reaches the browser. Attach personalization server-side instead:

```javascript Backend (sessionToken mode) theme={null}
// Your backend endpoint, called by your own frontend
const response = await fetch("https://api.fish.audio/v1/agent/sessions", {
  method: "POST",
  headers: {
    Authorization: `Bearer ${process.env.FISH_API_KEY}`,
    "Content-Type": "application/json",
  },
  body: JSON.stringify({
    agent_id: "YOUR_AGENT_ID",
    dynamic_variables: { customer_name: user.name, plan: user.plan },
  }),
});
const sessionToken = await response.json(); // pass to AgentSession.start({ sessionToken })
```

<Tip>
  In `agentId` mode, values arrive from the end user's browser. Treat them as untrusted input, and use `sessionToken` mode when personalization must come from data only your backend knows.
</Tip>

## System variables

The platform fills a handful of `{{system.*}}` placeholders itself, on every session. You cannot supply or override them: the names you pass in `dynamic_variables` cannot contain a dot, so the two namespaces never collide.

| Variable | Value | Notes |
| - | - | - |
| `system.channel` | `phone_inbound`, `phone_outbound`, or `web_voice` | Web SDK, API, preview, and Single Turn and Tool test sessions are `web_voice`. A Simulation test uses its [channel](/agents/test/agent-tests#channel). |
| `system.timezone` | The session's resolved IANA timezone, for example `Asia/Tokyo` | `UTC` when nothing resolves. Always equals the session's `timezone` field; see [which timezone a session uses](/agents/build/time-timezone#which-timezone-a-session-uses). |
| `system.today` | Today's date in that timezone, ISO 8601 `YYYY-MM-DD` | The calendar date at session creation. There is no `system.now`; see [World context](#world-context). |
| `system.language` | The session language code, one of the [52 supported languages](/agents/build/voice-language#speaking-language) | |
| The `language` [override](/agents/deploy/authenticated-sessions#overrides) when the request carries one, otherwise the agent's [speaking language](/agents/build/voice-language#speaking-language). | | |
| `system.caller_number` | The human party's number, E.164 | Inbound: the caller (empty if withheld or not a valid E.164 number). Outbound: the number you dialed. Empty on non-phone sessions. |
| `system.caller_number_spoken` | The same number written for speech, for example `+1; 4 1 5; 5 5 5; 0 1 2 3` | Use it wherever the agent should say the number out loud; see [Reading a number out loud](#reading-a-number-out-loud). Empty whenever `system.caller_number` is empty. |
| `system.dialed_number` | Your workspace number, E.164 | Inbound: the number that was called. Outbound: the number the call was placed from. Empty on non-phone sessions. |

### Reading a number out loud

Write `{{system.caller_number}}` where the number must stay a plain E.164 string (a tool argument, a lookup, a message the customer reads) and `{{system.caller_number_spoken}}` where the agent should say it. The spoken form puts the country code first, then groups of three or four digits, with a space between digits and a semicolon between groups: `+14155550123` becomes `+1; 4 1 5; 5 5 5; 0 1 2 3`. The agent repeats it as written, and the voice reads it digit by digit with a short pause at each semicolon, in every supported language. The semicolon is deliberate: a comma or a period is spoken as a word in several languages, a semicolon is silent in all of them. A raw E.164 string is read correctly in English and Chinese, but in most other languages it comes out as one long number, and asking the agent in the prompt to "read it digit by digit" leaves the spelling to the model, which occasionally drops a digit.

### Phone numbers

The phone numbers are the session's `caller_number` and `dialed_number` fields as shown in [session history](/agents/telephony/inbound-calls#phone-sessions-in-history). Only a value that is a well-formed E.164 number reaches the agent: anything else the carrier sends as the caller id (for example an `anonymous` marker) renders as an empty string, while the history field keeps the raw value. On every session that is not a phone call (web, text, preview, agent tests) both render as an empty string, never as literal `{{system.caller_number}}` text, so one agent can serve web and phone with the same prompt.

Write the prompt so it also reads well when the number is empty:

```text System prompt theme={null}
You answer the support line {{system.dialed_number}}. The caller's number is
"{{system.caller_number}}". If it is empty, ask for a callback number; otherwise
confirm the last four digits before discussing the account.
```

On an inbound call from +1 415 555 0123 to your number +1 408 555 0199 the model sees:

```text Rendered theme={null}
You answer the support line +14085550199. The caller's number is
"+14155550123". If it is empty, ask for a callback number; otherwise
confirm the last four digits before discussing the account.
```

On a web session the same prompt renders `The caller's number is "".`, and the agent asks for a number. To look the caller up in your own systems, template the number into a [webhook tool](#in-webhook-tools) instead of relying on the model to repeat it.

### Date and timezone

`system.today` is always ISO formatted, whatever the session language: in the system prompt the model reads it and speaks it in the session's language, while a fixed first message containing `{{system.today}}` is spoken verbatim as `2026-08-21`. A session that runs past midnight keeps its opening date. Template it, and `system.timezone`, when a request needs them explicitly, for example a webhook tool's query string; for the agent's own sense of time no variable is needed, see [World context](#world-context).

### In webhook tools

System variables render wherever custom variables do, and also inside [webhook tool](/agents/build/webhook-tools#argument-templating) requests: the URL, header values, and body template. Custom variables do not render into tools. In the URL the value is URL-encoded, like argument values: the leading `+` of an E.164 number travels as `%2B` and the `/` in a timezone name as `%2F`, both decoding back on your server.

```text Webhook tool URL theme={null}
https://crm.example.com/availability?phone={{system.caller_number}}&from={{system.today}}&tz={{system.timezone}}
```

### Unknown names

A reference to a system variable that does not exist (`{{system.foo}}`) is rejected with `422` when you save the configuration or the tool, and when you send it in session overrides. The error lists the available names.

<Note>
  There is no `system.session_id`. The configuration is rendered before the session receives its id, so the id cannot be templated into it. Read it from the `POST /v1/agent/sessions` [response](/agents/deploy/authenticated-sessions), the SDK's `connect` event, or the `session` object in [webhooks](/agents/monitor/webhooks).
</Note>

## World context

The agent knows the current date and time without any variable. The platform states the date and the session's timezone at the start of every session and refreshes the time on every turn, so "tomorrow morning" or "next Tuesday" resolve correctly even in a long conversation. You don't need a custom `{{today}}` or `{{now}}`, and there is no `system.now`: variables render once, when the session is created, so a templated time would be stale from the first reply onward. When you want the date or timezone as text in a template, for example in a webhook tool URL, use the [`system.today` and `system.timezone`](#system-variables) system variables.

See [Time & timezone](/agents/build/time-timezone) for how the timezone is resolved and how to turn the injection off for a session (`world_context: false`).

## Going further

<CardGroup>
  <Card title="Agent configuration" icon="sliders" href="/agents/build/configuration">
    The fields your placeholders act on.
  </Card>

  <Card title="Overrides" icon="server" href="/agents/deploy/authenticated-sessions#overrides">
    Replace whole configuration fields for one session.
  </Card>

  <Card title="Versions & publishing" icon="code-branch" href="/agents/deploy/versions-publishing">
    Sessions assemble the published configuration.
  </Card>

  <Card title="Web SDK" icon="js" href="/agents/deploy/web-sdk">
    Every `AgentSession.start()` option.
  </Card>
</CardGroup>


# Knowledge Base
Source: https://docs.fish.audio/agents/build/knowledge-base

Ground your agent in your own documents: add a source once, attach it to any agent in your workspace

The knowledge base is a workspace-wide document library. Add a source once (an uploaded Markdown or plain-text file) and attach it to any number of agents. In conversation, the agent draws on attached sources to answer questions its [instructions](/agents/build/configuration) alone can't cover: product details, policies, FAQs, procedures.

## Add sources

Sources come in two scopes:

* **Workspace sources** live in the shared library. Add them from the workspace-level **Knowledge base** page with **Add file**: choose a `.md` or `.txt` file (UTF-8 text, up to 1 MB) and give the source a name and an optional description. Workspace sources can be attached to any number of agents.
* **Agent-private sources** belong to a single agent. On your agent's **Knowledge base** tab, **New private file** creates a source that is attached to that agent immediately. It doesn't appear in the shared library, can't be attached to other agents, and is not listed by the [knowledge-sources API](#manage-sources-over-the-api).

To attach a shared source to an agent, open the agent's **Knowledge base** tab and click **Import from global knowledge base**.

Every source is plain text and stays editable after upload: open a source to view its content and edit the text in place.

<Note>
  PDF, Office, and scanned documents are not supported. Convert them to Markdown or plain text first.
</Note>

### Upload validation

Files are validated, parsed, and indexed as part of the upload itself. When the upload succeeds, the source is immediately available to attached agents. If the file is empty, over 1 MB, or not UTF-8 text, the upload is rejected with an error and nothing is created.

### Folders

Group sources into folders with **New folder**, and rename sources from the row menu. Deleting a folder never deletes its sources. Anything inside moves back to **All sources**.

## Share sources across agents

Workspace sources belong to the workspace, not to a single agent (agent-private files offer only **Edit** and **Delete**). In the library, the **Used by** column shows how many agents attach each source, and **Manage access** lists every agent with a toggle:

* **Attach**: Turn an agent on to give it access to the source.
* **Detach**: Turn an agent off. The source stays in the library and remains attached to other agents.

Editing a source propagates to every agent that attaches it. Each attached agent shows the edit as an unpublished draft change.

<Warning>
  **Delete** removes a source from *all* agents that reference it, not just the one you're editing. To remove a source from a single agent, detach it with **Manage access** instead.
</Warning>

## How your agent uses knowledge

### Small corpora are inlined

If the total size of an agent's attached sources is **8 KB or less**, the full content is inlined into the agent's instructions. This is the fastest path: the agent has everything up front, with zero per-turn lookup cost.

<Note>
  Inlined knowledge produces no retrieval events. If your corpus is small and you don't see retrieval activity, nothing is wrong.
</Note>

### Larger corpora are searched per turn

Above 8 KB, the agent searches attached sources on every turn, using the user's latest message as the query, and injects the best-matching passages into that turn's context. The search runs under a fixed time budget: if it can't complete in time, the turn is answered without retrieved context. Retrieval never delays the agent's reply.

### Publishing pins revisions

When you [publish](/agents/deploy/versions-publishing) an agent, the published version pins the then-current revision of every attached source. Sessions read the pinned revisions, so edits you make mid-call never leak into an in-progress conversation. Publish again to roll updated content out.

## Limits

| Limit | Value |
| - | - |
| File upload formats | `.md`, `.txt` |
| File size | 1 MB per file |
| Sources per agent | 100 |
| Inline threshold | 8 KB total attached content |

Attaching more than 100 sources to one agent is rejected with `422`.

## Manage sources over the API

Knowledge sources are exposed at `/v1/agent/knowledge-sources`. Only workspace sources appear on this surface. Agent-private files are a console-only convenience:

| Endpoint | Description |
| - | - |
| `GET /v1/agent/knowledge-sources` | List sources in your workspace |
| `POST /v1/agent/knowledge-sources` | Create a source: multipart upload with the `.md`/`.txt` file in the `source` field |
| `GET /v1/agent/knowledge-sources/{source_id}` | Retrieve a source |
| `PATCH /v1/agent/knowledge-sources/{source_id}` | Rename a source or replace its content. Multipart like create; omitted fields keep their values, and a new `source` file bumps `revision_number` |
| `DELETE /v1/agent/knowledge-sources/{source_id}` | Delete a source |
| `GET /v1/agent/knowledge-sources/{source_id}/agents` | List agents that attach the source |

```bash Create a source theme={null}
curl --request POST https://api.fish.audio/v1/agent/knowledge-sources \
  --header "Authorization: Bearer $FISH_API_KEY" \
  --form "source=@faq.md;type=text/markdown" \
  --form "name=Customer FAQ" \
  --form "description=Answers for the support agent"
```

```bash Replace content theme={null}
curl --request PATCH https://api.fish.audio/v1/agent/knowledge-sources/{source_id} \
  --header "Authorization: Bearer $FISH_API_KEY" \
  --form "source=@faq-v2.md;type=text/markdown"
```

```bash Check dependents theme={null}
curl https://api.fish.audio/v1/agent/knowledge-sources/{source_id}/agents \
  --header "Authorization: Bearer $FISH_API_KEY"
```

Only the `source` file part is required on create: `name` falls back to the uploaded file's name.

### Attach sources to an agent

Attach sources through the agent's configuration. `knowledge_source_ids` replaces the whole attached list. Attaching sources also turns the knowledge base on, unless you send `enabled: false` with them to attach now and switch on later. Publish for the change to reach live sessions.

```bash Attach sources theme={null}
curl --request PATCH https://api.fish.audio/v1/agent/agents/YOUR_AGENT_ID/config \
  --header "Authorization: Bearer $FISH_API_KEY" \
  --header "Content-Type: application/json" \
  --data '{
    "knowledge_base": {
      "knowledge_source_ids": ["SOURCE_ID_1", "SOURCE_ID_2"]
    }
  }'
```

| Field | Rules |
| - | - |
| `knowledge_source_ids` | Workspace source ids, up to 100. Replaces the whole attached list; an unknown id is rejected with `422`. Send `[]` to detach everything |
| `enabled` | Whether the agent uses the attached sources. Defaults to `true` when omitted together with a non-empty `knowledge_source_ids`; send `false` to attach sources without using them yet |

<Note>
  `DELETE` returns `409` while the source is still attached to any agent, in its draft or its published version. Call the dependents endpoint to see which agents use it, then detach it from each (remove its id from `knowledge_base.knowledge_source_ids` in an agent update) and republish any agent whose published version still references it. Deleting from the console instead removes the source from all agents at once, after a confirmation.
</Note>

## Going further

<CardGroup>
  <Card title="Agent configuration" icon="sliders" href="/agents/build/configuration">
    Instructions, voice, and everything else your agent is made of.
  </Card>

  <Card title="Versions & publishing" icon="code-branch" href="/agents/deploy/versions-publishing">
    How publishing snapshots your agent, knowledge revisions included.
  </Card>

  <Card title="Preview calls" icon="phone" href="/agents/test/preview-calls">
    Talk to your draft and check knowledge answers before publishing.
  </Card>

  <Card title="Conversation history" icon="clock-rotate-left" href="/agents/monitor/conversation-history">
    Review transcripts of past sessions.
  </Card>
</CardGroup>


# System Tools
Source: https://docs.fish.audio/agents/build/system-tools

Built-in agent capabilities like hanging up the call, enabled per agent

System tools are capabilities built into the platform. There is nothing to create or host. Each agent simply enables the ones it needs.

| Tool | Key | What it does |
| - | - | - |
| Hang up call | `hang_up_call` | Lets the agent end the call itself. |

<Note>
  Unlike [webhook tools](/agents/build/webhook-tools), system tools are not workspace resources shared across agents. The on/off state is stored per agent.
</Note>

## Toggle in the Builder

Open your agent's **Tools** page in the Builder. The **System tools** card holds the **Hang up call** switch.

## Toggle via the API

Over the REST API, system tool switches are part of the agent's configuration, in the same `tools` section that holds its attached webhook tools. See [Configuration](/agents/build/configuration) for reading and updating an agent's config.

## Transfer call

`transfer_call` has no switch of its own: it is enabled automatically once the agent has a transfer destination configured, and only on phone sessions (web sessions have no call to transfer). See [Transfers](/agents/telephony/transfers) for configuring destinations and everything else about transfers.

## Going further

<CardGroup>
  <Card title="Tools overview" icon="wrench" href="/agents/build/tools">
    How webhook, client, and system tools fit together.
  </Card>

  <Card title="Webhook tools" icon="webhook" href="/agents/build/webhook-tools">
    Call your own HTTP endpoints from a conversation.
  </Card>

  <Card title="Client tools" icon="browser" href="/agents/build/client-tools">
    Run tool calls inside your own app via the SDK.
  </Card>

  <Card title="Voice & language" icon="language" href="/agents/build/voice-language">
    Configure the agent's voice and speaking language.
  </Card>
</CardGroup>


# Time & Timezone
Source: https://docs.fish.audio/agents/build/time-timezone

Your agent knows the current date and time by default: how the timezone is chosen and how to opt out

Agents are time-aware out of the box: every session knows today's date and the current time in the session's timezone. You don't reference a variable or add anything to your prompt: "tomorrow morning" and "next Tuesday" ground correctly from the first turn, on every channel (voice, text, and phone).

<Note>
  The default timezone is **UTC**. Change it for the agent in its
  [configuration](/agents/build/configuration#timezone), and override it for a
  single session with `timezone` in the creation request. See
  [Which timezone a session uses](#which-timezone-a-session-uses).
</Note>

## What the agent knows

Fish Audio injects two pieces of world context server-side:

| When | What | Example |
| - | - | - |
| At session start | Today's date and the session timezone | `Today is Thursday, July 23, 2026. Session timezone: Asia/Tokyo (UTC+9).` |
| Before every reply | The current time, minute precision | `Current date and time: Thursday, July 23, 2026 at 13:00 (Asia/Tokyo).` |

The time is refreshed on every turn, so it stays accurate through long conversations and past midnight.

<Note>
  Injection happens outside your configuration, so it never counts toward the
  system prompt's character limit.
</Note>

Both facts also exist as [system variables](/agents/build/dynamic-variables#system-variables) for the cases where you want to template them yourself, for example into a webhook tool's URL: `{{system.today}}` is the date at session creation as ISO `YYYY-MM-DD`, and `{{system.timezone}}` is the resolved IANA name (`UTC` when nothing resolves). There is no `{{system.now}}`: variables render once, when the session is created, so a templated clock would freeze at the opening. The per-turn time above is the one to rely on.

## Which timezone a session uses

The timezone is resolved once, when the session is created, taking the first that applies:

1. **`timezone`** in the creation request: your explicit per-session choice, as an IANA name like `Asia/Tokyo`. An invalid name rejects the request with `422`.
2. **The agent's configured timezone**: set on the agent in the Builder (**Configuration → Timezone**) or via the [agent config API](/agents/build/configuration). Pin one when your agent serves a single region regardless of who calls.
3. **`client_timezone`**: a hint with the end user's browser timezone, sent automatically by the [Web SDK](/agents/deploy/web-sdk) in public-agent mode. Used only when neither of the above is set; an invalid hint is ignored rather than failing the session.
4. **The caller's phone number**: inbound calls infer the timezone from the caller's country when that country has a single timezone.
5. **UTC** otherwise.

In the browser, this means zero configuration: the SDK detects the visitor's real timezone and the agent talks about "today" in the user's local terms, not yours.

With [authenticated sessions](/agents/deploy/authenticated-sessions), the SDK's automatic hint doesn't apply. Your backend creates the session, so set `timezone` (or forward the browser's value as `client_timezone`) in the creation request:

```bash API (curl) theme={null}
curl --request POST https://api.fish.audio/v1/agent/sessions \
  --header "Authorization: Bearer $FISH_API_KEY" \
  --header "Content-Type: application/json" \
  --data '{
    "agent_id": "YOUR_AGENT_ID",
    "timezone": "Asia/Tokyo"
  }'
```

<Note>
  Phone-number inference is country-level only. Countries that span several
  timezones (the US, Canada, Australia, Russia, Brazil) are skipped. Set the
  agent's timezone for those. Calls that resolve nothing run in UTC.
</Note>

## Turn it off

Time awareness is on by default. To withhold both the date and the per-turn time from a session (for example a role-play agent set on a fictional date, or a test that must be reproducible), set `world_context: false` when creating the session:

<CodeGroup>
  ```bash API (curl) theme={null}
  curl --request POST https://api.fish.audio/v1/agent/sessions \
    --header "Authorization: Bearer $FISH_API_KEY" \
    --header "Content-Type: application/json" \
    --data '{
      "agent_id": "YOUR_AGENT_ID",
      "world_context": false
    }'
  ```

  ```javascript JavaScript SDK theme={null}
  import { AgentSession } from "@fishaudio/agent-client";

  // Public agent: the SDK forwards the opt-out in its creation request.
  const session = await AgentSession.start({
    agentId: "YOUR_AGENT_ID",
    worldContext: false,
  });
  ```
</CodeGroup>

## Going further

<CardGroup>
  <Card title="Authenticated sessions" icon="server" href="/agents/deploy/authenticated-sessions">
    All session-creation parameters, including `timezone` and `world_context`.
  </Card>

  <Card title="Dynamic variables" icon="brackets-curly" href="/agents/build/dynamic-variables">
    Personalize the rest of the prompt at session creation.
  </Card>
</CardGroup>


# Tools
Source: https://docs.fish.audio/agents/build/tools

Give your agent the ability to act during a conversation: call your backend, trigger actions in your app, or use built-in capabilities

Out of the box, an agent can only talk. Tools let it act: fetch an order status mid-call, open a page in your app, or hang up when the conversation is over. You describe what a tool does and what arguments it takes; the agent decides when to call it.

<CardGroup>
  <Card title="Webhook tools" icon="globe" href="/agents/build/webhook-tools">
    The platform calls your HTTP endpoint and speaks the result.
  </Card>

  <Card title="Client tools" icon="code" href="/agents/build/client-tools">
    Your application executes the tool through the SDK.
  </Card>

  <Card title="System tools" icon="toggle-on" href="/agents/build/system-tools">
    Built-in capabilities you switch on per agent.
  </Card>
</CardGroup>

## Choose a tool type

| Type | Executed by | Use it for |
| - | - | - |
| [Webhook](/agents/build/webhook-tools) | Fish Audio: we call your HTTP endpoint during the conversation | Order lookups, CRM reads, bookings, and anything else your backend can answer |
| [Client](/agents/build/client-tools) | Your app: the SDK hands the call to code you register | Navigation, UI updates, device actions, and anything else only the client can do |
| [System](/agents/build/system-tools) | The platform: no code involved | Hanging up the call |

Webhook and client tools are **custom tools**: you define a name, a description, and arguments, and the agent fills in the argument values when it calls. System tools are ready-made. Enable them with a switch on each agent.

## Where tools live

Custom tools are **workspace resources**, not part of a single agent. Define a `lookup_order` tool once and attach it to your support, sales, and after-hours agents. You manage them in two places:

* **Builder → Tools** shows the tools attached to the agent you are editing, plus that agent's system-tool switches. Tools you create here are attached to the agent immediately.
* **The workspace tool library** lists every custom tool with how many agents use it. Tools you create here start unattached. Grant agents access afterwards.

<Note>
  Editing a shared tool updates the draft of **every** agent that uses it. Those agents show unpublished changes until they are published again. See [Versions & publishing](/agents/deploy/versions-publishing).
</Note>

## Detach vs. delete

* **Detach** is agent-scoped. Removing a tool from an agent in the Builder leaves the tool (and its other attachments) untouched.
* **Delete** is workspace-wide. Deleting a tool from the library in the console detaches it from every agent that uses it, and those agents show unpublished changes.

Over the API, `DELETE` is stricter: it returns `409` while the tool is still attached to any agent. List the tool's dependent agents first and detach it from each one.

## Manage tools over the API

Every endpoint requires your API key. See the [API introduction](/api-reference/introduction).

| Endpoint | Description |
| - | - |
| `GET /v1/agent/tools` | List the custom tools in your workspace |
| `POST /v1/agent/tools` | Create a tool: starts unattached |
| `GET /v1/agent/tools/{tool_id}` | Retrieve a tool's configuration |
| `PATCH /v1/agent/tools/{tool_id}` | Update a tool: every field is optional |
| `DELETE /v1/agent/tools/{tool_id}` | Delete a tool: returns `409` while any agent still uses it |
| `GET /v1/agent/tools/{tool_id}/agents` | List the agents that use this tool |

Create a webhook tool:

```bash theme={null}
curl --request POST https://api.fish.audio/v1/agent/tools \
  --header "Authorization: Bearer $FISH_API_KEY" \
  --header "Content-Type: application/json" \
  --data '{
    "tool_type": "webhook",
    "name": "lookup_order",
    "description": "Look up the status of a customer order.",
    "arguments": [
      { "name": "order_number", "description": "The order number the caller provides." }
    ],
    "method": "GET",
    "url": "https://api.example.com/orders/{{order_number}}",
    "timeout_seconds": 10
  }'
```

Arguments can be referenced anywhere in the request with `{{name}}` placeholders. Before deleting a tool, check which agents depend on it:

```bash theme={null}
curl https://api.fish.audio/v1/agent/tools/{tool_id}/agents \
  --header "Authorization: Bearer $FISH_API_KEY"
```

The response lists each dependent agent's `agent_id` and `name`, plus a
`total` count.

Field-level configuration (custom headers, request body templates, timeouts, error handling, and mock responses) is covered in [Webhook tools](/agents/build/webhook-tools).

## Going further

<CardGroup>
  <Card title="Test it live" icon="phone" href="/agents/test/preview-calls">
    Call your draft agent before publishing.
  </Card>

  <Card title="Versions & publishing" icon="code-branch" href="/agents/deploy/versions-publishing">
    Drafts, published versions, and what "unpublished changes" means.
  </Card>

  <Card title="Web SDK" icon="js" href="/agents/deploy/web-sdk">
    Register client tool handlers in your application.
  </Card>

  <Card title="API introduction" icon="key" href="/api-reference/introduction">
    API keys for the endpoints above.
  </Card>
</CardGroup>


# Voice & Language
Source: https://docs.fish.audio/agents/build/voice-language

Choose the voice your agent speaks with and the language it converses in

Your agent speaks with a voice model from the Fish Audio Voice Library: the same voices you use for text to speech. Pick one in the Builder, or set it through the API, and choose the language the agent holds conversations in.

<CardGroup>
  <Card title="Configuration page" icon="sliders" href="/agents/build/configuration">
    Where the voice and language settings live in the Builder.
  </Card>

  <Card title="Voice Library" icon="book-open" href="/features/manage-voices">
    Browse public voices and manage your own.
  </Card>

  <Card title="Voice Cloning" icon="clone">
    Create a custom voice, then use it here. See [Voice Cloning](/features/voice-cloning).
  </Card>
</CardGroup>

## Pick a voice in the Builder

The voice picker on the **Configuration** page has two levels: a curated list for a fast start, and the full Voice Library when you want something specific.

<Steps>
  <Step title="Open the voice selector">
    In your agent's **Configuration** page, open the voice card. The **Choose a voice** view shows a curated selection of voices.
  </Step>

  <Step title="Browse the full library (optional)">
    Not seeing the right fit? Select **More voices** to open the **Select Voice** browser: the full Voice Library, with your own cloned voices under **My Voices**.
  </Step>

  <Step title="Save automatically">
    Your selection is written to the agent's draft configuration as soon as you pick it. Start a [preview call](/agents/test/preview-calls) to hear the voice in a real conversation.
  </Step>

  <Step title="Publish">
    Draft changes don't affect live sessions until you **Publish**. See [Versions & publishing](/agents/deploy/versions-publishing).
  </Step>
</Steps>

## Use any voice model

The agent's voice is a **voice model id** (`voice_id`): the same ids used as `reference_id` in [Text to Speech](/features/text-to-speech). Any public voice model from the Voice Library works, including:

* **Library voices**: Ready-made public voices. Find ids in the [Voice Library](/features/manage-voices).
* **Your cloned voices**: [Clone a voice](/features/voice-cloning) once, then use its model id as your agent's voice.

## Set the voice via API

Voice settings live in the `voice` section of the agent's configuration. Patches are partial: only the fields you send change, and the result is saved to the draft:

<CodeGroup>
  ```bash API (curl) theme={null}
  curl --request PATCH "https://api.fish.audio/v1/agent/agents/$AGENT_ID/config" \
    --header "Authorization: Bearer $FISH_API_KEY" \
    --header "Content-Type: application/json" \
    --data '{
      "voice": {
        "voice_id": "802e3bc2b27e49c2995d23ef70e6ac89"
      }
    }'
  ```
</CodeGroup>

`voice.speaking_language` lives in the same section and is patched the same way. As in the Builder, API edits land in the draft. Publish to roll them out.

## Speaking language

**Speaking language** sets the language for the agent's conversations (`voice.speaking_language` on the wire). Every session converses in this language. Any of the 52 languages below can be chosen.

<Accordion title="All 52 supported languages and their codes">
  | Code | Language | Code | Language |
  | - | - | - | - |
  | `en` | English | `da` | Danish |
  | `ja` | Japanese | `no` | Norwegian |
  | `zh` | Chinese | `fi` | Finnish |
  | `ko` | Korean | `et` | Estonian |
  | `es` | Spanish | `lv` | Latvian |
  | `fr` | French | `lt` | Lithuanian |
  | `de` | German | `ca` | Catalan |
  | `pt` | Portuguese | `he` | Hebrew |
  | `it` | Italian | `fa` | Persian |
  | `nl` | Dutch | `ur` | Urdu |
  | `ar` | Arabic | `kk` | Kazakh |
  | `hi` | Hindi | `ka` | Georgian |
  | `id` | Indonesian | `hy` | Armenian |
  | `tr` | Turkish | `bn` | Bengali |
  | `ru` | Russian | `ta` | Tamil |
  | `bg` | Bulgarian | `te` | Telugu |
  | `sr` | Serbian | `kn` | Kannada |
  | `hr` | Croatian | `mr` | Marathi |
  | `cs` | Czech | `gu` | Gujarati |
  | `sk` | Slovak | `pa` | Punjabi |
  | `pl` | Polish | `ne` | Nepali |
  | `uk` | Ukrainian | `th` | Thai |
  | `ro` | Romanian | `vi` | Vietnamese |
  | `hu` | Hungarian | `ms` | Malay |
  | `el` | Greek | `tl` | Tagalog |
  | `sv` | Swedish | `af` | Afrikaans |
</Accordion>

<Note>
  The voice model and the speaking language are independent settings: picking a voice does not change the language, and vice versa. Choose a voice that sounds natural in the language you configure.
</Note>

Both settings can also be replaced for a single session: send `overrides.voice_id` or `overrides.language` on the session request. See [Overrides](/agents/deploy/authenticated-sessions#overrides).

## Speech recognition model

**Speech recognition** picks the model that transcribes what callers say (`asr.model` on the wire). Latency is the typical wait from the caller's last word to the transcript. The agent starts its reply after that.

| Value | Model | Latency | Notes |
| - | - | - | - |
| `deepgram:nova-3` | Deepgram Nova-3 | ≈0.4 s | Default. Mid-call language switching covers English, Spanish, French, German, Hindi, Russian, Portuguese, Japanese, Italian, and Dutch. |
| `elevenlabs:scribe_v2_realtime` | ElevenLabs Scribe v2 Realtime | ≈0.9 s | Keyterms longer than 20 characters are ignored. |
| `elevenlabs:scribe_v2_medical` | ElevenLabs Scribe v2 Medical | ≈1.5 s | Tuned for clinical speech: medication names, anatomy, and pathology terms. Keyterms longer than 50 characters are ignored. |

Choose it with the **Speech recognition** selector in the Builder's voice section, or via the API:

<CodeGroup>
  ```bash API (curl) theme={null}
  curl --request PATCH "https://api.fish.audio/v1/agent/agents/$AGENT_ID/config" \
    --header "Authorization: Bearer $FISH_API_KEY" \
    --header "Content-Type: application/json" \
    --data '{
      "asr": {
        "model": "elevenlabs:scribe_v2_realtime"
      }
    }'
  ```
</CodeGroup>

Speech recognition applies to spoken sessions (voice calls and phone). As with every voice setting, the change lands in the draft: publish to roll it out.

## Multilingual recognition

**Multilingual recognition** lets the agent understand callers who switch to another language partway through a conversation. With it off, speech recognition listens only for the speaking language. That is more accurate when every caller speaks the same language, and it keeps short or accented phrases from being transcribed as another language.

It is off by default. Toggle it with the **Multilingual recognition** switch in the Builder's voice section, or via the API (`asr.multilingual`, default `false`):

<CodeGroup>
  ```bash API (curl) theme={null}
  curl --request PATCH "https://api.fish.audio/v1/agent/agents/$AGENT_ID/config" \
    --header "Authorization: Bearer $FISH_API_KEY" \
    --header "Content-Type: application/json" \
    --data '{
      "asr": {
        "multilingual": true
      }
    }'
  ```
</CodeGroup>

<Note>
  Which languages a caller can switch between depends on the [speech recognition model](#speech-recognition-model). With Deepgram Nova-3, a speaking language outside its switching set stays locked even with the setting on. With ElevenLabs Scribe v2, turning the setting off steers recognition toward the speaking language rather than locking it. To lock it, use [strict language](#strict-language).
</Note>

## Strict language

**Strict language** keeps the agent from understanding any language other than its speaking language. When speech recognition detects that the caller spoke another language, the agent receives `[unintelligible speech]` instead of the transcript and answers as if it did not catch what was said. It applies whether multilingual recognition is on or off.

It is off by default. Turn it on via the API (`asr.strict_language`, default `false`):

<CodeGroup>
  ```bash API (curl) theme={null}
  curl --request PATCH "https://api.fish.audio/v1/agent/agents/$AGENT_ID/config" \
    --header "Authorization: Bearer $FISH_API_KEY" \
    --header "Content-Type: application/json" \
    --data '{
      "asr": {
        "strict_language": true
      }
    }'
  ```
</CodeGroup>

<Note>
  With the ElevenLabs Scribe v2 models, speech in another language reaches the agent as `[unintelligible speech]`. With Deepgram Nova-3, recognition runs a model for the speaking language only, so speech in another language is usually not transcribed at all and the agent keeps listening.
</Note>

<Warning>
  Language is detected per utterance, so a short or heavily accented phrase in the speaking language can occasionally be detected as another language and replaced with `[unintelligible speech]` too. Only turn on strict language when you explicitly need to stop the agent from understanding other languages.
</Warning>

## Speaking speed

**Speaking speed** sets how fast the agent talks, as a multiplier from `0.5` (half speed) to `2.0` (double speed). The default is `1.0`. Many English-language agents sound more natural at a slightly faster rate, such as `1.2`.

Set it with the **Speaking speed** slider in the Builder's voice section, or via the API (`voice.speed`, default `1.0`):

<CodeGroup>
  ```bash API (curl) theme={null}
  curl --request PATCH "https://api.fish.audio/v1/agent/agents/$AGENT_ID/config" \
    --header "Authorization: Bearer $FISH_API_KEY" \
    --header "Content-Type: application/json" \
    --data '{
      "voice": {
        "speed": 1.2
      }
    }'
  ```
</CodeGroup>

The rate applies to everything the agent speaks in voice and phone sessions. As with every voice setting, the change lands in the draft: publish to roll it out.

## Expressive mode

**Expressive mode** makes the agent steer its own delivery: it opens sentences with emotion cues, adds natural pauses and emphasis, laughs where it genuinely fits, and speaks the way people talk, with contractions and the occasional "um". You get lively, emotionally aware speech without writing any delivery rules into your prompt.

It is on by default. Toggle it with the **Expressive mode** switch in the Builder's voice section, or via the API (`voice.expressive`, default `true`):

<CodeGroup>
  ```bash API (curl) theme={null}
  curl --request PATCH "https://api.fish.audio/v1/agent/agents/$AGENT_ID/config" \
    --header "Authorization: Bearer $FISH_API_KEY" \
    --header "Content-Type: application/json" \
    --data '{
      "voice": {
        "expressive": true
      }
    }'
  ```
</CodeGroup>

The delivery cues are rendered by the voice model. They are never spoken and never appear in transcripts or message history. Expressive mode applies to spoken sessions (voice calls and phone); text chat is unaffected.

<Tip>
  Keep your system prompt about **what** the agent says: who it is, what it knows, what it should do. With expressive mode on, **how** it sounds is handled for you.
</Tip>

## Going further

<CardGroup>
  <Card title="Agent configuration" icon="sliders" href="/agents/build/configuration">
    System prompt, first message, and conversation settings.
  </Card>

  <Card title="System tools" icon="wrench" href="/agents/build/system-tools">
    Built-in capabilities like hanging up the call.
  </Card>

  <Card title="Preview calls" icon="phone" href="/agents/test/preview-calls">
    Talk to your draft agent and hear the voice live.
  </Card>

  <Card title="Versions & publishing" icon="rocket" href="/agents/deploy/versions-publishing">
    How drafts become the live agent.
  </Card>
</CardGroup>


# Webhook Tools
Source: https://docs.fish.audio/agents/build/webhook-tools

Let your agent call your HTTP endpoints in the middle of a conversation

A webhook tool lets your agent call an HTTP endpoint mid-conversation (look up an order, create a ticket, check availability) and use the response in its next reply. You declare the arguments; the agent fills them in from the conversation and the platform makes the request.

<Note>
  Tools are workspace-level resources shared across agents. Removing a tool from an agent **detaches** it: the tool stays in your workspace and remains attached to other agents. Editing a tool updates every agent that uses it, and those agents show unpublished changes until you [publish](/agents/deploy/versions-publishing).
</Note>

## Create a webhook tool

In the Builder, open **Tools** and choose **Add tool → Webhook**. A tool created from the Builder is attached to the current agent immediately; use the tool's **Access** tab to enable it for other agents.

<Steps>
  <Step title="Name and describe the tool">
    The model decides when to call a tool based on its name and description. Write the description for the model: say what the tool does and when to use it, e.g. "Look up the status of an order. Use when the caller asks where their order is."
  </Step>

  <Step title="Declare arguments">
    Each argument is a name plus a description. Arguments are exposed to the agent as tool parameters. The description tells the model what value to extract from the conversation.
  </Step>

  <Step title="Configure the request">
    Pick a method (`GET`, `POST`, `PUT`, `PATCH`, or `DELETE`; default `POST`), the endpoint URL, a content type (default `application/json`), and an optional body template.
  </Step>

  <Step title="Add headers">
    Attach authentication or any custom headers. They are sent with every webhook request.
  </Step>

  <Step title="Choose an execution mode">
    Decide whether the agent waits for the response, fires the request and moves on, or keeps talking while the request runs in the background. See [Execution modes](#execution-modes).
  </Step>

  <Step title="Set response handling">
    Choose a timeout (1–120 seconds, or up to 300 for background tools; default 30) and how errors are surfaced to the agent.
  </Step>
</Steps>

## Argument templating

Reference any declared argument with `{{name}}`: in the URL, in header values, or in the body template. The platform substitutes the values the agent supplies before sending the request. [System variables](/agents/build/dynamic-variables#system-variables) such as `{{system.caller_number}}` or `{{system.today}}` are available in the same places; they are filled when the session starts, before the agent supplies any argument.

```text Endpoint URL theme={null}
https://api.example.com/orders/{{order_number}}
```

```json Body template theme={null}
{
  "subject": "{{subject}}",
  "details": "{{details}}",
  "source": "voice-agent"
}
```

## Authentication headers

Each header has a kind that identifies what it carries:

| Kind | Wire value | Sent as |
| - | - | - |
| Custom | `custom` | The header name and value you provide, e.g. `X-API-Key` |
| Bearer | `authorization_bearer` | An `Authorization` header. Provide the full value, e.g. `Bearer <token>` |
| Basic | `authorization_basic` | An `Authorization` header. Provide the full value, e.g. `Basic <credentials>` |

Header values are sent exactly as you store them. The Builder's Bearer and Basic presets create the `Authorization` header with the `Bearer ` or `Basic ` prefix pre-filled. Complete the value with your credential.

<Warning>
  Bearer and Basic authorization values are write-only. The API accepts them on create and update but never echoes them when you read the tool back: credential headers return `value: null`, with `has_secret: true` confirming a secret is stored. To rotate a secret, submit a new value.
</Warning>

## Execution modes

`execution_mode` decides whether the agent waits for your endpoint (default `blocking`):

| Mode | The agent | Your endpoint's response |
| - | - | - |
| `blocking` | Waits for the response before replying | Used in the agent's next reply |
| `fire_and_forget` | Continues immediately | Discarded: recorded in the tool-call history, never spoken |
| `background` | Keeps talking while the request runs | Announced at the next natural pause once it arrives |

**Blocking** is right when the answer drives the conversation, like an order lookup the caller is waiting on. Keep those endpoints fast: the caller hears silence while your backend works.

**Fire and forget** fits actions that need no spoken confirmation from your backend: recording a consent, logging an outcome to your CRM. The agent tells the model the request was dispatched and moves on. Failures appear only in the conversation's tool-call history. The agent is never told, so `error_handling` has no spoken effect in this mode.

**Background** fits slow work that would otherwise stall the call: aggregating a shipment status across carriers, generating a report, confirming a payment. The agent acknowledges that the request is running and continues the conversation; when the response arrives, it works the result into the conversation at the next natural pause. The model is explicitly told not to guess the result before it arrives. Failures are delivered the same way, shaped by `error_handling`.

```json Mark a tool as background theme={null}
{ "execution_mode": "background", "timeout_seconds": 180 }
```

A few things to know about background calls:

* Up to **10 background calls** can be in flight per conversation. Further calls fail immediately with an error the agent can react to ("let me finish checking the first thing you asked about").
* When the conversation ends, in-flight background calls are cancelled; the attempt stays visible in the conversation's tool-call history.
* In text-only runs ([agent tests](/agents/test/agent-tests), for example), every webhook tool executes as `blocking`, so results always appear in the transcript in order.

Client tools have their own lighter-weight equivalent: the [`expects_response` switch](/agents/build/client-tools), which makes a client tool fire-and-forget.

## Timeouts and error handling

`timeout_seconds` caps how long the platform waits for your endpoint: 1–120 seconds, default 30. Background tools may go up to 300 seconds, since they don't hold up the conversation while they run. For blocking tools especially, keep endpoints fast and lower the timeout so a slow backend can't stall the call.

`error_handling` controls what the agent learns when a call fails (default `passthrough`):

| Option | What the agent sees |
| - | - |
| `passthrough` | The error response, so the agent can react to it in conversation ("I couldn't find that order number"). |
| `hide` | Only that the call failed: no error details reach the agent. |

Use `hide` when error responses might leak internal details you don't want spoken aloud. Under `fire_and_forget` the agent never learns about failures at all, whichever option is set.

## Mock responses

A tool can store mock responses: canned payloads, each with a `name`, a `status_code` (100–599), a `content_type`, and a `body`. Mocks are saved with the tool's configuration for test scenarios. [Simulation tests](/agents/test/agent-tests#tool-mocks) that mock all tools answer the tool with its first mock response. Live calls always hit the real endpoint.

## Test your tool

The tool editor's **Test** tab fires a real request at your endpoint. Fill in the arguments as JSON (pre-filled with a sample based on your declared arguments) and send. You get back the status code, latency, response headers, and response body. A failing test never blocks saving the tool. There is no live call behind a test, so `{{system.*}}` placeholders have no value: in header values and the body template they render empty unless you add the values to the arguments JSON as a nested object, for example `{"system": {"caller_number": "+15551234567"}}`. The URL is sent exactly as saved; placeholders in it are not substituted by a test request.

Response bodies are captured up to 64 KB; larger bodies are cut off and flagged with `response_truncated`.

## Create tools via the API

`POST /v1/agent/tools` creates a tool in your workspace. Tools created this way are not attached to any agent. Enable them per agent afterwards.

```bash Create a webhook tool theme={null}
curl --request POST https://api.fish.audio/v1/agent/tools \
  --header "Authorization: Bearer $FISH_API_KEY" \
  --header "Content-Type: application/json" \
  --data '{
    "tool_type": "webhook",
    "name": "create_ticket",
    "description": "Open a support ticket. Use when the issue cannot be resolved in conversation.",
    "arguments": [
      { "name": "subject", "description": "One-line summary of the issue." },
      { "name": "details", "description": "Full description of the problem." }
    ],
    "method": "POST",
    "url": "https://api.example.com/tickets",
    "content_type": "application/json",
    "body_template": "{ \"subject\": \"{{subject}}\", \"details\": \"{{details}}\" }",
    "headers": [
      { "name": "X-API-Key", "value": "YOUR_API_KEY", "kind": "custom" }
    ],
    "timeout_seconds": 30,
    "error_handling": "passthrough"
  }'
```

## Escalate to a ticket mid-call

When the agent can't resolve an issue, the strongest close is a ticket opened while the caller is still on the line, with the ticket number read back aloud. The `create_ticket` tool above is the entire integration. What makes it work is the prompt around it and the response your endpoint returns.

**Set the escalation policy in the prompt.** The tool description says what the tool does; the [system prompt](/agents/build/configuration) says when escalating is the right move:

```text System prompt excerpt theme={null}
If you cannot resolve the caller's issue, offer to open a support ticket.
Confirm they agree before calling create_ticket. Write a one-line subject and
put what the caller reported and what you already tried into the details.
After the tool returns, read the ticket number back and say when to expect a
reply.
```

**Return something worth saying.** With `error_handling: passthrough` (the default), the agent sees your response body and uses it in its next reply, so respond with what the caller should hear:

```json Endpoint response theme={null}
{ "ticket_id": "T-1042", "expected_reply": "within 24 hours" }
```

Ticket numbers get spoken aloud: short, pronounceable IDs survive text-to-speech far better than UUIDs.

<Note>
  An in-call ticket depends on the model choosing to escalate. For a safety net
  that catches every unresolved call (including those where the agent never
  called the tool), pair this with [auto-ticketing from post-call
  analysis](/agents/monitor/webhooks#auto-ticket-unresolved-calls).
</Note>

## Limits

| Field | Limit |
| - | - |
| `name` | 1–120 characters. Display text; the model-facing function name is derived from it. [Client tool names](/agents/build/client-tools#naming-rules) are stricter: a 64-character pattern, since the SDK registers handlers by exact name |
| `description` | up to 2,000 characters |
| Argument `name` | 1–64 characters |
| Argument `description` | up to 500 characters |
| `url` | up to 4,000 characters |
| `body_template` | up to 100,000 characters |
| Mock response `body` | up to 100,000 characters |
| Mock response `status_code` | 100–599 |
| `timeout_seconds` | 1–120 seconds (background tools: 1–300), default 30 |
| `execution_mode` | `blocking` (default), `fire_and_forget`, or `background`; webhook tools only |
| Background calls in flight | 10 per conversation |
| Test response capture | first 64 KB (`response_truncated` set beyond that) |

### URL restrictions

Endpoint URLs must use `http` or `https`. Requests to localhost, private network ranges, and cloud metadata endpoints are rejected, and URLs may not embed credentials. The Builder validates the URL as you type; the API enforces the same rules.

## Going further

<CardGroup>
  <Card title="Client tools" icon="browser" href="/agents/build/client-tools">
    Run tool calls in your own app instead of over HTTP.
  </Card>

  <Card title="System tools" icon="gears" href="/agents/build/system-tools">
    Built-in capabilities you toggle per agent.
  </Card>

  <Card title="Preview calls" icon="phone" href="/agents/test/preview-calls">
    Talk to your agent and watch tool calls fire.
  </Card>

  <Card title="Agent tests" icon="vial" href="/agents/test/agent-tests">
    Assert on agent behavior, tools included.
  </Card>
</CardGroup>


# Core Concepts
Source: https://docs.fish.audio/agents/concepts

Workspaces, agents, versions, and sessions: the mental model behind Fish Agents

Four nouns cover the whole platform. A **workspace** holds everything your team builds. An **agent** is a voice assistant you configure. Publishing an agent's draft creates an immutable **version**. And every conversation with an agent (from your app, a phone call, or the console) is a **session**.

## Workspaces

A workspace is the isolation boundary for everything you create. Agents, tools, knowledge sources, and tests all belong to a workspace, and nothing in it is visible from any other workspace.

Tools, knowledge sources, and tests are **workspace-level resources**, not per-agent ones: create a webhook tool once and attach it to as many agents as you like. Access controls in the console decide which agents can use each shared resource.

Your API key belongs to your team: requests under `/v1/agent/*` can read and write agent resources across all of your team's workspaces, and each resource carries its `workspace_id`. Resources created through the API land in the key owner's default workspace:

```bash List your team's agents theme={null}
curl https://api.fish.audio/v1/agent/agents \
  --header "Authorization: Bearer $FISH_API_KEY"
```

Create keys under [API keys](https://fish.audio/app/api-keys/) in the console.

## Agents

An agent is a bundle of configuration: a system prompt that steers behavior, a first-message policy, a [voice and speaking language](/agents/build/voice-language), attached [tools](/agents/build/tools), and [knowledge sources](/agents/build/knowledge-base). You assemble it in the Builder (the console page you land on after creating an agent) or through the API at `/v1/agent/agents`.

Creating an agent takes only a name. Everything else is configured afterward, so start minimal and iterate.

## Drafts and published versions

Every agent has exactly one **draft** and any number of immutable **published versions**.

<Steps>
  <Step title="Edit the draft">
    Everything you change in the Builder autosaves to the draft. Watch for the
    `Saving… / Saved` indicator.
  </Step>

  <Step title="Publish">
    **Publish** snapshots the draft into a numbered, immutable version and makes
    it the version that live traffic uses. The button is only enabled when the
    draft actually differs from the latest published version.
  </Step>

  <Step title="Keep iterating">
    Continue editing the draft without affecting live conversations. When you're
    ready, publish again. The version number increments.
  </Step>
</Steps>

### Live vs. Draft

The agent list shows one of two states:

| Badge | Meaning |
| - | - |
| **Live** | The agent has been published at least once |
| **Draft** | The agent has never been published |

Editing the draft of a Live agent does not change its badge. The Builder shows an unpublished-changes indicator next to **Publish** instead.

### Which configuration runs

* [Preview calls](/agents/test/preview-calls) in the Builder always run the **current draft**, so you can hear changes before anyone else does.
* Sessions from the SDK, phone calls, and public agent links run the **latest published version**.

<Tip>
  This split is the core workflow: iterate on the draft, verify with a preview
  call, then publish to roll the change out to real traffic.
</Tip>

### Restore and clone

Past versions stay in the agent's history. Review them or restore one back into the draft from [Versions & publishing](/agents/deploy/versions-publishing).

**Clone** creates a new agent from an existing one's current draft: prompt, voice, and tool and knowledge attachments carry over (shared resources are referenced, not copied). The clone is named `{name} copy`, starts as Draft, and does not inherit the source's publish history.

## Sessions

A session is one conversation between one user and one agent. Sessions start from four places:

<CardGroup>
  <Card title="Your app" icon="code" href="/agents/deploy/web-sdk">
    A client connects with the [Web](/agents/deploy/web-sdk) or
    [React](/agents/deploy/react-sdk) SDK.
  </Card>

  <Card title="Preview call" icon="headset" href="/agents/test/preview-calls">
    You test the draft from the Builder.
  </Card>

  <Card title="Phone call" icon="phone" href="/agents/telephony/inbound-calls">
    A caller dials a connected [phone number](/agents/telephony/phone-numbers).
  </Card>

  <Card title="Public agent" icon="globe" href="/agents/deploy/public-agents">
    Anyone talks to an agent you've shared publicly.
  </Card>
</CardGroup>

Except [preview calls](/agents/test/preview-calls), which are ephemeral and never enter history, every session appears in [Conversation history](/agents/monitor/conversation-history) and `/v1/agent/sessions` (with its transcript, and its recording when the agent [records audio](/agents/monitor/conversation-history#what-gets-stored)) and can trigger [post-call analysis](/agents/monitor/post-call-analysis) when configured.

## Deleting resources

Deletion is immediate and, from the console, cannot be undone: a deleted agent disappears from lists, the Builder, and all API responses. The same applies to tools, knowledge sources, and tests.

<Note>
  Names are freed on deletion: you can immediately create a new agent (or tool)
  with the same name.
</Note>

## Where everything lives

| Concept | Console | API resource |
| - | - | - |
| Agent | **Agents** list → Builder | [`/v1/agent/agents`](/api-reference/endpoint/agent/list-agents) |
| Draft & versions | Builder (autosave + **Publish**) | [`/v1/agent/agents`](/api-reference/endpoint/agent/get-draft-config) |
| Session | [Preview calls](/agents/test/preview-calls), [Conversation history](/agents/monitor/conversation-history) | [`/v1/agent/sessions`](/api-reference/endpoint/agent/create-agent-session) |
| Tool | Builder **Tools** page | [`/v1/agent/tools`](/api-reference/endpoint/agent/list-tools) |
| Knowledge source | Builder **Knowledge base** page | [`/v1/agent/knowledge-sources`](/api-reference/endpoint/agent/list-knowledge-sources) |
| Phone number | **Phone numbers** page | [`/v1/agent/phone-numbers`](/api-reference/endpoint/agent/list-phone-numbers) |

## Going further

<CardGroup>
  <Card title="Quickstart" icon="rocket" href="/agents/quickstart">
    Create, configure, and talk to your first agent.
  </Card>

  <Card title="Configuration" icon="sliders" href="/agents/build/configuration">
    System prompt, first message, and conversation settings.
  </Card>

  <Card title="Versions & publishing" icon="code-branch" href="/agents/deploy/versions-publishing">
    Publish history, restore, and rollout details.
  </Card>

  <Card title="Integrate" icon="plug" href="/agents/deploy/overview">
    Put your agent in front of users with the SDKs.
  </Card>
</CardGroup>


# Authenticated Sessions
Source: https://docs.fish.audio/agents/deploy/authenticated-sessions

Create short-lived session tokens on your backend and run private agents in any client

An authenticated session starts on your server: your backend calls the session endpoint with your API key and receives a short-lived **session token**, which your client uses to connect. The API key never leaves your server, and every session parameter (user identity, overrides, dynamic variables) is set by code you trust. This is how private agents run in production.

The token is client-agnostic. Create it the same way regardless of which surface renders the conversation:

| Client | Where the token goes |
| - | - |
| [Web SDK](/agents/deploy/web-sdk) | `AgentSession.start({ sessionToken })` |
| [React SDK](/agents/deploy/react-sdk) | `startSession({ sessionToken })` |
| [Widget](/agents/deploy/widget#private-agents) | Returned from `sessionTokenProvider` |
| [Custom client](/agents/deploy/protocol) | Connect to the transport named in the response, per the [wire protocol](/agents/deploy/protocol) |

<Note>
  No backend, and anyone may talk to the agent? A [public
  agent](/agents/deploy/public-agents) lets the SDK create sessions with just an
  `agentId`: no token involved, gated by an origin allowlist and rate limits.
</Note>

## Create a token on your backend

Your backend exchanges your API key for a single-conversation token.

<Steps>
  <Step title="Create a session from your backend">
    Call `POST /v1/agent/sessions` with your API key. This is where you set
    per-session parameters: user identity, timezone, overrides, dynamic
    variables.
  </Step>

  <Step title="Return the response to your frontend">
    The response is the session token. Forward it verbatim.
  </Step>

  <Step title="Start the client with the token">
    Pass the object to `AgentSession.start({sessionToken})` unmodified. The SDK
    handles the connection from there.
  </Step>
</Steps>

<CodeGroup>
  ```bash API (curl) theme={null}
  curl --request POST https://api.fish.audio/v1/agent/sessions \
    --header "Authorization: Bearer $FISH_API_KEY" \
    --header "Content-Type: application/json" \
    --data '{
      "agent_id": "YOUR_AGENT_ID",
      "end_user_id": "user_42",
      "timezone": "Asia/Tokyo",
      "dynamic_variables": { "name": "Ada" }
    }'
  ```

  ```javascript Backend (Express) theme={null}
  app.post("/api/voice-session", async (req, res) => {
    const upstream = await fetch("https://api.fish.audio/v1/agent/sessions", {
      method: "POST",
      headers: {
        Authorization: `Bearer ${process.env.FISH_API_KEY}`,
        "Content-Type": "application/json",
      },
      body: JSON.stringify({
        agent_id: "YOUR_AGENT_ID",
        end_user_id: req.user.id,
        timezone: req.user.timezone, // IANA name, like "Asia/Tokyo"
        dynamic_variables: { name: req.user.name },
      }),
    });
    if (!upstream.ok) {
      console.error("session creation failed:", upstream.status);
      return res.status(502).json({ error: "session_unavailable" });
    }
    res.json(await upstream.json());
  });
  ```

  ```python Backend (Python) theme={null}
  import os
  import requests

  def create_voice_session(end_user_id: str, name: str, timezone: str) -> dict:
      response = requests.post(
          "https://api.fish.audio/v1/agent/sessions",
          headers={"Authorization": f"Bearer {os.environ['FISH_API_KEY']}"},
          json={
              "agent_id": "YOUR_AGENT_ID",
              "end_user_id": end_user_id,
              "timezone": timezone,  # IANA name, like "Asia/Tokyo"
              "dynamic_variables": {"name": name},
          },
      )
      response.raise_for_status()
      return response.json()
  ```
</CodeGroup>

`timezone` overrides the agent's configured timezone for this session, so the agent speaks in your user's local date and time. The default is UTC: see [Time & timezone](/agents/build/time-timezone).

On the client, fetch the token from your backend and pass it to the SDK unchanged:

```javascript Browser theme={null}
import { AgentSession } from "@fishaudio/agent-client";

const sessionToken = await fetch("/api/voice-session", { method: "POST" }).then(r =>
  r.json()
);

const session = await AgentSession.start({ sessionToken });
```

More on what the SDK can do once connected is in the [Web SDK](/agents/deploy/web-sdk) reference.

<Note>
  `overrides`, `dynamic_variables`, `tool_events`, `timezone`, and
  `world_context` belong in your backend's creation request. The SDK forwards
  these options (and its `language` shorthand for `overrides.language`) only in
  [public agent](/agents/deploy/public-agents) mode.
</Note>

## Request fields

| Field | Type | Description |
| - | - | - |
| `agent_id` | string, required | The agent to talk to. It must have a [published version](/agents/deploy/versions-publishing). |
| `name` | string, optional | Display name for this session in the console's Conversations list, up to 128 characters. Omit it to show the session's start time instead. API-key requests only: keyless (public) creation rejects it with `400`. |
| `overrides` | object, optional | Replace parts of the published configuration for this session. See [Overrides](#overrides). |
| `dynamic_variables` | object, optional | Up to 50 entries of string, number, or boolean values, substituted into `{{placeholders}}`. See [Dynamic variables](/agents/build/dynamic-variables). |
| `tool_events` | boolean, optional | Stream tool lifecycle events (`toolCallStarted` / `toolCallCompleted` / `toolCallFailed`) to the client. Default `true`; set `false` to keep tool inputs and outputs off the client. |
| `end_user_id` | string, optional | Your identifier for the end user, up to 256 characters. Stored on the session and echoed in [webhook](/agents/monitor/webhooks) payloads and [custom LLM](/agents/build/custom-llm) requests. |
| `metadata` | object, optional | Your own key-value namespace. Stored and returned verbatim on session queries and webhooks, never read or interpreted by the platform. |
| `llm_extra_body` | object, optional | JSON object (at most 16 KB) forwarded verbatim to your [custom LLM](/agents/build/custom-llm) endpoint on every request as `fishaudio_extra_body`, for example the chat or thread the user is in. Not stored or returned on session reads; ignored when the agent uses a platform model. |
| `record_audio` | boolean, optional | Whether to record this session's audio. Overrides the agent's [recording setting](/agents/monitor/conversation-history#what-gets-stored) for this session only; omit it to use the agent's configuration. |
| `timezone` | string, optional | IANA timezone (like `Asia/Tokyo`) for the agent's sense of local time. Invalid names are rejected with `422`. See [Time & timezone](/agents/build/time-timezone). |
| `client_timezone` | string, optional | The end user's device timezone as an IANA name. A hint, not a demand: it applies only when neither `timezone` nor the agent's configured timezone is set, and invalid values are ignored. The SDK fills it automatically in public-agent mode. From your backend, forward the value from your client. See the [resolution order](/agents/build/time-timezone). |
| `world_context` | boolean, optional | Whether the agent knows the current date and time. Default `true`; set `false` to withhold both from this session. |

Unknown fields (top-level or inside `overrides`) are rejected with `422`.

## Overrides

`overrides` replaces parts of the agent's published configuration for one session:

| Field | Type | Effect for this session |
| - | - | - |
| `first_message` | string, up to 10,000 characters | The agent opens with exactly this text, spoken verbatim, whatever first-message mode the configuration sets. `{{placeholders}}` render inside it. |
| `first_message_prompt` | string, up to 10,000 characters | Instructions the agent generates its opener from, replacing the configured first-message behavior. `{{placeholders}}` render inside it. Mutually exclusive with `first_message`: sending both is `422`. |
| `system_prompt` | string, up to 32,000 tokens | Replaces the configured system prompt entirely, under the same token budget. `{{placeholders}}` render inside it. |
| `voice_id` | string | The voice the agent speaks with: any [voice model id](/agents/build/voice-language#use-any-voice-model) from the Voice Library. Voices bias pronunciation toward their own language, so pair it with `language`. |
| `language` | the code of one of the [52 supported languages](/agents/build/voice-language#speaking-language) | Pins |
| the conversation language, taking precedence over the configured [speaking language](/agents/build/voice-language#speaking-language). | | |

```json theme={null}
{
  "agent_id": "YOUR_AGENT_ID",
  "overrides": {
    "first_message": "Welcome back, {{name}}, picking up where we left off.",
    "voice_id": "802e3bc2b27e49c2995d23ef70e6ac89",
    "language": "ja"
  }
}
```

On [public agents](/agents/deploy/public-agents), keyless (browser-created) sessions may override only `language` and `voice_id`. The prompt-shaping fields are rejected with `400` regardless of the agent. Create the session from your backend to use them.

Everything else on the request is not an override: `dynamic_variables`, `timezone`, `record_audio`, and `tool_events` are separate top-level fields. Session length is not settable per session: `max_duration_seconds` comes from the agent's [conversation configuration](/agents/build/configuration#call-duration).

## Response

```json JSON theme={null}
{
  "session_id": "abc123",
  "expires_at": "2026-07-23T12:00:00Z",
  "max_duration_seconds": 1800,
  "transport": "livekit",
  "livekit_url": "wss://example.livekit.cloud",
  "token": "<participant token>"
}
```

| Field | Description |
| - | - |
| `session_id` | The session's id. Use it later to look up the [conversation record](/agents/monitor/conversation-history). |
| `expires_at` | Deadline for the client to connect. Create the token right before starting, not ahead of time. |
| `max_duration_seconds` | Hard cap on session length. |
| `transport` | Which transport the SDK uses for this session; `livekit` today. |
| `livekit_url`, `token` | Connection details for that transport, consumed by the SDK. |

Treat the response as opaque and pass it to `start()` unmodified. If the SDK does not recognize the `transport` value, it fails fast with an `unsupported_transport` error asking you to upgrade the SDK.

## Token lifetime

* A session token is **single-use**: `start()` consumes it once to establish the conversation.
* On network drops the SDK reconnects at the transport level; it never re-creates the session, so you never need a fresh token mid-call.
* Once a session ends, the token is spent. Create a new token for each conversation.

## Ending sessions from your backend

A session normally ends from the client side: the user disconnects, or the agent [hangs up](/agents/build/system-tools). To force-end a live session server-side, call the end endpoint with your API key and the `session_id` from the creation response:

```bash Request theme={null}
curl --request POST https://api.fish.audio/v1/agent/sessions/$SESSION_ID/end \
  --header "Authorization: Bearer $FISH_API_KEY"
```

The call terminates and the response is `204`. The session record stays readable in [conversation history](/agents/monitor/conversation-history), along with the transcript and recording when the agent [stores them](/agents/monitor/conversation-history#what-gets-stored). The same endpoint ends in-progress [phone calls](/agents/telephony/inbound-calls) too.

## Keep API keys on the server

<Warning>
  Your API key grants full access to every resource in your team: agents,
  sessions, tools, phone numbers. Never embed it in a browser, mobile app, or
  any code you ship to users. The only credential that belongs in a client is
  the session token, and the only credential-free path is a public agent ID.
</Warning>

## Error responses

Session creation fails with standard HTTP statuses; the SDK surfaces them as a `FishAgentError` with `statusCode` set.

| Status | Meaning |
| - | - |
| `400` | A keyless (public-agent) request sent an override [public sessions don't accept](#overrides), or a `name`. |
| `401` | Invalid API key. A request with no `Authorization` header at all is treated as a public-agent request instead. |
| `402` | Quota exceeded. |
| `403` | Public-agent request rejected: the agent is not public, or the page's `Origin` is not on the allow-list. |
| `409` | Conflict: most commonly the agent has no published version yet. [Publish](/agents/deploy/versions-publishing) to resolve. |
| `422` | The request body failed validation: an unknown field, an unsupported `language` code, an invalid `timezone`, or an invalid dynamic variable. |
| `429` | Rate limited: a public-agent request exceeded the per-agent or per-IP limit. |

The complete status-code reference for every `/v1/agent` endpoint (error shapes, `400` vs `422`, conflict semantics) is on [Agents API errors](/api-reference/agent-errors).

## Going further

<CardGroup>
  <Card title="Public agents" icon="globe" href="/agents/deploy/public-agents">
    The credential-free alternative, with origin allow-lists.
  </Card>

  <Card title="Widget with private agents" icon="puzzle-piece" href="/agents/deploy/widget#private-agents">
    Feed the widget session tokens from your backend via `sessionTokenProvider`.
  </Card>

  <Card title="Web SDK" icon="js" href="/agents/deploy/web-sdk">
    Everything `AgentSession` can do once connected.
  </Card>

  <Card title="Dynamic variables" icon="brackets-curly" href="/agents/build/dynamic-variables">
    Personalize each session at creation time.
  </Card>
</CardGroup>


# Deploy Your Agent
Source: https://docs.fish.audio/agents/deploy/overview

Choose where users meet your agent and who may start sessions: widget, SDKs, or phone; public or authenticated

Your agent is one artifact with many front doors. Deploying it is two independent decisions:

1. **Where users meet your agent**: a prebuilt widget, your own UI built on the SDKs, a phone number, or a custom client.
2. **Who may start sessions**: anyone visiting your pages, or only clients your backend has vouched for.

Every surface runs the same agent with the same published configuration, and the browser surfaces work with either access mode.

<Note>
  Publishing gates every channel: sessions always run the agent's **published**
  configuration, and creating a session against an agent that has never been
  published fails. [Publish](/agents/deploy/versions-publishing) before wiring
  up any channel.
</Note>

## Where users meet your agent

In increasing order of integration effort:

<CardGroup>
  <Card title="Widget" icon="puzzle-piece" href="/agents/deploy/widget">
    Drop a prebuilt call widget into your site: two lines of HTML, zero build
    step.
  </Card>

  <Card title="Your own UI" icon="js" href="/agents/deploy/web-sdk">
    Build a custom experience with the [Web SDK](/agents/deploy/web-sdk) or
    [React SDK](/agents/deploy/react-sdk): events, transcripts, audio controls.
  </Card>

  <Card title="Phone" icon="phone" href="/agents/telephony/phone-numbers">
    Bind a number and callers dial straight in.
  </Card>

  <Card title="Custom client" icon="network-wired" href="/agents/deploy/protocol">
    On a stack the SDKs don't cover, implement the [wire
    protocol](/agents/deploy/protocol) directly.
  </Card>
</CardGroup>

## Who may start sessions

The widget and the SDKs each work with **either** access mode; pick per agent. Phone numbers skip this decision entirely: inbound calls connect through [telephony](/agents/telephony/inbound-calls), not a browser session.

<CardGroup>
  <Card title="Public agents" icon="globe" href="/agents/deploy/public-agents">
    The client connects with just an `agentId`: no backend, gated by an origin
    allowlist and rate limits.
  </Card>

  <Card title="Authenticated sessions" icon="server" href="/agents/deploy/authenticated-sessions">
    Your backend creates a short-lived session token with your API key: trusted
    attribution and full control.
  </Card>
</CardGroup>

| | Public agent ID | Session token |
| - | - | - |
| Credential in browser | None | Short-lived session token |
| Requires | Agent set to public + allowed origins | Your own backend endpoint |
| Session parameters set by | The SDK (`overrides`, `dynamicVariables`, `language`, `toolEvents` options) | Your backend, in the creation request |
| Best for | Demos, marketing pages, low-friction embeds | Production apps with signed-in users |

## Still deciding?

| You are building | Start with |
| - | - |
| A call button on an existing site, no build step | [Widget](/agents/deploy/widget) with [public access](/agents/deploy/public-agents) |
| A demo or marketing page without a backend | [Web SDK](/agents/deploy/web-sdk) on a [public agent](/agents/deploy/public-agents) |
| Voice calls inside a web app with signed-in users | [Web SDK](/agents/deploy/web-sdk) with [authenticated sessions](/agents/deploy/authenticated-sessions) |
| A React or Next.js frontend | [React SDK](/agents/deploy/react-sdk) |
| A phone line callers dial in to | [Phone numbers](/agents/telephony/phone-numbers) and [transfers](/agents/telephony/transfers) |
| Server-side session creation and history retrieval | [Authenticated sessions](/agents/deploy/authenticated-sessions) and [conversation history](/agents/monitor/conversation-history) |
| A client on a stack the SDKs don't cover | [Wire protocol](/agents/deploy/protocol) |

## How a browser session starts

For private agents, your backend creates the session. Your API key never ships to the browser. The browser SDK takes the resulting session token and handles audio and transport from there.

```bash Your backend theme={null}
curl --request POST https://api.fish.audio/v1/agent/sessions \
  --header "Authorization: Bearer $FISH_API_KEY" \
  --header "Content-Type: application/json" \
  --data '{ "agent_id": "YOUR_AGENT_ID" }'
# → 201 session token. Return it to your frontend as-is
```

```javascript Your frontend theme={null}
import { AgentSession } from "@fishaudio/agent-client";

// sessionToken = the create-session response, passed through unchanged
const session = await AgentSession.start({ sessionToken });
```

<Tip>
  Agents marked [public](/agents/deploy/public-agents) skip the backend step
  entirely: the SDK connects with just an `agentId`, gated by an origin
  allowlist and rate limits.
</Tip>

For phone calls there is no client integration: assign a number, and inbound calls connect directly to your agent, which can [transfer callers](/agents/telephony/transfers) to another number mid-call. See [Phone numbers](/agents/telephony/phone-numbers).

## Going further

<CardGroup>
  <Card title="Versions & publishing" icon="rocket" href="/agents/deploy/versions-publishing">
    The publish step every channel depends on.
  </Card>

  <Card title="Authenticated sessions" icon="server" href="/agents/deploy/authenticated-sessions">
    The full session-creation reference: request fields, token lifetime,
    errors.
  </Card>

  <Card title="Conversation history" icon="clock-rotate-left" href="/agents/monitor/conversation-history">
    List sessions and read full transcripts over REST.
  </Card>

  <Card title="Wire protocol" icon="code" href="/agents/deploy/protocol">
    The wire-level session contract, for building your own client.
  </Card>
</CardGroup>


# Wire Protocol
Source: https://docs.fish.audio/agents/deploy/protocol

The realtime message contract between clients and agent sessions, for platforms the SDKs don't cover

Everything the SDKs do rides a small, versioned wire protocol: two JSON message channels, plus the transport's standard transcription and state mechanisms. The transport is WebRTC, built on LiveKit: audio travels as WebRTC [media tracks](https://docs.livekit.io/home/client/tracks/), the message channels ride LiveKit's reliable [data channels](https://docs.livekit.io/home/client/data/messages/), and any platform with a [LiveKit client SDK](https://docs.livekit.io/home/client/connect/) can implement it. This page documents that contract for consumers that cannot use the SDKs: custom native stacks or ports to new platforms.

<Note>
  This is an escape hatch. For web and React apps, use the [Web
  SDK](/agents/deploy/web-sdk) or [React SDK](/agents/deploy/react-sdk)
  instead. They implement everything below (connection, reconnection, message
  parsing, tool dispatch) and stay current as the protocol evolves.
</Note>

## The protocol package

Every message shape on this page is published as TypeScript definitions in `@fishaudio/agent-protocol`, with zero runtime dependencies. This page documents the protocol as of package version **0.1.0**. The package is the source of truth for shapes; this page fixes the semantics.

```bash npm theme={null}
npm install @fishaudio/agent-protocol
```

## Connect to a session

Create a session server-side ([authenticated sessions](/agents/deploy/authenticated-sessions)), then connect to the transport named in the response. [Public agents](/agents/deploy/public-agents) can create sessions without the `Authorization` header.

```bash Create a session theme={null}
curl --request POST https://api.fish.audio/v1/agent/sessions \
  --header "Authorization: Bearer $FISH_API_KEY" \
  --header "Content-Type: application/json" \
  --data '{ "agent_id": "YOUR_AGENT_ID" }'
```

```json Response (201) theme={null}
{
  "session_id": "...",
  "expires_at": "2026-07-23T12:34:56Z",
  "max_duration_seconds": 1800,
  "transport": "livekit",
  "livekit_url": "wss://...",
  "token": "<participant JWT>"
}
```

The response is a discriminated union on `transport`. The `livekit` arm carries `livekit_url` and `token`: [connect](https://docs.livekit.io/home/client/connect/) a LiveKit-compatible WebRTC client with them before `expires_at` (the join deadline), then [publish your microphone track and subscribe](https://docs.livekit.io/home/client/tracks/) to the agent's audio track.

<Warning>
  If you receive a `transport` value you don't recognize, fail with an explicit
  "unsupported transport" error. New transport arms may be introduced, each
  with its own payload.
</Warning>

## Channels

Once connected, a session uses these channels. The two `*-event` topics carry reliable [data packets](https://docs.livekit.io/home/client/data/messages/): one complete JSON object per packet.

| Channel | Direction | Carries |
| - | - | - |
| Audio tracks | both | Your microphone up, the agent's speech down |
| `agent-event` topic | agent → client | Control events: client tool calls, tool lifecycle, errors, session end |
| `client-event` topic | client → agent | Text turns, activity, interrupts, hangup, tool results |
| `lk.transcription` text streams | agent → client | Streaming transcripts for both sides |
| `lk.agent.state` participant attribute | agent → client | Agent pipeline state (sticky) |

Data-channel message fields are camelCase; REST bodies are snake\_case.

Each channel maps to a standard LiveKit client mechanism:

* **`client-event` (sending)**: publish each message as a UTF-8 JSON payload with [`publishData`](https://docs.livekit.io/home/client/data/messages/), in **reliable** mode, with `topic` set to `client-event`.
* **`agent-event` (receiving)**: listen for the [`DataReceived`](https://docs.livekit.io/home/client/data/messages/) room event and keep only packets whose topic is `agent-event`; decode each payload as one JSON object.
* **Transcription**: register a handler for the `lk.transcription` topic with [`registerTextStreamHandler`](https://docs.livekit.io/home/client/data/text-streams/). Segment semantics are described [below](#transcription-and-agent-state).
* **Agent state**: read `lk.agent.state` from the agent participant's [attributes](https://docs.livekit.io/home/client/state/participant-attributes/) and subscribe to the `AttributesChanged` event for updates.

The complete wiring in the JS SDK (`livekit-client`) looks like this; every LiveKit SDK has equivalent APIs:

```ts Connect and wire up all channels theme={null}
import { Room, RoomEvent } from 'livekit-client';

const room = new Room();

// Register all handlers BEFORE connecting: the agent may already be
// speaking when you join, and `lk.agent.state` is delivered on join.

// Agent events: reliable data packets on the `agent-event` topic
room.on(RoomEvent.DataReceived, (payload, participant, kind, topic) => {
  if (topic !== 'agent-event') return;
  const event = JSON.parse(new TextDecoder().decode(payload));
  // event.type: 'client_tool.call' | 'tool.started' | 'tool.completed' | ...
});

// Transcription: text streams on the `lk.transcription` topic
room.registerTextStreamHandler('lk.transcription', async (reader, participant) => {
  const segmentId = reader.info.attributes?.['lk.segment_id'];
  const isFinal = reader.info.attributes?.['lk.transcription_final'] === 'true';
  const text = await reader.readAll();
  // participant.identity tells you whether this is the user's or the agent's segment
});

// Agent state: sticky `lk.agent.state` participant attribute
room.on(RoomEvent.ParticipantAttributesChanged, (changed, participant) => {
  if ('lk.agent.state' in changed) {
    console.log(participant.attributes['lk.agent.state']);
  }
});

// Agent speech: attach the agent's audio track when it arrives
room.on(RoomEvent.TrackSubscribed, (track) => {
  document.body.appendChild(track.attach());
});

// Now connect, using `livekit_url` and `token` from the
// session-creation response above, and publish your microphone
await room.connect(livekitUrl, token);
await room.localParticipant.setMicrophoneEnabled(true);

// Client events: publish JSON on the `client-event` topic
function send(event: object) {
  return room.localParticipant.publishData(
    new TextEncoder().encode(JSON.stringify(event)),
    { reliable: true, topic: 'client-event' },
  );
}

await send({ type: 'user.message', text: "What's my balance?" });
```

## Agent events: `agent-event`

### `client_tool.call`

The agent wants to run a [client tool](/agents/build/client-tools) in your app.

```json client_tool.call theme={null}
{
  "type": "client_tool.call",
  "callId": "call_abc",
  "toolName": "open_dashboard",
  "params": { "tab": "billing" },
  "expectsResponse": true
}
```

Run the tool, then publish a `client_tool.result` with the same `callId`. While `expectsResponse` is `true`, the agent suspends the model's tool call until your result arrives or the tool's configured timeout elapses (default 30 seconds, configurable from 1 to 120). When `expectsResponse` is `false`, the call is fire-and-forget: the agent continues immediately and any result you send is ignored.

### Tool lifecycle: `tool.started`, `tool.completed`, `tool.failed`

Each tool the agent invokes emits one `tool.started`, resolved by exactly one terminal message (`tool.completed` or `tool.failed`) with the same `callId`. Terminal messages repeat `toolName` and `toolSource`, so a client that missed the start can still render a complete entry.

| Field | Type | Notes |
| - | - | - |
| `callId` | string | Correlates a start with its terminal message |
| `toolName` | string | The tool name as the model sees it |
| `toolSource` | string | Where the tool runs: `client`, `webhook`, `mcp`, `builtin`, `background`, or `unknown`; treat as an open set |
| `input` / `output` | string | JSON-serialized payload, truncated at the source at 4 KB |
| `inputTruncated` / `outputTruncated` | boolean | Present when the payload was truncated |
| `error` | string | `tool.failed` only |

These events are on by default and their payloads travel to the end user's client. Pass `tool_events: false` when creating the session to keep tool data off the wire: the session then receives none of the three.

### `error`

```json error theme={null}
{ "type": "error", "code": "provider_error" }
```

`code` is a coarse category only: `provider_error` (an upstream model or voice provider failed) or `internal_error` (the runtime failed). The message deliberately carries no raw error detail.

### `session.ended`

```json session.ended theme={null}
{ "type": "session.ended", "reason": "conversation_timeout" }
```

Sent once, right before the server ends the call (before the agent leaves the room or the room is deleted), so a client learns the cause instead of inferring it from the disconnect. `reason` is `user_hangup` (the user hung up, or a backend ended the session through the API), `agent_hangup` (the agent hung up or the workflow reached an end node), `conversation_timeout` (the session reached its maximum duration), or `escalated` (the call was transferred to a human). The values match `end_reason` on the [session record](/agents/monitor/conversation-history#status-and-end-reason); treat the set as open, per the [compatibility rules](#compatibility-rules). A call that drops without this message has no announced reason: the SDKs report `connection_lost` once reconnection fails.

## Client events: `client-event`

Publish these on the `client-event` topic. The agent ignores malformed JSON and unknown types.

| Type | Effect |
| - | - |
| `user.message` | Injects a text user turn; the agent replies as if the user had spoken |
| `user.activity` | Signals user activity (typing); suppresses idle re-engagement |
| `user.interrupt` | Explicitly interrupts the agent's current speech |
| `user.hangup` | Ends the session gracefully |
| `client_tool.result` | Settles a pending `client_tool.call` |

```json user.message theme={null}
{ "type": "user.message", "text": "What's my balance?" }
```

```json client_tool.result theme={null}
{
  "type": "client_tool.result",
  "callId": "call_abc",
  "result": { "opened": true }
}
```

* `user.message` gets **no server echo**: you already hold the text, so render the bubble locally. Add `"audio": false` to have the agent answer that turn in text only: no speech is synthesized and the reply arrives over transcription. When the field is absent the agent speaks as usual; note the web SDK's `sendUserMessage` sends `"audio": false` unless called with `audio: true`.
* `client_tool.result` may carry `result` (any JSON value) or `"isError": true` to report the tool as failed to the model. Results for unknown or already-settled `callId`s are ignored.

## Transcription and agent state

These ride the transport's built-in mechanisms rather than custom messages.

**Transcription** arrives as [text streams](https://docs.livekit.io/home/client/data/text-streams/) on the `lk.transcription` topic. Segments are identified by the `lk.segment_id` stream attribute; `lk.transcription_final: "true"` marks a segment as final. Roles are distinguished by sender identity: the user's segments are sent under the user's own participant identity, the agent's under the agent participant.

* Agent segments stream incrementally, paced to audio playback. An interrupted segment closes containing only the words actually spoken; there is no residual text.
* User segments are interim until final; each interim update **replaces the entire segment text** under the same segment id.

**Agent state** is published as the sticky `lk.agent.state` [participant attribute](https://docs.livekit.io/home/client/state/participant-attributes/) with values `initializing`, `idle`, `listening`, `thinking`, and `speaking`. Sticky means a client that connects late or reconnects reads the current value immediately. The SDKs derive their three public modes from this attribute plus transcript segment open/close.

## Compatibility rules

1. **Ignore unknown `type` values and unknown fields.** This is required consumer behavior and the foundation of forward compatibility.
2. **Evolution is additive-only.** Published fields never change name or meaning and are never removed; new fields are always optional. A semantic change ships as a new `type`.
3. **No replay.** Data-channel delivery is reliable and ordered within a connection, but after a reconnect or late join, missed messages are gone. Never wait for history. Tool terminal messages repeat their identifying fields, and the state attribute is sticky, precisely to soften this.

## Going further

<CardGroup>
  <Card title="Web SDK" icon="js" href="/agents/deploy/web-sdk">
    The supported implementation of this protocol for browsers.
  </Card>

  <Card title="Client tools" icon="wrench" href="/agents/build/client-tools">
    Declare the tools your `client_tool.call` handlers implement.
  </Card>

  <Card title="Authenticated sessions" icon="server" href="/agents/deploy/authenticated-sessions">
    Session creation, API keys, and token handling.
  </Card>

  <Card title="Public agents" icon="globe" href="/agents/deploy/public-agents">
    Let clients create sessions without a backend.
  </Card>
</CardGroup>


# Public Agents
Source: https://docs.fish.audio/agents/deploy/public-agents

Let visitors talk to your agent directly from the browser, no backend required

A public agent accepts sessions straight from the browser: the SDK calls the session endpoint with just an `agent_id`, no API key and no server of your own. The platform gates access with three controls: an agent-level public switch (off by default), a required origin allowlist, and rate limiting.

Use this mode for demos, marketing pages, and support bubbles where standing up a backend isn't worth it. For production apps with their own users, [authenticated sessions](/agents/deploy/authenticated-sessions) are usually the better fit. See [when to use session tokens instead](#when-to-use-session-tokens-instead).

## Enable public access

<Steps>
  <Step title="Publish the agent">
    Public sessions always run the agent's **published** configuration. An
    unpublished agent rejects session creation. See [Versions &
    publishing](/agents/deploy/versions-publishing).
  </Step>

  <Step title="Turn on the public switch">
    On the agent's **Deploy** page in the console, turn on **Public access**. It
    is **off by default**. Until you enable it, sessions require
    authentication.
  </Step>

  <Step title="Add allowed origins">
    On the same page, list every web origin that may start sessions under
    **Allowed origins** (`allowed_origins` in the API). The allowlist is
    **required**: a request whose `Origin` doesn't match any entry is rejected
    with `403`.
  </Step>
</Steps>

## Start a session from the browser

Pass `agentId` to the SDK and it creates the session directly with the Fish Audio API, then connects the audio.

<CodeGroup>
  ```javascript JavaScript theme={null}
  import { AgentSession } from "@fishaudio/agent-client";

  const session = await AgentSession.start({
    agentId: "YOUR_AGENT_ID",
  });

  session.on("agentResponse", ({ text }) => console.log("Agent:", text));
  ```

  ```bash API (curl) theme={null}
  # No Authorization header. The Origin header is checked instead.
  # Browsers send Origin automatically; curl must set it explicitly.
  curl --request POST https://api.fish.audio/v1/agent/sessions \
    --header "Content-Type: application/json" \
    --header "Origin: https://app.example.com" \
    --data '{ "agent_id": "YOUR_AGENT_ID" }'
  ```
</CodeGroup>

The response is the same session token an authenticated request returns. The SDK consumes it internally. Everything else (events, methods, client tools) works exactly as in the [Web SDK](/agents/deploy/web-sdk).

## Origin matching

An entry matches only on **exact scheme + host + port**. The host is case-insensitive, and a trailing slash on an entry is tolerated.

| Allowlist entry | Browser origin | Allowed |
| - | - | - |
| `https://app.example.com` | `https://app.example.com` | Yes |
| `https://app.example.com` | `https://APP.example.com` | Yes (host is case-insensitive) |
| `https://app.example.com` | `http://app.example.com` | No (scheme differs) |
| `https://app.example.com` | `https://app.example.com:8443` | No (port differs) |
| `http://localhost:5173` | `http://127.0.0.1:5173` | No (different hosts) |

<Warning>
  `localhost` and `127.0.0.1` are **different origins**. If you develop against both, list both:

  ```text theme={null}
  http://localhost:5173
  http://127.0.0.1:5173
  ```
</Warning>

## Rate limiting

Public session creation is rate limited on two dimensions at once: per **client IP** and per **agent**. Either limit being exceeded rejects the request. Sessions started on a public agent are billed to the workspace that owns the agent, at the standard [conversation rate](/agents/pricing).

<Note>
  If legitimate traffic outgrows these limits, or you need per-user quotas, move
  to [authenticated sessions](/agents/deploy/authenticated-sessions): your
  backend becomes the gate, and the platform-side public limits no longer apply.
</Note>

## Treat public input as untrusted

On the public path, the entire session request originates in the visitor's browser. The platform enforces its own guardrails (public sessions accept only the `language` and `voice_id` [overrides](/agents/deploy/authenticated-sessions#overrides), rejecting everything else with `400`), but anything that passes through verbatim is attacker-controllable:

* **`metadata`** is stored and returned exactly as sent, never interpreted by the platform. When you read it back in [conversation history](/agents/monitor/conversation-history) or [webhooks](/agents/monitor/webhooks), treat it as untrusted data, never as proof of who the visitor is.
* **`dynamic_variables`** are chosen by the page that starts the session. Don't inject anything through them that the visitor shouldn't control. See [Dynamic variables](/agents/build/dynamic-variables).

For trusted attribution (a verified `end_user_id`, server-set metadata), create sessions from your backend instead.

## Errors

Public-mode failures surface as `FishAgentError` codes on `AgentSession.start()`:

| Code | Meaning |
| - | - |
| `agent_not_public` | The agent's public switch is off. |
| `origin_forbidden` | The page's origin is not in `allowed_origins` (HTTP `403`). |
| `session_request_failed` | Other creation failures, for example the agent has no published version. |

## When to use session tokens instead

Public mode trades control for zero setup. Prefer the `sessionToken` flow (your backend calls the session endpoint with an API key and hands the token to the browser) when any of these apply:

* The agent should stay **private** rather than callable by anyone who loads your page.
* You need **trusted attribution**: an `end_user_id` and `metadata` set by your server, not the browser.
* You want server-side control of `overrides`, `dynamic_variables`, or `tool_events` instead of accepting the browser's values.
* You already authenticate users and want **your own quotas** per account, not IP-based platform limits.

The SDK call changes by one field: pass `{ sessionToken }` instead of `{ agentId }`. See [Authenticated sessions](/agents/deploy/authenticated-sessions) for the full flow.

## Going further

<CardGroup>
  <Card title="Authenticated sessions" icon="server" href="/agents/deploy/authenticated-sessions">
    The session token flow in full: request fields, lifetime, errors.
  </Card>

  <Card title="Web SDK" icon="js" href="/agents/deploy/web-sdk">
    Events, methods, and client tools, identical in both modes.
  </Card>

  <Card title="React SDK" icon="react" href="/agents/deploy/react-sdk">
    Hooks and components for React apps.
  </Card>

  <Card title="Conversation history" icon="clock-rotate-left" href="/agents/monitor/conversation-history">
    Review what public visitors said to your agent.
  </Card>
</CardGroup>


# React SDK
Source: https://docs.fish.audio/agents/deploy/react-sdk

Add voice conversations to your React app with hooks and drop-in components

`@fishaudio/agent-react` wraps the [Web SDK](/agents/deploy/web-sdk) in idiomatic React: a `useConversation` hook for session control, an optional provider that shares one session across your component tree, a streaming chat-log hook, and a ready-made audio visualizer. The SDK handles microphone capture, audio playback, and transport internally. Authentication, events, client tools, and error semantics are the Web SDK's; everything documented there applies here.

<CardGroup>
  <Card title="Web SDK reference" icon="js" href="/agents/deploy/web-sdk">
    Every option, event, method, and error code.
  </Card>

  <Card title="Authenticated sessions" icon="server" href="/agents/deploy/authenticated-sessions">
    Create session tokens on your backend.
  </Card>

  <Card title="Public agents" icon="globe" href="/agents/deploy/public-agents">
    Connect with just an agent id, no backend.
  </Card>
</CardGroup>

## Install

<CodeGroup>
  ```bash npm theme={null}
  npm install @fishaudio/agent-react
  ```

  ```bash pnpm theme={null}
  pnpm add @fishaudio/agent-react
  ```

  ```bash yarn theme={null}
  yarn add @fishaudio/agent-react
  ```
</CodeGroup>

Requires React 18 or later. The package re-exports the Web SDK's main exports (`AgentSession`, `FishAgentError`, and their types), so app code can import everything from `@fishaudio/agent-react`.

## Quick start

A minimal call UI: start a call, show what the agent is doing, mute, hang up. This example connects to a [public agent](/agents/deploy/public-agents) by id.

```tsx App.tsx theme={null}
import { useConversation } from "@fishaudio/agent-react";

export function CallButton() {
  const {
    startSession,
    endSession,
    status,
    mode,
    isSpeaking,
    micMuted,
    setMicMuted,
  } = useConversation();

  if (status === "connected" || status === "reconnecting") {
    const label = isSpeaking
      ? "Agent is speaking"
      : mode === "thinking"
        ? "Thinking..."
        : "Listening";

    return (
      <div>
        <p>{label}</p>
        <button onClick={() => setMicMuted(!micMuted)}>
          {micMuted ? "Unmute" : "Mute"}
        </button>
        <button onClick={() => endSession()}>Hang up</button>
      </div>
    );
  }

  return (
    <button
      disabled={status === "connecting"}
      onClick={() => startSession({ agentId: "YOUR_AGENT_ID" })}
    >
      Start call
    </button>
  );
}
```

`status`, `mode`, and `isSpeaking` are React state, so your component re-renders as the conversation progresses. When the component unmounts, the session ends automatically.

<Note>
  Call `startSession` from a user gesture (such as a click handler) so the
  browser allows microphone capture and audio playback.
</Note>

### Connect to a private agent

For agents that are not public, your backend creates the session with your API key (`POST /v1/agent/sessions`) and returns the response to the browser. Pass it to `startSession` unchanged.

```tsx theme={null}
const res = await fetch("/api/voice-session", { method: "POST" });
const sessionToken = await res.json();
await startSession({ sessionToken });
```

See [Authenticated sessions](/agents/deploy/authenticated-sessions) for the backend side. `startSession` accepts the same options as `AgentSession.start` in the [Web SDK](/agents/deploy/web-sdk#options), including [`clientTools`](/agents/build/client-tools). Session settings such as [`overrides`](/agents/deploy/authenticated-sessions#overrides) and `dynamicVariables` apply when you connect with an `agentId`; with a `sessionToken`, your backend sets them in its session-creation request instead.

### Next.js

The SDK only touches browser APIs when a session starts, so importing it in Server Components is safe, but any component that calls the hooks needs the `"use client"` directive. Create session tokens in a Route Handler so your API key stays on the server:

<CodeGroup>
  ```ts app/api/voice-session/route.ts theme={null}
  export async function POST() {
    const upstream = await fetch("https://api.fish.audio/v1/agent/sessions", {
      method: "POST",
      headers: {
        Authorization: `Bearer ${process.env.FISH_API_KEY}`,
        "Content-Type": "application/json",
      },
      body: JSON.stringify({ agent_id: process.env.FISH_AGENT_ID }),
    });
    if (!upstream.ok) {
      return Response.json({ error: "session_unavailable" }, { status: 502 });
    }
    return Response.json(await upstream.json());
  }
  ```

  ```tsx app/call-button.tsx theme={null}
  "use client";

  import { useConversation } from "@fishaudio/agent-react";

  export function CallButton() {
    const { startSession, status } = useConversation();

    const start = async () => {
      const sessionToken = await fetch("/api/voice-session", { method: "POST" })
        .then(r => r.json());
      await startSession({ sessionToken });
    };

    return (
      <button disabled={status === "connecting"} onClick={start}>
        Start call
      </button>
    );
  }
  ```
</CodeGroup>

Start calls made from a user interaction are safe under React Strict Mode's effect replay.

## `useConversation(defaults?)`

Owns one session's lifecycle and re-renders on its state changes. `defaults` is merged into every `startSession(overrides?)` call, with `overrides` winning per key. Put long-lived options such as `clientTools`, `callbacks`, or a public `agentId` in `defaults`, and per-call values such as a freshly fetched `sessionToken` in `overrides`. The latest render's `defaults` are used, so inline objects are fine.

```tsx theme={null}
const conversation = useConversation({
  clientTools: { open_page: ({ page }) => showPanel(String(page)) },
  callbacks: { onDisconnect: ({ reason }) => showCallEnded(reason) },
});

const start = async () => {
  const sessionToken = await fetch("/api/voice-session", { method: "POST" })
    .then(r => r.json());
  await conversation.startSession({ sessionToken });
};
```

### Returned value

| Field | Type | Description |
| - | - | - |
| `startSession(overrides?)` | `(overrides?) => Promise<string>` | Create and connect a session (`agentId` or `sessionToken`); resolves with the session id |
| `endSession()` | `() => Promise<void>` | Hang up gracefully. Safe to call at any time |
| `status` | `"idle" \| SessionStatus` | `"idle"` before the first start and after a failed start, then `"connecting"` / `"connected"` / `"reconnecting"` / `"ended"` |
| `mode` | `AgentMode` | `"listening"` / `"thinking"` / `"speaking"`; drive a talking-orb UI with this |
| `isSpeaking` | `boolean` | `true` while the agent is audibly speaking |
| `micMuted`, `setMicMuted(muted)` | `boolean`, `(muted) => Promise<void>` | Microphone mute state. After a `microphone: false` start it begins `true`, and the first unmute captures the microphone |
| `sendUserMessage(text, options?)` | `(text, { audio? }?) => void` | Send a typed message as a user turn. The agent replies in text only by default; pass `{ audio: true }` to have it speak the reply. Pair with [`useAgentMessages`](#chat-log-with-useagentmessages) for a chat UI |
| `sendUserActivity()` | `() => void` | Signal that the user is typing, so the agent holds back |
| `interrupt()` | `() => void` | Explicitly stop the agent mid-response |
| `startAudio()` | `() => Promise<void>` | Unlock playback inside a user gesture if autoplay was blocked |
| `setOutputVolume(volume)` | `(volume) => void` | Playback volume, `0` to `1` |
| `setInputDevice(deviceId)` | `(deviceId) => Promise<void>` | Switch the microphone mid-call. Rejects with `device_change_failed` if the device cannot be activated. No-op before the first start |
| `setOutputDevice(deviceId)` | `(deviceId) => Promise<void>` | Route playback to another output device (`""` for the default). Rejects with `device_change_failed` where unsupported (common on mobile browsers). No-op before the first start; for a pre-call pick, pass `audio.outputDeviceId` to `startSession` |
| `session` | `AgentSession \| null` | The live `AgentSession` object (`null` before the first start) for direct `.on(...)` event access |

For transcripts, tool-call events, and error handling, subscribe to events on `session` or pass `callbacks` in `defaults`. See the [Web SDK event reference](/agents/deploy/web-sdk#events).

### Lifecycle details

* **Double-start safe.** If a session is already live, `startSession` resolves with its id instead of starting a second one; two concurrent calls (a double-click) share one start.

* **Hangup-during-start safe.** Calling `endSession` while a start is still waiting on session creation, the microphone permission, or the realtime connection marks the conversation ended and waits for that start to settle; a late connection is closed before the hook exposes it. A `startSession` requested during that cleanup waits, then creates a fresh session. If another `endSession` cancels that queued restart, it rejects with an error whose `name` is `"AbortError"`.

* **Failure resets.** If `startSession` rejects (see [Errors](/agents/deploy/web-sdk#errors)), `status` returns to `"idle"` and the hook is ready for another attempt:

  ```tsx theme={null}
  try {
    await conversation.startSession({ sessionToken });
  } catch (error) {
    if (error instanceof FishAgentError && error.code === "mic_permission_denied") {
      showMicHelp();
    }
  }
  ```

* **Unmount ends the call.** No leaked microphones after navigation. Keep the component mounted for the call's duration; lift it up, or use the [provider](#share-one-session-across-components), if the page around it changes.

* The message senders (`sendUserMessage`, `sendUserActivity`, `interrupt`) are no-ops before the session connects; they do not queue.

## Share one session across components

Wrap your tree in `AgentSessionProvider` when several components need the same conversation: call controls in the header, a transcript panel elsewhere. The provider hosts one `useConversation(options)`, so its `options` prop takes the hook's `defaults`.

```tsx App.tsx theme={null}
import {
  AgentSessionProvider,
  useAgentSessionContext,
  useAgentMessages,
  AgentAudioVisualizer,
} from "@fishaudio/agent-react";

export function App() {
  return (
    <AgentSessionProvider options={{ agentId: "YOUR_AGENT_ID" }}>
      <CallControls />
      <Transcript />
      <AgentAudioVisualizer bars={24} />
    </AgentSessionProvider>
  );
}

function CallControls() {
  const { status, startSession, endSession } = useAgentSessionContext();
  return status === "connected" ? (
    <button onClick={() => endSession()}>Hang up</button>
  ) : (
    <button onClick={() => startSession()}>Start call</button>
  );
}

function Transcript() {
  const messages = useAgentMessages();

  return (
    <ul>
      {messages.map(m => (
        <li key={m.key}>
          <strong>{m.role === "agent" ? "Agent" : "You"}</strong>: {m.text}
        </li>
      ))}
    </ul>
  );
}
```

Components inside the provider read the shared state with `useAgentSessionContext()` instead of calling `useConversation` themselves. It returns the same fields, so the quick-start `CallButton` only needs its hook call swapped.

| Hook | Purpose |
| - | - |
| `useAgentSessionContext()` | The nearest provider's conversation (the same value `useConversation` returns). Throws outside a provider |
| `useOptionalAgentSessionContext()` | Nullable variant, for components that also work standalone |

The session-consuming hooks and components (`useAgentMessages`, `useAudioLevels`, `<AgentAudioVisualizer />`) all resolve their session the same way: pass one explicitly (`useAgentMessages(session)`) or omit the argument to use the surrounding provider's. Passing `null` explicitly means "no session"; it does not fall back to the provider.

## Chat log with `useAgentMessages`

`useAgentMessages(session?)` returns the conversation as a live list of `ChatMessage` entries in order, aggregated from the session's transcript events so you do not have to reduce them yourself:

| Field | Type | Description |
| - | - | - |
| `key` | `string` | Stable per-turn key; use it as the React list key |
| `role` | `"user" \| "agent"` | Who is speaking |
| `text` | `string` | The whole turn so far (not a delta) |
| `final` | `boolean` | `false` while the turn is still streaming |

* A user turn appears as soon as transcription starts and its `text` is refined in place; always re-render from `text`, never append.
* An agent turn grows with each response delta and flips to `final: true` when the turn completes.
* Both directions update the existing entry (matched by `key`), so list order and keys stay stable across renders.
* A component mounted mid-session starts from `session.getTranscript()`, so turns that streamed before it subscribed still appear, without duplicates.
* The log resets when a new session starts. Persist it yourself (for example into app state on `disconnect`) if you need history across calls.
* Events dropped during a reconnection are not replayed, so the log may skip turns lost to an outage.

```tsx Voice and text chat panel theme={null}
import { useState } from "react";
import { useAgentMessages, useAgentSessionContext } from "@fishaudio/agent-react";

function ChatPanel() {
  const { status, sendUserMessage, sendUserActivity } = useAgentSessionContext();
  const messages = useAgentMessages();
  const [draft, setDraft] = useState("");

  const submit = (event: React.FormEvent) => {
    event.preventDefault();
    if (!draft.trim()) return;
    sendUserMessage(draft); // shows up in `messages` like a spoken turn
    setDraft("");
  };

  return (
    <div>
      <ul>
        {messages.map(message => (
          <li key={message.key} style={{ opacity: message.final ? 1 : 0.6 }}>
            <b>{message.role}:</b> {message.text}
          </li>
        ))}
      </ul>
      <form onSubmit={submit}>
        <input
          value={draft}
          disabled={status !== "connected"}
          onChange={event => {
            setDraft(event.target.value);
            sendUserActivity(); // hold the agent back while the user types
          }}
        />
        <button type="submit">Send</button>
      </form>
    </div>
  );
}
```

If you only want finalized messages in order, without streaming updates, subscribe to the session's [`message` event](/agents/deploy/web-sdk#events) directly instead.

## Audio levels and visualizer

`useAudioLevels(session?, fps?)` returns polled `{ input, output }` levels in `0` to `1`: `input` is the microphone, `output` is the agent's voice. It samples 20 times per second by default; raise `fps` for snappier meters, lower it to reduce re-renders. Both are `0` before the session connects.

```tsx Microphone meter theme={null}
function MicMeter() {
  const { input } = useAudioLevels();
  return (
    <div style={{ width: 120, background: "#eee" }}>
      <div style={{ width: `${input * 100}%`, height: 8, background: "#22c55e" }} />
    </div>
  );
}
```

`<AgentAudioVisualizer />` renders animated canvas frequency bars driven by the agent's output audio, a drop-in "the agent is talking" indicator. Inside an `AgentSessionProvider` it picks up the active session automatically; elsewhere, pass a `session` prop. The bars are colored by the canvas's CSS `color`, so it themes like text.

```tsx theme={null}
import { AgentAudioVisualizer } from "@fishaudio/agent-react";

function CallScreen() {
  return <AgentAudioVisualizer bars={32} width={320} height={80} className="agent-bars" />;
}
```

| Prop | Type | Default | Description |
| - | - | - | - |
| `session` | `AgentSession \| null` | provider's | Explicit session, or omit to use the provider's |
| `bars` | `number` | `24` | Number of frequency bars |
| `width` / `height` | `number` | `240` / `64` | Canvas size in pixels |
| `className` | `string` | | Passed to the `<canvas>`; set `color` through it to theme the bars |

For a fully custom visualization, build on `useAudioLevels` or the session's frequency-data methods from the [Web SDK](/agents/deploy/web-sdk#audio-controls).

## Going further

<CardGroup>
  <Card title="Web SDK" icon="js" href="/agents/deploy/web-sdk">
    Full option, event, and method reference behind these hooks.
  </Card>

  <Card title="Client tools" icon="wrench" href="/agents/build/client-tools">
    Let the agent call functions in your app.
  </Card>

  <Card title="Authenticated sessions" icon="server" href="/agents/deploy/authenticated-sessions">
    Create session tokens server-side for private agents.
  </Card>

  <Card title="Dynamic variables" icon="brackets-curly" href="/agents/build/dynamic-variables">
    Personalize each session at start time.
  </Card>
</CardGroup>


# Versions & Publishing
Source: https://docs.fish.audio/agents/deploy/versions-publishing

Edit safely in a draft, then publish an immutable version that live sessions run

Every agent has one **draft** and a linear history of **published versions**. Edits in the Builder or via the API always land in the draft. Production sessions never see them until you publish. Publishing snapshots the draft into an immutable, numbered version.

<CardGroup>
  <Card title="Edit the draft" icon="sliders" href="/agents/build/configuration">
    Prompt, voice, and conversation settings, all autosaved.
  </Card>

  <Card title="Test before publishing" icon="phone" href="/agents/test/preview-calls">
    Preview calls talk to the draft, not the live version.
  </Card>

  <Card title="Automate via API" icon="brackets-curly" href="/api-reference/endpoint/agent/publish-agent">
    Manage drafts, publishing, and version history programmatically.
  </Card>
</CardGroup>

## Draft vs. published

| | Draft | Published version |
| - | - | - |
| Mutability | Editable: Builder and `PATCH .../config` write here | Immutable snapshot |
| Used by | [Preview calls](/agents/test/preview-calls) and [agent tests](/agents/test/agent-tests) | Production sessions: web, SDK, and phone |
| History | One per agent, always current | `version_number` increments with each publish |

<Note>
  An agent that has never been published cannot take production sessions:
  session creation returns `409` until the first publish.
</Note>

Phone number settings are not part of the agent, so they are not versioned. Which agent a number is bound to, its [transfer caller ID](/agents/telephony/transfers#caller-id-on-transfers), and an [imported number's SIP configuration](/agents/telephony/byo-sip#update-the-configuration) live on the number. Changes to them apply from the next call without publishing, and publishing or rolling back the agent never changes them.

## Publish from the Builder

<Steps>
  <Step title="Edit: changes autosave">
    Every change is written to the draft automatically;
    the **Saving… / Draft saved** indicator in the top bar shows the current
    state.
  </Step>

  <Step title="Watch the unpublished-changes indicator">
    When the draft differs from the last published version, the **Publish**
    button becomes active with an unpublished-changes hint next to it. When
    there is nothing new to publish, the button is disabled. An agent that has
    never been published always counts as having changes.
  </Step>

  <Step title="Publish">
    Click **Publish**. Any pending edits are saved first, then the draft is
    snapshotted as the next version. A confirmation shows the new version
    number, and the agent shows as **Live** in your agents list.
  </Step>
</Steps>

New sessions use the newly published version from that point on. To hear a change before it goes live, run a [preview call](/agents/test/preview-calls). Previews always use the draft.

## Publish via the API

Configuration edits (`PATCH`) go to the draft; a separate `publish` call rolls them out. A `version_title` and `version_description` are optional and make the version history easier to audit:

```bash Update the draft theme={null}
curl --request PATCH "https://api.fish.audio/v1/agent/agents/$AGENT_ID/config" \
  --header "Authorization: Bearer $FISH_API_KEY" \
  --header "Content-Type: application/json" \
  --data '{
    "prompt": { "system_prompt": "You are a concise support agent for Acme." }
  }'
```

```bash Publish theme={null}
curl --request POST "https://api.fish.audio/v1/agent/agents/$AGENT_ID/publish" \
  --header "Authorization: Bearer $FISH_API_KEY" \
  --header "Content-Type: application/json" \
  --data '{ "version_title": "Tighten support tone" }'
```

The publish response includes the new `version_number`.

## Version history

List published versions, or fetch the complete configuration snapshot of any one of them, including the version currently serving production:

```bash List versions theme={null}
curl "https://api.fish.audio/v1/agent/agents/$AGENT_ID/versions" \
  --header "Authorization: Bearer $FISH_API_KEY"
```

```bash Get one version's snapshot theme={null}
curl "https://api.fish.audio/v1/agent/agents/$AGENT_ID/versions/3" \
  --header "Authorization: Bearer $FISH_API_KEY"
```

Each version record includes:

| Field | Description |
| - | - |
| `version_number` | Auto-incrementing, unique per agent |
| `version_title` | Optional label sent when publishing |
| `version_description` | Optional description sent when publishing |
| `config_hash` | Fingerprint of the configuration snapshot |
| `published_at` | When the version was published |

Fetching a single version returns the full configuration as it was published. Because `GET .../config` always shows the draft, the snapshot endpoint is how you read back what is actually running in production.

<Warning>
  Version snapshots never echo credential secrets. Write-only fields (such as
  webhook secrets and tool authorization headers) return a `has_secret: true`
  marker instead of the value, and the custom LLM
  [`api_key`](/agents/build/custom-llm) reads back as `null`, in every version
  including historical ones.
</Warning>

## Restore a previous version

Roll back by copying an old snapshot back into the draft: read the version you want, then `PATCH` its sections into the draft config.

```bash 1. Read the old snapshot theme={null}
curl "https://api.fish.audio/v1/agent/agents/$AGENT_ID/versions/2" \
  --header "Authorization: Bearer $FISH_API_KEY"
```

```bash 2. Patch it back into the draft theme={null}
curl --request PATCH "https://api.fish.audio/v1/agent/agents/$AGENT_ID/config" \
  --header "Authorization: Bearer $FISH_API_KEY" \
  --header "Content-Type: application/json" \
  --data '{ "prompt": { "system_prompt": "<system prompt from the snapshot>" } }'
```

Patching updates the draft. It does **not** publish. Review or [preview](/agents/test/preview-calls) the restored draft, then publish it as a new version; the rollback becomes part of the linear history, so the audit trail stays intact. Because snapshots never echo secrets (see above), credential fields keep their current draft values unless you set them again explicitly. Phone number settings, such as the transfer caller ID, stay as they are.

## Clone an agent

Cloning creates a new agent from an existing one's **draft** configuration. In the console, open the agents list and choose **Clone** from the agent's row menu.

The clone starts with a copy of the source's current draft. Published versions do not carry over: the new agent is unpublished until you publish it yourself. Use clones to template a base configuration across many agents, or to experiment without risking an agent that is already live.

## Going further

<CardGroup>
  <Card title="Agent configuration" icon="sliders" href="/agents/build/configuration">
    Everything that lives in the draft: prompt, voice, conversation settings.
  </Card>

  <Card title="Preview calls" icon="phone" href="/agents/test/preview-calls">
    Talk to the draft before it ships.
  </Card>

  <Card title="Agent tests" icon="vial" href="/agents/test/agent-tests">
    Run regression tests against the draft, then publish with confidence.
  </Card>

  <Card title="Integration overview" icon="plug" href="/agents/deploy/overview">
    Connect published agents to your product via SDK and API.
  </Card>
</CardGroup>


# Web SDK
Source: https://docs.fish.audio/agents/deploy/web-sdk

Voice sessions in the browser with @fishaudio/agent-client: options, events, transcripts, text input, audio controls, client tools, and errors

`@fishaudio/agent-client` runs a live voice conversation with your agent from any web page: open the microphone, stream audio both ways, and react to typed events for transcripts, agent state, and tool calls. The SDK handles the realtime transport (WebRTC) internally.

Using React? [`@fishaudio/agent-react`](/agents/deploy/react-sdk) wraps this SDK in hooks and a provider.

## Install

<CodeGroup>
  ```bash npm theme={null}
  npm install @fishaudio/agent-client
  ```

  ```bash pnpm theme={null}
  pnpm add @fishaudio/agent-client
  ```

  ```bash yarn theme={null}
  yarn add @fishaudio/agent-client
  ```
</CodeGroup>

## Start a session

`AgentSession.start()` creates the session, connects, and opens the microphone in one call. Authenticate one of two ways:

* **`agentId`**: for [public agents](/agents/deploy/public-agents). The SDK creates the session directly from the browser; no backend needed.
* **`sessionToken`**: for private agents. Your backend calls `POST /v1/agent/sessions` with your API key and hands the JSON response to the browser; pass it through unchanged. See [Authenticated sessions](/agents/deploy/authenticated-sessions).

<CodeGroup>
  ```javascript Public agent theme={null}
  import { AgentSession } from "@fishaudio/agent-client";

  const session = await AgentSession.start({
    agentId: "YOUR_AGENT_ID",
  });
  ```

  ```javascript Private agent theme={null}
  import { AgentSession } from "@fishaudio/agent-client";

  // Your backend calls POST /v1/agent/sessions and returns the
  // response body. Pass it to the SDK as-is.
  const resp = await fetch("/api/voice-session", { method: "POST" });
  const sessionToken = await resp.json();

  const session = await AgentSession.start({ sessionToken });
  ```
</CodeGroup>

`start()` resolves once the realtime connection is up and rejects with a [`FishAgentError`](#errors) if session creation, the microphone permission, or the connection fails. Call it from a user gesture (a click handler): browsers only grant the microphone and audio playback inside one.

### Options

| Option | Type | Description |
| - | - | - |
| `agentId` / `sessionToken` | `string` / `SessionToken` | Exactly one is required: a public agent id, or the JSON your backend received from `POST /v1/agent/sessions` |
| `serverUrl` | `string` | API base used in `agentId` mode. Default `https://api.fish.audio` |
| `clientTools` | `Record<string, ClientToolHandler>` | Handlers for [client tools](/agents/build/client-tools) declared on the agent |
| `clientToolTimeoutMs` | `number` | Per-handler timeout for client tools, in milliseconds. Default `15000` |
| `overrides` | `SessionOverrides` | Per-session config [overrides](/agents/deploy/authenticated-sessions#overrides) (`agentId` mode only; with `sessionToken`, your backend sends them when creating the session). Keyless sessions accept only `voice_id` and `language`; the prompt-shaping fields are rejected with `400` |
| `dynamicVariables` | `Record<string, string \| number \| boolean>` | Values for `{{placeholders}}` in the agent config; see [Dynamic variables](/agents/build/dynamic-variables) (`agentId` mode only) |
| `language` | `SessionLanguage`, the code of one of the [52 supported languages](/agents/build/voice-language#speaking-language) | Shorthand |
| for `overrides.language` (`agentId` mode only). Omit it to use the agent's configured [speaking language](/agents/build/voice-language#speaking-language); any other value is rejected with `422` | | |
| `toolEvents` | `boolean` | Whether tool lifecycle events reach this client (default `true`; `agentId` mode only, with `sessionToken` your backend sets `tool_events`) |
| `timezone` | `string` | IANA timezone for the agent's sense of local time: the top of the [resolution order](/agents/build/time-timezone), overriding the agent's configured timezone. When omitted, the SDK still sends the browser timezone as a lower-priority hint (`client_timezone`), which applies only if the agent has no timezone configured |
| `worldContext` | `boolean` | Whether the agent knows the current date and time (default `true`; `agentId` mode only, with `sessionToken` your backend sets `world_context`) |
| `endUserId` | `string` | Your identifier for the end user, stored on the session and echoed in [webhooks](/agents/monitor/webhooks) (`agentId` mode only; with `sessionToken` your backend sets `end_user_id`) |
| `metadata` | `object` | Your own key-values, stored on the session record and returned verbatim (`agentId` mode only; with `sessionToken` your backend sets `metadata`) |
| `microphone` | `boolean` | Capture the microphone on start (default `true`). `false` joins muted with no permission prompt, for text-first UIs; the first `setMicMuted(false)` captures it, so call that from a user gesture |
| `audio` | `{ inputDeviceId?, outputDeviceId? }` | Pick specific microphone and output devices. `start()` rejects with `device_change_failed` if the output device cannot be selected |
| `wakeLock` | `boolean` | Hold a screen wake lock while the session is live, so long calls survive the phone trying to sleep (default `true`). A denied or unsupported wake lock is silent |
| `callbacks` | `Partial<AgentSessionCallbacks>` | Shorthand for `.on()`: each `onXxx` key subscribes the `xxx` [event](#events), so `onUserTranscript` subscribes `userTranscript`. From 0.2.1, a key that is not a known event name throws a `TypeError` |

```javascript Callbacks shorthand theme={null}
const session = await AgentSession.start({
  sessionToken,
  callbacks: {
    onUserTranscript: ({ text, final }) => renderUserBubble(text, final),
    onAgentResponseDelta: ({ text }) => renderAgentBubble(text),
    onModeChange: mode => setOrbState(mode),
    onDisconnect: ({ reason }) => showCallEnded(reason),
    onError: error => console.error(error.code),
  },
});
```

## Session lifecycle

A session moves through a fixed state machine, surfaced by the `statusChange` event and the `session.status` property:

```text States theme={null}
connecting → connected ⇄ reconnecting → ended(reason)
      └───────────(failure)───────────→ ended
```

`start()` resolves once the realtime connection is up; the agent itself joins moments later. If it has not joined within 15 seconds, the session emits a `connection_failed` error and ends with reason `connection_lost` instead of idling on a dead call.

When the session ends, `disconnect` fires with a reason (also available as `session.endReason`):

| `EndReason` | Meaning |
| - | - |
| `user_hangup` | You called `session.end()`, the user left the page, or your backend [ended the session](/agents/deploy/authenticated-sessions#ending-sessions-from-your-backend) |
| `agent_hangup` | The agent ended the call: the hang-up [system tool](/agents/build/system-tools), a workflow end node, or an idle hangup |
| `conversation_timeout` | The session reached the agent's [maximum duration](/agents/build/configuration#call-duration) |
| `escalated` | The call was transferred to a human |
| `connection_lost` | The connection dropped and could not be recovered, the agent never joined, or the call ended before the server could say why |

Every reason except `connection_lost` comes from the server: it sends the protocol's [`session.ended` event](/agents/deploy/protocol#session-ended) just before tearing the call down, and the values match `end_reason` on the [session record](/agents/monitor/conversation-history#status-and-end-reason), so your client and your backend agree about how a session ended. This needs `@fishaudio/agent-client` 0.3.0 or later; older clients infer the reason from the disconnect and report every server-side end as `agent_hangup`.

Brief network drops don't end the session: the SDK moves to `reconnecting` and back to `connected` automatically, reusing the same session. It never creates a new session, so you never need a new token mid-call.

<Warning>
  Events are not replayed after a reconnect. Anything emitted while you were in
  `reconnecting` (transcript updates, tool events, errors) is dropped, not
  resent. Design your UI to tolerate gaps: tool `toolCallCompleted` /
  `toolCallFailed` events repeat the tool name and source, so a terminal event
  still renders even if you missed `toolCallStarted`.
</Warning>

The session also exposes `sessionId` (use it to look up the [conversation record](/agents/monitor/conversation-history) later), `status`, `mode`, `isSpeaking`, `micMuted`, and `endReason` as properties.

## Events

The session is a typed event emitter: subscribe with `session.on(event, handler)`, remove with `off`, or use `once`.

| Event | Payload | Fires when |
| - | - | - |
| `connect` | `{ sessionId }` | The session is connected and live |
| `disconnect` | `{ reason: EndReason }` | The session has ended |
| `statusChange` | `SessionStatus` | Status transitions (`connecting` / `connected` / `reconnecting` / `ended`) |
| `modeChange` | `AgentMode` | The agent switches between `listening`, `thinking`, and `speaking` |
| `userTranscript` | `{ segmentId, text, final }` | The user's speech is transcribed; interim updates replace the whole segment |
| `agentResponseDelta` | `{ segmentId, delta, text }` | The agent speaks: `delta` is the new text, `text` the segment so far |
| `agentResponse` | `{ segmentId, text }` | An agent segment is finalized |
| `message` | `{ role: "user" \| "agent", text }` | A finalized message from either side, a ready-made chat feed |
| `toolCallStarted` | `{ callId, toolName, source, input, inputTruncated }` | A tool call begins; `input` is a JSON string (truncated at 4 KB) |
| `toolCallCompleted` | `{ callId, toolName, source, output, outputTruncated }` | A tool call succeeds |
| `toolCallFailed` | `{ callId, toolName, source, error }` | A tool call fails |
| `error` | `FishAgentError` | A session or tool error occurs; see [Errors](#errors) |

```javascript Subscribe to events theme={null}
session.on("userTranscript", ({ segmentId, text, final }) => {
  upsertBubble("user", segmentId, text, final);
});

session.on("agentResponseDelta", ({ segmentId, text }) => {
  upsertBubble("agent", segmentId, text, false);
});

session.on("modeChange", mode => setOrbState(mode));
session.on("disconnect", ({ reason }) => showCallEnded(reason));
```

### Transcript semantics

Transcripts on both sides arrive as **segments**, one segment per utterance or response, identified by `segmentId`:

* **User segments**: interim results **replace the entire segment text** (they never append). Render by upserting on `segmentId`; `final: true` marks the segment as finalized.
* **Agent segments**: text streams in sync with audio playback: what you display matches what the user has actually heard. If the agent is interrupted, the segment finalizes containing only the words that were spoken. A reply that spans several speech segments (for example around a tool call) arrives as several segments.
* The `message` event delivers only finalized messages from both sides, in order. Use it when you want a simple transcript list without handling interim updates.

A listener attached late misses the events fired before it subscribed. `session.getTranscript()` returns every segment so far, as `{ segmentId, role, text, final }` in conversation order: seed your UI from it, then keep applying live events by `segmentId`.

### Tool call lifecycle

Every tool the agent runs (webhook, client, or system tool) emits one `toolCallStarted`, resolved by exactly one `toolCallCompleted` or `toolCallFailed` with the same `callId`. `input` and `output` are JSON strings truncated at 4 KB; parse them only when the matching `*Truncated` flag is `false`. These events carry tool arguments and results into the end user's browser: to keep them off the client, start with `toolEvents: false`, or create the session with `tool_events: false` on your backend.

## Agent modes

`session.mode` (and the `modeChange` event) tracks what the agent is doing, for driving an orb or status indicator:

| Mode | Meaning |
| - | - |
| `listening` | Default state: the agent is waiting for or hearing the user |
| `thinking` | The agent is preparing a response |
| `speaking` | Agent audio is playing; ends when playback finishes |

`session.isSpeaking` is a convenience boolean for the `speaking` mode. Mode does not react to the user's own speech. For instant "the mic hears you" feedback, poll `getInputVolume()` locally.

## Send text

Users can type instead of talking, in the same session:

```javascript Text input theme={null}
// Send a typed user turn. There is no server echo; the SDK emits
// the finalized `message` event locally from the text you passed.
// By default the agent answers a typed turn in text only
// (agentResponseDelta / agentResponse) without speaking out loud.
session.sendUserMessage("Do you ship to Norway?");

// Pass `audio: true` to have the agent speak its reply for this turn.
session.sendUserMessage("What's my order status?", { audio: true });

// Signal typing so the agent doesn't talk over the user.
input.addEventListener("input", () => session.sendUserActivity());

// Stop the agent's current speech immediately (explicit barge-in).
session.interrupt();
```

Text-only replies are not paced to audio playback: the transcript streams as fast as it generates. Messages sent right after `start()` resolves, before the agent has finished joining, are held and delivered in order once it is ready.

## Audio controls

| Method / property | Description |
| - | - |
| `setMicMuted(muted)` / `micMuted` | Mute or unmute the microphone. After a `microphone: false` start, the first unmute captures the microphone (permission prompt, so call it from a user gesture) and rejects with `mic_permission_denied` if refused |
| `setOutputVolume(v)` | Set playback volume, `0`–`1` |
| `getInputVolume()` / `getOutputVolume()` | Current mic / agent volume, `0`–`1`; poll per frame |
| `getInputFrequencyData()` / `getOutputFrequencyData()` | FFT data as `Uint8Array`, for visualizers |
| `startAudio()` | Unlock playback under browser autoplay policies; call inside a user gesture (for example the click that starts the call) |
| `setInputDevice(deviceId)` | Switch the microphone mid-call. Rejects with `device_change_failed` if the device cannot be activated, switching back to the previous microphone (best effort) |
| `setOutputDevice(deviceId)` | Route playback to another output device (`""` for the default). Rejects with `device_change_failed` where the browser does not support output selection (common on mobile); playback stays on the previous device |

Device ids come from `navigator.mediaDevices.enumerateDevices()`. To pick devices before the call starts, pass the `audio` option (`inputDeviceId` / `outputDeviceId`) to `start()` instead.

```javascript Audio visualizer theme={null}
function draw() {
  const fft = session.getOutputFrequencyData(); // Uint8Array
  renderBars(canvas, fft);
  rafId = requestAnimationFrame(draw);
}
draw();

// Stop drawing when the session ends.
session.on("disconnect", () => cancelAnimationFrame(rafId));
```

## Client tools

Register handlers for tools of type `client` declared on the agent. The agent calls them mid-conversation and your return value goes back to the model:

```javascript Register a client tool theme={null}
const session = await AgentSession.start({
  agentId: "YOUR_AGENT_ID",
  clientTools: {
    open_page: async params => {
      showPanel(String(params.page));
      return { opened: true };
    },
  },
});

// Or after start:
session.registerClientTool("highlight_product", handler);
```

Handlers can be sync or async; a thrown error or a timeout (default 15 s, set with `clientToolTimeoutMs`) is returned to the agent as a tool error. See [Client tools](/agents/build/client-tools) for declaration, naming rules, result size, and dispatch semantics.

## Errors

Failures surface as `FishAgentError`: thrown from `AgentSession.start()` when the session can't be created, emitted on the `error` event otherwise. Each carries a `code`, an optional `statusCode` (set on session-creation HTTP errors), and `cause`.

| Code | Meaning |
| - | - |
| `session_request_failed` | Session creation failed; check `statusCode` |
| `agent_not_public` | `agentId` mode, but the agent is not enabled for public access |
| `origin_forbidden` | The page's origin is not in the agent's allowed origins |
| `unsupported_transport` | The session token uses a transport this SDK version doesn't know; upgrade the SDK |
| `mic_permission_denied` | The user denied microphone access; the SDK ends the session |
| `device_change_failed` | A requested audio device could not be activated, or the browser doesn't support selecting it (output selection is unsupported on some mobile browsers). Thrown from `start()` with `audio.outputDeviceId` set, or from device-switching calls |
| `connection_failed` | The realtime connection could not be established or recovered, a message to the agent could not be sent, the agent never joined (15 s), or the microphone could not be toggled |
| `session_expired` | The session token's join deadline passed before connecting |
| `tool_failed` / `tool_timeout` | A client tool handler threw, was not registered, or its result could not be delivered / a handler exceeded `clientToolTimeoutMs` |
| `provider_error` / `internal_error` | The session failed server-side: upstream model/voice provider vs. platform runtime. Emitted on `error`; the session may recover |

<Note>
  `provider_error` and `internal_error` carry only the category code. Raw
  provider or infrastructure details are never sent to the browser.
</Note>

```javascript Handle errors theme={null}
import { AgentSession, FishAgentError } from "@fishaudio/agent-client";

try {
  const session = await AgentSession.start({ agentId: "YOUR_AGENT_ID" });
  session.on("error", err => console.warn("session error:", err.code));
} catch (err) {
  if (err instanceof FishAgentError && err.code === "mic_permission_denied") {
    showMicHelp();
  } else {
    showRetry(err);
  }
}
```

## End the session

```javascript Hang up theme={null}
await session.end(); // graceful hangup; disconnect fires with reason "user_hangup"
```

Calling `end()` more than once, or while a previous call is still finishing, is safe. `endReason` stays readable on the ended session.

## Underlying connection

The session API is transport-neutral: the server picks the transport when it creates the session, `livekit` (WebRTC) today. For needs the session API does not cover yet, such as connection-quality telemetry, `session.getRoom()` returns the live [livekit-client `Room`](https://docs.livekit.io/reference/client-sdk-js/), or `undefined` once the session has ended. Prefer the session API where one exists: code built on the `Room` couples to this SDK's transport choice and its pinned livekit-client major, with no compatibility promise across SDK versions. See [Wire protocol](/agents/deploy/protocol) for the messages underneath.

## Going further

<CardGroup>
  <Card title="React SDK" icon="react" href="/agents/deploy/react-sdk">
    Hooks, provider, and visualizer components on top of this SDK.
  </Card>

  <Card title="Authenticated sessions" icon="server" href="/agents/deploy/authenticated-sessions">
    Create session tokens on your backend for private agents.
  </Card>

  <Card title="Public agents" icon="globe" href="/agents/deploy/public-agents">
    Backend-free sessions with origin allow-lists and rate limits.
  </Card>

  <Card title="Wire protocol" icon="tower-broadcast" href="/agents/deploy/protocol">
    The wire-level events underneath the SDK.
  </Card>
</CardGroup>


# Widget
Source: https://docs.fish.audio/agents/deploy/widget

Embed a complete voice and chat UI on any website with two lines of HTML, no build step

The widget is the zero-code way to put your agent on a website. A `<fish-agent>` custom element renders the complete experience (a floating launcher that expands into a voice-first chat card with live transcript, typing during the call, and inline tool activity), and a single script tag registers it. Under the hood it runs the same sessions as the [Web SDK](/agents/deploy/web-sdk), so everything downstream (history, analysis, webhooks) works unchanged.

## Prerequisites

* An agent with a [published version](/agents/deploy/versions-publishing).
* **Public access** enabled on the agent, with your site's origin on the allowed-origins list. See [Public agents](/agents/deploy/public-agents). `localhost` and `127.0.0.1` count as different origins.
* To keep the agent private instead, skip public access and supply session tokens from your backend with [`sessionTokenProvider`](#private-agents).

## Two-line embed

Add the element and the script anywhere on the page:

```html theme={null}
<fish-agent agent-id="your-agent-id"></fish-agent>
<script
  src="https://unpkg.com/@fishaudio/agent-widget-embed"
  async
  type="text/javascript"
></script>
```

`@fishaudio/agent-widget-embed` is the widget pre-bundled as one IIFE file that registers `<fish-agent>` on load.

## Install from npm

Bundlers can install the element instead: `npm install @fishaudio/agent-widget`, then call `registerWidget()` once. Importing the package has no side effects.

```javascript theme={null}
import { registerWidget } from "@fishaudio/agent-widget";
registerWidget(); // defines <fish-agent>
```

### React

React apps get a real component: `<FishAgentWidget>` registers and renders the element with camelCase props, object props serialized for you, and the [page events](#page-events) as callback props. `clientTools` is just a prop:

```tsx theme={null}
import { FishAgentWidget } from "@fishaudio/agent-widget/react";

<FishAgentWidget
  agentId="your-agent-id"
  clientTools={{
    highlight_product: ({ product_id }) => scrollToProduct(product_id),
  }}
  onConnect={({ sessionId }) => console.log(sessionId)}
/>;
```

Every attribute below has a camelCase prop; `dynamicVariables` and `textContents` take objects, and `onCall(options)` still runs last for anything else. Importing the entry also types the raw `<fish-agent>` element in JSX, for pages that use the CDN script and install the package only for its types.

## Attributes

| Attribute | Description |
| - | - |
| `agent-id` | Public agent ID. Required unless the `sessionTokenProvider` property is set. See [Private agents](#private-agents). |
| `agent-name` | Display name in the header. |
| `greeting` | Home-screen headline. |
| `proactive-message` | Enables the attention bubble next to the launcher. |
| `proactive-delay` | Seconds before the bubble shows. Default `3`. |
| `transcript` / `text-input` / `mic-muting` | Feature switches, on by default; set `"false"` to disable. |
| `consent` | `"true"` shows a first-run terms card (default off). Acceptance is remembered in `localStorage`. |
| `consent-text`, `terms-url`, `privacy-url`, `consent-key` | Consent copy, linked policies, and the `localStorage` key (default `fish-agent-consent`). |
| `position` | `bottom-right` (default), `bottom-left`, `top-right`, `top-left`. |
| `language` | Pin the session language. See [Overrides](/agents/deploy/authenticated-sessions#overrides). |
| `dynamic-variables` | JSON object of `{{name}}` template values. See [Dynamic variables](/agents/build/dynamic-variables). |
| `user-id` | Your end-user identifier, stored on the session. |
| `server-url` | Fish API base override. Default `https://api.fish.audio`. |
| `text-contents` | JSON overriding any UI string (keys in `DEFAULT_TEXTS` of `@fishaudio/agent-widget`). |

## Private agents

Keep the agent non-public and set `sessionTokenProvider` instead of an `agent-id`. It's a JS property on the element (functions can't be attributes), called before every session start: fetch the session token from your backend (with whatever auth headers, payload, or credentials the request needs) and return the JSON; it's used verbatim.

```html theme={null}
<fish-agent agent-name="Support"></fish-agent>
<script>
  document.querySelector("fish-agent").sessionTokenProvider = async () => {
    const response = await fetch("/api/voice-session", {
      method: "POST",
      headers: { Authorization: `Bearer ${appSession.token}` },
    });
    return response.json();
  };
</script>
```

React apps pass the same function as a prop:

```tsx theme={null}
<FishAgentWidget sessionTokenProvider={getSessionToken} />
```

Your backend holds the API key and creates the session with `POST /v1/agent/sessions`; origin checks, user auth, and rate limiting on that endpoint are yours. See [Authenticated sessions](/agents/deploy/authenticated-sessions) for the token flow and a backend example.

## Theming

Set CSS custom properties on the element. The widget's internals live in a shadow root: page CSS can't leak in, but every `--fish-*` token is public:

```css theme={null}
fish-agent {
  --fish-accent: #7c3aed;
  --fish-orb-color-1: #c4b5fd;
  --fish-orb-color-2: #4c1d95;
  --fish-radius: 16px;
  --fish-offset-x: 32px;
  --fish-offset-y: 32px;
  --fish-z-index: 999999;
}
```

| Property | Controls |
| - | - |
| `--fish-accent` | Launcher and primary button color |
| `--fish-orb-color-1` / `--fish-orb-color-2` | The voice orb's gradient colors |
| `--fish-radius` | Corner radius of the panel |
| `--fish-offset-x` / `--fish-offset-y` | Distance from the viewport edges |
| `--fish-z-index` | Stacking order on your page |

Also available: `--fish-accent-text`, `--fish-bg`, `--fish-text`, `--fish-text-secondary`, `--fish-border`, `--fish-bubble-agent-bg/-text`, `--fish-bubble-user-bg/-text`, `--fish-danger`, `--fish-live`, `--fish-fab-size`, `--fish-font`.

## Page events

The element dispatches `CustomEvent`s (bubbling, composed):

| Event | `detail` | When |
| - | - | - |
| `fish-agent:call` | `{ options }`: **mutable** `AgentSession.start` options | Right before a session starts. Mutate `detail.options` to inject `clientTools`, `overrides`, anything. |
| `fish-agent:connect` | `{ sessionId }` | Session established. |
| `fish-agent:disconnect` | `{ reason }` | Session ended. |
| `fish-agent:error` | `{ code, message }` | Start failure or in-call error. |

Inbound: dispatch `fish-agent:expand` on the element or `document` to open the panel programmatically.

Registering [client tools](/agents/build/client-tools) is just the `:call` event (React apps pass the `clientTools` prop instead; same injection, wrapped):

```javascript theme={null}
document
  .querySelector("fish-agent")
  .addEventListener("fish-agent:call", event => {
    event.detail.options.clientTools = {
      highlight_product: ({ product_id }) => scrollToProduct(product_id),
    };
  });
```

## Console settings and precedence

On load, the widget anonymously fetches the agent's console widget settings from `GET /v1/agent/agents/{agent_id}/widget`, the same public-plus-origin gate as session creation, so it only answers for public agents to allowed origins. Settings resolve with a fixed precedence: **HTML attribute > console widget config > built-in default**. The endpoint being unreachable never breaks the widget: it renders from attributes and defaults.

## Going further

<CardGroup>
  <Card title="Public agents" icon="globe" href="/agents/deploy/public-agents">
    The public switch, origin allowlist, and rate limits behind the widget.
  </Card>

  <Card title="Authenticated sessions" icon="server" href="/agents/deploy/authenticated-sessions">
    The session-token flow your `sessionTokenProvider` implements.
  </Card>

  <Card title="Web SDK" icon="js" href="/agents/deploy/web-sdk">
    The `AgentSession` API underneath the widget, for building your own UI.
  </Card>

  <Card title="Client tools" icon="code" href="/agents/build/client-tools">
    Let the agent trigger actions on the embedding page.
  </Card>
</CardGroup>


# Conversation History
Source: https://docs.fish.audio/agents/monitor/conversation-history

List past sessions, replay the full timeline of messages and tool calls, and download recordings

Every production session your agents handle (from the API, the console, [public agents](/agents/deploy/public-agents), or [phone calls](/agents/telephony/inbound-calls)) is queryable over REST: a lightweight list for browsing, a merged timeline of messages and tool activity per session, and per-speaker recordings when the agent [records audio](#what-gets-stored). You can query a session while the call is still in progress.

This is a server-side API: authenticate with your API key. The client SDKs deliberately expose no history interface; fetch history from your backend and pass it to your frontend as needed.

## What gets stored

The session record (status, timing, caller attribution, name, and `metadata`) and the conversation transcript are always kept for production sessions. Audio recording is a per-agent choice, on by default: set `conversation.record_audio` in the console under your agent's **Settings** or in the `conversation` section of the [config API](/agents/build/configuration#configure-through-the-api), with a per-session override on the [session request](/agents/deploy/authenticated-sessions). Recording off means there is no audio to download.

The setting doesn't affect the live call: transcript events still stream to connected clients in real time.

<Warning>
  Recording defaults to on. Call-recording laws vary by jurisdiction. Make sure
  callers are informed and consent where required before going live.
</Warning>

## List sessions

`GET /v1/agent/sessions` returns session facts, newest first. The list is intentionally thin (no transcripts or tool details) so it stays fast at any volume.

<CodeGroup>
  ```bash API (curl) theme={null}
  curl "https://api.fish.audio/v1/agent/sessions?agent_id=YOUR_AGENT_ID&status=completed&page_size=50" \
    --header "Authorization: Bearer $FISH_API_KEY"
  ```

  ```python Python theme={null}
  import os
  import httpx

  resp = httpx.get(
      "https://api.fish.audio/v1/agent/sessions",
      params={"agent_id": "YOUR_AGENT_ID", "status": "completed", "page_size": 50},
      headers={"Authorization": f"Bearer {os.environ['FISH_API_KEY']}"},
  )
  sessions = resp.json()["sessions"]
  ```
</CodeGroup>

Each row carries the session facts:

```json theme={null}
{
  "sessions": [
    {
      "session_id": "…",
      "agent_id": "…",
      "agent_name": "Support agent",
      "name": "Order #4821 follow-up",
      "status": "completed",
      "end_reason": "user_hangup",
      "source": "phone",
      "caller_number": "+15551234567",
      "dialed_number": "+14155550100",
      "sip_call_id": "7f3a9c@sbc.example.com",
      "started_at": "2026-07-22T09:14:02Z",
      "ended_at": "2026-07-22T09:16:05Z",
      "duration_seconds": 123,
      "metadata": { "crm_ticket": "T-4821" }
    }
  ],
  "has_more": true,
  "next_cursor": "…"
}
```

`caller_number` and `dialed_number` are set for phone sessions only (E.164) and are `null` for web sessions. `sip_call_id` is the SIP `Call-ID` of the call's INVITE. It is `null` on web sessions and on outbound calls that were never answered. See [Match calls with your SIP records](/agents/telephony/byo-sip#match-calls-with-your-sip-records). `name` is the display name you passed when creating the session (`null` when omitted), and `metadata` echoes back whatever you attached when creating the session.

### Status and end reason

`status` says whether a conversation happened: `completed` means a participant connected and the call is over (it is billed even if the runtime ended it), `failed` means nobody ever connected, `active` means it is in progress, `pending` means the session was created and no one has connected yet, and `unknown` means the platform temporarily lost track of a live call.

`end_reason` says why it ended. It is `null` while a session is live, for a short moment after it ends, and on sessions that predate the field.

| `end_reason` | `status` | Meaning |
| - | - | - |
| `user_hangup` | `completed` | The caller hung up |
| `agent_hangup` | `completed` | The agent ended the call (end-call tool, workflow end node, or idle hangup) |
| `escalated` | `completed` | The call was transferred to a human |
| `conversation_timeout` | `completed` | The call reached its maximum duration |
| `heartbeat_timeout` | `completed` | The agent runtime stopped responding mid-call |
| `agent_join_timeout` | `completed` | The caller connected but no agent joined the call |
| `internal_error` | `completed` | The agent runtime hit an internal error |
| `llm_endpoint_failure` | `completed` | A [custom LLM](/agents/build/custom-llm) failed repeatedly and the agent hung up |
| `room_join_timeout` | `failed` | The caller never connected before the join deadline |
| `cancelled` | `failed` | The session was ended before anyone connected |
| `dial_failed` | `failed` | An outbound call was not answered; `dial_status` has the detail |

The same values appear on the [`call.ended` webhook](/agents/monitor/webhooks) and in the dashboard, so the three surfaces never disagree about a session.

### Filters

| Parameter | Behavior |
| - | - |
| `agent_id` | Sessions of a single agent |
| `status` | One of `pending`, `active`, `completed`, `failed`, `unknown`; comma-separate values to match several |
| `end_reason` | One or more of the `end_reason` values above, comma-separated |
| `caller_number` | Exact match on the caller's E.164 number; a bare number gets `+` prepended automatically |
| `sip_call_id` | Exact match on the SIP `Call-ID` of the call's INVITE, for looking up the session behind a carrier CDR entry |
| `created_after` / `created_before` | ISO 8601 timestamps |

<Note>
  `pending` sessions (a session was created but no participant ever connected)
  are excluded by default. Pass `status=pending` explicitly to see them. Preview
  calls made from the Builder never appear in this API.
</Note>

### Pagination

Two modes, mutually exclusive (combining them returns `400`):

| Mode | How | Use it for |
| - | - | - |
| Cursor | Pass the previous response's `next_cursor` as `cursor`; stop when `has_more` is `false` | Crawling, exports, syncing |
| Page | Pass `page` (1-based); the response always includes `total` | Paged UIs with jump-to-page |

`page_size` defaults to 30, maximum 100; values out of range are rejected. In cursor mode `total` is `null` unless you pass `include_total=true`. Page mode is capped at an offset of 100,000 rows (requests beyond it return `400`), and every page response still includes `next_cursor`, so you can switch to cursor crawling from any page for deep scans.

## Get a session

`GET /v1/agent/sessions/{session_id}` returns the session facts plus a single `items` timeline: messages and tool activity merged in the order they actually happened.

```bash API (curl) theme={null}
curl "https://api.fish.audio/v1/agent/sessions/SESSION_ID" \
  --header "Authorization: Bearer $FISH_API_KEY"
```

```json theme={null}
{
  "session_id": "…",
  "status": "completed",
  "end_reason": "user_hangup",
  "items": [
    {
      "type": "message",
      "role": "user",
      "content": "Can you check order A1?",
      "turn_id": 3,
      "created_at": "2026-07-22T09:14:05Z"
    },
    {
      "type": "tool_call",
      "call_id": "tool_8f2…",
      "tool_name": "lookup_order",
      "tool_source": "webhook",
      "input": "{\"order_id\":\"A1\"}",
      "created_at": "2026-07-22T09:14:06Z"
    },
    {
      "type": "tool_result",
      "call_id": "tool_8f2…",
      "tool_name": "lookup_order",
      "tool_source": "webhook",
      "status": "completed",
      "output": "{\"state\":\"shipped\"}",
      "output_truncated": false,
      "error": null,
      "latency_ms": 812,
      "created_at": "2026-07-22T09:14:07Z"
    },
    {
      "type": "message",
      "role": "assistant",
      "content": "Order A1 shipped yesterday.",
      "turn_id": 3,
      "created_at": "2026-07-22T09:14:07Z"
    }
  ],
  "analysis": {
    "status": "completed",
    "summary": "…",
    "data": [],
    "criteria_results": []
  }
}
```

### Item types

| `type` | What it is |
| - | - |
| `message` | One utterance. `role` is `user` or `assistant` (system prompt content is never exposed). `turn_id` groups items belonging to the same conversational turn. |
| `tool_call` | A tool invocation as it started. `input` is a JSON string, stored in full with no truncation. |
| `tool_result` | The paired outcome, matched by `call_id`. `status` is `completed` or `failed`. `latency_ms` is the execution time. |

Details worth knowing:

* **Role vocabulary**: the history API's `assistant` is the same speaker the [SDK's live events](/agents/deploy/web-sdk) call `agent`.
* **Order**: items are sorted by `created_at` ascending; a `tool_result`'s timestamp is its completion time, so long-running tools appear where they actually finished, with messages in between. Items sharing a timestamp order `message`, then `tool_call`, then `tool_result`.
* **`tool_source`**: where the tool ran, one of `client`, `webhook`, `builtin` (platform tools such as call transfer and hang-up), `mcp` (tools from a connected MCP server), `background` (work the agent delegated to a background task), or `unknown` (calls recorded before source attribution). Treat it as an open set. See [Tools](/agents/build/tools).
* **Payloads**: `input` and `output` are JSON strings, symmetric with the live SDK events, so one parser covers both. `output` and `error` are stored up to 256 KB; beyond that the text is cut and `output_truncated` is `true`.
* **Mid-call queries**: you can fetch an `active` session; you get the items persisted so far, and `analysis` is `null` until [post-call analysis](/agents/monitor/post-call-analysis) completes.
* A `tool_call` without a matching `tool_result` means the execution never resolved.

<Warning>
  `items` is a discriminated union on `type`, and it evolves additively. Ignore
  item types and fields you don't recognize. New modalities and item kinds will
  appear without a version bump.
</Warning>

### Correlate with live events

`call_id` is the same identifier your client receives in the SDK's `toolCallStarted` / `toolCallCompleted` / `toolCallFailed` events. Log it live, and you can align in-call UI with the post-call record: jump from a tool chip in your interface straight to the matching `tool_call` / `tool_result` pair in history. See the [Web SDK](/agents/deploy/web-sdk).

## Download recordings

`GET /v1/agent/sessions/{session_id}/recording` returns the recording status and signed download URLs: one audio track per speaker, so you can play or process the agent and the user separately. For sessions that never [recorded](#what-gets-stored) (recording turned off, or no audio produced), the endpoint returns `404`.

```bash API (curl) theme={null}
curl "https://api.fish.audio/v1/agent/sessions/SESSION_ID/recording" \
  --header "Authorization: Bearer $FISH_API_KEY"
```

Signed URLs are short-lived: store the `session_id` and request fresh URLs when you need the audio. For the same reason, recordings are not embedded in the session detail response.

## Access semantics

* Your API key sees every production session across your team's workspaces, from all sources.
* A session that doesn't exist, or belongs to another team, always returns `404`, never `403`. The [recording endpoint](#download-recordings) additionally returns `404` for a session that exists but was never recorded.

## Going further

<CardGroup>
  <Card title="Post-call analysis" icon="wand-magic-sparkles" href="/agents/monitor/post-call-analysis">
    What the embedded `analysis` object contains and how to configure it.
  </Card>

  <Card title="Webhooks" icon="webhook" href="/agents/monitor/webhooks">
    Push instead of poll: get notified when sessions end.
  </Card>

  <Card title="Web SDK" icon="js" href="/agents/deploy/web-sdk">
    The live `toolCall*` events that share `call_id` with history.
  </Card>

  <Card title="Inbound calls" icon="phone" href="/agents/telephony/inbound-calls">
    Where `caller_number` and `dialed_number` come from.
  </Card>
</CardGroup>


# Post-call Analysis
Source: https://docs.fish.audio/agents/monitor/post-call-analysis

Automatic summaries, structured data extraction, and success evaluation after every conversation

After a conversation ends, the platform runs an LLM pass over the [stored transcript](/agents/monitor/conversation-history#what-gets-stored) and produces three results: a **summary**, values for the **data fields** you define, and a verdict for each **success criterion**. Summaries are on by default for every agent. Extraction and evaluation are opt-in, configured per agent on the Builder's **Analysis** page.

<CardGroup>
  <Card title="See it on a test call" icon="phone" href="/agents/test/preview-calls">
    End a preview call and watch the analysis card fill in.
  </Card>

  <Card title="Read results via API" icon="brackets-curly" href="/agents/monitor/conversation-history">
    Analysis is embedded in every session detail response.
  </Card>

  <Card title="Push to your backend" icon="webhook" href="/agents/monitor/webhooks">
    Get notified the moment a conversation is analyzed.
  </Card>
</CardGroup>

## What a run produces

### Summary

A concise recap of the conversation, enabled by default.

* **Custom prompt** (optional, up to 1,000 characters): replace the platform's default summary instruction with your own, e.g. "Summarize in two sentences, focusing on the caller's request and the outcome."
* **Output language**: fixed per agent, chosen from the same [52 languages](/agents/build/voice-language#speaking-language) as the speaking language. Default is English.

<Note>
  Summaries and rationales are always written in the configured language, even
  when the conversation happened in another one, so downstream systems see
  consistent output.
</Note>

### Data fields

Structured values extracted from the transcript, up to **20 fields** per agent. Each field is:

| Property | Description |
| - | - |
| `name` | Unique per agent; lowercase letters, digits, and underscores, starting with a letter (up to 64 characters) |
| `type` | `boolean`, `text`, `number`, or `enum` |
| `description` | Instruction telling the model what to extract (up to 500 characters) |
| `enum_options` | For `enum` fields only: 2–20 allowed values |

When the conversation doesn't contain the information, the field's value is `null` with a rationale explaining why.

### Success criteria

Up to **10 criteria** per agent, each a `name` plus a `description` (up to 500 characters) stating what a successful conversation looks like, such as "The agent resolved the caller's issue or set clear next steps."

Every criterion gets a three-state verdict with a rationale:

* **`success`**: the transcript shows the expectation was met.
* **`failure`**: the transcript shows it was not.
* **`unknown`**: the transcript is incomplete, the answer was ambiguous, or the information needed to judge is missing.

<Tip>
  The `unknown` state is deliberate: rather than forcing an ambiguous call into
  a binary verdict, the model tells you it couldn't judge, so your success-rate
  numbers stay honest.
</Tip>

## Configure analysis

Open your agent's **Analysis** page in the Builder. Three cards (Summary, Data fields, and Criteria) autosave to the draft as you edit. You can also set the `analysis` section of the configuration via the API:

```bash theme={null}
curl --request PATCH "https://api.fish.audio/v1/agent/agents/$AGENT_ID/config" \
  --header "Authorization: Bearer $FISH_API_KEY" \
  --header "Content-Type: application/json" \
  --data '{
    "analysis": {
      "summary": { "enabled": true, "language": "en" },
      "data_fields": [
        {
          "name": "callback_requested",
          "type": "boolean",
          "description": "Did the caller ask to be called back?"
        },
        {
          "name": "issue_category",
          "type": "enum",
          "description": "The main topic of the call.",
          "enum_options": ["billing", "shipping", "product", "other"]
        }
      ],
      "criteria": [
        {
          "name": "issue_resolved",
          "description": "The agent fully resolved the caller'\''s issue, or set clear next steps before the call ended."
        }
      ]
    }
  }'
```

Like all configuration, edits land in the [draft](/agents/deploy/versions-publishing). Preview calls use the draft immediately; production sessions pick up analysis changes after you **publish**.

To disable analysis entirely, turn the summary off and leave both lists empty. Conversations then finish with analysis status `skipped` and no LLM call is made.

## When analysis runs

Analysis starts automatically when a conversation ends: production sessions from any source, and preview calls when you end them. A run moves through these statuses:

| Status | Meaning |
| - | - |
| `queued` | Scheduled, waiting to run |
| `running` | The LLM pass is in progress |
| `completed` | Results are available |
| `skipped` | Nothing to analyze: the conversation had no user messages, or analysis is disabled |
| `error` | The run failed; no results |

Results are typically ready within seconds of the conversation ending.

<Note>
  Production sessions are analyzed against the agent version pinned when the
  session started; editing your criteria afterwards never changes how past
  conversations were judged.
</Note>

## Read the results

In the Builder, analysis appears in the call panel after a preview call ends. Programmatically, the [session detail](/agents/monitor/conversation-history) response embeds it under `analysis`:

```bash theme={null}
curl "https://api.fish.audio/v1/agent/sessions/$SESSION_ID" \
  --header "Authorization: Bearer $FISH_API_KEY"
```

```json theme={null}
{
  "analysis": {
    "status": "completed",
    "summary": "The caller reported a delayed order and asked for a refund. The agent confirmed the delay, issued the refund, and offered a discount on the next purchase.",
    "data": [
      {
        "name": "callback_requested",
        "type": "boolean",
        "value": false,
        "rationale": "The caller did not ask to be called back."
      },
      {
        "name": "issue_category",
        "type": "enum",
        "value": "shipping",
        "rationale": "The conversation centered on a delayed delivery."
      }
    ],
    "criteria_results": [
      {
        "name": "issue_resolved",
        "result": "success",
        "rationale": "The refund was issued and the caller confirmed they were satisfied."
      }
    ]
  }
}
```

To push results into your CRM or data warehouse as soon as they're ready, configure a [post-call webhook](/agents/monitor/webhooks). `call.analyzed` fires as soon as the run settles, whatever the outcome, with `analysis.status` telling you which of the terminal statuses above it reached.

## Going further

<CardGroup>
  <Card title="Conversation history" icon="clock-rotate-left" href="/agents/monitor/conversation-history">
    Retrieve stored transcripts and session details over REST.
  </Card>

  <Card title="Webhooks" icon="webhook" href="/agents/monitor/webhooks">
    Deliver analysis results to your systems automatically.
  </Card>

  <Card title="Preview calls" icon="phone" href="/agents/test/preview-calls">
    Iterate on fields and criteria with instant feedback after each test call.
  </Card>

  <Card title="Versions & publishing" icon="code-branch" href="/agents/deploy/versions-publishing">
    How draft edits roll out to production sessions.
  </Card>
</CardGroup>


# Webhooks
Source: https://docs.fish.audio/agents/monitor/webhooks

Get an HTTP POST when a call ends and when its analysis settles, and personalize inbound phone calls before the agent answers

Point your agent at one or more endpoints on your server and Fish Audio calls them when things happen: `call.ended` the moment a session reaches a terminal state, `call.analyzed` when [post-call analysis](/agents/monitor/post-call-analysis) settles, and, on [outbound phone calls](/agents/telephony/outbound-calls), `phone_call.dial_finished` as soon as the dial attempt resolves. Use them to write results into your CRM, ticketing system, or data warehouse without polling the sessions API.

Inbound phone calls also have an [inbound call webhook](#inbound-call-webhook) that runs before the agent answers, so your server can look up the caller and hand back the values for the prompt's `{{placeholders}}`.

## Events

| Event | Fires | Payload |
| - | - | - |
| `call.ended` | When a session reaches a terminal state: every real call, whether or not analysis is configured | Session facts plus the reason the call ended |
| `call.analyzed` | When post-call analysis settles: completed, skipped, or error | Session facts plus the full `analysis` entity |
| `phone_call.dial_finished` | When an outbound call's dial attempt resolves: answered, busy, no answer, or failed | Session facts plus `dial_status` and `answered_by` |

For a given session, `call.ended` is delivered before `call.analyzed`, and every answered production call produces both events. When analysis has nothing to work with (the caller never spoke, or analysis is disabled) or fails, `call.analyzed` still arrives, with `analysis.status` set to `skipped` or `error` and empty result sections.

`phone_call.dial_finished` fires only for outbound calls, and it fires whether or not anyone picks up. An answered outbound call produces all three events, `phone_call.dial_finished` first; an outbound call that is never answered (a `dial_status` of `busy`, `no_answer`, or `failed`) produces `phone_call.dial_finished` and nothing else. Unanswered calls are not billed and not analyzed.

<Note>
  Transcripts are deliberately excluded from webhook payloads. Fetch them on
  demand with `GET /v1/agent/sessions/{session_id}`. See [conversation
  history](/agents/monitor/conversation-history).
</Note>

## Configure the endpoint

Webhooks live in the `webhooks` section of your agent's configuration. `post_call` is a list, so one agent can fan events out to several systems at once (the section's other entry, `conversation_init`, is the [inbound call webhook](#inbound-call-webhook)). Set them in the console on your agent's **Webhooks** page, or via the config API:

<CodeGroup>
  ```bash API (curl) theme={null}
  curl --request PATCH https://api.fish.audio/v1/agent/agents/YOUR_AGENT_ID/config \
    --header "Authorization: Bearer $FISH_API_KEY" \
    --header "Content-Type: application/json" \
    --data '{
      "webhooks": {
        "post_call": [
          {
            "url": "https://example.com/fish-webhooks",
            "secret": "your-signing-secret"
          },
          {
            "url": "https://crm.example.com/fish-webhooks",
            "secret": "another-signing-secret"
          }
        ]
      }
    }'
  ```

  ```python Python theme={null}
  import httpx, os

  httpx.patch(
      "https://api.fish.audio/v1/agent/agents/YOUR_AGENT_ID/config",
      headers={"Authorization": f"Bearer {os.environ['FISH_API_KEY']}"},
      json={
          "webhooks": {
              "post_call": [
                  {
                      "url": "https://example.com/fish-webhooks",
                      "secret": "your-signing-secret",
                  },
                  {
                      "url": "https://crm.example.com/fish-webhooks",
                      "secret": "another-signing-secret",
                  },
              ]
          }
      },
  )
  ```
</CodeGroup>

The list is replaced as a whole on each update, so send every endpoint you want to keep. Set it to `null` or `[]` to stop deliveries. A single `post_call` object is still accepted and is treated as a one-element list. Secrets are write-only, so an entry sent without a `secret` keeps the secret already stored for that `url`; you can read the configuration, edit the list, and send it back without re-entering secrets.

| Field | Rules |
| - | - |
| `post_call` | Up to 5 entries, and their `url` values must be unique. A longer list or a repeated URL is rejected with `422` |
| `url` | Required on every entry. Up to 4000 characters. Must resolve to a public address. Localhost and private-network hosts are rejected with `422`, and the check is repeated at every delivery |
| `secret` | Optional signing key, up to 256 characters, set per entry. **Write-only**: reads return `has_secret: true`, never the value. Omit it (or send `null`) to keep the stored secret for the same `url`; send `""` to remove it |

<Note>
  Webhook settings are part of the agent configuration, so they follow the
  draft-and-publish flow: [publish](/agents/deploy/versions-publishing) for
  changes to apply to production calls.
</Note>

## Payload

Every payload carries the `event` name and a `session` object with a fixed set of session facts. Note the field names differ from the [sessions API](/agents/monitor/conversation-history), which uses `session_id` / `started_at` / `ended_at` for the same facts. `call.ended` adds `ended_reason`; `call.analyzed` adds the `analysis` entity; `phone_call.dial_finished` adds top-level `dial_status` and `answered_by`:

<CodeGroup>
  ```json call.ended theme={null}
  {
    "event": "call.ended",
    "session": {
      "id": "5f3e…",
      "agent_id": "a1b2…",
      "branch_id": "b7d4…",
      "source": "phone",
      "status": "completed",
      "end_reason": "user_hangup",
      "conversation_started_at": "2026-07-23T12:01:12Z",
      "conversation_ended_at": "2026-07-23T12:04:16Z",
      "duration_seconds": 184,
      "end_user_id": "customer-42",
      "metadata": { "order_ref": "SO-1042" },
      "agent_name": "Support agent",
      "config_hash": "sha256:9c41…"
    },
    "ended_reason": "user_hangup"
  }
  ```

  ```json call.analyzed theme={null}
  {
    "event": "call.analyzed",
    "session": {
      "id": "5f3e…",
      "agent_id": "a1b2…",
      "branch_id": "b7d4…",
      "source": "phone",
      "status": "completed",
      "conversation_started_at": "2026-07-23T12:01:12Z",
      "conversation_ended_at": "2026-07-23T12:04:16Z",
      "duration_seconds": 184,
      "end_user_id": "customer-42",
      "metadata": { "order_ref": "SO-1042" },
      "agent_name": "Support agent",
      "config_hash": "sha256:9c41…"
    },
    "analysis": {
      "status": "completed",
      "summary": "Caller asked about a delayed order and accepted a reshipment.",
      "data": [
        {
          "name": "callback_requested",
          "type": "boolean",
          "value": false,
          "rationale": "…"
        }
      ],
      "criteria_results": [
        { "name": "issue_resolved", "result": "success", "rationale": "…" }
      ]
    }
  }
  ```

  ```json phone_call.dial_finished theme={null}
  {
    "event": "phone_call.dial_finished",
    "dial_status": "answered",
    "answered_by": "human",
    "session": {
      "id": "5f3e…",
      "agent_id": "a1b2…",
      "branch_id": "b7d4…",
      "source": "phone",
      "status": "active",
      "direction": "outbound",
      "dial_status": "answered",
      "answered_by": "human",
      "caller_number": "+15551234567",
      "dialed_number": "+14155550100",
      "sip_call_id": "7f3a9c@sbc.example.com",
      "batch_call_id": null,
      "conversation_started_at": "2026-07-23T12:01:12Z",
      "conversation_ended_at": null,
      "duration_seconds": null,
      "end_user_id": "",
      "metadata": { "lead_ref": "L-2041" },
      "agent_name": "Follow-up agent",
      "config_hash": "sha256:9c41…"
    }
  }
  ```
</CodeGroup>

`ended_reason` is the same value as `session.end_reason` and uses the [sessions API vocabulary](/agents/monitor/conversation-history#status-and-end-reason): `user_hangup`, `agent_hangup`, `escalated`, `conversation_timeout`, `heartbeat_timeout`, `agent_join_timeout`, `internal_error`, or `llm_endpoint_failure`. `call.ended` is only sent for sessions in which a participant connected, so the never-connected reasons (`room_join_timeout`, `cancelled`, `dial_failed`) do not appear here; query the sessions API with `status=failed` for those. In the rare case the reason is not settled yet at delivery time, both fields are `null`. New values may be added; treat unrecognized values as informational rather than rejecting the event.

`dial_status` is `answered`, `busy`, `no_answer`, or `failed`. `answered_by` is deprecated: answering-machine detection has been removed, so it is `unknown` once the call is answered and `null` before (older events may carry `human` or `voicemail`). Both facts also appear on the session, alongside `direction` (`inbound` or `outbound`) and `batch_call_id` (reserved, always `null` today); these session fields are present in every webhook payload, with the dial fields `null` on inbound sessions. The `phone_call.dial_finished` snapshot is taken when the dial resolves, so on an answered call `conversation_ended_at` and `duration_seconds` are still `null`. The final numbers arrive with `call.ended`.

`caller_number`, `dialed_number` and `sip_call_id` carry the same values as on the [sessions API](/agents/monitor/conversation-history): the E.164 numbers of the call (person-centric on outbound calls, so `caller_number` is the party you dialed) and the SIP `Call-ID` of the call's INVITE. All three are empty strings on web sessions, and `sip_call_id` is also empty on an unanswered outbound dial. How to use it against your carrier's records: [Match calls with your SIP records](/agents/telephony/byo-sip#match-calls-with-your-sip-records).

`end_user_id` and `metadata` are echoed exactly as you set them when creating the session. Use them to correlate the event with records in your own system. `branch_id` is an internal configuration-lineage identifier and is safe to ignore. What lands in `summary`, `data`, and `criteria_results` is defined by your [analysis configuration](/agents/monitor/post-call-analysis).

The example shows a `completed` analysis. `analysis.status` can also be `skipped` (nothing to analyze: the caller never spoke, or analysis is disabled) or `error` (the run failed, with the cause in `analysis.error`); both arrive with `summary: null` and empty `data` / `criteria_results`. Check the status before reading results.

## Verify the signature

When an entry has a `secret`, every request to that endpoint carries a signature header with the send time and an HMAC-SHA256 of `{timestamp}.{raw_body}`, keyed with that entry's own secret:

```http theme={null}
X-Fish-Webhook-Signature: t=1784808000,v1=8693a4b9…
```

`t` is the Unix time (seconds) the request was sent; each retry is signed fresh. `v1` is the hex HMAC. Verify in three steps:

1. Parse `t` and `v1` from the header.
2. Recompute HMAC-SHA256 over the string `{t}.` followed by the **raw request body bytes** (before any JSON parsing or re-serialization) and compare against `v1` in constant time.
3. Reject requests whose `t` is more than 5 minutes from your clock. Because the timestamp is inside the MAC, a replayed capture can't be refreshed.

<CodeGroup>
  ```javascript Node.js theme={null}
  import crypto from "node:crypto";

  const TOLERANCE_SECONDS = 300;

  function verifyWebhook(rawBody, signatureHeader, secret) {
    const elements = new Map(
      (signatureHeader ?? "").split(",").map(el => el.split("=", 2))
    );
    const timestamp = elements.get("t");
    const signature = elements.get("v1");
    if (
      !/^\d+$/.test(timestamp ?? "") ||
      !/^[0-9a-f]{64}$/.test(signature ?? "")
    ) {
      return false;
    }
    if (Math.abs(Date.now() / 1000 - Number(timestamp)) > TOLERANCE_SECONDS) {
      return false;
    }
    const expected = crypto
      .createHmac("sha256", secret)
      .update(`${timestamp}.`)
      .update(rawBody)
      .digest("hex");
    return crypto.timingSafeEqual(Buffer.from(signature), Buffer.from(expected));
  }
  ```

  ```python Python theme={null}
  import hashlib
  import hmac
  import time

  TOLERANCE_SECONDS = 300

  def verify_webhook(raw_body: bytes, signature_header: str, secret: str) -> bool:
      elements = dict(
          el.split("=", 1) for el in (signature_header or "").split(",") if "=" in el
      )
      timestamp = elements.get("t", "")
      signature = elements.get("v1", "")
      if not timestamp.isdigit():
          return False
      if abs(time.time() - int(timestamp)) > TOLERANCE_SECONDS:
          return False
      signed = f"{timestamp}.".encode() + raw_body
      expected = hmac.new(secret.encode(), signed, hashlib.sha256).hexdigest()
      return hmac.compare_digest(signature.encode(), expected.encode())
  ```
</CodeGroup>

<Warning>
  Reject requests with a missing or invalid signature. Without verification,
  anyone who discovers your endpoint URL can forge call results.
</Warning>

<Note>
  Future scheme revisions would ship under a new element (`v2=…`) alongside `v1`.
  Parse the elements you know and ignore the rest, as the snippets above do.
</Note>

## Delivery semantics

| Property | Behavior |
| - | - |
| Fan-out | Every configured endpoint receives every event |
| Guarantee | At-least-once, per endpoint |
| Timeout | 10 seconds per attempt |
| Retries | 2 after the first attempt (3 attempts total) per endpoint, with backoff of 1s / 5s, then that delivery is dropped |
| Ordering | `phone_call.dial_finished` (outbound only) before `call.ended` before `call.analyzed` for the same session |

Endpoints are delivered in parallel and independently: each gets its own attempts, its own retry budget, and its own signature keyed with its own secret. An endpoint that is down and exhausts all three attempts has no effect on the others.

Respond with a `2xx` status within the timeout; a `500` response or a timed-out request counts as a failed attempt. Acknowledge first and process asynchronously. Slow handlers burn their own retry budget.

**Idempotency.** At-least-once delivery means the same event can arrive more than once. Retries of one delivery carry an identical body, so dedupe `call.ended` and `phone_call.dial_finished` on (`event`, `session.id`), and `call.analyzed` on (`event`, `session.id`, `analysis.finished_at`). The extra element matters because a skipped or failed analysis can be re-run from the console: the recovered result arrives as a fresh `call.analyzed` with a newer `finished_at`, superseding the earlier one.

<Note>
  [Preview calls](/agents/test/preview-calls) made from the Builder never
  trigger webhooks. To test end to end, run a real session against your
  published agent.
</Note>

## Inbound call webhook

`webhooks.conversation_init` is a single endpoint called while an [inbound phone call](/agents/telephony/inbound-calls) is being set up, before the agent answers. It receives the caller's number and returns `dynamic_variables` that fill the `{{placeholders}}` in the agent's published prompt and first message, so the agent can greet a known customer by name or pick up an open order. In the console it is the **Inbound call webhook** on the agent's Webhooks page.

Only inbound phone calls use it. Web sessions and outbound calls carry their [dynamic variables](/agents/build/dynamic-variables) on the request that creates them, and [preview calls](/agents/test/preview-calls) never call it.

```bash Configure theme={null}
curl --request PATCH https://api.fish.audio/v1/agent/agents/YOUR_AGENT_ID/config \
  --header "Authorization: Bearer $FISH_API_KEY" \
  --header "Content-Type: application/json" \
  --data '{
    "webhooks": {
      "conversation_init": {
        "url": "https://example.com/fish-inbound",
        "secret": "your-signing-secret",
        "timeout_seconds": 5
      }
    }
  }'
```

| Field | Rules |
| - | - |
| `url` | Required. Same rules as a `post_call` URL: up to 4000 characters, public address only |
| `secret` | Optional signing key, up to 256 characters. Write-only: reads return `has_secret`. When set, requests carry the same [signature header](#verify-the-signature) as post-call events |
| `timeout_seconds` | How long the platform waits for your response before connecting the call without variables. `1` to `8`, default `5` |

Send `"conversation_init": null` to remove the endpoint. Like the rest of the configuration, the change applies to phone calls after the next [publish](/agents/deploy/versions-publishing).

### Request

The platform sends one `POST` with a JSON body:

```json call.inbound theme={null}
{
  "event": "call.inbound",
  "call": {
    "caller_number": "+15551234567",
    "dialed_number": "+14155550100",
    "agent_id": "a1b2…",
    "phone_number_id": "7e9f…",
    "twilio_call_sid": "CA5d3a…"
  }
}
```

`caller_number` and `dialed_number` are E.164. `caller_number` is an empty string when the carrier withholds the caller's number, and `twilio_call_sid` is empty when the call did not come through a Twilio number.

### Response

Respond with a `2xx` status and a JSON object whose `dynamic_variables` follow the same rules as on session creation: names match `[A-Za-z][A-Za-z0-9_]*`, values are strings (up to 1,000 characters), numbers, or booleans, and at most 50 entries are read.

```json Response theme={null}
{
  "dynamic_variables": {
    "customer_name": "Ada",
    "plan": "Pro",
    "open_orders": 2
  }
}
```

An entry that breaks a rule is dropped on its own; the call still connects with the rest. Other fields in the response are ignored, so the contract can grow without breaking deployed endpoints. Placeholders that receive no value stay in the text as written, as described under [dynamic variables](/agents/build/dynamic-variables#custom-variables).

```text System prompt theme={null}
You are the support line for Acme. The caller is {{customer_name}} on the {{plan}} plan
with {{open_orders}} open orders. Greet them by name.
```

### Failure behavior

The endpoint can personalize a call but never block one. The caller is waiting while the request runs, so keep the handler fast and the timeout short.

| Outcome | Effect |
| - | - |
| Response within the timeout | Variables are rendered into the prompt and first message |
| Timeout | The call connects without variables; the request is not retried |
| Connection error or non-`2xx` response | One immediate retry, then the call connects without variables |
| Body is not a JSON object | The call connects without variables |

## Auto-ticket unresolved calls

`call.analyzed` closes the loop on conversations the agent couldn't: judge every call with a success criterion, and open a ticket in your helpdesk whenever the verdict isn't `success`.

First give the agent's [analysis configuration](/agents/monitor/post-call-analysis) a criterion that captures resolution:

```json analysis.criteria theme={null}
[
  {
    "name": "issue_resolved",
    "description": "The agent fully resolved the caller's issue, or set clear next steps before the call ended."
  }
]
```

Then add your endpoint to `webhooks.post_call` (see [Configure the endpoint](#configure-the-endpoint)) and act on the verdict:

<CodeGroup>
  ```javascript Node.js (Express) theme={null}
  import express from "express";

  const app = express();
  const seen = new Set(); // swap for your database

  app.post(
    "/fish-webhooks",
    express.raw({ type: "application/json" }),
    (req, res) => {
      const signature = req.get("X-Fish-Webhook-Signature");
      if (!verifyWebhook(req.body, signature, process.env.FISH_WEBHOOK_SECRET)) {
        return res.status(401).end();
      }
      res.status(200).end(); // acknowledge first, then process

      const payload = JSON.parse(req.body);
      if (payload.event !== "call.analyzed") return;

      const key = `${payload.session.id}:${payload.analysis.finished_at}`;
      if (seen.has(key)) return; // at-least-once delivery: dedupe
      seen.add(key);

      if (payload.analysis.status !== "completed") {
        // skipped (the caller never spoke) or error: no verdict to trust.
        openTicket({
          subject: `Unjudged call ${payload.session.id}`,
          body:
            `Analysis ${payload.analysis.status}` +
            (payload.analysis.error ? `: ${payload.analysis.error}` : ""),
          customer: payload.session.end_user_id,
        });
        return;
      }

      const verdict = payload.analysis.criteria_results.find(
        c => c.name === "issue_resolved"
      );
      if (!verdict || verdict.result === "success") return;

      openTicket({
        subject: `Unresolved call ${payload.session.id}`,
        body:
          `${payload.analysis.summary}\n\n` +
          `issue_resolved: ${verdict.result}. ${verdict.rationale}`,
        customer: payload.session.end_user_id,
      });
    }
  );
  ```

  ```python Python (FastAPI) theme={null}
  import json
  import os

  from fastapi import FastAPI, Header, Request, Response

  app = FastAPI()
  seen: set[str] = set()  # swap for your database

  @app.post("/fish-webhooks")
  async def fish_webhooks(
      request: Request,
      x_fish_webhook_signature: str | None = Header(None),
  ):
      raw = await request.body()
      secret = os.environ["FISH_WEBHOOK_SECRET"]
      if not verify_webhook(raw, x_fish_webhook_signature, secret):
          return Response(status_code=401)

      payload = json.loads(raw)
      if payload["event"] != "call.analyzed":
          return {}

      analysis = payload["analysis"]
      key = f"{payload['session']['id']}:{analysis['finished_at']}"
      if key in seen:  # at-least-once delivery: dedupe
          return {}
      seen.add(key)

      if analysis["status"] != "completed":
          # skipped (the caller never spoke) or error: no verdict to trust.
          detail = f": {analysis['error']}" if analysis.get("error") else ""
          open_ticket(
              subject=f"Unjudged call {payload['session']['id']}",
              body=f"Analysis {analysis['status']}{detail}",
              customer=payload["session"]["end_user_id"],
          )
          return {}

      results = analysis["criteria_results"]
      verdict = next((c for c in results if c["name"] == "issue_resolved"), None)
      if verdict and verdict["result"] != "success":
          open_ticket(
              subject=f"Unresolved call {payload['session']['id']}",
              body=f"{analysis['summary']}\n\n"
              f"issue_resolved: {verdict['result']}. {verdict['rationale']}",
              customer=payload["session"]["end_user_id"],
          )
      return {}
  ```
</CodeGroup>

`verifyWebhook` is the function from [Verify the signature](#verify-the-signature); `openTicket` stands in for your helpdesk's API. Escalating on anything but `success` includes `unknown` verdicts: the model couldn't judge the call, which usually deserves human eyes too. Tighten the check to `failure` only if unknowns prove noisy.

Edges worth handling:

* Calls with nothing to analyze arrive with `analysis.status: "skipped"`. The handler above tickets them as unjudged, so every call reaches the helpdesk without also watching `call.ended`. Drop that branch if silent calls don't belong in your queue.
* Payloads carry no transcript. To include one in the ticket, fetch `GET /v1/agent/sessions/{session_id}` from your handler. See [conversation history](/agents/monitor/conversation-history).
* Set `end_user_id` and `metadata` when creating sessions so tickets attach to the right customer record without a lookup.

<Note>
  Post-call ticketing is silent: the caller has already hung up. When the
  caller should leave the call holding a ticket number, have the agent open it
  mid-call with a [webhook
  tool](/agents/build/webhook-tools#escalate-to-a-ticket-mid-call), and keep
  this recipe as the safety net behind it.
</Note>

## Going further

<CardGroup>
  <Card title="Post-call analysis" icon="chart-simple" href="/agents/monitor/post-call-analysis">
    Define the summary, data fields, and criteria that `call.analyzed` delivers.
  </Card>

  <Card title="Conversation history" icon="clock-rotate-left" href="/agents/monitor/conversation-history">
    Fetch the transcript, tool timeline, and recordings by `session_id`.
  </Card>

  <Card title="Versions & publishing" icon="code-branch" href="/agents/deploy/versions-publishing">
    How configuration changes (including webhooks) go live.
  </Card>

  <Card title="Agent configuration" icon="sliders" href="/agents/build/configuration">
    The full config schema behind `PATCH /v1/agent/agents/{agent_id}/config`.
  </Card>
</CardGroup>


# Agents Overview
Source: https://docs.fish.audio/agents/overview

Build voice agents in the console, ship them with the REST API and SDKs, and put them on the phone

Fish Agents is a hosted platform for real-time voice agents. Define an agent's prompt, voice, knowledge, and tools in the Builder console, publish it, and talk to it from your web app through the SDK, over a phone number, or anywhere the REST API reaches. Speech recognition, turn-taking, and speech synthesis are handled for you.

<Note>
  Agents is in public beta. Every Fish Audio account can use it: open
  [Agents](https://fish.audio/app/agents) in the console to get started.
  Features and APIs may still change before general availability.
</Note>

<CardGroup>
  <Card title="Quickstart" icon="rocket" href="/agents/quickstart">
    Create an agent and have your first conversation in minutes.
  </Card>

  <Card title="Concepts" icon="lightbulb" href="/agents/concepts">
    Agents, drafts, versions, sessions: the mental model.
  </Card>

  <Card title="Integration overview" icon="plug" href="/agents/deploy/overview">
    Pick the right surface for your app.
  </Card>
</CardGroup>

## Three surfaces, one agent

| Surface | What it does | Start here |
| - | - | - |
| **Console** | Design, test, and publish agents visually, with no code. | [Configuration](/agents/build/configuration) |
| **REST API** | Manage agents, sessions, tools, and knowledge programmatically under `/v1/agent/*`. | [API introduction](/api-reference/introduction) |
| **Client SDKs** | Run live voice conversations in the browser with `@fishaudio/agent-client` or `@fishaudio/agent-react`. | [Web SDK](/agents/deploy/web-sdk) |

All three operate on the same agents. The core console workflow (creating agents, editing configuration, publishing versions) is also available over the API with your standard Fish Audio API key:

```bash theme={null}
curl https://api.fish.audio/v1/agent/agents \
  --header "Authorization: Bearer $FISH_API_KEY"
```

## From draft to production

<Steps>
  <Step title="Configure">
    Write the system prompt, set the first message, and pick a [voice and
    speaking language](/agents/build/voice-language). Attach [knowledge
    sources](/agents/build/knowledge-base) and [tools](/agents/build/tools).
    Edits save to the agent's draft automatically.
  </Step>

  <Step title="Test">
    Talk to the draft in a [preview call](/agents/test/preview-calls), or run
    scripted [agent tests](/agents/test/agent-tests) that judge each reply pass
    or fail.
  </Step>

  <Step title="Publish">
    [Publishing](/agents/deploy/versions-publishing) freezes the draft into an
    immutable, numbered version. Live sessions always run the published
    configuration, so you can keep editing safely.
  </Step>

  <Step title="Integrate">
    Drop the [web SDK](/agents/deploy/web-sdk) into your app, create sessions
    over [REST](/agents/deploy/overview), or [bind a phone
    number](/agents/telephony/phone-numbers) so the agent answers inbound calls.
  </Step>

  <Step title="Monitor">
    Review transcripts, tool calls, and recordings in [conversation
    history](/agents/monitor/conversation-history), let [post-call
    analysis](/agents/monitor/post-call-analysis) summarize and evaluate every
    call, and push results to your backend with
    [webhooks](/agents/monitor/webhooks).
  </Step>
</Steps>

## Capabilities

<CardGroup>
  <Card title="Prompt & voice" icon="sliders" href="/agents/build/configuration">
    System prompt, first message, and any voice from the Fish Audio Voice
    Library.
  </Card>

  <Card title="Knowledge base" icon="book" href="/agents/build/knowledge-base">
    Upload documents the agent retrieves from during conversations. Update a
    source and every agent using it picks up the new content.
  </Card>

  <Card title="Tools" icon="wrench" href="/agents/build/tools">
    Call your backend with [webhook tools](/agents/build/webhook-tools), run
    code in your app with [client tools](/agents/build/client-tools), and toggle
    built-in [system tools](/agents/build/system-tools).
  </Card>

  <Card title="Dynamic variables" icon="brackets-curly" href="/agents/build/dynamic-variables">
    Personalize each session with values injected at call time.
  </Card>

  <Card title="Versions & publishing" icon="code-branch" href="/agents/deploy/versions-publishing">
    Draft safely, publish deliberately, and read back the full version history.
  </Card>

  <Card title="Telephony" icon="phone" href="/agents/telephony/inbound-calls">
    Answer [inbound calls](/agents/telephony/inbound-calls) on [phone
    numbers](/agents/telephony/phone-numbers) bound to your agents.
  </Card>

  <Card title="Conversation history" icon="clock-rotate-left" href="/agents/monitor/conversation-history">
    Transcripts, tool timelines, and per-speaker recordings. You choose what
    each agent stores.
  </Card>

  <Card title="Post-call analysis & webhooks" icon="chart-simple" href="/agents/monitor/post-call-analysis">
    Automatic summaries, structured data extraction, and success criteria,
    delivered to your systems via [webhooks](/agents/monitor/webhooks).
  </Card>
</CardGroup>


# Pricing
Source: https://docs.fish.audio/agents/pricing

Rates for conversations, LLM usage, telephony, and transfers, plus the trial balance and billing

Agents cost \*\*$0.06 per minute** of conversation, with a small number of add-ons for telephony and token-billed models. Every account starts with a $5 trial balance, and everything after that is charged against your API credit.

## Rates

| Item | Price | Notes |
| - | - | - |
| Agent conversation | \$0.06 per minute | All-inclusive: STT, TTS, the basic LLMs, knowledge base retrieval, and tool calls |
| LLM usage | Covered for the basic LLMs, by token usage for the rest | The other LLMs are billed on the tokens each request uses. See [LLM models](#llm-models). Custom LLM endpoints carry no LLM charge |
| Phone call surcharge | +\$0.01 per minute | Added to the conversation rate on phone calls. Some outbound destinations carry a higher rate instead. See [Outbound destinations](#outbound-destinations) |
| Phone number rental | \$1.20 per month | Charged in daily slices while you hold the number, and stops when you release it |
| Cold transfer | \$0.015 per minute | Charged for the time after the agent hands the call to a person. The conversation rate stops at that point |
| Warm transfer | \$0.025 per minute | Charged for the time after the caller and the person are joined. The joined call is still recorded and transcribed |

Usage is counted per second. A plus sign marks a charge on top of the conversation rate.

Calls on [imported SIP numbers](/agents/telephony/byo-sip) carry no telephony charges: no phone surcharge, no number rental, no transfer fees. They bill like web sessions.

### LLM models

The LLM your agent runs is chosen in the Builder. The basic models are:

* Gemini 3.5 Flash Lite
* GPT-5.6 Luna
* Gemma 4 26B

Every model has a per-token cost. For now the conversation rate covers it for the basic models as a promotional offer, so they carry no separate charge.

Every other model is billed on token usage. Prices are expressed in USD per 1 million tokens, with separate rates for input tokens, output tokens, and cached input. These rates are determined by the LLM providers and may change. Refer to the provider's documentation for current pricing.

What a minute costs on such a model therefore depends on your agent: how long the system prompt and retrieved knowledge are, how much conversation history the model re-reads on every turn, how often it is called, and how verbose its replies are. The Builder shows an estimated per-minute figure next to each token-billed model, computed from your current prompt and typical agent behavior. The estimate does not account for prompt caching, and actual charges are based on token usage.

After a conversation, each message in the dashboard's conversation view shows its model requests and token cost. A [custom LLM](/agents/build/custom-llm) endpoint carries no LLM charge.

### Outbound destinations

The standard phone surcharge applies to inbound calls and most outbound calls. Outbound calls to the destinations below replace it with a rate that depends on the number dialed:

| Destination | Landline | Mobile |
| - | - | - |
| Japan (landline includes 050 numbers) | +\$0.10 per minute | +\$0.22 per minute |
| United Kingdom (03 numbers: +\$0.04) | +\$0.02 per minute | +\$0.03 per minute |
| Italy | +\$0.02 per minute | +\$0.05 per minute |
| Australia | +\$0.03 per minute | +\$0.09 per minute |
| Mexico (mobile means the legacy `+52 1`) | +\$0.02 per minute | +\$0.05 per minute |

The countries purchased numbers can call are listed under [allowed destinations](/agents/telephony/outbound-calls#allowed-destinations).

## Trial balance

Every account starts with a one-time **\$5 trial balance**, about 60 minutes of conversation at the base rate. It pays for real usage at the normal rates:

* Every charge draws on it first: web sessions, phone calls and their surcharges, transferred call time, number rental, and LLM tokens on token-billed models. Only what the trial balance does not cover is charged to your API credit.
* A session can start on any model, including token-billed ones, as long as the trial balance or your API credit is above zero.
* [Preview calls](/agents/test/preview-calls) in the Builder are free on every LLM, including token-billed ones, and never draw on the trial balance.
* The [Agents billing page](https://fish.audio/app/agents/billing) shows how much of the trial balance is left, and the Agents home page shows it as a usage meter.

## Billing

Usage is charged to your **API credit**, the same prepaid balance as the Fish Audio speech APIs. Sessions are charged when they end, and number rental is charged daily.

Every charge is listed on the [Agents billing page](https://fish.audio/app/agents/billing).

## Examples

| Scenario | Breakdown | Total |
| - | - | - |
| Web conversation, 4 min 30 s, basic LLM | Conversation 4.5 min × $0.06 = $0.27 | \$0.27 |
| Inbound phone call, 6 min, basic LLM | Conversation 6 min × $0.06 = $0.36<br />Phone surcharge 6 min × $0.01 = $0.06 | \$0.42 |
| Outbound call, 10 min, Japanese mobile number | Conversation 10 min × $0.06 = $0.60<br />Japan mobile surcharge 10 min × $0.22 = $2.20 | \$2.80 |
| Inbound call, 3 min with the agent, then 5 min with a person after a warm transfer | Conversation 3 min × $0.06 = $0.18<br />Phone surcharge 3 min × $0.01 = $0.03<br />Warm transfer 5 min × $0.025 = $0.125 | \$0.335 |
| Web conversation, 5 min, token-billed LLM | Conversation 5 min × $0.06 = $0.30<br />LLM usage: the session's tokens at the LLM's rates | \$0.30 + LLM usage |

## Going further

<CardGroup>
  <Card title="Quickstart" icon="rocket" href="/agents/quickstart">
    Create an agent and talk to it on the trial balance.
  </Card>

  <Card title="Phone numbers" icon="hashtag" href="/agents/telephony/phone-numbers">
    Buy or import a number and bind it to an agent.
  </Card>

  <Card title="Transfers" icon="phone-arrow-right" href="/agents/telephony/transfers">
    Cold and warm transfers to a human.
  </Card>

  <Card title="Custom LLM" icon="brain" href="/agents/build/custom-llm">
    Run your own LLM endpoint at the base rate.
  </Card>
</CardGroup>


# Quickstart
Source: https://docs.fish.audio/agents/quickstart

Create a voice agent and have your first conversation in minutes

Build a voice agent and talk to it in a few minutes. Use the console for a no-code path, or provision everything over the API and connect from the browser with the SDK.

**Prerequisites**

* A Fish Audio account. Agents is in public beta and open to every account, no application needed.
* For the API path: a Fish Audio API key. Create one under [API keys](https://fish.audio/app/api-keys/) in the console
* Every account starts with a \$5 trial balance, about 60 minutes of agent conversation. It pays for web, phone, and API sessions at the normal rates, including token-billed LLMs. After that, usage is billed per second against your API credit.

<Tabs>
  <Tab title="Dashboard">
    <Steps>
      <Step title="Create an agent">
        Go to [Agents](https://fish.audio/app/agents) and click **New agent**. Give it a name. You land in the Builder immediately.
      </Step>

      <Step title="Write the system prompt">
        On the **Configuration** page, write the system prompt that defines who your agent is and how it should behave (up to 32,000 tokens). Optionally set a **First message** so the agent opens the conversation.

        Edits save automatically as a draft.
      </Step>

      <Step title="Pick a voice">
        Choose a voice and a speaking language for your agent. You can pick from the featured voices or browse the full library. See [Voice & language](/agents/build/voice-language) for details.
      </Step>

      <Step title="Test it with a preview call">
        Click **Test call** in the top bar, then **Start call** in the panel. Your browser asks for microphone access, then you talk to the agent directly, with a live transcript, call timer, and mute and hang-up controls in the side panel.

        Preview calls always run against your current draft, so you can iterate on the prompt and immediately hear the difference. See [Preview calls](/agents/test/preview-calls).
      </Step>

      <Step title="Publish">
        Click **Publish** to turn the draft into an immutable version. Published versions are what real sessions connect to.

        You can keep editing the draft afterwards; nothing goes live until you publish again. See [Versions & publishing](/agents/deploy/versions-publishing).
      </Step>
    </Steps>
  </Tab>

  <Tab title="API">
    <Steps>
      <Step title="Create an agent">
        Create the agent and configure it in one call. The `config` section is optional at creation time, and you can update it later with `PATCH /v1/agent/agents/{agent_id}/config`.

        ```bash Create an agent theme={null}
        curl --request POST https://api.fish.audio/v1/agent/agents \
          --header "Authorization: Bearer $FISH_API_KEY" \
          --header "Content-Type: application/json" \
          --data '{
            "name": "Support agent",
            "config": {
              "prompt": {
                "system_prompt": "You are a friendly support agent for Acme. Keep answers short and conversational."
              },
              "voice": {
                "voice_id": "802e3bc2b27e49c2995d23ef70e6ac89"
              }
            }
          }'
        ```

        The response includes the agent's `agent_id`. Use it as `$AGENT_ID` below. `voice_id` accepts any voice model id from the [Voice Library](/features/manage-voices).

        <Note>
          `system_prompt` is limited to 32,000 tokens; longer prompts return `422`. Keeping it under 2,000 tokens is recommended for latency and cost.
        </Note>
      </Step>

      <Step title="Publish it">
        Sessions only run **published** configuration. Creating a session for an agent that has never been published returns `409`.

        ```bash Publish the draft theme={null}
        curl --request POST https://api.fish.audio/v1/agent/agents/$AGENT_ID/publish \
          --header "Authorization: Bearer $FISH_API_KEY"
        ```

        Each publish creates an immutable version with an auto-incremented `version_number`.
      </Step>

      <Step title="Create a session">
        From your backend, exchange your API key for a short-lived session token. This is the credential the browser uses to join the call. Your API key never leaves your server.

        ```bash Create a session theme={null}
        curl --request POST https://api.fish.audio/v1/agent/sessions \
          --header "Authorization: Bearer $FISH_API_KEY" \
          --header "Content-Type: application/json" \
          --data '{ "agent_id": "'$AGENT_ID'", "timezone": "Asia/Tokyo" }'
        ```

        `timezone` overrides the agent's timezone for this session: replace it with your user's IANA timezone. The default is UTC, which you can change in the agent's configuration. See [Time & timezone](/agents/build/time-timezone).

        ```json Response theme={null}
        {
          "session_id": "...",
          "expires_at": "2026-07-23T12:34:56Z",
          "max_duration_seconds": 1800,
          "transport": "livekit",
          "livekit_url": "wss://...",
          "token": "<participant token>"
        }
        ```

        Return this response to your frontend as-is. `expires_at` is the deadline for joining the call.
      </Step>

      <Step title="Connect from the browser">
        Install the client SDK:

        ```bash Install theme={null}
        npm install @fishaudio/agent-client
        ```

        Pass the session token to `AgentSession.start`. The SDK requests the microphone, connects, and streams audio both ways:

        ```typescript Connect and talk theme={null}
        import { AgentSession } from "@fishaudio/agent-client";

        // sessionToken: the JSON response from POST /v1/agent/sessions,
        // fetched from your backend
        const session = await AgentSession.start({
          sessionToken,
          callbacks: {
            onUserTranscript: ({ text }) => console.log("You:", text),
            onAgentResponse: ({ text }) => console.log("Agent:", text),
          },
        });

        // When you are done:
        await session.end();
        ```

        Start talking. You hear the agent reply and both sides of the conversation arrive as transcript events. See the [Web SDK](/agents/deploy/web-sdk) for the full event and method surface, or the [React SDK](/agents/deploy/react-sdk) for hooks.
      </Step>

      <Step title="Put it together">
        A complete local setup is one backend route that creates the session with your API key, and a page that starts the call. The frontend is served by Vite, which also proxies `/api` to the backend so the browser never sees your key.

        ```bash Install theme={null}
        npm init -y
        npm install express @fishaudio/agent-client
        npm install --save-dev vite
        ```

        ```javascript server.mjs theme={null}
        import express from "express";

        const app = express();

        app.post("/api/voice-session", async (req, res) => {
          const upstream = await fetch("https://api.fish.audio/v1/agent/sessions", {
            method: "POST",
            headers: {
              Authorization: `Bearer ${process.env.FISH_API_KEY}`,
              "Content-Type": "application/json",
            },
            body: JSON.stringify({
              agent_id: process.env.AGENT_ID,
              timezone: "Asia/Tokyo", // your user's IANA timezone
            }),
          });
          if (!upstream.ok) {
            console.error("session creation failed:", upstream.status);
            return res.status(502).json({ error: "session_unavailable" });
          }
          res.json(await upstream.json());
        });

        app.listen(3001, () => console.log("backend on http://localhost:3001"));
        ```

        ```javascript vite.config.mjs theme={null}
        export default {
          server: {
            proxy: { "/api": "http://localhost:3001" },
          },
        };
        ```

        ```html index.html theme={null}
        <!doctype html>
        <button id="start">Start call</button>
        <button id="end" disabled>Hang up</button>
        <pre id="log"></pre>
        <script type="module" src="/main.js"></script>
        ```

        ```javascript main.js theme={null}
        import { AgentSession } from "@fishaudio/agent-client";

        const startButton = document.getElementById("start");
        const endButton = document.getElementById("end");
        const log = (line) => (document.getElementById("log").textContent += line + "\n");

        let session;

        // Start from a click: browsers only grant the microphone and
        // audio playback inside a user gesture.
        startButton.onclick = async () => {
          const sessionToken = await fetch("/api/voice-session", { method: "POST" })
            .then((r) => r.json());

          session = await AgentSession.start({
            sessionToken,
            callbacks: {
              onUserTranscript: ({ text, final }) => final && log("You: " + text),
              onAgentResponse: ({ text }) => log("Agent: " + text),
              onDisconnect: ({ reason }) => log("Call ended: " + reason),
            },
          });
          startButton.disabled = true;
          endButton.disabled = false;
        };

        endButton.onclick = () => session?.end();
        ```

        Run the two processes in separate terminals, then open the Vite URL (`http://localhost:5173`) and click **Start call**:

        ```bash Run theme={null}
        FISH_API_KEY=your_key AGENT_ID=your_agent_id node server.mjs
        npx vite
        ```

        Every request and response on this path is in the [Agents API reference](/api-reference/endpoint/agent/create-agent-session). For production, keep the same shape: your backend owns the API key and returns session tokens, and the SDK runs in the page.
      </Step>
    </Steps>
  </Tab>
</Tabs>

## Next steps

<CardGroup>
  <Card title="Core concepts" icon="sitemap" href="/agents/concepts">
    Agents, drafts, versions, and sessions: how the pieces fit together.
  </Card>

  <Card title="Web SDK" icon="code" href="/agents/deploy/web-sdk">
    Events, transcripts, client tools, and audio controls in the browser.
  </Card>

  <Card title="Agents API reference" icon="brackets-curly" href="/api-reference/endpoint/agent/create-agent">
    Every endpoint with full request and response schemas.
  </Card>

  <Card title="Knowledge base" icon="book" href="/agents/build/knowledge-base">
    Ground your agent in your own documents.
  </Card>

  <Card title="Conversation history" icon="clock-rotate-left" href="/agents/monitor/conversation-history">
    Review transcripts, tool calls, and recordings.
  </Card>
</CardGroup>


# Bring Your Own Number
Source: https://docs.fish.audio/agents/telephony/byo-sip

Import a number you already own by pointing your carrier's SIP trunk at Fish Audio: no rental fee, no telephony charges

If your numbers already live at a carrier, you can connect them to your agents without porting anything. Point the carrier's SIP trunk at Fish Audio and import the number: it stays with your carrier, who keeps billing you for the telephone-network legs, and on Fish Audio the calls bill as ordinary agent sessions. Imported numbers carry no monthly rental and no telephony charges of any kind: no phone surcharge, no transfer fees; the call bills like a web session.

This works with any carrier or PBX that speaks SIP trunking: Twilio Elastic SIP Trunking, Asterisk or FreePBX, and most SIP providers. It is also the only way to use non-US/CA numbers, which the purchasable inventory does not cover.

## How it works

A SIP trunk has two directions, and carriers name them from the carrier's point of view. **Origination** is traffic your carrier originates towards Fish Audio: an inbound call. **Termination** is traffic your carrier terminates for Fish Audio: an outbound call the platform hands to your carrier or PBX to complete. Twilio's trunk configuration uses the same two words, and so do the fields below.

* **Inbound (origination)**: your carrier routes calls for the number over its trunk to Fish Audio's SIP endpoint. The platform matches the dialed number and hands the call to the agent bound to it. From there it is a normal [inbound call](/agents/telephony/inbound-calls).
* **Outbound (termination)**, optional: give the import a termination host and the platform can also place calls from the number. Outbound calls and warm-transfer consult legs dial out through your trunk, with the imported number as the caller ID. You can also attach custom SIP headers that every call through the termination carries, for routing hints or tenant identification on your PBX.

The SIP endpoint to point your trunk at:

```text Origination URI theme={null}
sip:1pv316az391.sip.livekit.cloud;transport=tcp
```

## Prerequisites

* A number at your carrier and access to its SIP trunk configuration.
* Team owner or admin role (console), or an API key.

## Import the number

### In the console

On the workspace **Phone numbers** page, choose **Import number**. Enter the number in E.164 format, set at least one inbound authentication factor, and optionally fill in the **Outbound calling (termination)** section, including any **Custom SIP headers** (one `X-Name: value` per line). The same form is available later from the number's row menu as **Edit configuration**.

### Through the API

Import with the same endpoint that purchases numbers, using the `sip` provider variant:

```bash Request theme={null}
curl --request POST https://api.fish.audio/v1/agent/phone-numbers \
  --header "Authorization: Bearer $FISH_API_KEY" \
  --header "Content-Type: application/json" \
  --data '{
    "provider": "sip",
    "phone_number": "+14155550123",
    "label": "Main line (our carrier)",
    "inbound_auth_username": "fish-inbound",
    "inbound_auth_password": "long-random-password",
    "inbound_allowed_addresses": ["203.0.113.0/24"],
    "termination_uri": "pbx.example.com",
    "termination_transport": "tcp",
    "termination_auth_username": "fish-outbound",
    "termination_auth_password": "another-long-password",
    "termination_headers": { "X-Customer-Id": "acme" }
  }'
```

Returns `201` with the number object. Imported numbers land in your default workspace, like purchases.

| Field | Description |
| - | - |
| `provider` | Required: `sip`. |
| `phone_number` | Required: the E.164 number you own at the carrier. |
| `label` | Optional free-form label, up to 120 characters. |
| `agent_id` | Optional: bind an agent so the number answers right away. |
| `inbound_auth_username` | SIP digest username your trunk authenticates with; requires the password. |
| `inbound_auth_password` | The matching digest password. |
| `inbound_allowed_addresses` | IP addresses or CIDR ranges allowed to send calls for this number, up to 32. |
| `termination_uri` | Optional: your trunk's termination host as a bare `hostname[:port]`, without the `sip:` prefix. Enables outbound calls and warm transfers. |
| `termination_transport` | `auto` (default), `udp`, `tcp`, or `tls`. |
| `termination_auth_username` | Optional digest username for your termination; requires the password and a `termination_uri`. |
| `termination_auth_password` | The matching digest password. |
| `termination_headers` | Optional: custom `X-` headers for every call through the termination, up to 20; see [Custom SIP headers](#custom-sip-headers). |

At least one inbound factor (digest credentials and/or allowed addresses) is required; a `422` reports what is missing. A `409` means the number is already on the platform. A `502` means trunk provisioning failed; the number stays visible with status `error` and is safe to release and retry.

<Note>
  Digest passwords are stored only in the underlying trunk objects and are never
  echoed back by the API.
</Note>

## Inbound authentication

The SIP endpoint is shared, so an import must prove that calls really come from your trunk:

* **Digest credentials**: the platform challenges your trunk and verifies the username and password. Use this whenever your carrier or PBX answers digest challenges (Asterisk, FreePBX, most SIP providers).
* **Allowed source addresses**: calls are only accepted from the listed IPs or CIDR ranges. Use this for carriers that do not authenticate their origination traffic; Twilio Elastic SIP Trunking is one, so for Twilio this is the required factor.

Set both when your carrier supports it.

## Carrier walkthroughs

### Twilio Elastic SIP Trunking

<Steps>
  <Step title="Create a trunk">
    In the Twilio console, under **Elastic SIP Trunking**, create a trunk (or
    reuse an existing one).
  </Step>

  <Step title="Point origination at Fish Audio">
    Add an origination URI: `sip:1pv316az391.sip.livekit.cloud;transport=tcp`.
  </Step>

  <Step title="Attach your number">
    On the trunk's **Numbers** tab, add the phone number. Twilio routes its
    calls through the trunk from then on.
  </Step>

  <Step title="Import on Fish Audio">
    Twilio's origination does not answer digest challenges, so authenticate by
    source address: allow Twilio's published signaling IP ranges for the regions
    you use (see [Twilio's IP address
    list](https://www.twilio.com/docs/sip-trunking/ip-addresses)). In the
    console, the **Twilio Elastic SIP Trunking preset** button fills the ranges
    and sets the transport for you.
  </Step>

  <Step title="Enable outbound (optional)">
    On the trunk's **Termination** tab, note the termination SIP URI
    (`yourprefix.pstn.twilio.com`) and attach a **Credential List**. Pass the
    host as `termination_uri` and the credentials as `termination_auth_username`
    and `termination_auth_password`. Credentials are required here: Fish Audio's
    outbound traffic does not come from fixed IPs, so Twilio IP access control
    lists cannot authorize it.
  </Step>

  <Step title="Allow transfers (optional)">
    For [cold transfers](/agents/telephony/transfers), enable **Call Transfer
    (SIP REFER)** in the trunk's settings and also allow transfers to the PSTN.
    If transfers are limited to SIP destinations, Twilio rejects every transfer
    to a phone number, so transfers to phone numbers won't work. To choose the
    caller ID your team sees on a cold transfer, set **Caller ID for Transfer
    Target**: **Transferee** shows the caller's number, **Transferor** shows
    your imported number.
  </Step>
</Steps>

### Asterisk, FreePBX, and other SIP platforms

* Route the number's inbound calls to `sip:1pv316az391.sip.livekit.cloud;transport=tcp`.
* Configure digest credentials on the trunk and pass the same pair as `inbound_auth_username` and `inbound_auth_password`; add your PBX's public IPs to `inbound_allowed_addresses` for defense in depth.
* For outbound, expose a termination host reachable from the internet and pass it as `termination_uri`, with digest credentials if your PBX requires registration or authentication.

## Outbound calls and transfers

What an imported number can do depends on whether you configured a termination:

| | Inbound-only (no `termination_uri`) | With termination |
| - | - | - |
| **Inbound calls** | Yes | Yes |
| **Cold transfers** | Yes, when your carrier honors SIP REFER | Yes, when your carrier honors SIP REFER |
| **Warm transfers** | No | Yes; the consult leg dials through your trunk |
| **Outbound calls** | No | Yes; caller ID is the imported number |

The number object reports this as `supports_outbound`. Place calls with the same API as purchased numbers; see [Outbound calls](/agents/telephony/outbound-calls).

### Caller ID on transfers

Imported numbers don't have the [caller ID choice](/agents/telephony/transfers#caller-id-on-transfers) that bought numbers have. Instead:

* **Cold transfers** show whatever your carrier sends. On Twilio, you set this on the trunk with **Caller ID for Transfer Target** (see the [Twilio steps](#twilio-elastic-sip-trunking) above).
* **Warm transfers** always show your imported number.

### Custom SIP headers

`termination_headers` lets you tag every INVITE the platform sends through your termination, for example to identify the tenant or pick a route on your PBX:

```json theme={null}
"termination_headers": {
  "X-Customer-Id": "acme",
  "X-Route-Hint": "eu-west"
}
```

The headers ride along on outbound calls and on warm-transfer consult legs, since both dial through the termination. Cold transfers hand the call off on the inbound leg and do not carry them. Names must be `X-` tokens or `User-to-User`, so the request's own fields (`From`, `To`, `Via`) cannot be overridden, and `X-Fish-` is reserved for headers the platform sets itself. Values are 1 to 1024 printable ASCII characters (128 for `User-to-User`). Header names are case-insensitive, so a map with two spellings of the same name is rejected.

Values that change per call, such as a campaign or account id, go on the call instead: [`sip_headers`](/agents/telephony/outbound-calls#per-call-sip-headers) on the outbound call request is merged over these number-level headers. For matching calls to sessions you do not need a header of your own; see [Match calls with your SIP records](#match-calls-with-your-sip-records).

## Match calls with your SIP records

Your carrier, SBC or PBX keeps its own record of every call: a CDR, a log line, a recording file. Two identifiers let you line those records up with Fish Audio sessions, one in each direction, and neither needs any configuration.

| Identifier | What it is | Where you read it |
| - | - | - |
| `sip_call_id` | The SIP `Call-ID` of the call's INVITE, the value the SIP world uses as the identity of one call. Your SIP records carry it already. | On the session, from `GET /v1/agent/sessions/{session_id}`, as a filter on the sessions list (`?sip_call_id=`), and in the `session` object of every post-call webhook. |
| `X-Fish-Session-Id` | The Fish Audio session id, sent as a header on every INVITE and REFER the platform places through your trunk. | In your own SIP records, if your SBC or PBX logs custom headers. |

Which one you use depends on where you start:

* **From a Fish Audio webhook to your records**: read `sip_call_id` from the webhook's `session` object and look it up in your CDR. This works without any change on your side, because every SIP system already records `Call-ID`.
* **From your records to Fish Audio**: take the `Call-ID` from your CDR and list sessions with `GET /v1/agent/sessions?sip_call_id=…`, which returns the matching session. If your SBC logs custom headers, `X-Fish-Session-Id` from the INVITE gives you the session id directly.

When each identifier is present:

| Call | `sip_call_id` on the session | `X-Fish-Session-Id` in the INVITE |
| - | - | - |
| Inbound call | Yes, from your carrier's INVITE | Not applicable: the INVITE is yours |
| Outbound call, answered | Yes, from the INVITE the platform sent | Yes |
| Outbound call, not answered | `null`: no call was completed | Yes, on the INVITE that was refused |
| Warm transfer consult leg | Not exposed separately | Yes, on the consult INVITE |
| Cold transfer to a SIP target | Not applicable | Yes, as a header on the REFER |

The `X-Fish-Session-Id` header is only sent through your own trunk. Calls from platform-purchased numbers terminate on the platform's carrier and carry no such header, but `sip_call_id` is still exposed for them.

For your own business keys, such as a campaign or tenant id, add [custom SIP headers](#custom-sip-headers) on the number or [per call](/agents/telephony/outbound-calls#per-call-sip-headers). They complement the two identifiers above rather than replace them: a business key tells you which campaign a call belonged to, the identifiers tell you which call it was.

A `phone_call.dial_finished` payload carries everything needed for the webhook-to-CDR direction:

```json theme={null}
{
  "event": "phone_call.dial_finished",
  "dial_status": "answered",
  "session": {
    "id": "5f3e…",
    "caller_number": "+15551234567",
    "dialed_number": "+14155550100",
    "sip_call_id": "7f3a9c@sbc.example.com",
    "metadata": { "lead_ref": "L-2041" }
  }
}
```

## Update the configuration

Change any part of an imported number's trunk configuration in place: rotate digest credentials, adjust the allowed addresses, or add, change, and remove the termination. In the console, open **Edit configuration** from the number's row menu; over the API, `PUT` the full desired configuration:

```bash Request theme={null}
curl --request PUT https://api.fish.audio/v1/agent/phone-numbers/$PHONE_NUMBER_ID/sip-config \
  --header "Authorization: Bearer $FISH_API_KEY" \
  --header "Content-Type: application/json" \
  --data '{
    "inbound_auth_username": "fish-inbound",
    "inbound_auth_password": "",
    "inbound_allowed_addresses": ["203.0.113.0/24", "198.51.100.7"],
    "termination_uri": "pbx.example.com",
    "termination_transport": "tls",
    "termination_auth_username": "fish-outbound",
    "termination_auth_password": "",
    "termination_headers": { "X-Customer-Id": "acme" }
  }'
```

The body is the import payload without `phone_number`, `label`, and `agent_id`, and it replaces the whole configuration. Three things to know:

* An empty password next to a set username keeps the stored password, so you can edit other fields without re-entering secrets. Setting a username for the first time requires a password.
* An empty `termination_uri` removes the termination and makes the number inbound-only again.
* `termination_headers` replaces the stored map; omit it or send `{}` to remove all custom headers.

Updates apply in place: routing is never interrupted, and calls already in progress are unaffected. The response is the updated number object; for imported numbers it carries the non-secret configuration (`inbound_auth_username`, `inbound_allowed_addresses`, `termination_uri`, `termination_transport`, `termination_auth_username`, `termination_headers`) alongside `supports_outbound`.

## Billing

Imported numbers are free on Fish Audio: no monthly rental, no phone surcharge, no transfer fees. Calls on them bill like web sessions (see [Pricing](/agents/pricing)). Your carrier continues to bill you directly for its side of the traffic.

## Release

Releasing an imported number (`DELETE /v1/agent/phone-numbers/$PHONE_NUMBER_ID`) disconnects it from Fish Audio and returns `204`. The number itself stays yours at the carrier; you can import it again later.

## Going further

<CardGroup>
  <Card title="Inbound calls" icon="phone-arrow-down-left" href="/agents/telephony/inbound-calls">
    What happens when someone dials a bound number.
  </Card>

  <Card title="Call transfers" icon="phone-arrow-right" href="/agents/telephony/transfers">
    Cold and warm handoffs to a human.
  </Card>

  <Card title="Phone numbers" icon="phone" href="/agents/telephony/phone-numbers">
    The shared lifecycle: list, bind, label, release.
  </Card>

  <Card title="API introduction" icon="key" href="/api-reference/introduction">
    API keys and workspace scoping for every request.
  </Card>
</CardGroup>


# Inbound Calls
Source: https://docs.fish.audio/agents/telephony/inbound-calls

Bind an agent to a phone number and it answers every call to that number

Point a phone number at an agent and it picks up every inbound call. Phone calls are ordinary agent sessions: they appear in session history with caller attribution, they are [stored](/agents/monitor/conversation-history#what-gets-stored) and analyzed under the same per-agent settings as any other conversation, and they trigger the same webhooks.

<CardGroup>
  <Card title="Phone numbers" icon="hashtag" href="/agents/telephony/phone-numbers">
    Get a number and manage its agent binding.
  </Card>

  <Card title="Conversation history" icon="clock-rotate-left" href="/agents/monitor/conversation-history">
    Review transcripts, tool calls, and recordings.
  </Card>

  <Card title="Webhooks" icon="webhook" href="/agents/monitor/webhooks">
    Get notified when calls end and analysis completes.
  </Card>
</CardGroup>

## Set up inbound calling

<Steps>
  <Step title="Get a phone number">
    Add a number to your workspace; see [Phone numbers](/agents/telephony/phone-numbers). Numbers are stored and displayed in E.164 format (for example `+15551234567`).
  </Step>

  <Step title="Publish your agent">
    Calls connect to the agent's published configuration, not the draft. [Publish](/agents/deploy/versions-publishing) before pointing traffic at the number.
  </Step>

  <Step title="Bind the agent to the number">
    Bind in the console, or set `agent_id` on the number via the API:

    <CodeGroup>
      ```bash Bind an agent theme={null}
      curl --request PATCH https://api.fish.audio/v1/agent/phone-numbers/PHONE_NUMBER_ID \
        --header "Authorization: Bearer $FISH_API_KEY" \
        --header "Content-Type: application/json" \
        --data '{ "agent_id": "YOUR_AGENT_ID" }'
      ```

      ```bash Unbind theme={null}
      curl --request PATCH https://api.fish.audio/v1/agent/phone-numbers/PHONE_NUMBER_ID \
        --header "Authorization: Bearer $FISH_API_KEY" \
        --header "Content-Type: application/json" \
        --data '{ "agent_id": null }'
      ```
    </CodeGroup>
  </Step>
</Steps>

<Tip>
  The number-to-agent binding is resolved on each incoming call, so a rebind
  takes effect on the next call, useful for moving a number from a staging
  agent to a production agent.
</Tip>

## Phone sessions in history

Every answered call becomes a session with `source: "phone"` and these attribution fields:

| Field | Description |
| - | - |
| `caller_number` | The caller's number, E.164. `null` for non-phone sessions. |
| `dialed_number` | The workspace number that was called, E.164. `null` for non-phone sessions. |
| `sip_call_id` | The SIP `Call-ID` of the carrier's INVITE, for [matching your SIP records](/agents/telephony/byo-sip#match-calls-with-your-sip-records). |

List calls with the standard sessions endpoint. The `caller_number` filter is an exact match; bare numbers are automatically prefixed with `+`, so `15551234567` matches `+15551234567`:

```bash theme={null}
curl --request GET "https://api.fish.audio/v1/agent/sessions?agent_id=YOUR_AGENT_ID&caller_number=15551234567" \
  --header "Authorization: Bearer $FISH_API_KEY"
```

```json theme={null}
{
  "sessions": [
    {
      "session_id": "9c41f0d2e8a34b7f",
      "agent_id": "YOUR_AGENT_ID",
      "agent_name": "Support line",
      "status": "completed",
      "source": "phone",
      "caller_number": "+15551234567",
      "dialed_number": "+14155550100",
      "created_at": "2026-07-23T09:14:01Z",
      "started_at": "2026-07-23T09:14:02Z",
      "ended_at": "2026-07-23T09:16:45Z",
      "duration_seconds": 163,
      "metadata": {}
    }
  ],
  "has_more": false,
  "next_cursor": null
}
```

You can combine `caller_number` with the other list filters (`agent_id`, `status`, `created_after`, `created_before`); see [Conversation history](/agents/monitor/conversation-history) for the full parameter list.

The agent can read both numbers too, as `{{system.caller_number}}` and `{{system.dialed_number}}` in its system prompt, first message, and webhook tool requests. No variable needs to be passed; see [System variables](/agents/build/dynamic-variables#system-variables).

```text System prompt theme={null}
The caller is calling from {{system.caller_number}}. Confirm the last four digits before discussing the account.
```

## Personalize calls before the agent answers

Inbound calls have no session-creation request to carry [dynamic variables](/agents/build/dynamic-variables), so the agent has a dedicated hook instead: the [inbound call webhook](/agents/monitor/webhooks#inbound-call-webhook). When configured, the platform posts the caller's number to your endpoint while the call is being set up, and the `dynamic_variables` you return fill the `{{placeholders}}` in the published prompt and first message before the agent speaks.

```text First message theme={null}
Hi {{customer_name}}, thanks for calling Acme. Are you calling about order {{last_order}}?
```

Set it on the agent's **Webhooks** page in the console (**Inbound call webhook**) or through `webhooks.conversation_init` in the config API, then publish. A slow or failing endpoint never blocks the call: after the configured timeout (up to 8 seconds, default 5) the call connects without variables.

## Phone sessions behave like any session

Nothing about a phone call needs special handling downstream:

* **Transcript and tool timeline**: `GET /v1/agent/sessions/{session_id}` returns the conversation, including tool calls and results, when the agent [stores transcripts](/agents/monitor/conversation-history#what-gets-stored). See [Conversation history](/agents/monitor/conversation-history).
* **Recording**: `GET /v1/agent/sessions/{session_id}/recording` returns signed URLs, one track per speaker, when the session was recorded.
* **Post-call analysis**: summaries, data fields, and evaluation criteria run on phone sessions like any other. See [Post-call analysis](/agents/monitor/post-call-analysis).
* **Webhooks**: `call.ended` and `call.analyzed` fire for phone sessions too. See [Webhooks](/agents/monitor/webhooks).
* **Hang up via API**: end an in-progress call with `POST /v1/agent/sessions/{session_id}/end`.

## Going further

<CardGroup>
  <Card title="Phone numbers" icon="hashtag" href="/agents/telephony/phone-numbers">
    Number management and agent bindings.
  </Card>

  <Card title="Versions & publishing" icon="code-branch" href="/agents/deploy/versions-publishing">
    Control which configuration answers your calls.
  </Card>

  <Card title="Post-call analysis" icon="chart-simple" href="/agents/monitor/post-call-analysis">
    Extract structured data from every call.
  </Card>

  <Card title="Webhooks" icon="webhook" href="/agents/monitor/webhooks">
    React to call lifecycle events from your backend.
  </Card>
</CardGroup>


# Outbound Calls
Source: https://docs.fish.audio/agents/telephony/outbound-calls

Place calls from your phone numbers over the API; the agent speaks when the callee answers

Dial any allowed number from one of your workspace numbers and the agent takes the call the moment the callee picks up. Outbound calls are ordinary agent sessions with `direction: "outbound"`: they appear in session history, they are [stored](/agents/monitor/conversation-history#what-gets-stored) and analyzed under the same per-agent settings as any other conversation, and they trigger the same webhooks plus one extra, [`phone_call.dial_finished`](/agents/monitor/webhooks), that reports how the dial attempt ended.

<CardGroup>
  <Card title="Phone numbers" icon="hashtag" href="/agents/telephony/phone-numbers">
    Get a number that can place calls.
  </Card>

  <Card title="Webhooks" icon="webhook" href="/agents/monitor/webhooks">
    Get the dial outcome pushed to your backend.
  </Card>

  <Card title="Dynamic variables" icon="brackets-curly" href="/agents/build/dynamic-variables">
    Personalize the call with per-call values.
  </Card>
</CardGroup>

## Place a call

<Steps>
  <Step title="Get a number that supports outbound">
    Any [purchased number](/agents/telephony/phone-numbers) can place calls. An [imported BYO number](/agents/telephony/byo-sip) can too, once its termination is configured; the number object reports this as `supports_outbound`. The number you dial from is the caller ID the callee sees.
  </Step>

  <Step title="Publish your agent">
    Outbound calls run the agent's published configuration, not the draft. [Publish](/agents/deploy/versions-publishing) before dialing.
  </Step>

  <Step title="Create the phone call">
    ```bash theme={null}
    curl --request POST https://api.fish.audio/v1/agent/phone-calls \
      --header "Authorization: Bearer $FISH_API_KEY" \
      --header "Content-Type: application/json" \
      --header "Idempotency-Key: order-4711-reminder-1" \
      --data '{
        "agent_id": "YOUR_AGENT_ID",
        "phone_number_id": "YOUR_PHONE_NUMBER_ID",
        "to_number": "+14155550123"
      }'
    ```

    The request returns `201` as soon as the dial is dispatched:

    ```json theme={null}
    { "session_id": "9c41f0d2e8a34b7f", "status": "queued" }
    ```
  </Step>
</Steps>

This endpoint requires an API key; there is no anonymous variant. See the [API reference](/api-reference/endpoint/agent/create-phone-call) for the full schema.

### Request fields

| Field | Description |
| - | - |
| `agent_id` | Required: the agent that speaks on the call. Must have a published version. |
| `phone_number_id` | Required: the workspace number to dial from. |
| `to_number` | Required: the destination in E.164, for example `+14155550123`. |
| `dynamic_variables` | Optional: per-call values for `{{placeholders}}` in the agent's configured text, same rules as [session creation](/agents/build/dynamic-variables). Up to 50 entries. |
| `overrides` | Optional: replace whole configuration fields for this call, subject to the agent's [override allowlist](/agents/deploy/authenticated-sessions#overrides). |
| `metadata` | Optional: your own JSON object, returned verbatim on session reads and in webhook payloads. Never interpreted. |
| `llm_extra_body` | Optional: JSON object (at most 16 KB) forwarded to a [custom LLM](/agents/build/custom-llm) endpoint on every request as `fishaudio_extra_body`. Ignored on platform-model agents. |
| `sip_headers` | Optional, [imported SIP numbers](/agents/telephony/byo-sip) only: custom headers for this call's INVITE. See [Per-call SIP headers](#per-call-sip-headers). |

The session's [time and timezone context](/agents/build/time-timezone) resolves from the destination number when the agent has no fixed timezone configured, so "tomorrow morning" means the callee's morning.

### Per-call SIP headers

When the from-number is an imported SIP number, `sip_headers` adds headers to the INVITE the platform sends through your termination, so your SBC or PBX can route the call or tag its CDR with a value that changes per call: a campaign id, an account id, a consent reference.

```json theme={null}
"sip_headers": {
  "X-Campaign-Id": "spring-24",
  "X-Account-Id": "acct_8812"
}
```

They follow the same rules as the number's [`termination_headers`](/agents/telephony/byo-sip#custom-sip-headers) and are merged over them, so a per-call header replaces a number-level header of the same name. The same headers ride on a warm-transfer consult leg placed during the call. Every INVITE through your termination also carries `X-Fish-Session-Id` with the session id, so the SIP side can be joined to the platform side without any header of your own.

Platform-purchased numbers refuse `sip_headers` with `422 sip_headers_unsupported`: the shared trunk terminates on the carrier and nothing downstream would read them. `sip_headers` is not `metadata`: it leaves the platform for your SIP network, while `metadata` is only echoed back to you.

### Retry safely with an Idempotency-Key

Outbound dials spend money and ring real phones, so put an `Idempotency-Key` header on every create. For 24 hours, repeating the same key with the same body returns the call already placed instead of dialing again. The same key with a **different** body is refused with `422 idempotency_key_reuse`, and a retry that races an in-flight first attempt gets `409 idempotency_key_conflict`; back off and retry the same request. If the create fails with an ambiguous network error, retry with the same key: you get the placed session back if the first attempt went through.

## The dial outcome

Ringing is never billed; metering starts when the callee answers. A call that is never answered is not billed and not analyzed.

You learn how the dial ended in either of two ways:

* **Push**: subscribe to the [`phone_call.dial_finished` webhook](/agents/monitor/webhooks). It fires exactly once per outbound call, whether or not anyone picked up, before `call.ended`.
* **Poll**: read `GET /v1/agent/sessions/{session_id}` until `dial_status` is set.

```bash theme={null}
curl https://api.fish.audio/v1/agent/sessions/9c41f0d2e8a34b7f \
  --header "Authorization: Bearer $FISH_API_KEY"
```

Two session fields carry the outcome:

| Field | Values |
| - | - |
| `dial_status` | `answered`, `busy`, `no_answer`, or `failed`. `null` while the call is still ringing, and on inbound calls. |
| `answered_by` | Deprecated. Answering-machine detection has been removed, so new calls report `unknown` once answered (`null` before). Older sessions may still carry `human` or `voicemail`. |

Once answered, `sip_call_id` holds the SIP `Call-ID` of the leg the platform placed. It is `null` while ringing and on unanswered dials. See [Match calls with your SIP records](/agents/telephony/byo-sip#match-calls-with-your-sip-records).

On outbound sessions the attribution fields are person-centric: `caller_number` is the human you dialed and `dialed_number` is your workspace number. The same values reach the agent as `{{system.caller_number}}` and `{{system.dialed_number}}`, so a CRM lookup tool can use `https://crm.example.com/contacts?phone={{system.caller_number}}` without a per-call variable; see [System variables](/agents/build/dynamic-variables#system-variables). Everything else about the session (transcript, recording, [post-call analysis](/agents/monitor/post-call-analysis), `call.ended` and `call.analyzed` webhooks, hang-up via `POST /v1/agent/sessions/{session_id}/end`) works exactly as for [inbound calls](/agents/telephony/inbound-calls).

## Allowed destinations

Calls from purchased numbers can reach fixed-line and mobile numbers in the US, Canada, Japan, the UK, Italy, Australia and Mexico. Premium-rate numbers (for example UK `09`, Japan `0570` and `0990`, Australia `19`, Mexico `900`) and other service numbers such as toll-free lines are refused. The phone surcharge depends on the country and on whether the number is fixed-line or mobile; see the [pricing page](/agents/pricing).

Calls from [imported BYO numbers](/agents/telephony/byo-sip) dial out through your own trunk, so the country allowlist does not apply.

Numbers that live on the platform can never be dialed, so an agent cannot call another agent's number.

## Errors

Unlike most [Agents API errors](/api-reference/agent-errors), phone-call errors carry a machine-readable `reason` alongside the `message`:

```json theme={null}
{
  "status": 422,
  "reason": "destination_not_allowed",
  "message": "Outbound calls are limited to numbers in the US, Canada, Japan, the UK, Italy, Australia and Mexico"
}
```

| Status | Reason | Cause |
| - | - | - |
| `402` | `insufficient_credit` | Out of API credit; top up first. |
| `403` | `outbound_disabled` | Outbound calling is switched off for this workspace. |
| `404` | `number_not_found`, `agent_not_found` | The id doesn't exist, isn't yours, or the number was released. |
| `409` | `agent_not_published` | [Publish](/agents/deploy/versions-publishing) the agent first. |
| `409` | `idempotency_key_conflict` | A request with this key is still in flight; back off and retry the same request. |
| `422` | `number_provider_unsupported`, `number_termination_missing`, `number_inactive` | The from-number can't place calls; see [which numbers support outbound](/agents/telephony/byo-sip#outbound-calls-and-transfers). |
| `422` | `sip_headers_unsupported` | `sip_headers` was sent with a platform-purchased number. See [per-call SIP headers](#per-call-sip-headers). |
| `422` | `destination_invalid`, `destination_not_allowed`, `premium_destination_blocked`, `self_call_blocked` | The destination is refused; see [allowed destinations](#allowed-destinations). |
| `422` | `idempotency_key_reuse` | The key was already used with a different body; mint a fresh key per distinct call. |

`502` and `503` follow the [general guidance](/api-reference/agent-errors#5xx): retrying is safe, especially with an `Idempotency-Key`.

## Going further

<CardGroup>
  <Card title="Webhooks" icon="webhook" href="/agents/monitor/webhooks">
    The `phone_call.dial_finished` payload in full.
  </Card>

  <Card title="Transfers" icon="arrow-right-arrow-left" href="/agents/telephony/transfers">
    Hand an answered call to a human; consult legs dial from the same number.
  </Card>

  <Card title="Bring your own SIP numbers" icon="server" href="/agents/telephony/byo-sip">
    Place calls through your own trunk with your number as caller ID.
  </Card>

  <Card title="Conversation history" icon="clock-rotate-left" href="/agents/monitor/conversation-history">
    Transcripts, recordings, and session listing filters.
  </Card>
</CardGroup>


# Phone Numbers
Source: https://docs.fish.audio/agents/telephony/phone-numbers

Search, purchase, bind, and release phone numbers for your agents over the REST API

Phone numbers connect your agents to the telephone network. Each number lives in a workspace within your team (purchases land in your default workspace) and is bound to at most one agent: calls to the number are answered by that agent. The `/v1/agent/phone-numbers` API covers the whole lifecycle: search the purchasable inventory, buy a number (or [import one you already own](/agents/telephony/byo-sip)), bind it to an agent, and release it when you no longer need it.

## Search available numbers

Search the purchasable inventory for a number to buy.

```bash Request theme={null}
curl "https://api.fish.audio/v1/agent/available-phone-numbers?country_code=US&area_code=415" \
  --header "Authorization: Bearer $FISH_API_KEY"
```

The response is an `available_phone_numbers` array of inventory entries forwarded from the provider: number and region.

| Parameter | Description |
| - | - |
| `country_code` | ISO 3166-1 alpha-2 country code, `US` (default) or `CA`. |
| `area_code` | Restrict results to one area code, like `415`. |
| `number_type` | `local` (default). The managed inventory is US and Canada local numbers only; `toll_free` returns `400`. |
| `provider` | Inventory to search. Only `twilio` (the default) is available: managed numbers with [call transfer](/agents/telephony/transfers) support. |

<Note>
  A listed number can still be claimed by someone else before you buy it.
</Note>

## Purchase a number

Buy a number from the inventory with the same `provider` that listed it.

```bash Request theme={null}
curl --request POST https://api.fish.audio/v1/agent/phone-numbers \
  --header "Authorization: Bearer $FISH_API_KEY" \
  --header "Content-Type: application/json" \
  --data '{
    "provider": "twilio",
    "phone_number": "+14155550123",
    "label": "Support line (US)",
    "agent_id": "4f2a8c6e1b3d5f7a9c0e2d4b6a8f1c3e"
  }'
```

Returns `201` with the number object, the same shape the list endpoint returns. The number lands in your default workspace, and its monthly price is billed in daily slices; see [Pricing](/agents/pricing).

| Field | Description |
| - | - |
| `provider` | Required: `twilio`, the inventory the number came from. |
| `phone_number` | Required: an E.164 number from the search response. |
| `label` | Optional free-form label, up to 120 characters. |
| `agent_id` | Optional: bind an agent so the number answers inbound calls right away. |

A `409` means the number is already on the platform; a `502` means the provider refused the purchase; the number stays visible with status `error` and is safe to release.

<Note>
  Already have a number at your own carrier? The same endpoint imports it with
  `provider: "sip"` instead of buying one: no rental fee, no telephony charges.
  See [Bring your own number](/agents/telephony/byo-sip).
</Note>

## List your numbers

Returns your team's phone numbers across its workspaces, newest first; each carries its `workspace_id`.

```bash Request theme={null}
curl https://api.fish.audio/v1/agent/phone-numbers \
  --header "Authorization: Bearer $FISH_API_KEY"
```

```json Response theme={null}
{
  "phone_numbers": [
    {
      "phone_number_id": "7c9e6f2a4b8d4e1f9a3c5d7e8f0b1a2c",
      "phone_number": "+14155550123",
      "provider": "twilio",
      "label": "Support line (US)",
      "agent_id": "4f2a8c6e1b3d5f7a9c0e2d4b6a8f1c3e",
      "status": "active"
    }
  ],
  "has_more": false
}
```

| Field | Description |
| - | - |
| `phone_number_id` | Unique identifier; use it in URL paths |
| `phone_number` | The number in E.164 format |
| `provider` | Telephony provider backing the number |
| `label` | Free-form label you assign |
| `agent_id` | The agent that answers calls to this number; `null` when unbound |
| `status` | `provisioning`, `active`, or `error` |

## Get a single number

```bash Request theme={null}
curl https://api.fish.audio/v1/agent/phone-numbers/$PHONE_NUMBER_ID \
  --header "Authorization: Bearer $FISH_API_KEY"
```

Returns the same number object as the list endpoint.

## Bind or unbind an agent

`PATCH` the number with an `agent_id` to change which agent answers it. Binding is resolved per call, so the change applies from the next inbound call.

<CodeGroup>
  ```bash Bind theme={null}
  curl --request PATCH https://api.fish.audio/v1/agent/phone-numbers/$PHONE_NUMBER_ID \
    --header "Authorization: Bearer $FISH_API_KEY" \
    --header "Content-Type: application/json" \
    --data '{ "agent_id": "4f2a8c6e1b3d5f7a9c0e2d4b6a8f1c3e" }'
  ```

  ```bash Unbind theme={null}
  curl --request PATCH https://api.fish.audio/v1/agent/phone-numbers/$PHONE_NUMBER_ID \
    --header "Authorization: Bearer $FISH_API_KEY" \
    --header "Content-Type: application/json" \
    --data '{ "agent_id": null }'
  ```
</CodeGroup>

Omitted fields are left unchanged. Only an explicit `"agent_id": null` unbinds the number.

<Tip>
  Re-binding is how you promote an agent behind a stable number: point the
  number at a staging agent while you iterate, then `PATCH` it over to the
  production agent when you're ready. The number itself never changes.
</Tip>

## Label your numbers

Labels are free-form text for keeping an inventory readable: by team, region, campaign, or environment.

```bash Request theme={null}
curl --request PATCH https://api.fish.audio/v1/agent/phone-numbers/$PHONE_NUMBER_ID \
  --header "Authorization: Bearer $FISH_API_KEY" \
  --header "Content-Type: application/json" \
  --data '{ "label": "Support line (US)" }'
```

The response is the updated number object.

## Release a number

Release a number back to the provider's inventory. Returns `204`, daily billing stops, and the number disappears from the API immediately.

```bash Request theme={null}
curl --request DELETE https://api.fish.audio/v1/agent/phone-numbers/$PHONE_NUMBER_ID \
  --header "Authorization: Bearer $FISH_API_KEY"
```

<Warning>
  Releasing is irreversible. Anyone, including other platforms, can buy the
  number afterwards, so callers who saved it may reach a stranger.
</Warning>

## Going further

<CardGroup>
  <Card title="Inbound calls" icon="phone-arrow-down-left" href="/agents/telephony/inbound-calls">
    What happens when someone dials a bound number.
  </Card>

  <Card title="API introduction" icon="key" href="/api-reference/introduction">
    API keys and workspace scoping for every request.
  </Card>

  <Card title="Conversation history" icon="clock-rotate-left" href="/agents/monitor/conversation-history">
    Review phone sessions, transcripts, and recordings.
  </Card>

  <Card title="Versions & publishing" icon="rocket" href="/agents/deploy/versions-publishing">
    Publish configuration changes behind a stable number.
  </Card>
</CardGroup>


# Call Transfers
Source: https://docs.fish.audio/agents/telephony/transfers

Let the agent hand a phone call to a human: a cold carrier handoff, or a warm transfer with hold music and a private briefing

When a caller needs a person, the agent can transfer the call. Configure one or more destination numbers and the agent gains a built-in `transfer_call` tool it invokes when the caller asks for a human or the request clearly needs one. With several destinations, the agent picks the one whose name and description best match the conversation, so a single number can act as a switchboard. Transfers apply to phone calls only.

Two modes decide what the handoff feels like:

| | Cold | Warm |
| - | - | - |
| **Handoff** | The agent announces the transfer and drops out; the carrier connects the caller to the destination. | The caller waits on hold while the agent briefs the human on a private call, then the two are connected. |
| **What the human hears first** | The caller, directly. | The agent's briefing: who is calling and what they need. |
| **Recording & analysis** | Stop at the handoff: nothing after it appears in [post-call analysis](/agents/monitor/post-call-analysis). | Recording continues after the human joins, capturing the full call audio; the transcript (and therefore analysis) covers only the agent's segment, up to the merge. |

## Prerequisites

* A phone number bound to your agent, answering inbound calls. See [Inbound calls](/agents/telephony/inbound-calls).
* The number must support transfers. Numbers purchased from the platform inventory (`provider: "twilio"`) do; see [Phone numbers](/agents/telephony/phone-numbers). [Imported SIP numbers](/agents/telephony/byo-sip) support cold transfers when the carrier honors SIP REFER to the destination (on Twilio, enable Call Transfer and allow PSTN transfers on the trunk), and warm transfers when a termination is configured.

## Configure destinations

### In the console

On the agent's **Phone** page, under **Inbound calls**, turn on **Call transfer**. Give the destination a name, enter its phone number, describe when the agent should transfer there, choose its [announcement](#the-transfer-announcement), and pick the transfer mode. Choosing **Warm** adds a handoff choice (ask the human to accept first, or connect immediately after the briefing). **Add destination** adds another card; with more than one, every destination needs a distinct name. Like every config edit, the change lands in the agent's draft and takes effect on live calls after you [publish](/agents/deploy/versions-publishing).

### Through the API

Transfer destinations live in the `conversation` section of the [agent config](/agents/build/configuration), as `transfer_destinations`. An empty list disables transfers; the list is replaced as a whole on every write.

<CodeGroup>
  ```bash Enable theme={null}
  curl --request PATCH "https://api.fish.audio/v1/agent/agents/$AGENT_ID/config" \
    --header "Authorization: Bearer $FISH_API_KEY" \
    --header "Content-Type: application/json" \
    --data '{
      "conversation": {
        "transfer_destinations": [
          {
            "label": "Support desk",
            "phone_number": "+14155550123",
            "mode": "warm",
            "warm_connect": "confirm"
          }
        ]
      }
    }'
  ```

  ```bash Route by context theme={null}
  curl --request PATCH "https://api.fish.audio/v1/agent/agents/$AGENT_ID/config" \
    --header "Authorization: Bearer $FISH_API_KEY" \
    --header "Content-Type: application/json" \
    --data '{
      "conversation": {
        "transfer_destinations": [
          {
            "label": "Billing",
            "description": "Account information, invoices, payments and refunds",
            "phone_number": "+14155550100",
            "mode": "cold"
          },
          {
            "label": "Tech support",
            "description": "Troubleshooting, outages, anything not working",
            "phone_number": "+14155550200",
            "mode": "warm",
            "warm_connect": "confirm"
          }
        ]
      }
    }'
  ```

  ```bash Disable theme={null}
  curl --request PATCH "https://api.fish.audio/v1/agent/agents/$AGENT_ID/config" \
    --header "Authorization: Bearer $FISH_API_KEY" \
    --header "Content-Type: application/json" \
    --data '{ "conversation": { "transfer_destinations": [] } }'
  ```
</CodeGroup>

| Field | Description |
| - | - |
| `type` | Destination type; `phone` is the only value today and the default. |
| `label` | Name for the destination, shown in the Builder and used by the agent to pick one. At most 64 characters. Required, and unique (case-insensitive), once there is more than one entry. |
| `description` | When to transfer here, in plain language ("Invoices, payments, account balance"). The agent reads it when choosing between destinations. At most 500 characters. |
| `phone_number` | The destination in E.164 format, for example `+14155550123`. |
| `mode` | `cold` (default) or `warm`; see the comparison above. |
| `warm_connect` | Warm only: `confirm` (default) waits for the human's go-ahead before connecting; `direct` connects right after the briefing. |
| `announcement_mode` | The sentence the agent says right before transferring, cold or warm: `generated` (default) words it itself, `fixed` says `announcement_message`, `off` adds none. See [The transfer announcement](#the-transfer-announcement). |
| `announcement_message` | The line `fixed` says, at most 500 characters. [Dynamic variables](/agents/build/dynamic-variables) are filled in per call. It is kept when you switch modes, and an empty message falls back to `generated`. |
| `warm_briefing_instructions` | Warm only: what the agent says to your team member before asking them to take the call, replacing the default introduction. Notes from the conversation are always added after it, and [dynamic variables](/agents/build/dynamic-variables) are filled in per call. At most 1000 characters. See [Customizing the briefing](#customizing-the-briefing). |

Destination numbers must be E.164 and are limited to a set of supported countries. A number outside the list is rejected with `422 Unprocessable Entity` and an error naming the allowed country codes. The transfer card in the console shows the current list. A missing label, or two labels that differ only in case, is rejected the same way. There is no cap on the number of destinations, but every description is part of the agent's tool instructions on every turn, so keep the list to what a receptionist could hold in their head.

## How the agent decides to transfer

Configuring a destination is what enables the built-in `transfer_call` tool: an empty destination list removes it. Unlike the toggled [system tools](/agents/build/system-tools), you won't find it in the `tools` config section; it ships automatically on phone sessions whenever a valid destination exists.

Your system prompt decides when the agent transfers. Only when it says nothing about transfers does the built-in default apply, which is conservative: the agent transfers when the caller asks for a person, or when the request clearly needs one and the agent cannot help further. A switchboard agent should say so plainly, for example "Route the caller to the right team as soon as you know what they need; do not try to answer billing or technical questions yourself."

With several destinations, the tool gains a `destination` argument listing every label, and its instructions carry each destination's description. The agent picks the destination whose description matches what the caller needs based on the conversation so far, asks one short question when the need is unclear, and otherwise takes the closest match. Write descriptions the way you would brief a receptionist: what the team handles, in the caller's words.

<Tip>
  The prompt can loosen the policy as well as tighten it: "Transfer to a human
  whenever the caller mentions a refund" or "Never transfer before collecting
  the caller's name and account number." See [Tools](/agents/build/tools) for
  how the agent chooses between its tools.
</Tip>

## The transfer announcement

Right before a call is transferred, the agent says one short sentence so the caller knows what is happening, for example "One moment, I'm connecting you to billing." The **Transfer announcement** setting on each destination chooses where that sentence comes from.

| Option | What the agent says before transferring | Use it when |
| - | - | - |
| **Generated** (`generated`, default) | A short sentence in its own words, in the caller's language. If your system prompt says what to say when transferring, it says that instead. | You want a natural line, or your system prompt already scripts it. |
| **Fixed message** (`fixed`) | The text in `announcement_message`, word for word. | Every caller should hear the same line. |
| **Off** (`off`) | No extra sentence. The transfer starts as soon as the agent finishes its current reply. | You don't want transfers announced. |

These apply to every option:

* The sentence is said only once. If the agent already spoke in the same reply as the transfer, nothing is added. For example, if your system prompt has the agent say "Thanks, connecting you now." itself, the caller hears that line and the fixed message is skipped. With **Fixed message** or **Off**, remove any instruction in your system prompt about what to say when transferring.
* The caller cannot interrupt the sentence.
* Once the transfer starts, the caller can no longer answer, so the agent asks its questions before transferring. Don't put a question in a fixed message either.

In the console, the control sits on each destination card, above the transfer mode. **Fixed message** shows a field for the line.

## What happens on a cold transfer

The agent says its [announcement](#the-transfer-announcement) and, once that finishes playing, the call is handed to the carrier. The caller hears a dial tone while the destination rings; the agent drops out and its session ends at the handoff.

If the handoff is refused (the destination is unreachable, or the number doesn't support transfers), the agent stays on the line, apologizes, and keeps helping. It can attempt the transfer again later.

## What happens on a warm transfer

<Steps>
  <Step title="The caller goes on hold">
    The agent says its [announcement](#the-transfer-announcement), then hold
    music starts. With `off`, hold music starts as soon as the agent finishes
    its current reply.
  </Step>

  <Step title="The platform dials your human agent">
    A separate, private consult call rings the destination for up to 30 seconds.
    By default your team sees your agent's phone number as caller ID. You can
    show the caller's own number instead, see [Caller ID on
    transfers](#caller-id-on-transfers).
  </Step>

  <Step title="The agent briefs the human">
    The agent greets the human, reads out the caller's number digit by digit
    with a pause between groups when the caller ID is known, and relays short
    third-person notes on who is calling and what they need, generated from
    the conversation so far. The
    caller hears none of this. `warm_briefing_instructions` scripts the
    introduction; see [Customizing the briefing](#customizing-the-briefing).
  </Step>

  <Step title="The calls merge">
    With `warm_connect: "confirm"` the merge waits for the human to agree; with
    `direct` it happens right after the briefing. The hold music stops, the
    human joins the caller, and the agent leaves without another word.
  </Step>
</Steps>

If the human cannot be reached, declines, or the consult call hits voicemail, the caller comes off hold and the agent apologizes and continues helping. It can retry if the caller asks. If the caller hangs up while on hold, the agent briefly tells the human what happened and ends the consult call.

### Customizing the briefing

By default the agent introduces itself, reads out the caller's number, relays the notes, and asks whether the person can take the call. A warm destination's briefing instructions replace that introduction. The notes generated from the conversation and the handoff itself stay as they are: the agent still waits for a yes (or connects right away with `direct`), so the text only has to say what you want relayed and how.

The text is rendered with [dynamic variables](/agents/build/dynamic-variables) when the session starts, so it can carry facts the conversation may not mention. A custom text is spoken as written, so if it should include the caller's number, put it in yourself with `{{system.caller_number_spoken}}`, which arrives pre-grouped for speech (`+1; 4 1 5; 5 5 5; 0 1 2 3`) so the voice reads it digit by digit in any language; see [Reading a number out loud](/agents/build/dynamic-variables#reading-a-number-out-loud).

Two things to keep in mind. The variables are filled in once, when the session is created, so an order number the caller mentions later reaches the colleague only through the notes. And a withheld caller ID renders as an empty string, so write the text to read well without it.

#### In the console

On the destination card, choosing **Warm** shows a **Briefing instructions** field beneath the handoff choice. Leave it empty to keep the default introduction.

#### Through the API

Set `warm_briefing_instructions` on the destination; it is ignored on cold destinations.

<CodeGroup>
  ```bash Custom briefing theme={null}
  curl --request PATCH "https://api.fish.audio/v1/agent/agents/$AGENT_ID/config" \
    --header "Authorization: Bearer $FISH_API_KEY" \
    --header "Content-Type: application/json" \
    --data '{
      "conversation": {
        "transfer_destinations": [
          {
            "label": "Support desk",
            "phone_number": "+14155550123",
            "mode": "warm",
            "warm_connect": "confirm",
            "warm_briefing_instructions": "Say the caller number, {{system.caller_number_spoken}}, then explain in one sentence why they are calling and ask if the colleague can take the call. Skip the number if it is empty."
          }
        ]
      }
    }'
  ```
</CodeGroup>

## Caller ID on transfers

When the agent transfers a call, your team member's phone shows a caller ID. You choose which number it shows:

| Option | Your team sees | Good for |
| - | - | - |
| **Agent's number** (default) | The number the caller dialed, such as your support line | Recognizing that the call comes from the agent |
| **Caller's number** | The caller's own phone number | Seeing who is calling before answering, calling back |

The choice is set on each phone number, and applies to every transfer on that number, cold and warm. It is available on numbers you bought on Fish Audio. For imported numbers, see [Imported SIP numbers](/agents/telephony/byo-sip#caller-id-on-transfers).

If the caller hides their number, a warm transfer shows the agent's number instead.

### In the console

* **For all numbers of an agent**: open the agent's **Phone** page and use the **Show the caller's number on transfers** switch in the **Call transfer** section.
* **For one number**: on the **Phone numbers** page, open the number's menu and choose **Transfer caller ID**.

Only team owners and admins can change it. You don't need to publish the agent.

### Through the API

Set `transfer_caller_id` on the number: `original_caller` shows the caller's number, `agent_number` shows the agent's number.

```bash Request theme={null}
curl --request PATCH https://api.fish.audio/v1/agent/phone-numbers/$PHONE_NUMBER_ID \
  --header "Authorization: Bearer $FISH_API_KEY" \
  --header "Content-Type: application/json" \
  --data '{ "transfer_caller_id": "original_caller" }'
```

The caller ID is a setting on the number, not part of the agent, so it needs no publish and rolling back the agent leaves it as it is. Warm transfers use a new choice from the next call.

The response is the updated number, with `caller_id_sync_status` normally `synced`, which means cold transfers use the new choice too. If it reads `pending` or `error`, the change has not reached the phone carrier yet and cold transfers still use the previous choice. Send `{ "retry_caller_id_sync": true }` to try again.

The older boolean `cold_transfer_use_original_caller` still works, but it is deprecated. Use `transfer_caller_id` instead.

## Limitations

* **Phone calls only**: web and SDK sessions have no phone leg to hand off, so the `transfer_call` tool never ships for them.
* **Supported countries**: destination numbers are limited to an allowlist of country codes.

## Billing

Time after a transfer is billed separately from agent time, at a cold or warm transfer rate per minute. The agent rate stops when the call is handed over (cold) or when the caller and the person are joined (warm); from that point the call is billed as transfer minutes. Current rates are on the [Pricing](/agents/pricing) page.

## Going further

<CardGroup>
  <Card title="Inbound calls" icon="phone-arrow-down-left" href="/agents/telephony/inbound-calls">
    Bind a number and put your agent on the phone.
  </Card>

  <Card title="Phone numbers" icon="hashtag" href="/agents/telephony/phone-numbers">
    Number management, providers, and agent bindings.
  </Card>

  <Card title="System tools" icon="gears" href="/agents/build/system-tools">
    The other built-in capabilities, like hanging up.
  </Card>

  <Card title="Conversation history" icon="clock-rotate-left" href="/agents/monitor/conversation-history">
    Review transcripts and recordings of transferred calls.
  </Card>
</CardGroup>


# Agent Tests
Source: https://docs.fish.audio/agents/test/agent-tests

Script a conversation or simulate a user, then let an LLM judge score your agent to catch regressions before you publish

Write a test once, run it against any agent, and get a **Pass** or **Fail** verdict with the judge's reasoning. Tests live in a shared workspace library, run against your agent's current draft, and never touch what's published, so you can iterate on a prompt and re-run in seconds.

## Test types

| Type | What it checks | Use it for |
| - | - | - |
| **Single Turn** | The next reply to a scripted conversation, scored by an LLM judge against your expectation. | Wording, tone, and one specific answer. |
| **Tool** | Whether the next reply calls a given tool (or no tool at all), optionally with matching parameters. | Routing decisions and argument extraction. |
| **Simulation** | A whole conversation driven by a simulated user, scored by the judge against your success conditions, plus deterministic checks on what happened. | Flows that only show up over several exchanges, such as taking an order. |

Over the API, the type is the `test_type` field: `next_reply` for Single Turn, `tool`, or `simulation`. Each type's section below ends with its request body for [Create Test](/api-reference/endpoint/agent/create-test).

<Note>
  Tests run as text (no audio is synthesized), but the agent uses its full draft
  configuration: the [knowledge base](/agents/build/knowledge-base) is
  consulted, and the agent can invoke its attached tools. In Single Turn and
  Tool tests, [webhook tools](/agents/build/webhook-tools) send real HTTP
  requests, so point them at a staging endpoint. Simulation tests mock tools by
  default. For end-to-end verification with voice, use [preview
  calls](/agents/test/preview-calls).
</Note>

## Create a test

Tests are workspace-level resources, managed under **Library → Tests** in the console and shareable across every agent in your workspace.

<Steps>
  <Step title="Start a new test">
    Open **Library → Tests** and click **Add test**. Give it a name and pick a
    **Type**.
  </Step>

  <Step title="Describe what to check">
    Fill in the fields for that type. They are described in the sections below:
    [Single Turn](#single-turn-tests), [Tool](#tool-tests), and
    [Simulation](#simulation-tests).
  </Step>

  <Step title="Save">
    Click **Create Test**. The test is now in your library, ready to attach to
    agents.
  </Step>
</Steps>

To create tests from your own systems, send the same fields to [Create Test](/api-reference/endpoint/agent/create-test) with an [API key](/developer-guide/getting-started/api-key). For a complete Simulation test built step by step, follow the [rescheduling tutorial](/agents/test/simulation-tutorial).

## Single Turn tests

A Single Turn test hands your agent a scripted conversation and asks it to produce the next reply:

1. You script a conversation history of agent and user messages.
2. The agent generates the next reply using its current **draft** configuration and the same language model that answers in live conversations.
3. An LLM judge scores the reply against your **Expectation**, optionally calibrated by success and failure examples, and returns a verdict with its reasoning.

Under **Conversation**, click **Add message** to build the history the agent sees. Each message is either an **Agent** or **User** turn, and at least one must be a user turn. Under **Judging**, write the **Expectation**: what a correct reply must do. Optionally click **Add example** to provide success and failure examples. They calibrate the judge but aren't required.

### Single Turn over the API

| Console | API field |
| - | - |
| Conversation | `conversation`: messages with `role` (`agent` or `user`) and `text` |
| Expectation | `expectation`, required |
| Success and failure examples | `success_examples` and `failure_examples`, lists of strings |

```json theme={null}
{
  "name": "Quotes the late cancellation fee",
  "test_type": "next_reply",
  "agent_ids": ["<agent-id>"],
  "conversation": [
    {
      "role": "agent",
      "text": "Thanks for calling Bright Smile Dental. How can I help?"
    },
    {
      "role": "user",
      "text": "What happens if I cancel less than a day before my appointment?"
    }
  ],
  "expectation": "States the $50 late cancellation fee and offers to reschedule instead.",
  "success_examples": [
    "Cancelling within 24 hours costs $50. Would you like to move the appointment instead?"
  ],
  "failure_examples": ["There is no fee for cancelling."]
}
```

## Tool tests

A Tool test scripts the conversation the same way, but instead of judging the reply it checks which tool the agent called while producing it.

Under **Require tool execution**, pick a tool from the library and choose **Should have been called** or **Should not be called**. Leave the tool empty to check that the agent called no tool at all. Integration tools, such as calendar tools, can only be checked in a Simulation test. Under **Tool parameters**, optionally add the parameter values the call must carry, typed as string, number, or boolean. The result names the tool the agent actually called.

### Tool over the API

| Console | API field |
| - | - |
| Conversation | `conversation`, as in Single Turn |
| Tool | `referenced_tool`: `id`, `name`, and `type` (`webhook` or `client`). Omit it to check that no tool is called |
| Should have been called / Should not be called | `verify_absence`: `false` or `true` |
| Tool parameters | `tool_parameters`: `name`, `value`, and `type` (`string`, `number`, or `boolean`) |

[List Test Tools](/api-reference/endpoint/agent/list-test-tools) returns the tool ids an agent's tests can reference.

```json theme={null}
{
  "name": "Checks availability for the requested date",
  "test_type": "tool",
  "agent_ids": ["<agent-id>"],
  "conversation": [
    {
      "role": "user",
      "text": "Do you have anything open on Thursday, October 8?"
    }
  ],
  "referenced_tool": {
    "id": "<tool-id>",
    "name": "check_availability",
    "type": "webhook"
  },
  "tool_parameters": [
    { "name": "date", "value": "2026-10-08", "type": "string" }
  ],
  "verify_absence": false
}
```

## Simulation tests

A Simulation test does not script the user. Instead, a simulated user plays a role you describe, talks to your agent for up to a set number of turns, and an LLM judge scores the finished conversation against your success conditions.

### Write the scenario

The scenario is the simulated user's brief. Click **Insert template** to start from the four sections the simulator expects:

| Section | What to write |
| - | - |
| **PERSONA** | Who the user is and how they talk, for example an impatient customer on a lunch break. |
| **GOAL** | What they want out of the conversation. |
| **FACTS** | Details the agent may ask for, such as an order number or a date. Reveal one fact at a time. |
| **ENDING** | When the user hangs up: satisfied, out of patience, or after a fixed number of tries. |

Two rules make scenarios reliable. Reveal one fact at a time, so the agent has to ask for what it needs instead of receiving everything in the first message. And give an explicit ending, otherwise the conversation runs until the turn limit and the judge has to guess whether the user was done.

**Max turns** caps how many user turns the simulation runs. A **Conversation** script is optional here. Any messages you add are replayed first, then the simulated user takes over from the last message.

### Channel

**Channel** sets the register the agent speaks in during the simulation, the same one it uses on that channel in production: **Web voice**, **Phone inbound**, or **Phone outbound**. On a phone channel, for example, the agent repeats important numbers back in small groups. Phone outbound speaks in the same phone register as Phone inbound and only `{{system.channel}}` differs, so if the agent should open an outbound call differently, branch on `{{system.channel}}` in its prompt. **Auto**, the default, uses Phone inbound when a phone number is assigned to the agent and Web voice otherwise, so pick Phone outbound yourself for an agent that only places calls. As in production, client tools are not available on phone channels, and transfers only happen on phone channels. `{{system.channel}}` in the agent's prompt and in the scenario takes the same value. No real call is placed, so `{{system.caller_number}}` and `{{system.dialed_number}}` are empty on every channel.

### Success conditions

Add one to ten **Success conditions**, each a plain sentence describing something that must happen in the conversation, for example `The agent confirms the refund amount before closing`. Give each a short name, or leave it blank and the first words of the description become the name.

The judge reads the whole transcript and marks every condition **Success**, **Failure**, or **Unknown**. A run passes only when every condition succeeds. An **Unknown** verdict means the transcript did not contain enough evidence either way, and the run is flagged **Needs review** so you read the transcript before trusting the result. Success and failure examples, when you add them, calibrate the judge across all conditions.

### Tool mocks

Simulated conversations usually run many times, so by default tool calls never reach your real endpoints:

| Strategy | Behaviour |
| - | - |
| **Mock all** | Every tool answers from a mock. This is the default. A tool without any mock fails, see below. |
| **Mock selected** | Only the tools you list answer from a mock. Choose whether the rest return an error or call their real endpoint. |
| **Mock none** | Every tool with a real endpoint calls it. Webhooks may create real side effects, so reserve this for a safe test environment. |

A mock entry is the tool, the result it returns (JSON or plain text), an optional HTTP status for webhook tools, and optional parameter conditions so the mock only applies when the agent calls the tool with matching arguments (exact match, regular expression, or any value). Turn on **Return an error** to make the tool fail with that result instead, for example to test how the agent handles an outage. A webhook tool set to hide errors shows the agent only that the call failed, as in production. When a tool has several entries, the ones with conditions are checked first, in order, and an entry without conditions is the fallback that answers only when none of them match, wherever it sits in the list. Regular expressions, in mock conditions and in required tool call parameters alike, use JavaScript syntax without Unicode mode. Python forms such as `(?i)` or `(?P<name>...)` are refused when you save. Write a named group as `(?<name>...)`, and ignore case with `(?i:...)` around the part it applies to. Forms that JavaScript reads as literal text without Unicode mode are refused too: Unicode escapes such as `\p{L}` or `\u{1F600}` (list the characters in a class instead, for example `[A-Za-zÀ-ÿ]`) and Python escapes such as `\A`, `\z` or `\N{...}` (write `\A` as `^` and `\z` as `$`). The platform checks required tool call parameters itself, so there back-references, emoji and other characters outside the Basic Multilingual Plane, non-ASCII characters under `(?i:...)`, `\S` inside a class without `\s` (`[\s\S]` is fine) and `\c` are refused as well.

Under **Mock all**, a tool without an entry in the test answers with its own first mock response. A tool with neither fails, and the agent gets the same error it would get if the tool were unavailable in production. Tools whose result the agent never waits for, client tools that don't wait for an answer and webhook tools in fire and forget mode, need no mock: as in production, the agent only ever hears their acknowledgement, whether the call ran for real, answered from a mock, or was skipped. Such a client tool can't be mocked at all. A simulation has no client, so a client tool that waits for an answer always answers from its mock entries, under every strategy including **Mock none**, and needs one. The test form lists the agent's tools that would fail this way. To let specific tools reach their real endpoint while everything else stays mocked, add them under **Call the real endpoint**. Only webhook tools and read-only integration tools can go there. Every other tool still needs a mock, so a tool you attach to the agent later fails instead of reaching its endpoint until you mock it. A tool there can still have mock entries: a matching entry answers, and any other call reaches the endpoint. Integration tools, such as calendar tools, are mocked like any other tool. An integration that writes only after the caller confirms, such as creating a calendar event, answers the first call with a confirmation request and commits through its **Confirm write** tool. Mock both, the way production answers them. If you mock only the write tool and the model then calls Confirm write, that call goes unanswered and the run needs review.

A run in which a tool call went unanswered never passes: the tool had no mock, or none of its entries matched and neither the tool's own mock response nor **Call the real endpoint** could answer. The run fails with **Needs review** and **No mock** badges and lists the unanswered calls, even when the judge and every assertion passed, because the conversation left the path it would take in production. A call with no matching entry means either the agent passed arguments you didn't expect or the entry's conditions are too narrow. Calls to a tool the test forbids are not listed, they are the agent's failure and the assertion reports them. The judge is told the call failed because the test prepared no answer for it, and judges the agent as usual without treating the call as having run. Add or widen the mock and run the test again.

If you are used to tools without a mock calling their real endpoint, your first runs will show **No mock** on those calls. Add a mock for each tool the test form lists, or add read-only tools to **Call the real endpoint**.

Mocks and assertions apply to the tools attached to the agent under test. A tool the agent does not have is never mocked and counts as never called: a required call on it fails with *This tool is not on the agent*, while a forbidden tool or a maximum-only check passes. Client tools on a phone channel are treated the same way. This keeps a shared guard such as "never call issue\_refund" passing on agents that cannot call the tool.

### Assertions

Under **Advanced**, add deterministic checks that run alongside the judge:

* **Required tool calls**: the agent must call the tool, optionally with parameters that match, between a minimum and a maximum number of matching calls. The default is at least once. Set the maximum to 1 to catch a duplicate booking, or set both the minimum and the maximum to 0 to forbid calls with those arguments.
* **Forbidden tools**: the agent must not call the tool.
* **Ended by**: who ended the conversation, the agent, the user, a transfer, or any of them. A transfer only happens on a phone channel, so on Web voice a transfer check always fails.

A failed assertion fails the run regardless of the judge's verdict. Each assertion shows **Passed** or **Failed** with a short detail in the result.

### Repeat count

Set **Repeat** to run the same scenario up to 20 times in a batch. Simulated conversations vary from run to run, so repeats surface flaky behaviour that a single run would miss. The agent's Tests page shows how many repeats passed out of those that reached a verdict, for example `2 / 3 passed`, or `2 / 2 passed · 1 error` when one repeat ended in **Error**. While repeats are still running, the row counts them instead, for example `1 passed · 2 running`. Opening the test from that page lets you switch between the finished repeats of the latest batch to read each conversation. The test opens on the first failed repeat.

Repeats only apply to **Run all**. Running the test from its **Run** tab, or re-running one test from the row menu, runs the conversation once.

### Simulation over the API

Every Simulation field lives under `simulation`. An optional `conversation`, up to 50 messages, is replayed before the simulated user takes over.

| Console | API field under `simulation` |
| - | - |
| Scenario | `scenario` |
| Max turns | `max_turns`, 1 to 50, default 10 |
| Channel | `channel`: `web_voice`, `phone_inbound`, or `phone_outbound`. Omit it for **Auto** |
| Success conditions | `success_conditions`: `name` and `description` |
| Tool mocks strategy | `tool_mocks.strategy`: `all`, `selected`, or `none` |
| The rest, under **Mock selected** | `tool_mocks.fallback`: `error` or `real` |
| Mock entries | `tool_mocks.tools`: `tool`, `result`, and optionally `status`, `when`, and `error: true` |
| Call the real endpoint | `tool_mocks.real_tools` |
| Required tool calls | `assertions.tool_calls`: `tool`, `params`, `min_calls`, `max_calls` |
| Forbidden tools | `assertions.forbidden_tools` |
| Ended by | `assertions.ended_by`: `agent`, `user`, `transfer`, or `any` |
| Repeat | `repeat_count`, 1 to 20 |

Tools are referenced by `id`, `name`, and `type`, as in Tool tests, and integration tools use the `<provider_key>:<tool_name>` ids that [List Test Tools](/api-reference/endpoint/agent/list-test-tools) returns. Parameter conditions (`when` on a mock entry, `params` on a required call) map a parameter name to `{"type": "exact" | "regex" | "any", "value": "..."}`.

```json theme={null}
{
  "name": "Takes a pickup order",
  "test_type": "simulation",
  "agent_ids": ["<agent-id>"],
  "simulation": {
    "scenario": "PERSONA: Luca, a regular customer, brief and friendly.\nGOAL: Order one Margherita and one Diavola for pickup.\nFACTS: Name Luca, phone 555-0192. Give the phone number only when asked.\nENDING: Once the agent confirms the order number and pickup time, thank them and hang up.",
    "max_turns": 10,
    "success_conditions": [
      {
        "name": "Order confirmed",
        "description": "The agent confirmed both pizzas and told the caller when to pick them up."
      }
    ],
    "assertions": {
      "tool_calls": [
        {
          "tool": {
            "id": "<tool-id>",
            "name": "submit_order",
            "type": "webhook"
          },
          "params": { "phone": { "type": "regex", "value": "555-?0192" } },
          "min_calls": 1,
          "max_calls": 1
        }
      ]
    },
    "tool_mocks": {
      "strategy": "all",
      "tools": [
        {
          "tool": {
            "id": "<tool-id>",
            "name": "submit_order",
            "type": "webhook"
          },
          "result": { "ok": true, "order_id": "A17", "ready_in_minutes": 20 }
        }
      ]
    }
  }
}
```

The [rescheduling tutorial](/agents/test/simulation-tutorial) walks through a larger example with conditional mocks, an injected error, and forbidden tools.

## Attach tests to agents

A test only runs against agents it's attached to. Attach from either side:

* **From the library**: open the test's **Access** tab and toggle it on for each agent.
* **From the Builder**: on the agent's **Tests** page, click **Add tests** and pick from the library.

One test can be attached to many agents, and each agent keeps its own last result. **Remove from agent** detaches the test from that agent only. **Delete** in the library removes the test from all agents.

## Run a single test

Open the test and switch to its **Run** tab. Pick an agent that has access, then click **Run test**. Single Turn and Tool tests answer within seconds. A Simulation test runs in the background, shows **Queued** and then **Running**, and can take a few minutes.

The verdict card shows **Pass** or **Fail** with the time the run took, followed by details for the test type:

| Type | What you see |
| - | - |
| **Single Turn** | The full reply the agent generated and why the judge passed or failed it. |
| **Tool** | Which tool the agent called, if any, against what the test expected. |
| **Simulation** | The judge's summary, every success condition with its verdict and rationale, every assertion with its outcome, who ended the conversation, after how many turns and on which channel, the LLM cost of the agent, the simulated user, and the judge with their token breakdown, and the full transcript with each tool call marked by where its answer came from: **Mock** for an entry in the test, **Tool mock** for the tool's own mock response, **Real** when it reached the real endpoint, **No mock** or **No matching mock** when it failed for lack of one. |

A Simulation run with an **Unknown** condition fails and carries a **Needs review** badge. If a run can't complete, the card shows **Error** with a message that says whether the agent caused it, for example its custom LLM endpoint stopped responding, or the platform did, for example our language model did not answer. A conversation that broke off or hit its time limit is never judged, so it can't pass by accident. The transcript is still shown below the error, as it is when the conversation finished but could not be judged.

## Run every test for an agent

On the agent's **Tests** page in the Builder, click **Run all**. Each row moves through **queued → running → Pass/Fail** (or **Error** if a run can't complete). Simulation tests run once per repeat and the row shows how many repeats passed. The page header summarizes the latest batch, for example `4 passed, 1 failed on last run`, with the batch **pass rate**: green at 100%, amber from 80%, red below. The pass rate counts passed runs out of passed and failed ones, so a run that ended in **Error** does not lower it. Use the row menu to re-run a single test, edit it, or remove it from the agent.

## Tests run against the draft

Tests always exercise the agent's latest **draft** configuration, including unpublished changes to the system prompt. That makes the loop fast:

<Steps>
  <Step title="Edit the draft">
    Change the system prompt or first message in
    [Configuration](/agents/build/configuration).
  </Step>

  <Step title="Re-run">Click **Run all**.</Step>

  <Step title="Publish when green">
    Once results look right, [publish the
    draft](/agents/deploy/versions-publishing). Running tests never publishes
    anything.
  </Step>
</Steps>

## Run tests from the API and CI

Everything on this page is also available over the REST API with an [API key](/developer-guide/getting-started/api-key), so you can keep tests next to your agent configuration and gate releases on them. See the [Tests API reference](/api-reference/endpoint/agent/list-tests) for every endpoint.

* **Export and import.** [Get Test](/api-reference/endpoint/agent/get-test) returns a test's full definition. Post that body to [Create Test](/api-reference/endpoint/agent/create-test) as is to recreate it, for example from JSON files kept in your repository. Read-only fields such as `test_id` and `created_at` are ignored. Tool references use your workspace's tool ids, so an export imports as is within the same team. [List Test Tools](/api-reference/endpoint/agent/list-test-tools) returns the ids a test can reference for an agent, including integration tools. [Import and export tests](/agents/test/import-export-tests) has scripts for saving tests to files and for moving them to another team.
* **Attach.** Pass `agent_ids` when you create a test, or attach it later with [Attach Test](/api-reference/endpoint/agent/attach-test).
* **Run and wait.** [Run Tests](/api-reference/endpoint/agent/run-tests) starts a batch with every attached test, or only the `test_ids` you pass, and `repeat_count` overrides each test's repeat count for that batch. Poll [Get Test Batch](/api-reference/endpoint/agent/get-test-batch) until `completed` is `true`, then check `passed`, `failed`, `errors`, and `pass_rate`. `pass_rate` leaves out runs that ended in **Error**, so decide in your pipeline whether an error should fail the build or be retried.
* **History.** [List Test Batches](/api-reference/endpoint/agent/list-test-batches) and [List Test Runs](/api-reference/endpoint/agent/list-test-runs) return past results, filterable by test, batch, and status.

Runs always use the agent's current draft. A pipeline that changes the agent should update the draft first ([Configure through the API](/agents/build/configuration#configure-through-the-api)), run the tests, and publish only when they pass.

This GitHub Actions workflow runs every test attached to an agent and fails the build if any run fails or errors:

```yaml .github/workflows/agent-tests.yml theme={null}
name: Agent tests
on: [pull_request]

jobs:
  agent-tests:
    runs-on: ubuntu-latest
    timeout-minutes: 30
    steps:
      - name: Run the agent's tests
        env:
          FISH_API_KEY: ${{ secrets.FISH_API_KEY }}
          AGENT_ID: ${{ vars.AGENT_ID }}
        run: |
          set -euo pipefail
          api=https://api.fish.audio/v1/agent
          auth="Authorization: Bearer $FISH_API_KEY"

          batch=$(curl -sf -X POST "$api/agents/$AGENT_ID/tests/run" \
            -H "$auth" -H "Content-Type: application/json" -d '{}' | jq -r .batch_id)

          while true; do
            result=$(curl -sf "$api/agents/$AGENT_ID/test-batches/$batch" -H "$auth")
            [ "$(jq -r .completed <<<"$result")" = "true" ] && break
            sleep 15
          done

          jq '{passed, failed, errors, pass_rate}' <<<"$result"
          jq -e '.failed == 0 and .errors == 0' <<<"$result" > /dev/null
```

Simulation tests take minutes, so give the job a timeout that covers your longest scenarios. To tolerate occasional failures of repeated Simulation tests, gate on `pass_rate` instead, for example `jq -e '.pass_rate >= 0.8'`.

## Limits

| Field | Limit |
| - | - |
| Test name | 200 characters |
| Conversation message | 2,000 characters each, at least one message (optional for Simulation) |
| Conversation (Simulation) | 50 messages |
| Expectation | 400 characters |
| Success / failure example | 400 characters each |
| Scenario (Simulation) | 10,000 characters |
| Success conditions (Simulation) | 1 to 10, description 500 characters each |
| Max turns (Simulation) | 50 |
| Repeat count (Simulation) | 20 |
| Runs per API batch | 1,000 (tests times repeats) |
| Dynamic variables | 50 per test |

## Going further

<CardGroup>
  <Card title="Preview calls" icon="phone" href="/agents/test/preview-calls">
    Talk to your draft agent live, with voice, tools, and transcripts.
  </Card>

  <Card title="Versions & publishing" icon="rocket" href="/agents/deploy/versions-publishing">
    How drafts become immutable published versions.
  </Card>

  <Card title="Configuration" icon="sliders" href="/agents/build/configuration">
    System prompt, first message, and everything tests exercise.
  </Card>

  <Card title="Core concepts" icon="lightbulb" href="/agents/concepts">
    Agents, drafts, sessions, and the workspace library model.
  </Card>
</CardGroup>


# Import and Export Tests
Source: https://docs.fish.audio/agents/test/import-export-tests

Save tests as JSON files, recreate them in the same team, or move them to another team over the API

A test has one JSON format for reading and writing. [Get Test](/api-reference/endpoint/agent/get-test) returns it, and [Create Test](/api-reference/endpoint/agent/create-test) accepts the same body as is. That makes it easy to keep tests in version control, copy them to another agent, or move them to another team.

**Prerequisites**

* A Fish Audio [API key](/developer-guide/getting-started/api-key) for each team involved.
* `bash`, `curl`, and `jq` 1.6 or later for the scripts below.

## The export format

An exported test contains every field of its [test type](/agents/test/agent-tests#test-types), plus:

| Field | On import |
| - | - |
| `test_id`, `workspace_id`, `created_at`, `updated_at` | Ignored. The import gets its own values. |
| `agent_ids` | The agents the test was attached to. The import attaches to them again, so change it to target others. |

Tools appear as `{"id", "name", "type"}` wherever a test references them: `referenced_tool`, `simulation.assertions`, `simulation.tool_mocks.tools`, and `simulation.tool_mocks.real_tools`. Webhook and client tool ids belong to your team. Integration tool ids, such as `google_calendar:create_event`, are the same in every team.

Every call to Create Test creates a new test, so importing the same file twice gives you two copies.

## Export tests to files

This script saves every test attached to an agent as `tests/<test_id>.json`. Drop the `agent_id` parameter to export your whole test library instead.

```bash export-tests.sh theme={null}
set -euo pipefail
api=https://api.fish.audio/v1/agent
auth="Authorization: Bearer $FISH_API_KEY"
mkdir -p tests

cursor=""
while true; do
  page=$(curl -sfG "$api/tests" -H "$auth" \
    --data-urlencode "agent_id=$AGENT_ID" \
    --data-urlencode "page_size=100" \
    ${cursor:+--data-urlencode "cursor=$cursor"})
  for id in $(jq -r '.tests[].test_id' <<<"$page"); do
    curl -sf "$api/tests/$id" -H "$auth" > "tests/$id.json"
  done
  [ "$(jq -r .has_more <<<"$page")" = "true" ] || break
  cursor=$(jq -r .next_cursor <<<"$page")
done
```

## Import into the same team

Within one team, tool ids stay valid, so the files import as they are:

```bash import-tests.sh theme={null}
set -euo pipefail
api=https://api.fish.audio/v1/agent
auth="Authorization: Bearer $FISH_API_KEY"

for file in tests/*.json; do
  curl -sf -X POST "$api/tests" -H "$auth" -H "Content-Type: application/json" \
    --data @"$file" | jq -r '"\(.test_id)  \(.name)"'
done
```

Each test is attached to the agents in its `agent_ids`. To attach the copies to a different agent instead, rewrite that field on the way in:

```bash theme={null}
jq --arg agent "$AGENT_ID" '.agent_ids = [$agent]' "$file" |
  curl -sf -X POST "$api/tests" -H "$auth" -H "Content-Type: application/json" --data @-
```

All agents in `agent_ids` must be in the same workspace, and the test is created there.

## Import into another team

Webhook and client tool ids from the source team mean nothing in the target team, so each reference has to point at the target team's tool of the same name first.

<Warning>
  Create Test does not reject webhook and client tool ids it doesn't know. A
  test imported without remapping saves fine but misbehaves when it runs: its
  mocks never apply, required tool calls fail with *This tool is not on the
  agent*, and forbidden tool checks pass without checking anything. Always remap
  before importing into another team.
</Warning>

<Steps>
  <Step title="Recreate the tools">
    In the target team, create the tools the tests reference, with the same
    names, for example with [Create Tool](/api-reference/endpoint/agent/create-tool),
    and attach them to the target agent. Connect the same integrations on that
    agent if the tests reference integration tools.
  </Step>

  <Step title="Remap and import">
    Run this script with the target team's API key and agent. It reads the tools
    the target agent offers from [List Test Tools](/api-reference/endpoint/agent/list-test-tools),
    replaces every webhook and client tool id by name, attaches each test to
    the target agent, and stops with the tool's name if the agent has no tool
    with that name.

    ```bash import-into-team.sh theme={null}
    set -euo pipefail
    api=https://api.fish.audio/v1/agent
    auth="Authorization: Bearer $FISH_API_KEY"

    tools=$(curl -sf "$api/agents/$AGENT_ID/test-tools" -H "$auth" |
      jq '[.tools[] | select(.type != "integration") | {(.name): .id}] | add // {}')

    for file in tests/*.json; do
      jq --argjson tools "$tools" --arg agent "$AGENT_ID" '
        walk(
          if type == "object" and (.type == "webhook" or .type == "client")
            and has("id") and has("name")
          then .id = ($tools[.name] // error("No tool named \(.name) on the target agent"))
          else . end
        )
        | .agent_ids = [$agent]
      ' "$file" |
        curl -sf -X POST "$api/tests" -H "$auth" -H "Content-Type: application/json" \
          --data @- | jq -r '"\(.test_id)  \(.name)"'
    done
    ```
  </Step>
</Steps>

Integration tool ids are left as they are, since they are the same in every team.

## Going further

<CardGroup>
  <Card title="Agent tests" icon="vial" href="/agents/test/agent-tests">
    Every test type and how its fields map to the API.
  </Card>

  <Card title="Run tests from CI" icon="code-branch" href="/agents/test/agent-tests#run-tests-from-the-api-and-ci">
    Run an agent's tests from your pipeline and gate on the result.
  </Card>
</CardGroup>


# Preview Calls
Source: https://docs.fish.audio/agents/test/preview-calls

Talk to your draft agent in a live voice call, right in the Builder

Preview calls let you hold a live voice conversation with your agent without leaving the Builder. Every call runs against your current **draft** configuration: edit the system prompt, pick a different voice, and call again to hear the difference immediately, no publish required.

## Start a preview call

<Steps>
  <Step title="Open your agent in the Builder">
    Preview works from any Builder section: Configuration, Tools, Knowledge
    base, and so on.
  </Step>

  <Step title="Click Test call">
    The **Test call** button in the top bar opens the Preview panel on the right
    side of the Builder.
  </Step>

  <Step title="Click Start call">
    Your browser asks for microphone access the first time. The call cannot
    start without a microphone: if you deny permission, the panel tells you to
    allow access in your browser settings and try again.
  </Step>

  <Step title="Talk to your agent">
    Once connected, the status pill shows **Live** with a running call timer.
    The agent opens the call according to your [first message
    settings](/agents/build/configuration), and the conversation begins.
  </Step>
</Steps>

The panel stays connected while you switch between Builder sections, so you can read your configuration mid-call. Navigating away from the Builder (back to the agent list, for example) hangs up. If the call fails to connect, the panel shows an error state and lets you retry; a connection drop mid-call simply ends the call.

## During the call

### Live transcript

The panel shows both sides of the conversation as it happens:

* **Your speech** appears as it is transcribed, updating in place while you talk and settling when you finish a turn.
* **Agent replies** stream in as text while the agent speaks them aloud.

### Tool calls

When the agent invokes a [tool](/agents/build/tools), the call appears inline in the transcript as a compact entry that flips from running to done or failed. Expand an entry to inspect the exact arguments the model sent and the response it received, as formatted JSON truncated at 4 KB.

<Warning>
  Preview calls run your real configuration. [Webhook
  tools](/agents/build/webhook-tools) send real HTTP requests to your endpoints.
  Point them at a staging environment if you want to avoid side effects while
  testing.
</Warning>

### Controls

| Control | Behavior |
| - | - |
| Mute | Toggles your microphone without ending the call |
| End call | Ends the call and returns the panel to its ready state |
| Timer | Shows elapsed time; preview calls are capped at 10 minutes (600 seconds), or the agent's configured maximum call duration, whichever is lower |

## Drafts, not published versions

Preview calls and production sessions read different configurations:

| Session | Configuration used |
| - | - |
| Preview call | Latest draft, autosaved as you edit in the Builder |
| Production session | The published version, unchanged until you publish again |

This makes preview the fast inner loop: tweak the prompt, call, listen, repeat, and [publish](/agents/deploy/versions-publishing) when the draft is ready.

<Tip>
  After a config change, hang up and start a new call. A preview call reflects
  the draft as it stood when the session was created.
</Tip>

## After the call

Preview sessions are built for iteration, not record-keeping:

* **Post-call analysis runs in the panel.** If you have [post-call analysis](/agents/monitor/post-call-analysis) configured, ending the call triggers it and the results (summary, extracted data fields, and per-criterion success / failure / unknown verdicts) appear directly in the panel. Analysis runs however the call ends (including the automatic time limit or closing the tab mid-call), but results are viewable only in the panel.
* **Nothing enters conversation history.** Preview calls never appear in [conversation history](/agents/monitor/conversation-history), and the transcript is not preserved after you hang up.
* **No post-call webhook fires.** Preview sessions never trigger the `call.analyzed` [post-call webhook](/agents/monitor/webhooks), so Builder debugging doesn't push analysis results to your production systems.

## Going further

<CardGroup>
  <Card title="Agent tests" icon="vial" href="/agents/test/agent-tests">
    Scripted conversations with automated Pass/Fail judging: catch regressions
    without listening by ear.
  </Card>

  <Card title="Versions & publishing" icon="rocket" href="/agents/deploy/versions-publishing">
    Ship the draft you just previewed as an immutable published version.
  </Card>

  <Card title="Post-call analysis" icon="chart-simple" href="/agents/monitor/post-call-analysis">
    Configure summaries, data extraction, and success criteria. Results show up
    in the preview panel.
  </Card>

  <Card title="Web SDK" icon="code" href="/agents/deploy/web-sdk">
    Put the same voice conversation in your own app.
  </Card>
</CardGroup>


# Tutorial: Simulate a Rescheduling Call
Source: https://docs.fish.audio/agents/test/simulation-tutorial

Build a Simulation test step by step, from the scenario to tool mocks and assertions, and read what its results tell you

This tutorial builds one complete [Simulation test](/agents/test/agent-tests#simulation-tests) for a dental clinic agent that reschedules appointments. By the end, the test checks that the agent verifies the caller, handles a fully booked day, books exactly once, never cancels to rebook, and stays honest when a confirmation text fails.

**Prerequisites**

* An agent whose job includes rescheduling, with [webhook tools](/agents/build/webhook-tools) along these lines: `lookup_patient`, `check_availability` (takes a `date` as `YYYY-MM-DD`), `reschedule_appointment` (takes a `new_date`), `send_confirmation_sms`, and `cancel_appointment`. Your names and parameters will differ, so adapt the examples as you go.
* For the API path: a Fish Audio [API key](/developer-guide/getting-started/api-key).

## 1. Decide what can go wrong

Start from the failures you care about, not from the happy path. For rescheduling, these are the ones that cost a real clinic money or trust:

| Risk | How the test catches it |
| - | - |
| Changes an appointment without verifying the caller | Success condition, judged from the transcript |
| Books a full day anyway, or invents a free slot | A mock that returns no slots, plus a condition |
| Books twice | Assertion: reschedule called exactly once |
| Cancels the old appointment and books a new one | Assertion: cancel is a forbidden tool |
| Claims a confirmation text was sent when it failed | A mock that returns an error, plus a condition |

Judge-scored conditions cover what the agent says. Assertions cover what the agent does, and they are checked exactly, with no LLM involved.

## 2. Write the scenario

The scenario is the simulated user's brief, in four sections:

```text theme={null}
PERSONA: Dana Whitfield, a polite but busy patient calling from work. Short answers, slightly rushed.
GOAL: Move her cleaning from Tuesday October 6 at 10:00 AM to Thursday October 8 in the afternoon. If Thursday is not possible, Friday October 9 in the afternoon also works.
FACTS: Full name Dana Whitfield. Date of birth March 14, 1988. Only give the date of birth when asked. Do not mention Friday until the agent says Thursday is unavailable.
ENDING: Once the agent confirms the new day and time, thank them and hang up. If the agent books a time you did not agree to, correct it once, then hang up.
```

Two lines do most of the work. "Only give the date of birth when asked" means a pass proves the agent asked for it. "Do not mention Friday until the agent says Thursday is unavailable" forces the agent to find the alternative itself instead of being handed it. The ending gives the conversation a clear finish, so it doesn't run to the turn limit.

Set **Max turns** to 16, enough for verification, two availability checks, and a confirmation, with room to spare.

## 3. Write the success conditions

Each condition is one observable behaviour the judge checks against the whole transcript:

| Name | Description |
| - | - |
| Verified the patient | Before changing the appointment, the agent confirmed the caller's identity with her name and date of birth. |
| Handled the full day | When Thursday had no free slot, the agent said so plainly and offered another day instead of booking Thursday anyway. |
| Confirmed the new time | The agent read back the new appointment, Friday October 9 at 2:30 PM, and the caller agreed to it. |
| Honest about the text | If sending the confirmation text failed, the agent did not claim a text was sent and made clear that the appointment itself is booked. |

Write conditions the transcript can prove or disprove. "The agent was helpful" gives the judge nothing to check. If the transcript lacks evidence either way, the verdict is **Unknown** and the run is flagged **Needs review**.

## 4. Mock the tools

Keep the default strategy, **Mock all**, so no call reaches your real endpoints. Then add entries that steer the conversation:

| Tool | Condition | Result |
| - | - | - |
| `lookup_patient` | none | `{"patient_id": "pt_2291", "name": "Dana Whitfield", "next_appointment": "2026-10-06T10:00:00-07:00"}` |
| `check_availability` | `date` matches regex `^2026-10-08` | `{"date": "2026-10-08", "slots": []}` |
| `check_availability` | `date` matches regex `^2026-10-09` | `{"date": "2026-10-09", "slots": ["10:00", "14:30"]}` |
| `check_availability` | none | `{"slots": []}` |
| `reschedule_appointment` | none | `{"ok": true, "appointment_id": "apt_7731", "starts_at": "2026-10-09T14:30:00-07:00"}` |
| `send_confirmation_sms` | none, **Return an error**, status 503 | `{"error": "SMS provider unavailable"}` |

* **Conditional entries** give the same tool different answers in one conversation: Thursday is full, Friday has two slots. Entries with conditions are checked first, in order.
* **The unconditional `check_availability` entry** is the fallback for any other date, so an agent that checks Wednesday gets an answer instead of an unanswered call.
* **The error entry** makes the text fail the way an outage would, which is what the last success condition checks.
* `cancel_appointment` gets no mock on purpose. The test forbids it below, and a forbidden call is reported by its assertion.

Under **Mock all**, a call that neither a test entry nor the tool's own mock response answers fails, and the run is marked **Needs review** with a **No mock** badge. That is intentional: a run that left the path you prepared never passes by accident.

## 5. Add assertions

Under **Advanced**, add checks on what the agent actually did:

* **Required tool call** `reschedule_appointment`, with parameter `new_date` matching regex `^2026-10-09`, minimum 1, maximum 1. This catches both a wrong date and a double booking.
* **Required tool call** `check_availability`, minimum 1, maximum 4. The agent must check before offering times, and a loop of lookups fails.
* **Forbidden tool** `cancel_appointment`.

Any failed assertion fails the run, whatever the judge decided.

## 6. Pick the channel and the repeat count

Set **Channel** to **Phone inbound**, so the agent speaks the way it does on real calls, for example repeating dates and times back. Set **Repeat** to 5. Simulated users vary from run to run, and five runs show whether a pass is reliable or lucky.

## 7. Create the test

<Tabs>
  <Tab title="Dashboard">
    Open **Library → Tests**, click **Add test**, pick the **Simulation** type, and fill in the fields from the steps above. Then open the test's **Access** tab and turn it on for your agent.
  </Tab>

  <Tab title="API">
    Look up your tool ids with [List Test Tools](/api-reference/endpoint/agent/list-test-tools), replace the placeholders below, and send the body to [Create Test](/api-reference/endpoint/agent/create-test). `agent_ids` attaches the test in the same call.

    ```bash theme={null}
    curl --request POST https://api.fish.audio/v1/agent/tests \
      --header "Authorization: Bearer $FISH_API_KEY" \
      --header "Content-Type: application/json" \
      --data @reschedule-test.json
    ```

    ```json reschedule-test.json theme={null}
    {
      "name": "Reschedule: Thursday is full, caller takes Friday",
      "test_type": "simulation",
      "agent_ids": ["<agent-id>"],
      "simulation": {
        "scenario": "PERSONA: Dana Whitfield, a polite but busy patient calling from work. Short answers, slightly rushed.\nGOAL: Move her cleaning from Tuesday October 6 at 10:00 AM to Thursday October 8 in the afternoon. If Thursday is not possible, Friday October 9 in the afternoon also works.\nFACTS: Full name Dana Whitfield. Date of birth March 14, 1988. Only give the date of birth when asked. Do not mention Friday until the agent says Thursday is unavailable.\nENDING: Once the agent confirms the new day and time, thank them and hang up. If the agent books a time you did not agree to, correct it once, then hang up.",
        "max_turns": 16,
        "channel": "phone_inbound",
        "repeat_count": 5,
        "success_conditions": [
          {
            "name": "Verified the patient",
            "description": "Before changing the appointment, the agent confirmed the caller's identity with her name and date of birth."
          },
          {
            "name": "Handled the full day",
            "description": "When Thursday had no free slot, the agent said so plainly and offered another day instead of booking Thursday anyway."
          },
          {
            "name": "Confirmed the new time",
            "description": "The agent read back the new appointment, Friday October 9 at 2:30 PM, and the caller agreed to it."
          },
          {
            "name": "Honest about the text",
            "description": "If sending the confirmation text failed, the agent did not claim a text was sent and made clear that the appointment itself is booked."
          }
        ],
        "assertions": {
          "tool_calls": [
            {
              "tool": { "id": "<reschedule-tool-id>", "name": "reschedule_appointment", "type": "webhook" },
              "params": { "new_date": { "type": "regex", "value": "^2026-10-09" } },
              "min_calls": 1,
              "max_calls": 1
            },
            {
              "tool": { "id": "<availability-tool-id>", "name": "check_availability", "type": "webhook" },
              "min_calls": 1,
              "max_calls": 4
            }
          ],
          "forbidden_tools": [
            { "id": "<cancel-tool-id>", "name": "cancel_appointment", "type": "webhook" }
          ]
        },
        "tool_mocks": {
          "strategy": "all",
          "tools": [
            {
              "tool": { "id": "<patient-tool-id>", "name": "lookup_patient", "type": "webhook" },
              "result": { "patient_id": "pt_2291", "name": "Dana Whitfield", "next_appointment": "2026-10-06T10:00:00-07:00" }
            },
            {
              "tool": { "id": "<availability-tool-id>", "name": "check_availability", "type": "webhook" },
              "when": { "date": { "type": "regex", "value": "^2026-10-08" } },
              "result": { "date": "2026-10-08", "slots": [] }
            },
            {
              "tool": { "id": "<availability-tool-id>", "name": "check_availability", "type": "webhook" },
              "when": { "date": { "type": "regex", "value": "^2026-10-09" } },
              "result": { "date": "2026-10-09", "slots": ["10:00", "14:30"] }
            },
            {
              "tool": { "id": "<availability-tool-id>", "name": "check_availability", "type": "webhook" },
              "result": { "slots": [] }
            },
            {
              "tool": { "id": "<reschedule-tool-id>", "name": "reschedule_appointment", "type": "webhook" },
              "result": { "ok": true, "appointment_id": "apt_7731", "starts_at": "2026-10-09T14:30:00-07:00" }
            },
            {
              "tool": { "id": "<sms-tool-id>", "name": "send_confirmation_sms", "type": "webhook" },
              "error": true,
              "status": 503,
              "result": { "error": "SMS provider unavailable" }
            }
          ]
        }
      }
    }
    ```
  </Tab>
</Tabs>

## 8. Run it and read the result

On the agent's **Tests** page, click **Run all**. Over the API, start a batch with [Run Tests](/api-reference/endpoint/agent/run-tests) and poll [Get Test Batch](/api-reference/endpoint/agent/get-test-batch) until `completed` is `true`. Each run takes up to a few minutes.

Every run shows each success condition with its verdict and the judge's reasoning, each assertion with **Passed** or **Failed**, and the transcript with every tool call marked by where its answer came from. Here is what common failures mean:

| What you see | What it usually means |
| - | - |
| **Handled the full day** fails | The agent offered or booked Thursday without checking, or invented a slot. Tell it to check before offering times. |
| `reschedule_appointment` assertion reports 2 calls | A double booking, often a retry after the caller repeated herself. Make the prompt confirm once, then book once. |
| `cancel_appointment` forbidden tool fails | The agent treats rescheduling as cancel and rebook. Describe rescheduling as its own action in the prompt. |
| **No mock** on `check_availability` | The agent passed a date in another format, such as `Oct 8`. Describe the format in the tool, or widen the pattern. |
| **Unknown** and **Needs review** | The transcript didn't settle the condition. Read it, then make the condition more specific. |
| `3 / 5 passed` | The behaviour is unreliable. Open a failed repeat, fix the cause, and run again until all five pass. |

Once the test passes reliably, keep it attached. Every future prompt or tool change runs against it before you publish, and you can run it from CI as shown in [Run tests from the API and CI](/agents/test/agent-tests#run-tests-from-the-api-and-ci).

## Going further

<CardGroup>
  <Card title="Agent tests" icon="vial" href="/agents/test/agent-tests">
    Every field of every test type, and how each maps to the API.
  </Card>

  <Card title="Tests API" icon="flask" href="/api-reference/endpoint/agent/list-tests">
    Create, run, and read tests over the REST API.
  </Card>
</CardGroup>


# Agents API Errors
Source: https://docs.fish.audio/api-reference/agent-errors

Status codes, error shapes, and recovery guidance for every /v1/agent endpoint

Errors from `/v1/agent/*` come back as JSON with a `status` and a `message`:

```json theme={null}
{ "status": 404, "message": "Agent not found" }
```

The exceptions to that shape:

* **Validation failures** (`422`) return pydantic's error list, one entry per problem. Offending input values are stripped, so write-only credentials are never echoed back:

  ```json theme={null}
  [
    {
      "type": "value_error",
      "loc": ["body", "prompt", "system_prompt"],
      "msg": "Value error, system_prompt exceeds 32000 tokens (got 33001)",
      "ctx": { "error": "system_prompt exceeds 32000 tokens (got 33001)" }
    }
  ]
  ```

* **Phone-call creation** ([`POST /v1/agent/phone-calls`](/api-reference/endpoint/agent/create-phone-call)) adds a machine-readable `reason` field to its errors; see [Outbound calls](/agents/telephony/outbound-calls#errors) for the full list.

* A request body that isn't valid JSON returns `400` with `"Malformed JSON"`; a missing body or wrong `Content-Type` returns a bare `415`.

* A missing or malformed `Authorization` header returns a bare `401` with a `WWW-Authenticate: Bearer` header; a present-but-bad credential returns the JSON shape (`"Invalid token"`, `"Token expired"`).

## Status codes

| Status | Meaning | What to do |
| - | - | - |
| `400` | The request is well-formed JSON but semantically wrong | Read the `message`; it names the field or rule. |
| `401` | Missing, invalid, or expired API key | Send `Authorization: Bearer <key>`; check it on [API Keys](https://fish.audio/app/api-keys). |
| `402` | Out of API credit | Top up. Only session creation and phone-number purchase charge up front, so a `402` never leaves a partial charge. |
| `403` | The action is not allowed for this caller | See [403 causes](#403-what-was-refused) below. |
| `404` | The resource doesn't exist, or isn't yours | Cross-team access returns the same `404` as a missing id, so treat both identically. |
| `409` | A state conflict blocks the action | See [409 conflicts](#409-conflicts) below. |
| `415` | Missing body or wrong `Content-Type` | JSON endpoints need `application/json`; knowledge-source uploads need `multipart/form-data`. |
| `422` | Field-level validation failed | Fix the fields listed in the error array. |
| `429` | Public session creation is rate limited | Back off and retry; per-agent and per-IP windows apply to [public agents](/agents/deploy/public-agents) only. |
| `502` | An upstream dependency (conversation gateway, telephony provider) failed | Retry; for a failed phone-number purchase the row stays visible with status `error` and is safe to release. |
| `503` | A platform dependency is temporarily unavailable | Retry with backoff. |

## 400 vs 422

`422` is field-level validation: unknown fields (the public surface rejects them), length caps, enum values, invalid IANA timezones, dynamic-variable naming. `400` is semantic: an [override](/agents/deploy/authenticated-sessions#overrides) not enabled for the agent, mutually exclusive pagination parameters (`page` + `cursor`), an undecodable cursor, a page offset past 100,000 rows, or a knowledge upload that isn't UTF-8 plain text.

Two quirks worth coding around:

* **Explicit `null` is rejected** on most `PATCH` fields with `422`: omit a field to keep its value, send `""` to clear a text field. Write-only secrets (`llm.custom.api_key`, webhook `secret`) treat `null` like omission and keep the stored value.
* **Path-parameter validation renders `404`, not `422`**: a non-integer `{version_number}` looks like a missing version.

## 403: what was refused

| Message | Cause |
| - | - |
| `Agent is not public` | Anonymous session/widget request against an agent whose public switch is off. |
| `Origin header is required for public agents` | Anonymous request without an `Origin` header. Browsers send it automatically, curl must set it. |
| `Origin not allowed` | The page's origin is not on the agent's [allowed origins](/agents/deploy/public-agents#origin-matching). |
| `Agent is unavailable` | The agent's owner can't serve public sessions right now; visitors deliberately can't tell why. |
| `Phone number limit reached for this team (N)` | The team holds its maximum of live numbers; release one first. |

## 404: anti-enumeration

A resource that exists but belongs to another team returns the **identical** `404` body as one that never existed. This applies to agents, sessions, tools, knowledge sources, phone numbers, and versions. Released phone numbers and deleted knowledge sources behave the same. Never use `404` vs `403` to probe for existence.

The recording endpoint adds one more meaning: `Recording not found` on a real session means it was [never recorded](/agents/monitor/conversation-history#what-gets-stored).

## 409: conflicts

| Message | Cause and fix |
| - | - |
| `Agent has no published version` | Session creation against a never-published agent: [publish](/agents/deploy/versions-publishing) first. |
| `Knowledge source is attached to N agent(s)…` | Deletion is blocked while any agent references the source in its draft **or published** config: detach everywhere, republish, then delete. |
| `Tool is attached to N agent(s)…` | Deletion is blocked while any agent's **draft** references the tool (published snapshots are frozen copies and don't block): detach, then delete. |
| `This number is already on the platform` | The number is held by a team already; pick another from [search](/api-reference/endpoint/agent/search-available-phone-numbers). |

## Billing and rate limits

Exactly two endpoints charge before acting, and both can return `402 Out of API credit`: [`POST /v1/agent/sessions`](/api-reference/endpoint/agent/create-agent-session) (API-key callers) and [`POST /v1/agent/phone-numbers`](/api-reference/endpoint/agent/purchase-phone-number) (the first rental day). Anonymous public sessions never see `402`. An out-of-credit owner surfaces to visitors as `403 Agent is unavailable`. Usage is settled after each call and never fails a request.

`429` applies only to anonymous session creation on [public agents](/agents/deploy/public-agents), in fixed per-minute windows per agent and per client IP. API-key traffic is not rate limited on this surface; your backend is the gate.

## 5xx

`502` names the failing upstream in the message: the conversation gateway (`Agent gateway is unreachable`) or the telephony provider (`Twilio refused the request: …`). Retrying is safe. A failed phone-number purchase leaves the row visible with status `error`, refunds the day charge, and can be released; a failed release keeps the number live so releasing again retries. `503` means a platform dependency was briefly unreachable; retry with backoff.

## Going further

<CardGroup>
  <Card title="Authenticated sessions" icon="server" href="/agents/deploy/authenticated-sessions">
    The session token flow: request fields, lifetime, errors.
  </Card>

  <Card title="Public agents" icon="globe" href="/agents/deploy/public-agents">
    Origin allowlists and the errors unique to keyless access.
  </Card>
</CardGroup>


# Attach Test
Source: https://docs.fish.audio/api-reference/endpoint/agent/attach-test

put /v1/agent/agents/{agent_id}/tests/{test_id}
Attach a test to an agent so the agent's test runs include it.



# Create Agent
Source: https://docs.fish.audio/api-reference/endpoint/agent/create-agent

post /v1/agent/agents
Create an agent, optionally with its full initial configuration inline —
one call provisions a ready-to-publish agent.



# Create Agent Session
Source: https://docs.fish.audio/api-reference/endpoint/agent/create-agent-session

post /v1/agent/sessions
Start a conversation session with an agent and receive a join token for the
session transport (currently LiveKit WebRTC).



# Create Knowledge Source
Source: https://docs.fish.audio/api-reference/endpoint/agent/create-knowledge-source

post /v1/agent/knowledge-sources
Upload a plain-text or Markdown file (UTF-8, up to 1 MB) as a knowledge
source.



# Create Phone Call
Source: https://docs.fish.audio/api-reference/endpoint/agent/create-phone-call

post /v1/agent/phone-calls
Place an outbound call from one of your phone numbers; the agent speaks when
the callee answers.



# Create Test
Source: https://docs.fish.audio/api-reference/endpoint/agent/create-test

post /v1/agent/tests
Create a Single Turn, Tool or Simulation test, optionally attached to agents.



# Create Tool
Source: https://docs.fish.audio/api-reference/endpoint/agent/create-tool

post /v1/agent/tools
Create a workspace tool.



# Delete Agent
Source: https://docs.fish.audio/api-reference/endpoint/agent/delete-agent

delete /v1/agent/agents/{agent_id}
Delete an agent together with its configuration and version history.
Existing session records are retained.



# Delete Knowledge Source
Source: https://docs.fish.audio/api-reference/endpoint/agent/delete-knowledge-source

delete /v1/agent/knowledge-sources/{source_id}
Delete a knowledge source.



# Delete Test
Source: https://docs.fish.audio/api-reference/endpoint/agent/delete-test

delete /v1/agent/tests/{test_id}
Delete a test and detach it from every agent.



# Delete Tool
Source: https://docs.fish.audio/api-reference/endpoint/agent/delete-tool

delete /v1/agent/tools/{tool_id}
Delete a tool.



# Detach Test
Source: https://docs.fish.audio/api-reference/endpoint/agent/detach-test

delete /v1/agent/agents/{agent_id}/tests/{test_id}
Detach a test from an agent. The test stays in your library.



# End Agent Session
Source: https://docs.fish.audio/api-reference/endpoint/agent/end-agent-session

post /v1/agent/sessions/{session_id}/end
Hang up a live session: the agent disconnects and the call terminates.



# Get Agent
Source: https://docs.fish.audio/api-reference/endpoint/agent/get-agent

get /v1/agent/agents/{agent_id}
Fetch one agent's profile fields.



# Get Agent Session
Source: https://docs.fish.audio/api-reference/endpoint/agent/get-agent-session

get /v1/agent/sessions/{session_id}
Fetch one session's full detail: status, timing, and the conversation
timeline (transcript turns interleaved with tool calls, in order).



# Get Agent Session Recording
Source: https://docs.fish.audio/api-reference/endpoint/agent/get-agent-session-recording

get /v1/agent/sessions/{session_id}/recording
Return time-limited download URLs for the session's audio recording, one
per speaker track (agent and user are recorded separately).



# Get Agent Version
Source: https://docs.fish.audio/api-reference/endpoint/agent/get-agent-version

get /v1/agent/agents/{agent_id}/versions/{version_number}
One published version with its full frozen configuration snapshot —
including the current live version.



# Get Draft Config
Source: https://docs.fish.audio/api-reference/endpoint/agent/get-draft-config

get /v1/agent/agents/{agent_id}/config
Read the agent's current draft configuration — the state the next publish
will freeze.



# Get Knowledge Source
Source: https://docs.fish.audio/api-reference/endpoint/agent/get-knowledge-source

get /v1/agent/knowledge-sources/{source_id}
Fetch one knowledge source including its full text content and current
revision number.



# Get Phone Number
Source: https://docs.fish.audio/api-reference/endpoint/agent/get-phone-number

get /v1/agent/phone-numbers/{phone_number_id}
Fetch one phone number, including its current agent binding and
provisioning status.



# Get Test
Source: https://docs.fish.audio/api-reference/endpoint/agent/get-test

get /v1/agent/tests/{test_id}
Fetch one test's full definition, which is also the export format.



# Get Test Batch
Source: https://docs.fish.audio/api-reference/endpoint/agent/get-test-batch

get /v1/agent/agents/{agent_id}/test-batches/{batch_id}
The runs of one batch with their pass, fail and error counts.



# Get Test Run
Source: https://docs.fish.audio/api-reference/endpoint/agent/get-test-run

get /v1/agent/test-runs/{run_id}
Fetch one test run with its transcript, verdicts and tool calls.



# Get Tool
Source: https://docs.fish.audio/api-reference/endpoint/agent/get-tool

get /v1/agent/tools/{tool_id}
Fetch one tool's full definition.



# Get Widget Config
Source: https://docs.fish.audio/api-reference/endpoint/agent/get-widget-config

get /v1/agent/agents/{agent_id}/widget
Unauthenticated display configuration for the embeddable `<fish-agent>`
widget.



# List Agent Sessions
Source: https://docs.fish.audio/api-reference/endpoint/agent/list-agent-sessions

get /v1/agent/sessions
List your team's sessions, newest first.



# List Agent Versions
Source: https://docs.fish.audio/api-reference/endpoint/agent/list-agent-versions

get /v1/agent/agents/{agent_id}/versions
The publish history, newest first.



# List Agents
Source: https://docs.fish.audio/api-reference/endpoint/agent/list-agents

get /v1/agent/agents
List your team's agents, newest first.



# List Agents Using Knowledge Source
Source: https://docs.fish.audio/api-reference/endpoint/agent/list-agents-using-knowledge-source

get /v1/agent/knowledge-sources/{source_id}/agents
Every agent that references this source in its draft or currently
published configuration — the pre-flight check before a delete.



# List Agents Using Tool
Source: https://docs.fish.audio/api-reference/endpoint/agent/list-agents-using-tool

get /v1/agent/tools/{tool_id}/agents
Every agent whose draft configuration references this tool — the
pre-flight check before a delete.



# List Knowledge Sources
Source: https://docs.fish.audio/api-reference/endpoint/agent/list-knowledge-sources

get /v1/agent/knowledge-sources
List your team's knowledge sources, newest first.



# List Phone Numbers
Source: https://docs.fish.audio/api-reference/endpoint/agent/list-phone-numbers

get /v1/agent/phone-numbers
List your team's phone numbers, newest first.



# List Test Batches
Source: https://docs.fish.audio/api-reference/endpoint/agent/list-test-batches

get /v1/agent/agents/{agent_id}/test-batches
The agent's test batches, newest first, with their counts.



# List Test Runs
Source: https://docs.fish.audio/api-reference/endpoint/agent/list-test-runs

get /v1/agent/test-runs
List your team's test runs, newest first.



# List Test Tools
Source: https://docs.fish.audio/api-reference/endpoint/agent/list-test-tools

get /v1/agent/agents/{agent_id}/test-tools
The tool ids a test can reference for this agent.



# List Tests
Source: https://docs.fish.audio/api-reference/endpoint/agent/list-tests

get /v1/agent/tests
List your team's tests, newest first.



# List Tools
Source: https://docs.fish.audio/api-reference/endpoint/agent/list-tools

get /v1/agent/tools
List your team's tools, newest first.



# Publish Agent
Source: https://docs.fish.audio/api-reference/endpoint/agent/publish-agent

post /v1/agent/agents/{agent_id}/publish
Freeze the current draft into an immutable version (version_number
auto-increments) and make it the live configuration for new sessions.



# Purchase Phone Number
Source: https://docs.fish.audio/api-reference/endpoint/agent/purchase-phone-number

post /v1/agent/phone-numbers
Buy a number from the inventory.



# Release Phone Number
Source: https://docs.fish.audio/api-reference/endpoint/agent/release-phone-number

delete /v1/agent/phone-numbers/{phone_number_id}
Release a number back to the provider's inventory and stop its daily
billing.



# Run Tests
Source: https://docs.fish.audio/api-reference/endpoint/agent/run-tests

post /v1/agent/agents/{agent_id}/tests/run
Start a batch that runs the agent's attached tests against its draft.



# Search Available Phone Numbers
Source: https://docs.fish.audio/api-reference/endpoint/agent/search-available-phone-numbers

get /v1/agent/available-phone-numbers
Search the purchasable number inventory.



# Update Agent
Source: https://docs.fish.audio/api-reference/endpoint/agent/update-agent

patch /v1/agent/agents/{agent_id}
Update agent-level fields (name, description, status, public access and
session-override policy).



# Update Draft Config
Source: https://docs.fish.audio/api-reference/endpoint/agent/update-draft-config

patch /v1/agent/agents/{agent_id}/config
Patch the draft configuration section by section; omitted sections keep
their value.



# Update Knowledge Source
Source: https://docs.fish.audio/api-reference/endpoint/agent/update-knowledge-source

patch /v1/agent/knowledge-sources/{source_id}
Rename the source and/or replace its content by uploading a new file in
`source` (multipart).



# Update Phone Number
Source: https://docs.fish.audio/api-reference/endpoint/agent/update-phone-number

patch /v1/agent/phone-numbers/{phone_number_id}
Change the label and/or repoint the number at another agent — the
deployment-pipeline move (rebind from the staging agent to the production
one).



# Update Test
Source: https://docs.fish.audio/api-reference/endpoint/agent/update-test

patch /v1/agent/tests/{test_id}
Patch test fields. Omitted fields keep their value.



# Update Tool
Source: https://docs.fish.audio/api-reference/endpoint/agent/update-tool

patch /v1/agent/tools/{tool_id}
Patch tool fields; omitted fields keep their value (null is rejected —
send an empty string to clear a text field).



# Create Model
Source: https://docs.fish.audio/api-reference/endpoint/model/create-model

post /model
Create a new voice model

<Warning>
  Since this endpoint uploads files, use `multipart/form-data` for regular REST
  requests. Let your HTTP client set the multipart `Content-Type` boundary
  automatically.
</Warning>


# Delete Model
Source: https://docs.fish.audio/api-reference/endpoint/model/delete-model

delete /model/{id}
Delete an existing model



# Get Model
Source: https://docs.fish.audio/api-reference/endpoint/model/get-model

get /model/{id}
Get details of a specific model



# List Models
Source: https://docs.fish.audio/api-reference/endpoint/model/list-models

get /model
Get a list of all models



# Update Model
Source: https://docs.fish.audio/api-reference/endpoint/model/update-model

patch /model/{id}
Update an existing model



# Speech to Text
Source: https://docs.fish.audio/api-reference/endpoint/openapi-v1/speech-to-text

post /v1/asr
Transcribe audio with transcribe-1-pro, the recommended speech-to-text model: request and response fields, speaker turns, limits, formats, and errors

<Warning>
  This BETA endpoint transcribes one audio file per request. Send the audio as
  `multipart/form-data` (a file upload) or `application/msgpack`. Base64-encoded
  audio in a JSON body is not supported.
</Warning>

## Select a model

Send `POST https://api.fish.audio/v1/asr` with `Authorization: Bearer <API_KEY>` and a **`model` HTTP header** that selects the model. The request is synchronous: the response arrives when the whole file has been transcribed.

| Header value | Use it for |
| - | - |
| `transcribe-1-pro` | Recommended. Recordings up to 60 minutes, including multi-speaker conversations: speaker markers and speaker turns, with emotion and vocal-event cues preserved. |
| `transcribe-1` | General transcription of short recordings. Used when the `model` header is missing or not an exact match. |

Send `model: transcribe-1-pro` with every request, written exactly, in lowercase. If the header is missing, or its value is not an exact match (for example `Transcribe-1-Pro` or `transcribe-1pro`), the request is served and billed as `transcribe-1`, and no error is returned. If you expected `transcribe-1-pro` but the transcript has no speaker markers, check the header. This applies to the API playground on this page too: set its `model` header to `transcribe-1-pro`.

The model belongs in the header for both multipart and MessagePack requests; a `model` field in the request body is ignored. See [Limits](#limits) for recording length and request size.

To link the request to your own distributed trace, also send a W3C `traceparent` header. See [Tracing & Performance Analysis](/api-reference/observability).

## Request fields

Multipart and MessagePack bodies use the same field names. Multipart values are text; MessagePack values are typed.

| Field | Required | Default | Description |
| - | - | - | - |
| `audio` | Yes | — | Exactly one audio file per request: a file upload for multipart, or binary bytes for MessagePack. The format is detected from the file contents, not the file name. See [Supported audio formats](#supported-audio-formats). |
| `language` | No | Unset | Optional language hint, as a lowercase ISO 639-1 code such as `en`, `zh`, or `ja`. The language is detected automatically either way. An empty multipart value counts as unset. See [Language](#language). |
| `ignore_timestamps` | No | `true` | Default `true` (no timestamps). Set to `false` to get word-level `segments` and `speaker_turns`. Timestamps add processing time. |
| `tag_audio_events` | No | `true` | `true`: bracketed emotion and vocal-event cues such as `[laughter]` or `[高兴]` stay in `text` and in `speaker_turns`. `false`: the cues are removed. Timestamps, `duration`, and billing do not change. |
| `diarize` | No | `auto` | `auto` or `true`: return `speaker_turns` when timestamps are requested. `false`: omit `speaker_turns`. `false` does not change the transcript, and the speaker markers stay in `text`. |
| `num_speakers` | No | Unset | The expected number of speakers, an integer of 1 or more. Cannot be combined with `min_speakers` or `max_speakers`. |
| `min_speakers`, `max_speakers` | No | Unset | Lower and upper bounds on the number of speakers, integers of 1 or more. `min_speakers` must not exceed `max_speakers`. |

In multipart forms, send `true` or `false` for `ignore_timestamps`. Any value other than `true` (case-insensitive), including `1` or an empty value, is read as `false` and turns timestamps on. In MessagePack, send a boolean; a string or `nil` returns 400.

Speaker-count hints are applied on a best-effort basis: they guide speaker identification on longer recordings and may have no effect on short ones. They cannot be sent with `diarize=false`.

`tag_audio_events`, `diarize`, and the speaker-count hints are validated strictly:

* In multipart forms, `tag_audio_events` takes `true` or `false`, and `diarize` takes `auto`, `true`, or `false`, in any letter case. Speaker counts take digits only. Send each field at most once.
* In MessagePack, send `tag_audio_events` as a boolean, `diarize` as a boolean or one of the lowercase strings `auto`, `true`, or `false`, and speaker counts as integers, not floats or strings. `nil` means the default.
* Any other value, a repeated multipart field, or a conflicting combination returns 400 with code `invalid_parameter`.

**If you use `transcribe-1`:** send `audio`, `language`, and `ignore_timestamps`. The other fields apply to `transcribe-1-pro`; send them only with `model: transcribe-1-pro`.

### Examples

```bash theme={null}
curl --request POST https://api.fish.audio/v1/asr \
  --header "Authorization: Bearer $FISH_API_KEY" \
  --header "model: transcribe-1-pro" \
  --form audio=@conversation.wav \
  --form ignore_timestamps=false
```

This request returns word-level `segments` and `speaker_turns`. For a two-person interview, you can add a speaker-count hint, and remove emotion and vocal-event cues from the transcript:

```bash theme={null}
curl --request POST https://api.fish.audio/v1/asr \
  --header "Authorization: Bearer $FISH_API_KEY" \
  --header "model: transcribe-1-pro" \
  --form audio=@interview.mp3 \
  --form ignore_timestamps=false \
  --form num_speakers=2 \
  --form tag_audio_events=false
```

Let your HTTP client set the multipart `Content-Type` and boundary. For MessagePack, set `Content-Type: application/msgpack` and encode `audio` as binary bytes. See the [Speech to Text guide](/features/speech-to-text#direct-api-messagepack) for an example.

## Read the response

A successful response (HTTP 200) is a JSON object:

| Field | Description |
| - | - |
| `text` | The full transcript, with inline speaker markers such as `<\|speaker:0\|>` and, unless `tag_audio_events=false`, bracketed emotion and vocal-event cues such as `[高兴]` (happy) or `[laughter]`. See [Multi-speaker response format](#multi-speaker-response-format). |
| `duration` | Length of the audio in seconds, including silence. |
| `segments` | Word-level timestamps: an array of `{ "text", "start", "end" }`, with times in seconds. Empty (`[]`) when `ignore_timestamps=true` (the default), when no speech was found, or when timing is temporarily unavailable. See [Segments](#segments). |
| `speaker_turns` | The transcript as timed speaker turns. Returned when `ignore_timestamps=false` and `diarize` is not `false`; otherwise absent. See [Speaker turns](#speaker-turns). |
| `language` | The detected language's English name, such as `English` or `Chinese`. Omitted when it cannot be determined. Intended for display. |
| `language_code` | Two-letter ISO 639-1 code for the language, such as `en`. Omitted when unknown. If the language cannot be determined and you sent a `language` hint, this reports your hint. See [Language](#language). |
| `request_id` | A unique ID for this request, also present in `transcribe-1-pro` error bodies and in the `x-request-id` response header. Platform and network-edge errors do not carry it. Include it when you contact support. |

A field that is not returned is absent from the JSON, never `null`. Do not depend on the order of keys in the JSON.

**If you use `transcribe-1`:** rely on `text`, `duration`, `segments`, `language`, and `language_code`. Speaker markers, `speaker_turns`, and `request_id` are `transcribe-1-pro` features.

<Note>
  Both SDKs select `transcribe-1-pro` through the `model` header. In Python,
  pass
  `request_options=RequestOptions(additional_headers={"model": "transcribe-1-pro"})`
  to `asr.transcribe()`. In JavaScript, pass
  `{ headers: { model: "transcribe-1-pro" } }` as the second argument to
  `speechToText.convert()`. The Python SDK currently returns only `text`,
  `duration`, and `segments`, and neither SDK can send `tag_audio_events`,
  `diarize`, or the speaker-count hints. To use them, call the API directly, as
  in the examples above. The JavaScript SDK returns the whole response body,
  but its `STTResponse` type declares only `text`, `duration`, and `segments`.
</Note>

### Multi-speaker response format

`transcribe-1-pro` marks speaker changes **inside the `text` string** with `<|speaker:N|>` markers. The label `N` identifies the speaker for the text that follows, up to the next marker. A repeated label means that speaker is speaking again.

* Markers usually have a space on each side (`<|speaker:0|> 你好。 <|speaker:1|> ...`). Do not depend on exact spacing.
* Any text before the first marker belongs to the first turn.
* A transcript with no marker comes from a single speaker; treat it as speaker 0.
* Labels identify speakers within one response only. They are not names, and the same label in two requests is not the same person. Send a whole conversation as one file.

This illustrative response has two speakers and three turns. With `ignore_timestamps=true` (the default), timestamps are skipped and `speaker_turns` is absent:

```json theme={null}
{
  "text": "<|speaker:0|> 你好。 <|speaker:1|> [高兴]很开心认识你。 <|speaker:0|> 我也是。",
  "duration": 6.4,
  "segments": [],
  "language_code": "zh",
  "language": "Chinese",
  "request_id": "5f0c7e2a-9b1d-4c3e-8f6a-2d4b7c9e1a03"
}
```

Read this as speaker 0 saying `你好。`, speaker 1 saying `[高兴]很开心认识你。`, and speaker 0 saying `我也是。`. The `[高兴]` cue describes the delivery of speaker 1's speech. There is no separate emotion field; the cues are part of the text.

### Speaker turns

With `ignore_timestamps=false`, `transcribe-1-pro` also returns the turns as a structured `speaker_turns` array, unless you send `diarize=false`. When you request timestamps, prefer `speaker_turns` over parsing `text`.

* `speaker_turns` lists the turns in the order they occur, and is `[]` when no speech was found. Each turn is `{ "speaker": "speaker:N", "text": "...", "start": <seconds>, "end": <seconds> }`.
* `speaker` is the string `speaker:N`, where `N` matches the `<|speaker:N|>` marker in `text`. Like the markers, labels identify speakers within one response only.
* `text` is that turn's speech without speaker markers. It keeps emotion and vocal-event cues unless `tag_audio_events=false`.
* `start` and `end` come from the word timestamps. If word timing is unavailable (`segments` is empty), turn times are approximate and can cover the whole recording.
* Consecutive turns can have the same speaker. Do not assume turns are contiguous or non-overlapping.

This illustrative response is the same conversation with `ignore_timestamps=false`:

```json theme={null}
{
  "text": "<|speaker:0|> 你好。 <|speaker:1|> [高兴]很开心认识你。 <|speaker:0|> 我也是。",
  "duration": 6.4,
  "segments": [
    { "text": "你", "start": 0.24, "end": 0.52 },
    { "text": "好", "start": 0.52, "end": 0.8 },
    { "text": "很", "start": 1.6, "end": 1.84 },
    { "text": "开", "start": 1.84, "end": 2.1 },
    { "text": "心", "start": 2.1, "end": 2.36 },
    { "text": "认", "start": 2.36, "end": 2.7 },
    { "text": "识", "start": 2.7, "end": 2.98 },
    { "text": "你", "start": 2.98, "end": 3.3 },
    { "text": "我", "start": 4.5, "end": 4.78 },
    { "text": "也", "start": 4.78, "end": 5.02 },
    { "text": "是", "start": 5.02, "end": 5.4 }
  ],
  "speaker_turns": [
    { "speaker": "speaker:0", "text": "你好。", "start": 0.24, "end": 0.8 },
    {
      "speaker": "speaker:1",
      "text": "[高兴]很开心认识你。",
      "start": 1.6,
      "end": 3.3
    },
    { "speaker": "speaker:0", "text": "我也是。", "start": 4.5, "end": 5.4 }
  ],
  "language_code": "zh",
  "language": "Chinese",
  "request_id": "5f0c7e2a-9b1d-4c3e-8f6a-2d4b7c9e1a03"
}
```

See the [Speech to Text guide](/features/speech-to-text#multi-speaker-conversations) for parsing examples.

### Segments

A segment is usually one word; in Chinese and Japanese it is usually one character, or a few. Segment text has no punctuation and no speaker markers or cues, and can be normalized (for example `35` for `3.5`), so it does not always match `text` character for character. A segment can have `start` equal to `end`. Segments are not speaker turns and carry no speaker ID.

### Language

* `language` is optional. The language is detected automatically whether or not you send a hint, and a hint does not force the transcript into that language.
* Use a lowercase ISO 639-1 code such as `en`, `zh`, or `ja`. Other forms, such as `en-US` or `English`, may be rejected with 400.
* If the language cannot be determined (for example, very short audio), `language_code` reports your hint, which is not checked against the audio, and `language` may be absent.
* Responses report one language. For recordings that switch languages, `language` and `language_code` do not list every language spoken.

## Limits

* **Audio length:** up to 60 minutes per request. Longer audio returns 400 with code `audio_too_long`.
* **Request size:** for long recordings, send compressed audio such as MP3, Opus, or AAC. An hour of 128 kbps MP3 is about 55 MiB. A request that is too large returns 413.
* **One conversation per request:** send the whole recording in one request. Timestamps and speaker labels then cover the whole recording. Do not split a conversation, because labels are consistent only within one response.

**If you use `transcribe-1`:** it is designed for short recordings; for recordings longer than a few minutes, use `transcribe-1-pro`. It accepts up to 50 MiB per request; keep MP3 and Opus files under 25 MiB. A request that exceeds the size limit returns 413 or 400. A long request can fail with 503 if it exceeds the processing-time limit. Retry, and if it keeps failing, use `transcribe-1-pro` or split the audio.

### Processing time and timeouts

* Processing time grows with the length of the recording. Long `transcribe-1-pro` recordings can take several minutes.
* Set your HTTP client's timeout accordingly. The official SDKs' default timeout can be too short for long recordings, and many HTTP libraries default to even less (httpx defaults to 5 seconds). For long recordings, use a generous timeout, for example 15 minutes: `httpx.Timeout(900.0, connect=10.0)` with httpx, `RequestOptions(timeout=900)` with the Python SDK, or `{ timeoutInSeconds: 900 }` with the JavaScript SDK.
* In Node.js, the built-in `fetch`, which the JavaScript SDK uses, also stops waiting for a response after 5 minutes, even if you set a longer timeout. To wait longer, call `setGlobalDispatcher(new Agent({ headersTimeout: 900_000, bodyTimeout: 900_000 }))` from the `undici` package once at startup. See [Processing time and timeouts](/features/speech-to-text#processing-time-and-timeouts).
* If a long request fails with a 5xx error or the connection drops, retry it.
* Each request holds one of your account's [concurrent request slots](/developer-guide/models-pricing/pricing-and-rate-limits#concurrent-request-limits) until its response is returned.

## Supported audio formats

* `transcribe-1-pro` accepts WAV, MP3, AAC (including M4A/MP4), FLAC, Ogg (Opus or Vorbis), WebM/Matroska, and MOV, including browser recordings. For a video file, it uses the first audio track. Send the original file bytes; no conversion is needed.
* Not supported: AIFF, CAF, WMA, AMR, AC-3, and raw (headerless) PCM. These return 400.

**If you use `transcribe-1`:** it accepts WAV, MP3, AAC (including M4A/MP4), FLAC, and Ogg (Opus or Vorbis). Browser WebM recordings (for example, from `MediaRecorder`) may not be accepted; convert them to Ogg/Opus, MP3, or WAV first.

## Errors

Error responses have one of these shapes:

* **`transcribe-1-pro` errors** are JSON with `status`, `message`, `code`, and `request_id`. Branch on `code`, not on `message`.
* **Platform errors**, such as authentication, credit, and concurrency errors, are returned before the request reaches a model. They are JSON with exactly `status` and `message`.
* **Errors from the network edge** in front of the API, such as some 413 and 5xx responses, may not have a JSON body. Handle them by HTTP status.

This illustrative `transcribe-1-pro` error body shows the shape:

```json theme={null}
{
  "status": 400,
  "message": "A human-readable description of the problem.",
  "code": "invalid_parameter",
  "request_id": "5f0c7e2a-9b1d-4c3e-8f6a-2d4b7c9e1a03"
}
```

| Status | `code` | Meaning | Retry? |
| - | - | - | - |
| 400 | `invalid_request` | Malformed request: the body could not be parsed, `audio` is missing, or more than one `audio` was sent. | No |
| 400 | `invalid_parameter` | A parameter has a value it does not accept, or parameters conflict. | No |
| 400 | `invalid_audio` | The audio could not be decoded, or the format is not supported. | No |
| 400 | `audio_too_long` | The audio is longer than 60 minutes. | No |
| 400 | `audio_too_short` | The audio is too short to transcribe (under about 0.08 seconds). | No |
| 400 | — | The request body could not be read. | No |
| 401 | — | Missing or invalid API key. | No |
| 402 | — | Insufficient API credit. | No |
| 413 | `request_too_large` | The request is too large. It may come from the network edge without a JSON body. | No |
| 415 | `unsupported_media_type` | Unsupported `Content-Type`. | No |
| 429 | — (rarely `upstream_rejected`) | Usually your account is at its concurrency limit. No `Retry-After` header is sent. | Yes, with backoff |
| Other 4xx (not 429) | `upstream_rejected` | Rare: the request was rejected during transcription. | No |
| 500 | `internal_error` | Internal error. | Yes, with backoff |
| 502 | — | The service could not be reached. | Yes, with backoff |
| 503 | `upstream_unavailable` | The transcription service is temporarily unavailable. | Yes, with backoff |
| 503 | `upstream_timeout` | Processing took too long. | Yes, with backoff |
| 503 | `excessive_repetition` | The transcription produced unusable repeated output. | Yes, with backoff |
| 503 | `diarization_failed` | Speaker identification failed for this recording. | Yes, with backoff |
| 504 | — | Timed out connecting to the service. | Yes, with backoff |

* A dash means the error has no `code`; a 503 without a `code` also means the service is temporarily unavailable.
* New `code` values may be added; handle unknown codes by HTTP status.
* Retry 429 and 5xx responses with exponential backoff. Do not retry other 4xx responses; change the request first.
* Requests that return an error response are not billed.
* With the Python SDK, the raw error body is in the exception's `body` attribute; parse it with `json.loads` to read `code` and `request_id`.

**If you use `transcribe-1`:** its errors are JSON with `status` and `message`. Rely only on these two fields and on the HTTP status.

See [Errors](/api-reference/errors) for retry and SDK exception examples.


# Text to Speech
Source: https://docs.fish.audio/api-reference/endpoint/openapi-v1/text-to-speech

post /v1/tts
Convert text to speech

<Warning>
  This endpoint only accepts `application/json` and `application/msgpack`.

  For best results, upload reference audio using the [create model](/api-reference/endpoint/model/create-model) before using this one. This improves speech quality and reduces latency.

  To upload audio clips directly, without pre-uploading, serialize the request body with MessagePack as per the [instructions](/features/text-to-speech#direct-api-messagepack).
</Warning>

<Note>
  Audio formats supported:

  * WAV / PCM
    * Sample Rate: 8kHz, 16kHz, 24kHz, 32kHz, 44.1kHz
    * Default Sample Rate: 44.1kHz
    * 16-bit, mono
  * MP3
    * Sample Rate: 32kHz, 44.1kHz
    * Default Sample Rate: 44.1kHz
    * mono
    * Bitrate: 64kbps, 128kbps (default), 192kbps
  * Opus
    * Sample Rate: 48kHz
    * Default Sample Rate: 48kHz
    * mono
    * Bitrate: -1000 (auto), 24kbps, 32kbps (default), 48kbps, 64kbps
</Note>


# WebSocket TTS with Timestamps
Source: https://docs.fish.audio/api-reference/endpoint/openapi-v1/text-to-speech-live-with-timestamps

Stream text and receive speech with word-level timestamps over WebSocket

<Note>
  Connect to `wss://api.fish.audio/v1/tts/live/with-timestamp` with a WebSocket
  client. The endpoint requires a WebSocket upgrade and returns MessagePack
  binary frames. A normal HTTP GET request does not start synthesis.
</Note>

## Before you start

You need a [Fish Audio API key](/developer-guide/getting-started/api-key) and a
WebSocket client that supports custom authentication headers. To run the Python
example on this page, install its dependencies:

```bash theme={null}
pip install ormsgpack websockets
```

Replace `<token>` in the example with your API key and set `reference_id` to your
voice model ID. The `start.request` object accepts the same parameters as the
[Text to Speech API](/api-reference/endpoint/openapi-v1/text-to-speech).
Send text and receive audio concurrently, then keep reading after sending `stop`
until the server sends `finish`.

## Preserve the final timestamps

Process alignment metadata even when an `audio` event contains empty audio bytes:
a trailing event can carry a final alignment correction. Store each non-null
`alignment` by `chunk_seq`, replacing its previous snapshot. A null alignment
does not erase a snapshot you already received.

Add `chunk_audio_offset_sec` to each segment's `start` and `end` to place it on
the session's audio timeline. The optional `time` field measures elapsed server
session time in milliseconds; use the segment timestamps for captions.

After `finish`, you can send another `start` on the same socket. Reset your audio
buffer and stored alignment snapshots for the new session.

## Related endpoints

* [Text to Speech Stream with Timestamps](/api-reference/endpoint/openapi-v1/text-to-speech-stream-with-timestamps): send the full text in one HTTP request and receive audio and timestamps over SSE.
* [WebSocket TTS Streaming](/api-reference/endpoint/websocket/tts-live): stream text and audio without timestamps.


# Text to Speech Stream with Timestamps
Source: https://docs.fish.audio/api-reference/endpoint/openapi-v1/text-to-speech-stream-with-timestamps

post /v1/tts/stream/with-timestamp
Stream generated speech with timestamp alignment snapshots

<Warning>
  This endpoint returns `text/event-stream`. Each SSE `message` event contains
  one JSON payload with a base64-encoded audio chunk.
</Warning>

<Note>
  Use this endpoint when you need both progressive audio delivery and
  text-to-audio alignment data, such as karaoke-style highlighting, word or
  phrase progress indicators, captions synchronized to generated speech, or
  timeline editing.
</Note>

## How the Stream Works

The response is a Server-Sent Events stream. Every event includes:

| Field | Type | Description |
| - | - | - |
| `audio_base64` | `string` | One base64-encoded audio chunk. Concatenate all chunks in arrival order to reconstruct the complete audio. |
| `content` | `string` | Text content described by this event's latest alignment snapshot. Long input can be split into multiple content chunks. |
| `alignment` | `object \| null` | Latest cumulative timestamp snapshot for `chunk_seq`. When present, replace the previous snapshot for that `chunk_seq`; do not append segments. |
| `chunk_seq` | `integer` | Sequence number of the text chunk described by `alignment`. Bucket alignment snapshots by this value. |
| `chunk_audio_offset_sec` | `number` | Absolute start time of this text chunk within the full audio, in seconds. Add this to segment-local `start` and `end` values for a global audio timeline. |

`audio_base64` is the transport stream. `alignment` is a metadata snapshot for
`chunk_seq`. They are delivered together in the same SSE event, but the
alignment is not a per-audio-packet delta.

When `latency` is set to `balanced`, long input can be split into several text
chunks. A chunk may produce multiple non-null alignment snapshots as more audio
is rendered. Each newer snapshot supersedes the previous snapshot for the same
`chunk_seq`.

<Tip>
  Store alignments in a map keyed by `chunk_seq`. On every non-null `alignment`,
  replace the stored value for that key. Do not collect every non-null alignment
  as a separate final result.
</Tip>

## Alignment Shape

Each non-null `alignment` contains the current cumulative timing segments for a
single text chunk:

```json theme={null}
{
  "audio_base64": "SUQzBAAAAAAA...",
  "content": "Hello world",
  "chunk_seq": 0,
  "chunk_audio_offset_sec": 0.0,
  "alignment": {
    "audio_duration": 0.86,
    "segments": [
      {
        "text": "Hello",
        "start": 0,
        "end": 0.42
      },
      {
        "text": "world",
        "start": 0.42,
        "end": 0.86
      }
    ]
  }
}
```

`start` and `end` are measured in seconds from the start of that text chunk's
generated audio. Add `chunk_audio_offset_sec` to get timestamps on the complete
audio timeline.

`alignment` can be `null` before the first snapshot is available or when
alignment is unavailable. After a snapshot exists, later audio events may repeat
the latest snapshot so clients can continue using a simple latest-wins update
model.

## Minimal Request

```bash theme={null}
curl --no-buffer --request POST \
  --url https://api.fish.audio/v1/tts/stream/with-timestamp \
  --header 'Authorization: Bearer <token>' \
  --header 'Content-Type: application/json' \
  --header 'model: s2-pro' \
  --data '{
    "text": "Hello! Welcome to Fish Audio.",
    "reference_id": "model-id",
    "format": "opus",
    "latency": "balanced"
  }'
```

## Parsing the Stream

The stream payload uses standard SSE framing. Parse each `data:` line as JSON,
append every decoded `audio_base64` chunk to your audio buffer, and replace the
latest alignment snapshot for `chunk_seq` whenever `alignment` is non-null.

<Tabs>
  <Tab title="Python">
    ```python theme={null}
    import base64
    import json
    import requests

    response = requests.post(
        "https://api.fish.audio/v1/tts/stream/with-timestamp",
        headers={
            "Authorization": "Bearer <token>",
            "Content-Type": "application/json",
            "model": "s2-pro",
        },
        json={
            "text": "Hello! Welcome to Fish Audio.",
            "reference_id": "model-id",
            "format": "opus",
            "latency": "balanced",
        },
        stream=True,
    )

    audio_chunks = []
    alignment_by_chunk = {}

    for line in response.iter_lines(decode_unicode=True):
        if not line or not line.startswith("data: "):
            continue

        event = json.loads(line.removeprefix("data: "))
        audio_chunks.append(base64.b64decode(event["audio_base64"]))

        if event["alignment"] is not None:
            alignment_by_chunk[event["chunk_seq"]] = {
                "content": event["content"],
                "offset": event["chunk_audio_offset_sec"],
                "alignment": event["alignment"],
            }

    audio = b"".join(audio_chunks)
    ```
  </Tab>

  <Tab title="Node.js">
    ```javascript theme={null}
    const response = await fetch(
      "https://api.fish.audio/v1/tts/stream/with-timestamp",
      {
        method: "POST",
        headers: {
          Authorization: "Bearer <token>",
          "Content-Type": "application/json",
          model: "s2-pro",
        },
        body: JSON.stringify({
          text: "Hello! Welcome to Fish Audio.",
          reference_id: "model-id",
          format: "opus",
          latency: "balanced",
        }),
      }
    );

    const audioChunks = [];
    const alignmentByChunk = new Map();
    const decoder = new TextDecoder();
    let buffer = "";

    for await (const chunk of response.body) {
      buffer += decoder.decode(chunk, { stream: true });
      const events = buffer.split("\n\n");
      buffer = events.pop() ?? "";

      for (const eventText of events) {
        const dataLine = eventText
          .split("\n")
          .find((line) => line.startsWith("data: "));

        if (!dataLine) continue;

        const event = JSON.parse(dataLine.slice(6));
        audioChunks.push(Buffer.from(event.audio_base64, "base64"));

        if (event.alignment !== null) {
          alignmentByChunk.set(event.chunk_seq, {
            content: event.content,
            offset: event.chunk_audio_offset_sec,
            alignment: event.alignment,
          });
        }
      }
    }

    const audio = Buffer.concat(audioChunks);
    ```
  </Tab>
</Tabs>

## Handling Split Content Chunks

Long input can produce multiple text chunks. Treat audio and alignment as two
related streams:

1. Append every decoded `audio_base64` chunk in event order. Do this even when `alignment` is `null`.
2. For non-null `alignment`, replace the stored snapshot for `chunk_seq`.
3. Convert each snapshot's local segment times into global times by adding `chunk_audio_offset_sec`.

<Note>
  `audio_base64` chunks are transport chunks, not sentence or word boundaries.
  Do not try to align each audio chunk individually. Use `alignment.segments`
  plus `chunk_audio_offset_sec` for text timing.
</Note>

For example, if an event has `chunk_audio_offset_sec: 16.24`, add `16.24`
seconds to every segment in that event's `alignment` before rendering it on the
complete audio timeline.

<Tabs>
  <Tab title="Python">
    ```python theme={null}
    def build_global_timeline(alignment_by_chunk):
        timeline = []

        for chunk_seq, item in sorted(alignment_by_chunk.items()):
            offset_seconds = item["offset"]
            alignment = item["alignment"]

            for segment in alignment["segments"]:
                timeline.append({
                    "text": segment["text"],
                    "start": segment["start"] + offset_seconds,
                    "end": segment["end"] + offset_seconds,
                    "chunk_seq": chunk_seq,
                })

        return timeline
    ```
  </Tab>

  <Tab title="Node.js">
    ```javascript theme={null}
    function buildGlobalTimeline(alignmentByChunk) {
      const timeline = [];

      for (const [chunkSeq, item] of [...alignmentByChunk.entries()].sort(
        ([a], [b]) => a - b
      )) {
        for (const segment of item.alignment.segments) {
          timeline.push({
            text: segment.text,
            start: segment.start + item.offset,
            end: segment.end + item.offset,
            chunk_seq: chunkSeq,
          });
        }
      }

      return timeline;
    }
    ```
  </Tab>
</Tabs>

## Format Guidance

For timestamped streaming, we recommend `opus` with the default 48 kHz sample
rate when your client supports it. Opus is designed for streaming and gives the
best balance of quality, latency, and bandwidth for this endpoint.

`wav` and `pcm` avoid lossy codec artifacts and are straightforward to align,
but they produce much larger payloads. Use them when you need uncompressed
audio, direct sample-level processing, or a playback pipeline that already
expects raw audio.

<Warning>
  Use `mp3` only when broad playback compatibility is more important than the
  cleanest streaming boundaries. MP3 encoding uses overlapping audio windows, so
  its encoded chunks may not line up as neatly with timestamp snapshot updates
  as Opus.
</Warning>

This endpoint accepts the same TTS request fields as the [Text to Speech API](/api-reference/endpoint/openapi-v1/text-to-speech), including `reference_id`, `references`, `prosody`, `temperature`, `top_p`, `chunk_length`, `format`, and `latency`.


# Voice Design
Source: https://docs.fish.audio/api-reference/endpoint/openapi-v1/voice-design

post /v1/voice-design
Generate candidate voices from a prompt

<Warning>
  This endpoint only accepts `application/json`.

  You must include the `model: voice-design-1` header. Extra request fields are rejected.
</Warning>

<Note>
  A successful request returns generated voice candidates with `audio_base64`
  audio payloads. Decode the base64 value to write the candidate audio to a
  file.
</Note>

## Example

```bash theme={null}
curl --request POST https://api.fish.audio/v1/voice-design \
  --header "Authorization: Bearer $FISH_API_KEY" \
  --header "Content-Type: application/json" \
  --header "model: voice-design-1" \
  --data '{
    "instruction": "Warm, confident studio narrator with a natural tone",
    "reference_text": "Welcome to Fish Audio.",
    "language": "en",
    "n": 2,
    "speed": 1,
    "num_step": 32,
    "guidance_scale": 2,
    "instruct_guidance_scale": 0,
    "seed": 42
  }'
```

## Usage notes

* `instruction` is required and must be 1 to 2000 characters.
* `reference_text` is optional preview text and can be up to 150 characters.
* `n` controls how many candidates are returned. The supported range is 1 to 4.
* `seed` is optional and can help reproduce candidate generation.
* The endpoint is stateless: it does not create batches, samples, voice models, or presigned URLs.
* Billing happens once per successful generation request, not once per candidate.


# Get API Credit
Source: https://docs.fish.audio/api-reference/endpoint/wallet/get-api-credit

get /wallet/{user_id}/api-credit
Get current API credit balance



# Get User Package
Source: https://docs.fish.audio/api-reference/endpoint/wallet/get-user-package

get /wallet/{user_id}/package
Get current user premium information



# WebSocket TTS Streaming
Source: https://docs.fish.audio/api-reference/endpoint/websocket/tts-live

Real-time text-to-speech streaming via WebSocket

<Note>
  The WebSocket TTS endpoint enables bidirectional streaming for low-latency text-to-speech generation with MessagePack serialization.
</Note>

<Note>
  The `request` payload inside `StartEvent` uses the same parameters as the HTTP [Text to Speech API](/api-reference/endpoint/openapi-v1/text-to-speech). For more detailed field guidance, model-specific behavior, and examples, see that page. In WebSocket mode, `request.text` is typically empty in `StartEvent`, and the text content is sent through subsequent `TextEvent` messages.
</Note>


# Errors
Source: https://docs.fish.audio/api-reference/errors

HTTP status codes, the error response shape, and how to handle them in any language

Fish Audio errors come back as JSON with a `message` and a `status`:

```json theme={null}
{ "message": "Invalid Token", "status": 401 }
```

(A request whose body can't be parsed returns a plain-text parse error instead.)

Some endpoints add fields. For example, `transcribe-1-pro` speech-to-text errors also include `code` and `request_id`; see [Speech to Text errors](/api-reference/endpoint/openapi-v1/speech-to-text#errors). Errors returned by the network edge in front of the API, such as some `413` or `5xx` responses, may not be JSON.

## Status codes

| Status | Meaning | What to do |
| - | - | - |
| `400` | Bad request: invalid parameters, or a `reference_id` / voice that doesn't exist | Fix the request; read the `message`. |
| `401` | Invalid or missing API key | Send `Authorization: Bearer <key>`; check it on [API Keys](https://fish.audio/app/api-keys). |
| `402` | Insufficient credits | Top up on [API Billing](https://fish.audio/app/developers/billing/). |
| `403` | Not permitted for this key/resource | Check the key's scope and the resource owner. |
| `404` | Model or voice not found | Verify the `model_id` / `reference_id`. |
| `429` | Too many requests, for example when your account is at its [concurrency limit](/developer-guide/models-pricing/pricing-and-rate-limits#concurrent-request-limits) | Back off and retry (see below). |
| `5xx` | Server error | Retry with backoff; if it persists, contact support. |

## Retries

Retry `429` and `5xx` with exponential backoff. Don't retry other `4xx` codes; they won't succeed without a change to the request.

```python theme={null}
import time
from fishaudio import FishAudio
from fishaudio.exceptions import RateLimitError, APIError

client = FishAudio()

for attempt in range(5):
    try:
        audio = client.tts.convert(text="Hello!")
        break
    except RateLimitError:
        time.sleep(2 ** attempt)            # 1s, 2s, 4s, ...
    except APIError as e:
        if e.status >= 500:
            time.sleep(2 ** attempt)
        else:
            raise                            # 4xx: fix the request
```

## Handling errors in the SDKs

Both SDKs raise typed exceptions you can branch on. The base class carries the status and the parsed body.

<CodeGroup>
  ```python Python theme={null}
  from fishaudio.exceptions import (
      AuthenticationError,  # 401
      RateLimitError,       # 429
      NotFoundError,        # 404
      APIError,             # any other HTTP error; has .status and .message
      FishAudioError,       # base class for all SDK errors
  )

  try:
      audio = client.tts.convert(text="Hello!")
  except AuthenticationError:
      ...  # invalid or missing key
  except RateLimitError:
      ...  # back off and retry
  except NotFoundError:
      ...  # bad reference_id / model id
  except APIError as e:
      print(e.status, e.message)  # 400, 402, 5xx, ...
  ```

  ```javascript JavaScript theme={null}
  import {
    UnauthorizedError,         // 401
    TooEarlyError,             // 429
    NotFoundError,             // 404
    BadRequestError,           // 400
    UnprocessableEntityError,  // 422
    FishAudioError,            // base; has .statusCode and .body
  } from "fish-audio";

  try {
    await client.textToSpeech.convert({ text: "Hello!" }, "s2-pro");
  } catch (err) {
    if (err instanceof UnauthorizedError) {
      // invalid or missing key
    } else if (err instanceof TooEarlyError) {
      // back off and retry
    } else if (err instanceof FishAudioError) {
      console.error(err.statusCode, err.body);
    }
  }
  ```
</CodeGroup>

<Note>
  Audio playback via `play()` needs `ffmpeg`. If it's missing, the Python SDK raises
  `DependencyError`. Install `ffmpeg` or save the audio to a file instead.
</Note>


# Introduction
Source: https://docs.fish.audio/api-reference/introduction

How to use the Fish Audio API

## Welcome

You can generate a new API key at [https://fish.audio/app/api-keys/](https://fish.audio/app/api-keys/).

## Quick Start

See our [Quick Start](/developer-guide/getting-started/quickstart) guide to generate audio in under 2 minutes.

## Errors

Errors return a JSON body with a `message` and a `status`; some endpoints add fields. Errors from the network edge in front of the API, such as some `413` or `5xx` responses, may not have a JSON body. See [Errors](/api-reference/errors) for the full status-code table, retry guidance, and SDK exception handling.

## OpenAPI Schema

Fish Audio publishes a canonical OpenAPI schema at [https://api.fish.audio/openapi.json](https://api.fish.audio/openapi.json).
When working with AI coding agents or IDE assistants, mention this schema URL as part of your prompt or project context so the agent can understand Fish Audio's endpoints, request and response models, authentication requirements, and supported parameters directly from the machine-readable API contract.

## Distributed Tracing

Fish Audio inference APIs accept the W3C `traceparent` header so your business-side trace and Fish Audio's inference-side trace can share the same trace ID. See [Tracing & Performance Analysis](/api-reference/observability) for supported endpoints, examples, and enterprise performance analysis details.

## Create a Voice Clone

Use our [/model endpoint](/api-reference/endpoint/model/create-model) to create a voice clone model.

## Generate Speech

Use our [/v1/tts endpoint](/api-reference/endpoint/openapi-v1/text-to-speech) to generate speech.

## Design a Voice

Use our [/v1/voice-design endpoint](/api-reference/endpoint/openapi-v1/voice-design) to generate candidate voices from a prompt.

## Real-time Streaming

Use our [Python SDK](/features/realtime-streaming) or [JavaScript SDK](/features/realtime-streaming) for real-time audio streaming with WebSocket.

## Rate Limits

You can find the rate limits for each endpoint in the [Rate Limits](/developer-guide/models-pricing/pricing-and-rate-limits) section.


# Tracing & Performance Analysis
Source: https://docs.fish.audio/api-reference/observability

Correlate Fish Audio inference spans with your own distributed traces

Fish Audio inference APIs accept the standard W3C Trace Context `traceparent`
header. Send this header when your application already has an active trace and
you want Fish Audio edge, inference, alignment, and upstream ASR spans to appear
under the same trace ID.

Supported inference surfaces:

* `POST /v1/tts`
* `POST /v1/tts/stream/with-timestamp`
* `POST /v1/asr`
* `wss://api.fish.audio/v1/tts/live`

If `traceparent` is omitted or invalid, Fish Audio starts a new trace for the
request. Tracing does not change authentication, rate limits, billing, request
priority, or generated output.

## Header Format

Use the W3C `traceparent` format:

```text theme={null}
traceparent: 00-<trace-id>-<parent-id>-<trace-flags>
```

Example:

```text theme={null}
traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01
```

In this example, `4bf92f3577b34da6a3ce929d0e0e4736` is the trace ID. Fish Audio
continues that trace ID when it calls the inference backend and related services.

<Tip>
  Prefer letting your tracing SDK inject `traceparent` instead of hand-building
  it. For each Fish Audio request, create a child span in your business service
  and inject that span context into the outgoing HTTP or WebSocket headers.
</Tip>

## Bind Business and Inference Traces

To bind your business-side trace to Fish Audio's inference-side trace:

1. Start or continue a trace in your application for the user workflow.
2. Create a child span around the Fish Audio request.
3. Inject that span context into the request headers.
4. Send the request with `traceparent`.
5. Use the same trace ID in your observability tool to inspect both your
   business spans and Fish Audio inference spans.

For multiple Fish Audio calls in one workflow, keep the same trace ID by using
the same parent trace, but let your tracing SDK create a fresh parent/span ID for
each outgoing request.

## REST Example

```bash theme={null}
TRACEPARENT="00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01"

curl --request POST https://api.fish.audio/v1/tts \
  --header "Authorization: Bearer $FISH_API_KEY" \
  --header "Content-Type: application/json" \
  --header "model: s2-pro" \
  --header "traceparent: $TRACEPARENT" \
  --data '{
    "text": "Hello from a traced Fish Audio request.",
    "reference_id": "model-id",
    "format": "mp3"
  }' \
  --output out.mp3
```

## OpenTelemetry Examples

<Tabs>
  <Tab title="Python">
    ```python theme={null}
    import os
    import requests
    from opentelemetry import trace
    from opentelemetry.propagate import inject

    tracer = trace.get_tracer("my-service")

    with tracer.start_as_current_span("fish_audio.tts"):
        headers = {
            "Authorization": f"Bearer {os.environ['FISH_API_KEY']}",
            "Content-Type": "application/json",
            "model": "s2-pro",
        }
        inject(headers)

        response = requests.post(
            "https://api.fish.audio/v1/tts",
            headers=headers,
            json={
                "text": "Hello from a traced request.",
                "reference_id": "model-id",
                "format": "mp3",
            },
            timeout=60,
        )
        response.raise_for_status()
    ```
  </Tab>

  <Tab title="Node.js">
    ```javascript theme={null}
    import { context, propagation, trace } from "@opentelemetry/api";

    const tracer = trace.getTracer("my-service");
    const span = tracer.startSpan("fish_audio.tts");
    const ctx = trace.setSpan(context.active(), span);

    try {
      await context.with(ctx, async () => {
        const headers = {
          Authorization: `Bearer ${process.env.FISH_API_KEY}`,
          "Content-Type": "application/json",
          model: "s2-pro",
        };
        propagation.inject(ctx, headers);

        const response = await fetch("https://api.fish.audio/v1/tts", {
          method: "POST",
          headers,
          body: JSON.stringify({
            text: "Hello from a traced request.",
            reference_id: "model-id",
            format: "mp3",
          }),
        });

        if (!response.ok) {
          throw new Error(`${response.status} ${await response.text()}`);
        }
      });
    } finally {
      span.end();
    }
    ```
  </Tab>
</Tabs>

## WebSocket Tracing

Pass `traceparent` on the WebSocket upgrade request:

```javascript theme={null}
import WebSocket from "ws";

const ws = new WebSocket("wss://api.fish.audio/v1/tts/live", {
  headers: {
    Authorization: `Bearer ${process.env.FISH_API_KEY}`,
    model: "s2-pro",
    traceparent: "00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01",
  },
});
```

<Note>
  Browser WebSocket APIs do not allow custom headers. For browser clients,
  inject `traceparent` from a trusted server-side proxy or use REST endpoints
  where your frontend can send the header through `fetch`.
</Note>

## Speech-to-Text Request IDs

`transcribe-1-pro` responses from `POST /v1/asr` include a `request_id` field
in the JSON body, on success and on its own error responses, and the same value
in the `x-request-id` response header. Platform errors, such as authentication,
credit, and concurrency errors, and errors from the network edge do not carry
it. Include it when you contact support about a transcription request. See
[Speech to Text](/api-reference/endpoint/openapi-v1/speech-to-text#read-the-response).

## Enterprise Performance Analysis

<Info>
  Fish Audio supports trace-based performance analysis for enterprise customers
  with a signed enterprise agreement. When this support is enabled, share the
  W3C trace ID with Fish Audio support so we can correlate your business span
  with Fish Audio edge routing, backend inference, reference-audio encoding,
  TTFT, response-time, alignment, and upstream ASR spans.
</Info>

Only share the trace ID or `traceparent` value needed for investigation. Do not
place API keys, user identifiers, transcripts, audio URLs, or other sensitive
data in trace IDs or span names.


# JavaScript SDK Reference
Source: https://docs.fish.audio/api-reference/sdk/javascript/api-reference

Complete reference for Fish Audio JavaScript SDK

## Client

Import and initialize the client:

```typescript theme={null}
import { FishAudioClient } from "fish-audio";
const fishAudio = new FishAudioClient({ apiKey: process.env.FISH_API_KEY });
```

## Text to Speech

### convert()

Generate speech from text.

```typescript theme={null}
const audio = await fishAudio.textToSpeech.convert({ text: "Hello" });
```

Parameters: `request` (TTSRequest), `model?` (Backends) <br />
Returns: `Promise<ReadableStream<Uint8Array>>`

### convertRealtime()

Realtime streaming TTS over WebSocket.

```typescript theme={null}
async function* textStream() { yield "Hello, "; yield "world!"; }
const conn = await fishAudio.textToSpeech.convertRealtime({ text: "" }, textStream());
```

Parameters: `request` (TTSRequest with `text: ""`), `textStream` (`AsyncIterable<string>`), `backend?` (Backends) <br />
Returns: `RealtimeConnection` (`EventEmitter`-like connection) emitting `RealtimeEvents`

## Speech to Text

### convert()

Transcribe audio to text.

```typescript theme={null}
const res = await fishAudio.speechToText.convert(
  { audio: myAudio },
  { headers: { model: "transcribe-1-pro" } }
);
console.log(res.text);
```

Parameters: `request` (STTRequest), `requestOptions?` (for example `headers` and `timeoutInSeconds`) <br />
Returns: `STTResponse`

Select the model with the `model` header in `requestOptions.headers`. Without it, the request is served and billed as `transcribe-1`. See [Speech to Text](/features/speech-to-text).

## Voices

### search()

List/search available voice models.

```typescript theme={null}
const results = await fishAudio.voices.search();
```

Parameters: `request?` (ModelListRequest) <br />
Returns: `ModelListResponse`

### get()

Get model details.

```typescript theme={null}
const model = await fishAudio.voices.get("model_id");
```

Parameters: `voiceId` (string) <br />
Returns: `ModelEntity`

### ivc.create()

Create a new voice model from audio samples.

```typescript theme={null}
const res = await fishAudio.voices.ivc.create({ title, voices: [file], cover_image: file });
```

Parameters: `request` (ModelCreateRequest) <br />
Returns: `ModelEntity`

### update()

Update model metadata.

```typescript theme={null}
await fishAudio.voices.update("model_id", { title: "New Title" });
```

Parameters: `voiceId` (string), `request` (UpdateModelRequest) <br />
Returns: `UpdateVoiceResponse`

### delete()

Delete a model.

```typescript theme={null}
await fishAudio.voices.delete("model_id");
```

Parameters: `voiceId` (string) <br />
Returns: `DeleteVoiceResponse`

## User

### get\_api\_credit()

Check API credit balance.

```typescript theme={null}
await fishAudio.user.get_api_credit();
```

Returns: `APICreditResponse`

### get\_package()

Get subscription package details.

```typescript theme={null}
await fishAudio.user.get_package();
```

Returns: `PackageResponse`

## Request Classes

### TTSRequest

Text-to-speech parameters.

```typescript theme={null}
{
  text: "Hello",
  reference_id: "model_id",
  references: [ { audio: File, text: "sample" } ],
  format: "mp3",
  prosody: { speed: 1.0, volume: 0 },
}
```

Fields: `text`, `reference_id`, `references`, `format`, `mp3_bitrate`, `opus_bitrate`, `sample_rate`, `prosody`, `latency`, `chunk_length`, `normalize`, `temperature`, `top_p`

### STTRequest

Speech-to-text parameters.

```typescript theme={null}
{ audio: File, language?: "en", ignore_timestamps?: boolean }
```

Fields: `audio`, `language?`, `ignore_timestamps?`

### ReferenceAudio

Reference audio for voice cloning.

```typescript theme={null}
{ audio: File, text: "spoken text" }
```

Fields: `audio`, `text`

### Prosody

Speed and volume control.

```typescript theme={null}
{ speed: 1.2, volume: 5 }
```

Fields: `speed` (0.5–2.0), `volume` (-20 to 20)

### Backends

The backend model to use.

```typescript theme={null}
Backends = 's1' | 's2-pro';
```

The API also accepts `s2.1-pro` and `s2.1-pro-free`; the SDK's `Backends` type has not been updated yet, so pass them with a type assertion (for example `'s2.1-pro' as Backends`) or call the [REST API](/api-reference/endpoint/openapi-v1/text-to-speech) directly.

## Response Classes

### STTResponse

Transcription result.

```typescript theme={null}
response.text      // Complete transcription
response.duration  // Duration in seconds
response.segments  // ASRSegment[]
```

### ASRSegment

Timestamped text segment.

Fields: `text` (string), `start` (number, seconds), `end` (number, seconds)

### ModelEntity

Voice model information.

Fields: `_id`, `title`, `description`, `visibility`, `created_at`, `updated_at`, `tags`

### ModelListResponse

List response for voices.

Fields: `items` (ModelEntity\[]), `total` (number)

### APICreditResponse

API credit information.

Fields: `_id` (string), `user_id` (string), `credit` (string), `created_at` (string), `updated_at` (string), `has_phone_sha256` (boolean), `has_free_credit?` (boolean)

### PackageResponse

Subscription package details.

Fields: `user_id` (string), `type` (string), `total` (number), `balance` (number), `created_at` (string), `updated_at` (string), `finished_at` (string)

## WebSocket Classes

### RealtimeEvents

Events emitted by `convertRealtime` connections.

| Event | Meaning |
| - | - |
| `OPEN` | Connection established |
| `AUDIO_CHUNK` | Audio chunk received |
| `ERROR` | Error occurred |
| `CLOSE` | Connection closed |

## Event Classes

### StartEvent

Stream start event.

Fields: `event` ("start"), `request` (TTSRequest)

### TextEvent

Text chunk event.

Fields: `event` ("text"), `text` (string)

### FlushEvent

Flush text chunks event.

Fields: `event` ("flush")

### CloseEvent

Stream close event.

Fields: `event` ("stop")

## Exceptions

### FishAudioError

Generic error with status code, body, rawResponse.

### FishAudioTimeoutError

Connection timeout error.


# Client
Source: https://docs.fish.audio/api-reference/sdk/python/client



# fishaudio.client

Main Fish Audio client classes.

## FishAudio Objects

```python theme={null}
class FishAudio()
```

Synchronous Fish Audio API client.

**Example**:

```python theme={null}
from fishaudio import FishAudio

client = FishAudio(api_key="your_api_key")

# Generate speech
audio = client.tts.convert(text="Hello world")
with open("output.mp3", "wb") as f:
    for chunk in audio:
        f.write(chunk)

# List voices
voices = client.voices.list(page_size=20)
print(f"Found {voices.total} voices")
```

#### \_\_init\_\_

```python theme={null}
def __init__(*,
             api_key: Optional[str] = None,
             base_url: str = "https://api.fish.audio",
             timeout: float = 240.0,
             httpx_client: Optional[httpx.Client] = None)
```

Initialize Fish Audio client.

**Arguments**:

* `api_key` - API key (can also use FISH\_API\_KEY env var)
* `base_url` - API base URL
* `timeout` - Request timeout in seconds
* `httpx_client` - Optional custom HTTP client

#### tts

```python theme={null}
@property
def tts() -> TTSClient
```

Access TTS (text-to-speech) operations.

#### asr

```python theme={null}
@property
def asr() -> ASRClient
```

Access ASR (speech-to-text) operations.

#### voices

```python theme={null}
@property
def voices() -> VoicesClient
```

Access voice management operations.

#### account

```python theme={null}
@property
def account() -> AccountClient
```

Access account/billing operations.

#### close

```python theme={null}
def close() -> None
```

Close the HTTP client.

## AsyncFishAudio Objects

```python theme={null}
class AsyncFishAudio()
```

Asynchronous Fish Audio API client.

**Example**:

```python theme={null}
from fishaudio import AsyncFishAudio

async def main():
    client = AsyncFishAudio(api_key="your_api_key")

    # Generate speech
    audio = client.tts.convert(text="Hello world")
    async with aiofiles.open("output.mp3", "wb") as f:
        async for chunk in audio:
            await f.write(chunk)

    # List voices
    voices = await client.voices.list(page_size=20)
    print(f"Found {voices.total} voices")

asyncio.run(main())
```

#### \_\_init\_\_

```python theme={null}
def __init__(*,
             api_key: Optional[str] = None,
             base_url: str = "https://api.fish.audio",
             timeout: float = 240.0,
             httpx_client: Optional[httpx.AsyncClient] = None)
```

Initialize async Fish Audio client.

**Arguments**:

* `api_key` - API key (can also use FISH\_API\_KEY env var)
* `base_url` - API base URL
* `timeout` - Request timeout in seconds
* `httpx_client` - Optional custom async HTTP client

#### tts

```python theme={null}
@property
def tts() -> AsyncTTSClient
```

Access TTS (text-to-speech) operations.

#### asr

```python theme={null}
@property
def asr() -> AsyncASRClient
```

Access ASR (speech-to-text) operations.

#### voices

```python theme={null}
@property
def voices() -> AsyncVoicesClient
```

Access voice management operations.

#### account

```python theme={null}
@property
def account() -> AsyncAccountClient
```

Access account/billing operations.

#### close

```python theme={null}
async def close() -> None
```

Close the HTTP client.


# Core
Source: https://docs.fish.audio/api-reference/sdk/python/core



# fishaudio.core.client\_wrapper

HTTP client wrapper for managing requests and authentication.

## BaseClientWrapper Objects

```python theme={null}
class BaseClientWrapper()
```

Base wrapper with shared logic for sync/async clients.

#### get\_headers

```python theme={null}
def get_headers(
        additional_headers: Optional[dict[str, str]] = None) -> dict[str, str]
```

Build headers including authentication and user agent.

## ClientWrapper Objects

```python theme={null}
class ClientWrapper(BaseClientWrapper)
```

Wrapper for httpx.Client that handles authentication and error handling.

#### request

```python theme={null}
def request(method: str,
            path: str,
            *,
            request_options: Optional[RequestOptions] = None,
            **kwargs: Any) -> httpx.Response
```

Make an HTTP request with error handling.

**Arguments**:

* `method` - HTTP method (GET, POST, etc.)
* `path` - API endpoint path
* `request_options` - Optional request-level overrides
* `**kwargs` - Additional arguments to pass to httpx.request

**Returns**:

httpx.Response object

**Raises**:

* `APIError` - On non-2xx responses

#### client

```python theme={null}
@property
def client() -> httpx.Client
```

Get underlying httpx.Client for advanced usage (e.g., WebSockets).

#### close

```python theme={null}
def close() -> None
```

Close the HTTP client.

## AsyncClientWrapper Objects

```python theme={null}
class AsyncClientWrapper(BaseClientWrapper)
```

Wrapper for httpx.AsyncClient that handles authentication and error handling.

#### request

```python theme={null}
async def request(method: str,
                  path: str,
                  *,
                  request_options: Optional[RequestOptions] = None,
                  **kwargs: Any) -> httpx.Response
```

Make an async HTTP request with error handling.

**Arguments**:

* `method` - HTTP method (GET, POST, etc.)
* `path` - API endpoint path
* `request_options` - Optional request-level overrides
* `**kwargs` - Additional arguments to pass to httpx.request

**Returns**:

httpx.Response object

**Raises**:

* `APIError` - On non-2xx responses

#### client

```python theme={null}
@property
def client() -> httpx.AsyncClient
```

Get underlying httpx.AsyncClient for advanced usage (e.g., WebSockets).

#### close

```python theme={null}
async def close() -> None
```

Close the HTTP client.

# fishaudio.core.request\_options

Request-level options for API calls.

## RequestOptions Objects

```python theme={null}
class RequestOptions()
```

Options that can be provided on a per-request basis to override client defaults.

**Attributes**:

* `timeout` - Override the client's default timeout (in seconds)
* `max_retries` - Override the client's default max retries
* `additional_headers` - Additional headers to include in the request
* `additional_query_params` - Additional query parameters to include

#### get\_timeout

```python theme={null}
def get_timeout() -> Optional[httpx.Timeout]
```

Convert timeout to httpx.Timeout if set.

# fishaudio.core.iterators

Audio stream wrappers with collection utilities.

## AudioStream Objects

```python theme={null}
class AudioStream()
```

Wrapper for sync audio byte streams with collection utilities.

This class wraps an iterator of audio bytes and provides a convenient
`.collect()` method to gather all chunks into a single bytes object.

**Examples**:

```python theme={null}
from fishaudio import FishAudio

client = FishAudio(api_key="...")

# Collect all audio at once
audio = client.tts.stream(text="Hello!").collect()

# Or stream chunks manually
for chunk in client.tts.stream(text="Hello!"):
    process_chunk(chunk)
```

#### \_\_init\_\_

```python theme={null}
def __init__(iterator: Iterator[bytes])
```

Initialize the audio iterator wrapper.

**Arguments**:

* `iterator` - The underlying iterator of audio bytes

#### \_\_iter\_\_

```python theme={null}
def __iter__() -> Iterator[bytes]
```

Allow direct iteration over audio chunks.

#### collect

```python theme={null}
def collect() -> bytes
```

Collect all audio chunks into a single bytes object.

This consumes the iterator and returns all audio data as bytes.
After calling this method, the iterator cannot be used again.

**Returns**:

Complete audio data as bytes

**Examples**:

```python theme={null}
audio = client.tts.stream(text="Hello!").collect()
with open("output.mp3", "wb") as f:
    f.write(audio)
```

## AsyncAudioStream Objects

```python theme={null}
class AsyncAudioStream()
```

Wrapper for async audio byte streams with collection utilities.

This class wraps an async iterator of audio bytes and provides a convenient
`.collect()` method to gather all chunks into a single bytes object.

**Examples**:

```python theme={null}
from fishaudio import AsyncFishAudio

client = AsyncFishAudio(api_key="...")

# Collect all audio at once
stream = await client.tts.stream(text="Hello!")
audio = await stream.collect()

# Or stream chunks manually
async for chunk in await client.tts.stream(text="Hello!"):
    await process_chunk(chunk)
```

#### \_\_init\_\_

```python theme={null}
def __init__(async_iterator: AsyncIterator[bytes])
```

Initialize the async audio iterator wrapper.

**Arguments**:

* `async_iterator` - The underlying async iterator of audio bytes

#### \_\_aiter\_\_

```python theme={null}
def __aiter__() -> AsyncIterator[bytes]
```

Allow direct async iteration over audio chunks.

#### collect

```python theme={null}
async def collect() -> bytes
```

Collect all audio chunks into a single bytes object.

This consumes the async iterator and returns all audio data as bytes.
After calling this method, the iterator cannot be used again.

**Returns**:

Complete audio data as bytes

**Examples**:

```python theme={null}
stream = await client.tts.stream(text="Hello!")
audio = await stream.collect()
with open("output.mp3", "wb") as f:
    f.write(audio)
```

# fishaudio.core.websocket\_options

WebSocket-level options for WebSocket connections.

## WebSocketOptions Objects

```python theme={null}
class WebSocketOptions()
```

Options for configuring WebSocket connections.

These options are passed directly to httpx\_ws's connect\_ws/aconnect\_ws functions.
For complete documentation, see [https://frankie567.github.io/httpx-ws/reference/httpx\_ws/](https://frankie567.github.io/httpx-ws/reference/httpx_ws/)

**Attributes**:

* `keepalive_ping_timeout_seconds` - Maximum delay the client will wait for an answer
  to its Ping event. If the delay is exceeded, WebSocketNetworkError will be
  raised and the connection closed. Default: 20 seconds.
* `keepalive_ping_interval_seconds` - Interval at which the client will automatically
  send a Ping event to keep the connection alive. Set to None to disable this
  mechanism. Default: 20 seconds.
* `max_message_size_bytes` - Message size in bytes to receive from the server.
* `Default` - 65536 bytes (64 KiB).
* `queue_size` - Size of the queue where received messages will be held until they
  are consumed. If the queue is full, the client will stop receiving messages
  from the server until the queue has room available. Default: 512.

**Notes**:

Parameter descriptions adapted from httpx\_ws documentation.

#### to\_httpx\_ws\_kwargs

```python theme={null}
def to_httpx_ws_kwargs() -> dict[str, Any]
```

Convert to kwargs dict for httpx\_ws aconnect\_ws/connect\_ws.

# fishaudio.core.omit

OMIT sentinel for distinguishing None from not-provided parameters.


# Exceptions
Source: https://docs.fish.audio/api-reference/sdk/python/exceptions



# fishaudio.exceptions

Custom exceptions for the Fish Audio SDK.

## FishAudioError Objects

```python theme={null}
class FishAudioError(Exception)
```

Base exception for all Fish Audio SDK errors.

## APIError Objects

```python theme={null}
class APIError(FishAudioError)
```

Raised when the API returns an error response.

## AuthenticationError Objects

```python theme={null}
class AuthenticationError(APIError)
```

Raised when authentication fails (401).

## PermissionError Objects

```python theme={null}
class PermissionError(APIError)
```

Raised when permission is denied (403).

## NotFoundError Objects

```python theme={null}
class NotFoundError(APIError)
```

Raised when a resource is not found (404).

## RateLimitError Objects

```python theme={null}
class RateLimitError(APIError)
```

Raised when rate limit is exceeded (429).

## ServerError Objects

```python theme={null}
class ServerError(APIError)
```

Raised when the server encounters an error (5xx).

## WebSocketError Objects

```python theme={null}
class WebSocketError(FishAudioError)
```

Raised when WebSocket connection or streaming fails.

## ValidationError Objects

```python theme={null}
class ValidationError(FishAudioError)
```

Raised when request validation fails.

## DependencyError Objects

```python theme={null}
class DependencyError(FishAudioError)
```

Raised when a required dependency is missing.


# Overview
Source: https://docs.fish.audio/api-reference/sdk/python/overview

Fish Audio Python SDK for text-to-speech and voice cloning

![python.png](https://raw.githubusercontent.com/fishaudio/fish-audio-python/refs/heads/main/.github/assets/python.png)

# Fish Audio Python SDK

[![PyPI version](https://img.shields.io/pypi/v/fish-audio-sdk.svg)](https://badge.fury.io/py/fish-audio-sdk)
[![Python Version](https://img.shields.io/badge/python-3.9+-blue)](https://pypi.org/project/fish-audio-sdk/)
[![PyPI - Downloads](https://img.shields.io/pypi/dm/fish-audio-sdk)](https://pypi.org/project/fish-audio-sdk/)
[![codecov](https://img.shields.io/codecov/c/github/fishaudio/fish-audio-python)](https://codecov.io/gh/fishaudio/fish-audio-python)
[![License](https://img.shields.io/github/license/fishaudio/fish-audio-python)](https://github.com/fishaudio/fish-audio-python/blob/main/LICENSE)

The official Python library for the Fish Audio API

**Documentation:** [Python SDK Guide](https://docs.fish.audio/developer-guide/sdk-guide/python/) | [API Reference](https://docs.fish.audio/api-reference/sdk/python/)

> \[!IMPORTANT]
>
> ## Changes to PyPI Versioning
>
> For existing users on Fish Audio Python SDK, please note that the starting version is now `1.0.0`. The last version before this was `2025.6.3`. You may need to adjust your version constraints accordingly.
>
> The original API in the `fish_audio_sdk` package has NOT been removed, but you will not receive any updates if you continue using the old versioning scheme.
>
> The simplest fix is to update your dependency to `fish-audio-sdk>=1.0.0` to continue receiving updates, or by pinning to a specific version like `fish-audio-sdk==1.0.0` when installing via your package manager. There are no changes to the API itself in this transition.
>
> If you're using the legacy `fish_audio_sdk` and would like to switch to the newer, more robust `fishaudio` package, see the [migration guide](https://docs.fish.audio/archive/python-sdk-legacy/migration-guide) to upgrade.

## Installation

```bash theme={null}
pip install fish-audio-sdk

# With audio playback utilities
pip install fish-audio-sdk[utils]
```

## Authentication

Get your API key from [fish.audio/app/api-keys](https://fish.audio/app/api-keys):

```bash theme={null}
export FISH_API_KEY=your_api_key_here
```

Or provide directly:

```python theme={null}
from fishaudio import FishAudio

client = FishAudio(api_key="your_api_key")
```

## Quick Start

**Synchronous:**

```python theme={null}
from fishaudio import FishAudio
from fishaudio.utils import play, save

client = FishAudio()

# Generate audio
audio = client.tts.convert(text="Hello, world!")

# Play or save
play(audio)
save(audio, "output.mp3")
```

**Asynchronous:**

```python theme={null}
import asyncio
from fishaudio import AsyncFishAudio
from fishaudio.utils import play, save

async def main():
    client = AsyncFishAudio()
    audio = await client.tts.convert(text="Hello, world!")
    play(audio)
    save(audio, "output.mp3")

asyncio.run(main())
```

## Core Features

### Text-to-Speech

**With custom voice:**

```python theme={null}
# Use a specific voice by ID
audio = client.tts.convert(
    text="Custom voice",
    reference_id="9a9cf47702da476aa4629e2506d4a857"
)
```

**With speed control:**

```python theme={null}
audio = client.tts.convert(
    text="Speaking faster!",
    speed=1.5  # 1.5x speed
)
```

**Reusable configuration:**

```python theme={null}
from fishaudio.types import TTSConfig, Prosody

config = TTSConfig(
    prosody=Prosody(speed=1.2, volume=-5),
    reference_id="933563129e564b19a115bedd57b7406a",
    format="wav",
    latency="balanced"
)

# Reuse across generations
audio1 = client.tts.convert(text="First message", config=config)
audio2 = client.tts.convert(text="Second message", config=config)
```

**Chunk-by-chunk processing:**

```python theme={null}
# Stream and process chunks as they arrive
for chunk in client.tts.stream(text="Long content..."):
    send_to_websocket(chunk)

# Or collect all chunks
audio = client.tts.stream(text="Hello!").collect()
```

[Learn more](https://docs.fish.audio/features/text-to-speech)

### Speech-to-Text

```python theme={null}
from fishaudio.core import RequestOptions

# Transcribe audio with transcribe-1-pro (recommended).
# Without the model header, the request runs on transcribe-1.
with open("audio.wav", "rb") as f:
    result = client.asr.transcribe(
        audio=f.read(),
        language="en",
        request_options=RequestOptions(
            additional_headers={"model": "transcribe-1-pro"}
        ),
    )

print(result.text)

# Access timestamped segments
for segment in result.segments:
    print(f"[{segment.start:.2f}s - {segment.end:.2f}s] {segment.text}")
```

[Learn more](https://docs.fish.audio/features/speech-to-text)

### Real-time Streaming

Stream dynamically generated text for conversational AI and live applications:

**Synchronous:**

```python theme={null}
def text_chunks():
    yield "Hello, "
    yield "this is "
    yield "streaming!"

audio_stream = client.tts.stream_websocket(text_chunks(), latency="balanced")
play(audio_stream)
```

**Asynchronous:**

```python theme={null}
import asyncio
from fishaudio import AsyncFishAudio

async def text_chunks():
    yield "Hello, "
    yield "this is "
    yield "streaming!"

async def main():
    async with AsyncFishAudio() as client:
        # stream_websocket is an async generator: iterate it, don't await the call
        audio_stream = client.tts.stream_websocket(text_chunks(), latency="balanced")
        with open("out.mp3", "wb") as f:
            async for chunk in audio_stream:
                f.write(chunk)

asyncio.run(main())
```

[Learn more](https://docs.fish.audio/features/realtime-streaming)

### Voice Cloning

**Instant cloning:**

```python theme={null}
from fishaudio.types import ReferenceAudio

# Clone voice on-the-fly
with open("reference.wav", "rb") as f:
    audio = client.tts.convert(
        text="Cloned voice speaking",
        references=[ReferenceAudio(
            audio=f.read(),
            text="Text spoken in reference"
        )]
    )
```

**Persistent voice models:**

```python theme={null}
# Create voice model for reuse
with open("voice_sample.wav", "rb") as f:
    voice = client.voices.create(
        title="My Voice",
        voices=[f.read()],
        description="Custom voice clone"
    )

# Use the created model
audio = client.tts.convert(
    text="Using my saved voice",
    reference_id=voice.id
)
```

[Learn more](https://docs.fish.audio/features/voice-cloning)

## Resource Clients

| Resource | Description | Key Methods |
| - | - | - |
| `client.tts` | Text-to-speech | `convert()`, `stream()`, `stream_websocket()` |
| `client.asr` | Speech recognition | `transcribe()` |
| `client.voices` | Voice management | `list()`, `get()`, `create()`, `update()`, `delete()` |
| `client.account` | Account info | `get_credits()`, `get_package()` |

## Error Handling

```python theme={null}
from fishaudio.exceptions import (
    AuthenticationError,
    RateLimitError,
    NotFoundError,
    APIError,
    FishAudioError,
)

try:
    audio = client.tts.convert(text="Hello!")
except AuthenticationError:
    print("Invalid API key")
except RateLimitError:
    print("Rate limit exceeded")
except NotFoundError:
    print("Voice model not found")
except APIError as e:
    print(f"API error {e.status}: {e.message}")  # any other HTTP error, including 422 validation
except FishAudioError as e:
    print(f"SDK error: {e}")
```

## Resources

* **Documentation:** [SDK Guide](https://docs.fish.audio/developer-guide/sdk-guide/python/) | [API Reference](https://docs.fish.audio/api-reference/sdk/python/)
* **Package:** [PyPI](https://pypi.org/project/fish-audio-sdk/) | [GitHub](https://github.com/fishaudio/fish-audio-python)
* **Legacy SDK:** [Documentation](https://docs.fish.audio/archive/python-sdk-legacy) | [Migration Guide](https://docs.fish.audio/archive/python-sdk-legacy/migration-guide)

## License

This project is licensed under the Apache-2.0 License - see the [LICENSE](LICENSE) file for details.


# Resources
Source: https://docs.fish.audio/api-reference/sdk/python/resources



# fishaudio.resources.voices

Voice management namespace client.

## VoicesClient Objects

```python theme={null}
class VoicesClient()
```

Synchronous voice management operations.

#### list

```python theme={null}
def list(
    *,
    page_size: int = 10,
    page_number: int = 1,
    title: Optional[str] = OMIT,
    tags: Optional[Union[list[str], str]] = OMIT,
    self_only: bool = False,
    author_id: Optional[str] = OMIT,
    language: Optional[Union[list[str], str]] = OMIT,
    title_language: Optional[Union[list[str], str]] = OMIT,
    sort_by: str = "task_count",
    request_options: Optional[RequestOptions] = None
) -> PaginatedResponse[Voice]
```

List available voices/models.

**Arguments**:

* `page_size` - Number of results per page
* `page_number` - Page number (1-indexed)
* `title` - Filter by title
* `tags` - Filter by tags (single tag or list)
* `self_only` - Only return user's own voices
* `author_id` - Filter by author ID
* `language` - Filter by language(s)
* `title_language` - Filter by title language(s)
* `sort_by` - Sort field ("task\_count" or "created\_at")
* `request_options` - Request-level overrides

**Returns**:

Paginated response with total count and voice items

**Example**:

```python theme={null}
client = FishAudio(api_key="...")

# List all voices
voices = client.voices.list(page_size=20)
print(f"Total: {voices.total}")
for voice in voices.items:
    print(f"{voice.title}: {voice.id}")

# Filter by tags
tagged = client.voices.list(tags=["male", "english"])
```

#### get

```python theme={null}
def get(voice_id: str,
        *,
        request_options: Optional[RequestOptions] = None) -> Voice
```

Get voice by ID.

**Arguments**:

* `voice_id` - Voice model ID
* `request_options` - Request-level overrides

**Returns**:

Voice model details

**Example**:

```python theme={null}
client = FishAudio(api_key="...")
voice = client.voices.get("voice_id_here")
print(voice.title, voice.description)
```

#### create

```python theme={null}
def create(*,
           title: str,
           voices: builtins.list[bytes],
           description: Optional[str] = OMIT,
           texts: Optional[builtins.list[str]] = OMIT,
           tags: Optional[builtins.list[str]] = OMIT,
           cover_image: Optional[bytes] = OMIT,
           visibility: Visibility = "private",
           train_mode: str = "fast",
           enhance_audio_quality: bool = True,
           request_options: Optional[RequestOptions] = None) -> Voice
```

Create/clone a new voice.

**Arguments**:

* `title` - Voice model name
* `voices` - List of audio file bytes for training
* `description` - Voice description
* `texts` - Transcripts for voice samples
* `tags` - Tags for categorization
* `cover_image` - Cover image bytes
* `visibility` - Visibility setting (public, unlist, private)
* `train_mode` - Training mode (currently only "fast" supported)
* `enhance_audio_quality` - Whether to enhance audio quality
* `request_options` - Request-level overrides

**Returns**:

Created voice model

**Example**:

```python theme={null}
client = FishAudio(api_key="...")

with open("voice1.wav", "rb") as f1, open("voice2.wav", "rb") as f2:
    voice = client.voices.create(
        title="My Voice",
        voices=[f1.read(), f2.read()],
        description="Custom voice clone",
        tags=["custom", "english"]
    )
print(f"Created: {voice.id}")
```

#### update

```python theme={null}
def update(voice_id: str,
           *,
           title: Optional[str] = OMIT,
           description: Optional[str] = OMIT,
           cover_image: Optional[bytes] = OMIT,
           visibility: Optional[Visibility] = OMIT,
           tags: Optional[builtins.list[str]] = OMIT,
           request_options: Optional[RequestOptions] = None) -> None
```

Update voice metadata.

**Arguments**:

* `voice_id` - Voice model ID
* `title` - New title
* `description` - New description
* `cover_image` - New cover image bytes
* `visibility` - New visibility setting
* `tags` - New tags
* `request_options` - Request-level overrides

**Example**:

```python theme={null}
client = FishAudio(api_key="...")
client.voices.update(
    "voice_id_here",
    title="Updated Title",
    visibility="public"
)
```

#### delete

```python theme={null}
def delete(voice_id: str,
           *,
           request_options: Optional[RequestOptions] = None) -> None
```

Delete a voice.

**Arguments**:

* `voice_id` - Voice model ID
* `request_options` - Request-level overrides

**Example**:

```python theme={null}
client = FishAudio(api_key="...")
client.voices.delete("voice_id_here")
```

## AsyncVoicesClient Objects

```python theme={null}
class AsyncVoicesClient()
```

Asynchronous voice management operations.

#### list

```python theme={null}
async def list(
    *,
    page_size: int = 10,
    page_number: int = 1,
    title: Optional[str] = OMIT,
    tags: Optional[Union[list[str], str]] = OMIT,
    self_only: bool = False,
    author_id: Optional[str] = OMIT,
    language: Optional[Union[list[str], str]] = OMIT,
    title_language: Optional[Union[list[str], str]] = OMIT,
    sort_by: str = "task_count",
    request_options: Optional[RequestOptions] = None
) -> PaginatedResponse[Voice]
```

List available voices/models (async). See sync version for details.

#### get

```python theme={null}
async def get(voice_id: str,
              *,
              request_options: Optional[RequestOptions] = None) -> Voice
```

Get voice by ID (async). See sync version for details.

#### create

```python theme={null}
async def create(*,
                 title: str,
                 voices: builtins.list[bytes],
                 description: Optional[str] = OMIT,
                 texts: Optional[builtins.list[str]] = OMIT,
                 tags: Optional[builtins.list[str]] = OMIT,
                 cover_image: Optional[bytes] = OMIT,
                 visibility: Visibility = "private",
                 train_mode: str = "fast",
                 enhance_audio_quality: bool = True,
                 request_options: Optional[RequestOptions] = None) -> Voice
```

Create/clone a new voice (async). See sync version for details.

#### update

```python theme={null}
async def update(voice_id: str,
                 *,
                 title: Optional[str] = OMIT,
                 description: Optional[str] = OMIT,
                 cover_image: Optional[bytes] = OMIT,
                 visibility: Optional[Visibility] = OMIT,
                 tags: Optional[builtins.list[str]] = OMIT,
                 request_options: Optional[RequestOptions] = None) -> None
```

Update voice metadata (async). See sync version for details.

#### delete

```python theme={null}
async def delete(voice_id: str,
                 *,
                 request_options: Optional[RequestOptions] = None) -> None
```

Delete a voice (async). See sync version for details.

# fishaudio.resources.account

Account namespace client for billing and credits.

## AccountClient Objects

```python theme={null}
class AccountClient()
```

Synchronous account operations.

#### get\_credits

```python theme={null}
def get_credits(*,
                check_free_credit: Optional[bool] = OMIT,
                request_options: Optional[RequestOptions] = None) -> Credits
```

Get API credit balance.

**Arguments**:

* `check_free_credit` - Whether to check free credit availability
* `request_options` - Request-level overrides

**Returns**:

Credits information

**Example**:

```python theme={null}
client = FishAudio(api_key="...")
credits = client.account.get_credits()
print(f"Available credits: {float(credits.credit)}")

# Check free credit availability
credits = client.account.get_credits(check_free_credit=True)
if credits.has_free_credit:
    print("Free credits available!")
```

#### get\_package

```python theme={null}
def get_package(*,
                request_options: Optional[RequestOptions] = None) -> Package
```

Get package information.

**Arguments**:

* `request_options` - Request-level overrides

**Returns**:

Package information

**Example**:

```python theme={null}
client = FishAudio(api_key="...")
package = client.account.get_package()
print(f"Balance: {package.balance}/{package.total}")
```

## AsyncAccountClient Objects

```python theme={null}
class AsyncAccountClient()
```

Asynchronous account operations.

#### get\_credits

```python theme={null}
async def get_credits(
        *,
        check_free_credit: Optional[bool] = OMIT,
        request_options: Optional[RequestOptions] = None) -> Credits
```

Get API credit balance (async).

**Arguments**:

* `check_free_credit` - Whether to check free credit availability
* `request_options` - Request-level overrides

**Returns**:

Credits information

**Example**:

```python theme={null}
client = AsyncFishAudio(api_key="...")
credits = await client.account.get_credits()
print(f"Available credits: {float(credits.credit)}")

# Check free credit availability
credits = await client.account.get_credits(check_free_credit=True)
if credits.has_free_credit:
    print("Free credits available!")
```

#### get\_package

```python theme={null}
async def get_package(*,
                      request_options: Optional[RequestOptions] = None
                      ) -> Package
```

Get package information (async).

**Arguments**:

* `request_options` - Request-level overrides

**Returns**:

Package information

**Example**:

```python theme={null}
client = AsyncFishAudio(api_key="...")
package = await client.account.get_package()
print(f"Balance: {package.balance}/{package.total}")
```

# fishaudio.resources.tts

TTS (Text-to-Speech) namespace client.

## TTSClient Objects

```python theme={null}
class TTSClient()
```

Synchronous TTS operations.

#### stream

```python theme={null}
def stream(*,
           text: str,
           reference_id: Optional[str] = None,
           references: Optional[list[ReferenceAudio]] = None,
           format: Optional[AudioFormat] = None,
           latency: Optional[LatencyMode] = None,
           speed: Optional[float] = None,
           config: TTSConfig = TTSConfig(),
           model: Model = "s2-pro",
           request_options: Optional[RequestOptions] = None) -> AudioStream
```

Stream text-to-speech audio chunks.

**Arguments**:

* `text` - Text to synthesize
* `reference_id` - Voice reference ID (overrides config.reference\_id if provided)
* `references` - Reference audio samples (overrides config.references if provided)
* `format` - Audio format - "mp3", "wav", "pcm", or "opus" (overrides config.format if provided)
* `latency` - Latency mode - "normal" or "balanced" (overrides config.latency if provided)
* `speed` - Speech speed multiplier, e.g. 1.5 for 1.5x speed (overrides config.prosody.speed if provided)
* `config` - TTS configuration (audio settings, voice, model parameters)
* `model` - TTS model to use
* `request_options` - Request-level overrides

**Returns**:

AudioStream object that can be iterated for audio chunks

**Example**:

```python theme={null}
from fishaudio import FishAudio

client = FishAudio(api_key="...")

# Stream and process chunks
for chunk in client.tts.stream(text="Hello world"):
    process_audio_chunk(chunk)

# Or collect all at once
audio = client.tts.stream(text="Hello world").collect()
```

#### convert

```python theme={null}
def convert(*,
            text: str,
            reference_id: Optional[str] = None,
            references: Optional[list[ReferenceAudio]] = None,
            format: Optional[AudioFormat] = None,
            latency: Optional[LatencyMode] = None,
            speed: Optional[float] = None,
            config: TTSConfig = TTSConfig(),
            model: Model = "s2-pro",
            request_options: Optional[RequestOptions] = None) -> bytes
```

Convert text to speech and return complete audio as bytes.

This is a convenience method that streams all audio chunks and combines them.
For chunk-by-chunk processing, use stream() instead.

**Arguments**:

* `text` - Text to synthesize
* `reference_id` - Voice reference ID (overrides config.reference\_id if provided)
* `references` - Reference audio samples (overrides config.references if provided)
* `format` - Audio format - "mp3", "wav", "pcm", or "opus" (overrides config.format if provided)
* `latency` - Latency mode - "normal" or "balanced" (overrides config.latency if provided)
* `speed` - Speech speed multiplier, e.g. 1.5 for 1.5x speed (overrides config.prosody.speed if provided)
* `config` - TTS configuration (audio settings, voice, model parameters)
* `model` - TTS model to use
* `request_options` - Request-level overrides

**Returns**:

Complete audio as bytes

**Example**:

```python theme={null}
from fishaudio import FishAudio
from fishaudio.utils import play, save

client = FishAudio(api_key="...")

# Get complete audio
audio = client.tts.convert(text="Hello world")

# Play it
play(audio)

# Or save it
save(audio, "output.mp3")
```

#### stream\_websocket

```python theme={null}
def stream_websocket(
        text_stream: Iterable[Union[str, TextEvent, FlushEvent]],
        *,
        reference_id: Optional[str] = None,
        references: Optional[list[ReferenceAudio]] = None,
        format: Optional[AudioFormat] = None,
        latency: Optional[LatencyMode] = None,
        speed: Optional[float] = None,
        config: TTSConfig = TTSConfig(),
        model: Model = "s2-pro",
        max_workers: int = 10,
        ws_options: Optional[WebSocketOptions] = None) -> Iterator[bytes]
```

Stream text and receive audio in real-time via WebSocket.

Perfect for conversational AI, live captioning, and streaming applications.

**Arguments**:

* `text_stream` - Iterator of text chunks to stream
* `reference_id` - Voice reference ID (overrides config.reference\_id if provided)
* `references` - Reference audio samples (overrides config.references if provided)
* `format` - Audio format - "mp3", "wav", "pcm", or "opus" (overrides config.format if provided)
* `latency` - Latency mode - "normal" or "balanced" (overrides config.latency if provided)
* `speed` - Speech speed multiplier, e.g. 1.5 for 1.5x speed (overrides config.prosody.speed if provided)
* `config` - TTS configuration (audio settings, voice, model parameters)
* `model` - TTS model to use
* `max_workers` - ThreadPoolExecutor workers for concurrent sender
* `ws_options` - WebSocket connection options for configuring timeouts, message size limits, etc.
  Useful for long-running generations that may exceed default timeout values.
  See WebSocketOptions class for available parameters.

**Returns**:

Iterator of audio bytes

**Example**:

```python theme={null}
from fishaudio import FishAudio, TTSConfig, ReferenceAudio, WebSocketOptions

client = FishAudio(api_key="...")

def text_generator():
    yield "Hello, "
    yield "this is "
    yield "streaming text!"

# Simple usage with defaults
with open("output.mp3", "wb") as f:
    for audio_chunk in client.tts.stream_websocket(text_generator()):
        f.write(audio_chunk)

# With format and speed parameters
with open("output.wav", "wb") as f:
    for audio_chunk in client.tts.stream_websocket(
        text_generator(),
        format="wav",
        speed=1.3
    ):
        f.write(audio_chunk)

# With reference_id parameter
with open("output.mp3", "wb") as f:
    for audio_chunk in client.tts.stream_websocket(text_generator(), reference_id="your_model_id"):
        f.write(audio_chunk)

# With references parameter
with open("output.mp3", "wb") as f:
    for audio_chunk in client.tts.stream_websocket(
        text_generator(),
        references=[ReferenceAudio(audio=audio_bytes, text="sample")]
    ):
        f.write(audio_chunk)

# With WebSocket options for long-running generations
# Useful if you're generating very long responses that may take >20 seconds
ws_options = WebSocketOptions(keepalive_ping_timeout_seconds=60.0)
with open("output.mp3", "wb") as f:
    for audio_chunk in client.tts.stream_websocket(
        text_generator(),
        ws_options=ws_options
    ):
        f.write(audio_chunk)

# Parameters override config values
config = TTSConfig(format="mp3", latency="balanced")
with open("output.wav", "wb") as f:
    for audio_chunk in client.tts.stream_websocket(
        text_generator(),
        format="wav",  # Parameter wins
        config=config
    ):
        f.write(audio_chunk)
```

## AsyncTTSClient Objects

```python theme={null}
class AsyncTTSClient()
```

Asynchronous TTS operations.

#### stream

```python theme={null}
async def stream(
        *,
        text: str,
        reference_id: Optional[str] = None,
        references: Optional[list[ReferenceAudio]] = None,
        format: Optional[AudioFormat] = None,
        latency: Optional[LatencyMode] = None,
        speed: Optional[float] = None,
        config: TTSConfig = TTSConfig(),
        model: Model = "s2-pro",
        request_options: Optional[RequestOptions] = None) -> AsyncAudioStream
```

Stream text-to-speech audio chunks (async).

**Arguments**:

* `text` - Text to synthesize
* `reference_id` - Voice reference ID (overrides config.reference\_id if provided)
* `references` - Reference audio samples (overrides config.references if provided)
* `format` - Audio format - "mp3", "wav", "pcm", or "opus" (overrides config.format if provided)
* `latency` - Latency mode - "normal" or "balanced" (overrides config.latency if provided)
* `speed` - Speech speed multiplier, e.g. 1.5 for 1.5x speed (overrides config.prosody.speed if provided)
* `config` - TTS configuration (audio settings, voice, model parameters)
* `model` - TTS model to use
* `request_options` - Request-level overrides

**Returns**:

AsyncAudioStream object that can be iterated for audio chunks

**Example**:

```python theme={null}
from fishaudio import AsyncFishAudio

client = AsyncFishAudio(api_key="...")

# Stream and process chunks
async for chunk in await client.tts.stream(text="Hello world"):
    await process_audio_chunk(chunk)

# Or collect all at once
stream = await client.tts.stream(text="Hello world")
audio = await stream.collect()
```

#### convert

```python theme={null}
async def convert(*,
                  text: str,
                  reference_id: Optional[str] = None,
                  references: Optional[list[ReferenceAudio]] = None,
                  format: Optional[AudioFormat] = None,
                  latency: Optional[LatencyMode] = None,
                  speed: Optional[float] = None,
                  config: TTSConfig = TTSConfig(),
                  model: Model = "s2-pro",
                  request_options: Optional[RequestOptions] = None) -> bytes
```

Convert text to speech and return complete audio as bytes (async).

This is a convenience method that streams all audio chunks and combines them.
For chunk-by-chunk processing, use stream() instead.

**Arguments**:

* `text` - Text to synthesize
* `reference_id` - Voice reference ID (overrides config.reference\_id if provided)
* `references` - Reference audio samples (overrides config.references if provided)
* `format` - Audio format - "mp3", "wav", "pcm", or "opus" (overrides config.format if provided)
* `latency` - Latency mode - "normal" or "balanced" (overrides config.latency if provided)
* `speed` - Speech speed multiplier, e.g. 1.5 for 1.5x speed (overrides config.prosody.speed if provided)
* `config` - TTS configuration (audio settings, voice, model parameters)
* `model` - TTS model to use
* `request_options` - Request-level overrides

**Returns**:

Complete audio as bytes

**Example**:

```python theme={null}
from fishaudio import AsyncFishAudio
from fishaudio.utils import play, save

client = AsyncFishAudio(api_key="...")

# Get complete audio
audio = await client.tts.convert(text="Hello world")

# Play it
play(audio)

# Or save it
save(audio, "output.mp3")
```

#### stream\_websocket

```python theme={null}
async def stream_websocket(text_stream: AsyncIterable[Union[str, TextEvent,
                                                            FlushEvent]],
                           *,
                           reference_id: Optional[str] = None,
                           references: Optional[list[ReferenceAudio]] = None,
                           format: Optional[AudioFormat] = None,
                           latency: Optional[LatencyMode] = None,
                           speed: Optional[float] = None,
                           config: TTSConfig = TTSConfig(),
                           model: Model = "s2-pro",
                           ws_options: Optional[WebSocketOptions] = None)
```

Stream text and receive audio in real-time via WebSocket (async).

Perfect for conversational AI, live captioning, and streaming applications.

**Arguments**:

* `text_stream` - Async iterator of text chunks to stream
* `reference_id` - Voice reference ID (overrides config.reference\_id if provided)
* `references` - Reference audio samples (overrides config.references if provided)
* `format` - Audio format - "mp3", "wav", "pcm", or "opus" (overrides config.format if provided)
* `latency` - Latency mode - "normal" or "balanced" (overrides config.latency if provided)
* `speed` - Speech speed multiplier, e.g. 1.5 for 1.5x speed (overrides config.prosody.speed if provided)
* `config` - TTS configuration (audio settings, voice, model parameters)
* `model` - TTS model to use
* `ws_options` - WebSocket connection options for configuring timeouts, message size limits, etc.
  Useful for long-running generations that may exceed default timeout values.
  See WebSocketOptions class for available parameters.

**Returns**:

Async iterator of audio bytes

**Example**:

```python theme={null}
from fishaudio import AsyncFishAudio, TTSConfig, ReferenceAudio, WebSocketOptions

client = AsyncFishAudio(api_key="...")

async def text_generator():
    yield "Hello, "
    yield "this is "
    yield "async streaming!"

# Simple usage with defaults
async with aiofiles.open("output.mp3", "wb") as f:
    async for audio_chunk in client.tts.stream_websocket(text_generator()):
        await f.write(audio_chunk)

# With format and speed parameters
async with aiofiles.open("output.wav", "wb") as f:
    async for audio_chunk in client.tts.stream_websocket(
        text_generator(),
        format="wav",
        speed=1.3
    ):
        await f.write(audio_chunk)

# With reference_id parameter
async with aiofiles.open("output.mp3", "wb") as f:
    async for audio_chunk in client.tts.stream_websocket(text_generator(), reference_id="your_model_id"):
        await f.write(audio_chunk)

# With references parameter
async with aiofiles.open("output.mp3", "wb") as f:
    async for audio_chunk in client.tts.stream_websocket(
        text_generator(),
        references=[ReferenceAudio(audio=audio_bytes, text="sample")]
    ):
        await f.write(audio_chunk)

# With WebSocket options for long-running generations
# Useful if you're generating very long responses that may take >20 seconds
ws_options = WebSocketOptions(keepalive_ping_timeout_seconds=60.0)
async with aiofiles.open("output.mp3", "wb") as f:
    async for audio_chunk in client.tts.stream_websocket(
        text_generator(),
        ws_options=ws_options
    ):
        await f.write(audio_chunk)

# Parameters override config values
config = TTSConfig(format="mp3", latency="balanced")
async with aiofiles.open("output.wav", "wb") as f:
    async for audio_chunk in client.tts.stream_websocket(
        text_generator(),
        format="wav",  # Parameter wins
        config=config
    ):
        await f.write(audio_chunk)
```

# fishaudio.resources.realtime

Real-time WebSocket streaming helpers.

#### iter\_websocket\_audio

```python theme={null}
def iter_websocket_audio(ws) -> Iterator[bytes]
```

Process WebSocket audio messages (sync).

Receives messages from WebSocket, yields audio chunks, handles errors.
Unknown events are ignored and iteration continues.

**Arguments**:

* `ws` - WebSocket connection from httpx\_ws.connect\_ws

**Yields**:

Audio bytes

**Raises**:

* `WebSocketError` - On disconnect or error finish event

#### aiter\_websocket\_audio

```python theme={null}
async def aiter_websocket_audio(ws) -> AsyncIterator[bytes]
```

Process WebSocket audio messages (async).

Receives messages from WebSocket, yields audio chunks, handles errors.
Unknown events are ignored and iteration continues.

**Arguments**:

* `ws` - WebSocket connection from httpx\_ws.aconnect\_ws

**Yields**:

Audio bytes

**Raises**:

* `WebSocketError` - On disconnect or error finish event

# fishaudio.resources.asr

ASR (Automatic Speech Recognition) namespace client.

## ASRClient Objects

```python theme={null}
class ASRClient()
```

Synchronous ASR operations.

#### transcribe

```python theme={null}
def transcribe(
        *,
        audio: bytes,
        language: Optional[str] = OMIT,
        include_timestamps: bool = True,
        request_options: Optional[RequestOptions] = None) -> ASRResponse
```

Transcribe audio to text.

**Arguments**:

* `audio` - Audio file bytes
* `language` - Language code (e.g., "en", "zh"). Auto-detected if not provided.
* `include_timestamps` - Whether to include timestamp information for segments
* `request_options` - Request-level overrides

**Returns**:

ASRResponse with transcription text, duration, and segments

**Example**:

```python theme={null}
client = FishAudio(api_key="...")

with open("audio.mp3", "rb") as f:
    audio_bytes = f.read()

result = client.asr.transcribe(audio=audio_bytes, language="en")
print(result.text)
for segment in result.segments:
    print(f"{segment.start}-{segment.end}: {segment.text}")
```

## AsyncASRClient Objects

```python theme={null}
class AsyncASRClient()
```

Asynchronous ASR operations.

#### transcribe

```python theme={null}
async def transcribe(
        *,
        audio: bytes,
        language: Optional[str] = OMIT,
        include_timestamps: bool = True,
        request_options: Optional[RequestOptions] = None) -> ASRResponse
```

Transcribe audio to text (async).

**Arguments**:

* `audio` - Audio file bytes
* `language` - Language code (e.g., "en", "zh"). Auto-detected if not provided.
* `include_timestamps` - Whether to include timestamp information for segments
* `request_options` - Request-level overrides

**Returns**:

ASRResponse with transcription text, duration, and segments

**Example**:

```python theme={null}
client = AsyncFishAudio(api_key="...")

async with aiofiles.open("audio.mp3", "rb") as f:
    audio_bytes = await f.read()

result = await client.asr.transcribe(audio=audio_bytes, language="en")
print(result.text)
for segment in result.segments:
    print(f"{segment.start}-{segment.end}: {segment.text}")
```


# Types
Source: https://docs.fish.audio/api-reference/sdk/python/types



# fishaudio.types.voices

Voice and model management types.

## Sample Objects

```python theme={null}
class Sample(BaseModel)
```

A sample audio for a voice model.

**Attributes**:

* `title` - Title/name of the audio sample
* `text` - Transcription of the spoken content in the sample
* `task_id` - Unique identifier for the sample task
* `audio` - URL or path to the audio file

## Author Objects

```python theme={null}
class Author(BaseModel)
```

Voice model author information.

**Attributes**:

* `id` - Unique author identifier
* `nickname` - Author's display name
* `avatar` - URL to author's avatar image

## Voice Objects

```python theme={null}
class Voice(BaseModel)
```

A voice model.

Represents a TTS voice that can be used for synthesis.

**Attributes**:

* `id` - Unique voice model identifier (use as reference\_id in TTS)
* `type` - Model type. Options: "svc" (singing voice conversion), "tts" (text-to-speech)
* `title` - Voice model title/name
* `description` - Detailed description of the voice model
* `cover_image` - URL to the voice model's cover image
* `train_mode` - Training mode used. Options: "fast"
* `state` - Current model state: "created", "training", "trained", or "failed"
* `tags` - List of tags for categorization (e.g., \["male", "english", "young"])
* `samples` - List of audio samples demonstrating the voice
* `created_at` - Timestamp when the model was created
* `updated_at` - Timestamp when the model was last updated
* `languages` - List of supported language codes (e.g., \["en", "zh"])
* `visibility` - Model visibility. Options: "public", "private", "unlist"
* `lock_visibility` - Whether visibility setting is locked
* `like_count` - Number of likes the model has received
* `mark_count` - Number of bookmarks/favorites
* `shared_count` - Number of times the model has been shared
* `task_count` - Number of times the model has been used for generation
* `liked` - Whether the current user has liked this model. Default: False
* `marked` - Whether the current user has bookmarked this model. Default: False
* `author` - Information about the voice model's creator

# fishaudio.types.account

Account-related types (credits, packages, etc.).

## Credits Objects

```python theme={null}
class Credits(BaseModel)
```

User's API credit balance.

**Attributes**:

* `id` - Unique credits record identifier
* `user_id` - User identifier
* `credit` - Current credit balance (decimal for precise accounting)
* `created_at` - Timestamp when the credits record was created
* `updated_at` - Timestamp when the credits were last updated
* `has_phone_sha256` - Whether the user has a verified phone number. Optional
* `has_free_credit` - Whether the user has received free credits. Optional

## Package Objects

```python theme={null}
class Package(BaseModel)
```

User's prepaid package information.

**Attributes**:

* `id` - Unique package identifier
* `user_id` - User identifier
* `type` - Package type identifier
* `total` - Total units in the package
* `balance` - Remaining units in the package
* `created_at` - Timestamp when the package was purchased
* `updated_at` - Timestamp when the package was last updated
* `finished_at` - Timestamp when the package was fully consumed. None if still active

# fishaudio.types.tts

TTS-related types.

## ReferenceAudio Objects

```python theme={null}
class ReferenceAudio(BaseModel)
```

Reference audio for voice cloning/style.

**Attributes**:

* `audio` - Audio file bytes for the reference sample
* `text` - Transcription of what is spoken in the reference audio. Should match exactly
  what's spoken and include punctuation for proper prosody.

## Prosody Objects

```python theme={null}
class Prosody(BaseModel)
```

Speech prosody settings (speed and volume).

**Attributes**:

* `speed` - Speech speed multiplier. Range: 0.5-2.0. Default: 1.0.
* `Examples` - 1.5 = 50% faster, 0.8 = 20% slower
* `volume` - Volume adjustment in decibels. Range: -20.0 to 20.0. Default: 0.0 (no change).
  Positive values increase volume, negative values decrease it.

#### from\_speed\_override

```python theme={null}
@classmethod
def from_speed_override(cls,
                        speed: float,
                        base: Optional["Prosody"] = None) -> "Prosody"
```

Create Prosody with speed override, preserving volume from base.

**Arguments**:

* `speed` - Speed value to use
* `base` - Base prosody to preserve volume from (if any)

**Returns**:

New Prosody instance with overridden speed

## TTSConfig Objects

```python theme={null}
class TTSConfig(BaseModel)
```

TTS generation configuration.

Reusable configuration for text-to-speech requests. Create once, use multiple times.
All parameters have sensible defaults.

**Attributes**:

* `format` - Audio output format. Options: "mp3", "wav", "pcm", "opus". Default: "mp3"
* `sample_rate` - Audio sample rate in Hz. If None, uses format-specific default.
* `mp3_bitrate` - MP3 bitrate in kbps. Options: 64, 128, 192. Default: 128
* `opus_bitrate` - Opus bitrate in kbps. Options: -1000, 24, 32, 48, 64. Default: 32
* `normalize` - Whether to normalize/clean the input text. Default: True
* `chunk_length` - Characters per generation chunk. Range: 100-300. Default: 200.
  Lower values = faster initial response, higher values = better quality
* `latency` - Generation mode. Options: "normal" (higher quality), "balanced" (faster). Default: "balanced"
* `reference_id` - Voice model ID from fish.audio (e.g., "9a9cf47702da476aa4629e2506d4a857").
  Find IDs in voice URLs or via voices.list()
* `references` - List of reference audio samples for instant voice cloning. Default: \[]
* `prosody` - Speech speed and volume settings. Default: None (uses natural prosody)
* `top_p` - Nucleus sampling parameter for token selection. Range: 0.0-1.0. Default: 0.7
* `temperature` - Randomness in generation. Range: 0.0-1.0. Default: 0.7.
  Higher = more varied, lower = more consistent
* `max_new_tokens` - Maximum number of tokens to generate. Default: 1024
* `repetition_penalty` - Penalty for repeated tokens. Default: 1.2
* `min_chunk_length` - Minimum chunk length for generation. Default: 50
* `condition_on_previous_chunks` - Whether to condition generation on previous chunks. Default: True
* `early_stop_threshold` - Threshold for early stopping. Default: 1.0

## TTSRequest Objects

```python theme={null}
class TTSRequest(BaseModel)
```

Request parameters for text-to-speech generation.

This model is used internally for WebSocket streaming.
For the HTTP API, parameters are passed directly to methods.

**Attributes**:

* `text` - Text to synthesize into speech
* `chunk_length` - Characters per generation chunk. Range: 100-300. Default: 200
* `format` - Audio output format. Options: "mp3", "wav", "pcm", "opus". Default: "mp3"
* `sample_rate` - Audio sample rate in Hz. If None, uses format-specific default
* `mp3_bitrate` - MP3 bitrate in kbps. Options: 64, 128, 192. Default: 128
* `opus_bitrate` - Opus bitrate in kbps. Options: -1000, 24, 32, 48, 64. Default: 32
* `references` - List of reference audio samples for voice cloning. Default: \[]
* `reference_id` - Voice model ID for using a specific voice. Default: None
* `normalize` - Whether to normalize/clean the input text. Default: True
* `latency` - Generation mode. Options: "normal", "balanced". Default: "balanced"
* `prosody` - Speech speed and volume settings. Default: None
* `top_p` - Nucleus sampling for token selection. Range: 0.0-1.0. Default: 0.7
* `temperature` - Randomness in generation. Range: 0.0-1.0. Default: 0.7
* `max_new_tokens` - Maximum number of tokens to generate. Default: 1024
* `repetition_penalty` - Penalty for repeated tokens. Default: 1.2
* `min_chunk_length` - Minimum chunk length for generation. Default: 50
* `condition_on_previous_chunks` - Whether to condition generation on previous chunks. Default: True
* `early_stop_threshold` - Threshold for early stopping. Default: 1.0

## StartEvent Objects

```python theme={null}
class StartEvent(BaseModel)
```

WebSocket start event to initiate TTS streaming.

**Attributes**:

* `event` - Event type identifier, always "start"
* `request` - TTS configuration for the streaming session

## TextEvent Objects

```python theme={null}
class TextEvent(BaseModel)
```

WebSocket event to send a text chunk for synthesis.

**Attributes**:

* `event` - Event type identifier, always "text"
* `text` - Text chunk to synthesize

## FlushEvent Objects

```python theme={null}
class FlushEvent(BaseModel)
```

WebSocket event to force immediate audio generation from buffered text.

Use this to ensure all buffered text is synthesized without waiting for more input.

**Attributes**:

* `event` - Event type identifier, always "flush"

## CloseEvent Objects

```python theme={null}
class CloseEvent(BaseModel)
```

WebSocket event to end the streaming session.

**Attributes**:

* `event` - Event type identifier, always "stop"

# fishaudio.types.shared

Shared types used across the SDK.

## PaginatedResponse Objects

```python theme={null}
class PaginatedResponse(BaseModel, Generic[T])
```

Generic paginated response.

**Attributes**:

* `total` - Total number of items across all pages
* `items` - List of items on the current page

#### warn\_if\_deprecated\_model

```python theme={null}
def warn_if_deprecated_model(model: str) -> None
```

Emit a deprecation warning if a legacy model is used.

# fishaudio.types.asr

ASR (Automatic Speech Recognition) related types.

## ASRSegment Objects

```python theme={null}
class ASRSegment(BaseModel)
```

A timestamped segment of transcribed text.

**Attributes**:

* `text` - The transcribed text for this segment
* `start` - Segment start time in seconds
* `end` - Segment end time in seconds

## ASRResponse Objects

```python theme={null}
class ASRResponse(BaseModel)
```

Response from speech-to-text transcription.

**Attributes**:

* `text` - Complete transcription of the entire audio
* `duration` - Total audio duration in seconds
* `segments` - List of timestamped text segments. Empty if include\_timestamps=False

#### duration

Duration in seconds


# Utils
Source: https://docs.fish.audio/api-reference/sdk/python/utils



# fishaudio.utils.play

Audio playback utility.

#### play

```python theme={null}
def play(audio: Union[bytes, Iterable[bytes]],
         *,
         notebook: bool = False,
         use_ffmpeg: bool = True) -> None
```

Play audio using various playback methods.

**Arguments**:

* `audio` - Audio bytes or iterable of bytes
* `notebook` - Use Jupyter notebook playback (IPython.display.Audio)
* `use_ffmpeg` - Use ffplay for playback (default, falls back to sounddevice)

**Raises**:

* `DependencyError` - If required playback tool is not installed

**Examples**:

```python theme={null}
from fishaudio import FishAudio, play

client = FishAudio(api_key="...")
audio = client.tts.convert(text="Hello world")

# Play directly
play(audio)

# In Jupyter notebook
play(audio, notebook=True)

# Force sounddevice fallback
play(audio, use_ffmpeg=False)
```

# fishaudio.utils.save

Audio saving utility.

#### save

```python theme={null}
def save(audio: Union[bytes, Iterable[bytes]], filename: str) -> None
```

Save audio to a file.

**Arguments**:

* `audio` - Audio bytes or iterable of bytes
* `filename` - Path to save the audio file

**Examples**:

```python theme={null}
from fishaudio import FishAudio, save

client = FishAudio(api_key="...")
audio = client.tts.convert(text="Hello world")

# Save to file
save(audio, "output.mp3")

# Works with iterators too
audio_stream = client.tts.convert(text="Another example")
save(audio_stream, "another.mp3")
```

# fishaudio.utils.stream

Audio streaming utility.

#### stream

```python theme={null}
def stream(audio_stream: Iterator[bytes]) -> bytes
```

Stream audio in real-time while playing it with mpv.

This function plays the audio as it's being generated and
simultaneously captures it to return the complete audio buffer.

**Arguments**:

* `audio_stream` - Iterator of audio byte chunks

**Returns**:

Complete audio bytes after streaming finishes

**Raises**:

* `DependencyError` - If mpv is not installed

**Examples**:

```python theme={null}
from fishaudio import FishAudio, stream

client = FishAudio(api_key="...")
audio_stream = client.tts.convert(text="Hello world")

# Stream and play in real-time, get complete audio
complete_audio = stream(audio_stream)

# Save the captured audio
with open("output.mp3", "wb") as f:
    f.write(complete_audio)
```


# Legacy
Source: https://docs.fish.audio/archive/python-sdk-legacy/index

Archived documentation for the legacy Session-based Python SDK

<Warning>
  This documentation is for the legacy Python SDK using the Session-based API. This API is deprecated.

  **Please migrate to the [new Python SDK](/developer-guide/sdk-guide/python)** which uses a modern client-based architecture.

  See the [migration guide](/archive/python-sdk-legacy/migration-guide) for help upgrading.
</Warning>

## About the Legacy SDK

This archive contains documentation for the `fish_audio_sdk` module using the Session-based API. While this API still functions, it is no longer actively maintained and lacks the modern features available in the new SDK.

### What's Different in the New SDK

The new Python SDK (`fishaudio` module) offers:

* **Modern client-based architecture** - More intuitive and consistent with modern Python libraries
* **Full async support** - Native asyncio integration for better performance
* **Better type safety** - Comprehensive type hints and better IDE support
* **Improved error handling** - More detailed error messages and exception hierarchy
* **Enhanced utilities** - Built-in audio playback, streaming, and file management
* **Active maintenance** - Regular updates and new features

### Migration Path

We strongly recommend migrating to the new SDK. The [migration guide](/archive/python-sdk-legacy/migration-guide) provides:

* Side-by-side code comparisons
* Complete list of breaking changes
* Common migration patterns
* Troubleshooting tips

## Migration

<Card title="Migration Guide" icon="arrow-right" href="/archive/python-sdk-legacy/migration-guide">
  Complete guide to upgrading from the legacy SDK to the new client-based API
</Card>

## Legacy Documentation Pages

<CardGroup>
  <Card title="Installation" icon="download" href="/archive/python-sdk-legacy/installation">
    How to install the legacy SDK
  </Card>

  <Card title="Authentication" icon="key" href="/archive/python-sdk-legacy/authentication">
    Session initialization and API keys
  </Card>

  <Card title="Text-to-Speech" icon="microphone" href="/archive/python-sdk-legacy/text-to-speech">
    TTS with the Session-based API
  </Card>

  <Card title="Voice Cloning" icon="clone" href="/archive/python-sdk-legacy/voice-cloning">
    Reference audio and voice models
  </Card>

  <Card title="Speech-to-Text" icon="waveform" href="/archive/python-sdk-legacy/speech-to-text">
    ASR transcription with legacy API
  </Card>

  <Card title="WebSocket Streaming" icon="bolt" href="/archive/python-sdk-legacy/websocket">
    Real-time streaming with WebSocketSession
  </Card>
</CardGroup>


# Contributing
Source: https://docs.fish.audio/contributing

Help improve Fish Audio and contribute to our open source projects.

# Contributing to Fish Audio

First off, thanks for taking the time to contribute!

All types of contributions are encouraged and valued. See the sections below for different ways to help and details about how this project handles them. Please make sure to read the relevant section before making your contribution. It will make it a lot easier for us maintainers and smooth out the experience for all involved. The community looks forward to your contributions.

<Note>
  If you like the project but don't have time to contribute, there are other easy ways to support Fish Audio:

  * Star our repositories
  * Tweet about it
  * Reference Fish Audio in your project's readme
  * Mention the project at local meetups and tell your friends/colleagues
</Note>

## Code of Conduct

This project and everyone participating in it is governed by the Fish Audio Code of Conduct. By participating, you are expected to uphold this code. Please report unacceptable behavior to our community team.

## I Have a Question

Before you ask a question, please read the available [Documentation](https://docs.fish.audio).

It's best to search for existing [Issues](https://github.com/fishaudio) that might help you. In case you have found a suitable issue and still need clarification, you can write your question in that issue. It is also advisable to search the internet for answers first.

If you still need to ask a question:

1. Open an [Issue](https://github.com/fishaudio) in the relevant repository
2. Provide as much context as you can about what you're running into
3. Provide project and platform versions (Node.js, Python, OS, etc.), depending on what seems relevant

We will take care of the issue as soon as possible.

## I Want To Contribute

<Warning>
  **Legal Notice**

  When contributing to this project, you must agree that you have authored 100% of the content, that you have the necessary rights to the content, and that the content you contribute may be provided under the project license.
</Warning>

### Reporting Bugs

#### Before Submitting a Bug Report

A good bug report shouldn't leave others needing to chase you up for more information. Please investigate carefully, collect information, and describe the issue in detail:

* Make sure you are using the latest version
* Determine if your bug is really a bug and not an error on your side (e.g., incompatible environment components/versions)
* Check if there is already a bug report for your issue in the bug tracker
* Search the internet (including Stack Overflow) to see if others have discussed the issue
* Collect information about the bug:
  * Stack trace (Traceback)
  * OS, Platform and Version (Windows, Linux, macOS, x86, ARM)
  * Version of the interpreter, compiler, SDK, runtime environment, package manager
  * Your input and the output
  * Can you reliably reproduce the issue? Can you reproduce it with older versions?

#### How Do I Submit a Good Bug Report?

<Warning>
  You must never report security-related issues, vulnerabilities, or bugs including sensitive information to the issue tracker. Instead, sensitive bugs must be sent by email to our security team.
</Warning>

We use GitHub issues to track bugs and errors. If you run into an issue:

1. Open an [Issue](https://github.com/fishaudio) in the relevant repository
2. Explain the behavior you would expect and the actual behavior
3. Provide as much context as possible and describe the **reproduction steps** that someone else can follow to recreate the issue on their own
4. Provide the information you collected in the previous section

Once filed:

* The project team will label the issue accordingly
* A team member will try to reproduce the issue with your provided steps
* If there are no reproduction steps, the team will ask for them and mark the issue as `needs-repro`
* If the team reproduces the issue, it will be marked `needs-fix` and left to be implemented

### Suggesting Enhancements

This section guides you through submitting an enhancement suggestion for Fish Audio, including completely new features and minor improvements to existing functionality.

#### Before Submitting an Enhancement

* Make sure you are using the latest version
* Read the [documentation](https://docs.fish.audio) carefully to see if the functionality already exists
* Perform a [search](https://github.com/fishaudio) to see if the enhancement has already been suggested
* Consider whether your idea fits with the scope and aims of the project

#### How Do I Submit a Good Enhancement Suggestion?

Enhancement suggestions are tracked as GitHub issues:

* Use a **clear and descriptive title** for the issue
* Provide a **step-by-step description** of the suggested enhancement in as many details as possible
* **Describe the current behavior** and **explain which behavior you expected to see instead** and why
* Include **screenshots or screen recordings** if applicable
* **Explain why this enhancement would be useful** to most Fish Audio users

### Your First Code Contribution

We welcome first-time contributors! Here's how to get started:

1. **Fork the repository** you want to contribute to
2. **Clone your fork** locally
3. **Create a new branch** for your changes
4. **Make your changes** following our styleguides
5. **Test your changes** thoroughly
6. **Commit your changes** with clear commit messages
7. **Push to your fork** and submit a pull request

<Tip>
  Look for issues labeled `good first issue` or `help wanted` for beginner-friendly tasks.
</Tip>

### Improving The Documentation

Documentation improvements are always welcome! This includes:

* Fixing typos and grammatical errors
* Adding missing information or clarifications
* Improving code examples
* Adding new guides or tutorials
* Translating documentation

See our [documentation repository](https://github.com/fishaudio/docs) to get started.

## Styleguides

### Commit Messages

* Use clear and meaningful commit messages
* Start with a verb in the present tense (e.g., "Add", "Fix", "Update", "Remove")
* Keep the first line under 72 characters
* Reference issues and pull requests when relevant
* Provide additional context in the commit body if needed

Example:

```
Add voice cloning support for Python SDK

- Implement VoiceCloneClient class
- Add comprehensive error handling
- Include usage examples in docstrings

Closes #123
```

### Code Style

* Follow the existing code style in each repository
* Use meaningful variable and function names
* Add comments for complex logic
* Write tests for new features
* Ensure all tests pass before submitting

## Attribution

This contribution guide is based on the **contributing.md** generator. Fish Audio is committed to open source and welcomes contributions from developers worldwide.


# Emotion & Expression Control
Source: https://docs.fish.audio/developer-guide/best-practices/emotion-control

Make your AI voices express emotions naturally

## Overview

Control how your AI voice expresses emotions, from happy and excited to sad and contemplative. Add natural pauses, laughter, and other human-like elements to make speech more engaging.

<Tip>
  The `(parenthesis)` syntax on this page applies to the S1 model. S2 uses `[bracket]` syntax with natural language descriptions and is not limited to a fixed set of tags. See the [Models Overview](/developer-guide/models-pricing/models-overview#s2-natural-language-control) for details.
</Tip>

## How to Use

Simply wrap emotion tags in parentheses before your text:

```
(happy) What a beautiful day!
(sad) I'm sorry to hear that.
(excited) This is amazing news!
```

Include tone markers or audio effects:

```
(whispering) Let me tell you something.
(laughing) Ha ha ha, wow that's so funny!
```

## Important Rules

### Placement Matters

**For all languages:**

* Emotion tags MUST go at the beginning of sentences
* Tone controls can go anywhere in the text
* Sound effects can go anywhere in the text

**Correct:**

```
(happy) What a wonderful day!
```

**Incorrect:**

```
What a (happy) wonderful day!
```

## Best Practices

**Do:**

* Use one emotion per sentence
* Add sounds after relevant words
* Keep tags simple and clear
* Test different combinations

**Don't:**

* Overuse tags in short text
* Mix conflicting emotions
* Create custom tags
* Forget the parentheses

## Available Emotions

See the [Emotion Control guide](/developer-guide/core-features/emotions) for the full list of supported emotions.

## Scene Examples

**Customer Service:**

```
(friendly) Hello! How can I help you today?
(empathetic) I understand your frustration.
(confident) I'll resolve this for you right away.
```

**Storytelling:**

```
(mysterious)(whispering) Once upon a midnight dreary...
(excited) Suddenly, the door burst open!
(scared)(shouting) Run for your lives!
```

**Educational Content:**

```
(enthusiastic) Welcome to today's lesson!
(curious) Have you ever wondered why the sky is blue?
(proud) Great job! You got it right!
```

## Real-World Examples

### Virtual Assistant

```
(friendly) Good morning! 
(helpful) I've prepared your schedule for today.
(concerned) You have three urgent emails.
(encouraging) Let's tackle them together!
```

### Audiobook Narration

```
(narrator) Chapter One: The Beginning
(mysterious) The old house stood silent in the fog.
(scared)(whispering) "Is anyone there?" she asked.
(relieved)(sighing) No one answered. Phew.
```

### Game Character

```
(brave) I'll defeat the dragon!
(struggling)(panting) This is... harder than... I thought!
(triumphant)(shouting) Victory is mine!
(laughing) Ha ha ha!
```

## Advanced Techniques

### Emotion Transitions

Gradually change emotions:

```
(happy) I got the promotion!
(uncertain) But... it means moving away.
(sad) I'll miss everyone here.
```

### Background Effects

Add atmosphere:

```
The comedy show was amazing (audience laughing)
Everyone was having fun (background laughter)
The crowd loved it (crowd laughing)
```

## Troubleshooting

### Emotion Not Working?

1. Check tag placement (beginning of sentence for emotions)
2. Verify spelling exactly matches the list
3. Don't use quotes around tags
4. Include parentheses

### Unnatural Sound?

* Add appropriate text after sound tags
* Don't overuse in short sentences
* Space out emotional changes
* Test with different voices

### Tips for Success

1. **Start simple** - Use basic emotions first
2. **Preview often** - Test how it sounds
3. **Be consistent** - Keep character emotions logical
4. **Less is more** - Don't overuse tags

## Get Creative

Experiment with combinations to create unique character voices and engaging narratives. The key is finding the right balance between emotional expression and natural speech flow.

## Support

Need help with emotions?

* **Try it live:** [fish.audio](https://fish.audio)
* **Community:** [Discord](https://discord.gg/fish-audio)
* **Email:** [support@fish.audio](mailto:support@fish.audio)


# Real-time Voice Streaming
Source: https://docs.fish.audio/developer-guide/best-practices/real-time-streaming

Stream voice generation in real-time for interactive applications

## Overview

Real-time streaming lets you generate speech as you type or speak, perfect for chatbots, virtual assistants, and live applications.

## When to Use Streaming

**Perfect for:**

* Live chat applications
* Virtual assistants
* Interactive storytelling
* Real-time translations
* Gaming dialogue

**Not ideal for:**

* Pre-recorded content
* Batch processing

## Getting Started

### Web Playground

Try real-time streaming instantly:

1. Visit [fish.audio](https://fish.audio)
2. Enable "Streaming Mode"
3. Start typing and hear voice generation in real-time

### Using the SDK

Stream text as it's being written:

<Tabs>
  <Tab title="Python">
    ```python theme={null}
    from fishaudio import FishAudio

    # Initialize client
    client = FishAudio(api_key="your_api_key")

    # Stream text word by word
    def stream_text():
        text = "Hello, this is being generated in real time"
        for word in text.split():
            yield word + " "

    # Generate speech as text streams
    audio_stream = client.tts.stream_websocket(
        stream_text(),
        reference_id="your_voice_model_id",
        temperature=0.7,  # Controls variation
        top_p=0.7,  # Controls diversity
        latency="balanced"
    )

    with open("output.mp3", "wb") as f:
        for audio_chunk in audio_stream:
            f.write(audio_chunk)
    ```
  </Tab>

  <Tab title="JavaScript">
    ```javascript theme={null}
    import { FishAudioClient, RealtimeEvents } from "fish-audio";
    import { writeFile } from "fs/promises";
    import path from "path";

    const apiKey = "your_api_key";
    const referenceId = "your_voice_model_id";

    async function* makeTextStream() {
      const chunks = [
        "Hello from Fish Audio! ",
        "This is a realtime text-to-speech test. ",
        "We are streaming multiple chunks over WebSocket.",
      ];
      for (const chunk of chunks) {
        yield chunk;
        await new Promise((r) => setTimeout(r, 200));
      }
    }

    async function main() {
      const client = new FishAudioClient({ apiKey });

      // For realtime, set text to "" and stream content via makeTextStream
      const request = {
        text: "",
        reference_id: referenceId,
      };

      const connection = await client.textToSpeech.convertRealtime(
        request,
        makeTextStream()
      );

      // Collect audio and write to a file when the stream ends
      const chunks = [];
      connection.on(RealtimeEvents.OPEN, () => console.log("WebSocket opened"));
      connection.on(RealtimeEvents.AUDIO_CHUNK, (audio) => {
        if (audio instanceof Uint8Array || Buffer.isBuffer(audio)) {
          chunks.push(Buffer.from(audio));
        }
      });
      connection.on(RealtimeEvents.ERROR, (err) =>
        console.error("WebSocket error:", err)
      );
      connection.on(RealtimeEvents.CLOSE, async () => {
        const outPath = path.resolve(process.cwd(), "out.mp3");
        await writeFile(outPath, Buffer.concat(chunks));
        console.log("Saved to", outPath);
      });
    }

    main().catch((err) => {
      console.error(err);
      process.exit(1);
    });
    ```
  </Tab>
</Tabs>

## Configuration Options

### Speed vs Quality

**Latency Modes:**

* **Normal:** Best quality, \~500ms latency
* **Balanced:** Good quality, \~300ms latency

<Tabs>
  <Tab title="Python">
    ```python theme={null}
    # Use latency parameter with stream_websocket
    audio_stream = client.tts.stream_websocket(
        text_chunks(),
        reference_id="model_id",
        latency="balanced"  # For faster response
    )
    ```
  </Tab>

  <Tab title="JavaScript">
    ```javascript theme={null}
    const request = {
      text: "",
      reference_id: "model_id",
      latency: "balanced", // For faster response
    };
    ```
  </Tab>
</Tabs>

### Voice Control

**Temperature** (0.1 - 1.0):

* Lower: More consistent, predictable
* Higher: More varied, expressive

**Top-p** (0.1 - 1.0):

* Lower: More focused
* Higher: More diverse

## Real-time Applications

### Chatbot Integration

Stream responses as they're generated:

<Tabs>
  <Tab title="Python">
    ```python theme={null}
    def chatbot_response(user_input):
        # Get AI response (streaming)
        ai_text = get_ai_response(user_input)

        # Convert to speech in real-time
        audio_stream = client.tts.stream_websocket(ai_text)
        for audio_chunk in audio_stream:
            play_audio(audio_chunk)
    ```
  </Tab>

  <Tab title="JavaScript">
    ```javascript theme={null}
    async function chatbotResponse(userInput) {
      // Get AI response (streaming)
      const aiTextStream = getAiResponse(userInput); // async iterable of strings

      // Convert to speech in real-time
      for await (const textChunk of aiTextStream) {
        for await (const audioChunk of ttsStream(textChunk)) {
          playAudio(audioChunk);
        }
      }
    }
    ```
  </Tab>
</Tabs>

### Live Translation

Translate and speak simultaneously:

<Tabs>
  <Tab title="Python">
    ```python theme={null}
    def live_translate(source_audio):
        # Transcribe source audio
        text = transcribe(source_audio)
        
        # Translate text
        translated = translate(text, target_language)
        
        # Stream translated speech
        for chunk in stream_text(translated):
            generate_speech(chunk)
    ```
  </Tab>

  <Tab title="JavaScript">
    ```javascript theme={null}
    async function liveTranslate(sourceAudio) {
      // Transcribe source audio
      const text = await transcribe(sourceAudio);

      // Translate text
      const translated = await translate(text, targetLanguage);

      // Stream translated speech
      for await (const chunk of streamText(translated)) {
        generateSpeech(chunk);
      }
    }
    ```
  </Tab>
</Tabs>

## Best Practices

### Text Buffering

**Do:**

* Send complete words with spaces
* Use punctuation for natural pauses
* Buffer 5-10 words for smoothness

**Don't:**

* Send individual characters
* Forget spaces between words
* Send huge chunks at once

### Connection Management

1. **Keep connections alive** for multiple generations
2. **Handle disconnections** gracefully
3. **Implement retry logic** for reliability

### Audio Playback

For smooth playback:

* Buffer 2-3 audio chunks
* Use cross-fading between chunks
* Handle network delays gracefully

## Common Use Cases

### Interactive Story

<Tabs>
  <Tab title="Python">
    ```python theme={null}
    def interactive_story():
        story_parts = [
            "Once upon a time,",
            "in a land far away,",
            "there lived a brave knight..."
        ]
        
        for part in story_parts:
            # Generate and play each part
            stream_speech(part)
            # Wait for user input
            user_choice = get_user_input()
            # Continue based on choice
    ```
  </Tab>

  <Tab title="JavaScript">
    ```javascript theme={null}
    function interactiveStory() {
      const storyParts = [
        "Once upon a time,",
        "in a land far away,",
        "there lived a brave knight...",
      ];

      for (const part of storyParts) {
        // Generate and play each part
        streamSpeech(part);
        // Wait for user input
        const userChoice = getUserInput();
        // Continue based on choice
      }
    }
    ```
  </Tab>
</Tabs>

### Virtual Assistant

<Tabs>
  <Tab title="Python">
    ```python theme={null}
    def virtual_assistant():
        while True:
            # Listen for wake word
            if detect_wake_word():
                # Start streaming response
                response = process_command()
                stream_speech(response)
    ```
  </Tab>

  <Tab title="JavaScript">
    ```javascript theme={null}
    async function virtualAssistant() {
      while (true) {
        // Listen for wake word
        if (detectWakeWord()) {
          // Start streaming response
          const response = processCommand();
          streamSpeech(response);
        }
      }
    }
    ```
  </Tab>
</Tabs>

### Live Commentary

<Tabs>
  <Tab title="Python">
    ```python theme={null}
    def live_commentary(event_stream):
        for event in event_stream:
            # Generate commentary
            commentary = generate_commentary(event)
            # Stream immediately
            stream_speech(commentary)
    ```
  </Tab>

  <Tab title="JavaScript">
    ```javascript theme={null}
    async function liveCommentary(eventStream) {
      for await (const event of eventStream) {
        // Generate commentary
        const commentary = generateCommentary(event);
        // Stream immediately
        streamSpeech(commentary);
      }
    }
    ```
  </Tab>
</Tabs>

## Troubleshooting

### Audio Gaps

**Problem:** Gaps between audio chunks<br />
**Solution:**

* Increase buffer size
* Use balanced latency mode
* Check network connection

### Delayed Response

**Problem:** Long wait before audio starts<br />
**Solution:**

* Use balanced latency mode
* Send initial text immediately
* Reduce chunk size

### Choppy Playback

**Problem:** Audio cuts in and out<br />
**Solution:**

* Buffer more chunks before playing
* Check network stability
* Use consistent chunk sizes

## Advanced Features

### Dynamic Voice Switching

Change voices mid-stream:

<Tabs>
  <Tab title="Python">
    ```python theme={null}
    # Start with one voice
    def text1():
        yield "Hello from voice one."

    audio1 = client.tts.stream_websocket(text1(), reference_id="voice1")
    for chunk in audio1:
        play_audio(chunk)

    # Switch to another
    def text2():
        yield "And now voice two!"

    audio2 = client.tts.stream_websocket(text2(), reference_id="voice2")
    for chunk in audio2:
        play_audio(chunk)
    ```
  </Tab>

  <Tab title="JavaScript">
    ```javascript theme={null}
    // Start with one voice
    const request1 = { reference_id: "voice1" };
    streamSpeech("Hello from voice one.", request1);

    // Switch to another
    const request2 = { reference_id: "voice2" };
    streamSpeech("And now voice two!", request2);
    ```
  </Tab>
</Tabs>

### Emotion Injection

Add emotions dynamically:

<Tabs>
  <Tab title="Python">
    ```python theme={null}
    def emotional_speech(text, emotion):
        emotional_text = f"({emotion}) {text}"
        stream_speech(emotional_text)
    ```
  </Tab>

  <Tab title="JavaScript">
    ```javascript theme={null}
    function emotionalSpeech(text, emotion) {
      const emotionalText = `(${emotion}) ${text}`;
      streamSpeech(emotionalText);
    }
    ```
  </Tab>
</Tabs>

### Speed Control

Adjust speaking speed:

<Tabs>
  <Tab title="Python">
    ```python theme={null}
    from fishaudio.types import Prosody

    # Use speed and volume with stream_websocket
    audio_stream = client.tts.stream_websocket(
        text_chunks(),
        speed=1.5  # 1.5x speed
    )
    # Note: For full prosody control including volume, use TTSConfig
    ```
  </Tab>

  <Tab title="JavaScript">
    ```javascript theme={null}
    const request = {
      text: "",
      prosody: {
        speed: 1.5, // 1.5x speed
        volume: 0,  // Normal volume
      },
    };
    ```
  </Tab>
</Tabs>

## Performance Tips

1. **Pre-load voices** for instant start
2. **Use connection pooling** for multiple streams
3. **Monitor latency** and adjust settings
4. **Cache common phrases** for instant playback

## Get Support

Need help with streaming?

* **Discord Community:** [Join our Discord](https://discord.gg/fish-audio)
* **Email Support:** [support@fish.audio](mailto:support@fish.audio)
* **Status Page:** [status.fish.audio](https://status.fish.audio)


# Voice Cloning Best Practices
Source: https://docs.fish.audio/developer-guide/best-practices/voice-cloning

Simple tips to get the best voice cloning results with Fish Audio

## Getting Started

Voice cloning lets you create a digital version of any voice. Use at least 10 seconds of audio recording for studio-quality results right in the Playground or via the API.

## Recording Your Voice

### Find a Quiet Space

**Good places to record:**

* A bedroom with curtains and carpet
* Inside a parked car
* A quiet office or study room
* Any room with soft furniture

**Avoid recording near:**

* Open windows with traffic noise
* Running appliances (AC, fans, refrigerators)
* Other people talking
* TVs or music playing

### Use What You Have

**Best options:**

* USB microphone or gaming headset
* Phone voice recorder app (place it on a stable surface)
* Earbuds with microphone (hold them steady)

**Quick tip:** Keep the microphone about a hand's width from your mouth and speak normally.

## What to Say

**Best approach:** Record 2-3 clips of 15-20 seconds each that form a complete paragraph.

Here's a sample script you can read naturally:

```
"Hello, my name is Alex, and I enjoy reading books about technology 
and science. Yesterday, I walked through the park, observing the 
beautiful autumn leaves. The weather was quite pleasant, with a 
gentle breeze and warm sunshine. I often think about how amazing 
our world is, full of interesting discoveries waiting to be made."
```

### Recording Tips

**Must Have:**

* Only one person speaking
* Steady volume throughout
* Consistent tone and emotion
* Small pauses between sentences (about half a second)

**Nice to Have:**

* No background noise
* No room echo
* Professional mic (but phone is fine too!)

**Avoid:**

* Multiple speakers in one recording
* Big changes in volume or emotion
* Background music or TV
* Rushing through without pauses

## Troubleshooting

### Common Problems

**Voice sounds robotic?**

* Try recording for longer, 30-60 seconds
* Speak more naturally and add pauses

**Voice doesn't sound like you?**

* Make sure you're the only person speaking in the recording
* Check that there's no background music or TV

**Poor audio quality?**

* Find a quieter room to record
* Move closer to your microphone
* Try using a different recording device

## Important: Getting Permission

<Warning>
  Only clone voices you have permission to use:

  * Your own voice
  * Someone who gave you written permission
  * Never use voices from the internet without permission
  * Never use celebrity or public figure voices without permission
</Warning>

## How to Upload Your Recording

<Steps>
  <Step title="Go to Fish Audio">
    Visit [fish.audio](https://fish.audio) and log in
  </Step>

  <Step title="Click 'Create Voice'">
    Find the voice creation button in your dashboard
  </Step>

  <Step title="Upload Your Audio">
    Select your recorded file and give your voice a name
  </Step>

  <Step title="Wait for Processing">
    It usually takes just a few seconds
  </Step>

  <Step title="Try It Out">
    Type some text and hear your cloned voice speak!
  </Step>
</Steps>

## Making Different Voices

Want to create character voices or different styles? Try these:

### Different Emotions

Record the same text with different feelings:

* Happy and energetic
* Calm and relaxed
* Serious and professional

### Different Characters

Create unique voices for:

* Storytelling and audiobooks
* Game characters
* Educational content
* Podcast intros

## Get Help

Need assistance? We're here to help:

* **Community Forum**: [Join our Discord](https://discord.gg/fish-audio)
* **Email Support**: [support@fish.audio](mailto:support@fish.audio)
* **Video Tutorials**: Coming soon!


# Compatibility
Source: https://docs.fish.audio/developer-guide/compat/capabilities

The compatibility contract: supported endpoints, how parameters behave, and exactly what is refused.

Everything under `https://api.fish.audio/compat` follows the contract on this
page.

## How parameters are handled

**Unsupported options return an explicit error, never a silent no-op.** Every
parameter is either **mapped** to the native equivalent (`stability` →
temperature, `speed` → speech rate, bitrates snap to a tier), **ignored** when
it doesn't affect synthesis (`similarity_boost`, `style`, `seed`), or
**refused** with a 4xx in the protocol's own error format when honoring it would
change the response content or structure
([full list](#what-is-explicitly-refused)). Unknown JSON fields are ignored,
matching OpenAI behavior.

## What each protocol supports

Two protocols: OpenAI at `/v1` (equivalently `/api/v1` and `/openai/v1`; the
OpenAI, OpenRouter, and Groq SDKs all speak it), and ElevenLabs at
`/elevenlabs`.

| Capability | OpenAI / OpenRouter / Groq | ElevenLabs |
| - | - | - |
| TTS, streamed as synthesized | Yes | Yes |
| TTS with character timestamps | – | Yes (REST and NDJSON stream) |
| Realtime TTS over WebSocket | Yes (Realtime API, GA protocol) | Yes (`stream-input`) |
| STT from file | Yes (multipart and JSON) | Yes |
| STT word timing | Yes (`verbose_json`) | Yes |
| Realtime STT over WebSocket | Yes | Yes (manual commit) |
| Subtitles (`srt` / `vtt`) | Yes | – |
| Chat-modality audio over SSE | Yes | – |
| Voice library listing | – (not part of the protocol) | Yes (`/v1/voices`, `/v2/voices`) |
| Model catalog | Yes (list; no retrieve-by-id) | Yes |

Voices are always Fish voice IDs. Empty or `default` selects the model's
default voice, and anything that isn't a real voice ID, preset names (`nova`,
`echo`, `Rachel`, …) included, is a 400. One exception: lowercase `alloy`
matches a real Fish voice and synthesizes as an unrelated voice, not OpenAI's
Alloy (`Alloy` is refused). Where to find voice IDs:
[Voices](/developer-guide/getting-started/migration#voices).

## Audio formats

| Codec | Sample rates (kHz) |
| - | - |
| MP3 | 32, 44.1 |
| WAV / PCM | 8, 16, 24, 32, 44.1 |
| Opus | 48 |

Requesting a sample rate resamples the audio; a rate the codec doesn't take is
a 400, including all of 22.05 kHz (`mp3_22050_32`, `pcm_22050`, …). On REST,
OpenAI `pcm16` defaults to 24 kHz and `pcm` to 44.1 kHz; an explicit
`sample_rate` overrides that.

μ-law and A-law (`ulaw_*`, `alaw_*`, `audio/pcmu`, `audio/pcma`) are refused
everywhere. How formats are written on the Realtime socket, and when rates
reset, is in the
[Realtime protocol reference](/developer-guide/compat/realtime-protocol#audio-formats).

<Warning>
  **WAV responses carry a placeholder header: trust the byte count, not the
  header.** The RIFF size field reads `4294967076` and the `data` chunk size
  `4294967040` however short the clip is, on buffered responses and streams
  alike. Anything that reads the header gets the duration wrong: Python's
  `wave` module reports a one-second clip as 13.5 hours; players that decode
  to end-of-stream are unaffected. For a correct duration use the response
  length, re-mux (`ffmpeg -i in.wav -c copy out.wav`), or request MP3 or Opus.
</Warning>

### Bitrate snapping

MP3 is encoded at 64, 128, or 192 kbps. An out-of-tier request snaps **up**
to the nearest tier (`mp3_32000_32` → 64 kbps at 32 kHz); anything above 192
kbps is a 400. To pick the tier exactly, set it as a
[Fish-native parameter](#reaching-fish-native-parameters); a non-tier value
there is refused too. Opus bitrate is automatic: requested values are accepted
and ignored.

## What is explicitly refused

Each returns a 4xx in the protocol's own error format (or, on WebSockets, an
in-band error event):

| Endpoint | Refused options |
| - | - |
| ElevenLabs STT | `diarize`, `entity_redaction`, `entity_detection`, `entity_redaction_mode`, `use_multi_channel`, `additional_formats`, `tag_audio_events`, `no_verbatim`, `webhook`, `webhook_id`, `source_url`, `cloud_storage_url`, `enable_logging=false`, `timestamps_granularity=character` |
| ElevenLabs realtime STT | any `commit_strategy` other than `manual` |
| ElevenLabs TTS | `ulaw_*` / `alaw_*` output formats |
| OpenAI TTS | `response_format` of `aac` or `flac`; `instructions`; any `stream_format` other than `audio` |
| OpenAI STT | `include`, `known_speaker_names`, `known_speaker_references` |
| OpenAI chat | `n ≠ 1`; audio input combined with audio output in one call |
| OpenAI Realtime | `audio/pcmu`, `audio/pcma`; any `turn_detection` with a type (`server_vad`, `semantic_vad`, anything else; only `null` and `"none"` pass); `output_modalities` that omits `"audio"`; a second concurrent response (`response_already_active`) |

ElevenLabs STT refuses `diarize` and `tag_audio_events`. For speaker turns and
event tags, use the native
[Speech to Text API](/features/speech-to-text#multi-speaker-conversations) with
`transcribe-1-pro`.

On OpenAI transcription, `languages`, `keywords`, `prompt`, and `temperature`
are in the [ignored tier](#how-parameters-are-handled).

Booleans count as requested only when true: `false`, `"False"`, `0`, `none`,
and `off` are treated as unset, so SDK defaults never trigger a refusal.

## Placeholder values

Protocol fields with no Fish counterpart are filled with neutral constants;
don't build logic on them:

* OpenAI `verbose_json`: the Whisper-engine internals (`avg_logprob`,
  `no_speech_prob`, `compression_ratio`, `temperature`, `seek`, `tokens`) are
  neutral constants.
* ElevenLabs STT `language_probability` is always `1.0`, and `words[].logprob`
  is always `0`. Neither is a confidence score.
* `speaker_id` is always `null`; requesting diarization returns an explicit
  error instead.
* `language` in responses echoes your request, never a detection result.
  Send none and you get `language: ""` alongside `language_probability: 1.0`.

## Reaching Fish-native parameters

The full native TTS parameter set (`temperature`, `top_p`, `chunk_length`,
`latency`, `normalize`, `prosody`, reference audio) is reachable on
**synthesis endpoints**, in two ways. One exception: **reference audio is not
available on the timestamped ElevenLabs endpoints** (`…/with-timestamps`,
`…/stream/with-timestamps`, the `stream-input` WebSocket): a 400 on the REST
variants, an in-band error frame at flush time on the WebSocket.

### `provider.options` in the request body

On OpenAI-protocol speech and chat-synthesis requests, a
`provider.options.fish-audio` block sets the corresponding native TTS fields:

<Accordion title="Code examples (Python · Node · curl)">
  <CodeGroup>
    ```python Python theme={null}
    resp = client.audio.speech.create(
        model="fish-audio/s2.1-pro",
        input="Hello from Fish Audio!",
        voice="",
        response_format="mp3",
        extra_body={"provider": {"options": {"fish-audio": {
            "latency": "normal", "temperature": 0.4,
        }}}},
    )
    ```

    ```javascript Node theme={null}
    const resp = await client.audio.speech.create({
      model: "fish-audio/s2.1-pro",
      input: "Hello from Fish Audio!",
      voice: "",
      response_format: "mp3",
      provider: { options: { "fish-audio": { latency: "normal", temperature: 0.4 } } },
    });
    ```

    ```bash curl theme={null}
    curl https://api.fish.audio/compat/v1/audio/speech \
      -H "Authorization: Bearer $FISH_AUDIO_API_KEY" \
      -H "Content-Type: application/json" \
      -d '{"model": "fish-audio/s2.1-pro",
           "input": "Hello from Fish Audio!",
           "response_format": "mp3",
           "provider": {"options": {"fish-audio": {"latency": "normal", "temperature": 0.4}}}}' \
      --output hello.mp3
    ```
  </CodeGroup>
</Accordion>

Inline `references` (base64 audio plus its transcript) enable zero-shot
voice cloning through any OpenAI SDK.

### `X-Fish-*` request headers

On every synthesis endpoint (OpenAI `POST /v1/audio/speech` and chat
synthesis, the ElevenLabs endpoints, and OpenAI Realtime synthesis sessions,
including their WebSocket handshakes), an `X-Fish-<Param>` header sets the
corresponding native TTS field:

<Accordion title="Code examples (Python · Node · curl)">
  <CodeGroup>
    ```python Python theme={null}
    resp = client.audio.speech.create(
        model="fish-audio/s2.1-pro",
        input="Hello from Fish Audio!",
        voice="",
        response_format="mp3",
        extra_headers={"X-Fish-Latency": "normal", "X-Fish-Mp3-Bitrate": "64"},
    )
    ```

    ```javascript Node theme={null}
    const resp = await client.audio.speech.create(
      {
        model: "fish-audio/s2.1-pro",
        input: "Hello from Fish Audio!",
        voice: "",
        response_format: "mp3",
      },
      { headers: { "X-Fish-Latency": "normal", "X-Fish-Mp3-Bitrate": "64" } },
    );
    ```

    ```bash curl theme={null}
    curl https://api.fish.audio/compat/v1/audio/speech \
      -H "Authorization: Bearer $FISH_AUDIO_API_KEY" \
      -H "Content-Type: application/json" \
      -H "X-Fish-Latency: normal" \
      -H "X-Fish-Mp3-Bitrate: 64" \
      -d '{"model": "fish-audio/s2.1-pro", "input": "Hello!", "response_format": "mp3"}' \
      --output hello.mp3
    ```
  </CodeGroup>
</Accordion>

Values parse as JSON when possible, and headers win over mapped values.

Neither channel applies to transcription: every STT endpoint returns a 400
when sent `provider.options.fish-audio` or `X-Fish-*`. That includes a
Realtime socket whose handshake carried `X-Fish-*` headers: the
`input_audio_buffer.commit` there is refused with
`unsupported_native_options`, so open a second socket without the headers to
transcribe. On the native `/v1/tts/live` WebSocket, put native fields in
each `start.request` payload rather than in handshake headers.

<Note>
  Synthesis on the Compatible APIs defaults to `latency: "low"`
  (streaming-optimized). For maximum quality on non-interactive workloads, set
  `X-Fish-Latency: normal`; your explicit value always wins.
</Note>

## Limits and operational contract

| Limit | Value | On excess |
| - | - | - |
| TTS / chat request body (OpenAI) | 32 MiB | 413 |
| JSON body (ElevenLabs) | 140 MiB | 413 |
| Transcription request body | 140 MiB | 413 |
| Transcription audio (REST) | 100 MiB | 413 |
| WebSocket frame | 32 MiB | connection closed |
| Realtime: text per conversation item | 1 MiB | `conversation_too_large` error event |
| Realtime: pending items per session | 1024 | `conversation_too_large` error event |
| Realtime: buffered audio awaiting commit | 100 MiB | error event, audio not buffered |
| ElevenLabs `stream-input`: buffered text | 1 MiB | error frame `code: 1009`, then close |
| ElevenLabs realtime STT: buffered audio | 100 MiB | `input_error`; append dropped, socket stays open |
| Single connection (SSE / WebSocket) | 3600 s | closed; reconnect before expiry |
| Time to first response byte | 5 min | 502 (streaming bodies are not time-capped) |

The 32 MiB frame limit bounds one WebSocket message, not a session. On a
Realtime socket, item-borne audio and `input_audio_buffer` bytes share the
same 100 MiB budget.

* **Billing**: Like native usage, against your key. Billable requests are
  never retried automatically.
* **Rate limits**: The Compatible APIs add none of their own today; a 429
  means you reached your account's concurrency limit, the same one native
  calls count against. Every 429 carries a `Retry-After` header.
* **Truncation is an error**: A stream that fails mid-response returns an
  error frame or 502, never a truncated 200.
* Every synthesis or transcription response carries an `X-Generation-Id`
  header; include it in support requests.
* During service incidents, requests may return 503; back off exponentially.

## Known client-side limitations

* **OpenAI Realtime**: the endpoint speaks the GA event names
  (`response.output_audio.*`). Current `openai` packages emit these; very old
  beta-era clients expecting `response.audio.*` will not match.
* **ElevenLabs Python `convert_realtime`**: pass `model_id` and
  `voice_settings` explicitly. Left to the SDK's `OMIT` sentinel, `model_id`
  reaches the handshake URL as the literal `model_id=Ellipsis` (a 400) and
  `voice_settings` raises `AttributeError` on the first frame. The SDK also
  force-upgrades `http://` base URLs to `wss://`.


# Migrate from ElevenLabs
Source: https://docs.fish.audio/developer-guide/compat/migrate-from-elevenlabs

Point the ElevenLabs SDK at Fish Audio: TTS, timestamps, realtime streaming input, speech-to-text, and your voice library.

Configure the client once:

<CodeGroup>
  ```python Python theme={null}
  import os

  from elevenlabs import ElevenLabs

  client = ElevenLabs(
      base_url="https://api.fish.audio/compat/elevenlabs",
      # sent as the standard xi-api-key header
      api_key=os.environ["FISH_AUDIO_API_KEY"],
  )
  ```

  ```javascript Node theme={null}
  import { ElevenLabsClient } from "@elevenlabs/elevenlabs-js";

  const client = new ElevenLabsClient({
    baseUrl: "https://api.fish.audio/compat/elevenlabs",
    apiKey: process.env.FISH_AUDIO_API_KEY, // sent as the standard xi-api-key header
  });
  ```
</CodeGroup>

The two changes that matter when migrating:

1. **Voices are your Fish voices.** ElevenLabs preset names (`Rachel`, …)
   don't exist here. Pass the ID of a voice from your
   [Fish voice library](/developer-guide/getting-started/migration#voices), or
   `default` (also `-`) for the model's default voice.
   `client.voices.get_all()` lists your voices ([Voice library](#voice-library)).
2. **Keep your `model_id`, or pick a Fish model.** Existing values like
   `eleven_multilingual_v2` are accepted as-is and served by Fish `s2.1-pro`,
   so your code runs unchanged. To choose a different model, pass its Fish
   name, e.g. `fish-audio/s2-pro`; see
   [Models](/developer-guide/getting-started/migration#models).

| Feature | Supported |
| - | - |
| Text to speech (+ character timestamps) | ✓ |
| Realtime input streaming (WebSocket) | ✓ |
| Speech to text + word timestamps | ✓ |
| Realtime STT (manual commit) | ✓ |
| Voice library & model catalog | ✓ |
| Diarization / redaction / webhooks | – |
| μ-law / A-law output | – |

## Text to speech

<CodeGroup>
  ```python Python theme={null}
  audio = b"".join(client.text_to_speech.convert(
      voice_id="default",
      text="Hello from Fish Audio!",
      model_id="eleven_multilingual_v2",
      output_format="mp3_44100_128",
  ))
  ```

  ```javascript Node theme={null}
  const stream = await client.textToSpeech.convert("default", {
    text: "Hello from Fish Audio!",
    modelId: "eleven_multilingual_v2",
    outputFormat: "mp3_44100_128",
  });
  const chunks = [];
  for await (const chunk of stream) chunks.push(chunk);
  const audio = Buffer.concat(chunks);
  ```

  ```bash curl theme={null}
  curl "https://api.fish.audio/compat/elevenlabs/v1/text-to-speech/default?output_format=mp3_44100_128" \
    -H "xi-api-key: $FISH_AUDIO_API_KEY" \
    -H "Content-Type: application/json" \
    -d '{"text": "Hello from Fish Audio!",
         "model_id": "eleven_multilingual_v2"}' \
    --output hello.mp3
  ```
</CodeGroup>

`convert`, `stream`, `convert_with_timestamps`, and `stream_with_timestamps`
all work.

`voice_settings` maps onto Fish's synthesis controls: `speed` sets the speech
rate, and `stability` becomes Fish `temperature = 1 − stability`. The
vendor-specific tuning knobs (`similarity_boost`, `style`,
`use_speaker_boost`, `language_code`, `seed`) are accepted and ignored, never
an error.

### Output formats

`mp3_*`, `pcm_*`, `opus_*`, and `wav_*` (a Fish Audio extension) work at the
supported sample rates: 32000 or 44100 for `mp3_*`, 48000 for `opus_*`.
The 22.05 kHz family (`mp3_22050_32`, `pcm_22050`, …) is refused. The
sample-rate segment of the format name is honored; the `mp3_*` bitrate segment
snaps **up** to the nearest Fish tier
([details](/developer-guide/compat/capabilities#bitrate-snapping)), while Opus
bitrate is automatic. `ulaw_*` and `alaw_*` return a 400.

### Timestamps

The `with-timestamps` variants return the official shapes: REST uses snake\_case
keys in **seconds**; the WebSocket uses camelCase keys in **milliseconds**
(`charStartTimesMs`). That asymmetry matches ElevenLabs' own API. Character
timings are interpolated from Fish segment timestamps at millisecond precision.

* **`normalizedAlignment` is identical to `alignment`**, field for field. Fish
  synthesizes the text as given, so there is no normalization pass to report.
  Pick either; don't diff them expecting to recover normalization.
* **On the WebSocket, the milliseconds are absolute on the session timeline**
  (alignment frames need `?sync_alignment=true`; see
  [Realtime input streaming](#realtime-input-streaming-websocket)), not
  relative to the current turn. Turn two of a `stream-input` session starts
  where turn one's audio ended (\~1022 ms, say), not back at 0. That is what lets
  a client concatenate every frame's audio and alignment and have the captions
  line up. Don't add your own per-turn offset; you would double-count it.

## Realtime input streaming (WebSocket)

`wss://api.fish.audio/compat/elevenlabs/v1/text-to-speech/{voice_id}/stream-input`

The official protocol, unchanged:

1. Send the init message: a single-space `text`, plus optional
   `voice_settings`.
2. Send text chunks; `try_trigger_generation` or `flush` forces synthesis now.
3. Send an empty-string `text` to finish.

The server streams audio frames, then sends `{"isFinal": true}` and closes the
connection, so SDK iterators that read until close terminate correctly.

Frame shapes, precisely, because they are not all the same:

* A normal frame is `{"audio": "<base64>"}`. Timing data is opt-in: add
  `?sync_alignment=true` to the handshake URL and frames become
  `{"audio": …, "alignment": {…}, "normalizedAlignment": {…}}`. With
  alignment on, a turn's first audio arrives only after its first text chunk
  has fully synthesized, so expect a later first byte than the default mode.
  Both alignment keys are omitted when there is nothing to report for that
  chunk, so treat them as optional rather than indexing into them blindly.
* With `sync_alignment`, a turn can end with an **audio-less frame** carrying
  only `alignment` and `normalizedAlignment`: trailing punctuation and
  whitespace that no audio segment spoke. Code that assumes every frame has
  `audio` will throw on it.
* Errors arrive in band as `{"error": "…", "code": …}` on the open socket.
  The code is an HTTP status, or a WebSocket close code for transport-level
  limits (an over-1 MiB text buffer closes the socket with `1009`).
* The stream ends with `{"isFinal": true}` (a frame with no `audio`), followed
  by a close.

An `output_format` that is invalid on its face (`ulaw_*`, or a name that
can't be parsed) fails with HTTP 400 before the upgrade. The 22.05 kHz family
fails after it instead: the socket opens, then carries a `{"code": 400, …}`
frame and closes with a normal close code.

```python theme={null}
from elevenlabs import VoiceSettings

def sentences():
    yield "Streaming text, "
    yield "sentence by sentence."

audio = b"".join(client.text_to_speech.convert_realtime(
    voice_id="default",
    text=sentences(),
    model_id="eleven_multilingual_v2",
    voice_settings=VoiceSettings(stability=0.5, speed=1.0),
))
```

`text` is typed `Iterator[str]`. A plain string does run (the SDK re-chunks it
character by character), but a generator is what the signature asks for: you
yield text as it becomes available, rather than having it all up front.

<Note>
  Pass both `model_id` and `voice_settings` explicitly with `convert_realtime`.
  Both default to the SDK's `OMIT` sentinel (`Ellipsis`), and each fails its own
  way: `model_id` goes into the handshake URL as the literal string
  `model_id=Ellipsis` and comes back a 400, while `voice_settings` opens the
  socket fine and then raises `AttributeError` as the first frame is serialized.
  Both are quirks of the SDK itself.

  The same SDK force-upgrades `http://` base URLs to `wss://`, so it cannot be
  pointed at a plaintext endpoint, including a local test proxy.
</Note>

## Speech to text

<CodeGroup>
  ```python Python theme={null}
  result = client.speech_to_text.convert(
      file=open("meeting.wav", "rb"),
      model_id="scribe_v1",  # accepted and ignored; Fish's ASR model is used
      language_code="zh",
  )
  ```

  ```javascript Node theme={null}
  import fs from "node:fs";

  const buf = fs.readFileSync("meeting.wav");
  const result = await client.speechToText.convert({
    file: new File([buf], "meeting.wav"),
    modelId: "scribe_v1", // accepted and ignored; Fish's ASR model is used
  });
  console.log(result.text);
  ```

  ```bash curl theme={null}
  curl https://api.fish.audio/compat/elevenlabs/v1/speech-to-text \
    -H "xi-api-key: $FISH_AUDIO_API_KEY" \
    -F model_id=scribe_v1 \
    -F file=@meeting.wav
  ```
</CodeGroup>

The response matches the official shape: `text`, plus `words` entries of
`type: "word"` and `type: "spacing"`. Word granularity is the default, so those
entries come back whether or not you pass `timestamps_granularity="word"`;
`timestamps_granularity="character"` is not supported and returns a 400.

Three fields are constants, not measurements; don't build quality gates or
speaker logic on them:

* `language_probability` is always `1.0`.
* `words[].logprob` is always `0`.
* `words[].speaker_id` is always `null`; requesting diarization returns an
  explicit error instead.

And `language_code` in the response **echoes what you sent**, never a
detection result. The value is still forwarded to Fish's ASR; only the reported
field is an echo rather than a measurement. Fish needs no language hint, so
sending none is normal; you then get `language_code: ""` alongside
`language_probability: 1.0`, which reads as total confidence in an empty answer.
Code branching on `language_probability > 0.9` will act on that empty string.

(On the *synthesis* side `language_code` is a different story: there it is
accepted and dropped, along with the other vendor tuning knobs.)

Options that would change the response contract are **rejected with 400**
rather than silently ignored:

| Option | Why it's refused |
| - | - |
| `diarize` | speaker attribution is not available here |
| `use_multi_channel` | per-channel transcription is not available |
| `entity_detection` | entities would be missing from the response |
| `entity_redaction` | text would come back unredacted |
| `entity_redaction_mode` | same |
| `no_verbatim` | the transcript would still be verbatim |
| `tag_audio_events` | event tags are not available here |
| `additional_formats` | the extra formats would be absent |
| `timestamps_granularity=character` | only word granularity exists |
| `webhook`, `webhook_id` | there is no delivery callback |
| `source_url`, `cloud_storage_url` | remote audio would not be fetched |
| `enable_logging=false` | zero-retention mode cannot be honored |

For speaker turns and event tags, use the native
[Speech to Text API](/features/speech-to-text#multi-speaker-conversations) with
`transcribe-1-pro` instead.

<Warning>
  **`tag_audio_events` is the one that bites on migration.** ElevenLabs defaults
  it to `true` and their examples pass it explicitly, so code copied from an
  ElevenLabs project frequently sends it, and gets a 400 here. Drop the
  parameter; the transcript itself is unaffected.
</Warning>

Booleans count as requested only when true: `False`, `"False"`, `0`, and `off`
are treated as unset, so default SDK serialization never trips any of these.

### Realtime STT

The realtime WebSocket (`/v1/speech-to-text/realtime`) supports
`commit_strategy=manual` only: stream `input_audio_chunk` messages and set
`commit: true` to receive the committed transcript. Your application decides
the segment boundaries, so `commit_strategy=vad` is refused at the handshake
instead of leaving a session waiting.

<Warning>
  **Set `audio_format` to match your audio.** This socket carries bare PCM
  samples with no container, so the `audio_format` query parameter is the only
  thing that says how to interpret them. It defaults to **`pcm_16000`** (the
  ElevenLabs SDK's own default).

  Feed 44.1 kHz samples to a session that says 16000 and you get a **`200` with
  a fluent, confident, completely wrong transcript**, word timestamps and all:
  the samples are read at the wrong rate, so the audio is effectively played at
  the wrong speed before it ever reaches the recognizer. There is no error to
  catch. Pass `?audio_format=pcm_44100` (or resample to 16 kHz before sending).

  Only `pcm_<rate>` values are accepted; `ulaw_8000` and other containerized
  names are refused at the handshake rather than opening a session that would
  fail on every commit.
</Warning>

Audio goes in the `audio_base_64` field of an `input_audio_chunk` message;
other message shapes are ignored by design. Each commit answers with a
`committed_transcript` message followed by
`committed_transcript_with_timestamps`.

## Voice library

Your Fish voice library and the model catalog are available in the ElevenLabs
shapes:

* `GET /v1/voices`, `GET /v2/voices`, `GET /v1/voices/{id}`: your voices in
  the ElevenLabs `Voice` shape (`category: "cloned"`, tags mapped to `labels`).
* `GET /v1/models`: the Fish model catalog in the ElevenLabs shape.
* `GET /v1/user`: a stub for SDK health checks.

<CodeGroup>
  ```python Python theme={null}
  voices = client.voices.get_all()
  for v in voices.voices:
      print(v.voice_id, v.name)
  ```

  ```javascript Node theme={null}
  const voices = await client.voices.search();
  for (const v of voices.voices) console.log(v.voiceId, v.name);
  ```

  ```bash curl theme={null}
  curl https://api.fish.audio/compat/elevenlabs/v1/voices \
    -H "xi-api-key: $FISH_AUDIO_API_KEY"
  ```
</CodeGroup>

Errors on this protocol use the ElevenLabs envelope:
`{"detail": {"status": "...", "message": "..."}}`. The statuses you can see are
`invalid_request`, `unauthorized`, `voice_not_found`, `model_not_found`,
`not_found`, `payload_too_large`, `rate_limit_error`, `upstream_error`, and
`provider_error`.

`provider_error` is the one that tells you where the failure happened: the
request reached the Fish Audio API and its message is passed through verbatim.
An *invalid* key surfaces this way too, as a 401 `provider_error` carrying
"No permission", while `unauthorized` is reserved for requests that sent no key
at all. Every other status is raised by the compatibility layer itself.

<Accordion title="Telling the two failure kinds apart: examples worth logging differently">
  `voice_not_found` and `provider_error` are the easy pair to confuse. Looking
  up a voice that isn't yours (`GET /v1/voices/{id}`) gives you
  `voice_not_found` with a 404. Naming that same voice in a *synthesis* request
  instead fails as a 400 `provider_error` carrying "Reference not found".

  So `{"status": "invalid_request", "message": "unsupported output_format
      \"ulaw_8000\""}` reports a parameter that is never accepted, while
  `{"status": "provider_error", "message": "Invalid sample rate 22050 for
      format audio/mpeg…"}` comes from synthesis itself.
</Accordion>

## Going further

<CardGroup>
  <Card title="Compatibility" icon="list-check" href="/developer-guide/compat/capabilities">
    The full contract: mappings, limits, and explicit refusals.
  </Card>

  <Card title="Fish-native parameters" icon="sliders" href="/developer-guide/compat/capabilities#reaching-fish-native-parameters">
    Reach `latency`, `temperature`, and reference audio via `X-Fish-*` headers.
  </Card>
</CardGroup>


# Migrate from Groq
Source: https://docs.fish.audio/developer-guide/compat/migrate-from-groq

Point the official Groq SDKs at Fish Audio: TTS, transcription, and the model catalog.

Configure the client once. The Groq SDK speaks the OpenAI protocol and appends
`/openai/v1` to its base URL itself, so give it the bare `/compat` base:

<CodeGroup>
  ```python Python theme={null}
  import os

  from groq import Groq

  client = Groq(
      base_url="https://api.fish.audio/compat",
      api_key=os.environ["FISH_AUDIO_API_KEY"],
  )
  ```

  ```javascript Node theme={null}
  import Groq from "groq-sdk";

  const client = new Groq({
    apiKey: process.env.FISH_AUDIO_API_KEY,
    baseURL: "https://api.fish.audio/compat",
  });
  ```
</CodeGroup>

`/openai/v1` speaks the same protocol the OpenAI SDK uses, so everything
in [Migrate from OpenAI](/developer-guide/compat/migrate-from-openai) applies
here too.

Model names your Groq code already sends keep working: `playai-tts` maps to
`fish-audio/s2.1-pro`, and `whisper-large-v3`, `whisper-large-v3-turbo`, and
`distil-whisper-large-v3-en` map to `fish-audio/transcribe-1`. Anything
outside the catalog returns a 400.

| Feature | Supported |
| - | - |
| Text to speech (streamed) | ✓ |
| Speech to text + word timestamps | ✓ |
| Model catalog (list) | ✓ |
| Realtime WebSocket | via the OpenAI SDK |
| TTS character timestamps | – |
| Speech translation | – |
| Voice library listing | – |

## Text to speech

<CodeGroup>
  ```python Python theme={null}
  resp = client.audio.speech.create(
      model="fish-audio/s2.1-pro",
      input="Hello from Fish Audio!",
      voice="",  # a voice ID from your Fish voice library; "" = model default
      response_format="mp3",
  )
  open("hello.mp3", "wb").write(resp.read())
  ```

  ```javascript Node theme={null}
  import fs from "node:fs";

  const resp = await client.audio.speech.create({
    model: "fish-audio/s2.1-pro",
    input: "Hello from Fish Audio!",
    voice: "", // a voice ID from your Fish voice library; "" = model default
    response_format: "mp3",
  });
  fs.writeFileSync("hello.mp3", Buffer.from(await resp.arrayBuffer()));
  ```

  ```bash curl theme={null}
  curl https://api.fish.audio/compat/openai/v1/audio/speech \
    -H "Authorization: Bearer $FISH_AUDIO_API_KEY" \
    -H "Content-Type: application/json" \
    -d '{"model": "fish-audio/s2.1-pro",
         "input": "Hello from Fish Audio!",
         "response_format": "mp3"}' \
    --output hello.mp3
  ```
</CodeGroup>

Audio streams as it is synthesized. Set `response_format` explicitly; it
defaults to `pcm` on this endpoint, not `mp3`. `voice` is a Fish voice ID, not
a vendor preset; see
[Voices](/developer-guide/getting-started/migration#voices).

The accepted formats are `mp3`, `pcm`, `pcm16`, `wav`, and `opus`, the same
set as the rest of `/openai/v1`; the `flac`, `mulaw`, and `ogg`
the SDK types also offer are formats Fish cannot produce and return a 400.
`sample_rate` (Hz) is accepted as an extension field on the request body
(untyped: `extra_body` in Python, an extra property in Node; defaults to
44100\); asking for a rate the codec doesn't take is a 400, and the per-codec
rate table is in
[Audio formats](/developer-guide/compat/capabilities#audio-formats).

<Accordion title="Groq SDK type quirks: casting pcm, pcm16, opus, and srt, vtt">
  The SDK's `response_format` types omit `pcm`, `pcm16`, and `opus`, and the
  same `Literal` narrows the Python SDK, so type checkers flag them there
  too. Cast at the call site:

  <CodeGroup>
    ```python Python theme={null}
    response_format="opus",  # type: ignore[arg-type]
    ```

    ```typescript TypeScript theme={null}
    response_format: "opus" as SpeechCreateParams["response_format"],
    ```
  </CodeGroup>

  The transcription side types `response_format` as
  `json | text | verbose_json`, so `srt`/`vtt` need the same cast, and the
  SDK hands the subtitle document back as a plain string even though the
  method is typed to return `Transcription`.
</Accordion>

## Transcription

<CodeGroup>
  ```python Python theme={null}
  tr = client.audio.transcriptions.create(
      model="fish-audio/transcribe-1",
      file=open("hello.mp3", "rb"),
  )
  print(tr.text)
  ```

  ```javascript Node theme={null}
  import fs from "node:fs";

  const buf = fs.readFileSync("hello.mp3");
  const tr = await client.audio.transcriptions.create({
    model: "fish-audio/transcribe-1",
    file: new File([buf], "hello.mp3"),
  });
  console.log(tr.text);
  ```

  ```bash curl theme={null}
  curl https://api.fish.audio/compat/openai/v1/audio/transcriptions \
    -H "Authorization: Bearer $FISH_AUDIO_API_KEY" \
    -F model=fish-audio/transcribe-1 \
    -F file=@hello.mp3
  ```
</CodeGroup>

`verbose_json`, word-level timestamp granularity, and the subtitle formats
(`srt`, `vtt`) work as they do on the OpenAI protocol; details and placeholder-field
caveats in [Migrate from OpenAI](/developer-guide/compat/migrate-from-openai#transcription);
`srt`/`vtt` need the SDK type cast shown above.

Translation has no counterpart: Fish has no translation model, so
`client.audio.translations` returns a 404. Transcribe in the source language and
translate the text downstream.

## Model catalog

<CodeGroup>
  ```python Python theme={null}
  ids = [m.id for m in client.models.list().data]
  # ["fish-audio/s2.1-pro", "fish-audio/s2.1-pro-free", ...]
  ```

  ```javascript Node theme={null}
  const models = await client.models.list();
  const ids = models.data.map(m => m.id);
  // ["fish-audio/s2.1-pro", "fish-audio/s2.1-pro-free", ...]
  ```

  ```bash curl theme={null}
  # No auth required
  curl https://api.fish.audio/compat/openai/v1/models
  ```
</CodeGroup>

Entries follow the OpenRouter model schema: `id` and `created` are populated,
but the Groq SDK's other declared `Model` fields (`object`, `owned_by`,
`active`, `context_window`) come back `undefined`, and the richer Fish fields
(`pricing`, `architecture`, `supported_parameters`) are present but untyped.

Listing only: there is no per-model route, so `models.retrieve(...)` returns a
404\.

## Error handling

Errors map to the Groq SDK's typed exceptions (`AuthenticationError`,
`RateLimitError`, …) with the same envelope as the OpenAI protocol: the HTTP
status as an integer `error.code` on every error. Failures from the Fish Audio
API keep their original status with `type: "provider_error"` and add
`metadata.provider_name: "fish-audio"`; errors from the compatibility layer
itself carry no `metadata`.

## Going further

<CardGroup>
  <Card title="Migrate from OpenAI" icon="message-code" href="/developer-guide/compat/migrate-from-openai">
    Everything behind `/openai/v1`: chat-modality audio, Realtime
    WebSocket, error mapping.
  </Card>

  <Card title="Compatibility" icon="list-check" href="/developer-guide/compat/capabilities">
    The full contract: mappings, limits, and explicit refusals.
  </Card>
</CardGroup>


# Migrate from OpenAI
Source: https://docs.fish.audio/developer-guide/compat/migrate-from-openai

Point the official OpenAI SDKs at Fish Audio: speech, transcription, chat-modality audio, and the Realtime WebSocket.

Configure your client once; every call shape below then works unchanged.

<CodeGroup>
  ```python Python theme={null}
  import os

  from openai import OpenAI

  client = OpenAI(
      base_url="https://api.fish.audio/compat/v1",
      api_key=os.environ["FISH_AUDIO_API_KEY"],
  )
  ```

  ```javascript Node theme={null}
  import OpenAI from "openai";

  const client = new OpenAI({
    baseURL: "https://api.fish.audio/compat/v1",
    apiKey: process.env.FISH_AUDIO_API_KEY,
  });
  ```
</CodeGroup>

The same change works in any framework that takes an OpenAI-compatible base
URL. Authentication is the standard `Authorization: Bearer` header; on
WebSockets, `?api_key=<key>` in the URL also works. The
Groq SDK speaks the same protocol; see
[Migrate from Groq](/developer-guide/compat/migrate-from-groq).

| Feature | Supported |
| - | - |
| Text to speech (streamed) | ✓ |
| Speech to text + word timestamps | ✓ (REST) |
| Subtitles (`srt` / `vtt`) | ✓ |
| Chat-modality audio | ✓ |
| Realtime WebSocket (TTS + STT) | ✓ |
| Model catalog | ✓ |
| TTS character timestamps | – |
| Voice library listing | – |

## Text to speech

`POST /v1/audio/speech`

<CodeGroup>
  ```python Python theme={null}
  with client.audio.speech.with_streaming_response.create(
      model="fish-audio/s2.1-pro",
      input="Hello from Fish Audio!",
      voice="",  # a voice ID from your Fish voice library; "" = model default
      response_format="mp3",  # mp3 | wav | opus | pcm | pcm16
      speed=1.1,
  ) as r:
      r.stream_to_file("out.mp3")
  ```

  ```javascript Node theme={null}
  import fs from "node:fs";

  const response = await client.audio.speech.create({
    model: "fish-audio/s2.1-pro",
    input: "Hello from Fish Audio!",
    voice: "", // a voice ID from your Fish voice library; "" = model default
    response_format: "mp3", // mp3 | wav | opus | pcm | pcm16
    speed: 1.1,
  });
  fs.writeFileSync("out.mp3", Buffer.from(await response.arrayBuffer()));
  ```

  ```bash curl theme={null}
  curl https://api.fish.audio/compat/v1/audio/speech \
    -H "Authorization: Bearer $FISH_AUDIO_API_KEY" \
    -H "Content-Type: application/json" \
    -d '{"model": "fish-audio/s2.1-pro",
         "input": "Hello from Fish Audio!",
         "response_format": "mp3",
         "speed": 1.1}' \
    --output out.mp3
  ```
</CodeGroup>

Differences from OpenAI:

* **Set `response_format` explicitly.** It defaults to `pcm` here, not `mp3`
  as on OpenAI. Code that omits it and writes the bytes to `out.mp3` produces a
  file that won't play.
* **`voice` is a Fish voice ID**; see
  [Voices](/developer-guide/getting-started/migration#voices). Preset names
  (`nova`, `echo`, …) are a 400, except `alloy`, which happens to synthesize in
  an unrelated voice; see
  [capabilities](/developer-guide/compat/capabilities#what-each-protocol-supports).
* **`instructions` is refused with a 400.** Use `X-Fish-*` headers or
  `provider.options` for delivery control instead.
* **`stream_format` must be `audio`.** `stream_format: "sse"` is a first-class
  parameter in current OpenAI SDKs and is refused with a 400 here; the
  response is always the raw audio stream.

`speed` works as on OpenAI. `sample_rate` (Hz) triggers a real resample, but
each codec accepts only certain rates and asking for one it doesn't take is a
**400, not a silent fallback**; the per-codec rate table is in
[Audio formats](/developer-guide/compat/capabilities#audio-formats).

One asymmetry between the two SDKs: `sample_rate` is not a first-class
parameter in the OpenAI **Python** SDK, so passing it to `create()` raises
`TypeError`. Send it through `extra_body`. The Node SDK is more permissive and
accepts it inline in the request object.

```python theme={null}
with client.audio.speech.with_streaming_response.create(
    model="fish-audio/s2.1-pro",
    input="Hello from Fish Audio!",
    voice="",
    response_format="wav",
    extra_body={"sample_rate": 16000},
) as r:
    r.stream_to_file("out.wav")
```

Your existing model names carry over: `tts-1`, `tts-1-hd`, and
`gpt-4o-mini-tts` are accepted as aliases for `fish-audio/s2.1-pro`. A name
that is neither a known alias nor a Fish model is refused with a 400, never
silently substituted.

## Transcription

`POST /v1/audio/transcriptions`

<CodeGroup>
  ```python Python theme={null}
  tr = client.audio.transcriptions.create(
      model="fish-audio/transcribe-1",
      file=open("meeting.wav", "rb"),
      response_format="verbose_json",  # json | verbose_json | text | srt | vtt
      timestamp_granularities=["word"],
  )
  ```

  ```javascript Node theme={null}
  import fs from "node:fs";

  const tr = await client.audio.transcriptions.create({
    model: "fish-audio/transcribe-1",
    file: fs.createReadStream("meeting.wav"),
    response_format: "verbose_json", // json | verbose_json | text | srt | vtt
    timestamp_granularities: ["word"],
  });
  ```

  ```bash curl theme={null}
  curl https://api.fish.audio/compat/v1/audio/transcriptions \
    -H "Authorization: Bearer $FISH_AUDIO_API_KEY" \
    -F model=fish-audio/transcribe-1 \
    -F file=@meeting.wav \
    -F response_format=verbose_json \
    -F 'timestamp_granularities[]=word'
  ```
</CodeGroup>

`timestamp_granularities=["word"]` is what adds the top-level `words` array;
without it `verbose_json` returns `segments[]` alone. (Fish times every word, so
`segments[]` is per-word either way; the flag controls the `words` array, not
the precision.)

Subtitle formats (`srt`, `vtt`) are aggregated from Fish's word-level
timestamps into phrase-length cues, so they won't line up one-to-one with the
per-word entries `verbose_json` returns in `segments[]`.

In `verbose_json`, the `language` field echoes the language **you** sent, not a
detection result; `transcribe-1` needs no hint to work. Send none and the field
comes back as an empty string, not a failure. Don't route on it. (The value you send is still passed to Fish's ASR;
it's only the reported field that is an echo rather than a measurement.)

Transcription model names carry over the same way: `whisper-1`,
`gpt-4o-transcribe`, and `gpt-4o-mini-transcribe` are accepted as aliases for
`fish-audio/transcribe-1`; an unrecognized name is refused with the same 400.

For speaker turns, long recordings, and inline emotion cues, use
`transcribe-1-pro` on the native
[Speech to Text API](/features/speech-to-text) instead.

<Note>
  In `verbose_json`, the Whisper-engine internals (`avg_logprob`,
  `no_speech_prob`, `compression_ratio`, `temperature`) are neutral constants;
  don't build quality filters on them.
</Note>

The OpenAI SDK's `client.audio.translations` has no counterpart here:
`POST /v1/audio/translations` is a 404. Transcribe in the source language and
translate the text downstream.

## Chat-modality audio

`POST /v1/chat/completions` serves TTS and STT in the chat-completions shape,
for SDKs and frameworks that only speak chat: LangChain, Vercel AI SDK, and
similar.

For **speech output**, request the audio modality; the last user message's text
is synthesized:

<CodeGroup>
  ```python Python theme={null}
  import base64

  resp = client.chat.completions.create(
      model="fish-audio/s2.1-pro",
      modalities=["text", "audio"],
      audio={"voice": "", "format": "mp3"},
      messages=[{"role": "user", "content": "Hello from Fish Audio!"}],
  )
  open("hello.mp3", "wb").write(base64.b64decode(resp.choices[0].message.audio.data))
  ```

  ```javascript Node theme={null}
  import fs from "node:fs";

  const resp = await client.chat.completions.create({
    model: "fish-audio/s2.1-pro",
    modalities: ["text", "audio"],
    audio: { voice: "", format: "mp3" },
    messages: [{ role: "user", content: "Hello from Fish Audio!" }],
  });
  fs.writeFileSync(
    "hello.mp3",
    Buffer.from(resp.choices[0].message.audio.data, "base64"),
  );
  ```

  ```bash curl theme={null}
  curl https://api.fish.audio/compat/v1/chat/completions \
    -H "Authorization: Bearer $FISH_AUDIO_API_KEY" \
    -H "Content-Type: application/json" \
    -d '{"model": "fish-audio/s2.1-pro",
         "modalities": ["text", "audio"],
         "audio": {"voice": "", "format": "mp3"},
         "messages": [{"role": "user", "content": "Hello from Fish Audio!"}]}'
  ```
</CodeGroup>

With `stream: true` the audio arrives over SSE in base64 chunks that are safe
to concatenate as they come.

For **speech input**, put an `input_audio` content part in the last user
message; the reply's `content` is the transcript.

Two refusals on this endpoint: `n` must be 1, and one call cannot combine audio
input with audio output.

## Realtime WebSocket

`wss://api.fish.audio/compat/v1/realtime?model=fish-audio/s2.1-pro`

The SDK derives the WebSocket URL from the same client:

```javascript theme={null}
import OpenAI from "openai";
import { OpenAIRealtimeWS } from "openai/beta/realtime/ws";

const client = new OpenAI({
  baseURL: "https://api.fish.audio/compat/v1",
  apiKey: process.env.FISH_AUDIO_API_KEY,
});
const rt = new OpenAIRealtimeWS({ model: "fish-audio/s2.1-pro" }, client);
// Stand-in for your audio pipeline; deltas are base64 chunks.
const playChunk = b64 => process.stdout.write(`audio: ${b64.length} b64 chars\n`);

rt.on("session.created", () => {
  rt.send({
    type: "conversation.item.create",
    item: {
      type: "message",
      role: "user",
      content: [{ type: "input_text", text: "Hello from Fish Audio!" }],
    },
  });
  rt.send({ type: "response.create" });
});
rt.on("response.output_audio.delta", ev => playChunk(ev.delta));
```

The essentials: the endpoint speaks the Realtime **GA** event names, you drive
turn detection yourself (`input_audio_buffer.commit` ends an utterance;
server-side VAD is refused, not ignored), and transcription runs over the same
socket or in a dedicated transcription session. The full event list, format
and voice rules, and transcription-session handshakes are in the
[Realtime protocol reference](/developer-guide/compat/realtime-protocol).

## Error handling

Errors map to your SDK's typed exceptions (`AuthenticationError`,
`RateLimitError`, `APIStatusError`, …). The envelope carries the HTTP status as
an integer `code`:

```json theme={null}
{
  "error": {
    "code": 400,
    "message": "…",
    "type": "invalid_request_error"
  }
}
```

Errors from the Fish Audio API keep their original status (for example 402
when your account is out of credit) and carry `type: "provider_error"` plus
`"metadata": {"provider_name": "fish-audio"}`. Errors from the compatibility
layer itself carry no `metadata`. Rate limits return 429 with a `Retry-After`
header.

Authentication produces both kinds, and the `type` tells you which problem you
have. No key, or a header that can't be parsed, is a 401 with
`type: "authentication_error"` and no `metadata`: nothing was checked against
your account. A key that fails validation is a 401 with
`type: "provider_error"` and `metadata.provider_name: "fish-audio"`: the key
itself is invalid. Both surface as your SDK's `AuthenticationError`.

`GET /v1/models` is unauthenticated and answers even without a key, so it is
not a way to check whether a key is valid.

## Going further

<CardGroup>
  <Card title="Compatibility" icon="list-check" href="/developer-guide/compat/capabilities">
    The full contract: mappings, limits, and explicit refusals.
  </Card>

  <Card title="Fish-native parameters" icon="sliders" href="/developer-guide/compat/capabilities#reaching-fish-native-parameters">
    `latency`, `temperature`, and zero-shot cloning through the OpenAI SDK.
  </Card>
</CardGroup>


# Migrate from OpenRouter
Source: https://docs.fish.audio/developer-guide/compat/migrate-from-openrouter

Point the official OpenRouter TypeScript SDK at Fish Audio: TTS, STT, and the model catalog.

Configure the client once. `serverURL` takes the `/api/v1` prefix:

```typescript theme={null}
import { OpenRouter } from "@openrouter/sdk";

const client = new OpenRouter({
  apiKey: process.env.FISH_AUDIO_API_KEY,
  serverURL: "https://api.fish.audio/compat/api/v1",
});
```

From Python, OpenRouter's own SDK works the same way: `pip install openrouter`,
then pass `server_url`:

```python theme={null}
import os

from openrouter import OpenRouter

client = OpenRouter(
    api_key=os.environ["FISH_AUDIO_API_KEY"],
    server_url="https://api.fish.audio/compat/api/v1",
)

res = client.tts.create_speech(
    model="fish-audio/s2.1-pro",
    input="Hello from Fish Audio!",
    response_format="mp3",
)
open("hello.mp3", "wb").write(res.read())
```

The OpenAI SDK works against the same base URL too; the Python tabs below
use it.

| Feature | Supported |
| - | - |
| Text to speech (streamed) | ✓ |
| Speech to text + word timestamps | ✓ |
| Model catalog (full schema) | ✓ |
| `temperature` / `top_p` / `repetition_penalty` | ✓ (top level) |
| `input_references` (stateless cloning) | ✓ |
| Chat-modality audio & Realtime | via the OpenAI SDK |
| TTS character timestamps | – |
| Voice library listing | – |

## Text to speech

<CodeGroup>
  ```typescript Node theme={null}
  import fs from "node:fs";

  const stream = await client.tts.createSpeech({
    speechRequest: {
      model: "fish-audio/s2.1-pro",
      input: "Hello from Fish Audio!",
      responseFormat: "mp3", // defaults to pcm when omitted
    },
  });

  const chunks = [];
  for await (const chunk of stream) chunks.push(chunk);
  fs.writeFileSync("hello.mp3", Buffer.concat(chunks));
  ```

  ```python Python theme={null}
  import os

  from openai import OpenAI

  client = OpenAI(
      base_url="https://api.fish.audio/compat/api/v1",
      api_key=os.environ["FISH_AUDIO_API_KEY"],
  )

  with client.audio.speech.with_streaming_response.create(
      model="fish-audio/s2.1-pro",
      input="Hello from Fish Audio!",
      voice="",  # a voice ID from your Fish voice library; "" = model default
      response_format="mp3",
  ) as r:
      r.stream_to_file("hello.mp3")
  ```

  ```bash curl theme={null}
  curl https://api.fish.audio/compat/api/v1/audio/speech \
    -H "Authorization: Bearer $FISH_AUDIO_API_KEY" \
    -H "Content-Type: application/json" \
    -d '{"model": "fish-audio/s2.1-pro",
         "input": "Hello from Fish Audio!",
         "response_format": "mp3"}' \
    --output hello.mp3
  ```
</CodeGroup>

The response is a `ReadableStream`: audio bytes arrive as they are synthesized.
The optional `voice` field takes a voice ID from your Fish voice library
(omitted or empty = model default; see
[Voices](/developer-guide/getting-started/migration#voices)).

`input_references` (OpenRouter's first-class field for stateless voice
cloning, reference audio plus its transcript) is supported: no saved voice
model required. Both SDKs carry the field on their `SpeechRequest` type
(`inputReferences` in TypeScript, `input_references` in Python).

### `temperature`, `top_p`, `repetition_penalty`

These three sit **at the top level of the speech request body**, next to
`model` and `input` (no `provider.options` wrapper required), matching what
the catalog advertises in each model's `supported_parameters`.

```json theme={null}
{
  "model": "fish-audio/s2.1-pro",
  "input": "Hello from Fish Audio!",
  "response_format": "mp3",
  "temperature": 0.4,
  "top_p": 0.7,
  "repetition_penalty": 1.1
}
```

Neither SDK's speech type declares them, so they still have to go around the
typed request: as raw JSON via `fetch`, or through the OpenAI SDK's
`extra_body`.

The same three are also reachable through `provider.options.fish-audio` and the
`X-Fish-*` headers, and if you set a value two ways the more specific one wins:
**`X-Fish-*` header beats `provider.options`, and `provider.options` beats the
top level.** Everything else in the Fish parameter set (`latency`,
`chunk_length`, `normalize`, `prosody`, references) is available only through
those two channels, not at the top level. See
[Fish-native parameters](/developer-guide/compat/capabilities#reaching-fish-native-parameters).

## Transcription

<CodeGroup>
  ```typescript Node theme={null}
  const audio = fs.readFileSync("hello.mp3");

  const result = await client.stt.createTranscriptionMultipart({
    requestBody: {
      model: "fish-audio/transcribe-1",
      file: { fileName: "hello.mp3", content: audio },
    },
  });
  console.log(result.text);
  ```

  ```python Python theme={null}
  import os

  from openai import OpenAI

  client = OpenAI(
      base_url="https://api.fish.audio/compat/api/v1",
      api_key=os.environ["FISH_AUDIO_API_KEY"],
  )

  tr = client.audio.transcriptions.create(
      model="fish-audio/transcribe-1",
      file=open("hello.mp3", "rb"),
  )
  print(tr.text)
  ```

  ```bash curl theme={null}
  curl https://api.fish.audio/compat/api/v1/audio/transcriptions \
    -H "Authorization: Bearer $FISH_AUDIO_API_KEY" \
    -F model=fish-audio/transcribe-1 \
    -F file=@hello.mp3
  ```
</CodeGroup>

JSON bodies with base64 `input_audio` also work, matching OpenRouter's schema.
For word-level timing, ask for `response_format="verbose_json"` together with
`timestamp_granularities=["word"]`.

For speaker turns, long recordings, and inline emotion cues, use
`transcribe-1-pro` on the native
[Speech to Text API](/features/speech-to-text) instead.

## Model catalog

<CodeGroup>
  ```typescript Node theme={null}
  const page = await client.models.list({});
  const ids = page.result.data.map(m => m.id);
  // ["fish-audio/s2.1-pro", "fish-audio/s2.1-pro-free", ...]
  ```

  ```python Python theme={null}
  import os

  from openai import OpenAI

  client = OpenAI(
      base_url="https://api.fish.audio/compat/api/v1",
      api_key=os.environ["FISH_AUDIO_API_KEY"],
  )

  ids = [m.id for m in client.models.list()]
  # ["fish-audio/s2.1-pro", "fish-audio/s2.1-pro-free", ...]
  ```

  ```bash curl theme={null}
  # No auth required
  curl https://api.fish.audio/compat/api/v1/models
  ```
</CodeGroup>

Entries validate against the OpenRouter `Model` schema: **every required field
is present, and the ones with no Fish counterpart are explicitly `null` rather
than missing**, so SDK model validation passes and `model.context_length` reads
as `None`/`null` instead of raising.

Four fields are always `null`: `context_length`, `default_parameters`,
`per_request_limits`, and `supported_voices`. Six optional ones
(`alias_target`, `benchmarks`, `expiration_date`, `hugging_face_id`,
`knowledge_cutoff`, and `reasoning`) are simply absent; don't write code that
expects the key to exist. Filter with `output_modalities=speech` or `text` to
list one direction only.

## Error handling

Errors use the OpenRouter envelope, with the HTTP status as an integer in
`error.code`. Failures from the Fish Audio API surface as
`type: "provider_error"` with their original status and
`metadata.provider_name: "fish-audio"`; errors from the compatibility layer
itself carry no `metadata`, just `code`, `message`, and `type`.

## Going further

<CardGroup>
  <Card title="Compatibility" icon="list-check" href="/developer-guide/compat/capabilities">
    The full contract: mappings, limits, and explicit refusals.
  </Card>

  <Card title="Fish-native parameters" icon="sliders" href="/developer-guide/compat/capabilities#reaching-fish-native-parameters">
    Reach `latency`, `chunk_length`, `normalize`, and more via
    `provider.options` or `X-Fish-*` headers.
  </Card>
</CardGroup>


# Realtime protocol reference
Source: https://docs.fish.audio/developer-guide/compat/realtime-protocol

What the OpenAI-compatible Realtime WebSocket speaks: events, formats, turn detection, and transcription sessions.

```
wss://api.fish.audio/compat/v1/realtime?model=fish-audio/s2.1-pro
```

Authentication is the standard `Authorization: Bearer` header, or
`?api_key=<key>` in the URL. The OpenAI SDK derives this URL from the same
client you configured for REST; a working example is in
[Migrate from OpenAI](/developer-guide/compat/migrate-from-openai#realtime-websocket).

## Events

The endpoint speaks the Realtime **GA** event names
(`response.output_audio.delta` and the rest of the GA set), which is what
current `openai` packages emit, even though the Node module path still says
`beta`.

Supported client events: `session.update`, `transcription_session.update`,
`conversation.item.create`, `conversation.item.delete`,
`input_audio_buffer.append` / `commit` / `clear`, `response.create`,
`response.cancel`. `conversation.item.truncate` is accepted and does nothing
(it cuts a previous assistant message's audio, and none is retained
server-side), so it is the one client event that sends no acknowledgement back. Any other event
type comes back as an `unknown_event_type` error.

One response at a time per session: a second `response.create` while one is
running returns the `response_already_active` error.

## Audio formats

Output formats: `pcm16`, `wav`, `mp3`, `opus`, as a string (`"mp3"`) or the
GA object form (`{"type": "audio/mp3", "rate": 32000}`). `audio/pcmu` and
`audio/pcma` are rejected explicitly. **Both spellings reset the rate when
they carry no `rate` of their own**: to that format's default (`audio/pcm` →
24000, `wav` and `mp3` → 44100, `opus` → 48000), not to what the session was
using before, so set `rate` explicitly when you switch formats mid-session.
`session.updated` echoes the resulting output *and* input format, rate
included, so you can read back what the session settled on. Supported sample
rates per codec are in
[Audio formats](/developer-guide/compat/capabilities#audio-formats).

## Voice and speed

`voice` is a Fish voice ID, accepted both at the legacy top-level
`session.voice` and at the GA `session.audio.output.voice` position; session
events echo it at the GA position. `speed` follows the same two-position rule
and the REST endpoint's 0.25–4.0 range; out of range returns an
`invalid_speed` error event instead of synthesizing.

## Turn detection

Turn detection is yours to drive: send `input_audio_buffer.commit` to end an
utterance. Any `turn_detection` with a type (`server_vad`, `semantic_vad`, or
anything else, in either the GA or the legacy top-level position) is refused
with `unsupported_turn_detection` rather than accepted and ignored, and
`session.created` reports `"turn_detection": null`. Likewise, an
`output_modalities` that omits `"audio"` is refused: this socket always
produces audio.

## Transcription

Transcription works over the same socket: `input_audio_buffer.append` base64
audio, then `input_audio_buffer.commit`. The transcript arrives as
`conversation.item.input_audio_transcription.completed`, and a failed
transcription arrives as the matching
`conversation.item.input_audio_transcription.failed` (the buffered audio is
kept, so you can retry). The socket returns the transcript text only; for
word timestamps use `POST /v1/audio/transcriptions`.

Both official ways of opening a **dedicated** transcription session work: send
`transcription_session.update`, or connect with `?intent=transcription` in the
URL. Either way the session object comes back as
`realtime.transcription_session` in `transcription_session.created` /
`transcription_session.updated`. With `?intent=transcription` and no `model`
in the URL, the session defaults to `fish-audio/transcribe-1` rather than a
TTS model.

## Limits

Session limits (text and audio caps per conversation item, item counts, frame
size) are in the
[limits table](/developer-guide/compat/capabilities#limits-and-operational-contract).

## Related

<CardGroup>
  <Card title="Migrate from OpenAI" icon="message-code" href="/developer-guide/compat/migrate-from-openai#realtime-websocket">
    A runnable Realtime example with the official SDK.
  </Card>

  <Card title="Compatibility" icon="list-check" href="/developer-guide/compat/capabilities">
    The full contract: mappings, limits, and explicit refusals.
  </Card>
</CardGroup>


# Emotion Control
Source: https://docs.fish.audio/developer-guide/core-features/emotions

Add natural emotions and expressions to your AI-generated speech

<Card title="Try it live in the API playground" icon="flask" href="/api-reference/endpoint/openapi-v1/text-to-speech">
  Drop text with the markers below into the `text` field and send a real request to hear the emotion.
</Card>

## Overview

Fish Audio models support 64+ emotional expressions and voice styles that can be controlled through text markers in your input. Add natural pauses, laughter, and other human-like elements to make speech more engaging and realistic.

<Tip>
  This page shows S2 usage with `[bracket]` cues. If you use the legacy S1 model,
  wrap markers in parentheses instead; see [S1 (legacy) syntax](#s1-legacy-syntax)
  below for the full list, or the [Models
  Overview](/developer-guide/models-pricing/models-overview#s2-natural-language-control).
</Tip>

## How It Works

Add emotional or stylistic cues in square brackets within your text:

```text theme={null}
[happy] What a beautiful day!
[sad] I'm sorry to hear that.
[excited] This is amazing news!
```

The S2 TTS models will interpret these markers and adjust the voice accordingly.

## Complete Emotion Reference

### Basic Emotions (24 expressions)

| Emotion | Tag | Description | Example Context |
| - | - | - | - |
| Happy | `[happy]` | Cheerful, upbeat tone | Good news, greetings |
| Sad | `[sad]` | Melancholic, downcast | Sympathy, bad news |
| Angry | `[angry]` | Frustrated, aggressive | Complaints, warnings |
| Excited | `[excited]` | Energetic, enthusiastic | Announcements, celebrations |
| Calm | `[calm]` | Peaceful, relaxed | Instructions, meditation |
| Nervous | `[nervous]` | Anxious, uncertain | Disclaimers, apologies |
| Confident | `[confident]` | Assertive, self-assured | Presentations, sales |
| Surprised | `[surprised]` | Shocked, amazed | Reactions, discoveries |
| Satisfied | `[satisfied]` | Content, pleased | Confirmations, reviews |
| Delighted | `[delighted]` | Very pleased, joyful | Celebrations, compliments |
| Scared | `[scared]` | Frightened, fearful | Warnings, horror stories |
| Worried | `[worried]` | Concerned, troubled | Concerns, questions |
| Upset | `[upset]` | Disturbed, distressed | Complaints, problems |
| Frustrated | `[frustrated]` | Annoyed, exasperated | Technical issues, delays |
| Depressed | `[depressed]` | Very sad, hopeless | Serious topics |
| Empathetic | `[empathetic]` | Understanding, caring | Support, counseling |
| Embarrassed | `[embarrassed]` | Ashamed, awkward | Apologies, mistakes |
| Disgusted | `[disgusted]` | Repelled, revolted | Negative reviews |
| Moved | `[moved]` | Emotionally touched | Heartfelt moments |
| Proud | `[proud]` | Accomplished, satisfied | Achievements, praise |
| Relaxed | `[relaxed]` | At ease, casual | Casual conversation |
| Grateful | `[grateful]` | Thankful, appreciative | Thanks, appreciation |
| Curious | `[curious]` | Inquisitive, interested | Questions, exploration |
| Sarcastic | `[sarcastic]` | Ironic, mocking | Humor, criticism |

### Advanced Emotions (25 expressions)

| Emotion | Tag | Description | Example Context |
| - | - | - | - |
| Disdainful | `[disdainful]` | Contemptuous, scornful | Criticism, rejection |
| Unhappy | `[unhappy]` | Discontent, dissatisfied | Complaints, feedback |
| Anxious | `[anxious]` | Very worried, uneasy | Urgent matters |
| Hysterical | `[hysterical]` | Uncontrollably emotional | Extreme reactions |
| Indifferent | `[indifferent]` | Uncaring, neutral | Neutral responses |
| Uncertain | `[uncertain]` | Doubtful, unsure | Speculation, questions |
| Doubtful | `[doubtful]` | Skeptical, questioning | Disbelief, questioning |
| Confused | `[confused]` | Puzzled, perplexed | Clarification requests |
| Disappointed | `[disappointed]` | Let down, dissatisfied | Unmet expectations |
| Regretful | `[regretful]` | Sorry, remorseful | Apologies, mistakes |
| Guilty | `[guilty]` | Culpable, responsible | Confessions, apologies |
| Ashamed | `[ashamed]` | Deeply embarrassed | Serious mistakes |
| Jealous | `[jealous]` | Envious, resentful | Comparisons |
| Envious | `[envious]` | Wanting what others have | Admiration with desire |
| Hopeful | `[hopeful]` | Optimistic about future | Future plans |
| Optimistic | `[optimistic]` | Positive outlook | Encouragement |
| Pessimistic | `[pessimistic]` | Negative outlook | Warnings, doubts |
| Nostalgic | `[nostalgic]` | Longing for the past | Memories, stories |
| Lonely | `[lonely]` | Isolated, alone | Emotional content |
| Bored | `[bored]` | Uninterested, weary | Disinterest |
| Contemptuous | `[contemptuous]` | Showing contempt | Strong criticism |
| Sympathetic | `[sympathetic]` | Showing sympathy | Condolences |
| Compassionate | `[compassionate]` | Showing deep care | Support, help |
| Determined | `[determined]` | Resolved, decided | Goals, commitments |
| Resigned | `[resigned]` | Accepting defeat | Giving up, acceptance |

## Sound & Delivery Markers

These markers aren't emotions. They shape *how* a line is delivered, add natural human sounds, or layer in ambient effects. Combine them with the emotion cues above.

### Tone Markers (6 expressions)

Control volume, intensity, and emphasis. Place `[emphasis]` right before the word or phrase you want to stress:

```text theme={null}
This is [emphasis] really important.
```

| Tone | Tag | Description | When to Use |
| - | - | - | - |
| Hurried | `[in a hurry tone]` | Rushed, urgent | Time-sensitive information |
| Shouting | `[shouting]` | Loud, calling out | Getting attention |
| Screaming | `[screaming]` | Very loud, panicked | Emergencies, fear |
| Whispering | `[whispering]` | Very soft, secretive | Secrets, quiet scenes |
| Soft | `[soft tone]` | Gentle, quiet | Comfort, lullabies |
| Emphasis | `[emphasis]` | Stress a word/phrase | Highlighting key words |

### Audio Effects (11 expressions)

Add natural human sounds:

| Effect | Tag | Description | Suggested Text |
| - | - | - | - |
| Laughing | `[laughing]` | Full laughter | Ha, ha, ha |
| Chuckling | `[chuckling]` | Light laugh | Heh, heh |
| Sobbing | `[sobbing]` | Crying heavily | Optional text |
| Crying Loudly | `[crying loudly]` | Intense crying | Optional text |
| Sighing | `[sighing]` | Exhale of relief/frustration | sigh |
| Groaning | `[groaning]` | Sound of frustration | ugh |
| Panting | `[panting]` | Out of breath | huff, puff |
| Gasping | `[gasping]` | Sharp intake of breath | gasp |
| Yawning | `[yawning]` | Tired sound | yawn |
| Snoring | `[snoring]` | Sleep sound | zzz |
| Clear Throat | `[clear throat]` | Throat-clearing sound | ahem |

### Special Effects

Additional markers for atmosphere and context:

| Effect | Tag | Description |
| - | - | - |
| Audience Laughter | `[audience laughing]` | Crowd laughing sound |
| Background Laughter | `[background laughter]` | Ambient laughter |
| Crowd Laughter | `[crowd laughing]` | Large group laughing |
| Short Pause | `[break]` | Brief pause in speech |
| Long Pause | `[long-break]` | Extended pause in speech |

You can also use natural expressions like "Ha,ha,ha" for laughter without tags.

## Usage Guidelines

### Placement Rules

**For S2:**

* Sentence-level emotion cues usually work best at the beginning of sentences
* Tone controls can go anywhere in the text
* Sound effects can go anywhere in the text
* Bracket cues can use natural language descriptions and are not limited to a fixed set of tags

**Correct:**

```text theme={null}
[happy] What a wonderful day!
What a [warm and happy] wonderful day!
```

## Advanced Techniques

### Combining Effects

You can layer multiple emotions for complex expressions:

```text theme={null}
[sad][whispering] I miss you so much.
[angry][shouting] Get out of here now!
[excited][laughing] We won! Ha ha!
```

### Emotion Transitions

Create natural emotional progressions:

```text theme={null}
[happy] I got the promotion!
[uncertain] But... it means relocating.
[sad] I'll miss everyone here.
[hopeful] Though it's a great opportunity.
[determined] I'm going to make it work!
```

### Background Effects

Add atmospheric sounds:

```text theme={null}
The comedy show was amazing [audience laughing]
Everyone was having fun [background laughter]
The crowd loved it [crowd laughing]
```

### Intensity Modifiers

Fine-tune emotional intensity with descriptive modifiers:

```text theme={null}
[slightly sad] I'm a bit disappointed.
[very excited] This is absolutely amazing!
[extremely angry] This is unacceptable!
```

## Language Support

All 13 supported languages can use emotion markers. For sentence-level control, cues usually work best at the sentence start in these languages:

* **English, Chinese, Japanese, German, French, Spanish, Korean, Arabic, Russian, Dutch, Italian, Polish, Portuguese**

## Best Practices

### Do's

* Use one primary emotion per sentence
* Test different emotion combinations
* Match emotions to context logically
* Add appropriate text after sound effects (e.g., "Ha ha" after laughing)
* Use natural expressions when possible
* Space out emotional changes for realism

### Don'ts

* Don't overuse emotion tags in short text
* Don't mix conflicting emotions
* Don't make bracket descriptions so long that they interrupt readability
* Don't forget brackets
* Don't place sentence-level emotion cues far from the sentence they control

## Common Use Cases

### Customer Service

```text theme={null}
[friendly] Hello! How can I help you today?
[empathetic] I understand your frustration.
[confident] I'll resolve this for you right away.
[grateful] Thank you for your patience!
```

### Storytelling

```text theme={null}
[narrator] Once upon a time...
[mysterious][whispering] The old house stood silent.
[scared] "Is anyone there?" she called out.
[relieved][sighing] No one answered. Phew.
```

### Educational Content

```text theme={null}
[enthusiastic] Welcome to today's lesson!
[curious] Have you ever wondered why?
[encouraging] That's a great question!
[proud] Excellent work!
```

### Marketing & Sales

```text theme={null}
[excited] Introducing our newest product!
[confident] You won't find better quality anywhere.
[urgent] Limited time offer!
[satisfied] Join thousands of happy customers!
```

## Troubleshooting

### Emotion Not Working?

1. **Check placement** - Put the cue where the emotion or effect should begin
2. **Keep wording clear** - Use concise natural language descriptions
3. **Use the right syntax** - S2 cues use square brackets; S1 cues must use parentheses

### Unnatural Sound?

* Space out emotional changes
* Use appropriate intensity
* Test with different voices
* Add context text after sound effects

### Performance Notes

* Emotion markers don't count toward token limits
* No additional latency for emotion processing
* All emotions available on all pricing tiers
* Maximum of 3 combined emotions per sentence recommended

## Quick Reference Tables

### Emotion Intensity Scale

| Base Emotion | Mild | Moderate | Intense |
| - | - | - | - |
| Happy | satisfied | happy | delighted |
| Sad | disappointed | sad | depressed |
| Angry | frustrated | angry | furious |
| Scared | nervous | scared | terrified |
| Excited | interested | excited | ecstatic |

### Common Combinations

| Scenario | Emotion Combo | Example |
| - | - | - |
| Whispered Secret | `[mysterious][whispering]` | "I have something to tell you..." |
| Angry Shout | `[angry][shouting]` | "Stop right there!" |
| Sad Sigh | `[sad][sighing]` | "I wish things were different. Sigh." |
| Excited Laugh | `[excited][laughing]` | "We did it! Ha ha!" |
| Nervous Question | `[nervous][uncertain]` | "Are you sure about this?" |

## S1 (legacy) syntax

The **S2-Pro** and **S2.1-Pro** family models use `[bracket]` cues with free-form natural language. The previous-generation **S1** model uses the same emotion names but requires `(parentheses)` and a fixed tag set:

```text theme={null}
(happy) What a beautiful day!
(sad)(whispering) I'll miss you so much.
```

<AccordionGroup>
  <Accordion title="Basic emotions (S1)">
    | Emotion | Tag | Description | Example Context |
    | - | - | - | - |
    | Happy | `(happy)` | Cheerful, upbeat tone | Good news, greetings |
    | Sad | `(sad)` | Melancholic, downcast | Sympathy, bad news |
    | Angry | `(angry)` | Frustrated, aggressive | Complaints, warnings |
    | Excited | `(excited)` | Energetic, enthusiastic | Announcements, celebrations |
    | Calm | `(calm)` | Peaceful, relaxed | Instructions, meditation |
    | Nervous | `(nervous)` | Anxious, uncertain | Disclaimers, apologies |
    | Confident | `(confident)` | Assertive, self-assured | Presentations, sales |
    | Surprised | `(surprised)` | Shocked, amazed | Reactions, discoveries |
    | Satisfied | `(satisfied)` | Content, pleased | Confirmations, reviews |
    | Delighted | `(delighted)` | Very pleased, joyful | Celebrations, compliments |
    | Scared | `(scared)` | Frightened, fearful | Warnings, horror stories |
    | Worried | `(worried)` | Concerned, troubled | Concerns, questions |
    | Upset | `(upset)` | Disturbed, distressed | Complaints, problems |
    | Frustrated | `(frustrated)` | Annoyed, exasperated | Technical issues, delays |
    | Depressed | `(depressed)` | Very sad, hopeless | Serious topics |
    | Empathetic | `(empathetic)` | Understanding, caring | Support, counseling |
    | Embarrassed | `(embarrassed)` | Ashamed, awkward | Apologies, mistakes |
    | Disgusted | `(disgusted)` | Repelled, revolted | Negative reviews |
    | Moved | `(moved)` | Emotionally touched | Heartfelt moments |
    | Proud | `(proud)` | Accomplished, satisfied | Achievements, praise |
    | Relaxed | `(relaxed)` | At ease, casual | Casual conversation |
    | Grateful | `(grateful)` | Thankful, appreciative | Thanks, appreciation |
    | Curious | `(curious)` | Inquisitive, interested | Questions, exploration |
    | Sarcastic | `(sarcastic)` | Ironic, mocking | Humor, criticism |
  </Accordion>

  <Accordion title="Advanced emotions (S1)">
    | Emotion | Tag | Description | Example Context |
    | - | - | - | - |
    | Disdainful | `(disdainful)` | Contemptuous, scornful | Criticism, rejection |
    | Unhappy | `(unhappy)` | Discontent, dissatisfied | Complaints, feedback |
    | Anxious | `(anxious)` | Very worried, uneasy | Urgent matters |
    | Hysterical | `(hysterical)` | Uncontrollably emotional | Extreme reactions |
    | Indifferent | `(indifferent)` | Uncaring, neutral | Neutral responses |
    | Uncertain | `(uncertain)` | Doubtful, unsure | Speculation, questions |
    | Doubtful | `(doubtful)` | Skeptical, questioning | Disbelief, questioning |
    | Confused | `(confused)` | Puzzled, perplexed | Clarification requests |
    | Disappointed | `(disappointed)` | Let down, dissatisfied | Unmet expectations |
    | Regretful | `(regretful)` | Sorry, remorseful | Apologies, mistakes |
    | Guilty | `(guilty)` | Culpable, responsible | Confessions, apologies |
    | Ashamed | `(ashamed)` | Deeply embarrassed | Serious mistakes |
    | Jealous | `(jealous)` | Envious, resentful | Comparisons |
    | Envious | `(envious)` | Wanting what others have | Admiration with desire |
    | Hopeful | `(hopeful)` | Optimistic about future | Future plans |
    | Optimistic | `(optimistic)` | Positive outlook | Encouragement |
    | Pessimistic | `(pessimistic)` | Negative outlook | Warnings, doubts |
    | Nostalgic | `(nostalgic)` | Longing for the past | Memories, stories |
    | Lonely | `(lonely)` | Isolated, alone | Emotional content |
    | Bored | `(bored)` | Uninterested, weary | Disinterest |
    | Contemptuous | `(contemptuous)` | Showing contempt | Strong criticism |
    | Sympathetic | `(sympathetic)` | Showing sympathy | Condolences |
    | Compassionate | `(compassionate)` | Showing deep care | Support, help |
    | Determined | `(determined)` | Resolved, decided | Goals, commitments |
    | Resigned | `(resigned)` | Accepting defeat | Giving up, acceptance |
  </Accordion>

  <Accordion title="Tone markers (S1)">
    | Tone | Tag | Description | When to Use |
    | - | - | - | - |
    | Hurried | `(in a hurry tone)` | Rushed, urgent | Time-sensitive information |
    | Shouting | `(shouting)` | Loud, calling out | Getting attention |
    | Screaming | `(screaming)` | Very loud, panicked | Emergencies, fear |
    | Whispering | `(whispering)` | Very soft, secretive | Secrets, quiet scenes |
    | Soft | `(soft tone)` | Gentle, quiet | Comfort, lullabies |
  </Accordion>

  <Accordion title="Audio effects (S1)">
    | Effect | Tag | Description | Suggested Text |
    | - | - | - | - |
    | Laughing | `(laughing)` | Full laughter | Ha, ha, ha |
    | Chuckling | `(chuckling)` | Light laugh | Heh, heh |
    | Sobbing | `(sobbing)` | Crying heavily | (optional) |
    | Crying Loudly | `(crying loudly)` | Intense crying | (optional) |
    | Sighing | `(sighing)` | Exhale of relief/frustration | sigh |
    | Groaning | `(groaning)` | Sound of frustration | ugh |
    | Panting | `(panting)` | Out of breath | huff, puff |
    | Gasping | `(gasping)` | Sharp intake of breath | gasp |
    | Yawning | `(yawning)` | Tired sound | yawn |
    | Snoring | `(snoring)` | Sleep sound | zzz |
  </Accordion>

  <Accordion title="Special effects (S1)">
    | Effect | Tag | Description |
    | - | - | - |
    | Audience Laughter | `(audience laughing)` | Crowd laughing sound |
    | Background Laughter | `(background laughter)` | Ambient laughter |
    | Crowd Laughter | `(crowd laughing)` | Large group laughing |
    | Short Pause | `(break)` | Brief pause in speech |
    | Long Pause | `(long-break)` | Extended pause in speech |
  </Accordion>
</AccordionGroup>

## See Also

* [API Reference](/api-reference/introduction) - Implementation details
* [Text-to-Speech Guide and Best Practices](/features/text-to-speech)


# Fine-grained Control
Source: https://docs.fish.audio/developer-guide/core-features/fine-grained-control

Advanced control over speech generation

<Card title="Try it live in the API playground" icon="flask" href="/api-reference/endpoint/openapi-v1/text-to-speech">
  Put your phoneme or paralanguage tags into the `text` field and send a real
  request to hear the result.
</Card>

<Card title="Use pronunciation dictionaries" icon="book-open" href="/developer-guide/core-features/fine-grained-control/pronunciation-dictionaries">
  Create, preview, and publish pronunciation rules in the web app, then reuse
  them across TTS requests.
</Card>

## Getting Started

To use fine-grained control, you can use either our SDK, API, or Playground.

SDK/API: Phoneme tags are preserved by text normalization, so you can keep the default normalization behavior for pronunciation control. Set `"normalize": false` only when you want to prevent normalization from rewriting the surrounding text, such as numbers, dates, or URLs.

Playground: You can use V1.6 Control Model, without setting any other options.

<Note>
  Disabling normalization may reduce the stability of reading numbers, dates,
  and URLs. You'll need to handle these cases manually for best results.
</Note>

## Phoneme Control

Phoneme control allows you to specify exact pronunciations for words, characters, or short phrases. Wrap the desired pronunciation in `<|phoneme_start|>` and `<|phoneme_end|>` tags.

The replacement scope depends on the language:

* English: replace one word with CMU Arpabet.
* Chinese: replace one character or syllable with tone-number pinyin.
* Japanese: replace a short Japanese word or phrase with OpenJTalk-style romaji and pitch accent markers.

<CardGroup>
  <Card title="English" icon="language" href="/developer-guide/core-features/fine-grained-control/english">
    CMU Arpabet examples for names, homographs, acronyms, and technical terms.
  </Card>

  <Card title="Chinese" icon="language" href="/developer-guide/core-features/fine-grained-control/chinese">
    Tone-number pinyin examples for multi-character words, tones, and polyphonic
    characters.
  </Card>

  <Card title="Japanese" icon="language" href="/developer-guide/core-features/fine-grained-control/japanese">
    OpenJTalk romaji phonemes with pitch accent digits.
  </Card>
</CardGroup>

### Quick Examples

English:

```text theme={null}
I am an <|phoneme_start|>EH1 N JH AH0 N IH1 R<|phoneme_end|>.
```

Chinese:

```text theme={null}
我是一个<|phoneme_start|>gong1<|phoneme_end|><|phoneme_start|>cheng2<|phoneme_end|><|phoneme_start|>shi1<|phoneme_end|>。
```

Japanese:

```text theme={null}
<|phoneme_start|>ha0shi1ga0<|phoneme_end|>見えます。
```

## Paralanguage

Paralanguage controls allow you to add natural speech elements and pauses to make the generated speech sound more human-like. There are two main types of controls:

### Pause Words

You can use common pause words like "um", "uh", "嗯", "啊" to control the rhythm of the speech.

### Special Effects

The following special effects can be added using parentheses:

| Effect | Description | First Available | Stage |
| - | - | - | - |
| `(break)` | Short pause | V1.6 | Experimental |
| `(long-break)` | Extended pause | V1.6 | Experimental |
| `(breath)` | Breathing sound | V1.6 | Experimental |
| `(laugh)` | Laughter sound | V1.6 | Experimental |
| `(cough)` | Coughing sound | V1.6 | Experimental |
| `(lip-smacking)` | Lip smacking sound | V1.6 | Experimental |
| `(sigh)` | Sighing sound | V1.6 | Experimental |

<Warning>
  The effects `(laugh)`, `(cough)`, `(lip-smacking)`, and `(sigh)` are
  developing. You may need to repeat them multiple times for better results.
</Warning>

Example:

```text theme={null}
I am, um, an (break) engineer.
```

You can combine paralanguage and phoneme control in the same text:

```text theme={null}
I am, um, an (break) <|phoneme_start|>EH1 N JH AH0 N IH1 R<|phoneme_end|>.
```


# Chinese Phoneme Control
Source: https://docs.fish.audio/developer-guide/core-features/fine-grained-control/chinese

Control Chinese pronunciation with tone-number pinyin

## Overview

Chinese phoneme control uses pinyin with tone numbers, also known as tone3 pinyin. Wrap one syllable in each `<|phoneme_start|>` and `<|phoneme_end|>` tag.

```text theme={null}
我是一个<|phoneme_start|>gong1<|phoneme_end|><|phoneme_start|>cheng2<|phoneme_end|><|phoneme_start|>shi1<|phoneme_end|>。
```

This format is especially useful for polyphonic characters, names, and domain-specific terms where the default reading may be ambiguous.

## Tone Numbers

Put the tone number at the end of each pinyin syllable:

| Tone | Example | Description |
| - | - | - |
| 1 | `ma1` | High level |
| 2 | `ma2` | Rising |
| 3 | `ma3` | Dipping |
| 4 | `ma4` | Falling |
| 5 | `ma5` | Neutral |

Use lowercase pinyin and keep punctuation outside the phoneme tag.

## Multi-character Words

For a multi-character word, place adjacent phoneme tags in the same order as the original characters:

```text theme={null}
Standard: 我是一个工程师。
With phoneme control: 我是一个<|phoneme_start|>gong1<|phoneme_end|><|phoneme_start|>cheng2<|phoneme_end|><|phoneme_start|>shi1<|phoneme_end|>。
```

You can also tag only the ambiguous character and leave the rest of the sentence unchanged:

```text theme={null}
请把这个字读作<|phoneme_start|>hang2<|phoneme_end|>。
```

## Polyphonic Characters

For polyphonic characters, choose the pinyin that matches the phrase meaning:

```text theme={null}
重庆: <|phoneme_start|>chong2<|phoneme_end|><|phoneme_start|>qing4<|phoneme_end|>
重要: <|phoneme_start|>zhong4<|phoneme_end|><|phoneme_start|>yao4<|phoneme_end|>
```

```text theme={null}
银行: <|phoneme_start|>yin2<|phoneme_end|><|phoneme_start|>hang2<|phoneme_end|>
行走: <|phoneme_start|>xing2<|phoneme_end|><|phoneme_start|>zou3<|phoneme_end|>
```

```text theme={null}
音乐: <|phoneme_start|>yin1<|phoneme_end|><|phoneme_start|>yue4<|phoneme_end|>
快乐: <|phoneme_start|>kuai4<|phoneme_end|><|phoneme_start|>le4<|phoneme_end|>
```

## Generate Pinyin

The training pipeline uses the `pypinyin` dictionary and converts entries to tone3 pinyin. The helper below mirrors that behavior for single characters:

```bash theme={null}
pip install pypinyin
```

```python theme={null}
from pypinyin.contrib.tone_convert import to_tone3
from pypinyin.pinyin_dict import pinyin_dict


def chinese_char_to_pinyin(char: str) -> str | None:
    pinyin = pinyin_dict.get(ord(char))
    if pinyin is None:
        return None
    if "," in pinyin:
        raise ValueError(f"{char} has multiple readings; choose one manually")
    return to_tone3(pinyin)


print(chinese_char_to_pinyin("工"))
# gong1
```

Phrase-level words can require a phrase dictionary or manual selection. For example, `重` should be `chong2` in `重庆` but `zhong4` in `重要`.

## Practical Tips

* Use one phoneme tag per Chinese character or syllable.
* Keep Chinese punctuation, brackets, and spaces outside the tag.
* Choose readings manually for names and polyphonic characters.
* Use `ma5`-style tone 5 when you need to mark a neutral tone explicitly.


# English Phoneme Control
Source: https://docs.fish.audio/developer-guide/core-features/fine-grained-control/english

Control English pronunciation with CMU Arpabet

## Overview

English phoneme control uses CMU Arpabet, the pronunciation format used by CMUdict.

Wrap the pronunciation for one word in `<|phoneme_start|>` and `<|phoneme_end|>`, and keep surrounding punctuation outside the tag.

```text theme={null}
I am an <|phoneme_start|>EH1 N JH AH0 N IH1 R<|phoneme_end|>.
```

<Note>
  IPA is not supported for English phoneme tags. Convert IPA pronunciations to
  CMU Arpabet before using phoneme control.
</Note>

## CMU Arpabet

CMU Arpabet is written as space-separated uppercase symbols. Vowels can include stress digits:

* `0` for unstressed vowels.
* `1` for primary stress.
* `2` for secondary stress.

For the full symbol inventory, see the CMUdict [`cmudict.symbols`](https://github.com/cmusphinx/cmudict/blob/master/cmudict.symbols) list. You can also look up words on the [CMU Pronouncing Dictionary](http://www.speech.cs.cmu.edu/cgi-bin/cmudict) page.

Example:

```text theme={null}
Standard: I am an engineer.
With phoneme control: I am an <|phoneme_start|>EH1 N JH AH0 N IH1 R<|phoneme_end|>.
```

You can omit stress digits when you only need a rough pronunciation, but CMUdict-style output with stress digits usually gives the model the clearest signal.

## Common Examples

Use phoneme control when spelling alone is ambiguous:

```text theme={null}
The <|phoneme_start|>R IY1 D<|phoneme_end|> endpoint returns the current state.
The book was <|phoneme_start|>R EH1 D<|phoneme_end|> yesterday.
```

```text theme={null}
The <|phoneme_start|>B EY1 S<|phoneme_end|> line is too loud.
The <|phoneme_start|>B AE1 S<|phoneme_end|> swam upstream.
```

```text theme={null}
The <|phoneme_start|>P OW1 L IH0 SH<|phoneme_end|> team joined the call.
Please <|phoneme_start|>P AA1 L IH0 SH<|phoneme_end|> the final mix.
```

Use it for product names, acronyms, and technical terms:

```text theme={null}
Deploy with <|phoneme_start|>K UW2 B ER0 N EH1 T IY0 Z<|phoneme_end|>.
The query uses <|phoneme_start|>EH1 S K Y UW1 EH1 L<|phoneme_end|>.
```

## Generate CMU Arpabet

The training pipeline uses CMUdict-style pronunciations. You can generate the same format with the `cmudict` package:

```bash theme={null}
pip install cmudict
```

```python theme={null}
import cmudict


entries = cmudict.dict()


def cmu_pronunciation(word: str) -> str | None:
    phones = entries.get(word.lower())
    if not phones:
        return None
    return " ".join(phones[0])


print(cmu_pronunciation("engineer"))
# EH1 N JH AH0 N IH1 R
```

CMUdict may contain multiple pronunciations for the same word. Listen to the result and choose the variant that matches your intended accent or context.

## Practical Tips

* Replace only the word whose pronunciation needs control.
* Strip punctuation before dictionary lookup, then place punctuation after the tag.
* Use CMU Arpabet for English phoneme tags.
* For names and brands, write the pronunciation that you want the listener to hear, not necessarily the spelling.


# Japanese Phoneme Control
Source: https://docs.fish.audio/developer-guide/core-features/fine-grained-control/japanese

Control Japanese pronunciation with romaji phonemes and pitch accent markers

## Overview

Japanese phoneme control uses OpenJTalk-style romaji phonemes plus pitch accent information. This is useful for Japanese homographs that have the same plain phoneme sequence but different pitch accents, such as `端が`, `箸が`, and `橋が`.

```text theme={null}
Standard: 橋が見えます。
With phoneme control: <|phoneme_start|>ha0shi1ga0<|phoneme_end|>見えます。
```

Unlike Chinese, Japanese phoneme control is usually applied to a short word or phrase, not one tag per character.

## Format

Put the pitch level digit immediately after each vowel-bearing mora:

* `0` means the current mora is low.
* `1` means the current mora is high.
* `N` can also carry a pitch digit.
* Consonants are written without spaces before the vowel they belong to, for example `ha`, `shi`, and `ga`.
* Use OpenJTalk phoneme symbols such as `a`, `i`, `u`, `e`, `o`, `N`, `cl`, `ky`, `sh`, `ch`, and `ts`.

The following examples all share the plain phoneme sequence `h a sh i g a`, but the pitch markers disambiguate the word:

* `端が` (end + subject marker): `<|phoneme_start|>ha0shi1ga1<|phoneme_end|>`
* `箸が` (chopsticks + subject marker): `<|phoneme_start|>ha1shi0ga0<|phoneme_end|>`
* `橋が` (bridge + subject marker): `<|phoneme_start|>ha0shi1ga0<|phoneme_end|>`

<Note>
  Japanese pitch accent depends on the dictionary, reading, and dialect.
  Generate the phoneme string from the same text you send to TTS, then listen
  and adjust the digits when you need a specific accent.
</Note>

## Relation to ttslearn Prosody Symbols

The [ttslearn Japanese Tacotron recipe](https://r9y9.github.io/ttslearn/latest/notebooks/ch10_Recipe-Tacotron.html#%E3%83%95%E3%83%AB%E3%82%B3%E3%83%B3%E3%83%86%E3%82%AD%E3%82%B9%E3%83%88%E3%83%A9%E3%83%99%E3%83%AB%E3%81%8B%E3%82%89%E3%81%AE%E9%9F%B3%E7%B4%A0%E5%88%97%E3%81%8A%E3%82%88%E3%81%B3%E9%9F%BB%E5%BE%8B%E8%A8%98%E5%8F%B7%E3%81%AE%E6%8A%BD%E5%87%BA) shows how to extract phonemes and prosody symbols from OpenJTalk full-context labels. That recipe prints symbols such as `[` for a pitch rise and `]` for a pitch fall.

Fish Audio phoneme tags should not contain literal `[` or `]`. Convert that prosody into digit notation, such as `ha0shi1ga0`.

## Generate Japanese Phonemes

You can generate Japanese phoneme strings with `pyopenjtalk`. The converter below follows the same full-context label logic used in training:

```bash theme={null}
pip install pyopenjtalk
```

```python theme={null}
import re

import pyopenjtalk


JAPANESE_VOWELS = "aiueoAIUEON"


def japanese_to_romaji_with_accent(sentence: str) -> str:
    text = ""
    labels = pyopenjtalk.extract_fullcontext(sentence)
    level = -1

    for index, label in enumerate(labels):
        phoneme = re.search(r"\-([^\+]*)\+", label).group(1)
        if phoneme in ["sil", "pau"]:
            continue

        text += phoneme

        a1 = int(re.search(r"/A:(\-?[0-9]+)\+", label).group(1))
        a2 = int(re.search(r"\+(\d+)\+", label).group(1))
        a3 = int(re.search(r"\+(\d+)/", label).group(1))

        next_phoneme = re.search(r"\-([^\+]*)\+", labels[index + 1]).group(1)
        if next_phoneme in ["sil", "pau"]:
            a2_next = -1
        else:
            a2_next = int(re.search(r"\+(\d+)\+", labels[index + 1]).group(1))

        # Accent phrase boundary
        if a3 == 1 and a2_next == 1:
            if level >= 0:
                text += str(level)
            level = -1
        # Falling
        elif a1 == 0 and a2_next == a2 + 1:
            level = 0
            text += "1"
        # Rising
        elif a2 == 1 and a2_next == 2:
            level = 1
            text += "0"
        elif phoneme in JAPANESE_VOWELS:
            if level < 0:
                level = 0
            text += str(level)

    return text


print(japanese_to_romaji_with_accent("橋が"))
# ha0shi1ga0
```

Then place the result inside the phoneme tags:

```text theme={null}
<|phoneme_start|>ha0shi1ga0<|phoneme_end|>
```

Minimal request body:

```json theme={null}
{
  "text": "<|phoneme_start|>ha0shi1ga0<|phoneme_end|>見えます。"
}
```

## Processing Longer Text

For long Japanese text, split on punctuation and tag short Japanese runs instead of wrapping an entire paragraph. The training augmentation used short segments and skipped empty or very long spans.

Good:

```text theme={null}
<|phoneme_start|>ha0shi1ga0<|phoneme_end|>、見えます。
```

Avoid:

```text theme={null}
<|phoneme_start|>very long paragraph with multiple clauses...<|phoneme_end|>
```

If your text contains symbols that OpenJTalk should read as words, normalize them before conversion. For example, the training preprocessor converted `％` to `パーセント` before extracting phonemes.


# Pronunciation Dictionaries
Source: https://docs.fish.audio/developer-guide/core-features/fine-grained-control/pronunciation-dictionaries

Create, preview, and publish pronunciation dictionaries in the web app and use them in TTS requests

## Overview

Pronunciation dictionaries are available in the [Fish Audio web app](https://fish.audio/app/pronunciation-dictionary) and the [TTS API](/api-reference/endpoint/openapi-v1/text-to-speech). Use them when the same names, brands, acronyms, or technical terms appear across many requests and you want consistent pronunciation without adding phoneme tags to every script.

Create and publish a dictionary in the web app, then reference its dictionary ID and version ID in your TTS requests. For rules that only apply to one request, send an [inline dictionary](#inline-dictionaries) instead.

<Card title="Open pronunciation dictionaries" icon="book-open" href="https://fish.audio/app/pronunciation-dictionary">
  Create rules, preview how they sound, and publish reusable versions.
</Card>

## When To Use It

Use a pronunciation dictionary for:

* Product names, brand names, people names, or place names that are often misread.
* Acronyms and technical terms such as `SQL`, `Kubernetes`, or internal project names.
* Scripts where the same word list is reused across batches or episodes.
* Homographs when a larger phrase can remove ambiguity, such as `read endpoint` versus `read yesterday`.

Avoid broad entries for common words unless every occurrence should be read the same way. If pronunciation depends on context, use a longer phrase as the key or place a [phoneme tag](/developer-guide/core-features/fine-grained-control#phoneme-control) directly in the text.

## Create and Publish a Dictionary

1. Sign in to [Pronunciation dictionary](https://fish.audio/app/pronunciation-dictionary) and select **New Dictionary**. Enter a name and an optional description, then select **Create Dictionary**.
2. Select **Add a new rule**. Enter the written word or phrase in **Input** and its phoneme pronunciation in **Output**. Turn on **Case sensitive** if capitalization should affect matching.
3. Select a voice and model, then use **Play preview** on a rule to hear its pronunciation. Previewing a rule does not publish your edits.
4. Select **Publish New Version** to save the rules as an immutable version.
5. Open the dictionary's **More actions** menu and copy **Dictionary ID** and **Latest Version ID**. Use these as `id` and `version` in the [TTS request below](#use-a-published-dictionary).

<Note>
  Unpublished edits do not change a published version. After publishing an
  update, change the `version` in your TTS request to use it. Requests pinned to
  an older version keep using that version's rules.
</Note>

Open **Version history** to inspect published versions and compare their rules.

### Import and Export Rules

You can upload an existing `.pls`, `.xml`, or `.txt` file when creating a dictionary, or use **Import PLS** in the editor. Imported pronunciations must use the [phoneme formats below](#write-pronunciation-rules); importing a file does not convert IPA to Fish Audio phonemes.

For a `.txt` file, put one rule on each line, separating the input and output with a comma or tab:

```text theme={null}
Kubernetes,K UW2 B ER0 N EH1 T IY0 Z
SQL,EH1 S K Y UW1 EH1 L
```

Text imports default to case-insensitive matching. Review the rules, preview them, and publish any changes before using the version in TTS. To export a published dictionary as a PLS file, select **Download** from its **More actions** menu.

## Write Pronunciation Rules

Each rule maps the written text (`key`, shown as **Input** in the web app) to a phoneme pronunciation (`value`, shown as **Output**):

```json theme={null}
{
  "key": "Kubernetes",
  "value": "K UW2 B ER0 N EH1 T IY0 Z",
  "case_sensitive": false
}
```

`key` is a literal substring to match in the TTS input, not a regular expression or a whole-word pattern. Use the spelling, spacing, and punctuation you expect in the text. A short key can also match inside a longer word, so include enough context to avoid unintended replacements.

`value` is the phoneme string that should replace the key during synthesis. Do not include `<|phoneme_start|>` or `<|phoneme_end|>`; the synthesis pipeline wraps dictionary values internally.

Use the same phoneme notation as manual phoneme control:

| Language | Value format | Example |
| - | - | - |
| [English](/developer-guide/core-features/fine-grained-control/english) | CMU Arpabet | `K UW2 B ER0 N EH1 T IY0 Z` |
| [Chinese](/developer-guide/core-features/fine-grained-control/chinese) | Tone-number pinyin | `chong2` |
| [Japanese](/developer-guide/core-features/fine-grained-control/japanese) | OpenJTalk-style romaji with pitch digits | `ha0shi1ga0` |

`case_sensitive` defaults to `false`, which ignores ASCII capitalization when matching. Set it to `true` when a rule should only match the exact capitalization you entered. Managed dictionaries reject duplicate keys; when case sensitivity is off, keys that differ only by case count as duplicates.

## Use a Published Dictionary

After publishing a dictionary, pass its dictionary ID and version ID in `pronunciation_dictionary`. Set `FISH_API_KEY` to your [API key](/developer-guide/getting-started/api-key), and replace the placeholder IDs with the values copied from the web app:

```bash theme={null}
curl --request POST https://api.fish.audio/v1/tts \
  --header "Authorization: Bearer $FISH_API_KEY" \
  --header "Content-Type: application/json" \
  --header "model: s2-pro" \
  --data '{
    "text": "Our Kubernetes service reads SQL.",
    "reference_id": "YOUR_VOICE_ID",
    "format": "mp3",
    "pronunciation_dictionary": [
      {
        "id": "YOUR_DICTIONARY_ID",
        "version": "YOUR_VERSION_ID"
      }
    ]
  }' \
  --output speech.mp3
```

`reference_id` selects the voice; it is separate from the dictionary ID. Find a voice ID in the [Voice Library](https://fish.audio/app/discovery).

Always send an explicit `version`. Use the copied version ID, not the display number such as `V1` or the string `latest`. Publishing a new version does not automatically update existing requests.

## Inline Dictionaries

Pass inline dictionaries directly in a TTS request when the rules are request-specific:

```json theme={null}
{
  "text": "Our Kubernetes service reads SQL from DynamoDB.",
  "reference_id": "YOUR_VOICE_ID",
  "pronunciation_dictionary": [
    {
      "items": [
        {
          "key": "Kubernetes",
          "value": "K UW2 B ER0 N EH1 T IY0 Z",
          "case_sensitive": false
        },
        {
          "key": "SQL",
          "value": "EH1 S K Y UW1 EH1 L",
          "case_sensitive": true
        },
        {
          "key": "DynamoDB",
          "value": "D AY1 N AH0 M OW0 D IY1 B IY1",
          "case_sensitive": false
        }
      ]
    }
  ]
}
```

Inline dictionaries are best for temporary overrides, generated scripts, or per-customer terms that you do not need to manage as reusable platform resources.

## Request Limits and Matching

The TTS API accepts either managed references or inline dictionaries in one request. Do not mix both forms in the same `pronunciation_dictionary` array.

Send pronunciation dictionaries with `application/json` or `application/msgpack`. Nested dictionary arrays are not representable in `multipart/form-data`.

| Limit | Maximum |
| - | - |
| Dictionaries per request | 3 |
| Rules per dictionary | 5,000 |
| Characters per `key` | 256 |
| Characters per `value` | 1,024 |

Keys and values must be nonempty and must not contain `<|phoneme_start|>` or `<|phoneme_end|>` markers.

Dictionaries are combined in the order you send them. When the same key appears more than once, the first entry wins. For overlapping matches, the earliest match in the text wins; if matches start at the same position, the longest key wins. Replacements are applied during synthesis, and replacement values are not matched again.

<Note>
  If a dictionary ID or version ID cannot be resolved, the API skips that
  reference and continues synthesis. A successful audio response does not prove
  that the dictionary was applied. Check the IDs and listen to representative
  text when testing a new version.
</Note>

## Practical Tips

* Write `key` as the listener-facing surface form, not as a regex or pattern.
* Write `value` as phonemes only, without marker tokens.
* Use narrow keys for context-sensitive words; use phrase keys when a single word is ambiguous.
* Keep dictionaries scoped by product, customer, language, or domain so broad rules do not surprise unrelated scripts.
* Test a new dictionary version with representative text before using it in production traffic.


# Get Your API Key
Source: https://docs.fish.audio/developer-guide/getting-started/api-key

Create a Fish Audio account, generate an API key, and make your first request

Everything you build with Fish Audio (the API, the Python library, JavaScript) authenticates with a single **API key**. Here's how to get one and make your first call in a couple of minutes.

## 1. Create an account and key

<Steps>
  <Step title="Sign up">
    Go to [fish.audio/auth/signup](https://fish.audio/auth/signup), create an account, and verify your email.
  </Step>

  <Step title="Open the API Keys page">
    Sign in and open [fish.audio/app/api-keys](https://fish.audio/app/api-keys).
  </Step>

  <Step title="Create a key">
    Click **Create New Key**, give it a descriptive name (and an expiration if you want), then **copy the key and store it securely**.

    <Warning>Never commit your API key to version control or share it publicly.</Warning>
  </Step>
</Steps>

## 2. Store it as an environment variable

The SDKs and the examples throughout these docs read your key from `FISH_API_KEY`:

```bash theme={null}
export FISH_API_KEY="your_api_key_here"
```

This keeps the key out of your code and lets you use different keys for development and production.

## 3. Make your first request

<CodeGroup>
  ```python Python theme={null}
  from fishaudio import FishAudio
  from fishaudio.utils import save

  client = FishAudio()  # reads FISH_API_KEY
  audio = client.tts.convert(text="Hello from Fish Audio!")
  save(audio, "hello.mp3")
  ```

  ```bash API (curl) theme={null}
  curl --request POST https://api.fish.audio/v1/tts \
    --header "Authorization: Bearer $FISH_API_KEY" \
    --header "Content-Type: application/json" \
    --header "model: s2-pro" \
    --data '{ "text": "Hello from Fish Audio!", "format": "mp3" }' \
    --output hello.mp3
  ```
</CodeGroup>

You just generated your first audio. Where to next:

<CardGroup>
  <Card title="Text to Speech" icon="microphone" href="/features/text-to-speech">
    Voices, formats, speed, and streaming.
  </Card>

  <Card title="Quick Start" icon="rocket" href="/developer-guide/getting-started/quickstart">
    A fuller first-request walkthrough.
  </Card>

  <Card title="Voice Cloning" icon="clone" href="/features/voice-cloning">
    Build a custom voice from your own audio.
  </Card>

  <Card title="Speech to Text" icon="waveform" href="/features/speech-to-text">
    Transcribe audio with timestamps.
  </Card>
</CardGroup>

<Tip>
  **Building with an AI coding agent?** Install the Fish Audio skill so it writes correct SDK/API code: `npx skills add docs.fish.audio`. See [AI Coding Agents](/developer-guide/resources/coding-agents).
</Tip>


# Changelog
Source: https://docs.fish.audio/developer-guide/getting-started/changelog

Complete release history and version updates for all Fish Audio products

<Update label="Drama 3 Preview" description="September 23, 2026">
  ## Drama 3 Preview

  Preview text-to-speech model you direct in plain language. Describe tone, pacing, and character without a fixed set of audio tags.

  Use model ID `drama-3-preview` in the API. Behavior and availability may change while it is in preview.

  [Text to Speech API](/api-reference/endpoint/openapi-v1/text-to-speech)
</Update>

<Update label="Transcribe 1 Pro" description="September 18, 2026">
  ## Transcribe 1 Pro

  Speech-to-text model for short clips and long recordings, including multi-speaker conversations. Transcripts can include inline speaker markers, speaker turns, and emotion and vocal-event cues.

  Use model ID `transcribe-1-pro` on `POST /v1/asr`

  [Speech to Text](/features/speech-to-text) | [Models overview](/developer-guide/models-pricing/models-overview)
</Update>

<Update label="Fish Audio S2.1 Pro" description="June 23, 2026">
  ## Fish Audio S2.1 Pro

  Production text-to-speech model that improves on S2-Pro in quality, latency, and throughput. Supports 83 languages on one model, with the same `[bracket]` natural-language control and multi-speaker dialogue as S2.

  Use model ID `s2.1-pro` for production workloads that need TTFA and DPA guarantees. `s2.1-pro-free` is the same model at no cost through November 30, 2026 for testing, prototyping, development, and smaller businesses, under fair-use limits and without those guarantees.

  [Read more about S2.1 Pro](https://fish.audio/blog/s2-1-pro-free-api/) | [Models overview](/developer-guide/models-pricing/models-overview)
</Update>

<Update label="Fish Audio S2" description="March 2026">
  ## Fish Audio S2

  Next-generation text-to-speech model with inline emotion cues, multi-speaker dialogue support, and 80+ languages.

  S2 introduces `[bracket]` syntax for natural language control over emotion and paralinguistic cues (e.g., `[whisper]`, `[laugh]`, `[emphasis]`). Tags are treated as standard text rather than dedicated control tokens, so you are not limited to a fixed set of expressions. Built on the Qwen3-4B backbone and fully open-source.

  Use model ID `s2-pro` in the API. S1 remains supported for existing integrations.

  [GitHub](https://github.com/fishaudio/fish-speech) | [HuggingFace](https://huggingface.co/fishaudio)
</Update>

<Update label="Fish Audio S1" description="June 2025">
  ## Fish Audio S1

  Historic rebrand from Fish Speech to Fish Audio. #1 ranking on TTS-Arena2 with industry-leading performance.

  S1 (4B params): 0.008 WER, 0.004 CER - Available on Fish Audio Playground
  S1-mini (0.5B params): 0.011 WER, 0.005 CER - Open source on Hugging Face

  64+ emotional expressions with RLHF integration and multilingual support for English, Chinese, Japanese, and more.

  [Read More about S1](https://fish.audio/blog/introducing-s1/)
</Update>

<Update label="v1.5.1" description="May 27, 2025">
  ## v1.5.1

  Fixed critical PyTorch security settings and improved inference speed significantly. Added ONNX export support for better deployment options and enhanced text processing for Arabic and Hebrew languages. Includes bug fixes for Apple Silicon (MPS) compatibility and reorganized library structure for cleaner codebase.
</Update>

<Update label="v1.5.0" description="December 21, 2024">
  ## v1.5.0

  Introduced v1.5 model architecture with improved dataset handling and bearer token authentication for APIs.

  Added reference audio caching by hash for faster performance and better Apple Silicon support. Includes OpenAPI documentation refactoring and base64 reference data support in JSON format.
</Update>

<Update label="v1.4.3" description="November 23, 2024">
  ## v1.4.3

  Introduced Fish Agent for conversational AI with streaming capabilities and real-time interactions.

  Added comprehensive Korean language documentation and fixed critical non-English speech issues. Improved WebUI streaming functionality and PyTorch version compatibility.
</Update>

<Update label="v1.4.2" description="October 25, 2024">
  ## v1.4.2

  Documentation-focused release with comprehensive updates for v1.4, macOS support, and multiple language translations.

  Improved Docker support and API enhancements for JSON format handling. Added audio selection to WebUI and fixed various stability issues including cache handling and backend performance.
</Update>

<Update label="v1.4.1" description="September 15, 2024">
  ## v1.4.1

  Infrastructure improvements focused on Docker optimization and multi-platform builds.

  Updated PyTorch version and replaced audio backend from sox for better performance. Enhanced CI/CD pipeline with buildx support and fixed various Docker-related issues.
</Update>

<Update label="v1.4.0" description="September 12, 2024">
  ## v1.4.0

  Major release with new VQGAN architecture for improved audio quality and faster inference.

  Updated WebUI with enhanced interface and better language switching. Added Japanese documentation translation and fixed inference warmup issues for better performance.
</Update>

<Update label="v1.2.1" description="September 8, 2024">
  ## v1.2.1

  Replaced Whisper with SenseVoice for better ASR and added native Apple Silicon support.

  Includes Portuguese (Brazil) localization, streaming audio functionality, and CPU-only inference improvements. Pinned PyTorch to 2.3.1 to fix inference speed issues and aligned API with official closed-source version.
</Update>

<Update label="v1.2" description="July 18, 2024">
  ## v1.2

  Introduced auto-reranking system for better results along with bilingual support and model quantization.

  Replaced standard Whisper with Faster Whisper for improved speed and added Japanese documentation. Enhanced model stability and inference performance with optimized v1.2 architecture.
</Update>

<Update label="v1.1.2" description="June 27, 2024">
  ## v1.1.2

  Minor release adding Chinese text normalization support and a streaming audio download button in the WebUI.

  Fixed LoRA merging issues and improved Firefly performance.
</Update>

<Update label="v1.1.1" description="June 8, 2024">
  ## v1.1.1

  Breaking changes: Replaced zibai with uvicorn for API server, new text-splitter with byte-based length calculation, and license change to CC-BY-NC-SA 4.0.

  Added Apple Silicon (MPS) support, Windows one-click installation, and automatic model downloading with resume capability. Improved WebUI with better file selection and download progress indicators.
</Update>

<Update label="v1.1.0" description="May 11, 2024">
  ## v1.1.0

  Added VITS decoder integration with full streaming support and queue management for real-time audio generation.

  Introduced internationalization (i18n) with Spanish translation and improved Windows packaging. Optimized GPU memory usage and CPU-only inference performance while adding LoRA support to the Gradio UI.
</Update>

<Update label="v1.0.0" description="April 30, 2024">
  ## v1.0.0

  Major milestone release introducing new VQ-GAN architecture with VITS decoder support, LoRA fine-tuning, and streaming inference capabilities.

  Breaking changes include removal of the Rust-based data server, new tokenizer replacing phonemizer, and updated model architecture (VQ + DiT + Reflow). Achieved 4x memory reduction during loading and added WebUI for training and annotation.
</Update>

<Update label="v0.2.0" description="December 23, 2023">
  ## v0.2.0

  First public release of Fish Speech featuring a complete text-to-speech pipeline with VQ-GAN audio codec and LLAMA-based language model.

  Includes multi-language support (Chinese, English, Japanese), Gradio WebUI for inference, HTTP API server, and Docker support. Added special optimizations for Chinese users including mirror downloads and localized documentation.
</Update>


# Migrate to Fish Audio
Source: https://docs.fish.audio/developer-guide/getting-started/migration

Run your existing OpenAI, OpenRouter, ElevenLabs, or Groq code on Fish Audio.

| Your SDK | Base URL |
| - | - |
| OpenAI | `https://api.fish.audio/compat/v1` |
| OpenRouter | `https://api.fish.audio/compat/api/v1` |
| ElevenLabs | `https://api.fish.audio/compat/elevenlabs` |
| Groq | `https://api.fish.audio/compat` |

<Tabs>
  <Tab title="OpenAI">
    <Steps>
      <Step title="Set your API key">
        Store your [Fish Audio API key](/developer-guide/getting-started/api-key)
        as an environment variable:

        ```bash theme={null}
        export FISH_AUDIO_API_KEY="your_api_key"
        ```
      </Step>

      <Step title="Swap the base URL and generate">
        <CodeGroup>
          ```python Python theme={null}
          import os
          from openai import OpenAI

          client = OpenAI(
              base_url="https://api.fish.audio/compat/v1",
              api_key=os.environ["FISH_AUDIO_API_KEY"],
          )

          with client.audio.speech.with_streaming_response.create(
              model="fish-audio/s2.1-pro",
              input="Hello from Fish Audio!",
              voice="",  # "" = default voice; see Voices below to pick one
              response_format="mp3",
          ) as response:
              response.stream_to_file("hello.mp3")
          ```

          ```javascript Node theme={null}
          import OpenAI from "openai";
          import fs from "node:fs";

          const client = new OpenAI({
            baseURL: "https://api.fish.audio/compat/v1",
            apiKey: process.env.FISH_AUDIO_API_KEY,
          });

          const response = await client.audio.speech.create({
            model: "fish-audio/s2.1-pro",
            input: "Hello from Fish Audio!",
            voice: "", // "" = default voice; see Voices below to pick one
            response_format: "mp3",
          });
          fs.writeFileSync("hello.mp3", Buffer.from(await response.arrayBuffer()));
          ```

          ```bash curl theme={null}
          curl https://api.fish.audio/compat/v1/audio/speech \
            -H "Authorization: Bearer $FISH_AUDIO_API_KEY" \
            -H "Content-Type: application/json" \
            -d '{"model": "fish-audio/s2.1-pro",
                 "input": "Hello from Fish Audio!",
                 "response_format": "mp3"}' \
            --output hello.mp3
          ```
        </CodeGroup>
      </Step>
    </Steps>

    Full guide: [Migrate from OpenAI](/developer-guide/compat/migrate-from-openai),
    covering transcription, chat audio, the Realtime WebSocket, and what each
    feature supports.
  </Tab>

  <Tab title="OpenRouter">
    <Steps>
      <Step title="Set your API key">
        Store your [Fish Audio API key](/developer-guide/getting-started/api-key)
        as an environment variable:

        ```bash theme={null}
        export FISH_AUDIO_API_KEY="your_api_key"
        ```
      </Step>

      <Step title="Swap the base URL and generate">
        From Python, use the OpenAI SDK against the same `/api/v1` base; that
        is how OpenRouter itself is used from Python.

        <CodeGroup>
          ```typescript Node theme={null}
          import { OpenRouter } from "@openrouter/sdk";
          import fs from "node:fs";

          const client = new OpenRouter({
            apiKey: process.env.FISH_AUDIO_API_KEY,
            serverURL: "https://api.fish.audio/compat/api/v1",
          });

          const stream = await client.tts.createSpeech({
            speechRequest: {
              model: "fish-audio/s2.1-pro",
              input: "Hello from Fish Audio!",
              responseFormat: "mp3",
            },
          });
          const chunks = [];
          for await (const chunk of stream) chunks.push(chunk);
          fs.writeFileSync("hello.mp3", Buffer.concat(chunks));
          ```

          ```python Python theme={null}
          import os
          from openai import OpenAI

          client = OpenAI(
              base_url="https://api.fish.audio/compat/api/v1",
              api_key=os.environ["FISH_AUDIO_API_KEY"],
          )

          with client.audio.speech.with_streaming_response.create(
              model="fish-audio/s2.1-pro",
              input="Hello from Fish Audio!",
              voice="",  # "" = default voice; see Voices below to pick one
              response_format="mp3",
          ) as response:
              response.stream_to_file("hello.mp3")
          ```

          ```bash curl theme={null}
          curl https://api.fish.audio/compat/api/v1/audio/speech \
            -H "Authorization: Bearer $FISH_AUDIO_API_KEY" \
            -H "Content-Type: application/json" \
            -d '{"model": "fish-audio/s2.1-pro",
                 "input": "Hello from Fish Audio!",
                 "response_format": "mp3"}' \
            --output hello.mp3
          ```
        </CodeGroup>
      </Step>
    </Steps>

    Full guide: [Migrate from OpenRouter](/developer-guide/compat/migrate-from-openrouter),
    covering transcription, the model catalog, and what each feature supports.
  </Tab>

  <Tab title="ElevenLabs">
    <Steps>
      <Step title="Set your API key">
        Store your [Fish Audio API key](/developer-guide/getting-started/api-key)
        as an environment variable:

        ```bash theme={null}
        export FISH_AUDIO_API_KEY="your_api_key"
        ```
      </Step>

      <Step title="Swap the base URL and generate">
        The key is sent as the standard `xi-api-key` header:

        <CodeGroup>
          ```python Python theme={null}
          import os
          from elevenlabs import ElevenLabs

          client = ElevenLabs(
              base_url="https://api.fish.audio/compat/elevenlabs",
              api_key=os.environ["FISH_AUDIO_API_KEY"],
          )

          audio = b"".join(client.text_to_speech.convert(
              voice_id="default",  # "default" = default voice; see Voices below
              text="Hello from Fish Audio!",
              model_id="fish-audio/s2.1-pro",
              output_format="mp3_44100_128",
          ))
          open("hello.mp3", "wb").write(audio)
          ```

          ```javascript Node theme={null}
          import { ElevenLabsClient } from "@elevenlabs/elevenlabs-js";
          import fs from "node:fs";

          const client = new ElevenLabsClient({
            baseUrl: "https://api.fish.audio/compat/elevenlabs",
            apiKey: process.env.FISH_AUDIO_API_KEY,
          });

          // "default" = default voice; see Voices below to pick one
          const stream = await client.textToSpeech.convert("default", {
            text: "Hello from Fish Audio!",
            modelId: "fish-audio/s2.1-pro",
            outputFormat: "mp3_44100_128",
          });
          const chunks = [];
          for await (const chunk of stream) chunks.push(chunk);
          fs.writeFileSync("hello.mp3", Buffer.concat(chunks));
          ```

          ```bash curl theme={null}
          curl "https://api.fish.audio/compat/elevenlabs/v1/text-to-speech/default?output_format=mp3_44100_128" \
            -H "xi-api-key: $FISH_AUDIO_API_KEY" \
            -H "Content-Type: application/json" \
            -d '{"text": "Hello from Fish Audio!",
                 "model_id": "fish-audio/s2.1-pro"}' \
            --output hello.mp3
          ```
        </CodeGroup>
      </Step>
    </Steps>

    Full guide: [Migrate from ElevenLabs](/developer-guide/compat/migrate-from-elevenlabs),
    covering speech-to-text, timestamps, realtime streaming, your voice
    library, and what each feature supports.
  </Tab>

  <Tab title="Groq">
    <Steps>
      <Step title="Set your API key">
        Store your [Fish Audio API key](/developer-guide/getting-started/api-key)
        as an environment variable:

        ```bash theme={null}
        export FISH_AUDIO_API_KEY="your_api_key"
        ```
      </Step>

      <Step title="Swap the base URL and generate">
        The Groq SDK appends `/openai/v1` itself, so give it the bare `/compat`
        base:

        <CodeGroup>
          ```python Python theme={null}
          import os
          from groq import Groq

          client = Groq(
              base_url="https://api.fish.audio/compat",
              api_key=os.environ["FISH_AUDIO_API_KEY"],
          )

          resp = client.audio.speech.create(
              model="fish-audio/s2.1-pro",
              input="Hello from Fish Audio!",
              voice="",  # "" = default voice; see Voices below to pick one
              response_format="mp3",
          )
          open("hello.mp3", "wb").write(resp.read())
          ```

          ```javascript Node theme={null}
          import Groq from "groq-sdk";
          import fs from "node:fs";

          const client = new Groq({
            apiKey: process.env.FISH_AUDIO_API_KEY,
            baseURL: "https://api.fish.audio/compat",
          });

          const resp = await client.audio.speech.create({
            model: "fish-audio/s2.1-pro",
            input: "Hello from Fish Audio!",
            voice: "", // "" = default voice; see Voices below to pick one
            response_format: "mp3",
          });
          fs.writeFileSync("hello.mp3", Buffer.from(await resp.arrayBuffer()));
          ```

          ```bash curl theme={null}
          curl https://api.fish.audio/compat/openai/v1/audio/speech \
            -H "Authorization: Bearer $FISH_AUDIO_API_KEY" \
            -H "Content-Type: application/json" \
            -d '{"model": "fish-audio/s2.1-pro",
                 "input": "Hello from Fish Audio!",
                 "response_format": "mp3"}' \
            --output hello.mp3
          ```
        </CodeGroup>
      </Step>
    </Steps>

    Full guide: [Migrate from Groq](/developer-guide/compat/migrate-from-groq),
    covering transcription, the model catalog, and what each feature supports.
  </Tab>
</Tabs>

## From a framework

Frameworks that speak the OpenAI protocol work the same way: point them at
`/compat/v1`:

<CodeGroup>
  ```python LangChain theme={null}
  import base64
  import os
  from langchain_openai import ChatOpenAI

  llm = ChatOpenAI(
      base_url="https://api.fish.audio/compat/v1",
      api_key=os.environ["FISH_AUDIO_API_KEY"],
      model="fish-audio/s2.1-pro",
      model_kwargs={"modalities": ["text", "audio"],
                    "audio": {"voice": "", "format": "mp3"}},
  )

  resp = llm.invoke("Hello from Fish Audio!")
  audio = base64.b64decode(resp.additional_kwargs["audio"]["data"])
  open("hello.mp3", "wb").write(audio)
  ```

  ```javascript Vercel AI SDK theme={null}
  import { experimental_generateSpeech as generateSpeech } from "ai";
  import { createOpenAI } from "@ai-sdk/openai";
  import fs from "node:fs";

  const openai = createOpenAI({
    baseURL: "https://api.fish.audio/compat/v1",
    apiKey: process.env.FISH_AUDIO_API_KEY,
  });

  const { audio } = await generateSpeech({
    model: openai.speech("fish-audio/s2.1-pro"),
    text: "Hello from Fish Audio!",
    voice: "",
    outputFormat: "mp3",
  });
  fs.writeFileSync("hello.mp3", audio.uint8Array);
  ```
</CodeGroup>

Building a voice agent with Pipecat or LiveKit? Fish Audio has
[native integrations](/developer-guide/integrations/pipecat) for those.

## Models

| Model | What it is |
| - | - |
| `fish-audio/s2.1-pro` | Production multilingual TTS with voice cloning (recommended) |
| `fish-audio/s2.1-pro-free` | Free tier of S2.1 Pro, for prototyping (no latency guarantee) |
| `fish-audio/s2-pro` | Expressive TTS for narration and multi-speaker content |
| `fish-audio/s1` | Multilingual TTS with emotion control |
| `fish-audio/transcribe-1` | Multilingual speech-to-text with word-level timestamps |

`transcribe-1` needs no language hint. The `language` field in a transcription
response echoes what you sent: empty when you sent none, never a detection
result (the [ElevenLabs protocol](/developer-guide/compat/migrate-from-elevenlabs#speech-to-text)
echoes `language_code` the same way).

For speaker turns, long recordings, and inline emotion cues, use
`transcribe-1-pro` on the native
[Speech to Text API](/features/speech-to-text).

## Voices

The examples above use the model's default voice. To pick a specific one, pass
a **voice ID**: browse the [Voice Library](/overview/platform) and copy the id
of any voice, or make your own with
[Voice Cloning](/features/voice-cloning) or
[Voice Design](/features/voice-design). Vendor preset names (`nova`, `echo`,
`Rachel`, …) don't exist here.

```python theme={null}
voice="9a9cf47702da476aa4629e2506d4a857"      # OpenAI / OpenRouter / Groq
voice_id="9a9cf47702da476aa4629e2506d4a857"   # ElevenLabs
```

## Going further

Wondering whether an option you rely on is supported? See
[Compatibility](/developer-guide/compat/capabilities).

Starting fresh instead? The [native API](/api-reference/introduction) and
[official SDK](/developer-guide/sdk-guide/quickstart) expose every Fish Audio
capability directly.


# Quick Start
Source: https://docs.fish.audio/developer-guide/getting-started/quickstart

Generate your first AI voice with Fish Audio in under 5 minutes

## Overview

This guide will walk you through generating your first text-to-speech audio with Fish Audio. By the end, you'll have converted text into natural-sounding speech using our API.

<Info>
  **Already using the OpenAI, OpenRouter, ElevenLabs, or Groq SDK?** Swap in a
  Fish Audio base URL and API key and it keeps working.
  [Migrate to Fish Audio](/developer-guide/getting-started/migration) has the
  URL for each SDK.
</Info>

## Prerequisites

<AccordionGroup>
  <Accordion icon="user-plus" title="Create a Fish Audio account">
    Sign up for a free Fish Audio account to get started with our API.

    1. Go to [fish.audio/auth/signup](https://fish.audio/auth/signup)
    2. Fill in your details to create an account, complete steps to verify your account.
    3. Log in to your account and navigate to the [API section](https://fish.audio/app/api-keys)
  </Accordion>

  <Accordion icon="key" title="Get your API key">
    Once you have an account, you'll need an API key to authenticate your requests.

    1. Log in to your [Fish Audio Dashboard](https://fish.audio/app/api-keys/)
    2. Navigate to the API Keys section
    3. Click "Create New Key" and give it a descriptive name, set a expiration if desired
    4. Copy your key and store it securely

    <Warning>Keep your API key secret! Never commit it to version control or share it publicly.</Warning>
  </Accordion>
</AccordionGroup>

## Your First TTS Request

Choose your preferred method to generate speech:

<Tabs>
  <Tab title="cURL">
    <Steps>
      <Step title="Set your API key">
        Store your API key as an environment variable (recommended approach):

        ```bash theme={null}
        export FISH_API_KEY="replace_me"
        ```
      </Step>

      <Step title="Make the TTS request">
        Run this [cURL](https://curl.se/) command to generate your first speech:

        ```bash theme={null}
        curl -X POST https://api.fish.audio/v1/tts \
          -H "Authorization: Bearer $FISH_API_KEY" \
          -H "Content-Type: application/json" \
          -H "model: s2.1-pro-free" \
          -d '{
            "text": "Hello! Welcome to Fish Audio. This is my first AI-generated voice.",
            "format": "mp3"
          }' \
          --output welcome.mp3
        ```
      </Step>

      <Step title="Play your audio">
        The audio has been saved as `welcome.mp3`. You can play it by:

        * Double-clicking the file or opening it in any media player
        * Or using the command line:

        ```bash theme={null}
        # On macOS
        afplay welcome.mp3

        # On Linux
        mpg123 welcome.mp3

        # On Windows
        start welcome.mp3
        ```
      </Step>
    </Steps>
  </Tab>

  <Tab title="Python">
    <Steps>
      <Step title="Install the SDK">
        ```bash theme={null}
        pip install fish-audio-sdk
        ```
      </Step>

      <Step title="Generate speech">
        Create a Python script:

        ```python theme={null}
        from fishaudio import FishAudio
        from fishaudio.utils import save

        # Initialize with your API key
        client = FishAudio()  # reads FISH_API_KEY

        # Generate speech
        audio = client.tts.convert(text="Hello! Welcome to Fish Audio.")
        save(audio, "welcome.mp3")

        print("✓ Audio saved to welcome.mp3")
        ```
      </Step>

      <Step title="Run the script">
        ```bash theme={null}
        python generate_speech.py
        ```
      </Step>

      <Step title="Play your audio">
        The audio has been saved as `welcome.mp3`. You can play it by:

        * Double-clicking the file or opening it in any media player
        * Or using the command line:

        ```bash theme={null}
        # On macOS
        afplay welcome.mp3

        # On Linux
        mpg123 welcome.mp3

        # On Windows
        start welcome.mp3
        ```
      </Step>
    </Steps>
  </Tab>

  <Tab title="JavaScript">
    <Steps>
      <Step title="Install the SDK">
        ```bash theme={null}
        npm install fish-audio
        ```
      </Step>

      <Step title="Generate speech">
        Create a JavaScript script:

        ```javascript theme={null}
        import { FishAudioClient } from "fish-audio";
        import { writeFile } from "fs/promises";

        const fishAudio = new FishAudioClient({ apiKey: process.env.FISH_API_KEY });

        const audio = await fishAudio.textToSpeech.convert({
          text: "Hello, world!",
        });

        const buffer = Buffer.from(await new Response(audio).arrayBuffer());
        await writeFile("welcome.mp3", buffer);

        console.log("✓ Audio saved to welcome.mp3");
        ```
      </Step>

      <Step title="Run the script">
        ```bash theme={null}
        node generate_speech.mjs
        ```
      </Step>

      <Step title="Play your audio">
        The audio has been saved as `welcome.mp3`. You can play it by:

        * Double-clicking the file or opening it in any media player
        * Or using the command line:

        ```bash theme={null}
        # On macOS
        afplay welcome.mp3

        # On Linux
        mpg123 welcome.mp3

        # On Windows
        start welcome.mp3
        ```
      </Step>
    </Steps>
  </Tab>
</Tabs>

## If your first request fails

| Status | Likely cause | Fix |
| - | - | - |
| `401` | Invalid or missing API key | Check `FISH_API_KEY` and your key on [API Keys](https://fish.audio/app/api-keys). |
| `402` | Out of credits | Top up on [Billing](https://fish.audio/app/billing). |
| `400` | Bad `reference_id` / parameters | Verify the voice id; read the error `message`. |
| `429` | Rate limit | Wait and retry with backoff. |

See [Errors](/api-reference/errors) for the full table and retry handling.

## Customizing Your Voice

The examples above use the default voice. To use a different voice, add the `reference_id` parameter with a model ID from [fish.audio](https://fish.audio). You can find the model ID in the URL or use the copy button when viewing any voice.

Choose a voice to try:

<Tabs>
  <Tab title="E-Girl Voice">
    From: [https://fish.audio/m/ca3007f96ae7499ab87d27ea3599956a](https://fish.audio/m/ca3007f96ae7499ab87d27ea3599956a)

    ```bash theme={null}
    export REFERENCE_ID="ca3007f96ae7499ab87d27ea3599956a"
    ```
  </Tab>

  <Tab title="Energetic Male">
    From: [https://fish.audio/m/9a9cf47702da476aa4629e2506d4a857](https://fish.audio/m/9a9cf47702da476aa4629e2506d4a857)

    ```bash theme={null}
    export REFERENCE_ID="9a9cf47702da476aa4629e2506d4a857"
    ```
  </Tab>
</Tabs>

Then generate speech with your chosen voice:

<Tabs>
  <Tab title="cURL">
    ```bash theme={null}
    curl -X POST https://api.fish.audio/v1/tts \
      -H "Authorization: Bearer $FISH_API_KEY" \
      -H "Content-Type: application/json" \
      -H "model: s2-pro" \
      -d '{
        "text": "This is a custom voice from Fish Audio! You can explore hundreds of different voices on the platform, or even create your own.",
        "reference_id": "'"$REFERENCE_ID"'",
        "format": "mp3"
      }' \
      --output custom_voice.mp3
    ```
  </Tab>

  <Tab title="Python">
    ```python theme={null}
    import os
    from fishaudio import FishAudio
    from fishaudio.utils import save

    client = FishAudio()  # reads FISH_API_KEY

    # Generate speech with custom voice
    audio = client.tts.convert(
        text="This is a custom voice from Fish Audio! You can explore hundreds of different voices on the platform, or even create your own.",
        reference_id=os.environ.get("REFERENCE_ID")
    )
    save(audio, "custom_voice.mp3")
    ```
  </Tab>

  <Tab title="JavaScript">
    ```javascript theme={null}
    import { FishAudioClient } from "fish-audio";
    import { writeFile } from "fs/promises";

    const fishAudio = new FishAudioClient({ apiKey: process.env.FISH_API_KEY });

    const audio = await fishAudio.textToSpeech.convert({
      text: "This is a custom voice from Fish Audio! You can explore hundreds of different voices on the platform, or even create your own.",
      reference_id: process.env.REFERENCE_ID,
    });

    const buffer = Buffer.from(await new Response(audio).arrayBuffer());
    await writeFile("custom_voice.mp3", buffer);

    console.log("✓ Audio saved to custom_voice.mp3");
    ```
  </Tab>
</Tabs>

## Support

Need help? Check out these resources:

* [API Reference](/api-reference/introduction) - Complete API documentation
* [Create a Voice Clone](/api-reference/endpoint/model/create-model) - Create a voice clone model
* [Generate Speech](/api-reference/endpoint/openapi-v1/text-to-speech) - Generate realistic speech
* [Real-time Streaming](/features/realtime-streaming) - WebSocket for real-time streaming
* [Discord Community](https://discord.com/invite/dF9Db2Tt3Y) - Get help from the community
* [Support Email](mailto:support@fish.audio) - Contact our support team


# LiveKit
Source: https://docs.fish.audio/developer-guide/integrations/livekit

Build real-time voice AI agents with Fish Audio and LiveKit

[LiveKit Agents](https://github.com/livekit/agents) is an open source framework for building real-time voice and multimodal AI agents. It handles streaming audio pipelines, turn detection, interruptions, and LLM orchestration so you can focus on your agent's behavior.

Fish Audio integrates with LiveKit through the `fishaudio` plugin, providing text-to-speech synthesis with support for both chunked and real-time WebSocket streaming modes.

## Prerequisites

* A [Fish Audio account](https://fish.audio) with an API key
* Python 3.9 or higher

## Installation

Install LiveKit Agents with Fish Audio support:

```bash theme={null}
pip install "livekit-agents[fishaudio]"
```

## Configuration

Set your Fish Audio API key as an environment variable:

```bash theme={null}
export FISH_API_KEY=your_api_key_here
```

## Basic usage

Add Fish Audio TTS to your LiveKit agent:

```python theme={null}
from livekit.plugins.fishaudio import TTS

tts = TTS(
    reference_id="your_voice_model_id",  # Optional: use a specific voice
    model="s1",
    sample_rate=24000,
    latency_mode="balanced"
)
```

### Key parameters

| Parameter | Description |
| - | - |
| `api_key` | Your Fish Audio API key (or use `FISH_API_KEY` env var) |
| `model` | TTS model/backend to use (default: `s1`) |
| `reference_id` | Voice model ID from the [Fish Audio library](https://fish.audio/discover) |
| `output_format` | Audio format: `pcm`, `mp3`, `wav`, or `opus` (default: `pcm`) |
| `sample_rate` | Audio sample rate in Hz (default: `24000`) |
| `num_channels` | Number of audio channels (default: `1`) |
| `base_url` | Custom API endpoint (default: `https://api.fish.audio`) |
| `latency_mode` | `normal` (\~500ms) or `balanced` (\~300ms, default) |

### Streaming modes

The plugin supports two synthesis modes:

```python theme={null}
# Chunked (non-streaming) synthesis
stream = tts.synthesize("Hello, world!")

# Real-time WebSocket streaming
stream = tts.stream()
```

## Resources

* [LiveKit Agents Documentation](https://docs.livekit.io/agents/)
* [LiveKit GitHub](https://github.com/livekit/agents)
* [Fish Audio Plugin Reference](https://docs.livekit.io/reference/python/v1/livekit/plugins/fishaudio/index.html)
* [Fish Audio Voice Library](https://fish.audio/discovery)


# n8n
Source: https://docs.fish.audio/developer-guide/integrations/n8n

Automate workflows with Fish Audio and n8n

[n8n](https://n8n.io/) is a fair-code licensed workflow automation platform. The Fish Audio community node brings text-to-speech, speech-to-text, and voice cloning capabilities to your n8n workflows.

## Installation

Install from n8n community nodes:

1. Go to **Settings** > **Community Nodes**
2. Select **Install**
3. Enter `n8n-nodes-fishaudio`
4. Accept the risks and install

See the [n8n community nodes guide](https://docs.n8n.io/integrations/community-nodes/installation/) for details.

## Configuration

1. Go to **Credentials** > **Add Credential**
2. Search for "Fish Audio API"
3. Enter your API key from [fish.audio/app/api-keys](https://fish.audio/app/api-keys)

## Features

The node supports:

* **Text-to-Speech**: Generate audio from text using any voice model
* **Speech-to-Text**: Transcribe audio files
* **Voice Models**: List, create, and manage custom voices
* **Account**: Check credit balance

The node is also available as an AI tool for use with n8n's AI Agent nodes.

## Resources

* [npm package](https://www.npmjs.com/package/n8n-nodes-fishaudio)
* [GitHub](https://github.com/fishaudio/fish-audio-n8n)
* [n8n Community Nodes](https://docs.n8n.io/integrations/community-nodes/)


# Pipecat
Source: https://docs.fish.audio/developer-guide/integrations/pipecat

Build voice AI agents with Fish Audio and Pipecat

[Pipecat](https://github.com/pipecat-ai/pipecat) is an open source framework for building voice and multimodal conversational AI. It handles the orchestration of audio, AI services, and conversation pipelines so you can focus on what makes your agent unique.

Fish Audio integrates with Pipecat through `FishAudioTTSService`, which provides real-time text-to-speech synthesis using WebSocket streaming for low-latency conversational applications.

## Prerequisites

* A [Fish Audio account](https://fish.audio) with an API key
* Python 3.9 or higher

## Installation

Install Pipecat with Fish Audio support:

```bash theme={null}
pip install "pipecat-ai[fish]"
```

## Configuration

Set your Fish Audio API key as an environment variable:

```bash theme={null}
export FISH_API_KEY=your_api_key_here
```

## Basic usage

Add `FishAudioTTSService` to your Pipecat pipeline:

```python theme={null}
from pipecat.services.fish import FishAudioTTSService

tts = FishAudioTTSService(
    api_key=os.getenv("FISH_API_KEY"),
    reference_id="your_voice_model_id",  # Optional: use a specific voice
    model_id="s1",
    params=FishAudioTTSService.InputParams(
        latency="normal",
        prosody_speed=1.0
    )
)
```

### Key parameters

| Parameter | Description |
| - | - |
| `api_key` | Your Fish Audio API key |
| `reference_id` | Voice model ID from the [Fish Audio library](https://fish.audio/discover) |
| `model_id` | TTS model version (default: `s1`) |
| `output_format` | Audio format: `pcm`, `mp3`, `wav`, or `opus` |

### Prosody controls

Customize speech characteristics with `InputParams`:

```python theme={null}
params=FishAudioTTSService.InputParams(
    latency="balanced",      # "normal" or "balanced"
    prosody_speed=1.2,       # 0.5 to 2.0
    prosody_volume=0,        # Volume adjustment in dB
    normalize=True           # Audio normalization
)
```

## Resources

* [Pipecat Documentation](https://docs.pipecat.ai/server/services/tts/fish)
* [Pipecat GitHub](https://github.com/pipecat-ai/pipecat)
* [Fish Audio Voice Library](https://fish.audio/discovery)


# SLNG
Source: https://docs.fish.audio/developer-guide/integrations/slng

Access Fish Audio's real-time speech models through SLNG, our preferred regional hosting provider

Run Fish Audio's real-time speech models closer to your users with [SLNG](https://slng.ai), our **preferred provider for regional hosting**. Europe and Australia are available now, with more [regions coming soon](#regions). Generate speech through its HTTP API or stream audio in real time over WebSocket.

## Prerequisites

* A SLNG account with an [API key](https://docs.slng.ai/authentication)
* `curl` for the HTTP example
* Python 3.9 or higher with `websockets` 14+ and `msgpack` for the WebSocket example (`pip install 'websockets>=14' msgpack`)

Set your SLNG API key in the environment before running the examples:

```bash theme={null}
export SLNG_API_KEY="YOUR_SLNG_API_KEY"
```

Both HTTP and WebSocket requests authenticate with `Authorization: Bearer <SLNG_API_KEY>`.

## Available models

Choose a model and use its endpoint path with `https://api.slng.ai` for HTTP or `wss://api.slng.ai` for WebSocket. Each model's API references describe its supported request fields and response format.

| Model | Endpoint path | API references |
| - | - | - |
| S2.1 Pro | `/v1/tts/slng/fish/tts:s2.1-pro` | [HTTP POST](https://docs.slng.ai/api-reference/tts/fish-tts-2-1-pro/fish-tts-2-1-pro-http), [WebSocket](https://docs.slng.ai/api-reference/tts/fish-tts-2-1-pro/fish-tts-2-1-pro-ws) |

The examples below use S2.1 Pro in Europe (`eu`). See [Regions](#regions) for availability, region selection, and requests for additional regions.

## Voices

Browse [SLNG's official Fish Audio voice list](https://docs.slng.ai/voices/fish-audio) for available voices and audio samples. Set `reference_id` to the `Voice ID` listed for your chosen voice. The examples use `9a9cf47702da476aa4629e2506d4a857` (Hannah).

For voice cloning with inline reference audio, SLNG accepts `references` containing audio bytes and their transcripts through MessagePack requests.

## REST API

Synthesize a complete utterance with a single request:

```bash theme={null}
curl --fail-with-body --request POST \
  --url "https://api.slng.ai/v1/tts/slng/fish/tts:s2.1-pro" \
  --header "Authorization: Bearer $SLNG_API_KEY" \
  --header "X-World-Part-Override: eu" \
  --header 'Content-Type: application/json' \
  --data '{
    "text": "Hello from Fish Audio on SLNG.",
    "reference_id": "9a9cf47702da476aa4629e2506d4a857",
    "format": "mp3"
  }' \
  --output output.mp3
```

The response contains binary audio in the requested format. Supported formats are `mp3`, `wav`, `pcm`, and `opus`.

## WebSocket streaming

Send text incrementally and receive audio chunks over a persistent connection. SLNG's Fish Audio WebSocket endpoint uses **MessagePack** for both outgoing and incoming messages. MessagePack is a binary serialization format that can carry audio bytes directly.

```python theme={null}
import asyncio
import os

import msgpack
from websockets.asyncio.client import connect


async def tts_stream():
    url = "wss://api.slng.ai/v1/tts/slng/fish/tts:s2.1-pro"
    headers = {
        "Authorization": f"Bearer {os.environ['SLNG_API_KEY']}",
        "X-World-Part-Override": "eu",
    }

    async with connect(url, additional_headers=headers) as ws:
        # Start a session with the voice and output format.
        await ws.send(msgpack.packb({
            "event": "start",
            "request": {
                "text": "",
                "reference_id": "9a9cf47702da476aa4629e2506d4a857",
                "format": "mp3",
            },
        }, use_bin_type=True))

        # Send one or more text frames, then signal the end of input.
        await ws.send(msgpack.packb({
            "event": "text",
            "text": "Hello from Fish Audio on SLNG.",
        }, use_bin_type=True))
        await ws.send(msgpack.packb({"event": "stop"}, use_bin_type=True))

        with open("output.mp3", "wb") as output:
            async for message in ws:
                data = msgpack.unpackb(message, raw=False)

                if data["event"] == "audio":
                    output.write(data["audio"])
                elif data["event"] == "error":
                    raise RuntimeError(data["error"])
                elif data["event"] == "finish":
                    if data["reason"] == "error":
                        raise RuntimeError("SLNG ended synthesis with an error")
                    break


asyncio.run(tts_stream())
```

Use a `flush` event to synthesize buffered text while keeping the session open. The `stop` event ends input and lets the server finish synthesis.

## Regions

Current and upcoming regional availability for Fish Audio through SLNG:

| Region | Region code | Availability |
| - | - | - |
| Europe | `eu` | Available |
| Australia | `au` | Available |
| Brazil | — | Coming soon |
| Singapore | — | Coming soon |
| India | — | Coming soon |
| Saudi Arabia | — | Coming soon |
| United Arab Emirates | — | Coming soon |
| Indonesia | — | Coming soon |
| South Korea | — | Coming soon |

For available regions, set `X-World-Part-Override` to the region code in your HTTP request or WebSocket connection headers. If you omit the header, SLNG selects the region automatically. Region codes for upcoming regions will be added when they become available.

For upcoming region availability or to request a new region, [contact Fish Audio](https://fish.audio/contact-sales/). Include your preferred region, model, and expected usage so our team can discuss availability and timelines.


# Telnyx
Source: https://docs.fish.audio/developer-guide/integrations/telnyx

Use Fish Audio voices for real-time and in-call text-to-speech with Telnyx

[Telnyx](https://telnyx.com) is a carrier-owned global communications platform providing infrastructure for real-time agents: voice AI, SIP trunking, programmable voice, and messaging. Fish Audio is available on Telnyx as a hosted text-to-speech provider: you synthesize Fish Audio voices directly through the Telnyx API and play them in live phone calls via Call Control and TeXML, with no Fish Audio API key required.

## Prerequisites

* A [Telnyx account](https://telnyx.com/sign-up) with an API key
* Python 3.9 or higher with `websockets` 14+ (`pip install websockets`) for the streaming example

## Voices

Telnyx exposes a curated shortlist of voices from the [Fish Audio Voice Library](https://fish.audio/discovery) rather than accepting arbitrary voice IDs. Voices use the format `FishAudio.<Model>.<VoiceId>`, where the model is `s2.1-pro` (latest, default), `s2-pro`, or `s1`:

```text theme={null}
FishAudio.s2.1-pro.933563129e564b19a115bedd57b7406a
```

All curated voices are cross-lingual: any voice can speak any language present in the input text. On S2 models, you can also control delivery with inline emotion tags like `[happy]` or `[whispering]`; see [emotion control](/developer-guide/best-practices/emotion-control) for the full syntax.

See the [Telnyx Fish Audio provider page](https://developers.telnyx.com/docs/voice/tts/providers/fishaudio) for the current voice roster, supported audio formats, and sample rates.

## WebSocket streaming

Stream text in and receive base64-encoded audio chunks in real time:

```python theme={null}
import asyncio
import json
import base64
import websockets

async def tts_stream():
    url = "wss://api.telnyx.com/v2/text-to-speech/speech?voice=FishAudio.s2.1-pro.933563129e564b19a115bedd57b7406a"
    headers = {"Authorization": "Bearer YOUR_TELNYX_API_KEY"}

    async with websockets.connect(url, additional_headers=headers) as ws:
        # 1. Handshake
        await ws.send(json.dumps({
            "text": " ",
            "voice_settings": {"format": "mp3", "sample_rate": 44100}
        }))

        # 2. Send text
        await ws.send(json.dumps({"text": "Hello from Fish Audio on Telnyx."}))

        # 3. Signal end of input
        await ws.send(json.dumps({"text": ""}))

        # 4. Collect audio
        audio_chunks = []
        async for message in ws:
            data = json.loads(message)

            if data.get("error"):
                print(f"Error: {data['error']}")
                break

            if data.get("audio"):
                audio_chunks.append(base64.b64decode(data["audio"]))

            if data.get("isFinal"):
                break

    with open("output.mp3", "wb") as f:
        for chunk in audio_chunks:
            f.write(chunk)

asyncio.run(tts_stream())
```

The handshake requests MP3 output via `voice_settings`; without it, audio is delivered as raw PCM at 24 kHz.

## REST API

Synthesize a complete utterance with a single HTTP request:

```bash theme={null}
curl --request POST \
  --url https://api.telnyx.com/v2/text-to-speech/speech \
  --header 'Authorization: Bearer YOUR_TELNYX_API_KEY' \
  --header 'Content-Type: application/json' \
  --data '{
    "text": "Hello from Fish Audio on Telnyx.",
    "voice": "FishAudio.s2.1-pro.933563129e564b19a115bedd57b7406a",
    "voice_settings": {"format": "mp3", "sample_rate": 44100}
  }' \
  --output output.mp3
```

The response streams back audio bytes. Set `output_type` to `base64_output` or `audio_id` for JSON responses instead.

## Resources

* [Telnyx Fish Audio provider reference](https://developers.telnyx.com/docs/voice/tts/providers/fishaudio)
* [Telnyx WebSocket streaming examples](https://developers.telnyx.com/docs/voice/tts/websocket-streaming/examples)
* [Telnyx REST API examples](https://developers.telnyx.com/docs/voice/tts/rest-api/examples)
* [Fish Audio Voice Library](https://fish.audio/discovery)


# Choosing a Model
Source: https://docs.fish.audio/developer-guide/models-pricing/choosing-a-model

Select the right Fish Audio model for your use case and requirements

We recommend using **Fish Audio S2.1-Pro** for production projects. It improves on S2-Pro quality, latency, and throughput, and is the right choice when you need production TTFA and DPA guarantees.

Use **`s2.1-pro-free`** for testing, prototyping, development, and smaller businesses. It is the same model as S2.1-Pro at \$0, but it does not guarantee TTFA or DPA.

## Support

Need help? Check out these resources:

* [API Reference](/api-reference/introduction) - Complete API documentation
* [Create a Voice Clone](/api-reference/endpoint/model/create-model) - Create a voice clone model
* [Generate Speech](/api-reference/endpoint/openapi-v1/text-to-speech) - Generate realistic speech
* [Real-time Streaming](/features/realtime-streaming) - WebSocket for real-time streaming
* [Discord Community](https://discord.com/invite/dF9Db2Tt3Y) - Get help from the community
* [Support Email](mailto:support@fish.audio) - Contact our support team


# Model Deprecations
Source: https://docs.fish.audio/developer-guide/models-pricing/deprecations

Track deprecated models and migration timelines for Fish Audio services

## Available Models

Currently available models:

* **Fish Audio S2** (Recommended) - Latest generation with best performance
* **Fish Audio S1** - Highly expressive and natural sounding

## Deprecated Models

* **speech-1.6** - Fish Speech v1.6 has been deprecated on February, 28th, 2026
* **speech-1.5** - Fish Speech v1.5 has been deprecated on February, 28th, 2026

<Note>
  We strongly recommend using **Fish Audio S1** for all new projects to access the latest capabilities and performance improvements.
</Note>

## Support

Need help? Check out these resources:

* [API Reference](/api-reference/introduction) - Complete API documentation
* [Create a Voice Clone](/api-reference/endpoint/model/create-model) - Create a voice clone model
* [Generate Speech](/api-reference/endpoint/openapi-v1/text-to-speech) - Generate realistic speech
* [Real-time Streaming](/features/realtime-streaming) - WebSocket for real-time streaming
* [Discord Community](https://discord.com/invite/dF9Db2Tt3Y) - Get help from the community
* [Support Email](mailto:support@fish.audio) - Contact our support team


# Models Overview
Source: https://docs.fish.audio/developer-guide/models-pricing/models-overview

Explore Fish Audio's speech generation and transcription models and their capabilities

## Available Models

Fish Audio offers text-to-speech models for voice generation and ASR models for audio transcription.

### Recommended Model

<Card title="s2.1-pro" icon="star">
  **Fish Audio S2.1-Pro** - Our recommended production TTS model and an improved version of S2-Pro

  * Natural language control with `[bracket]` syntax, not limited to a fixed set (e.g., `[whispers sweetly]`, `[laughing nervously]`)
  * Multi-speaker dialogue support
  * 83 languages
  * Improved quality, latency, and throughput over S2-Pro
  * Production option for workloads that need TTFA and DPA guarantees
</Card>

### Free Development Model

<Card title="s2.1-pro-free" icon="flask">
  **Fish Audio S2.1-Pro Free** - The same model as S2.1-Pro, available at \$0 for development and testing

  * Use the `s2.1-pro-free` model string with the same TTS API endpoint
  * Same model quality and language coverage as `s2.1-pro`
  * Free to use under fair-use limits
  * No TTFA or DPA guarantees
  * Best for testing, prototyping, development, and smaller businesses
</Card>

### Previous S2 Model

<Card title="s2-pro" icon="microchip">
  **Fish Audio S2-Pro** - Previous-generation S2 TTS model

  * Natural language control with `[bracket]` syntax, not limited to a fixed set (e.g., `[whispers sweetly]`, `[laughing nervously]`)
  * Multi-speaker dialogue support
  * 80+ languages
  * 100ms time-to-first-audio
  * Full SGLang-based serving stack
  * Open-source
</Card>

<Note>
  We recommend using `s2.1-pro` for production projects. Use `s2.1-pro-free` when you want the same model for evaluation, prototyping, development, and smaller businesses without TTFA or DPA guarantees. S1 remains available for existing integrations.
</Note>

### Previous Model

<Card title="s1" icon="microchip">
  **Fish Audio S1** - High-quality voice generation

  * 4 billion parameters
  * 0.008 WER (0.8% word error rate)
  * Full emotional control capabilities with `(parenthesis)` syntax
</Card>

## Speech-to-Text Models

We recommend `transcribe-1-pro` for speech to text. Both ASR models use `POST /v1/asr`, and you select one with the `model` HTTP header, written exactly as shown. Send `model: transcribe-1-pro` with every request: if the header is missing or its value is not an exact match, the request is served and billed as `transcribe-1`, and no error is returned.

| Model | Capabilities |
| - | - |
| `transcribe-1-pro` (recommended) | Anything from short clips to long recordings (up to 60 minutes), including multi-speaker conversations. Speaker markers in `text`, structured `speaker_turns` with start and end times, speaker-count hints, and inline emotion and vocal-event cues such as `[高兴]` (happy) and `[laughter]` (optional via `tag_audio_events`). Supports optional timestamps. |
| `transcribe-1` | General transcription of short recordings, with optional timestamps. Serves requests whose `model` header is missing or not an exact match. |

Pro labels speakers with inline `<|speaker:N|>` markers in `text`, alongside emotion cues. A label identifies the same speaker within one response only. When you request timestamps (`ignore_timestamps=false`), Pro also returns `speaker_turns`, a structured list of turns with each turn's speaker, text, and start and end times, unless you send `diarize=false`. Timestamped `segments` are word-level, contain spoken words without these markers, and do not expose per-segment speaker IDs. See the [multi-speaker response example](/features/speech-to-text#multi-speaker-conversations) and [Speaker turns](/features/speech-to-text#speaker-turns).

Pro has its own optional request fields: `tag_audio_events`, `diarize`, and the speaker-count hints `num_speakers`, `min_speakers`, and `max_speakers`. Speaker-count hints are applied on a best-effort basis and may have no effect on short recordings. For supported audio formats, file size, and length, see [Input audio](/features/speech-to-text#input-audio) and [Limits](/features/speech-to-text#limits). For costs, see [ASR pricing](/developer-guide/models-pricing/pricing-and-rate-limits#automatic-speech-recognition-asr-models).

<Note>
  **If you use `transcribe-1`:** it is designed for short recordings; for recordings longer than a few minutes, use `transcribe-1-pro`. Speaker markers, speaker turns, and emotion and vocal-event cues are `transcribe-1-pro` features, so send the Pro request fields above only with `model: transcribe-1-pro`.
</Note>

## Model Specifications

### Fish Audio S1 Performance Metrics

* **Word Error Rate (WER)**: 0.008 (0.8%)
* **Character Error Rate (CER)**: 0.004 (0.4%)
* **Real-time Factor**: \~1:7 on standard hardware
* **TTS-Arena2 Ranking**: #1 worldwide

## Supported Languages

### S2.1-Pro and S2-Pro

S2.1-Pro supports 83 languages, while S2-Pro supports 80+ languages. Both use automatic language detection and support inline emotion and paralinguistic cues.

<Info>
  Language detection is automatic - simply provide text in your target language.
</Info>

### S1

S1 supports text-to-speech generation in 13 languages with full emotional expression capabilities.

```
English, Chinese, Japanese, German,
French, Spanish, Korean, Arabic,
Russian, Dutch, Italian, Polish, Portuguese
```

## Voice Styles and Emotions

Fish Audio models support emotional expressions and voice styles that can be controlled through text markers in your input.

### S2.1-Pro and S2-Pro Natural Language Control

S2.1-Pro and S2-Pro treat `[bracket]` tags as standard text rather than dedicated control tokens. Through training on massive datasets, the models learned implicit mappings between natural language descriptions and acoustic variations. You can use any descriptive expression and the model will interpret it, such as `[whispers sweetly]` or `[laughing nervously]`.

Common examples include:

```
[whisper] [laugh] [emphasis] [sigh] [gasp] [pause]
[angry] [excited] [sad] [surprised] [inhale] [exhale]
```

<Tip>
  S2 cues can be placed anywhere in your text to control emotion at specific positions. For example: `"I can't believe it [gasp] you actually did it [laugh]"`
</Tip>

### S1 Voice Styles and Emotions

S1 supports 64+ emotional expressions using `(parenthesis)` syntax.

### Basic Emotions (24 expressions)

```
(angry) (sad) (excited) (surprised) (satisfied) (delighted)
(scared) (worried) (upset) (nervous) (frustrated) (depressed)
(empathetic) (embarrassed) (disgusted) (moved) (proud) (relaxed)
(grateful) (confident) (interested) (curious) (confused) (joyful)
```

### Advanced Emotions (25 expressions)

```
(disdainful) (unhappy) (anxious) (hysterical) (indifferent)
(impatient) (guilty) (scornful) (panicked) (furious) (reluctant)
(keen) (disapproving) (negative) (denying) (astonished) (serious)
(sarcastic) (conciliative) (comforting) (sincere) (sneering)
(hesitating) (yielding) (painful) (awkward) (amused)
```

### Tone Markers (5 expressions)

```
(in a hurry tone) (shouting) (screaming) (whispering) (soft tone)
```

### Audio Effects (10 expressions)

```
(laughing) (chuckling) (sobbing) (crying loudly) (sighing)
(panting) (groaning) (crowd laughing) (background laughter) (audience laughing)
```

<Tip>
  You can also use natural expressions like "Ha,ha,ha" for laughter. Experiment with combinations to achieve the perfect emotional tone for your application.
</Tip>

## Support

Need help? Check out these resources:

* [API Reference](/api-reference/introduction) - Complete API documentation
* [Create a Voice Clone](/api-reference/endpoint/model/create-model) - Create a voice clone model
* [Generate Speech](/api-reference/endpoint/openapi-v1/text-to-speech) - Generate realistic speech
* [Real-time Streaming](/features/realtime-streaming) - WebSocket for real-time streaming
* [Discord Community](https://discord.com/invite/dF9Db2Tt3Y) - Get help from the community
* [Support Email](mailto:support@fish.audio) - Contact our support team


# Pricing & Rate Limits
Source: https://docs.fish.audio/developer-guide/models-pricing/pricing-and-rate-limits

Understand Fish Audio pricing plans, usage costs, and API rate limits

## API Pricing

The Fish Audio API uses pay-as-you-go pricing based on actual usage. There are no subscription fees or monthly minimums for API access.

### Text-to-Speech (TTS) Models

TTS pricing is based on the size of input text, measured in millions of UTF-8 bytes.

| Model Name | Price (USD) |
| - | - |
| `s2.1-pro` | \$15.00 / M UTF-8 bytes |
| `s2.1-pro-free` | \$0.00 / M UTF-8 bytes |
| `s2-pro` | \$15.00 / M UTF-8 bytes |
| `s1` | \$15.00 / M UTF-8 bytes |

<Info>
  1M UTF-8 bytes is approximately 180,000 English words, or about 12 hours of
  speech
</Info>

### Automatic Speech Recognition (ASR) Models

| Model Name | Price (USD) |
| - | - |
| `transcribe-1-pro` (recommended) | \$0.36 / audio hour |
| `transcribe-1` | \$0.36 / audio hour |

We recommend `transcribe-1-pro`, which handles multi-speaker conversations (speaker markers and speaker turns), long recordings, and inline emotion cues. Both models cost the same and use the [Speech to Text API](/api-reference/endpoint/openapi-v1/speech-to-text). Send the `model: transcribe-1-pro` HTTP header to select Pro; a request whose `model` header is missing or not an exact match is served and billed as `transcribe-1`. See the [model comparison](/features/speech-to-text#choose-an-asr-model).

**How ASR billing works:**

* Charges are based on the duration of audio processed
* Duration is rounded up to the nearest second
* The full duration of the recording is billed, including silence, once per successful request
* Requests that return an error response are not billed

### Voice Design

| Model Name | Price (USD) |
| - | - |
| `voice-design-1` | \$0.01 / successful API request |

**How Voice Design billing works:**

* Charges are based on successful `POST /v1/voice-design` requests
* One successful request is charged once, even when it returns multiple candidates
* Authentication, validation, balance, concurrency, and service errors are not billed

## Rate Limits

These limits help us ensure fair usage and maintain service quality for all users.

### Concurrent Request Limits

| Tier | Spending Threshold | Concurrent Requests |
| - | - | - |
| Starter | \< \$100 paid | 5 requests |
| Elevated | ≥ \$100 paid | 15 requests |
| High Volume | ≥ \$1,000 paid | 50 requests |
| Enterprise | Custom | Custom limits |

<Tip>
  Concurrency tiers unlock as soon as your total prepaid amount reaches the
  threshold. You do not need to spend the full balance first. If your workload
  needs a higher concurrency tier, you can top up in advance to unlock the next
  tier immediately.
</Tip>

Speech-to-text requests count against the same concurrency limit as your other
API requests, shared by all API keys on your account. Each `/v1/asr` request
holds one slot until its response is returned, which for long
`transcribe-1-pro` recordings can be several minutes.

When you are at the limit, the [native API](/api-reference/introduction),
including `POST /v1/asr`, returns 429 without a `Retry-After` header. Retry
with exponential backoff.

### Convert Concurrency to QPS or QPM

Fish Audio rate limits are based on **concurrent requests**: the number of
requests that can be in progress at the same time. This is different from QPS
(queries per second) or QPM (queries per minute), which measure how many
requests complete over time.

The conversion depends on how long your Fish Audio request occupies a
concurrency slot. Short interactive requests can produce much higher QPM from
the same concurrency tier than long-form synthesis requests.

Use this planning formula:

```text theme={null}
QPS ~= concurrency / average_request_duration_seconds
QPM ~= concurrency * 60 / average_request_duration_seconds
required_concurrency ~= target_QPM * average_request_duration_seconds / 60
```

Before you estimate throughput, measure your own average request duration:

1. For TTS workloads, send representative production-shaped requests with
   `latency="normal"` to get a stable quality-first baseline. For other APIs,
   use the same request shape you expect in production.
2. Measure from request start until the full response completes. For WebSocket
   streaming, measure until the final audio chunk is received or the stream is
   closed.
3. Calculate the average duration for each workload type. If one business
   workflow makes multiple Fish Audio calls, estimate each call separately. Use
   p95 duration when you need safer capacity planning for bursts, retries, or
   uneven traffic.

<Info>
  There is no single fixed conversion from concurrency to QPM. A customer whose
  average request occupies a slot for 2 seconds can complete about 30 times as
  many requests per minute as a customer whose average request occupies a slot
  for 60 seconds, even on the same concurrency tier.
</Info>

Example planning estimates:

| Workload | Average occupied time | 5 concurrency | 15 concurrency | 50 concurrency |
| - | - | - | - | - |
| AI companion short reply | 2 seconds | \~150 QPM | \~450 QPM | \~1,500 QPM |
| Voice-agent support turn | 4 seconds | \~75 QPM | \~225 QPM | \~750 QPM |
| Narration paragraph | 12 seconds | \~25 QPM | \~75 QPM | \~250 QPM |
| Audiobook or long-form synthesis | 60 seconds | \~5 QPM | \~15 QPM | \~50 QPM |

Treat these numbers as examples, not guarantees. Your actual QPS and QPM depend
on text length, model, reference audio, output format, network path, streaming
behavior, and the latency distribution of your own application.

<Note>
  Please reach out to our team to enable enterprise volume pricing, rate limits,
  and billing.
</Note>

## Support

Need help? Check out these resources:

* [API Reference](/api-reference/introduction) - Complete API documentation
* [Create a Voice Clone](/api-reference/endpoint/model/create-model) - Create a voice clone model
* [Generate Speech](/api-reference/endpoint/openapi-v1/text-to-speech) - Generate realistic speech
* [Real-time Streaming](/features/realtime-streaming) - WebSocket for real-time streaming
* [Discord Community](https://discord.com/invite/dF9Db2Tt3Y) - Get help from the community
* [Support Email](mailto:support@fish.audio) - Contact our support team


# Coding Assistant Quickstart
Source: https://docs.fish.audio/developer-guide/resources/agent-quickstart

Build with Fish Audio using your AI coding assistant: install the skill and start prompting in a minute

Install the Fish Audio **agent skill** and your coding agent (Claude Code, Cursor, Codex, and others) writes correct, current Fish Audio code: right method names, units, and error types, instead of guessing. Here's the fastest path.

<Steps>
  <Step title="Install the skill">
    ```bash theme={null}
    npx skills add https://docs.fish.audio
    ```

    This installs both Fish Audio skills (a canonical copy in `.agents/skills/`, with symlinks for Claude Code and Cursor). Run `npx skills update` later to refresh them.

    <CardGroup>
      <Card title="fish-audio-sdk" icon="cube">
        Python (`fish-audio-sdk`) and JavaScript (`fish-audio`): exact method signatures, sync + async, model selection, and the real exception types.
      </Card>

      <Card title="fish-audio-api" icon="code">
        Raw REST + WebSocket for any language: auth, endpoints, MessagePack/JSON/multipart rules, and the streaming protocol.
      </Card>
    </CardGroup>
  </Step>

  <Step title="Set your API key">
    [Create a key](/developer-guide/getting-started/api-key) and export it; the code your agent writes reads it from the environment:

    ```bash theme={null}
    export FISH_API_KEY="your_api_key_here"
    ```
  </Step>

  <Step title="Ask your agent">
    Prompt in plain language; it uses the correct client, methods, and error types:

    <CardGroup>
      <Card title="TTS in a cloned voice" icon="microphone">
        "Generate speech with Fish Audio in a cloned voice and save it to a file."
      </Card>

      <Card title="Transcribe with timestamps" icon="waveform">
        "Transcribe `speech.wav` with Fish Audio and print the segments."
      </Card>

      <Card title="Stream from an LLM" icon="bolt">
        "Stream an LLM's tokens to Fish Audio TTS over the WebSocket."
      </Card>

      <Card title="Raw API, any language" icon="code">
        "Call the Fish Audio TTS REST API from Go, no SDK."
      </Card>
    </CardGroup>
  </Step>
</Steps>

## Install options

<CodeGroup>
  ```bash All skills theme={null}
  npx skills add https://docs.fish.audio
  ```

  ```bash One skill theme={null}
  npx skills add https://docs.fish.audio --skill fish-audio-sdk
  ```

  ```bash Target an agent theme={null}
  # claude-code, cursor, codex, ...
  npx skills add https://docs.fish.audio -a claude-code
  ```

  ```bash List / inspect first theme={null}
  npx skills add https://docs.fish.audio --list
  ```
</CodeGroup>

<Card title="AI Coding Assistants: full guide" icon="laptop-code" href="/developer-guide/resources/coding-agents">
  Targeting specific agents, the Fish Audio MCP server, and reading the skills
  before you install.
</Card>

## Next steps

<CardGroup>
  <Card title="Get your API key" icon="key" href="/developer-guide/getting-started/api-key">
    Create a key and make your first request.
  </Card>

  <Card title="Quick Start" icon="rocket" href="/developer-guide/getting-started/quickstart">
    Generate your first audio by hand, in any language.
  </Card>

  <Card title="Text to Speech" icon="microphone" href="/features/text-to-speech">
    Voices, formats, streaming, and the direct API.
  </Card>

  <Card title="API reference" icon="book-open" href="/api-reference/introduction">
    Endpoints, parameters, and the OpenAPI schema.
  </Card>
</CardGroup>

## For autonomous agents & RAG pipelines

Not a coding agent installing a skill, but an autonomous agent, RAG pipeline, or crawler? Start from these low-noise, machine-readable entry points:

* [llms.txt](https://docs.fish.audio/llms.txt): curated documentation index (read this first)
* [llms-full.txt](https://docs.fish.audio/llms-full.txt): broader context across the whole site
* [OpenAPI](https://docs.fish.audio/api-reference/openapi.json): REST schemas, parameters, and examples
* [AsyncAPI](https://docs.fish.audio/api-reference/asyncapi.yml): the WebSocket streaming protocol

<AccordionGroup>
  <Accordion title="Canonical API facts">
    - Base API URL: `https://api.fish.audio`
    - Authentication: `Authorization: Bearer <FISH_API_KEY>`
    - TTS model selection: optional `model` header (`s1`, `s2-pro`, `s2.1-pro`, `s2.1-pro-free`). Recommended: `s2.1-pro` for production, `s2.1-pro-free` for the free developer tier. If omitted, the server uses `s2.1-pro`
    - ASR model selection: `model` header on `POST /v1/asr` (`transcribe-1-pro`, `transcribe-1`). Recommended: `transcribe-1-pro` (speaker turns, long recordings, emotion cues); send it explicitly. If the header is missing or not an exact match, the request is served and billed as `transcribe-1`. The `/v1/asr` entry in `openapi.json` does not yet list the `transcribe-1-pro` fields, `request_id`, `speaker_turns`, or the error `code`; for those, follow the [Speech to Text API reference](https://docs.fish.audio/api-reference/endpoint/openapi-v1/speech-to-text.md)
    - Main REST endpoints:
      * `POST /v1/tts`
      * `POST /v1/asr`
      * `GET /model`
      * `POST /model`
      * `GET /model/{id}`
      * `PATCH /model/{id}`
      * `DELETE /model/{id}`
    - Real-time streaming endpoint: `wss://api.fish.audio/v1/tts/live`
  </Accordion>

  <Accordion title="Retrieval order">
    1. Read [llms.txt](https://docs.fish.audio/llms.txt) for the curated
       documentation index. 2. Read
       [llms-full.txt](https://docs.fish.audio/llms-full.txt) when broad site context
       is needed. 3. Read
       [OpenAPI](https://docs.fish.audio/api-reference/openapi.json) for REST
       schemas, parameters, and examples. 4. Read
       [AsyncAPI](https://docs.fish.audio/api-reference/asyncapi.yml) for the
       WebSocket streaming protocol. 5. Fetch individual `.md` pages only after
       narrowing to a specific task.
  </Accordion>

  <Accordion title="High-value URLs by task">
    **API specs**

    * [OpenAPI](https://docs.fish.audio/api-reference/openapi.json)
    * [AsyncAPI](https://docs.fish.audio/api-reference/asyncapi.yml)
    * [API Introduction](https://docs.fish.audio/api-reference/introduction.md)

    **Auth & SDK setup**

    * [Python Authentication](https://docs.fish.audio/developer-guide/sdk-guide/python/authentication.md)
    * [JavaScript Authentication](https://docs.fish.audio/developer-guide/getting-started/api-key.md)
    * [Python SDK Overview](https://docs.fish.audio/api-reference/sdk/python/overview.md)
    * [JavaScript SDK Reference](https://docs.fish.audio/api-reference/sdk/javascript/api-reference.md)

    **Core product tasks**

    * [Text to Speech Guide](https://docs.fish.audio/features/text-to-speech.md)
    * [Speech to Text Guide](https://docs.fish.audio/features/speech-to-text.md)
    * [Creating Voice Models](https://docs.fish.audio/features/voice-cloning.md)
    * [Emotion Control](https://docs.fish.audio/developer-guide/core-features/emotions.md)
    * [Fine-grained Control](https://docs.fish.audio/developer-guide/core-features/fine-grained-control.md)

    **Real-time & integrations**

    * [WebSocket TTS Streaming](https://docs.fish.audio/api-reference/endpoint/websocket/tts-live.md)
    * [Real-time Streaming Best Practices](https://docs.fish.audio/developer-guide/best-practices/real-time-streaming.md)
    * [Realtime Streaming (SDK)](https://docs.fish.audio/features/realtime-streaming.md)
    * [LiveKit Integration](https://docs.fish.audio/developer-guide/integrations/livekit.md)
    * [Pipecat Integration](https://docs.fish.audio/developer-guide/integrations/pipecat.md)

    **Models, pricing & lifecycle**

    * [Models Overview](https://docs.fish.audio/developer-guide/models-pricing/models-overview.md)
    * [Choosing a Model](https://docs.fish.audio/developer-guide/models-pricing/choosing-a-model.md)
    * [Pricing & Rate Limits](https://docs.fish.audio/developer-guide/models-pricing/pricing-and-rate-limits.md)
    * [Model Deprecations](https://docs.fish.audio/developer-guide/models-pricing/deprecations.md)
  </Accordion>

  <Accordion title="Task routing">
    * **Generate speech** → Quick Start, the Text to Speech guide, and `POST
        /v1/tts`. - **Transcribe audio** → the Speech to Text guide and `POST /v1/asr`
      with `model: transcribe-1-pro`. - **Clone or manage voices** → Creating Voice
      Models and the `/model` endpoints. - **Stream audio in real time** → AsyncAPI,
      WebSocket TTS Streaming, and the realtime guides. - **Pick a model or estimate
      cost** → Models Overview and Pricing & Rate Limits.
  </Accordion>

  <Accordion title="Notes for agents">
    * Prefer `openapi.json` and `asyncapi.yml` for machine-readable schemas, except for the `/v1/asr` gaps listed under Canonical API facts.
    * Append `.md` to any page URL to fetch the human-authored page as plain Markdown.
    * Some richer pages use interactive MDX widgets. If a fetched page contains UI or component noise, fall back to `llms.txt`, `llms-full.txt`, or the API spec files.
  </Accordion>
</AccordionGroup>


# Brand Guidelines
Source: https://docs.fish.audio/developer-guide/resources/brand

Design guidelines for using Fish Audio brand assets

## Logo

### Wordmark

Our preferred logo format combines the [Fish Audio Icon](#icon) with the wordmark side by side.
This is the primary version of our logo and should be used whenever possible for maximum brand recognition and clarity.

<Frame>
  <img alt="Fish Audio Clearspace Wordmark" />

  <img alt="Fish Audio Clearspace Wordmark" />
</Frame>

### Icon

Our icon features a whale composed of audio bars and sound waves, symbolizing the fusion of marine life with audio technology. This design represents our brand's commitment to natural, flowing, and powerful voice generation.

The Fish Audio icon should only be used when space constraints or context make it impractical to display the full wordmark. Always prefer the wordmark with icon combination when possible.

<Frame>
  <img alt="Fish Audio Clearspace Logo" />

  <img alt="Fish Audio Clearspace Logo" />
</Frame>

### Avoid

To maintain the integrity of our brand identity, please do not alter our logo in any of the following ways:

<Columns>
  <Frame>
    <img alt="Incorrect logo usage - distorted" />

    <img alt="Incorrect logo usage - distorted" />
  </Frame>

  <Frame>
    <img alt="Incorrect logo usage - rotated" />

    <img alt="Incorrect logo usage - rotated" />
  </Frame>
</Columns>

<Columns>
  <Frame>
    <img alt="Incorrect logo usage - wrong colors" />

    <img alt="Incorrect logo usage - wrong colors" />
  </Frame>

  <Frame>
    <img alt="Incorrect logo usage - effects" />

    <img alt="Incorrect logo usage - effects" />
  </Frame>
</Columns>

## Colors

Our official brand colors consist of black and white for primary logo applications, complemented by secondary grays for subtle variations and an accent coral for visual highlights in marketing materials.

<Frame>
  <div>
    <ColorSwatch name="Black (Primary)" />

    <ColorSwatch name="White (Primary)" />

    <ColorSwatch name="Light Gray" />

    <ColorSwatch name="Dark Gray" />

    <ColorSwatch name="Coral (Accent)" />
  </div>
</Frame>

## Typography

Our brand uses **Onest Semibold** in the logo wordmark. This documentation is also set in Onest, so you're experiencing our brand typography right now.

[Download Onest on Google Fonts](https://fonts.google.com/specimen/Onest)

## Usage Guidelines

The Fish Audio name and logos are trademarks of Hanabi AI Inc. You may freely use and redistribute our brand assets when referencing Fish Audio. By using our brand assets, you agree that we own them and that any goodwill generated by your use benefits Fish Audio.

### Do

* Use our brand assets freely in your projects, applications, and content
* Share our brand assets in blog posts, tutorials, documentation, and educational materials
* Follow the visual guidelines shown above (spacing, colors, sizing)
* Link to fish.audio when using our brand online

### Don't

* Use our logo as part of your own product name or branding
* Imply partnership, sponsorship, or endorsement without permission
* Feature our logo more prominently than your own brand

### Questions?

If you're unsure whether your use case is appropriate or need special permission, please contact us at [support@fish.audio](mailto:support@fish.audio).

## Download Assets

<DownloadAssets />


# AI Coding Assistants
Source: https://docs.fish.audio/developer-guide/resources/coding-agents

Install the Fish Audio skill so your coding agent writes correct SDK and API code

Install the Fish Audio **agent skill**, and your coding agent (Claude Code, Cursor, Codex, and others) writes correct, current Fish Audio code: right method names, units, and error types, instead of guessing.

## Install the skill

```bash theme={null}
npx skills add https://docs.fish.audio
```

This installs both Fish Audio skills into your agent (a canonical copy in `.agents/skills/`, with symlinks for Claude Code and Cursor). Run `npx skills update` later to refresh them.

<CardGroup>
  <Card title="fish-audio-sdk" icon="cube">
    Python (`fish-audio-sdk`) and JavaScript (`fish-audio`): exact method signatures and defaults, sync + async, model selection, and the real exception types.
  </Card>

  <Card title="fish-audio-api" icon="code">
    Raw REST + WebSocket for any language or edge runtime: auth, endpoints, MessagePack/JSON/multipart rules, and the streaming protocol.
  </Card>
</CardGroup>

### Install options

<CodeGroup>
  ```bash All skills theme={null}
  npx skills add https://docs.fish.audio
  ```

  ```bash One skill theme={null}
  npx skills add https://docs.fish.audio --skill fish-audio-sdk
  ```

  ```bash Target an agent theme={null}
  # claude-code, cursor, codex, ...
  npx skills add https://docs.fish.audio -a claude-code
  ```

  ```bash List / inspect first theme={null}
  npx skills add https://docs.fish.audio --list
  ```
</CodeGroup>

Want to read them before installing? The skills are served at [/.well-known/agent-skills/index.json](https://docs.fish.audio/.well-known/agent-skills/index.json), with each skill's markdown at `/.well-known/agent-skills/<name>/SKILL.md`.

## Try it

Once installed, ask your agent in plain language; it will use the correct client, methods, and error types:

<CardGroup>
  <Card title="TTS in a cloned voice" icon="microphone">
    "Generate speech with Fish Audio in a cloned voice and save it to a file."
  </Card>

  <Card title="Transcribe with timestamps" icon="waveform">
    "Transcribe `speech.wav` with Fish Audio and print the segments."
  </Card>

  <Card title="Stream from an LLM" icon="bolt">
    "Stream an LLM's tokens to Fish Audio TTS over the WebSocket."
  </Card>

  <Card title="Raw API, any language" icon="code">
    "Call the Fish Audio TTS REST API from Go, no SDK."
  </Card>
</CardGroup>

## Or connect via MCP

The [Fish Audio MCP server](/overview/mcp) at `https://api.fish.audio/mcp` gives your agent tools instead of docs: it can search the voice library, generate speech, and transcribe audio with your Fish Audio account (OAuth sign-in, no API key). See the [MCP server guide](/overview/mcp) for client setup and the tool list.

<Note>
  This site also exposes [llms.txt](https://docs.fish.audio/llms.txt) and [llms-full.txt](https://docs.fish.audio/llms-full.txt) for agents that fetch docs directly.
</Note>

## Next steps

<CardGroup>
  <Card title="Get your API key" icon="key" href="/developer-guide/getting-started/api-key">
    Create a key and make your first request.
  </Card>

  <Card title="Text to Speech" icon="microphone" href="/features/text-to-speech">
    Voices, formats, streaming, and the direct API.
  </Card>

  <Card title="API reference" icon="book-open" href="/api-reference/introduction">
    Endpoints, parameters, and the OpenAPI schema.
  </Card>

  <Card title="Errors" icon="triangle-exclamation" href="/api-reference/errors">
    Status codes, retries, and SDK exception handling.
  </Card>
</CardGroup>

## Support

* **Technical support**: [support@fish.audio](mailto:support@fish.audio)
* **Issues**: [GitHub](https://github.com/fishaudio)
* **Community**: [Discord](https://discord.gg/dF9Db2Tt3Y)


# Batch-transcribe files with a language hint
Source: https://docs.fish.audio/developer-guide/sdk-guide/cookbook/batch-transcribe-with-language-hint

Loop over local audio files and transcribe each with an explicit language, collecting text and duration per file

## Prerequisites

<AccordionGroup>
  <Accordion icon="user-plus" title="Create a Fish Audio account">
    Sign up for a free Fish Audio account to get started with our API.

    1. Go to [fish.audio/auth/signup](https://fish.audio/auth/signup)
    2. Fill in your details to create an account, complete steps to verify your account.
    3. Log in to your account and navigate to the [API section](https://fish.audio/app/api-keys)
  </Accordion>

  <Accordion icon="key" title="Get your API key">
    Once you have an account, you'll need an API key to authenticate your requests.

    1. Log in to your [Fish Audio Dashboard](https://fish.audio/app/api-keys/)
    2. Navigate to the API Keys section
    3. Click "Create New Key" and give it a descriptive name, set a expiration if desired
    4. Copy your key and store it securely

    <Warning>Keep your API key secret! Never commit it to version control or share it publicly.</Warning>
  </Accordion>
</AccordionGroup>

## Recipe

This recipe uses `transcribe-1-pro`, which handles recordings up to 60 minutes long. Every tab selects it with the `model` request header; without that header, the request is served and billed as `transcribe-1`.

Read each file's bytes from disk and pass them to [`asr.transcribe()`](/api-reference/sdk/python/resources#transcribe) with `language` set to a lowercase ISO 639-1 code such as `en`, `zh`, or `ja`. Other forms, such as `en-US` or `English`, may be rejected with 400. Pass `language` when you know the source language. Detection still runs, and the hint does not force the transcript into that language; the hint is reported as `language_code` when the language cannot be determined. See [Language](/features/speech-to-text#language) for details.

Collect one result row per file as you go. If a file cannot be read, its request fails with a network error such as a timeout, or the API returns an error for it, record the error in that file's row and move on to the next file.

<CodeGroup>
  ```python Synchronous theme={null}
  import httpx
  from fishaudio import FishAudio
  from fishaudio.core import RequestOptions
  from fishaudio.exceptions import APIError

  client = FishAudio()

  paths = ["speech.wav"]  # add more file paths here
  language = "en"
  options = RequestOptions(
      timeout=900,  # long recordings can take several minutes
      additional_headers={"model": "transcribe-1-pro"},
  )

  results = []
  for path in paths:
      try:
          with open(path, "rb") as f:
              audio = f.read()
          transcript = client.asr.transcribe(
              audio=audio,
              language=language,
              include_timestamps=False,
              request_options=options,
          )
      except (OSError, httpx.HTTPError, APIError) as e:
          # Record the failure and continue with the next file.
          results.append({"file": path, "error": f"{type(e).__name__}: {e}"})
          continue
      results.append({
          "file": path,
          "text": transcript.text,
          "duration": transcript.duration,  # seconds
      })

  failed = [row for row in results if "error" in row]
  for row in results:
      if "error" in row:
          print(f"{row['file']}: FAILED: {row['error']}")
      else:
          print(f"{row['file']} ({row['duration']:.1f}s): {row['text']}")

  if failed:
      raise SystemExit(f"{len(failed)} of {len(results)} files failed")
  ```

  ```python Asynchronous theme={null}
  import asyncio

  import httpx
  from fishaudio import AsyncFishAudio
  from fishaudio.core import RequestOptions
  from fishaudio.exceptions import APIError

  async def main():
      async with AsyncFishAudio() as client:
          paths = ["speech.wav"]  # add more file paths here
          language = "en"
          options = RequestOptions(
              timeout=900,  # long recordings can take several minutes
              additional_headers={"model": "transcribe-1-pro"},
          )

          results = []
          for path in paths:
              try:
                  with open(path, "rb") as f:
                      audio = f.read()
                  transcript = await client.asr.transcribe(
                      audio=audio,
                      language=language,
                      include_timestamps=False,
                      request_options=options,
                  )
              except (OSError, httpx.HTTPError, APIError) as e:
                  # Record the failure and continue with the next file.
                  results.append({"file": path, "error": f"{type(e).__name__}: {e}"})
                  continue
              results.append({
                  "file": path,
                  "text": transcript.text,
                  "duration": transcript.duration,  # seconds
              })
      return results

  results = asyncio.run(main())

  failed = [row for row in results if "error" in row]
  for row in results:
      if "error" in row:
          print(f"{row['file']}: FAILED: {row['error']}")
      else:
          print(f"{row['file']} ({row['duration']:.1f}s): {row['text']}")

  if failed:
      raise SystemExit(f"{len(failed)} of {len(results)} files failed")
  ```

  ```javascript JavaScript theme={null}
  import { FishAudioClient } from "fish-audio";
  import { readFile } from "fs/promises";

  const client = new FishAudioClient({ apiKey: process.env.FISH_API_KEY });

  const paths = ["speech.wav"]; // add more file paths here
  const language = "en";
  const options = {
    headers: { model: "transcribe-1-pro" },
    timeoutInSeconds: 900, // long recordings can take several minutes
  };

  const results = [];
  for (const path of paths) {
    try {
      const audio = new File([await readFile(path)], path);
      const transcript = await client.speechToText.convert(
        { audio, language },
        options
      );
      results.push({
        file: path,
        text: transcript.text,
        duration: transcript.duration, // seconds
      });
    } catch (err) {
      // Record the failure and continue with the next file.
      const error = err.statusCode
        ? `HTTP ${err.statusCode}: ${err.body?.message ?? "(no JSON body)"}`
        : err.message;
      results.push({ file: path, error });
    }
  }

  const failed = results.filter(row => row.error);
  for (const row of results) {
    if (row.error) {
      console.log(`${row.file}: FAILED: ${row.error}`);
    } else {
      console.log(`${row.file} (${row.duration.toFixed(1)}s): ${row.text}`);
    }
  }

  if (failed.length > 0) {
    console.error(`${failed.length} of ${results.length} files failed`);
    // Exit now: a request that failed with a network error can leave its timeout running.
    process.exit(1);
  }
  ```
</CodeGroup>

Each call returns an [`ASRResponse`](/api-reference/sdk/python/types#asrresponse-objects) with `.text` and `.duration` (seconds). With `transcribe-1-pro`, `text` can contain inline speaker markers such as `<|speaker:0|>` and cues such as `[laughter]`; see [Speaker markers](/features/speech-to-text#speaker-markers) to split it into turns. This recipe skips timestamps; pass `include_timestamps=True` (Python) or `ignore_timestamps: false` (JavaScript) for word-level `.segments`. The Python SDK currently returns only `text`, `duration`, and `segments`; to read `language_code`, call the API directly (see [Direct API](/features/speech-to-text#direct-api-messagepack)).

A file that fails is recorded with its error, and the loop continues. After the summary, the script exits with a non-zero status if any file failed, so a scheduled job does not report success. A file that failed with 429, a 5xx status, or a network error such as a timeout is worth retrying later with backoff; for other 4xx errors, fix the file or the request first.

Long recordings can take several minutes to transcribe, so every tab sets a 15-minute request timeout, longer than the SDKs' default; in Node.js, also see the note below.

<Note>
  In Node.js, the built-in `fetch` that the JavaScript SDK uses stops waiting
  for a response after 5 minutes, even with a longer `timeoutInSeconds`. To
  wait longer, install `undici@7` and, once at startup, call its
  `setGlobalDispatcher(new Agent({ headersTimeout: 900_000, bodyTimeout: 900_000 }))`.
  See [Processing time and
  timeouts](/features/speech-to-text#processing-time-and-timeouts).
</Note>

<Tip>
  Use one `language` per batch; split mixed-language files into separate lists.
</Tip>

## Related

* [Speech-to-Text guide](/features/speech-to-text)
* [Instant voice cloning](/developer-guide/sdk-guide/cookbook/instant-voice-cloning)


# Clone a voice and wait until it is ready
Source: https://docs.fish.audio/developer-guide/sdk-guide/cookbook/clone-and-wait-until-ready

Create a persistent voice from a reference clip, poll until training finishes, then synthesize with it

A persistent voice is trained asynchronously: `voices.create()` returns immediately with a voice whose `state` is `created` or `training`. Before you can synthesize with it, you need to wait until its `state` becomes `trained`.

This recipe creates a voice from `reference.wav`, polls [`voices.get()`](/api-reference/sdk/python/resources#get) until training finishes (with a timeout), then synthesizes with `reference_id`.

## Prerequisites

<AccordionGroup>
  <Accordion icon="user-plus" title="Create a Fish Audio account">
    Sign up for a free Fish Audio account to get started with our API.

    1. Go to [fish.audio/auth/signup](https://fish.audio/auth/signup)
    2. Fill in your details to create an account, complete steps to verify your account.
    3. Log in to your account and navigate to the [API section](https://fish.audio/app/api-keys)
  </Accordion>

  <Accordion icon="key" title="Get your API key">
    Once you have an account, you'll need an API key to authenticate your requests.

    1. Log in to your [Fish Audio Dashboard](https://fish.audio/app/api-keys/)
    2. Navigate to the API Keys section
    3. Click "Create New Key" and give it a descriptive name, set a expiration if desired
    4. Copy your key and store it securely

    <Warning>Keep your API key secret! Never commit it to version control or share it publicly.</Warning>
  </Accordion>
</AccordionGroup>

## Recipe

Poll `voices.get(voice.id).state` on an interval, stopping when it reaches `trained` (or raising if it `failed` or the timeout elapses). Then pass the voice id as `reference_id` on `convert()`.

<CodeGroup>
  ```python Synchronous theme={null}
  import time

  from fishaudio import FishAudio
  from fishaudio.utils import save

  client = FishAudio()

  # 1. Create a persistent voice from a reference clip.
  with open("reference.wav", "rb") as f:
      voice = client.voices.create(title="My Voice", voices=[f.read()])

  # 2. Poll until the voice finishes training.
  deadline = time.time() + 300  # 5-minute timeout
  while voice.state != "trained":
      if voice.state == "failed":
          raise RuntimeError(f"Voice {voice.id} failed to train")
      if time.time() > deadline:
          raise TimeoutError(f"Voice {voice.id} not ready (state={voice.state})")
      time.sleep(5)
      voice = client.voices.get(voice.id)

  # 3. Synthesize with the trained voice.
  audio = client.tts.convert(
      text="My voice is ready to use.",
      reference_id=voice.id,
  )
  save(audio, "out.mp3")
  ```

  ```python Asynchronous theme={null}
  import asyncio

  from fishaudio import AsyncFishAudio
  from fishaudio.utils import save


  async def main():
      async with AsyncFishAudio() as client:
          # 1. Create a persistent voice from a reference clip.
          with open("reference.wav", "rb") as f:
              voice = await client.voices.create(title="My Voice", voices=[f.read()])

          # 2. Poll until the voice finishes training.
          deadline = asyncio.get_event_loop().time() + 300  # 5-minute timeout
          while voice.state != "trained":
              if voice.state == "failed":
                  raise RuntimeError(f"Voice {voice.id} failed to train")
              if asyncio.get_event_loop().time() > deadline:
                  raise TimeoutError(f"Voice {voice.id} not ready (state={voice.state})")
              await asyncio.sleep(5)
              voice = await client.voices.get(voice.id)

          # 3. Synthesize with the trained voice.
          audio = await client.tts.convert(
              text="My voice is ready to use.",
              reference_id=voice.id,
          )
      save(audio, "out.mp3")


  asyncio.run(main())
  ```

  ```javascript JavaScript theme={null}
  import { readFile, writeFile } from "fs/promises";

  import { FishAudioClient } from "fish-audio";

  const client = new FishAudioClient({ apiKey: process.env.FISH_API_KEY });

  // 1. Create a persistent voice from a reference clip.
  const sample = new File([await readFile("reference.wav")], "reference.wav");
  let voice = await client.voices.ivc.create({
    title: "My Voice",
    visibility: "private",
    voices: [sample],
  });

  // 2. Poll until the voice finishes training.
  const deadline = Date.now() + 300_000; // 5-minute timeout
  while (voice.state !== "trained") {
    if (voice.state === "failed") {
      throw new Error(`Voice ${voice._id} failed to train`);
    }
    if (Date.now() > deadline) {
      throw new Error(`Voice ${voice._id} not ready (state=${voice.state})`);
    }
    await new Promise((resolve) => setTimeout(resolve, 5000));
    voice = await client.voices.get(voice._id);
  }

  // 3. Synthesize with the trained voice.
  const stream = await client.textToSpeech.convert(
    { text: "My voice is ready to use.", reference_id: voice._id },
    "s2-pro",
  );
  const chunks = [];
  for await (const chunk of stream) chunks.push(Buffer.from(chunk));
  await writeFile("out.mp3", Buffer.concat(chunks));
  ```
</CodeGroup>

A voice moves through `created` → `training` → `trained`, or ends in `failed`. Always handle `failed` and the timeout so a stuck voice cannot loop forever.

<Tip>
  Training a persistent voice takes time, so only create one when you will reuse the voice across many requests. For one-off synthesis, skip the wait entirely and pass a `ReferenceAudio` inline. See [Instant voice cloning](/developer-guide/sdk-guide/cookbook/instant-voice-cloning).
</Tip>

## Related

* [Instant voice cloning](/developer-guide/sdk-guide/cookbook/instant-voice-cloning)
* [Voice Cloning guide](/features/voice-cloning)
* [Voices API reference](/api-reference/sdk/python/resources#voices)


# Discover and reuse a Voice Library voice
Source: https://docs.fish.audio/developer-guide/sdk-guide/cookbook/discover-library-voice

Search the public Voice Library by title, pick a result, and synthesize speech with it as your reference_id

## Prerequisites

<AccordionGroup>
  <Accordion icon="user-plus" title="Create a Fish Audio account">
    Sign up for a free Fish Audio account to get started with our API.

    1. Go to [fish.audio/auth/signup](https://fish.audio/auth/signup)
    2. Fill in your details to create an account, complete steps to verify your account.
    3. Log in to your account and navigate to the [API section](https://fish.audio/app/api-keys)
  </Accordion>

  <Accordion icon="key" title="Get your API key">
    Once you have an account, you'll need an API key to authenticate your requests.

    1. Log in to your [Fish Audio Dashboard](https://fish.audio/app/api-keys/)
    2. Navigate to the API Keys section
    3. Click "Create New Key" and give it a descriptive name, set a expiration if desired
    4. Copy your key and store it securely

    <Warning>Keep your API key secret! Never commit it to version control or share it publicly.</Warning>
  </Accordion>
</AccordionGroup>

## Recipe

Set `self_only=False` on [`voices.list()`](/api-reference/sdk/python/resources#list) to search the public [Voice Library](/features/manage-voices) instead of only your own models. The response carries `total` (matches across all pages) and `items` (this page). Pick a result's `id` and pass it straight to [`tts.convert()`](/api-reference/sdk/python/resources#convert) as `reference_id`.

<CodeGroup>
  ```python Python theme={null}
  from fishaudio import FishAudio
  from fishaudio.utils import save

  client = FishAudio()  # reads FISH_API_KEY

  # Search the public library by title (not just your own voices)
  page = client.voices.list(title="narration", self_only=False, page_size=10)
  print(f"{page.total} public voices match")

  # Pick the first result; fall back to a known id if the search is empty
  reference_id = "<voice-id>"
  for voice in page.items:
      print(voice.id, voice.title, voice.languages)
      reference_id = reference_id if reference_id != "<voice-id>" else voice.id

  # Synthesize with the discovered voice as the reference
  audio = client.tts.convert(
      text="Speaking with a voice I found in the public library.",
      reference_id=reference_id,
  )
  save(audio, "out.mp3")
  ```

  ```javascript JavaScript theme={null}
  import { FishAudioClient } from "fish-audio";
  import { writeFile } from "fs/promises";

  const client = new FishAudioClient({ apiKey: process.env.FISH_API_KEY });

  // Search the public Voice Library by title (not just your own voices)
  const page = await client.voices.search({ title: "narration", page_size: 10 });
  console.log(`${page.total} public voices match`);

  // Pick the first result; fall back to a known id if the search is empty
  let referenceId = "<voice-id>";
  for (const voice of page.items) {
    console.log(voice._id, voice.title, voice.languages);
    referenceId = referenceId !== "<voice-id>" ? referenceId : voice._id;
  }

  // Synthesize with the discovered voice as the reference
  const stream = await client.textToSpeech.convert(
    {
      text: "Speaking with a voice I found in the public library.",
      reference_id: referenceId,
      format: "mp3",
    },
    "s2-pro"
  );

  const chunks = [];
  for await (const chunk of stream) chunks.push(Buffer.from(chunk));
  await writeFile("out.mp3", Buffer.concat(chunks));
  ```
</CodeGroup>

`total > len(page.items)` tells you there are more pages; bump `page_number` to walk them. Any public voice `id` works as `reference_id`; nothing is saved to your account.

You can hit the same endpoint directly:

```bash theme={null}
curl "https://api.fish.audio/model?title=narration&page_size=10" \
  --header "Authorization: Bearer $FISH_API_KEY"

# Response: { "total": 128, "items": [ { "_id": "...", "title": "...", ... } ] }
```

<Tip>
  Title search is fuzzy and ranked by usage, so the top result is usually the
  most popular match. Add `language=["en"]` to narrow by spoken language, or
  raise `page_size` and page with `page_number` to scan deeper.
</Tip>

## Related

* [Manage Voices](/features/manage-voices)
* [Instant voice cloning](/developer-guide/sdk-guide/cookbook/instant-voice-cloning)
* [Python reference: voices](/api-reference/sdk/python/resources)


# Instant voice cloning
Source: https://docs.fish.audio/developer-guide/sdk-guide/cookbook/instant-voice-cloning

Clone a voice on the fly from a short reference clip, with no model to manage

## Prerequisites

<AccordionGroup>
  <Accordion icon="user-plus" title="Create a Fish Audio account">
    Sign up for a free Fish Audio account to get started with our API.

    1. Go to [fish.audio/auth/signup](https://fish.audio/auth/signup)
    2. Fill in your details to create an account, complete steps to verify your account.
    3. Log in to your account and navigate to the [API section](https://fish.audio/app/api-keys)
  </Accordion>

  <Accordion icon="key" title="Get your API key">
    Once you have an account, you'll need an API key to authenticate your requests.

    1. Log in to your [Fish Audio Dashboard](https://fish.audio/app/api-keys/)
    2. Navigate to the API Keys section
    3. Click "Create New Key" and give it a descriptive name, set a expiration if desired
    4. Copy your key and store it securely

    <Warning>Keep your API key secret! Never commit it to version control or share it publicly.</Warning>
  </Accordion>
</AccordionGroup>

## Recipe

Pass a [`ReferenceAudio`](/api-reference/sdk/python/types#referenceaudio-objects) (raw audio bytes + an exact transcript) on the `convert` call. Nothing is saved server-side; the clone applies to that request only.

<CodeGroup>
  ```python Synchronous theme={null}
  from fishaudio import FishAudio
  from fishaudio.types import ReferenceAudio
  from fishaudio.utils import save

  client = FishAudio()

  with open("reference.wav", "rb") as f:
      audio = client.tts.convert(
          text="This sentence is spoken in the cloned voice.",
          references=[ReferenceAudio(
              audio=f.read(),
              text="Exact transcript of what is said in reference.wav.",
          )],
      )

  save(audio, "cloned.mp3")
  ```

  ```python Asynchronous theme={null}
  import asyncio
  from fishaudio import AsyncFishAudio
  from fishaudio.types import ReferenceAudio
  from fishaudio.utils import save

  async def main():
      async with AsyncFishAudio() as client:
          with open("reference.wav", "rb") as f:
              audio = await client.tts.convert(
                  text="This sentence is spoken in the cloned voice.",
                  references=[ReferenceAudio(
                      audio=f.read(),
                      text="Exact transcript of what is said in reference.wav.",
                  )],
              )
      save(audio, "cloned.mp3")

  asyncio.run(main())
  ```

  ```javascript JavaScript theme={null}
  import { FishAudioClient } from "fish-audio";
  import { readFile, writeFile } from "fs/promises";

  const client = new FishAudioClient({ apiKey: process.env.FISH_API_KEY });

  const reference = new File(
    [await readFile("reference.wav")],
    "reference.wav",
  );

  const stream = await client.textToSpeech.convert(
    {
      text: "This sentence is spoken in the cloned voice.",
      references: [
        {
          audio: reference,
          text: "Exact transcript of what is said in reference.wav.",
        },
      ],
      format: "mp3",
    },
    "s2-pro",
  );

  const chunks = [];
  for await (const chunk of stream) chunks.push(Buffer.from(chunk));

  await writeFile("cloned.mp3", Buffer.concat(chunks));
  ```
</CodeGroup>

<Tip>
  Use 10–30 s of clean speech, and make `text` match the audio exactly
  (including punctuation) for the best prosody.
</Tip>

## Reuse a voice across many requests

If you'll use the voice repeatedly, create a persistent model once and pass its id as `reference_id`. See the [Voice Cloning guide](/features/voice-cloning).

```python theme={null}
with open("sample.wav", "rb") as f:
    voice = client.voices.create(title="My Voice", voices=[f.read()])

audio = client.tts.convert(text="Reusing my saved voice.", reference_id=voice.id)
```

## Related

* [Voice Cloning guide](/features/voice-cloning)
* [Stream TTS to a file](/developer-guide/sdk-guide/cookbook/streaming-to-file)


# One-shot vs persistent cloning: pick the right approach
Source: https://docs.fish.audio/developer-guide/sdk-guide/cookbook/oneshot-vs-persistent-cloning

Choose between instant per-request cloning and a saved, reusable voice model

## Prerequisites

<AccordionGroup>
  <Accordion icon="user-plus" title="Create a Fish Audio account">
    Sign up for a free Fish Audio account to get started with our API.

    1. Go to [fish.audio/auth/signup](https://fish.audio/auth/signup)
    2. Fill in your details to create an account, complete steps to verify your account.
    3. Log in to your account and navigate to the [API section](https://fish.audio/app/api-keys)
  </Accordion>

  <Accordion icon="key" title="Get your API key">
    Once you have an account, you'll need an API key to authenticate your requests.

    1. Log in to your [Fish Audio Dashboard](https://fish.audio/app/api-keys/)
    2. Navigate to the API Keys section
    3. Click "Create New Key" and give it a descriptive name, set a expiration if desired
    4. Copy your key and store it securely

    <Warning>Keep your API key secret! Never commit it to version control or share it publicly.</Warning>
  </Accordion>
</AccordionGroup>

## Recipe

There are two ways to clone a voice. Pick by how often you'll reuse it:

* **One-shot (instant)**: pass a [`ReferenceAudio`](/api-reference/sdk/python/types#referenceaudio-objects) (raw bytes + exact transcript) on each `convert` call. Nothing is stored server-side; the clone lives only for that request.
* **Persistent**: call `voices.create` once to train a model, then reuse its id as `reference_id` on every request. No reference upload per call, and the same voice is shared across processes.

Start with one-shot. Below, a single reference clip is cloned inline:

<CodeGroup>
  ```python Synchronous theme={null}
  from fishaudio import FishAudio
  from fishaudio.types import ReferenceAudio
  from fishaudio.utils import save

  client = FishAudio()

  with open("reference.wav", "rb") as f:
      audio = client.tts.convert(
          text="This line is spoken in the cloned voice, no model required.",
          references=[ReferenceAudio(
              audio=f.read(),
              text="Exact transcript of what is said in reference.wav.",
          )],
      )

  save(audio, "oneshot.mp3")
  ```

  ```python Asynchronous theme={null}
  import asyncio
  from fishaudio import AsyncFishAudio
  from fishaudio.types import ReferenceAudio
  from fishaudio.utils import save

  async def main():
      async with AsyncFishAudio() as client:
          with open("reference.wav", "rb") as f:
              audio = await client.tts.convert(
                  text="This line is spoken in the cloned voice, no model required.",
                  references=[ReferenceAudio(
                      audio=f.read(),
                      text="Exact transcript of what is said in reference.wav.",
                  )],
              )
      save(audio, "oneshot.mp3")

  asyncio.run(main())
  ```

  ```javascript JavaScript theme={null}
  import { FishAudioClient } from "fish-audio";
  import { readFile, writeFile } from "fs/promises";

  const client = new FishAudioClient({ apiKey: process.env.FISH_API_KEY });

  // One-shot: clone inline by sending the reference bytes + exact transcript.
  // Nothing is stored server-side; the clone lives only for this request.
  const reference = await readFile("reference.wav");

  const stream = await client.textToSpeech.convert(
    {
      text: "This line is spoken in the cloned voice, no model required.",
      references: [
        {
          audio: new File([reference], "reference.wav"),
          text: "Exact transcript of what is said in reference.wav.",
        },
      ],
      format: "mp3",
    },
    "s2-pro"
  );

  const chunks = [];
  for await (const chunk of stream) chunks.push(Buffer.from(chunk));
  await writeFile("oneshot.mp3", Buffer.concat(chunks));
  ```
</CodeGroup>

<Tip>
  One-shot re-sends the reference bytes on every request, so it's ideal for
  one-off or rarely-repeated voices. Once a voice is used more than a handful of
  times, switch to a persistent model to skip the per-call upload.
</Tip>

## Train a persistent voice once, reuse forever

Call [`voices.create`](/api-reference/sdk/python/resources#create) to train a model, then pass `voice.id` as `reference_id`. The same id works from any process and across SDK and REST.

```python theme={null}
with open("reference.wav", "rb") as f:
    voice = client.voices.create(title="My Narrator", voices=[f.read()])

# reuse the same id on every later request; no reference upload
audio = client.tts.convert(
    text="Reusing my saved voice across many requests.",
    reference_id=voice.id,
)
save(audio, "persistent.mp3")
```

Already have a trained voice id? Skip training and pass it directly:

```python theme={null}
audio = client.tts.convert(text="Hello again.", reference_id="<voice-id>")
```

## Which to choose

| | One-shot | Persistent |
| - | - | - |
| Setup | None | One `voices.create` call |
| Per request | Re-uploads reference bytes | Sends only `reference_id` |
| Stored server-side | No | Yes (manage with `voices.update` / `voices.delete`) |
| Best for | One-off or experimental clones | Voices reused many times or across services |

<Tip>
  For either path, give the reference 10–30 s of clean speech and make the
  transcript match the audio exactly (including punctuation) for the best prosody.
</Tip>

## Related

* [Instant voice cloning](/developer-guide/sdk-guide/cookbook/instant-voice-cloning)
* [Voice Cloning guide](/features/voice-cloning)
* [Manage voices](/features/manage-voices)


# Realtime: LLM tokens → speech
Source: https://docs.fish.audio/developer-guide/sdk-guide/cookbook/realtime-llm-to-speech

Pipe a streaming LLM response straight into speech over a WebSocket as tokens arrive

## Prerequisites

<AccordionGroup>
  <Accordion icon="user-plus" title="Create a Fish Audio account">
    Sign up for a free Fish Audio account to get started with our API.

    1. Go to [fish.audio/auth/signup](https://fish.audio/auth/signup)
    2. Fill in your details to create an account, complete steps to verify your account.
    3. Log in to your account and navigate to the [API section](https://fish.audio/app/api-keys)
  </Accordion>

  <Accordion icon="key" title="Get your API key">
    Once you have an account, you'll need an API key to authenticate your requests.

    1. Log in to your [Fish Audio Dashboard](https://fish.audio/app/api-keys/)
    2. Navigate to the API Keys section
    3. Click "Create New Key" and give it a descriptive name, set a expiration if desired
    4. Copy your key and store it securely

    <Warning>Keep your API key secret! Never commit it to version control or share it publicly.</Warning>
  </Accordion>
</AccordionGroup>

## Recipe

[`stream_websocket()`](/api-reference/sdk/python/resources#stream_websocket) takes an iterable of text chunks and yields audio chunks in real time. Feed it your LLM's token stream and play or forward the audio as it's produced.

<CodeGroup>
  ```python Synchronous theme={null}
  from fishaudio import FishAudio
  from fishaudio.utils import play

  client = FishAudio()

  def llm_tokens():
      # Replace with your real streaming LLM call
      for token in ["The ", "first ", "move ", "sets ", "everything ", "in ", "motion."]:
          yield token

  audio_stream = client.tts.stream_websocket(llm_tokens(), reference_id="<voice-id>")
  play(audio_stream)  # or: for chunk in audio_stream: send_to_client(chunk)
  ```

  ```python Asynchronous theme={null}
  import asyncio
  from fishaudio import AsyncFishAudio

  async def llm_tokens():
      for token in ["The ", "first ", "move ", "sets ", "everything ", "in ", "motion."]:
          yield token

  async def main():
      async with AsyncFishAudio() as client:
          audio_stream = client.tts.stream_websocket(llm_tokens())
          with open("out.mp3", "wb") as f:
              async for chunk in audio_stream:
                  f.write(chunk)

  asyncio.run(main())
  ```
</CodeGroup>

## Force generation at a boundary

By default the engine buffers text until it has enough for natural prosody. Yield a [`FlushEvent`](/api-reference/sdk/python/types#flushevent-objects) to force synthesis of what's buffered, useful for turn-taking in a conversation:

```python theme={null}
from fishaudio.types import TextEvent, FlushEvent

def turns():
    yield TextEvent(text="Are you ready?")
    yield FlushEvent()              # speak the question now
    yield TextEvent(text="Let's begin.")
```

The SDK sends the start/stop frames for you.

<Tip>
  Errors mid-stream surface as `WebSocketError`. Reconnect with a fresh call
  rather than retrying on the same socket.
</Tip>

## Related

* [Realtime WebSocket guide](/features/realtime-streaming)
* [Errors & Retries](/developer-guide/sdk-guide/python/errors)


# Stream TTS to a file
Source: https://docs.fish.audio/developer-guide/sdk-guide/cookbook/streaming-to-file

Generate long audio and write it to disk chunk-by-chunk, without buffering it all in memory

## Prerequisites

<AccordionGroup>
  <Accordion icon="user-plus" title="Create a Fish Audio account">
    Sign up for a free Fish Audio account to get started with our API.

    1. Go to [fish.audio/auth/signup](https://fish.audio/auth/signup)
    2. Fill in your details to create an account, complete steps to verify your account.
    3. Log in to your account and navigate to the [API section](https://fish.audio/app/api-keys)
  </Accordion>

  <Accordion icon="key" title="Get your API key">
    Once you have an account, you'll need an API key to authenticate your requests.

    1. Log in to your [Fish Audio Dashboard](https://fish.audio/app/api-keys/)
    2. Navigate to the API Keys section
    3. Click "Create New Key" and give it a descriptive name, set a expiration if desired
    4. Copy your key and store it securely

    <Warning>Keep your API key secret! Never commit it to version control or share it publicly.</Warning>
  </Accordion>
</AccordionGroup>

## Recipe

For long text, use [`stream()`](/api-reference/sdk/python/resources#stream) and write each chunk as it arrives instead of holding the whole file in memory.

<CodeGroup>
  ```python Synchronous theme={null}
  from fishaudio import FishAudio

  client = FishAudio()

  with open("output.mp3", "wb") as f:
      for chunk in client.tts.stream(text="A very long passage of text..."):
          f.write(chunk)
  ```

  ```python Asynchronous theme={null}
  import asyncio
  from fishaudio import AsyncFishAudio

  async def main():
      async with AsyncFishAudio() as client:
          audio_stream = await client.tts.stream(text="A very long passage of text...")
          with open("output.mp3", "wb") as f:
              async for chunk in audio_stream:
                  f.write(chunk)

  asyncio.run(main())
  ```

  ```javascript JavaScript theme={null}
  import { FishAudioClient } from "fish-audio";
  import { createWriteStream } from "fs";

  const client = new FishAudioClient({ apiKey: process.env.FISH_API_KEY });

  // convert() returns a ReadableStream<Uint8Array>. Iterate it and write each
  // chunk as it arrives, so you never hold the whole file in memory.
  const stream = await client.textToSpeech.convert(
    { text: "A very long passage of text...", format: "mp3" },
    "s2-pro"
  );

  const file = createWriteStream("output.mp3");
  for await (const chunk of stream) {
    file.write(Buffer.from(chunk));
  }
  file.end();
  ```
</CodeGroup>

## Collect instead of iterate

If you just want the full bytes, call `.collect()`:

```python theme={null}
audio = client.tts.stream(text="Hello!").collect()  # -> bytes
```

<Tip>
  `convert()` already returns the complete audio as `bytes`. Reach for
  `stream()` when you want to start writing/forwarding bytes before generation
  finishes, or to avoid buffering large files.
</Tip>

## Related

* [Text-to-Speech guide](/features/text-to-speech)
* [Realtime: LLM tokens → speech](/developer-guide/sdk-guide/cookbook/realtime-llm-to-speech)


# Telephony-grade audio (8 kHz) for IVR and phone
Source: https://docs.fish.audio/developer-guide/sdk-guide/cookbook/telephony-8khz-audio

Generate 8 kHz mono WAV/PCM that matches the narrowband sample rate phone networks expect for IVR and call-center playback

## Prerequisites

<AccordionGroup>
  <Accordion icon="user-plus" title="Create a Fish Audio account">
    Sign up for a free Fish Audio account to get started with our API.

    1. Go to [fish.audio/auth/signup](https://fish.audio/auth/signup)
    2. Fill in your details to create an account, complete steps to verify your account.
    3. Log in to your account and navigate to the [API section](https://fish.audio/app/api-keys)
  </Accordion>

  <Accordion icon="key" title="Get your API key">
    Once you have an account, you'll need an API key to authenticate your requests.

    1. Log in to your [Fish Audio Dashboard](https://fish.audio/app/api-keys/)
    2. Navigate to the API Keys section
    3. Click "Create New Key" and give it a descriptive name, set a expiration if desired
    4. Copy your key and store it securely

    <Warning>Keep your API key secret! Never commit it to version control or share it publicly.</Warning>
  </Accordion>
</AccordionGroup>

## Recipe

Phone networks carry narrowband audio at 8 kHz. Generating at a higher rate just forces the carrier to downsample on the way through, wasting bandwidth and often softening the result. Synthesize at 8 kHz directly and the bytes are ready to hand to your IVR or SIP stack.

Set the sample rate on [`TTSConfig`](/api-reference/sdk/python/types#ttsconfig-objects) (it is not a top-level argument) and write the WAV to disk.

<CodeGroup>
  ```python Synchronous theme={null}
  from fishaudio import FishAudio
  from fishaudio.types import TTSConfig
  from fishaudio.utils import save

  client = FishAudio()

  audio = client.tts.convert(
      text="Thank you for calling. Press one to speak with an agent.",
      config=TTSConfig(format="wav", sample_rate=8000),
  )

  save(audio, "out.wav")
  ```

  ```python Asynchronous theme={null}
  import asyncio
  from fishaudio import AsyncFishAudio
  from fishaudio.types import TTSConfig
  from fishaudio.utils import save

  async def main():
      async with AsyncFishAudio() as client:
          audio = await client.tts.convert(
              text="Thank you for calling. Press one to speak with an agent.",
              config=TTSConfig(format="wav", sample_rate=8000),
          )
      save(audio, "out.wav")

  asyncio.run(main())
  ```

  ```javascript JavaScript theme={null}
  import { FishAudioClient } from "fish-audio";
  import { writeFile } from "fs/promises";

  const client = new FishAudioClient({ apiKey: process.env.FISH_API_KEY });

  const stream = await client.textToSpeech.convert(
    {
      text: "Thank you for calling. Press one to speak with an agent.",
      format: "wav",
      sample_rate: 8000,
    },
    "s2-pro"
  );

  const chunks = [];
  for await (const chunk of stream) chunks.push(Buffer.from(chunk));

  await writeFile("out.wav", Buffer.concat(chunks));
  ```
</CodeGroup>

The output is a mono 8 kHz WAV, the standard for G.711 PCM telephony. For a headerless stream to feed straight into a SIP or RTP pipeline, switch to raw PCM with `format="pcm"`; the sample rate stays on `TTSConfig`.

```python theme={null}
audio = client.tts.convert(
    text="Thank you for calling. Press one to speak with an agent.",
    config=TTSConfig(format="pcm", sample_rate=8000),
)
```

```bash API (curl) theme={null}
curl --request POST https://api.fish.audio/v1/tts \
  --header "Authorization: Bearer $FISH_API_KEY" \
  --header "Content-Type: application/json" \
  --header "model: s2-pro" \
  --data '{ "text": "Thank you for calling. Press one to speak with an agent.", "format": "wav", "sample_rate": 8000 }' \
  --output out.wav
```

<Tip>
  8 kHz discards everything above \~4 kHz, so plosives and sibilance lose detail.
  Keep prompts short and articulate, and reserve higher sample rates (16/24 kHz)
  for VoIP or recordings that never touch the legacy phone network.
</Tip>

## Related

* [Text-to-Speech guide](/features/text-to-speech)
* [Stream TTS to a file](/developer-guide/sdk-guide/cookbook/streaming-to-file)


# Transcribe audio to SRT/VTT captions
Source: https://docs.fish.audio/developer-guide/sdk-guide/cookbook/transcribe-to-captions

Transcribe audio with word-level timestamps, group the words into caption cues, and write valid SRT and WebVTT files

## Prerequisites

<AccordionGroup>
  <Accordion icon="user-plus" title="Create a Fish Audio account">
    Sign up for a free Fish Audio account to get started with our API.

    1. Go to [fish.audio/auth/signup](https://fish.audio/auth/signup)
    2. Fill in your details to create an account, complete steps to verify your account.
    3. Log in to your account and navigate to the [API section](https://fish.audio/app/api-keys)
  </Accordion>

  <Accordion icon="key" title="Get your API key">
    Once you have an account, you'll need an API key to authenticate your requests.

    1. Log in to your [Fish Audio Dashboard](https://fish.audio/app/api-keys/)
    2. Navigate to the API Keys section
    3. Click "Create New Key" and give it a descriptive name, set a expiration if desired
    4. Copy your key and store it securely

    <Warning>Keep your API key secret! Never commit it to version control or share it publicly.</Warning>
  </Accordion>
</AccordionGroup>

## Recipe

This recipe uses `transcribe-1-pro`, which handles recordings up to 60 minutes long. Both tabs select it with the `model` request header; without that header, the request is served and billed as `transcribe-1`.

Call [`asr.transcribe()`](/api-reference/sdk/python/resources#transcribe) with `include_timestamps=True` (in JavaScript, `ignore_timestamps: false`) to get timed segments ([`ASRSegment`](/api-reference/sdk/python/types#asrsegment-objects)). Segments are word-level (one character, or a few, in Chinese and Japanese) and carry no punctuation, so group consecutive segments into caption-sized cues before you format them. Segment `start` / `end` are in **seconds**; SRT wants `HH:MM:SS,mmm` (comma), WebVTT wants `HH:MM:SS.mmm` (dot).

The recipe starts a new cue after a pause longer than 0.6 seconds, or when the cue would grow past 42 characters or 6 seconds. It joins words with a space, except between Chinese or Japanese characters, which are written without spaces. A segment can have `start` equal to `end`, so a cue with no duration gets a short one that ends no later than the next cue starts. Every cue therefore ends after it starts, as SRT and WebVTT players expect.

<CodeGroup>
  ```python Python theme={null}
  import re

  from fishaudio import FishAudio
  from fishaudio.core import RequestOptions

  client = FishAudio()

  MAX_CHARS = 42  # longest cue text, in characters
  MAX_MS = 6000  # longest cue, in milliseconds
  MAX_GAP_MS = 600  # a longer pause between words starts a new cue
  MIN_MS = 500  # display time for a cue whose words have no duration

  # Chinese and Japanese are written without spaces between words.
  NO_SPACE = re.compile(r"[\u3040-\u30ff\u3400-\u4dbf\u4e00-\u9fff]")


  def join_words(left: str, right: str) -> str:
      if NO_SPACE.match(left[-1]) and NO_SPACE.match(right[0]):
          return left + right
      return f"{left} {right}"


  def group_cues(segments) -> list[dict]:
      """Group word-level segments into cues, with start and end in milliseconds."""
      cues = []
      for segment in segments:
          word = segment.text.strip()
          if not word:
              continue
          start, end = round(segment.start * 1000), round(segment.end * 1000)
          if cues:
              cue = cues[-1]
              text = join_words(cue["text"], word)
              if (
                  start - cue["end"] <= MAX_GAP_MS
                  and len(text) <= MAX_CHARS
                  and end - cue["start"] <= MAX_MS
              ):
                  cue["text"], cue["end"] = text, max(cue["end"], end)
                  continue
          cues.append({"start": start, "end": end, "text": word})

      # A segment can have start == end. Give a cue with no duration a short one,
      # ending no later than the next cue starts, so every cue ends after it starts.
      for i, cue in enumerate(cues):
          if cue["end"] <= cue["start"]:
              end = cue["start"] + MIN_MS
              if i + 1 < len(cues):
                  end = min(end, cues[i + 1]["start"])
              cue["end"] = max(end, cue["start"] + 1)
      return cues


  def format_timestamp(ms: int, separator: str) -> str:
      """HH:MM:SS,mmm for SRT (separator ","), HH:MM:SS.mmm for WebVTT (".")."""
      hours, ms = divmod(ms, 3_600_000)
      minutes, ms = divmod(ms, 60_000)
      seconds, ms = divmod(ms, 1000)
      return f"{hours:02d}:{minutes:02d}:{seconds:02d}{separator}{ms:03d}"


  with open("speech.wav", "rb") as f:
      result = client.asr.transcribe(
          audio=f.read(),
          include_timestamps=True,
          request_options=RequestOptions(
              timeout=900,  # long recordings can take several minutes
              additional_headers={"model": "transcribe-1-pro"},
          ),
      )

  cues = group_cues(result.segments)

  # SRT: 1-based index, comma decimal separator, blank line between cues.
  with open("captions.srt", "w", encoding="utf-8") as srt:
      for i, cue in enumerate(cues, start=1):
          start = format_timestamp(cue["start"], ",")
          end = format_timestamp(cue["end"], ",")
          srt.write(f"{i}\n{start} --> {end}\n{cue['text']}\n\n")

  # WebVTT: same cues, "WEBVTT" header, dot decimal separator.
  with open("captions.vtt", "w", encoding="utf-8") as vtt:
      vtt.write("WEBVTT\n\n")
      for cue in cues:
          start = format_timestamp(cue["start"], ".")
          end = format_timestamp(cue["end"], ".")
          vtt.write(f"{start} --> {end}\n{cue['text']}\n\n")

  print(f"Wrote {len(cues)} cues to captions.srt and captions.vtt")
  ```

  ```javascript JavaScript theme={null}
  import { FishAudioClient } from "fish-audio";
  import { readFile, writeFile } from "fs/promises";

  const client = new FishAudioClient({ apiKey: process.env.FISH_API_KEY });

  const MAX_CHARS = 42; // longest cue text, in characters
  const MAX_MS = 6000; // longest cue, in milliseconds
  const MAX_GAP_MS = 600; // a longer pause between words starts a new cue
  const MIN_MS = 500; // display time for a cue whose words have no duration

  // Chinese and Japanese are written without spaces between words.
  const NO_SPACE = /[\u3040-\u30ff\u3400-\u4dbf\u4e00-\u9fff]/;

  function joinWords(left, right) {
    if (NO_SPACE.test(left.at(-1)) && NO_SPACE.test(right[0])) {
      return left + right;
    }
    return `${left} ${right}`;
  }

  // Group word-level segments into cues, with start and end in milliseconds.
  function groupCues(segments) {
    const cues = [];
    for (const segment of segments) {
      const word = segment.text.trim();
      if (!word) continue;
      const start = Math.round(segment.start * 1000);
      const end = Math.round(segment.end * 1000);
      const cue = cues.at(-1);
      if (cue) {
        const text = joinWords(cue.text, word);
        if (
          start - cue.end <= MAX_GAP_MS &&
          text.length <= MAX_CHARS &&
          end - cue.start <= MAX_MS
        ) {
          cue.text = text;
          cue.end = Math.max(cue.end, end);
          continue;
        }
      }
      cues.push({ start, end, text: word });
    }

    // A segment can have start == end. Give a cue with no duration a short one,
    // ending no later than the next cue starts, so every cue ends after it starts.
    cues.forEach((cue, i) => {
      if (cue.end <= cue.start) {
        let end = cue.start + MIN_MS;
        if (i + 1 < cues.length) end = Math.min(end, cues[i + 1].start);
        cue.end = Math.max(end, cue.start + 1);
      }
    });
    return cues;
  }

  // HH:MM:SS,mmm for SRT (separator ","), HH:MM:SS.mmm for WebVTT (".").
  function formatTimestamp(ms, separator) {
    const pad = (n, width) => String(n).padStart(width, "0");
    const hours = Math.floor(ms / 3_600_000);
    const minutes = Math.floor((ms % 3_600_000) / 60_000);
    const seconds = Math.floor((ms % 60_000) / 1000);
    const millis = ms % 1000;
    return `${pad(hours, 2)}:${pad(minutes, 2)}:${pad(seconds, 2)}${separator}${pad(millis, 3)}`;
  }

  const result = await client.speechToText.convert(
    {
      audio: new File([await readFile("speech.wav")], "speech.wav"),
      ignore_timestamps: false,
    },
    {
      headers: { model: "transcribe-1-pro" },
      timeoutInSeconds: 900, // long recordings can take several minutes
    }
  );

  const cues = groupCues(result.segments);
  const timing = (cue, separator) =>
    `${formatTimestamp(cue.start, separator)} --> ${formatTimestamp(cue.end, separator)}`;

  // SRT: 1-based index, comma decimal separator, blank line between cues.
  const srt = cues.map(
    (cue, i) => `${i + 1}\n${timing(cue, ",")}\n${cue.text}\n\n`
  );
  await writeFile("captions.srt", srt.join(""), "utf-8");

  // WebVTT: same cues, "WEBVTT" header, dot decimal separator.
  const vtt = cues.map(cue => `${timing(cue, ".")}\n${cue.text}\n\n`);
  await writeFile("captions.vtt", "WEBVTT\n\n" + vtt.join(""), "utf-8");

  console.log(`Wrote ${cues.length} cues to captions.srt and captions.vtt`);
  ```
</CodeGroup>

Both tabs send the same request and write the same `captions.srt` and `captions.vtt`. Long recordings can take several minutes to transcribe, so both tabs set a 15-minute request timeout, longer than the SDKs' default; in Node.js, also see the note below. Tune `MAX_CHARS`, `MAX_MS`, and `MAX_GAP_MS` for your player and audience. `segments` is empty when no speech was found or word timing was temporarily unavailable, and the files then contain no cues, so check the cue count before you publish them.

<Note>
  In Node.js, the built-in `fetch` that the JavaScript SDK uses stops waiting
  for a response after 5 minutes, even with a longer `timeoutInSeconds`. To
  wait longer, install `undici@7` and, once at startup, call its
  `setGlobalDispatcher(new Agent({ headersTimeout: 900_000, bodyTimeout: 900_000 }))`.
  See [Processing time and
  timeouts](/features/speech-to-text#processing-time-and-timeouts).
</Note>

<Tip>
  Captions built from `segments` have no punctuation. Because this recipe
  requests timestamps from `transcribe-1-pro`, the response also includes
  `speaker_turns`, with each turn's punctuated text and its start and end time.
  To keep speakers in separate cues, start a new cue at each turn boundary. In
  JavaScript, read `result.speaker_turns`; it is not in the SDK's TypeScript
  type. The Python SDK does not return `speaker_turns`; to read it from the API,
  see [Speaker turns](/features/speech-to-text#speaker-turns).
</Tip>

## Related

* [Speech-to-Text guide](/features/speech-to-text)
* [ASR Types Reference](/api-reference/sdk/python/types#fishaudio.types.asr)


# Build a voice agent loop: speech in, reply, speech out
Source: https://docs.fish.audio/developer-guide/sdk-guide/cookbook/voice-agent-loop

Transcribe an utterance, generate a reply with your own LLM, and stream that reply back out as speech

## Prerequisites

<AccordionGroup>
  <Accordion icon="user-plus" title="Create a Fish Audio account">
    Sign up for a free Fish Audio account to get started with our API.

    1. Go to [fish.audio/auth/signup](https://fish.audio/auth/signup)
    2. Fill in your details to create an account, complete steps to verify your account.
    3. Log in to your account and navigate to the [API section](https://fish.audio/app/api-keys)
  </Accordion>

  <Accordion icon="key" title="Get your API key">
    Once you have an account, you'll need an API key to authenticate your requests.

    1. Log in to your [Fish Audio Dashboard](https://fish.audio/app/api-keys/)
    2. Navigate to the API Keys section
    3. Click "Create New Key" and give it a descriptive name, set a expiration if desired
    4. Copy your key and store it securely

    <Warning>Keep your API key secret! Never commit it to version control or share it publicly.</Warning>
  </Accordion>
</AccordionGroup>

## Recipe

A voice agent is three stages chained together: [`asr.transcribe()`](/api-reference/sdk/python/resources#transcribe) turns the caller's audio into text, your own LLM turns that text into a reply, and [`tts.stream()`](/api-reference/sdk/python/resources#stream) turns the reply back into speech. The transcript and the reply are just strings, so the only Fish Audio-specific parts are the first and last calls. Streaming the reply lets you start writing (or forwarding) audio before the whole sentence is synthesized.

The recipe transcribes with `transcribe-1-pro`. Every tab selects it with the `model` request header; without that header, the request is served and billed as `transcribe-1`.

<CodeGroup>
  ```python Synchronous theme={null}
  import re

  from fishaudio import FishAudio
  from fishaudio.core import RequestOptions
  from fishaudio.utils import save

  client = FishAudio()

  def reply_from_llm(text: str) -> str:
      # ---- PLACEHOLDER ----
      # Call your own LLM here and return its reply as a string.
      # e.g. return openai_client.chat.completions.create(...).choices[0].message.content
      return f"You said: {text}. How can I help?"

  def voice_agent_turn(audio_path: str, out_path: str) -> str:
      with open(audio_path, "rb") as f:
          heard = client.asr.transcribe(
              audio=f.read(),
              include_timestamps=False,
              request_options=RequestOptions(
                  additional_headers={"model": "transcribe-1-pro"}
              ),
          )

      # Drop speaker markers such as <|speaker:0|>; keep cues such as [laughter].
      said = re.sub(r"\s*<\|speaker:\d+\|>\s*", " ", heard.text).strip()
      reply = reply_from_llm(said)

      audio_stream = client.tts.stream(text=reply, reference_id="<voice-id>")
      save(audio_stream, out_path)  # writes chunks as they arrive
      return reply

  reply = voice_agent_turn("speech.wav", "reply.mp3")
  print("Agent:", reply)
  ```

  ```python Asynchronous theme={null}
  import asyncio
  import re

  from fishaudio import AsyncFishAudio
  from fishaudio.core import RequestOptions
  from fishaudio.utils import save

  def reply_from_llm(text: str) -> str:
      # ---- PLACEHOLDER ----
      # Call your own LLM here and return its reply as a string.
      return f"You said: {text}. How can I help?"

  async def main():
      async with AsyncFishAudio() as client:
          with open("speech.wav", "rb") as f:
              heard = await client.asr.transcribe(
                  audio=f.read(),
                  include_timestamps=False,
                  request_options=RequestOptions(
                      additional_headers={"model": "transcribe-1-pro"}
                  ),
              )

          # Drop speaker markers such as <|speaker:0|>; keep cues such as [laughter].
          said = re.sub(r"\s*<\|speaker:\d+\|>\s*", " ", heard.text).strip()
          reply = reply_from_llm(said)

          audio_stream = await client.tts.stream(text=reply, reference_id="<voice-id>")
          with open("reply.mp3", "wb") as out:
              async for chunk in audio_stream:
                  out.write(chunk)
          print("Agent:", reply)

  asyncio.run(main())
  ```

  ```javascript JavaScript theme={null}
  import { FishAudioClient } from "fish-audio";
  import { readFile, writeFile } from "fs/promises";

  const client = new FishAudioClient({ apiKey: process.env.FISH_API_KEY });

  function replyFromLlm(text) {
    // ---- PLACEHOLDER ----
    // Call your own LLM here and return its reply as a string.
    // e.g. return openaiClient.chat.completions.create(...).choices[0].message.content
    return `You said: ${text}. How can I help?`;
  }

  async function voiceAgentTurn(audioPath, outPath) {
    const heard = await client.speechToText.convert(
      {
        audio: new File([await readFile(audioPath)], audioPath),
        language: "en",
      },
      { headers: { model: "transcribe-1-pro" } }
    );

    // Drop speaker markers such as <|speaker:0|>; keep cues such as [laughter].
    const said = heard.text.replace(/\s*<\|speaker:\d+\|>\s*/g, " ").trim();
    const reply = replyFromLlm(said);

    const stream = await client.textToSpeech.convert(
      { text: reply, reference_id: "<voice-id>", format: "mp3" },
      "s2-pro"
    );
    const chunks = [];
    for await (const chunk of stream) chunks.push(Buffer.from(chunk));
    await writeFile(outPath, Buffer.concat(chunks));
    return reply;
  }

  const reply = await voiceAgentTurn("speech.wav", "reply.mp3");
  console.log("Agent:", reply);
  ```
</CodeGroup>

`heard` is an [`ASRResponse`](/api-reference/sdk/python/types#asrresponse-objects): `heard.text` is the full transcript and `heard.duration` is the clip length in seconds. With `transcribe-1-pro`, `heard.text` can contain inline speaker markers such as `<|speaker:0|>` and cues such as `[laughter]`. The recipe removes the speaker markers before it calls your LLM and keeps the cues as context; see [Speaker markers](/features/speech-to-text#speaker-markers). The Python calls pass `include_timestamps=False` because the loop needs only the text, and word timestamps add processing time. Pass `language="en"` as a hint when you know the caller's language; see [Language](/features/speech-to-text#language).

<Tip>
  For the lowest latency, feed your LLM's token stream straight into
  [`stream_websocket()`](/api-reference/sdk/python/resources#stream_websocket)
  instead of waiting for the full reply string. See [Realtime: LLM tokens →
  speech](/developer-guide/sdk-guide/cookbook/realtime-llm-to-speech).
</Tip>

## Reply in the caller's voice

`reference_id` points the reply at a saved voice. Drop it to use the default voice, or clone the caller's voice from the same clip you just transcribed by passing `references` instead. See [Instant voice cloning](/developer-guide/sdk-guide/cookbook/instant-voice-cloning).

## Related

* [Speech-to-Text guide](/features/speech-to-text)
* [Realtime: LLM tokens → speech](/developer-guide/sdk-guide/cookbook/realtime-llm-to-speech)
* [Stream TTS to a file](/developer-guide/sdk-guide/cookbook/streaming-to-file)


# Authentication
Source: https://docs.fish.audio/developer-guide/sdk-guide/python/authentication

Configure API authentication for the Fish Audio Python SDK

## Get Your API Key

<AccordionGroup>
  <Accordion icon="user-plus" title="Create a Fish Audio account">
    Sign up for a free Fish Audio account to get started with our API.

    1. Go to [fish.audio/auth/signup](https://fish.audio/auth/signup)
    2. Fill in your details to create an account, complete steps to verify your account.
    3. Log in to your account and navigate to the [API section](https://fish.audio/app/api-keys)
  </Accordion>

  <Accordion icon="key" title="Get your API key">
    Once you have an account, you'll need an API key to authenticate your requests.

    1. Log in to your [Fish Audio Dashboard](https://fish.audio/app/api-keys/)
    2. Navigate to the API Keys section
    3. Click "Create New Key" and give it a descriptive name, set a expiration if desired
    4. Copy your key and store it securely

    <Warning>Keep your API key secret! Never commit it to version control or share it publicly.</Warning>
  </Accordion>
</AccordionGroup>

## Client Initialization

Initialize the [`FishAudio`](/api-reference/sdk/python/client#fishaudio-objects) client with your API key:

<Tabs>
  <Tab title="Environment Variable (Recommended)">
    The most secure approach is using environment variables:

    ```python theme={null}
    from fishaudio import FishAudio

    # Automatically reads from FISH_API_KEY environment variable
    client = FishAudio()
    ```

    Set the environment variable in your shell:

    ```bash theme={null}
    export FISH_API_KEY=your_api_key_here
    ```

    Or create a `.env` file in your project root:

    ```bash theme={null}
    FISH_API_KEY=your_api_key_here
    ```

    Then load it using `python-dotenv`:

    ```python theme={null}
    from dotenv import load_dotenv
    from fishaudio import FishAudio

    # Load environment variables from .env file
    load_dotenv()

    client = FishAudio()
    ```

    <Tip>
      Using environment variables keeps your API key out of your codebase and makes it easy to use different keys for development and production.
    </Tip>
  </Tab>

  <Tab title="Direct API Key">
    Provide the API key directly when initializing the client:

    ```python theme={null}
    from fishaudio import FishAudio

    client = FishAudio(api_key="your_api_key_here")
    ```

    <Warning>
      This approach is less secure. Never commit code containing your actual API key. Use this only for quick testing or when loading the key from a secure secrets manager.
    </Warning>
  </Tab>

  <Tab title="Custom Base URL">
    If you're using a proxy or custom endpoint:

    ```python theme={null}
    from fishaudio import FishAudio

    client = FishAudio(
        api_key="your_api_key",
        base_url="https://your-proxy-domain.com"
    )
    ```

    This is useful for:

    * Corporate proxies
    * Development/staging environments
    * Self-hosted deployments
  </Tab>
</Tabs>

## Verifying Authentication

Test your authentication by making a simple API call to check your account credits:

```python focus={7-9} theme={null}
from fishaudio import FishAudio
from fishaudio.exceptions import AuthenticationError

try:
    client = FishAudio()

    # Check account credits (requires valid authentication)
    credits = client.account.get_credits()
    print(f"Authentication successful! Credits: {credits.credit}")

except AuthenticationError:
    print("Authentication failed. Check your API key.")
```

Handle [`AuthenticationError`](/api-reference/sdk/python/exceptions#authenticationerror-objects) when verifying authentication. The example uses [`get_credits()`](/api-reference/sdk/python/resources#get_credits) to verify the authentication works.

## Next Steps

<CardGroup>
  <Card title="Text-to-Speech" icon="microphone" href="/features/text-to-speech">
    Generate speech with the authenticated client
  </Card>

  <Card title="Voice Cloning" icon="clone" href="/features/voice-cloning">
    Clone voices and create custom models
  </Card>

  <Card title="Account Management" icon="user" href="/api-reference/sdk/python/resources#account">
    Check credits and manage your account
  </Card>

  <Card title="Error Handling" icon="triangle-exclamation" href="/api-reference/sdk/python/exceptions">
    Handle authentication errors properly
  </Card>
</CardGroup>


# Errors & Retries
Source: https://docs.fish.audio/developer-guide/sdk-guide/python/errors

Exception types, retry strategy, and timeouts in the Fish Audio Python SDK

## Prerequisites

<AccordionGroup>
  <Accordion icon="user-plus" title="Create a Fish Audio account">
    Sign up for a free Fish Audio account to get started with our API.

    1. Go to [fish.audio/auth/signup](https://fish.audio/auth/signup)
    2. Fill in your details to create an account, complete steps to verify your account.
    3. Log in to your account and navigate to the [API section](https://fish.audio/app/api-keys)
  </Accordion>

  <Accordion icon="key" title="Get your API key">
    Once you have an account, you'll need an API key to authenticate your requests.

    1. Log in to your [Fish Audio Dashboard](https://fish.audio/app/api-keys/)
    2. Navigate to the API Keys section
    3. Click "Create New Key" and give it a descriptive name, set a expiration if desired
    4. Copy your key and store it securely

    <Warning>Keep your API key secret! Never commit it to version control or share it publicly.</Warning>
  </Accordion>
</AccordionGroup>

## Exception hierarchy

Every SDK error inherits from [`FishAudioError`](/api-reference/sdk/python/exceptions#fishaudioerror-objects). HTTP failures raise [`APIError`](/api-reference/sdk/python/exceptions#apierror-objects) or one of its subclasses, which expose `.status`, `.message`, and `.body`.

| Exception | Raised when | Notes |
| - | - | - |
| `AuthenticationError` | `401` | Missing or invalid API key |
| `PermissionError` | `403` | Key lacks permission for the resource |
| `NotFoundError` | `404` | Voice model id not found |
| `RateLimitError` | `429` | Rate limit / quota exceeded |
| `ServerError` | `5xx` | Transient server-side failure |
| `APIError` | any other non-2xx | Base for the above; `status == 422` for invalid parameters |
| `WebSocketError` | realtime stream failed mid-session | Reconnect rather than retrying the same socket |
| `DependencyError` | a required system tool is missing (e.g. ffmpeg for `play()`) | Carries `.dependency` and `.install_command` |

<Warning>
  Invalid request parameters come back as an `APIError` with `status == 422`.
  Catch `APIError`, not a separate `ValidationError` (none is raised at
  runtime).
</Warning>

## Handling errors

<CodeGroup>
  ```python Synchronous theme={null}
  from fishaudio import FishAudio
  from fishaudio.exceptions import (
      AuthenticationError,
      RateLimitError,
      NotFoundError,
      APIError,
      FishAudioError,
  )

  client = FishAudio()

  try:
      audio = client.tts.convert(text="Hello!", reference_id="maybe-missing")
  except AuthenticationError:
      print("Invalid API key")
  except RateLimitError:
      print("Rate limited: back off and retry")
  except NotFoundError:
      print("That voice model does not exist")
  except APIError as e:
      print(f"API error {e.status}: {e.message}")  # includes 422 validation
  except FishAudioError as e:
      print(f"SDK error: {e}")  # e.g. WebSocketError, DependencyError
  ```

  ```python Asynchronous theme={null}
  import asyncio
  from fishaudio import AsyncFishAudio
  from fishaudio.exceptions import RateLimitError, APIError, FishAudioError

  async def main():
      async with AsyncFishAudio() as client:
          try:
              audio = await client.tts.convert(text="Hello!")
          except RateLimitError:
              print("Rate limited: back off and retry")
          except APIError as e:
              print(f"API error {e.status}: {e.message}")
          except FishAudioError as e:
              print(f"SDK error: {e}")

  asyncio.run(main())
  ```
</CodeGroup>

## Retries

The Python client does **not** retry automatically. Add your own backoff where it matters, typically around `RateLimitError` and `ServerError`:

```python theme={null}
import time
from fishaudio import FishAudio
from fishaudio.exceptions import RateLimitError, ServerError

client = FishAudio()

def convert_with_retry(text: str, max_retries: int = 3) -> bytes:
    for attempt in range(max_retries):
        try:
            return client.tts.convert(text=text)
        except (RateLimitError, ServerError):
            if attempt == max_retries - 1:
                raise
            time.sleep(2 ** attempt)  # exponential backoff
    raise RuntimeError("unreachable")
```

<Note>
  `RequestOptions` accepts a `max_retries` field, but the current client does
  not act on it. Use an explicit loop like the one above.
</Note>

## Timeouts

The request timeout is set on the client (seconds; default `240`):

```python theme={null}
from fishaudio import FishAudio

client = FishAudio(timeout=30.0)
```

Override headers or timeout for a single request with `request_options`:

```python theme={null}
from fishaudio.core.request_options import RequestOptions

audio = client.tts.convert(
    text="Hello!",
    request_options=RequestOptions(timeout=15.0, additional_headers={"X-Trace": "abc"}),
)
```

<Warning>
  If you inject your own `httpx_client`, the client-level `timeout`,
  `base_url`, and `Authorization` header are **not** applied to it. Configure
  those on the client you pass in.
</Warning>

## Related

* [Exceptions API reference](/api-reference/sdk/python/exceptions)
* [Real-time WebSocket](/features/realtime-streaming): `WebSocketError` handling


# SDK Quickstart
Source: https://docs.fish.audio/developer-guide/sdk-guide/quickstart

Install a Fish Audio SDK, authenticate, and generate your first audio in under a minute

The fastest path from zero to playable audio with the official Fish Audio SDKs. By the end you'll have a script that turns text into an MP3.

<Note>
  The Python SDK is the recommended starting point and is fully covered below.
  The JavaScript SDK is in early release. See the [JavaScript SDK
  guide](/api-reference/sdk/javascript/api-reference) for its current
  surface.
</Note>

## 1. Install

<CodeGroup>
  ```bash Python theme={null}
  pip install fish-audio-sdk

  # optional: local audio playback (needs ffmpeg)
  pip install "fish-audio-sdk[utils]"
  ```

  ```bash JavaScript theme={null}
  npm install fish-audio
  ```
</CodeGroup>

## 2. Authenticate

<AccordionGroup>
  <Accordion icon="user-plus" title="Create a Fish Audio account">
    Sign up for a free Fish Audio account to get started with our API.

    1. Go to [fish.audio/auth/signup](https://fish.audio/auth/signup)
    2. Fill in your details to create an account, complete steps to verify your account.
    3. Log in to your account and navigate to the [API section](https://fish.audio/app/api-keys)
  </Accordion>

  <Accordion icon="key" title="Get your API key">
    Once you have an account, you'll need an API key to authenticate your requests.

    1. Log in to your [Fish Audio Dashboard](https://fish.audio/app/api-keys/)
    2. Navigate to the API Keys section
    3. Click "Create New Key" and give it a descriptive name, set a expiration if desired
    4. Copy your key and store it securely

    <Warning>Keep your API key secret! Never commit it to version control or share it publicly.</Warning>
  </Accordion>
</AccordionGroup>

Both SDKs read your key from the `FISH_API_KEY` environment variable:

```bash theme={null}
export FISH_API_KEY=your_api_key_here
```

## 3. Generate your first audio

<CodeGroup>
  ```python Python theme={null}
  from fishaudio import FishAudio
  from fishaudio.utils import save

  client = FishAudio()  # reads FISH_API_KEY

  audio = client.tts.convert(text="Hello from Fish Audio!")
  save(audio, "output.mp3")
  ```

  ```typescript JavaScript theme={null}
  import { FishAudioClient, play } from "fish-audio";

  const client = new FishAudioClient(); // reads FISH_API_KEY
  const audio = await client.textToSpeech.convert({ text: "Hello from Fish Audio!" });
  await play(audio);
  ```
</CodeGroup>

Run it, and you'll have `output.mp3` (Python) or local playback (JavaScript).

<Tip>
  Want async in Python? Every method mirrors onto `AsyncFishAudio`: `async with
      AsyncFishAudio() as client: audio = await client.tts.convert(text="...")`.
</Tip>

## Next steps

<CardGroup>
  <Card title="Text-to-Speech" icon="microphone" href="/features/text-to-speech">
    Voices, formats, prosody, and model selection
  </Card>

  <Card title="Voice Cloning" icon="clone" href="/features/voice-cloning">
    Instant cloning and persistent voice models
  </Card>

  <Card title="Realtime WebSocket" icon="bolt" href="/features/realtime-streaming">
    Stream LLM tokens to speech as they arrive
  </Card>

  <Card title="Errors & Retries" icon="triangle-exclamation" href="/developer-guide/sdk-guide/python/errors">
    Exception types, retries, and timeouts
  </Card>

  <Card title="Cookbook" icon="book" href="/developer-guide/sdk-guide/cookbook/streaming-to-file">
    Task-focused recipes
  </Card>

  <Card title="API Reference" icon="book-open" href="/api-reference/sdk/python/overview">
    Full Python SDK reference
  </Card>
</CardGroup>


# Air-gapped deployments
Source: https://docs.fish.audio/developer-guide/self-hosting/air-gapped

Running Fish Audio Enterprise on a network that cannot reach Fish Audio at all

Both offline modes make no outbound calls at runtime. Neither one, however,
installs itself out of thin air: a Helm install pulls container images and the chart
from a registry, and the appliance needs its image on the host. An air-gapped
deployment is about getting those artifacts across the boundary. The license bundle
has to cross it too, and unlike an image it expires — plan a recurring transfer, not
a one-off.

<Note>
  Fish Audio does this with you as part of an air-gapped delivery, and the step-by-step
  procedure is in the deployment runbook for the version you install. See
  [Releases](/developer-guide/self-hosting/enterprise-releases). This page is what to
  expect and what to plan for.
</Note>

## Which one to choose

**The air-gapped All-in-One appliance** is the straightforward answer to a strict air
gap: one image, moved to the disconnected host as a file and loaded there. Beyond the
image, only the license bundle — and, in the external-weights modes, the weight
fileset — has to be carried across. Its online counterpart cannot run air
gapped: it authorises every request against Fish Audio.

**The Helm chart** is more work, because the release is many images rather than one.
Every image has to be mirrored into a registry the cluster can reach and the release
pointed at it, component by component. Plan for the whole
set, and expect to do it with your account team rather than alone.

**What you give up either way:** images are pinned to the exact content Fish Audio
published, so a deployment cannot quietly run anything else. Copying an image into
another registry drops that pin, so check what you mirrored while both sides are still
there to compare.

## Offline usage accounting

An offline deployment cannot call a billing service, so it records usage locally
instead, in a signed ledger on persistent storage. It is durable data rather than
cache: include it in your backups, and agree a reconciliation cadence with your account
team. Usage is settled from the ledger files themselves, not from totals compiled off
them.

Because there is no service to validate a token against, any non-empty bearer token is
accepted and recorded as the billing identity. Use a stable, distinct token per tenant.

## Proving there is no egress

Regulated deployments usually need evidence rather than a configuration review, and the
runbook carries the exercise. Two things are worth knowing before you plan it.

It answers two questions, and they need different methods: whether the deployment
*needs* the internet, and whether it *calls out* when allowed to. The second is what a
telemetry or data-residency review actually asks.

It is also a cluster-level exercise rather than a namespace one. Confirm early that
whoever runs your cluster can take part, because a namespace-scoped account cannot
complete it.

## Next steps

* [Requirements](/developer-guide/self-hosting/requirements): hardware, platform, and network baselines
* [All-in-One container](/developer-guide/self-hosting/all-in-one): the single-container platform
* [Kubernetes deployment](/developer-guide/self-hosting/kubernetes): the Helm platform


# All-in-One container
Source: https://docs.fish.audio/developer-guide/self-hosting/all-in-one

The single-container appliance: what it is, what it cannot do, and what running it involves

The All-in-One image packages the whole speech stack (edge API, model API layer,
inference router and worker, vocoder, text normalizer, and Redis) into one container,
with the model weights baked into the image, except in the external-weights modes. It runs as a turnkey single-node appliance, no Kubernetes.

The two billing modes, **Offline All-in-One** and **Online All-in-One**, differ only in how
usage is accounted for. The offline mode records usage to a local
ledger and needs no network once the image is on the host. The online mode authorises
each request against Fish Audio and charges the key that made it, so it needs outbound
access and a key with credit. The accounting is compiled in, so an image is one or the
other; the container logs which at startup.

<Note>
  This page covers what the appliance is and what to plan for. The commands, tuning
  options, and troubleshooting are in the **All-in-One guide**, which ships in the
  deployment guide download for the image version you run. See
  [Releases](/developer-guide/self-hosting/enterprise-releases).
</Note>

## What it cannot do

The appliance runs one inference worker and one vocoder — a GPU each; the
Single-GPU special case shares one H100-class 80 GB card, with lower throughput.
It does not
autoscale, does not shard across more GPUs or nodes, and does not ship the forced
aligner, so it returns no word or segment timings. For elastic or higher-throughput
deployments, use the [Kubernetes chart](/developer-guide/self-hosting/kubernetes), which
scales replicas across all GPUs and nodes.

## What running it involves

One `docker run` on a host that meets the
[All-in-One host requirements](/developer-guide/self-hosting/requirements#all-in-one-docker).
It needs:

* **Two GPUs** — one for the inference worker, one for the vocoder. The Single-GPU
  mode is a special case: one H100-class 80 GB card shared by both, with lower
  throughput.
* **One exposed port** for the API.
* **One persistent volume.** Compile caches, the vocoder's built engine, reference
  voice archives, and — in the offline mode — the usage ledger all live there. Model
  weights are in the image, not on the volume; the external-weights modes mount
  them at run time instead — the delivered guide covers that mount.
* **The license bundle.** A host directory holding the certificate and key pair from
  **Developer → Self Host**, mounted read-only. Without it the API never comes up; once
  the certificate expires the container starts but refuses requests. Renewing is
  replacing those files in place; only a bundle that carries a new key needs the
  container restarted.

Everything inside the container runs as a non-root user, so a reused volume or a host
bind mount has to be writable by it.

**Plan for a slow first start.** The worker compiles its inference graphs and the
vocoder builds its engine before either serves, and the health endpoint verifies the
speech backend end to end rather than reporting immediate liveness. Both artifacts are
cached on the volume, so later starts take minutes. The vocoder engine is specific to
the GPU model, so moving to different cards rebuilds it once.

## Usage accounting and tenancy

The billing mode determines how usage is accounted for and what the bearer token has to be.

**Offline.** Usage goes to a local, signed, append-only ledger on the volume. Two
consequences worth designing around:

* **Any non-empty bearer token is accepted**; a missing or empty one is rejected. The
  appliance has nothing to validate a token against.
* **The token is recorded verbatim as the billing identity.** Use a stable, distinct
  token per tenant. Two tenants sharing a token are indistinguishable in the ledger.

Backups and reconciliation: [Offline usage accounting](/developer-guide/self-hosting/air-gapped#offline-usage-accounting).

**Online.** Each request is authorised against Fish Audio and charged to the key that
made it. The token must be a [Fish Audio API key](/developer-guide/getting-started/api-key)
with credit — not the deploy token used to pull the image — and anything else is
rejected. No ledger is written; usage appears in your Fish Audio account instead.

Authorisation happens on every request, not only at startup, and the appliance has no
local fallback. Treat outbound reachability as a production dependency.

## Capacity

The single worker admits a bounded number of in-flight requests; beyond that, requests
queue and time-to-first-audio climbs. The ceiling is set by the worker's key-value
cache VRAM, so larger cards support a higher cap. It is adjustable at launch without
rebuilding the image. Validate latency and error rate at any new value before
committing to it.

Reference-id requests resolve only from local archives placed on the volume, one zip
per voice.

## Next steps

* [Requirements](/developer-guide/self-hosting/requirements#all-in-one-docker): host baseline
* [Registry & license](/developer-guide/self-hosting/registry-access): how your team gets the image
* [Air-gapped deployments](/developer-guide/self-hosting/air-gapped): moving the image to a disconnected host


# Releases
Source: https://docs.fish.audio/developer-guide/self-hosting/enterprise-releases

How Fish Audio Enterprise versions are identified and how to move between them

Fish Audio publishes a version for each mode once it has been tested.
**Developer → Self Host** lists the versions available to your team and fills the
one you pick into the install commands on that page.

## Version identifiers

| Platform | Identified by | Applied with |
| - | - | - |
| Helm | A chart version | `--version` on `helm pull`/`install` |
| All-in-One | An image tag | The tag in the image reference |

Both Helm profiles are one chart, so they share a version. Which profile you deploy is
decided by the values file, not by the version. Each All-in-One mode — the
combination of billing, GPU topology, and how weights are delivered — is released
and tagged independently; **Developer → Self Host** shows the ones your team
is granted.

## Upgrading

A deployment stays on its version until you change it. Take the new version from
**Developer → Self Host**, then follow the upgrade
procedure in the deployment runbook for the Helm platform, or pull the new tag and
recreate the container against the same volume for
[All-in-One](/developer-guide/self-hosting/all-in-one). Air-gapped deployments
mirror the new version first. See
[Air-gapped deployments](/developer-guide/self-hosting/air-gapped).

<Note>
  Talk to your account team before skipping versions, and before upgrading a
  production deployment, so the change can be checked against how yours is
  configured.
</Note>

## Next step

Return to [Kubernetes deployment](/developer-guide/self-hosting/kubernetes) or
[All-in-One container](/developer-guide/self-hosting/all-in-one) to apply the
version you picked.


# Introduction
Source: https://docs.fish.audio/developer-guide/self-hosting/introduction

Run the Fish Audio Enterprise speech stack inside your own infrastructure

Fish Audio Enterprise can be deployed into infrastructure you control: your own
cloud account, an on-premise data center, or a network with no internet access at
all. Fish Audio delivers container images and a Helm chart through a private
registry; you own the cluster, the network boundary, and the data.

The self-hosted stack runs the same speech engine as the hosted API, so requests,
audio formats, and voice behavior match what you already build against.

## Why self-host

| Reason | What it gives you |
| - | - |
| Colocation | Run inference in the same region, VPC, or rack as your application and remove public-internet round trips from time-to-first-audio. |
| Single-tenant isolation | Dedicated GPUs and queues. Capacity is not shared with other tenants, and you decide when the deployment is upgraded. |
| Security posture | Voice traffic never leaves your network. The offline modes make no outbound calls at runtime and run on disconnected networks. |
| Data sovereignty | Input text, generated audio, and reference voices stay inside your boundary and under your own retention policy. |

## Platforms and modes

The same engine ships on two platforms. Your enterprise agreement determines
which platforms — and which modes of each — your team is granted.

| | Helm (Kubernetes) | All-in-One (Docker) |
| - | - | - |
| Shape | The chart on your cluster: replicas across GPUs and nodes, rolling upgrades. | One container on a single machine; the smallest footprint. |
| Best for | Production clusters and elastic capacity. | Evaluation, fixed-size appliances, strict air gaps. |

A **mode** is a combination of the axes below; on Helm only billing is a mode —
the other axes are cluster configuration. **Developer → Self Host** shows the
modes your team can install.

**Online** billing authorizes and bills each request against Fish Audio and
needs outbound access; **Offline** records usage to a local signed ledger and
runs air-gapped.

| Axis | All-in-One | Helm |
| - | - | - |
| Billing | Online or Offline. | The same two, as deployment profiles of one chart — a values file selects the profile. |
| Topology | **Dual-GPU** — worker and vocoder on a card each. **Single-GPU** — a special case: both share one H100-class 80 GB card, with lower throughput. | Not a mode — each component takes a whole GPU; you add replicas to grow. |
| Weights | **Baked** into the image, or **external** — mounted at run time. | In-cluster by default; the online profile can instead pull from S3-compatible storage — see [Kubernetes deployment](/developer-guide/self-hosting/kubernetes). |

The offline modes are air-gap capable — the Helm profile after mirroring images
and the chart into your own registry.

Choose the platform and mode that fit your constraints, then agree them with
your Fish Audio account team — your agreement determines what **Developer →
Self Host** offers you. Combinations beyond this standard set can be discussed
there too.

Both Helm profiles share one deployment procedure and differ only in a few values.
See [Kubernetes deployment](/developer-guide/self-hosting/kubernetes) for the
Helm platform and [All-in-One container](/developer-guide/self-hosting/all-in-one)
for the single-container platform.

## What is included

| Capability | Online Helm | Offline Helm | All-in-One (all modes) |
| - | - | - | - |
| Text to speech over `POST /v1/tts` | Included | Included | Included |
| WebSocket streaming | Included | Included | Included |
| Reference-voice requests (`reference_id`) | From pre-staged archives | From pre-staged archives | From pre-staged archives |
| Word and segment timestamps | Optional, needs an extra GPU | Not included | Not included |
| Horizontal scaling and autoscaling | Included | Included | Not included |
| Prometheus metrics | Included | Included | Container logs only |

<Note>
  Timestamp alignment is served by a separate forced-aligner component. It is
  disabled by default, is not part of the offline model bundle, and is not built
  into the All-in-One image, so `/v1/tts/stream/with-timestamp` returns audio
  without alignment data in those modes. Contact Fish Audio if your deployment
  needs timestamps.
</Note>

Reference voices are resolved from archives you stage yourself. A self-hosted
deployment does not read from the hosted voice library, and voice models created
on fish.audio are not automatically available to it. Products other than text to
speech are hosted-only unless your agreement says otherwise.

## Architecture at a glance

The table describes the Helm deployment; the All-in-One runs the same components
inside one container.

| Component | Role |
| - | - |
| Edge API | Entry point. Validates requests, applies product logic, and routes work to the model services. |
| Model API layer | Coordinates normalization, inference, and audio decoding. |
| Inference router | Distributes work across the GPU workers. |
| Inference workers | GPU-backed generation. |
| Vocoder | GPU-backed audio decoding. |
| Text normalizer | Text normalization ahead of inference. |
| Redis | Runtime state and cache. Installed by the chart into the release namespace. |
| Shared storage | Model cache, reference archives, and the usage ledger. Mounted on every node. |

Fish Audio delivers the deployment at the Kubernetes service level. Ingress, DNS,
TLS, external load balancing, and network allowlists are yours to choose and
configure.

## Getting access

<Steps>
  <Step title="Agree the deployment scope">
    Self-hosting is enabled per team under an enterprise agreement. Reach out
    through [fish.audio/enterprise](https://fish.audio/enterprise) with your GPU
    target, expected concurrency, and whether you need air-gapped operation.
  </Step>

  <Step title="Get your team enabled">
    Once your agreement is in place, Fish Audio grants your team the artifacts
    it is entitled to: the Helm chart, the component images, and the All-in-One
    image where applicable.
  </Step>

  <Step title="Create a deploy token">
    Sign in to fish.audio and open **Developer → Self Host** to see the
    platforms and modes your team is granted and to create the token that
    authenticates against the registry. See
    [Registry & license](/developer-guide/self-hosting/registry-access).
  </Step>
</Steps>

## Next steps

<CardGroup>
  <Card title="Requirements" icon="microchip" href="/developer-guide/self-hosting/requirements">
    GPU, CPU, memory, storage, and platform baselines.
  </Card>

  <Card title="Registry & license" icon="key" href="/developer-guide/self-hosting/registry-access">
    Create a deploy token and authenticate Docker and Helm.
  </Card>

  <Card title="Kubernetes deployment" icon="dharmachakra" href="/developer-guide/self-hosting/kubernetes">
    Install, upgrade, roll back, and validate the Helm release.
  </Card>

  <Card title="All-in-One container" icon="box" href="/developer-guide/self-hosting/all-in-one">
    Run the whole stack from a single `docker run`.
  </Card>

  <Card title="Air-gapped deployments" icon="network-wired" href="/developer-guide/self-hosting/air-gapped">
    Mirror artifacts, account for usage offline, and prove zero egress.
  </Card>

  <Card title="Operations" icon="chart-line" href="/developer-guide/self-hosting/operations">
    Monitoring signals, scaling, backups, and troubleshooting.
  </Card>

  <Card title="Releases" icon="code-branch" href="/developer-guide/self-hosting/enterprise-releases">
    How versions are published, and which one to deploy.
  </Card>
</CardGroup>


# Kubernetes deployment
Source: https://docs.fish.audio/developer-guide/self-hosting/kubernetes

What the Helm delivery installs, what you decide before installing, and what to expect

Fish Audio Enterprise ships as a Helm chart that installs the whole speech stack into
a cluster you run. Both Helm profiles use the same chart and the same procedure;
they differ only in how usage is accounted.

<Note>
  This page covers what the deployment involves and what you need to decide. The
  commands, values, and troubleshooting are in the **deployment runbook**, which ships
  in the deployment guide download for the chart version you install. See
  [Releases](/developer-guide/self-hosting/enterprise-releases). The runbook is
  versioned with the chart; this page is not, so follow the runbook when they differ.
</Note>

## Choose a billing profile

| | Offline | Online |
| - | - | - |
| Usage accounting | Local signed ledger on shared storage | Validated and billed against the Fish Audio service |
| Runtime egress | None | The billing endpoint on 443 — plus the model-source endpoint, if you use one |
| Model assets | Served in-cluster by the bundled model warehouse | In-cluster by default; can instead pull from S3-compatible storage |

The bearer-token rules follow the profile: online requests must carry a
[Fish Audio API key](/developer-guide/getting-started/api-key) with credit, while
offline accepts any stable non-empty token and records it verbatim as the billing
identity — the same semantics as the
[All-in-One's usage accounting](/developer-guide/self-hosting/all-in-one#usage-accounting-and-tenancy).

Both profiles serve model weights from inside the cluster by default. The online
profile can point the workers at S3-compatible storage instead — Fish Audio's
hosted bucket or your own; the deployment runbook's "Model source" section covers
the how. The offline profile stays in-cluster, since it runs with no egress.
"Offline" means runtime-offline: installation still pulls
images and the chart from a registry. For a cluster with no network access at all, see
[Air-gapped deployments](/developer-guide/self-hosting/air-gapped).

## What gets installed

The chart deploys the edge API, the model API layer, an inference router and worker,
a vocoder, a text normalizer, the model warehouse that serves the weights when
you use the default in-cluster model source, and Redis.
Everything lands in one namespace, in the single-worker shape the delivery was sized
against.

Ingress and TLS are off by default. Put your own ingress controller or load balancer
in front of the edge API service, or enable the chart's if you want Kubernetes to
manage that layer.

## What you decide before installing

| Decision | Notes |
| - | - |
| Namespace | `fish-audio` is the supported default. A different one only needs Helm's `--namespace` argument. |
| Shared storage path | Mounted at the same path on **every** node. See [Requirements](/developer-guide/self-hosting/requirements#shared-storage). |
| GPU scheduling | Tolerations and node selectors, if your GPU nodes are tainted or you run more than one GPU model. |
| Metrics | The chart exposes Prometheus annotations and creates no ServiceMonitors, so an annotation-scraping Prometheus works as-is and kube-prometheus-stack needs a scrape config. |
| Reference voices | Requests carrying a `reference_id` resolve from a local archive, never from the network. One zip per voice, staged on shared storage before that traffic starts. |
| Replica counts | Talk to your account team first. The shipped shape is what the delivery was sized and validated against. |

Three Kubernetes Secrets have to exist before the install: registry credentials, the
license bundle, and one shared between the in-cluster model store and the workers that
read from it (with an external model source, that secret instead carries the real
access keys for the storage — the runbook covers it). The license bundle is the certificate and key pair downloaded from
**Developer → Self Host**; the services that enforce it stay down until the Secret
exists, and refuse requests once the certificate lapses — renewal is described in
[Operations](/developer-guide/self-hosting/operations#license-renewal). The model-store
secret authenticates nothing outside the cluster, so the value is yours to choose. For
production, prefer External Secrets, Sealed Secrets, or your cloud secret manager over
plain Secrets.

## What to expect

Image references are pinned in the chart, so you select a version with a Helm argument
rather than by editing tags.

The first install is slow: the inference worker compiles its GPU graphs and the
vocoder builds its inference engine before either becomes ready, which takes far longer
than any later rollout. Both artifacts are cached on shared storage, so subsequent
starts are quick, as long as that storage persists.

Upgrades, rollback, and uninstall are all standard Helm operations. The runbook covers
the order to do them in and what to check at each step.

## Next steps

* [Requirements](/developer-guide/self-hosting/requirements): hardware, platform, and network baselines
* [Registry & license](/developer-guide/self-hosting/registry-access): how your team gets the chart
* [Operations](/developer-guide/self-hosting/operations): running it once it is live


# Operations
Source: https://docs.fish.audio/developer-guide/self-hosting/operations

Monitoring, scaling, backups, and troubleshooting for a self-hosted deployment

Day-2 guidance for the Helm platform. For the single-container platform, see
[All-in-One container](/developer-guide/self-hosting/all-in-one).

## Ownership

| Area | Owner |
| - | - |
| Kubernetes cluster, node lifecycle, platform add-ons | You |
| Fish Audio chart and application configuration | Fish Audio |
| Secrets and credentials | You, with Fish Audio inputs where required |
| Monitoring and alerting | You. Fish Audio can advise on expected signals and thresholds |
| Incident response | Joint during the deployment window, yours after handoff |

Agree an escalation path before production traffic starts.

## Monitoring signals

Prometheus is the metrics baseline. Watch:

* Pod readiness and restart counts for every service in the release, including Redis.
* Request success rate, error rate, latency, and time-to-first-audio.
* Queue depth, where the application exposes it.
* GPU utilization and GPU memory.
* CPU and memory usage against the configured requests and limits.
* Shared storage usage and latency.
* Redis availability and latency.
* License validity and time to expiry, exported as metrics; the services also log
  warnings well before expiry.
* Image pull failures and Kubernetes events in the release namespace.

Centralize application logs and retain Kubernetes events long enough to
investigate a bad rollout. Redact secrets before sharing any logs outside your
environment.

## Scaling

Capacity is changed through the release configuration: API replicas, GPU worker
replicas, CPU and memory requests, GPU resource requests, and per-worker concurrency.
All of it takes effect on an upgrade. Scaling on load instead of by hand is included
and off by default; while it is on, the replica count is the autoscaler's rather than
the file's. Your deployment guide covers turning it on.

Add a GPU replica only when a GPU is actually free: a pod that requests one on a full
cluster stays `Pending` indefinitely.

Validate every scale change with the smoke test and a benchmark run at your
expected concurrency, and remember that the first start of a new GPU worker pays
the compile cost before it becomes ready.

## Backup and retention

Back up:

* Reference voice archives, if your traffic uses reference ids. In a self-hosted
  deployment these are durable data, not cache. Nothing can re-download them.
* The offline usage ledger directories, before they are pruned by your own
  archival process.
* The values file used for the production release.
* Secret manager entries.
* Dashboards and alert rules.

Compile and model caches on shared storage do not need backing up. Losing them
costs a slow first start, nothing more.

## License renewal

Update the license Secret in place with the renewed bundle; no restart is needed.
Kubernetes propagates the change to the running pods, and each service re-verifies on
its own short cycle, so expect up to a couple of minutes of mixed responses before
it settles. One exception: a bundle
that carries a new key rather than a re-signed certificate. The offline profile's edge API
reads the ledger-signing key only at startup, so restart that deployment after
replacing a key.

Pods with an expired certificate keep running but refuse requests, then recover once
a renewal propagates.

## Upgrades

Commands are in the deployment runbook. Before running them, capture the release
history so you know the revision to roll back to, and keep that revision available
until the new one is accepted.

## Troubleshooting

The failures a Kubernetes deployment actually produces, and where each one usually
comes from. The commands to diagnose each, and the fixes, are in the troubleshooting
guide that ships with your deployment guide.

| Symptom | Likely causes |
| - | - |
| `ImagePullBackOff` or `ErrImagePull` | Missing pull secret, a deploy token that was rotated or deleted, a tag that is not in your mirror, or blocked egress to the registry. |
| Pods stay `Pending` | Not enough CPU, memory, or GPU capacity; GPU node taints without matching tolerations; a node selector that matches nothing. |
| Pods stuck `ContainerCreating` | A Secret the pod mounts does not exist yet — most often the license bundle. The pods recover on their own once it is created. |
| GPU not available, or `nvidia-smi` fails in a pod | Driver missing or unhealthy, device plugin or GPU Operator not ready, container runtime not configured for NVIDIA, or an unsupported GPU for the driver stack. |
| hostPath volume errors, or missing model and cache paths | Shared storage is not mounted on that node, a newly added node never got the mount, the mount target is unreachable, or permissions prevent writes. |
| Requests fail after reaching the edge API, with Redis errors in its logs | The Redis master or replica is not ready, the service is missing, or a network policy blocks it. |
| `Reference not found` | The archive was never staged, the filename does not exactly match the reference id, the zip is malformed, or the reference root was changed without mounting the new path into every edge API pod. |
| Metrics missing from Prometheus | Scrape annotations disabled in values, Prometheus not configured for annotation-based discovery, a port mismatch, or a network policy blocking scrapes. |
| `CrashLoopBackOff` | A missing environment variable, a license file missing or unreadable inside the Secret, model assets unreachable, or a path the container cannot write. |
| Requests refused with a license error | The certificate expired, or a renewal has not finished propagating — see [License renewal](#license-renewal). |
| Time-to-first-audio high, GPU utilization low | Concurrency above what the current replicas can serve, cold caches after a restart, storage latency, or CPU saturation on the API nodes. |

## Escalating to Fish Audio

Include:

* Kubernetes version, cloud provider, region, node types, and GPU type.
* Chart version and the values file with secrets removed.
* Pod status, relevant events, and logs with secrets redacted.
* The exact command that failed, with its output.
* Timestamp and time zone.


# Registry & license
Source: https://docs.fish.audio/developer-guide/self-hosting/registry-access

The Self Host dashboard: deploy tokens, registry sign-in, and your license bundle

Self-hosted images, charts, and documentation are distributed from a private Fish Audio
registry. Your team authenticates to it with a **deploy token** created in the fish.audio
dashboard.

## The Self Host page

Sign in to fish.audio and open **Developer → Self Host**. The page opens with
**Before you start** — the platforms and modes your team is granted, with the
prerequisites it checks. If it is empty, or reports that self-host deployment
is not enabled, contact your account manager. Then four steps:

1. **Deploy token** — create the credential the install commands authenticate with.
2. **Sign in** — one registry login with that token.
3. **License** — download the certificate and key bundle the deployment needs
   before it will serve, at first install and at every renewal. Some contracts
   bind it to specific GPUs — have the card UUIDs ready when you request it.
4. **Install** — pick your mode and version; the download commands are built for
   your team, with the registry host, artifact references, and the version you pick
   already filled in. The same step downloads the **deployment guide** for that
   version — the runbook or guide the other pages here refer to. See
   [Releases](/developer-guide/self-hosting/enterprise-releases) for how versions work.

<Note>
  The registry host, the artifact references, and the versions available to you are
  specific to your team and are shown only in the dashboard. Copy them from the Self
  Host page.
</Note>

## Prerequisites

* Self-hosting enabled for your team under an enterprise agreement.
* A fish.audio account that is a member of that team.
* Docker, and Helm 3.8 or newer for the OCI chart commands.

## Create a deploy token

<Steps>
  <Step title="Open the Deploy token step">
    On **Developer → Self Host**, select **Create Deploy Token**.
  </Step>

  <Step title="Name it for where it will be used">
    Use a name that identifies the consumer, such as `prod-cluster` or `ci-mirror`. The
    name appears in the token list alongside the creation date and last-used time.
  </Step>

  <Step title="Copy the token immediately">
    The token value is shown once, at creation. Store it in your secret manager before
    closing the dialog. If you lose it, rotate the token to issue a new one.
  </Step>
</Steps>

A team can hold up to five deploy tokens at a time. Tokens carry the grants of the team
that owns them, not of the person who created them, and follow those grants as they
change, so there is no token to recreate when your entitlement is updated.

Authenticate with your account email as the username and the deploy token as the
password. Docker, Helm, and the cluster's image pull secret all use the same pair; the
exact commands are on the Self Host page and in the deployment runbook.

## Managing tokens

| Action | Effect |
| - | - |
| Rotate | Issues a new token value and invalidates the old one immediately. Update every consumer before rotating, or new pods will fail to pull. |
| Delete | Revokes the token immediately. Any deployment still using it stops pulling images. Running pods keep running until they are rescheduled. |
| Last used | Shows when the token last authenticated, which identifies tokens that are safe to retire. |

Recommended practice:

* Issue one token per consumer (production cluster, staging cluster, CI mirror) so a
  single revocation never takes down more than one of them.
* Store tokens in your secret manager, not in values files or version control.
* Rotate on your normal credential schedule and whenever someone with access to a token
  leaves the team.

## The license

The license is the product permit: every platform and billing mode needs one to
run, and it is separate from how usage is billed. Fish Audio issues the
license against your contract, and the dashboard's License step is
where you download the bundle — a certificate (`license.crt`) and key
(`license.key`) that the deployment mounts, as its guide's License section
shows. Some contracts bind the license to specific GPU UUIDs; the dashboard
shows the binding.

The two failure modes are worth knowing before they happen. **Missing**, the
services never come up. **Expired**, they start but refuse every request with a
license error (403) until the renewed bundle is in place.

**Validity and renewal.** The certificate carries a validity window bounded by
your contract term; the dashboard shows the current expiry. Renewals are driven
by Fish Audio: before the expiry, settle the period's invoice — and, on offline
billing, send the usage ledger for reconciliation — and a renewed license is
issued. A renewal replaces only `license.crt`, in place, and the running
services reload it without a restart; only a bundle that carries a new key
needs one. The Helm-side mechanics are in
[License renewal](/developer-guide/self-hosting/operations#license-renewal);
the All-in-One's are in its deployment guide.

## Next step

With access in place, continue to
[Kubernetes deployment](/developer-guide/self-hosting/kubernetes) or the
[All-in-One container](/developer-guide/self-hosting/all-in-one).


# Requirements
Source: https://docs.fish.audio/developer-guide/self-hosting/requirements

Hardware, platform, and network baselines for a self-hosted deployment

These are the baselines Fish Audio deploys against. Confirm final sizing with your
account team once your traffic profile and target GPU are known: time-to-first-audio
and throughput depend on the model, GPU, text length, and concurrency, and should be
measured on your own hardware before you commit to a capacity plan.

## Helm (Kubernetes)

Everything in this section — cluster topology, GPUs, shared storage, platform
baseline, and network — is for the Helm platform. The All-in-One's host requirements,
including its own network row, are the [next section](#all-in-one-docker).

### Cluster topology

| Node type | Minimum | Purpose |
| - | - | - |
| CPU-only control-plane or system nodes | 3 nodes, roughly 4 vCPU and 16 GB RAM each | Control-plane high availability on self-managed Kubernetes, or a system node group for platform add-ons on managed Kubernetes. |
| CPU workload capacity | Roughly 28 vCPU and 125 GiB of memory requests | Non-GPU workloads: Redis, the edge API, the text normalizer, the model API layer, and the inference router. Provide a CPU node group, or leave GPU nodes schedulable so these can land there. |
| GPU capacity | 2 GPUs | One for the inference worker, one for the vocoder. Spread them across two nodes if the deployment also has to survive a node drain or failure. |
| Shared storage | 1 TB usable, mounted on every node | Model cache, reference archives, compile caches, and the offline usage ledger. |
| Network | Low-latency east-west networking | GPU workload stability, service-to-service calls, and shared storage access. |

Verify these against your own chart version by rendering the release with a deployment
profile applied (the chart defaults alone stop on a required value) and summing the
requests.

### GPUs

| Requirement | Detail |
| - | - |
| Count | 2 GPUs minimum: one inference worker, one vocoder. Enabling timestamp alignment adds a third. |
| Type | NVIDIA H100 or H200. H200 is the preferred target for its larger memory and bandwidth. |
| Memory | The worker pre-allocates most of its card for the KV cache, so a near-full card is expected, not a fault. A larger card raises the concurrency ceiling rather than leaving headroom. |
| Newer architectures | Contact Fish Audio before standardizing on a GPU generation that is not H100 or H200, so the image, CUDA stack, and driver combination can be confirmed. |
| MIG | Disable MIG unless the configuration has been validated with Fish Audio. |

On AWS, `p5.48xlarge` (H100) is an acceptable baseline and `p5en.48xlarge` (H200)
is the preferred target. Validate GPU instance quota in the target region before
scheduling a deployment window.

### Shared storage

Provide at least 1 TB of usable shared storage backed by EFS, NFS, or an
equivalent service, mounted at the same path on every node that runs Fish Audio
workloads (`/mnt/share` by default, and configurable). The chart mounts it into containers with
hostPath volumes and creates no PersistentVolumeClaims.

<Warning>
  The hostPath mount is created with `DirectoryOrCreate`. A node that is missing
  the shared mount silently gets a local directory instead, and the deployment
  looks healthy while data splits across nodes. Verify the mount, and that it is
  writable, on **every** node before installing. The deployment runbook gives a check
  that does this without node access.
</Warning>

Confirm the following with your storage team: how the mount is applied to newly
added nodes, the throughput mode, the backup policy, mount target reachability,
and expected growth of the model and cache data.

Redis is installed by the chart and does not use shared storage. Its append-only
file is backed by node-local ephemeral storage, so provision ephemeral capacity
on the nodes that host it, plus headroom for rewrites.

### Platform baseline

| Layer | Recommendation |
| - | - |
| Kubernetes | A currently supported minor version. Stay one minor behind the newest release if your GPU add-ons have not been validated on it yet. |
| Helm | A current maintained release supported by your platform. |
| Container runtime | The provider-managed containerd runtime where available. |
| GPU runtime | NVIDIA GPU Operator, or the provider-managed driver and device plugin stack. GPU nodes must expose `nvidia.com/gpu`. |
| GPU telemetry | DCGM exporter or the provider equivalent. |
| Storage | EFS, NFS, or an equivalent shared filesystem mounted on every node. |
| Object storage | Not required by default: models are served from inside the cluster. The online profile can pull them from S3-compatible storage instead — see [Kubernetes deployment](/developer-guide/self-hosting/kubernetes). |
| Redis | Installed by the chart into the release namespace. Do not point the release at a shared Redis without discussing it first. |
| Metrics | Prometheus, scraping `prometheus.io/*` pod annotations. The chart creates no ServiceMonitors, so kube-prometheus-stack users must add an annotation-based scrape config. |
| Access layer | Yours to choose. Ingress controller, DNS, TLS, and load balancing are not part of the delivery. |
| License bundle | The certificate and key from **Developer → Self Host**, loaded into the cluster as a Secret before the install. |

The cluster also needs working in-cluster DNS, a dedicated namespace, node labels
and taints for GPU scheduling, and a Pod Security level in that namespace that
permits `hostPID` and `hostIPC`, which the inference workers require.

### Network

| Direction | Requirement |
| - | - |
| Install-time egress | Access to the Fish Audio registry to pull images and the chart. Mirror both into your own registry for air-gapped installs. |
| Runtime egress, offline profile | None. No model download and no billing call. |
| Runtime egress, online profile | HTTPS to the Fish Audio authorization and billing endpoint. Models are in-cluster by default; if you point the workers at an external model source, allowlist that endpoint too. Ask your account team for the billing hostname to allowlist. Enabling the timestamp aligner adds one more host. |
| Ingress | Customer-approved ingress or a private endpoint, with DNS and TLS in place before production traffic. |

## All-in-One (Docker)

| Requirement | Detail |
| - | - |
| OS | Linux x86-64. |
| GPUs | 2 GPUs, no NVLink required — one for the inference worker, one for the vocoder. The Single-GPU mode is a special case: one H100-class 80 GB card shared by both, with lower throughput. |
| NVIDIA driver | Must support CUDA 13.x and your card's compute capability. This applies to the Helm platform too: a driver capped at CUDA 12 reports a healthy GPU and then fails the workloads. |
| Docker | Docker Engine 24 or newer. |
| Container toolkit | NVIDIA Container Toolkit installed and the `nvidia` runtime registered, so `--gpus all` exposes GPUs. |
| RAM | 128 GiB minimum, 192 GiB recommended. The container runs the whole stack in one process tree. |
| CPU | 32 vCPU minimum, 48 to 64 recommended. |
| Disk | 100 GB free if the host pulls the image itself; 120 GB if you move it as a file, since `docker load` needs the \~30 GB archive and the \~80 GB unpacked image at once. Plus the compile and engine caches. |
| Network | Offline mode: none at runtime. Online mode: outbound HTTPS to the Fish Audio authorization and billing endpoint, on every request. Ask your account team for the hostname to allowlist. |
| License bundle | The certificate and key pair from **Developer → Self Host**, in a host directory readable by the container's non-root user. |

## Next step

Once the platform checks pass, authenticate to the Fish Audio registry in
[Registry & license](/developer-guide/self-hosting/registry-access).


# Manage Voices
Source: https://docs.fish.audio/features/manage-voices

List, inspect, update, and delete your voice models

Every voice you [clone](/features/voice-cloning) becomes a model you own. List your library, look up a model's details, rename or re-share it, and delete what you no longer need, all from the API directly, the Python library, or JavaScript.

<CardGroup>
  <Card title="Use it in the web app" icon="browser" href="https://fish.audio/app/my-voices">
    No code: manage voices in the browser.
  </Card>

  <Card title="API reference" icon="brackets-curly" href="/api-reference/endpoint/model/list-models">
    Every endpoint for voice models.
  </Card>

  <Card title="Cookbooks" icon="book-open" href="/developer-guide/sdk-guide/cookbook/discover-library-voice">
    Search and reuse Library voices.
  </Card>
</CardGroup>

## When to use it

<CardGroup>
  <Card title="Browse your library" icon="folder-open">
    List the voices you've created or saved.
  </Card>

  <Card title="Find a voice id" icon="magnifying-glass">
    Look up the `reference_id` to use in [Text to Speech](/features/text-to-speech).
  </Card>

  <Card title="Update metadata" icon="pen">
    Rename, re-tag, or change a model's visibility.
  </Card>

  <Card title="Clean up" icon="trash">
    Delete models you no longer use.
  </Card>
</CardGroup>

## List your voices

Page through your library. The response carries the `total` count and the `items` for the current page.

<CodeGroup>
  ```python Python theme={null}
  from fishaudio import FishAudio

  client = FishAudio()  # reads FISH_API_KEY

  page = client.voices.list(self_only=True, page_size=20)

  print(f"{page.total} voices")
  for v in page.items:
      print(v.id, v.title, v.state, v.visibility)
  ```

  ```bash API (curl) theme={null}
  curl "https://api.fish.audio/model?self=true&page_size=20" \
    --header "Authorization: Bearer $FISH_API_KEY"

  # Response: { "total": 42, "items": [ ... ], "has_more": true }
  ```

  ```javascript JavaScript theme={null}
  import { FishAudioClient } from "fish-audio";

  const client = new FishAudioClient({ apiKey: process.env.FISH_API_KEY });

  const page = await client.voices.search({ page_size: 20 });

  console.log(`${page.total} voices`);
  for (const v of page.items) {
    console.log(v._id ?? v.id, v.title, v.state, v.visibility);
  }
  ```
</CodeGroup>

## Get, update, and delete

Use a voice **id** to inspect a single model, change its metadata, or remove it.

<CodeGroup>
  ```python Python theme={null}
  # Inspect one model
  voice = client.voices.get("YOUR_VOICE_ID")
  print(voice.title, voice.state)

  # Update metadata (only the fields you pass change)
  client.voices.update(
      "YOUR_VOICE_ID",
      title="Updated title",
      visibility="unlist",
  )

  # Delete
  client.voices.delete("YOUR_VOICE_ID")
  ```

  ```bash API (curl) theme={null}
  # Inspect one model
  curl https://api.fish.audio/model/YOUR_VOICE_ID \
    --header "Authorization: Bearer $FISH_API_KEY"

  # Update metadata
  curl --request PATCH https://api.fish.audio/model/YOUR_VOICE_ID \
    --header "Authorization: Bearer $FISH_API_KEY" \
    --header "Content-Type: application/json" \
    --data '{ "title": "Updated title", "visibility": "unlist" }'

  # Delete
  curl --request DELETE https://api.fish.audio/model/YOUR_VOICE_ID \
    --header "Authorization: Bearer $FISH_API_KEY"
  ```
</CodeGroup>

## Implementation details

### Filtering and pagination

Narrow the list with `title`, `tags`, or `language`, and page with `page_size` and `page_number`. Omit `self_only` (API: `self`) to search the public [Voice Library](/overview/platform) instead of just your own models.

<CodeGroup>
  ```python Python theme={null}
  page = client.voices.list(
      self_only=True,
      title="narration",
      page_size=50,
      page_number=2,
  )
  ```

  ```bash API (curl) theme={null}
  curl "https://api.fish.audio/model?self=true&title=narration&page_size=50&page_number=2" \
    --header "Authorization: Bearer $FISH_API_KEY"
  ```
</CodeGroup>

### Visibility

Switch a model between `private`, `unlist` (shareable link), and `public` (listed in the Voice Library) with `update`. Publishing a model lets anyone use it as a `reference_id`.

## Going further

<CardGroup>
  <Card title="Clone a voice" icon="clone" href="/features/voice-cloning">
    Create a new model from audio samples.
  </Card>

  <Card title="Speak with a voice" icon="microphone" href="/features/text-to-speech">
    Use any voice id as `reference_id`.
  </Card>

  <Card title="Voice Models API" icon="book-open" href="/api-reference/endpoint/model/list-models">
    Every endpoint for listing and managing models.
  </Card>

  <Card title="Python reference" icon="python" href="/api-reference/sdk/python/resources">
    The full `voices` resource surface.
  </Card>
</CardGroup>


# Realtime Streaming
Source: https://docs.fish.audio/features/realtime-streaming

Stream audio as it generates for the lowest latency

Start playing audio before the whole clip is ready. Fish Audio streams speech in chunks, so your users hear the first words in a fraction of a second, essential for voice agents and live narration. Two modes: **HTTP streaming** for text you already have, and **WebSocket** for text that arrives incrementally (like LLM tokens).

<CardGroup>
  <Card title="API reference" icon="brackets-curly" href="/api-reference/endpoint/websocket/tts-live">
    The live TTS WebSocket protocol.
  </Card>

  <Card title="Cookbooks" icon="book-open" href="/developer-guide/sdk-guide/cookbook/realtime-llm-to-speech">
    LLM-to-speech and voice agents.
  </Card>

  <Card title="Best practices" icon="gauge-high" href="/developer-guide/best-practices/real-time-streaming">
    Tuning latency for production.
  </Card>
</CardGroup>

## When to use it

<CardGroup>
  <Card title="Voice agents" icon="headset">
    Conversational AI where time-to-first-audio matters.
  </Card>

  <Card title="LLM to speech" icon="robot">
    Speak tokens as your model produces them.
  </Card>

  <Card title="Live narration" icon="tower-broadcast">
    Long-form content that should start playing immediately.
  </Card>

  <Card title="Interactive apps" icon="bolt">
    Anywhere a few hundred milliseconds of latency is noticeable.
  </Card>
</CardGroup>

## Stream text you already have

When you have the full string, stream the audio chunks as they generate and write or play them immediately.

<CodeGroup>
  ```python Python theme={null}
  from fishaudio import FishAudio

  client = FishAudio()  # reads FISH_API_KEY

  with open("out.mp3", "wb") as f:
      for chunk in client.tts.stream(text="Streaming keeps latency low."):
          f.write(chunk)  # or send to a speaker / socket as it arrives

  # Or collect the whole stream into one bytes object:
  audio = client.tts.stream(text="Streaming keeps latency low.").collect()
  ```

  ```bash API (curl) theme={null}
  curl --request POST https://api.fish.audio/v1/tts \
    --header "Authorization: Bearer $FISH_API_KEY" \
    --header "Content-Type: application/json" \
    --header "model: s2-pro" \
    --no-buffer \
    --data '{ "text": "Streaming keeps latency low.", "format": "mp3" }' \
    --output out.mp3
  ```

  ```javascript JavaScript theme={null}
  import { FishAudioClient } from "fish-audio";
  import { createWriteStream } from "fs";

  const client = new FishAudioClient({ apiKey: process.env.FISH_API_KEY });

  // convert() returns a ReadableStream<Uint8Array>. Write each chunk the
  // moment it arrives instead of waiting for the whole clip.
  const stream = await client.textToSpeech.convert(
    { text: "Streaming keeps latency low.", format: "mp3" },
    "s2-pro"
  );

  const file = createWriteStream("out.mp3");
  for await (const chunk of stream) {
    file.write(Buffer.from(chunk)); // or forward to a speaker / socket as it arrives
  }
  file.end();
  ```
</CodeGroup>

`--no-buffer` tells curl to write each chunk as it arrives instead of waiting for the full response.

## Stream from an LLM

When text arrives token by token, feed a generator to `stream_websocket`. It opens a WebSocket, sends text as you produce it, and yields audio chunks back, so speech keeps pace with your model.

<CodeGroup>
  ```python Python theme={null}
  from fishaudio import FishAudio
  from fishaudio.utils import play

  client = FishAudio()

  def llm_tokens():
      # Replace with your real streaming LLM call
      for token in ["The ", "first ", "move ", "sets ", "everything ", "in ", "motion."]:
          yield token

  for chunk in client.tts.stream_websocket(llm_tokens(), reference_id="YOUR_VOICE_ID"):
      play(chunk)  # play each chunk the moment it arrives
  ```

  ```bash API (WebSocket) theme={null}
  # Token-level streaming uses the WebSocket endpoint, not curl.
  # The Python SDK's stream_websocket() handles the protocol for you.
  # To build it directly, see the WebSocket reference:
  #   /api-reference/endpoint/websocket/tts-live
  ```
</CodeGroup>

## Implementation details

### Which mode to use

* **HTTP streaming (`tts.stream`)**: you have the full text up front and want low time-to-first-audio. Simplest option.
* **WebSocket (`tts.stream_websocket`)**: text is still being produced (LLM output, live captions). Lets you start speaking before the sentence is finished.

### Lower the latency further

* Use a streaming-friendly format like `mp3` or `pcm`.
* Keep the connection warm for back-to-back generations.
* Pair with a cloned voice via `reference_id` (see [Voice Cloning](/features/voice-cloning)).

## Control where audio generates

The WebSocket buffers incoming text and generates audio once it has enough context for natural-sounding speech, so you don't need to batch tokens yourself. When you *do* want a clean break (end of a sentence, a deliberate pause, or the end of a turn), yield a `FlushEvent` to force generation immediately. Wrap text in a `TextEvent` if you prefer explicit events over bare strings.

```python theme={null}
from fishaudio import FishAudio
from fishaudio.types import TextEvent, FlushEvent

client = FishAudio()

def script():
    yield TextEvent(text="First sentence. ")
    yield "Second sentence. "
    yield FlushEvent()        # generate everything buffered so far, now
    yield "Third sentence."

for chunk in client.tts.stream_websocket(script(), reference_id="YOUR_VOICE_ID"):
    ...  # play or forward each chunk
```

## Tune latency vs. quality

Both streaming paths take a `latency` mode:

* `latency="balanced"` (default): lowest time-to-first-audio. Use it for voice agents and live LLM output.
* `latency="normal"`: slightly higher latency, best audio quality. Use it for narration where you can afford a beat.

```python theme={null}
for chunk in client.tts.stream_websocket(llm_tokens(), latency="balanced"):
    ...
```

For finer control, pass a `TTSConfig` with chunk tuning. Smaller chunks emit audio sooner (lower latency); larger chunks give the model more context (smoother prosody):

```python theme={null}
from fishaudio.types import TTSConfig

config = TTSConfig(
    latency="balanced",
    chunk_length=200,       # target tokens per generated chunk
    min_chunk_length=100,   # don't emit a chunk shorter than this
)

for chunk in client.tts.stream(text="...", config=config):
    ...
```

## Stream asynchronously

For asyncio apps, `AsyncFishAudio` exposes the same streaming methods. `stream_websocket` accepts an async generator, so you can pipe an async LLM client straight into speech.

```python theme={null}
import asyncio
from fishaudio import AsyncFishAudio

async def main():
    client = AsyncFishAudio()

    async def llm_tokens():
        async for token in your_async_llm():
            yield token

    # stream_websocket is an async generator: iterate it, don't await the call
    async for chunk in client.tts.stream_websocket(
        llm_tokens(), reference_id="YOUR_VOICE_ID", latency="balanced"
    ):
        ...  # play or forward each chunk

asyncio.run(main())
```

## Direct API (no SDK)

Token-level streaming runs over the WebSocket endpoint. The SDK's `stream_websocket()` handles framing for you. To speak the protocol directly, send MessagePack frames over the socket; the same `application/msgpack` payload format also works for one-shot HTTP streaming, which is faster to serialize than JSON for large reference audio:

```python theme={null}
import os
import httpx
import ormsgpack

payload = {"text": "Streaming keeps latency low.", "format": "mp3", "latency": "balanced"}

with httpx.stream(
    "POST",
    "https://api.fish.audio/v1/tts",
    headers={
        "Authorization": f"Bearer {os.environ['FISH_API_KEY']}",
        "Content-Type": "application/msgpack",
        "model": "s2-pro",
    },
    content=ormsgpack.packb(payload),
) as r:
    for chunk in r.iter_bytes():
        ...  # write each chunk as it arrives
```

For the full WebSocket frame sequence, see the [live TTS protocol reference](/api-reference/endpoint/websocket/tts-live).

## Going further

<CardGroup>
  <Card title="Text to Speech" icon="microphone" href="/features/text-to-speech">
    Voices, formats, and prosody for every generation.
  </Card>

  <Card title="WebSocket reference" icon="plug" href="/api-reference/endpoint/websocket/tts-live">
    The live TTS protocol, message by message.
  </Card>

  <Card title="Streaming best practices" icon="gauge-high" href="/developer-guide/best-practices/real-time-streaming">
    Tuning latency for production voice apps.
  </Card>

  <Card title="Python reference" icon="python" href="/api-reference/sdk/python/resources">
    `tts.stream` and `tts.stream_websocket`.
  </Card>
</CardGroup>


# Speech to Text
Source: https://docs.fish.audio/features/speech-to-text

Transcribe audio with transcribe-1-pro, including word-level timestamps, speaker turns for multi-speaker conversations, long recordings, and emotion cues

Turn spoken audio into text with `transcribe-1-pro`, Fish Audio's recommended automatic speech recognition (ASR) model. Send an audio file to `POST /v1/asr` with the header `model: transcribe-1-pro` to receive a transcript, its duration, and optional word-level timestamps. Pro handles multi-speaker conversations and recordings up to 60 minutes: it marks speakers inline in the transcript, returns a structured list of speaker turns when you request timestamps, and keeps emotion and vocal-event cues.

<CardGroup>
  <Card title="Use it in the web app" icon="browser" href="https://fish.audio/app/speech-to-text">
    No code: upload audio, get a transcript.
  </Card>

  <Card title="API reference" icon="brackets-curly" href="/api-reference/endpoint/openapi-v1/speech-to-text">
    Every parameter for `POST /v1/asr`.
  </Card>

  <Card title="Cookbooks" icon="book-open" href="/developer-guide/sdk-guide/cookbook/transcribe-to-captions">
    Captions, batch transcription, and more.
  </Card>
</CardGroup>

## Choose an ASR model

| Model | Use it for |
| - | - |
| `transcribe-1-pro` (recommended) | Short clips, multi-speaker conversations, and long recordings (up to 60 minutes). Speaker markers and speaker turns, with emotion and vocal-event cues preserved. |
| `transcribe-1` | General transcription of short recordings. Also used when the `model` header is missing or does not match. |

Select the model with the **`model` HTTP header**, and send `model: transcribe-1-pro` with every request; `model` is not a form or body field. Write the value exactly, in lowercase. If the header is missing, or its value is not an exact match (for example `Transcribe-1-Pro` or `transcribe-1pro`), the request is served and billed as `transcribe-1`, and no error is returned. If you expected `transcribe-1-pro` but the transcript has no speaker markers, check the header. See [ASR pricing](/developer-guide/models-pricing/pricing-and-rate-limits#automatic-speech-recognition-asr-models) for usage costs.

`transcribe-1-pro` accepts these fields:

| Field | Purpose |
| - | - |
| `audio` | Required. Exactly one audio file per request. See [Input audio](#input-audio). |
| `language` | Optional language hint. See [Language](#language). |
| `ignore_timestamps` | Default `true`. Set to `false` for word-level timestamps and speaker turns. See [Read the timestamps](#read-the-timestamps). |
| `tag_audio_events` | Default `true`. Set to `false` to remove emotion and vocal-event cues. See [Emotion and vocal-event cues](#emotion-and-vocal-event-cues). |
| `diarize` | `auto` (default), `true`, or `false`. Set to `false` to omit `speaker_turns`. See [Speaker options](#speaker-options). |
| `num_speakers`, `min_speakers`, `max_speakers` | Speaker-count hints, applied on a best-effort basis. See [Speaker options](#speaker-options). |

If you use `transcribe-1`, select it with `model: transcribe-1` and send only `audio`, `language`, and `ignore_timestamps`. The other fields apply to `transcribe-1-pro` only.

## When to use it

<CardGroup>
  <Card title="Captions & subtitles" icon="closed-captioning" href="/developer-guide/sdk-guide/cookbook/transcribe-to-captions">
    Group word-level timestamps into SRT/VTT cues.
  </Card>

  <Card title="Meeting & call notes" icon="users">
    Use Pro to transcribe multi-speaker recordings for summaries and search.
  </Card>

  <Card title="Voice commands & notes" icon="microphone-lines">
    Turn short utterances into text your app can act on.
  </Card>

  <Card title="Accessibility" icon="universal-access">
    Make audio and video content readable.
  </Card>
</CardGroup>

## Quick start

Create an [API key](/developer-guide/getting-started/api-key) and set `FISH_API_KEY` in your environment. For the Python examples, [install the Fish Audio SDK](/developer-guide/sdk-guide/quickstart).

Every example on this page selects `transcribe-1-pro`. The examples below also request timestamps.

<CodeGroup>
  ```python Python theme={null}
  from fishaudio import FishAudio
  from fishaudio.core import RequestOptions

  client = FishAudio()  # reads FISH_API_KEY

  with open("speech.wav", "rb") as f:
      result = client.asr.transcribe(
          audio=f.read(),
          include_timestamps=True,
          request_options=RequestOptions(
              timeout=900,  # long recordings can take several minutes
              additional_headers={"model": "transcribe-1-pro"},
          ),
      )

  print(result.text)

  ```

  ```bash API (curl) theme={null}
  curl --request POST https://api.fish.audio/v1/asr \
    --header "Authorization: Bearer $FISH_API_KEY" \
    --header "model: transcribe-1-pro" \
    --form audio=@speech.wav \
    --form ignore_timestamps=false
  ```

  ```javascript JavaScript (Node.js 20+) theme={null}
  import { readFile } from "node:fs/promises";

  const form = new FormData();
  form.append("audio", new Blob([await readFile("speech.wav")]), "speech.wav");
  form.append("ignore_timestamps", "false");

  const response = await fetch("https://api.fish.audio/v1/asr", {
    method: "POST",
    headers: {
      Authorization: `Bearer ${process.env.FISH_API_KEY}`,
      model: "transcribe-1-pro",
    },
    body: form,
  });

  if (!response.ok) throw new Error(await response.text());
  const result = await response.json();
  console.log(result.text);
  ```
</CodeGroup>

The response contains `text`, `duration` (seconds), `segments`, and `request_id`, plus [`speaker_turns`](#speaker-turns) when timestamps are requested, and `language_code` and `language` when the language is known. Do not depend on the order of keys in the JSON. If you use `transcribe-1`, rely on `text`, `duration`, `segments`, and the language fields.

The Python SDK currently returns only `text`, `duration`, and `segments`. To read the other fields, call the API directly (see [Direct API](#direct-api-messagepack)).

<Note>
  The Python SDK selects Pro through `RequestOptions.additional_headers`;
  `asr.transcribe()` has no `model` argument, and it cannot send the
  `transcribe-1-pro` fields such as `diarize`. The JavaScript example calls the
  REST API directly so the header is explicit. For multipart uploads, let your
  HTTP client set `Content-Type` and its boundary.
</Note>

## Read the timestamps

Request timestamps with `ignore_timestamps=false`. The default is `true`, which skips them. Timestamps add processing time. In multipart forms, send `true` or `false`: any other value, including `1` or an empty value, is read as `false` and turns timestamps on.

Each segment is `{ "text", "start", "end" }`, with times in **seconds**. Segments are word-level: a segment is usually one word, or in Chinese and Japanese usually one character or a few. Segment text has no punctuation and no speaker markers or cues, and can be normalized (for example `35` for `3.5`), so it does not always match `text` character for character. A segment can have `start` equal to `end`.

`segments` is an empty array, never omitted, when you do not request timestamps, when no speech is found, or when timing is temporarily unavailable. Segments are not speaker turns; to see who spoke when, use [`speaker_turns`](#speaker-turns).

<CodeGroup>
  ```python Python theme={null}
  result = client.asr.transcribe(
      audio=audio_bytes,
      include_timestamps=True,
      request_options=RequestOptions(
          additional_headers={"model": "transcribe-1-pro"}
      ),
  )

  print(f"{result.duration:.1f}s total")
  for seg in result.segments:
      print(f"[{seg.start:6.2f} - {seg.end:6.2f}] {seg.text}")

  ```

  ```bash API (curl) theme={null}
  curl --request POST https://api.fish.audio/v1/asr \
    --header "Authorization: Bearer $FISH_API_KEY" \
    --header "model: transcribe-1-pro" \
    --form audio=@speech.wav \
    --form ignore_timestamps=false | jq '.segments'

  # Each segment: { "text": "One", "start": 0.0, "end": 0.24 }
  ```
</CodeGroup>

For captions, group consecutive segments into caption-sized cues instead of writing one cue per word. The [captions cookbook](/developer-guide/sdk-guide/cookbook/transcribe-to-captions) shows how.

<Note>
  In the Python SDK, segment timestamps are **on by default**. Pass
  `include_timestamps=False` to skip them. That's the *inverse* of the
  API/JavaScript flag `ignore_timestamps`.
</Note>

## Multi-speaker conversations

`transcribe-1-pro` supports recordings with multiple speakers, such as interviews, meetings, and calls. Send the recording as one audio file. Speaker labels are consistent within one response, not across requests, so do not split a conversation into several requests.

### Speaker markers

Pro identifies speakers with **inline `<|speaker:N|>` markers in the response's `text` string**, where `N` is a numeric speaker label. A marker assigns the following text to that speaker until the next marker. The same label can appear again when that speaker resumes talking.

Markers usually have a space on each side (`<|speaker:0|> 你好。 <|speaker:1|> ...`); do not rely on exact spacing. Any text before the first marker belongs to the first turn. A transcript with no marker comes from a single speaker; treat it as speaker 0.

For example, this illustrative response contains three turns from two speakers. It uses `ignore_timestamps=true`, so `segments` is empty:

```json theme={null}
{
  "text": "<|speaker:0|> 你好。 <|speaker:1|> [高兴]很开心认识你。 <|speaker:0|> 我也是。",
  "duration": 6.4,
  "segments": [],
  "language_code": "zh",
  "language": "Chinese",
  "request_id": "0b6f4c1e-7d2a-4e8b-9f3c-5a1d2e3f4b6c"
}
```

| Marker | How to read this example |
| - | - |
| `<\|speaker:0\|>` | Speaker 0 says `你好。`, then later `我也是。`. |
| `<\|speaker:1\|>` | Speaker 1 says `很开心认识你。`, with the emotion cue `[高兴]` (happy). |

Treat speaker labels as identifiers within that response, not as names or identities you can match across separate requests. To display turns without requesting timestamps, split `text` at the speaker markers, keeping any emotion cues in each turn. For example, after decoding the JSON response into `result`:

```python theme={null}
import re

parts = re.split(r"<\|speaker:(\d+)\|>", result["text"])
lead, rest = parts[0].strip(), parts[1:]
turns = [(rest[i], rest[i + 1].strip()) for i in range(0, len(rest), 2)]
if not turns:
    turns = [("0", lead)]  # no marker: a single speaker
elif lead:
    turns[0] = (turns[0][0], f"{lead} {turns[0][1]}".strip())
for speaker, text in turns:
    if text:
        print(f"Speaker {speaker}: {text}")
```

```text theme={null}
Speaker 0: 你好。
Speaker 1: [高兴]很开心认识你。
Speaker 0: 我也是。
```

When you request timestamps, prefer `speaker_turns` over parsing `text`.

### Speaker turns

With `ignore_timestamps=false`, `transcribe-1-pro` also returns `speaker_turns`, a list of who spoke when. The field is present when `ignore_timestamps=false` and `diarize` is not `false`; otherwise it is absent. Speaker identification is a `transcribe-1-pro` feature.

This illustrative response is the same conversation with timestamps requested:

```json theme={null}
{
  "text": "<|speaker:0|> 你好。 <|speaker:1|> [高兴]很开心认识你。 <|speaker:0|> 我也是。",
  "duration": 6.4,
  "segments": [
    { "text": "你", "start": 0.32, "end": 0.56 },
    { "text": "好", "start": 0.56, "end": 0.88 },
    { "text": "很", "start": 1.84, "end": 2.04 },
    { "text": "开", "start": 2.04, "end": 2.24 },
    { "text": "心", "start": 2.24, "end": 2.48 },
    { "text": "认", "start": 2.48, "end": 2.68 },
    { "text": "识", "start": 2.68, "end": 2.88 },
    { "text": "你", "start": 2.88, "end": 3.2 },
    { "text": "我", "start": 4.56, "end": 4.8 },
    { "text": "也", "start": 4.8, "end": 5.0 },
    { "text": "是", "start": 5.0, "end": 5.36 }
  ],
  "speaker_turns": [
    { "speaker": "speaker:0", "text": "你好。", "start": 0.32, "end": 0.88 },
    {
      "speaker": "speaker:1",
      "text": "[高兴]很开心认识你。",
      "start": 1.84,
      "end": 3.2
    },
    { "speaker": "speaker:0", "text": "我也是。", "start": 4.56, "end": 5.36 }
  ],
  "language_code": "zh",
  "language": "Chinese",
  "request_id": "0b6f4c1e-7d2a-4e8b-9f3c-5a1d2e3f4b6c"
}
```

* `speaker_turns` lists the turns in the order they occur, and is `[]` when no speech was found. Each turn is `{ "speaker", "text", "start", "end" }`.
* `speaker` is the string `speaker:N`, where `N` matches the `<|speaker:N|>` marker in `text`. Labels identify speakers within one response only. They are not names, and the same label in two requests is not the same person.
* `text` is that turn's speech without speaker markers. It keeps emotion and event cues unless `tag_audio_events=false`.
* `start` and `end` are in seconds and come from the word timestamps. If word timing is unavailable (`segments` is empty), turn times are approximate and can cover the whole recording.
* Consecutive turns can have the same speaker. Do not assume turns are contiguous or non-overlapping.

### Speaker options

These `transcribe-1-pro` fields control speaker identification:

* `diarize`: `auto` (default) or `true` returns `speaker_turns` when timestamps are requested. `false` omits `speaker_turns`. It does not change the transcript, and the speaker markers stay in `text`.
* `num_speakers`: the expected number of speakers. This is a hint, applied on a best-effort basis; it guides speaker identification on longer recordings and may have no effect on short ones. It cannot be combined with `min_speakers` or `max_speakers`.
* `min_speakers`, `max_speakers`: bounds on the number of speakers, with the same best-effort rule. `min_speakers` must not exceed `max_speakers`.

Speaker counts are integers of 1 or more. Speaker-count hints cannot be sent with `diarize=false`. Invalid values return 400 with the code `invalid_parameter`.

For example, to transcribe an interview with two speakers and print each turn:

<CodeGroup>
  ```python Python (httpx) theme={null}
  import os

  import httpx

  with open("interview.mp3", "rb") as f:
      resp = httpx.post(
          "https://api.fish.audio/v1/asr",
          headers={
              "Authorization": f"Bearer {os.environ['FISH_API_KEY']}",
              "model": "transcribe-1-pro",
          },
          files={"audio": ("interview.mp3", f)},
          data={"ignore_timestamps": "false", "num_speakers": "2"},
          # Long recordings can take several minutes to process.
          timeout=httpx.Timeout(900.0, connect=10.0),
      )
  resp.raise_for_status()
  result = resp.json()

  for turn in result.get("speaker_turns", []):
      print(f"[{turn['start']:7.2f}s] {turn['speaker']}: {turn['text']}")
  ```

  ```bash API (curl) theme={null}
  curl --request POST https://api.fish.audio/v1/asr \
    --header "Authorization: Bearer $FISH_API_KEY" \
    --header "model: transcribe-1-pro" \
    --form audio=@interview.mp3 \
    --form ignore_timestamps=false \
    --form num_speakers=2 | jq '.speaker_turns'
  ```
</CodeGroup>

## Emotion and vocal-event cues

Pro can retain cues about how speech sounds alongside the spoken words. These appear as inline bracketed text, such as `[高兴]` (happy) or `[laughter]`, in the response's `text` field and in the turn text of `speaker_turns`. For example, a transcript might contain:

```text theme={null}
你好，[高兴]很开心认识你
```

These are annotations inferred from the audio, not words you need to add to the request. The API has no separate `emotion` field or fixed emotion enum; preserve the returned text when your application needs these cues.

To get a transcript without cues, send `tag_audio_events=false` (`transcribe-1-pro`). Cues are then removed from `text` and `speaker_turns`; timestamps and billing do not change.

Word timestamps ignore speaker markers and cues. Read `text` for the annotated transcript and `segments` for timed speech; the segment text may therefore differ from the full transcript.

## Implementation details

### Language

`language` is optional. The language is detected automatically whether or not you send a hint, and a hint does not force the transcript into that language. With `transcribe-1-pro`, the hint does not change the transcript; it matters only when the language cannot be determined, as described below.

Use a lowercase ISO 639-1 code such as `en`, `zh`, or `ja`. Other forms, such as `en-US` or `English`, may be rejected with 400.

The response reports the language in two fields, which are omitted when unknown (never `null`):

* `language_code`: the two-letter ISO 639-1 code, such as `en`. Use it in application logic.
* `language`: the language's English name, such as `English` or `Chinese`. Use it for display.

If the language cannot be determined (for example, very short audio), `language_code` reports your hint, which is not checked against the audio, and `language` may be absent. Responses report one language: for recordings that switch languages, `language` and `language_code` do not list every language spoken. The Python SDK does not return these fields; call the API directly to read them.

<CodeGroup>
  ```python Python theme={null}
  pro = RequestOptions(additional_headers={"model": "transcribe-1-pro"})

  # Auto-detect
  result = client.asr.transcribe(audio=audio_bytes, request_options=pro)

  # Provide a language hint
  result = client.asr.transcribe(
      audio=audio_bytes, language="zh", request_options=pro
  )

  ```

  ```bash API (curl) theme={null}
  # Omit the form field to auto-detect, or set it explicitly:
  curl --request POST https://api.fish.audio/v1/asr \
    --header "Authorization: Bearer $FISH_API_KEY" \
    --header "model: transcribe-1-pro" \
    --form audio=@speech.wav \
    --form language=zh
  ```
</CodeGroup>

### Input audio

`transcribe-1-pro` accepts WAV, MP3, AAC (including M4A/MP4), FLAC, Ogg (Opus or Vorbis), WebM/Matroska, and MOV, including browser recordings, and uses the first audio track of a video file. Send the original file bytes; no conversion is needed.

Not supported: AIFF, CAF, WMA, AMR, AC-3, and raw (headerless) PCM. These return 400.

Send audio as `multipart/form-data` (a file upload, shown above) or `application/msgpack` (see [Direct API](#direct-api-messagepack)). Base64-encoded audio in a JSON body is not supported.

If you use `transcribe-1`, send WAV, MP3, AAC (including M4A/MP4), FLAC, or Ogg (Opus or Vorbis), and convert WebM recordings (for example, from a browser's MediaRecorder) to Ogg/Opus, MP3, or WAV first, or use `transcribe-1-pro`.

### Limits

These limits apply to `transcribe-1-pro`:

* Recordings up to 60 minutes long. Longer audio returns 400 `audio_too_long`.
* For long recordings, send compressed audio such as MP3, Opus, or AAC. An hour of 128 kbps MP3 is about 55 MiB. A request that is too large returns 413.
* Very short clips (under about 0.08 seconds) return 400 `audio_too_short`.

If you use `transcribe-1`, send up to 50 MiB per request and keep MP3 and Opus files under 25 MiB; a request that exceeds the size limit returns 413 or 400. `transcribe-1` is designed for short recordings; for recordings longer than a few minutes, use `transcribe-1-pro`. A long request can fail with 503 if it exceeds the processing-time limit. Retry, and if it keeps failing, use `transcribe-1-pro` or split the audio.

### Long recordings

With `transcribe-1-pro`, send the whole recording, up to 60 minutes, in one request. Timestamps and speaker labels cover the whole recording. Do not split conversations: speaker labels are consistent only within one response. Long recordings can take several minutes, so raise your [client timeout](#processing-time-and-timeouts) and retry on 5xx errors. You are billed for the full duration of the recording once.

If you use `transcribe-1`, keep each request short. For a long recording, use `transcribe-1-pro`, or split the audio into shorter clips, transcribe each, and offset each clip's `start`/`end` by where it began in the full recording. Check that each response covers the complete clip before combining transcripts.

### Processing time and timeouts

Each request returns only when the whole file has been transcribed. Processing time grows with the length of the recording, and timestamps add to it. Long `transcribe-1-pro` recordings can take several minutes.

Set your HTTP client's timeout accordingly. The official SDKs' default timeout can be too short for long recordings, and many HTTP libraries default to even less. To wait up to 15 minutes, as several examples on this page do:

* Python SDK: `RequestOptions(timeout=900)` for one request, or `FishAudio(timeout=900)` for the client.
* httpx: `timeout=httpx.Timeout(900.0, connect=10.0)`. Without it, httpx waits only 5 seconds.
* Node.js: the built-in `fetch` stops waiting for a response after 5 minutes, even if you pass a longer `AbortSignal`. For longer requests, use `fetch` from the `undici` package with a dispatcher that waits longer. `undici` 7 runs on Node.js 20.18.1 and later; `undici` 8 requires Node.js 22.19 or later.

```javascript theme={null}
import { readFile } from "node:fs/promises";
import { Agent, FormData, fetch } from "undici"; // npm install undici@7

// Node's built-in fetch stops waiting for a response after 5 minutes.
// This dispatcher waits up to 15 minutes.
const dispatcher = new Agent({
  headersTimeout: 900_000,
  bodyTimeout: 900_000,
});

const form = new FormData();
form.append("audio", new Blob([await readFile("meeting.mp3")]), "meeting.mp3");
form.append("ignore_timestamps", "false");

const response = await fetch("https://api.fish.audio/v1/asr", {
  method: "POST",
  headers: {
    Authorization: `Bearer ${process.env.FISH_API_KEY}`,
    model: "transcribe-1-pro",
  },
  body: form,
  dispatcher,
});

if (!response.ok) throw new Error(await response.text());
const result = await response.json();
console.log(result.text);
```

These values are examples, not a server guarantee. If a long request fails with a 5xx error or the connection drops, retry it. Each request also holds one of your account's concurrent request slots until its response is returned; see [concurrent request limits](/developer-guide/models-pricing/pricing-and-rate-limits#concurrent-request-limits).

### Request IDs

`transcribe-1-pro` returns a unique `request_id` in the response body, both on success and on its own errors, and the same value in the `x-request-id` response header. Platform and network-edge errors do not carry it. Log it, and include it when you contact support. The Python SDK does not return it on success; read it with a [direct API call](#direct-api-messagepack), or from the error body as shown in [Errors](#errors).

### Errors

The error body depends on where the error comes from:

* `transcribe-1-pro` errors are JSON with `status`, `message`, `code`, and `request_id`. Branch on `code`, not on `message`.
* Platform errors, such as a missing or invalid API key (401), insufficient API credit (402), or the concurrency limit (429), are JSON with `status` and `message`.
* Errors from the network edge in front of the API, such as some 413 and 5xx responses, may not have a JSON body.

| Status | `code` | What to do |
| - | - | - |
| 400 | `invalid_request`, `invalid_parameter` | Fix the request or the parameter value. |
| 400 | `invalid_audio` | Convert the audio to a [supported format](#input-audio). |
| 400 | `audio_too_long`, `audio_too_short` | Send audio within the [limits](#limits). |
| 413 | `request_too_large` | Send a smaller file, for example compressed audio. |
| 415 | `unsupported_media_type` | Send `multipart/form-data` or `application/msgpack`. |
| 429 | | Usually your account is at its concurrency limit. Retry with backoff. |
| 5xx | For example `upstream_unavailable`, `upstream_timeout`, `diarization_failed` | The service is temporarily unavailable or could not finish. Retry with backoff. |

Retry 429 and 5xx responses with exponential backoff; 429 responses have no `Retry-After` header. Do not retry other 4xx responses; change the request first. New `code` values may be added; handle unknown codes by HTTP status. See the [API reference](/api-reference/endpoint/openapi-v1/speech-to-text#errors) for every status and code.

With the Python SDK, a failed request raises `APIError` (`RateLimitError` for 429, `ServerError` for 5xx). `e.status` is the HTTP status, and `e.body` is the raw response body, where `transcribe-1-pro` errors carry `code` and `request_id`:

```python theme={null}
import json

from fishaudio.exceptions import APIError

try:
    result = client.asr.transcribe(
        audio=audio_bytes,
        request_options=RequestOptions(
            additional_headers={"model": "transcribe-1-pro"}
        ),
    )
except APIError as e:
    try:
        error = json.loads(e.body or "{}")
    except ValueError:
        error = {}  # for example, an error without a JSON body
    print(e.status, error.get("code"), error.get("request_id"))
    raise
```

If you use `transcribe-1`, its errors are JSON with `status` and `message`; rely only on these two fields.

### Async transcription

The Python SDK ships an async client with the same surface, useful when you're transcribing many files concurrently or already running inside an event loop. Use `AsyncFishAudio` and `await` the call:

```python theme={null}
import asyncio
from fishaudio import AsyncFishAudio
from fishaudio.core import RequestOptions

async def main():
    client = AsyncFishAudio()  # reads FISH_API_KEY
    with open("speech.wav", "rb") as f:
        result = await client.asr.transcribe(
            audio=f.read(),
            request_options=RequestOptions(
                timeout=900,
                additional_headers={"model": "transcribe-1-pro"},
            ),
        )
    print(result.text)

asyncio.run(main())
```

To run several files in parallel, gather the coroutines:

```python theme={null}
import asyncio
from fishaudio import AsyncFishAudio
from fishaudio.core import RequestOptions

async def transcribe_all(paths):
    client = AsyncFishAudio()
    pro = RequestOptions(
        timeout=900, additional_headers={"model": "transcribe-1-pro"}
    )
    clips = [open(p, "rb").read() for p in paths]
    return await asyncio.gather(*[
        client.asr.transcribe(audio=clip, request_options=pro) for clip in clips
    ])

for result in asyncio.run(transcribe_all(["speech.wav"])):
    print(result.text)
```

### Direct API (MessagePack)

`POST /v1/asr` also accepts a [MessagePack](https://msgpack.org) body instead of multipart form data, the same path the API reference links to for low-overhead, server-side calls. Calling the API directly also gives you the response fields the Python SDK does not return. Pack the audio bytes and options into one payload and set `Content-Type: application/msgpack`:

```python theme={null}
import os
import httpx
import ormsgpack

with open("speech.wav", "rb") as f:
    audio = f.read()

payload = {"audio": audio, "language": "en", "ignore_timestamps": False}

resp = httpx.post(
    "https://api.fish.audio/v1/asr",
    content=ormsgpack.packb(payload),
    headers={
        "Authorization": f"Bearer {os.environ['FISH_API_KEY']}",
        "Content-Type": "application/msgpack",
        "model": "transcribe-1-pro",
    },
    # httpx waits 5 seconds by default; long recordings can take minutes.
    timeout=httpx.Timeout(900.0, connect=10.0),
)
resp.raise_for_status()
result = resp.json()
print(result["text"])
print(result.get("language_code"), result.get("request_id"))
for turn in result.get("speaker_turns", []):
    print(turn["speaker"], turn["start"], turn["end"], turn["text"])
```

The response is the same as for multipart: `text`, `duration` (seconds), `segments`, and `request_id`, plus `speaker_turns` when timestamps are requested, and `language_code` and `language` when the language is known. Model selection stays in the HTTP header for both formats. MessagePack values are typed: send booleans for `ignore_timestamps` and `tag_audio_events`, integers for speaker counts, and a boolean or `"auto"` for `diarize`.

## Going further

<CardGroup>
  <Card title="Generate speech" icon="microphone" href="/features/text-to-speech">
    The reverse direction: text to lifelike audio.
  </Card>

  <Card title="Full API parameters" icon="book-open" href="/api-reference/endpoint/openapi-v1/speech-to-text">
    Every field and the raw response schema.
  </Card>

  <Card title="Python reference" icon="python" href="/api-reference/sdk/python/resources">
    `asr.transcribe` options and the `ASRResponse` type.
  </Card>
</CardGroup>


# Text to Speech
Source: https://docs.fish.audio/features/text-to-speech

Turn text into lifelike speech

Generate natural speech from text with the `s2.1-pro`, `s2-pro`, and `s1` models, from the API directly, the Python library, or JavaScript.

<CardGroup>
  <Card title="Use it in the web app" icon="browser" href="https://fish.audio/app/text-to-speech">
    No code: type, pick a voice, generate.
  </Card>

  <Card title="API reference" icon="brackets-curly" href="/api-reference/endpoint/openapi-v1/text-to-speech">
    Every parameter for `POST /v1/tts`.
  </Card>

  <Card title="Cookbooks" icon="book-open" href="/developer-guide/sdk-guide/cookbook/streaming-to-file">
    Ready-made recipes: streaming, telephony, and more.
  </Card>
</CardGroup>

## When to use it

<CardGroup>
  <Card title="Voiceovers & narration" icon="film">
    Audiobooks, explainers, ads, and video narration.
  </Card>

  <Card title="Conversational AI" icon="comments">
    Speak an assistant's replies. Pair with [streaming](/features/realtime-streaming) for low latency.
  </Card>

  <Card title="Accessibility & IVR" icon="universal-access">
    Read content aloud, phone menus, notifications.
  </Card>

  <Card title="Custom voices" icon="clone">
    Speak in a [cloned voice](/features/voice-cloning) you own.
  </Card>
</CardGroup>

## Quick start

Send text, get back audio. Choose your implementation:

<CodeGroup>
  ```python Python theme={null}
  from fishaudio import FishAudio
  from fishaudio.utils import save

  client = FishAudio()  # reads FISH_API_KEY
  audio = client.tts.convert(text="Hello from Fish Audio!")
  save(audio, "out.mp3")
  ```

  ```bash API (curl) theme={null}
  curl --request POST https://api.fish.audio/v1/tts \
    --header "Authorization: Bearer $FISH_API_KEY" \
    --header "Content-Type: application/json" \
    --header "model: s2-pro" \
    --data '{ "text": "Hello from Fish Audio!", "format": "mp3" }' \
    --output out.mp3
  ```

  ```javascript JavaScript theme={null}
  import { FishAudioClient } from "fish-audio";
  import { writeFile } from "fs/promises";

  const client = new FishAudioClient({ apiKey: process.env.FISH_API_KEY });

  // `s2-pro` is passed explicitly (the SDK default is `s1`).
  const stream = await client.textToSpeech.convert(
    { text: "Hello from Fish Audio!" },
    "s2-pro",
  );

  const chunks = [];
  for await (const chunk of stream) chunks.push(Buffer.from(chunk));
  await writeFile("hello.mp3", Buffer.concat(chunks));
  ```
</CodeGroup>

## Use a specific voice

Pass a **voice model id** (`reference_id`). Find ids in the [Voice Library](/overview/platform) or create your own via [Voice Cloning](/features/voice-cloning).

<CodeGroup>
  ```python Python theme={null}
  audio = client.tts.convert(
      text="This uses a specific voice.",
      reference_id="9a9cf47702da476aa4629e2506d4a857",
  )
  ```

  ```bash API (curl) theme={null}
  curl --request POST https://api.fish.audio/v1/tts \
    --header "Authorization: Bearer $FISH_API_KEY" \
    --header "Content-Type: application/json" \
    --header "model: s2-pro" \
    --data '{
      "text": "This uses a specific voice.",
      "reference_id": "9a9cf47702da476aa4629e2506d4a857",
      "format": "mp3"
    }' \
    --output out.mp3
  ```
</CodeGroup>

## Implementation details

### Models

* **`s2.1-pro`**: recommended for production, with improved quality, latency, and throughput over S2-Pro.
* **`s2.1-pro-free`**: the same model at \$0 for testing, prototyping, development, and smaller businesses, without TTFA or DPA guarantees.
* **`s2-pro`**: previous-generation S2 model with multi-speaker and natural-language expression control.
* **`s1`**: previous generation, `(parenthesis)` emotion tags.

In the API, select with the `model` request header; requests without it use `s2.1-pro`. The Python SDK always sends a model and defaults to `model="s2-pro"`. See [Choosing a Model](/developer-guide/models-pricing/choosing-a-model).

### Output formats

`mp3` (default), `wav`, `pcm`, `opus`. Set `format` (and optionally `mp3_bitrate`, `sample_rate`).

<CodeGroup>
  ```python Python theme={null}
  from fishaudio.types import TTSConfig

  audio = client.tts.convert(
      text="High quality",
      config=TTSConfig(format="wav", sample_rate=44100),
  )
  ```

  ```bash API (curl) theme={null}
  curl --request POST https://api.fish.audio/v1/tts \
    --header "Authorization: Bearer $FISH_API_KEY" \
    --header "Content-Type: application/json" \
    --header "model: s2-pro" \
    --data '{ "text": "High quality", "format": "wav", "sample_rate": 44100 }' \
    --output out.wav
  ```
</CodeGroup>

### Speed & prosody

Adjust speech speed (0.5–2.0) and volume.

<CodeGroup>
  ```python Python theme={null}
  audio = client.tts.convert(text="Speaking faster.", speed=1.5)
  ```

  ```bash API (curl) theme={null}
  curl --request POST https://api.fish.audio/v1/tts \
    --header "Authorization: Bearer $FISH_API_KEY" \
    --header "Content-Type: application/json" \
    --header "model: s2-pro" \
    --data '{ "text": "Speaking faster.", "prosody": { "speed": 1.5 } }' \
    --output out.mp3
  ```
</CodeGroup>

### Generation methods (Python)

The Python SDK exposes three ways to generate, depending on whether you have the full text upfront and how you want to consume the audio:

| Method | Returns | Use it for |
| - | - | - |
| `tts.convert()` | complete audio `bytes` | most cases: you have the text, you want the file |
| `tts.stream()` | `AudioStream` (iterate chunks, or `.collect()`) | memory-efficient transfer of large audio; write chunks to disk as they arrive |
| `tts.stream_websocket()` | iterator of audio `bytes` | text arriving in real time (LLM tokens, live captions) |

```python theme={null}
# Memory-efficient: write each chunk as it arrives instead of buffering
audio_stream = client.tts.stream(text="A very long passage...")
with open("out.mp3", "wb") as f:
    for chunk in audio_stream:
        f.write(chunk)
```

For real-time text streaming with `stream_websocket()`, see [Realtime Streaming](/features/realtime-streaming).

### Instant voice cloning (reference audio)

Instead of a saved `reference_id`, pass raw audio plus its transcript to clone a voice on the fly, with no training step. Best with a clean 10–30s sample.

```python theme={null}
from fishaudio.types import ReferenceAudio

with open("sample.wav", "rb") as f:
    audio = client.tts.convert(
        text="Spoken in the reference voice.",
        references=[ReferenceAudio(audio=f.read(), text="Transcript of the sample.")],
    )
```

To reuse a voice across many requests, [clone it once](/features/voice-cloning) and pass the resulting `reference_id` instead.

### Format & bitrate

Pick a format for your delivery channel, and tune bitrate to trade size against quality:

| Format | Notes |
| - | - |
| `mp3` (default) | good size/quality balance; set `mp3_bitrate` to `64`, `128`, or `192` |
| `wav` | uncompressed, highest quality; set `sample_rate` (e.g. `44100`) |
| `pcm` | raw samples, no container; for low-latency playback and telephony pipelines |
| `opus` | efficient for streaming; bitrate is automatic (`opus_bitrate=-1000`) |

```python theme={null}
from fishaudio.types import TTSConfig

audio = client.tts.convert(
    text="Smaller file, lower bitrate.",
    config=TTSConfig(format="mp3", mp3_bitrate=64),
)
```

### Latency & chunk length

`latency` trades stability for speed; `chunk_length` controls how much text the engine batches before it starts generating.

* `latency="balanced"` (default): lower time-to-first-audio (\~300ms). Good for interactive use.
* `latency="normal"`: most stable output, at slightly higher latency.
* `chunk_length` (`100`–`300`, default `200`): smaller chunks start audio sooner; larger chunks are more efficient for long text.

<CodeGroup>
  ```python Python theme={null}
  from fishaudio.types import TTSConfig

  audio = client.tts.convert(
      text="Quick, responsive output.",
      config=TTSConfig(latency="balanced", chunk_length=150),
  )
  ```

  ```bash API (curl) theme={null}
  curl --request POST https://api.fish.audio/v1/tts \
    --header "Authorization: Bearer $FISH_API_KEY" \
    --header "Content-Type: application/json" \
    --header "model: s2-pro" \
    --data '{ "text": "Quick, responsive output.", "latency": "balanced", "chunk_length": 150 }' \
    --output out.mp3
  ```
</CodeGroup>

### Direct API (MessagePack)

`POST /v1/tts` also accepts a MessagePack body (`Content-Type: application/msgpack`), the path the [API reference](/api-reference/endpoint/openapi-v1/text-to-speech) is built around. Use it to send binary reference audio in the request without base64 overhead, or when you don't want the SDK.

```python theme={null}
import os
import httpx
import ormsgpack

payload = {"text": "Hello from the direct API.", "reference_id": "YOUR_VOICE_ID", "format": "mp3"}

resp = httpx.post(
    "https://api.fish.audio/v1/tts",
    content=ormsgpack.packb(payload),
    headers={
        "Authorization": f"Bearer {os.environ['FISH_API_KEY']}",
        "Content-Type": "application/msgpack",
        "model": "s2-pro",
    },
)
with open("out.mp3", "wb") as f:
    f.write(resp.content)
```

If the `model` header is omitted or set to an unrecognized value, the request falls back to `s2.1-pro`. JSON and MessagePack accept the same fields.

### Advanced generation tuning

For finer control, `TTSConfig` exposes the model's sampling parameters. The defaults are well-tuned. Reach for these only when you need to dial in determinism or curb artifacts.

```python theme={null}
from fishaudio.types import TTSConfig, Prosody

config = TTSConfig(
    prosody=Prosody(speed=1.1, volume=0),
    temperature=0.7,            # lower = more deterministic
    top_p=0.7,
    repetition_penalty=1.2,     # >1.0 curbs repeated sounds
    max_new_tokens=1024,        # cap audio length per chunk
    normalize=True,             # expand numbers/dates for natural reading
)

audio = client.tts.convert(text="Carefully tuned output.", config=config)
```

Define a `TTSConfig` once and pass it to many `convert()` calls. See the [full field list](/api-reference/sdk/python/types#ttsconfig-objects) for every parameter and default.

## Going further

<CardGroup>
  <Card title="Stream as it generates" icon="bolt" href="/features/realtime-streaming">
    Lowest latency for conversational and live apps.
  </Card>

  <Card title="Emotion & expression" icon="face-smile" href="/developer-guide/core-features/emotions">
    Direct delivery with tags and prosody.
  </Card>

  <Card title="Full API parameters" icon="book-open" href="/api-reference/endpoint/openapi-v1/text-to-speech">
    Every field, type, and default.
  </Card>

  <Card title="Python reference" icon="python" href="/api-reference/sdk/python/resources">
    `tts.convert` / `stream` / `stream_websocket`.
  </Card>
</CardGroup>


# Voice Cloning
Source: https://docs.fish.audio/features/voice-cloning

Create a custom voice from audio samples, then speak with it

Build a reusable voice model from your own audio, then use it anywhere you generate speech. You get back a voice **id**. Pass it as `reference_id` to [Text to Speech](/features/text-to-speech) and every generation speaks in that voice. Works from the API directly, the Python library, or JavaScript.

<CardGroup>
  <Card title="Use it in the web app" icon="browser" href="https://fish.audio/app/voice-cloning">
    No code: clone a voice in the browser.
  </Card>

  <Card title="API reference" icon="brackets-curly" href="/api-reference/endpoint/model/create-model">
    Every field for `POST /model`.
  </Card>

  <Card title="Cookbooks" icon="book-open" href="/developer-guide/sdk-guide/cookbook/instant-voice-cloning">
    Instant clones, training, and reuse.
  </Card>
</CardGroup>

## When to use it

<CardGroup>
  <Card title="Brand voice" icon="building">
    One consistent voice across product, ads, and IVR.
  </Card>

  <Card title="Personal voice" icon="user">
    Clone your own voice for narration or assistants.
  </Card>

  <Card title="Characters" icon="masks-theater">
    Distinct voices for games, stories, and dialogue.
  </Card>

  <Card title="Dubbing & localization" icon="language">
    Keep a speaker's identity across languages.
  </Card>
</CardGroup>

## Quick start

Send one or more audio samples, get back a voice model. Choose your implementation:

<CodeGroup>
  ```python Python theme={null}
  from fishaudio import FishAudio

  client = FishAudio()  # reads FISH_API_KEY

  with open("sample.wav", "rb") as f:
      voice = client.voices.create(
          title="My Voice",
          voices=[f.read()],
          description="Cloned from a studio sample",
          visibility="private",
      )

  print(voice.id, voice.state)
  ```

  ```bash API (curl) theme={null}
  curl --request POST https://api.fish.audio/model \
    --header "Authorization: Bearer $FISH_API_KEY" \
    --form type=tts \
    --form title="My Voice" \
    --form "description=Cloned from a studio sample" \
    --form visibility=private \
    --form train_mode=fast \
    --form voices=@sample.wav

  # Returns the new model, including its "_id" and "state".
  ```

  ```javascript JavaScript theme={null}
  import { FishAudioClient } from "fish-audio";
  import { readFile } from "fs/promises";

  const client = new FishAudioClient({ apiKey: process.env.FISH_API_KEY });

  const sample = await readFile("reference.wav");

  const voice = await client.voices.ivc.create({
    title: "My Voice",
    voices: [new File([sample], "reference.wav")],
    description: "Cloned from a studio sample",
    visibility: "private",
  });

  console.log(voice._id, voice.state);
  ```
</CodeGroup>

## Use your cloned voice

Pass the voice **id** as `reference_id` to Text to Speech, exactly like any other voice.

<CodeGroup>
  ```python Python theme={null}
  audio = client.tts.convert(
      text="Now I speak in my cloned voice.",
      reference_id=voice.id,
  )
  ```

  ```bash API (curl) theme={null}
  curl --request POST https://api.fish.audio/v1/tts \
    --header "Authorization: Bearer $FISH_API_KEY" \
    --header "Content-Type: application/json" \
    --header "model: s2-pro" \
    --data '{ "text": "Now I speak in my cloned voice.", "reference_id": "YOUR_VOICE_ID" }' \
    --output out.mp3
  ```
</CodeGroup>

## Implementation details

### Sample quality

Clean, mono, single-speaker audio gives the best result. A short clip works for a quick clone; a minute or two of clear speech improves fidelity. Avoid background music, reverb, and overlapping voices.

### Multiple samples

Pass several clips to capture more range. You can also supply the matching transcripts as `texts` to sharpen pronunciation.

<CodeGroup>
  ```python Python theme={null}
  voice = client.voices.create(
      title="My Voice",
      voices=[open("a.wav", "rb").read(), open("b.wav", "rb").read()],
      texts=["Transcript of clip A.", "Transcript of clip B."],
  )
  ```

  ```bash API (curl) theme={null}
  curl --request POST https://api.fish.audio/model \
    --header "Authorization: Bearer $FISH_API_KEY" \
    --form type=tts \
    --form title="My Voice" \
    --form voices=@a.wav \
    --form voices=@b.wav
  ```
</CodeGroup>

### Visibility

Models are `private` by default. Set `unlist` for a shareable link, or `public` to publish to the [Voice Library](/overview/platform). You can change this later in [Manage Voices](/features/manage-voices).

## Instant vs. persistent clones

There are two ways to clone:

* **Persistent model** (above): train once with `voices.create()`, get back a reusable `id`. Best when you'll use the same voice repeatedly.
* **Instant clone**: pass reference audio inline on each generation with no model to manage. Best for one-off or per-request voices.

For an instant clone, send the reference audio (and its transcript) directly to Text to Speech via `references` instead of `reference_id`:

```python Python theme={null}
from fishaudio import FishAudio
from fishaudio.types import ReferenceAudio

client = FishAudio()

with open("reference.wav", "rb") as f:
    audio = client.tts.convert(
        text="This will sound like the reference voice.",
        references=[ReferenceAudio(
            audio=f.read(),
            text="The exact words spoken in the reference clip.",
        )],
    )
```

Pass several `ReferenceAudio` entries to capture more range, just as you would with multiple samples in a persistent model. The matching `text` for each clip sharpens pronunciation.

## Sample audio requirements

Samples can be `.wav`, `.mp3`, `.m4a`, or `.opus`. Aim for at least 10 seconds per clip; a minute or two of clear, single-speaker speech improves fidelity.

`enhance_audio_quality` (on by default) removes background noise and normalizes levels before training:

```python Python theme={null}
voice = client.voices.create(
    title="My Voice",
    voices=[open("sample.wav", "rb").read()],
    enhance_audio_quality=True,
)
```

Leave it on for noisy or lower-quality recordings. If your audio is already clean and studio-grade, turning it off (`enhance_audio_quality=False`) avoids any extra processing.

## Model state

A new model reports a `state` field that moves from `created` to `trained` (or `failed`). With `train_mode="fast"` (the default) the voice is usable almost immediately, so most clones return already `trained`.

```python Python theme={null}
voice = client.voices.create(title="My Voice", voices=[sample])
print(voice.state)  # "trained"
```

If a generation rejects the `reference_id`, re-fetch the model and confirm its state before using it in Text to Speech:

```python Python theme={null}
voice = client.voices.get(voice.id)
if voice.state == "trained":
    audio = client.tts.convert(text="Hello.", reference_id=voice.id)
```

## Going further

<CardGroup>
  <Card title="Speak with your voice" icon="microphone" href="/features/text-to-speech">
    Use `reference_id` in any generation.
  </Card>

  <Card title="Manage your voices" icon="sliders" href="/features/manage-voices">
    List, update, and delete your voice models.
  </Card>

  <Card title="Cloning best practices" icon="lightbulb" href="/developer-guide/best-practices/voice-cloning">
    Get the most natural results from your samples.
  </Card>

  <Card title="Create Model API" icon="book-open" href="/api-reference/endpoint/model/create-model">
    Every field for `POST /model`.
  </Card>
</CardGroup>


# Voice Design
Source: https://docs.fish.audio/features/voice-design

Generate candidate voices from a prompt

Voice Design creates short voice candidates from a natural-language prompt. Use it when you want to explore a voice direction before building a longer text-to-speech workflow or creating a persistent voice model.

<CardGroup>
  <Card title="API reference" icon="brackets-curly" href="/api-reference/endpoint/openapi-v1/voice-design">
    Every parameter for `POST /v1/voice-design`.
  </Card>

  <Card title="Pricing" icon="dollar-sign" href="/developer-guide/models-pricing/pricing-and-rate-limits#voice-design">
    Voice Design is billed per successful generation request.
  </Card>

  <Card title="Voice cloning" icon="clone" href="/features/voice-cloning">
    Create a reusable voice model from reference audio.
  </Card>
</CardGroup>

## When to use it

<CardGroup>
  <Card title="Explore voice concepts" icon="lightbulb">
    Generate several candidate voices from a short creative brief.
  </Card>

  <Card title="Preview narration styles" icon="headphones">
    Provide preview text to hear how a generated voice reads a specific line.
  </Card>

  <Card title="Seed creative workflows" icon="sparkles">
    Use generated candidates to choose a voice direction before longer TTS
    production.
  </Card>

  <Card title="Stateless API calls" icon="bolt">
    Get generated audio directly without creating batches, samples, or voice
    models.
  </Card>
</CardGroup>

## Quick start

Send a JSON request with a prompt and receive generated candidates. The current candidate audio payload is WAV bytes encoded as base64.

<CodeGroup>
  ```bash API (curl) theme={null}
  curl --request POST https://api.fish.audio/v1/voice-design \
    --header "Authorization: Bearer $FISH_API_KEY" \
    --header "Content-Type: application/json" \
    --header "model: voice-design-1" \
    --data '{
      "instruction": "Warm, confident studio narrator with a natural tone",
      "reference_text": "Welcome to Fish Audio.",
      "language": "en",
      "n": 2
    }' | jq -r '.candidates[0].audio_base64' | base64 --decode > voice.wav
  ```

  ```python Python theme={null}
  import base64
  import os
  import requests

  response = requests.post(
      "https://api.fish.audio/v1/voice-design",
      headers={
          "Authorization": f"Bearer {os.environ['FISH_API_KEY']}",
          "Content-Type": "application/json",
          "model": "voice-design-1",
      },
      json={
          "instruction": "Warm, confident studio narrator with a natural tone",
          "reference_text": "Welcome to Fish Audio.",
          "language": "en",
          "n": 2,
      },
      timeout=120,
  )
  response.raise_for_status()

  candidate = response.json()["candidates"][0]
  with open("voice.wav", "wb") as f:
      f.write(base64.b64decode(candidate["audio_base64"]))

  print(candidate["sample_rate"], candidate["duration_ms"])
  ```

  ```javascript JavaScript theme={null}
  import { writeFile } from "node:fs/promises";

  const response = await fetch("https://api.fish.audio/v1/voice-design", {
    method: "POST",
    headers: {
      Authorization: `Bearer ${process.env.FISH_API_KEY}`,
      "Content-Type": "application/json",
      model: "voice-design-1",
    },
    body: JSON.stringify({
      instruction: "Warm, confident studio narrator with a natural tone",
      reference_text: "Welcome to Fish Audio.",
      language: "en",
      n: 2,
    }),
  });

  if (!response.ok)
    throw new Error(`${response.status} ${await response.text()}`);

  const { candidates } = await response.json();
  await writeFile("voice.wav", Buffer.from(candidates[0].audio_base64, "base64"));
  console.log(candidates[0].sample_rate, candidates[0].duration_ms);
  ```
</CodeGroup>

## Prompt and preview text

`instruction` is the main voice design prompt. Describe the voice, age, delivery, tone, accent, pacing, and context in natural language.

```json theme={null}
{
  "instruction": "Energetic young presenter, bright tone, crisp diction, friendly but not cartoonish",
  "reference_text": "Here is your weekly product update.",
  "language": "en",
  "n": 3
}
```

`reference_text` is optional. When you provide it, candidates read that text so you can compare voices on the same line. Keep it short; the API accepts up to 300 characters.

## Parameters

| Field | Default | Notes |
| - | - | - |
| `instruction` | Required | Voice design prompt. 1 to 2000 characters. |
| `reference_text` | `null` | Optional preview text. Up to 300 characters. |
| `language` | `null` | Optional language hint such as `en`, `zh`, or `ja`. |
| `n` | `2` | Number of candidates to generate. Range: 1 to 4. |
| `speed` | `1.0` | Speaking speed multiplier. Must be greater than 0 and at most 3. |
| `num_step` | `32` | Diffusion steps. Range: 1 to 128. |
| `guidance_scale` | `2.0` | Higher values follow the prompt more strongly. Must be at least 0. |
| `instruct_guidance_scale` | `0.0` | Prompt conditioning guidance. Must be at least 0. |
| `seed` | `null` | Optional deterministic seed for candidate generation. |

<Warning>
  Voice Design accepts JSON only. Do not send MessagePack, multipart form data,
  inline reference audio, or service-internal fields such as `features`,
  `features_json_file`, or `include_audio_base64`.
</Warning>

## Response

The response contains one or more generated candidates:

```json theme={null}
{
  "candidates": [
    {
      "id": "candidate-id",
      "index": 0,
      "audio_base64": "UklGRg...",
      "sample_rate": 44100,
      "duration_ms": 3100,
      "text": "Welcome to Fish Audio.",
      "language": "en"
    }
  ]
}
```

Use `index` to preserve the order returned by the model. `id` is a stable candidate identifier for this response. Optional fields such as `text`, `instruct`, and `language` appear only when available.

## Billing and errors

Voice Design is billed once per successful generation request, not once per candidate. Authentication errors, validation errors, insufficient API credit, concurrency limits, upstream service errors, and empty candidate responses are not billed.

For the full error format and retry guidance, see [Errors](/api-reference/errors).


# Overview
Source: https://docs.fish.audio/overview/capabilities

Everything Fish Audio can do, and how to build with it

Fish Audio is a voice AI platform. Every core feature is available three ways: in the [web app](/overview/platform) (no code), through the [REST API](/api-reference/introduction), and via the official [SDK](/developer-guide/sdk-guide/quickstart).

## Core features

<CardGroup>
  <Card title="Text to Speech" icon="microphone" href="/features/text-to-speech">
    Convert text into lifelike speech with the `s2.1-pro`, `s2-pro`, and `s1` models.
  </Card>

  <Card title="Speech to Text" icon="waveform" href="/features/speech-to-text">
    Transcribe audio with `transcribe-1-pro`: multi-speaker conversations, long recordings, emotion cues, and optional timestamps.
  </Card>

  <Card title="Voice Cloning" icon="clone" href="/features/voice-cloning">
    Clone a voice instantly from a clip, or train a persistent model.
  </Card>

  <Card title="Realtime Streaming" icon="bolt" href="/features/realtime-streaming">
    Stream audio as it generates, for voice agents and live apps.
  </Card>

  <Card title="Manage Voices" icon="sliders" href="/features/manage-voices">
    List, inspect, update, and delete your voice models.
  </Card>
</CardGroup>

## Also in the web app

These run in the browser, no code required. See the [Platform guide](/overview/platform).

<CardGroup>
  <Card title="Voice Changer" icon="wand-magic-sparkles" href="/overview/platform">
    Transform existing audio into a different voice.
  </Card>

  <Card title="Story Studio" icon="book-open" href="/overview/platform">
    Produce multi-speaker, long-form audio: audiobooks and narration.
  </Card>

  <Card title="Music & Sound Effects" icon="music" href="/overview/platform">
    Generate music and cinematic sound effects from a prompt.
  </Card>

  <Card title="Audio Separation" icon="scissors" href="/overview/platform">
    Split audio into stems, and related processing utilities.
  </Card>
</CardGroup>

## Models

These text-to-speech models power most capabilities:

* **`s2.1-pro`**: the recommended production model, with improved quality, latency, and throughput over S2-Pro.
* **`s2.1-pro-free`**: the same model at \$0 for testing, prototyping, development, and smaller businesses, without TTFA or DPA guarantees.
* **`s2-pro`**: the previous-generation S2 model, with multi-speaker and natural-language expression control.
* **`s1`**: the previous generation, with `(parenthesis)` emotion tags.

For speech to text, select the model with the `model` header on `POST /v1/asr`:

* **`transcribe-1-pro`**: the recommended model, for everything from short clips to multi-speaker conversations (speaker markers and speaker turns) and long recordings, with inline emotion cues. Send `model: transcribe-1-pro` explicitly.
* **`transcribe-1`**: general transcription of short recordings. A request whose `model` header is missing or not an exact match is served by this model.

See [Models Overview](/developer-guide/models-pricing/models-overview) and [Choosing a Model](/developer-guide/models-pricing/choosing-a-model) for the full lineup and languages, and [Speech to Text](/features/speech-to-text#input-audio) for transcription formats and limits.

## Pick your path

<CardGroup>
  <Card title="Use the web app" icon="browser" href="/overview/platform">
    No code: generate audio, clone voices, and produce projects in your browser.
  </Card>

  <Card title="Build with the SDK" icon="code" href="/developer-guide/sdk-guide/quickstart">
    The Python library for your application.
  </Card>

  <Card title="Call the API" icon="terminal" href="/api-reference/introduction">
    Raw REST and WebSocket endpoints for any language.
  </Card>

  <Card title="Use your AI coding agent" icon="robot" href="/developer-guide/resources/coding-agents">
    Install the Fish Audio skill so your agent writes correct code.
  </Card>
</CardGroup>


# MCP Server
Source: https://docs.fish.audio/overview/mcp

Connect Claude, Cursor, or any MCP client to your Fish Audio account: search voices, generate speech, transcribe audio, and create images and videos

The Fish Audio MCP server gives AI agents direct access to your Fish Audio account: they can browse the voice library, generate speech, transcribe audio, and generate or edit images and videos for you.

* **Endpoint**: `https://api.fish.audio/mcp` (streamable HTTP)
* **Authentication**: OAuth. Your MCP client opens a browser window, you sign in with your Fish Audio account, and choose the team the connection works in.
* **Billing**: usage draws from your plan's package credits, exactly like the [web app](https://fish.audio/app). It does not consume developer API credits. Image and video generation is quoted first, and the agent asks you to confirm the price before it generates.

## Connect

<Tabs>
  <Tab title="Claude Code">
    ```bash theme={null}
    claude mcp add --transport http fish-audio https://api.fish.audio/mcp
    ```

    Run `/mcp` inside Claude Code to sign in, then ask "Generate a short greeting with a warm English voice."
  </Tab>

  <Tab title="Claude.ai">
    Go to **Settings → Connectors → Add custom connector** and enter
    `https://api.fish.audio/mcp`. Claude walks you through the sign-in.
  </Tab>

  <Tab title="Cursor">
    Open the command palette (`Cmd/Ctrl+Shift+P`) → "Open MCP settings" → "Add custom MCP", and add:

    ```json theme={null}
    {
      "mcpServers": {
        "fish-audio": { "url": "https://api.fish.audio/mcp" }
      }
    }
    ```

    Cursor prompts you to sign in on first use.
  </Tab>

  <Tab title="Codex CLI">
    ```bash theme={null}
    codex mcp add fish-audio https://api.fish.audio/mcp
    codex mcp login fish-audio
    ```

    `codex mcp login` opens the browser sign-in. Verify with `codex mcp list`.
  </Tab>

  <Tab title="Windsurf">
    Go to `Settings → Cascade → MCP Servers → View raw config` (`~/.codeium/windsurf/mcp_config.json`) and add:

    ```json theme={null}
    {
      "mcpServers": {
        "fish-audio": { "url": "https://api.fish.audio/mcp" }
      }
    }
    ```
  </Tab>
</Tabs>

## Try it

Once connected, ask in plain language and the agent picks the right tools. Click a card to copy its prompt; the copied text also names the server, so the agent can help you connect it if you have not yet.

<CardGroup>
  <PromptCard title="Narrate a script" icon="microphone" />

  <PromptCard title="Voice casting" icon="magnifying-glass" />

  <PromptCard title="Transcribe a recording" icon="waveform" />

  <PromptCard title="Check spend" icon="wallet" />

  <PromptCard title="Generate an image" icon="image" />

  <PromptCard title="Edit an image" icon="paintbrush" />

  <PromptCard title="Generate a video" icon="video" />

  <PromptCard title="Animate an image" icon="film" />
</CardGroup>

In coding agents like Claude Code and Codex, tools compose with the shell: the agent can download generated audio, images, and videos with `curl`, upload local recordings for transcription or reference images for editing, or batch-generate a directory of scripts.

## Good to know

* Add audio tags in square brackets inside the text to control delivery: `[whispering]`, `[excited]`, `[laughing]`. Tags are performed, never spoken.
* If no voice is specified, a curated default voice for the requested language is used.
* Local files can be transcribed too. The agent uploads them to a temporary slot that is deleted automatically after 7 days.
* Before an image or video is generated, the agent requests a credit estimate and asks you to confirm it. The server re-checks the price when the job is submitted and never charges more than the amount you approved.
* Video generation needs a Plus plan or higher. Free plans have a daily image allowance and a smaller image count per request; the estimate reports these limits before you are asked to approve anything.
* Reference images for editing or image-to-video are JPEG, PNG, or WebP files up to 20 MB. The agent uploads them for you, and an image it just generated can be reused as input without downloading it.
* Generated audio, images, and videos are returned as permanent URLs; failed generations are refunded.
* A connection is bound to the team you chose at sign-in. To work in another team, add the server again and choose that team.


# Platform (Web App)
Source: https://docs.fish.audio/overview/platform

Use Fish Audio in your browser, no code required

The [Fish Audio web app](https://fish.audio/app) gives you every capability without writing code: generate speech, clone voices, produce long-form projects, and manage your account. Sign in at [fish.audio/app](https://fish.audio/app).

<Note>
  This page is an orientation map of the web app. Use the links below to open
  each tool. To build with code instead, see the
  [SDK Quickstart](/developer-guide/sdk-guide/quickstart) or the [API
  Reference](/api-reference/introduction).
</Note>

## Create audio

<CardGroup>
  <Card title="Text-to-Speech" icon="microphone" href="https://fish.audio/app/text-to-speech">
    Type or paste text, pick a voice and model, and generate speech.
  </Card>

  <Card title="Voice Changer" icon="wand-magic-sparkles" href="https://fish.audio/app/voice-changer">
    Upload audio and transform it into another voice.
  </Card>

  <Card title="Speech-to-Text" icon="waveform" href="https://fish.audio/app/speech-to-text">
    Upload audio to transcribe it, with timestamps.
  </Card>

  <Card title="Sound Effects" icon="volume-high" href="https://fish.audio/app/sound-effects">
    Generate cinematic sound effects from a text prompt.
  </Card>

  <Card title="Music" icon="music" href="https://fish.audio/app/music-generation">
    Generate music from a description.
  </Card>

  <Card title="Audio Separation" icon="scissors" href="https://fish.audio/app/audio-separation">
    Split audio into stems (e.g. vocals and background).
  </Card>
</CardGroup>

## Voices

<CardGroup>
  <Card title="Voice Cloning" icon="clone" href="https://fish.audio/app/voice-cloning">
    Create a custom voice from your own audio samples.
  </Card>

  <Card title="Voice Library" icon="book-open" href="https://fish.audio/app/discovery">
    Browse and use thousands of community and official voices.
  </Card>

  <Card title="My Voices" icon="user" href="https://fish.audio/app/my-voices">
    Manage the voice models you've created or saved.
  </Card>

  <Card title="Pronunciation Dictionaries" icon="book-open" href="https://fish.audio/app/pronunciation-dictionary">
    Create, preview, and publish reusable pronunciation rules. Follow the
    [dictionary guide](/developer-guide/core-features/fine-grained-control/pronunciation-dictionaries)
    to use a published version in TTS requests.
  </Card>
</CardGroup>

## Produce projects

<Card title="Story Studio" icon="book" href="https://fish.audio/app/story-studio">
  Assemble multi-speaker, long-form audio (audiobooks, dialogue, and narration)
  in a project editor.
</Card>

## Library & history

<CardGroup>
  <Card title="Generation History" icon="clock-rotate-left" href="https://fish.audio/app/generation-history">
    Find, replay, and download everything you've generated.
  </Card>

  <Card title="My Library" icon="folder" href="https://fish.audio/app/my-library">
    Your saved audio, models, and collections.
  </Card>
</CardGroup>

## Account & billing

<CardGroup>
  <Card title="API Keys" icon="key" href="https://fish.audio/app/api-keys">
    Create and manage keys for the API and SDKs.
  </Card>

  <Card title="Plans & Billing" icon="credit-card" href="https://fish.audio/app/billing">
    Subscription, credits, and invoices. See [Pricing &
    Rate Limits](/developer-guide/models-pricing/pricing-and-rate-limits).
  </Card>

  <Card title="Usage" icon="chart-line" href="https://fish.audio/app/usage">
    Track your consumption.
  </Card>

  <Card title="Teams" icon="users" href="https://fish.audio/app/team">
    Shared workspaces, members, and billing for organizations.
  </Card>
</CardGroup>