Web SDK

@fishaudio/agent-client runs a live voice conversation with your agent from any web page: open the microphone, stream audio both ways, and react to typed events for transcripts, agent state, and tool calls. The SDK handles the realtime transport (WebRTC) internally. Using React? @fishaudio/agent-react wraps this SDK in hooks and a provider.

Install

npm install @fishaudio/agent-client

pnpm add @fishaudio/agent-client

yarn add @fishaudio/agent-client

Start a session

AgentSession.start() creates the session, connects, and opens the microphone in one call. Authenticate one of two ways:

agentId: for public agents. The SDK creates the session directly from the browser; no backend needed.
sessionToken: for private agents. Your backend calls POST /v1/agent/sessions with your API key and hands the JSON response to the browser; pass it through unchanged. See Authenticated sessions.

import { AgentSession } from "@fishaudio/agent-client";

const session = await AgentSession.start({
  agentId: "YOUR_AGENT_ID",
});

import { AgentSession } from "@fishaudio/agent-client";

// Your backend calls POST /v1/agent/sessions and returns the
// response body. Pass it to the SDK as-is.
const resp = await fetch("/api/voice-session", { method: "POST" });
const sessionToken = await resp.json();

const session = await AgentSession.start({ sessionToken });

start() resolves once the realtime connection is up and rejects with a FishAgentError if session creation, the microphone permission, or the connection fails. Call it from a user gesture (a click handler): browsers only grant the microphone and audio playback inside one.

Options

Option	Type	Description
`agentId` / `sessionToken`	`string` / `SessionToken`	Exactly one is required: a public agent id, or the JSON your backend received from `POST /v1/agent/sessions`
`serverUrl`	`string`	API base used in `agentId` mode. Default `https://api.fish.audio`
`clientTools`	`Record<string, ClientToolHandler>`	Handlers for client tools declared on the agent
`clientToolTimeoutMs`	`number`	Per-handler timeout for client tools, in milliseconds. Default `15000`
`overrides`	`SessionOverrides`	Per-session config overrides (`agentId` mode only; with `sessionToken`, your backend sends them when creating the session). Keyless sessions accept only `voice_id` and `language`; the prompt-shaping fields are rejected with `400`
`dynamicVariables`	`Record<string, string \| number \| boolean>`	Values for `{{placeholders}}` in the agent config; see Dynamic variables (`agentId` mode only)
`language`	`SessionLanguage`, the code of one of the 52 supported languages	Shorthand
for `overrides.language` (`agentId` mode only). Omit it to use the agent’s configured speaking language; any other value is rejected with `422`
`toolEvents`	`boolean`	Whether tool lifecycle events reach this client (default `true`; `agentId` mode only, with `sessionToken` your backend sets `tool_events`)
`timezone`	`string`	IANA timezone for the agent’s sense of local time: the top of the resolution order, overriding the agent’s configured timezone. When omitted, the SDK still sends the browser timezone as a lower-priority hint (`client_timezone`), which applies only if the agent has no timezone configured
`worldContext`	`boolean`	Whether the agent knows the current date and time (default `true`; `agentId` mode only, with `sessionToken` your backend sets `world_context`)
`endUserId`	`string`	Your identifier for the end user, stored on the session and echoed in webhooks (`agentId` mode only; with `sessionToken` your backend sets `end_user_id`)
`metadata`	`object`	Your own key-values, stored on the session record and returned verbatim (`agentId` mode only; with `sessionToken` your backend sets `metadata`)
`microphone`	`boolean`	Capture the microphone on start (default `true`). `false` joins muted with no permission prompt, for text-first UIs; the first `setMicMuted(false)` captures it, so call that from a user gesture
`audio`	`{ inputDeviceId?, outputDeviceId? }`	Pick specific microphone and output devices. `start()` rejects with `device_change_failed` if the output device cannot be selected
`wakeLock`	`boolean`	Hold a screen wake lock while the session is live, so long calls survive the phone trying to sleep (default `true`). A denied or unsupported wake lock is silent
`callbacks`	`Partial<AgentSessionCallbacks>`	Shorthand for `.on()`: each `onXxx` key subscribes the `xxx` event, so `onUserTranscript` subscribes `userTranscript`. From 0.2.1, a key that is not a known event name throws a `TypeError`

Callbacks shorthand

const session = await AgentSession.start({
  sessionToken,
  callbacks: {
    onUserTranscript: ({ text, final }) => renderUserBubble(text, final),
    onAgentResponseDelta: ({ text }) => renderAgentBubble(text),
    onModeChange: mode => setOrbState(mode),
    onDisconnect: ({ reason }) => showCallEnded(reason),
    onError: error => console.error(error.code),
  },
});

Session lifecycle

A session moves through a fixed state machine, surfaced by the statusChange event and the session.status property:

States

connecting → connected ⇄ reconnecting → ended(reason)
      └───────────(failure)───────────→ ended

start() resolves once the realtime connection is up; the agent itself joins moments later. If it has not joined within 15 seconds, the session emits a connection_failed error and ends with reason connection_lost instead of idling on a dead call. When the session ends, disconnect fires with a reason (also available as session.endReason):

`EndReason`	Meaning
`user_hangup`	You called `session.end()`, or the user left the page
`agent_hangup`	The server ended the session: the agent hung up (for example via the hang-up system tool), the session hit its maximum duration, or it was force-ended; the protocol does not distinguish these
`connection_lost`	The connection dropped and could not be recovered, or the agent never joined

Brief network drops don’t end the session: the SDK moves to reconnecting and back to connected automatically, reusing the same session. It never creates a new session, so you never need a new token mid-call.

Events are not replayed after a reconnect. Anything emitted while you were in reconnecting (transcript updates, tool events, errors) is dropped, not resent. Design your UI to tolerate gaps: tool toolCallCompleted / toolCallFailed events repeat the tool name and source, so a terminal event still renders even if you missed toolCallStarted.

The session also exposes sessionId (use it to look up the conversation record later), status, mode, isSpeaking, micMuted, and endReason as properties.

Events

The session is a typed event emitter: subscribe with session.on(event, handler), remove with off, or use once.

Event	Payload	Fires when
`connect`	`{ sessionId }`	The session is connected and live
`disconnect`	`{ reason: EndReason }`	The session has ended
`statusChange`	`SessionStatus`	Status transitions (`connecting` / `connected` / `reconnecting` / `ended`)
`modeChange`	`AgentMode`	The agent switches between `listening`, `thinking`, and `speaking`
`userTranscript`	`{ segmentId, text, final }`	The user’s speech is transcribed; interim updates replace the whole segment
`agentResponseDelta`	`{ segmentId, delta, text }`	The agent speaks: `delta` is the new text, `text` the segment so far
`agentResponse`	`{ segmentId, text }`	An agent segment is finalized
`message`	`{ role: "user" \| "agent", text }`	A finalized message from either side, a ready-made chat feed
`toolCallStarted`	`{ callId, toolName, source, input, inputTruncated }`	A tool call begins; `input` is a JSON string (truncated at 4 KB)
`toolCallCompleted`	`{ callId, toolName, source, output, outputTruncated }`	A tool call succeeds
`toolCallFailed`	`{ callId, toolName, source, error }`	A tool call fails
`error`	`FishAgentError`	A session or tool error occurs; see Errors

Subscribe to events

session.on("userTranscript", ({ segmentId, text, final }) => {
  upsertBubble("user", segmentId, text, final);
});

session.on("agentResponseDelta", ({ segmentId, text }) => {
  upsertBubble("agent", segmentId, text, false);
});

session.on("modeChange", mode => setOrbState(mode));
session.on("disconnect", ({ reason }) => showCallEnded(reason));

Transcript semantics

Transcripts on both sides arrive as segments, one segment per utterance or response, identified by segmentId:

User segments: interim results replace the entire segment text (they never append). Render by upserting on segmentId; final: true marks the segment as finalized.
Agent segments: text streams in sync with audio playback: what you display matches what the user has actually heard. If the agent is interrupted, the segment finalizes containing only the words that were spoken. A reply that spans several speech segments (for example around a tool call) arrives as several segments.
The message event delivers only finalized messages from both sides, in order. Use it when you want a simple transcript list without handling interim updates.

A listener attached late misses the events fired before it subscribed. session.getTranscript() returns every segment so far, as { segmentId, role, text, final } in conversation order: seed your UI from it, then keep applying live events by segmentId.

Tool call lifecycle

Every tool the agent runs (webhook, client, or system tool) emits one toolCallStarted, resolved by exactly one toolCallCompleted or toolCallFailed with the same callId. input and output are JSON strings truncated at 4 KB; parse them only when the matching *Truncated flag is false. These events carry tool arguments and results into the end user’s browser: to keep them off the client, start with toolEvents: false, or create the session with tool_events: false on your backend.

Agent modes

session.mode (and the modeChange event) tracks what the agent is doing, for driving an orb or status indicator:

Mode	Meaning
`listening`	Default state: the agent is waiting for or hearing the user
`thinking`	The agent is preparing a response
`speaking`	Agent audio is playing; ends when playback finishes

session.isSpeaking is a convenience boolean for the speaking mode. Mode does not react to the user’s own speech. For instant “the mic hears you” feedback, poll getInputVolume() locally.

Send text

Users can type instead of talking, in the same session:

Text input

// Send a typed user turn. There is no server echo; the SDK emits
// the finalized `message` event locally from the text you passed.
// By default the agent answers a typed turn in text only
// (agentResponseDelta / agentResponse) without speaking out loud.
session.sendUserMessage("Do you ship to Norway?");

// Pass `audio: true` to have the agent speak its reply for this turn.
session.sendUserMessage("What's my order status?", { audio: true });

// Signal typing so the agent doesn't talk over the user.
input.addEventListener("input", () => session.sendUserActivity());

// Stop the agent's current speech immediately (explicit barge-in).
session.interrupt();

Text-only replies are not paced to audio playback: the transcript streams as fast as it generates. Messages sent right after start() resolves, before the agent has finished joining, are held and delivered in order once it is ready.

Audio controls

Method / property	Description
`setMicMuted(muted)` / `micMuted`	Mute or unmute the microphone. After a `microphone: false` start, the first unmute captures the microphone (permission prompt, so call it from a user gesture) and rejects with `mic_permission_denied` if refused
`setOutputVolume(v)`	Set playback volume, `0`–`1`
`getInputVolume()` / `getOutputVolume()`	Current mic / agent volume, `0`–`1`; poll per frame
`getInputFrequencyData()` / `getOutputFrequencyData()`	FFT data as `Uint8Array`, for visualizers
`startAudio()`	Unlock playback under browser autoplay policies; call inside a user gesture (for example the click that starts the call)
`setInputDevice(deviceId)`	Switch the microphone mid-call. Rejects with `device_change_failed` if the device cannot be activated, switching back to the previous microphone (best effort)
`setOutputDevice(deviceId)`	Route playback to another output device (`""` for the default). Rejects with `device_change_failed` where the browser does not support output selection (common on mobile); playback stays on the previous device

Device ids come from navigator.mediaDevices.enumerateDevices(). To pick devices before the call starts, pass the audio option (inputDeviceId / outputDeviceId) to start() instead.

Audio visualizer

function draw() {
  const fft = session.getOutputFrequencyData(); // Uint8Array
  renderBars(canvas, fft);
  rafId = requestAnimationFrame(draw);
}
draw();

// Stop drawing when the session ends.
session.on("disconnect", () => cancelAnimationFrame(rafId));

Client tools

Register handlers for tools of type client declared on the agent. The agent calls them mid-conversation and your return value goes back to the model:

const session = await AgentSession.start({
  agentId: "YOUR_AGENT_ID",
  clientTools: {
    open_page: async params => {
      showPanel(String(params.page));
      return { opened: true };
    },
  },
});

// Or after start:
session.registerClientTool("highlight_product", handler);

Handlers can be sync or async; a thrown error or a timeout (default 15 s, set with clientToolTimeoutMs) is returned to the agent as a tool error. See Client tools for declaration, naming rules, result size, and dispatch semantics.

Errors

Failures surface as FishAgentError: thrown from AgentSession.start() when the session can’t be created, emitted on the error event otherwise. Each carries a code, an optional statusCode (set on session-creation HTTP errors), and cause.

Code	Meaning
`session_request_failed`	Session creation failed; check `statusCode`
`agent_not_public`	`agentId` mode, but the agent is not enabled for public access
`origin_forbidden`	The page’s origin is not in the agent’s allowed origins
`unsupported_transport`	The session token uses a transport this SDK version doesn’t know; upgrade the SDK
`mic_permission_denied`	The user denied microphone access; the SDK ends the session
`device_change_failed`	A requested audio device could not be activated, or the browser doesn’t support selecting it (output selection is unsupported on some mobile browsers). Thrown from `start()` with `audio.outputDeviceId` set, or from device-switching calls
`connection_failed`	The realtime connection could not be established or recovered, a message to the agent could not be sent, the agent never joined (15 s), or the microphone could not be toggled
`session_expired`	The session token’s join deadline passed before connecting
`tool_failed` / `tool_timeout`	A client tool handler threw, was not registered, or its result could not be delivered / a handler exceeded `clientToolTimeoutMs`
`provider_error` / `internal_error`	The session failed server-side: upstream model/voice provider vs. platform runtime. Emitted on `error`; the session may recover

provider_error and internal_error carry only the category code. Raw provider or infrastructure details are never sent to the browser.

Handle errors

import { AgentSession, FishAgentError } from "@fishaudio/agent-client";

try {
  const session = await AgentSession.start({ agentId: "YOUR_AGENT_ID" });
  session.on("error", err => console.warn("session error:", err.code));
} catch (err) {
  if (err instanceof FishAgentError && err.code === "mic_permission_denied") {
    showMicHelp();
  } else {
    showRetry(err);
  }
}

End the session

Hang up

await session.end(); // graceful hangup; disconnect fires with reason "user_hangup"

Calling end() more than once, or while a previous call is still finishing, is safe. endReason stays readable on the ended session.

Underlying connection

The session API is transport-neutral: the server picks the transport when it creates the session, livekit (WebRTC) today. For needs the session API does not cover yet, such as connection-quality telemetry, session.getRoom() returns the live livekit-client Room, or undefined once the session has ended. Prefer the session API where one exists: code built on the Room couples to this SDK’s transport choice and its pinned livekit-client major, with no compatibility promise across SDK versions. See Wire protocol for the messages underneath.

Going further

React SDK

Hooks, provider, and visualizer components on top of this SDK.

Authenticated sessions

Create session tokens on your backend for private agents.

Public agents

Backend-free sessions with origin allow-lists and rate limits.

Wire protocol

The wire-level events underneath the SDK.

Get Started

Build

Test

Deploy

Telephony

Monitor

Install

Start a session

Options

Session lifecycle

Events

Transcript semantics

Tool call lifecycle

Agent modes

Send text

Audio controls

Client tools

Errors

End the session

Underlying connection

Going further

React SDK

Authenticated sessions

Public agents

Wire protocol

​Install

​Start a session

​Options

​Session lifecycle

​Events

​Transcript semantics

​Tool call lifecycle

​Agent modes

​Send text

​Audio controls

​Client tools

​Errors

​End the session

​Underlying connection

​Going further

React SDK

Authenticated sessions

Public agents

Wire protocol

Install

Start a session

Options

Session lifecycle

Events

Transcript semantics

Tool call lifecycle

Agent modes

Send text

Audio controls

Client tools

Errors

End the session

Underlying connection

Going further