{
"type": "session.start",
"voice": {
"reference_id": "1c8ce946e6c847e6a3b32a88cce2a1e4"
},
"audio": {
"encoding": "pcm_s16le",
"sample_rate": 24000,
"noise_cancellation": true
},
"turns": {
"eager": true,
"backchannel": true,
"interim": true
},
"playback": {
"delay_ms": 260
},
"session_ending": true
}{
"type": "audio",
"data": "<960 bytes raw pcm_s16le (20 ms @ 24 kHz)>"
}{
"type": "reply.text",
"reply_id": "r-1-9f3a",
"for": "e_01",
"text": "Got it, adding a burger meal. ",
"flush": true,
"last": false
}{
"type": "reply.cancel",
"reply_id": "r-1-9f3a"
}{
"type": "listen.pause"
}{
"type": "listen.resume"
}{
"seq": 0,
"type": "session.started",
"session_id": "20261116-090715-6462dd",
"limits": {
"max_session_s": 1800,
"idle_s": 30,
"max_text_bytes": 4096,
"max_held_bytes": 8192,
"max_held_s": 15,
"max_message_bytes": 131072
},
"text_rules": {
"numbers": "written",
"ack_in_t": false,
"avoid": [
"colon",
"semicolon",
"spaced_dash"
]
},
"audio": {
"encoding": "pcm_s16le",
"sample_rate": 24000,
"output_sample_rate": 24000
},
"noise_cancellation": {
"on": true,
"delay_ms": 28
}
}{
"seq": 1,
"type": "audio",
"data": "<raw pcm_s16le mono bytes>"
}{
"seq": 42,
"type": "turn.eager",
"eager_id": "e_01",
"turn_id": "t_01",
"text": "Can I get a burger meal",
"agent": {
"reply_id": null,
"state": "idle",
"heard_text": "",
"basis": "aligned"
}
}{
"seq": 45,
"type": "turn.resumed",
"eager_id": "e_01"
}{
"seq": 49,
"type": "turn.committed",
"turn_id": "t_01",
"text": "Can I get a burger meal",
"kind": "turn",
"approved_eager_id": "e_01",
"lead_ms": 280,
"agent": {
"reply_id": "r-1-9f3a",
"state": "pending",
"heard_text": "",
"basis": "aligned"
}
}{
"seq": 123,
"type": "<string>",
"reason": "<string>",
"agent": {
"state": "<string>",
"heard_text": "<string>",
"basis": "<string>"
}
}{
"seq": 123,
"type": "<string>",
"text": "<string>",
"at_ms": 123
}{
"seq": 123,
"type": "<string>",
"turn_id": "<string>",
"text": "<string>"
}{
"seq": 123,
"type": "<string>",
"reply_id": "<string>",
"at_ms": 123
}{
"seq": 123,
"type": "<string>",
"reply_id": "<string>",
"spoken_chars": 123,
"basis": "<string>"
}{
"seq": 123,
"type": "<string>",
"reply_id": "<string>",
"by": "<string>",
"heard_text": "<string>",
"basis": "<string>",
"final": true
}{
"seq": 88,
"type": "reply.done",
"reply_id": "r-1-9f3a",
"status": "interrupted",
"reason": null,
"by": "caller",
"heard_text": "Got it, adding a burger meal.",
"unheard_text": " What drink would you like with that?",
"heard_chars": 30,
"basis": "aligned",
"audio_end_ms": 4120
}{
"seq": 123,
"type": "<string>",
"kind": "<string>",
"start_ms": 123,
"end_ms": 123
}{
"seq": 123,
"type": "<string>",
"in_s": 123
}{
"seq": 123,
"type": "<string>",
"code": "<string>",
"message": "<string>",
"recoverable": true
}Duplex API Converse
Voice conversation WebSocket with server-side turn detection, streamed speech, and interruption handling.
session.start.voice.reference_id. See Voice presets for Clara and the other ready-to-use reference IDs, or supply a custom Fish Voice Library ID. Use the ID, not the voice’s display name.
Every message is a binary WebSocket frame containing a MessagePack map. Audio
payloads use MessagePack binary data, not base64 strings.{
"type": "session.start",
"voice": {
"reference_id": "1c8ce946e6c847e6a3b32a88cce2a1e4"
},
"audio": {
"encoding": "pcm_s16le",
"sample_rate": 24000,
"noise_cancellation": true
},
"turns": {
"eager": true,
"backchannel": true,
"interim": true
},
"playback": {
"delay_ms": 260
},
"session_ending": true
}{
"type": "audio",
"data": "<960 bytes raw pcm_s16le (20 ms @ 24 kHz)>"
}{
"type": "reply.text",
"reply_id": "r-1-9f3a",
"for": "e_01",
"text": "Got it, adding a burger meal. ",
"flush": true,
"last": false
}{
"type": "reply.cancel",
"reply_id": "r-1-9f3a"
}{
"type": "listen.pause"
}{
"type": "listen.resume"
}{
"seq": 0,
"type": "session.started",
"session_id": "20261116-090715-6462dd",
"limits": {
"max_session_s": 1800,
"idle_s": 30,
"max_text_bytes": 4096,
"max_held_bytes": 8192,
"max_held_s": 15,
"max_message_bytes": 131072
},
"text_rules": {
"numbers": "written",
"ack_in_t": false,
"avoid": [
"colon",
"semicolon",
"spaced_dash"
]
},
"audio": {
"encoding": "pcm_s16le",
"sample_rate": 24000,
"output_sample_rate": 24000
},
"noise_cancellation": {
"on": true,
"delay_ms": 28
}
}{
"seq": 1,
"type": "audio",
"data": "<raw pcm_s16le mono bytes>"
}{
"seq": 42,
"type": "turn.eager",
"eager_id": "e_01",
"turn_id": "t_01",
"text": "Can I get a burger meal",
"agent": {
"reply_id": null,
"state": "idle",
"heard_text": "",
"basis": "aligned"
}
}{
"seq": 45,
"type": "turn.resumed",
"eager_id": "e_01"
}{
"seq": 49,
"type": "turn.committed",
"turn_id": "t_01",
"text": "Can I get a burger meal",
"kind": "turn",
"approved_eager_id": "e_01",
"lead_ms": 280,
"agent": {
"reply_id": "r-1-9f3a",
"state": "pending",
"heard_text": "",
"basis": "aligned"
}
}{
"seq": 123,
"type": "<string>",
"reason": "<string>",
"agent": {
"state": "<string>",
"heard_text": "<string>",
"basis": "<string>"
}
}{
"seq": 123,
"type": "<string>",
"text": "<string>",
"at_ms": 123
}{
"seq": 123,
"type": "<string>",
"turn_id": "<string>",
"text": "<string>"
}{
"seq": 123,
"type": "<string>",
"reply_id": "<string>",
"at_ms": 123
}{
"seq": 123,
"type": "<string>",
"reply_id": "<string>",
"spoken_chars": 123,
"basis": "<string>"
}{
"seq": 123,
"type": "<string>",
"reply_id": "<string>",
"by": "<string>",
"heard_text": "<string>",
"basis": "<string>",
"final": true
}{
"seq": 88,
"type": "reply.done",
"reply_id": "r-1-9f3a",
"status": "interrupted",
"reason": null,
"by": "caller",
"heard_text": "Got it, adding a burger meal.",
"unheard_text": " What drink would you like with that?",
"heard_chars": 30,
"basis": "aligned",
"audio_end_ms": 4120
}{
"seq": 123,
"type": "<string>",
"kind": "<string>",
"start_ms": 123,
"end_ms": 123
}{
"seq": 123,
"type": "<string>",
"in_s": 123
}{
"seq": 123,
"type": "<string>",
"code": "<string>",
"message": "<string>",
"recoverable": true
}Pass your Fish Audio API key on the server-to-server WebSocket upgrade request:
Authorization: Bearer YOUR_API_KEY
Sent once as the very first frame within 5 s of connecting (or within 1 s when server capacity is full). Configures voice, audio format, and eager turn options for the entire call.
Continuous real-time caller audio stream (20 ms to 100 ms per frame), silence included. Acts as the session clock; stopping for over 30 s closes the call (4408 idle_timeout) unless listen.pause is active.
Streams text chunks from your LLM for the agent to speak. Starting a new reply_id preempts any currently speaking reply.
Cancels an in-progress or held speculative reply (for example, when turn.resumed arrives).
Mutes caller input processing (during IVR keypad entry, call transfer, or hold) and extends idle timeout from 30 s to 300 s.
Resumes caller audio listening and restores the 30 s idle timeout.
Confirms the session is live, returning active limits, audio rates, and LLM prompt text_rules. Begin streaming real-time caller audio upon receipt.
Continuous pcm_s16le mono agent audio stream at output_sample_rate (silence included). Buffer ~240–300 ms on playback.
Speculative end-of-turn signal emitted as soon as the caller pauses. Start your LLM stream immediately and send reply.text with "for": eager_id (gate mutating database/order writes until turn.committed).
Caller continued speaking after turn.eager. Cancel your in-flight speculative LLM stream and send reply.cancel.
Authoritative finalized caller turn. If approved_eager_id matches your active speculative run, Fish Audio immediately releases the pre-synthesized audio buffer (lead_ms reports latency saved) and your server can commit any gated write tools.
Sound stopped agent speech or triggered VAD without becoming a valid caller turn.
Short caller acknowledgement ("mm-hmm", "okay") while the agent holds the floor.
Live streaming ASR partial transcript for UI captions (turns.interim: true).
Emitted when the first audible speech sample of reply_id enters the output audio stream.
Periodic spoken character progress along reply_id.
Emitted up to twice when a reply is cut short (final: false at cut onset for immediate client speaker fade-out; final: true once agent audio settles).
Single authoritative receipt concluding every reply. Always append heard_text (never unplayed unheard_text) to your LLM conversation history.
Autonomous conversational backchannel or natural vocalization produced by the voice model outside explicit replies.
Warning emitted 120 s and 30 s before limits.max_session_s.
Recoverable or fatal protocol/subsystem error notification.
Was this page helpful?

