Operations

Day-2 guidance for the Kubernetes delivery forms. For the single-container form, see All-in-One container.

​

Ownership

Agree an escalation path before production traffic starts.

​

Monitoring signals

Prometheus is the metrics baseline. Watch:

• Pod readiness and restart counts for every service in the release, including Redis.
• Request success rate, error rate, latency, and time-to-first-audio.
• Queue depth, where the application exposes it.
• GPU utilization and GPU memory.
• CPU and memory usage against the configured requests and limits.
• Shared storage usage and latency.
• Redis availability and latency.
• Image pull failures and Kubernetes events in the release namespace.

Centralize application logs and retain Kubernetes events long enough to investigate a bad rollout. Redact secrets before sharing any logs outside your environment.

​

Scaling

Capacity is changed through the release configuration: API replicas, GPU worker replicas, CPU and memory requests, GPU resource requests, and per-worker concurrency. All of it takes effect on an upgrade. Scaling on load instead of by hand is included and off by default; while it is on, the replica count is the autoscaler’s rather than the file’s. Your documentation bundle covers turning it on. Add a GPU replica only when a GPU is actually free: a pod that requests one on a full cluster stays Pending indefinitely. Validate every scale change with the smoke test and a benchmark run at your expected concurrency, and remember that the first start of a new GPU worker pays the compile cost before it becomes ready.

​

Backup and retention

Back up:

• Reference voice archives, if your traffic uses reference ids. In a self-hosted deployment these are durable data, not cache — nothing can re-download them.
• The offline usage ledger directories, before they are pruned by your own archival process.
• The values file used for the production release.
• Secret manager entries.
• Dashboards and alert rules.

Compile and model caches on shared storage do not need backing up. Losing them costs a slow first start, nothing more.

​

Upgrades

Commands are in the deployment runbook. Before running them, capture the release history so you know the revision to roll back to, and keep that revision available until the new one is accepted.

​

Troubleshooting

The failures a Kubernetes deployment actually produces, and where each one usually comes from. The commands to diagnose each, and the fixes, are in the troubleshooting guide that ships in your documentation bundle.

​

Escalating to Fish Audio

Include:

• Kubernetes version, cloud provider, region, node types, and GPU type.
• Chart version and the values file with secrets removed.
• Pod status, relevant events, and logs with secrets redacted.
• The exact command that failed, with its output.
• Timestamp and time zone.