Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 8 additions & 4 deletions developer-guide/self-hosting/air-gapped.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -7,20 +7,24 @@
Both offline delivery forms make no outbound calls at runtime. Neither one, however,
installs itself out of thin air: a Helm install pulls container images and the chart
from a registry, and the appliance needs its image on the host. An air-gapped
deployment is about getting those artifacts across the boundary.
deployment is about getting those artifacts across the boundary. The license bundle
has to cross it too, and unlike an image it expires — plan a recurring transfer, not
a one-off.

<Note>
Fish Audio does this with you as part of an air-gapped delivery, and the step-by-step
procedure is in the deployment runbook for the version you install. See

Check warning on line 16 in developer-guide/self-hosting/air-gapped.mdx

View check run for this annotation

Mintlify / Mintlify Validation (hanabiaiinc) - vale-spellcheck

developer-guide/self-hosting/air-gapped.mdx#L16

Did you really mean 'runbook'?
[Releases](/developer-guide/self-hosting/enterprise-releases). This page is what to
expect and what to plan for.
</Note>

## Which form to choose

**The All-in-One appliance** is the straightforward answer to a strict air gap: one
image with every weight baked in, moved to the disconnected host as a file and loaded
there. There is nothing else to mirror.
**The air-gapped All-in-One appliance** is the straightforward answer to a strict air
gap: one image with every weight baked in, moved to the disconnected host as a file and
loaded there. There is nothing else to mirror, though the license bundle still has to
be carried across. Its online counterpart cannot run air
gapped: it authorises every request against Fish Audio.

Check warning on line 27 in developer-guide/self-hosting/air-gapped.mdx

View check run for this annotation

Mintlify / Mintlify Validation (hanabiaiinc) - vale-spellcheck

developer-guide/self-hosting/air-gapped.mdx#L27

Did you really mean 'gapped'?

**The Helm chart** is more work, because the release is many images rather than one.
Every image has to be mirrored into a registry the cluster can reach and the release
Expand All @@ -46,13 +50,13 @@
## Proving there is no egress

Regulated deployments usually need evidence rather than a configuration review, and the
runbook carries the exercise. Two things are worth knowing before you plan it.

Check warning on line 53 in developer-guide/self-hosting/air-gapped.mdx

View check run for this annotation

Mintlify / Mintlify Validation (hanabiaiinc) - vale-spellcheck

developer-guide/self-hosting/air-gapped.mdx#L53

Did you really mean 'runbook'?

It answers two questions, and they need different methods: whether the deployment
*needs* the internet, and whether it *calls out* when allowed to. The second is what a
telemetry or data-residency review actually asks.

It is also a cluster-level exercise rather than a namespace one. Confirm early that

Check warning on line 59 in developer-guide/self-hosting/air-gapped.mdx

View check run for this annotation

Mintlify / Mintlify Validation (hanabiaiinc) - vale-spellcheck

developer-guide/self-hosting/air-gapped.mdx#L59

Did you really mean 'namespace'?
whoever runs your cluster can take part, because a namespace-scoped account cannot
complete it.

Expand Down
43 changes: 31 additions & 12 deletions developer-guide/self-hosting/all-in-one.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -5,10 +5,15 @@
---

The All-in-One image packages the whole speech stack (edge API, model API layer,
inference router and worker, vocoder, text normalizer, and Redis) into one container,

Check warning on line 8 in developer-guide/self-hosting/all-in-one.mdx

View check run for this annotation

Mintlify / Mintlify Validation (hanabiaiinc) - vale-spellcheck

developer-guide/self-hosting/all-in-one.mdx#L8

Did you really mean 'vocoder'?
with every model weight baked in. Once the image is on the host it runs with no
Kubernetes and no internet access, which makes it the turnkey option for single-node
appliances and strict air gaps.
with every model weight baked in. It runs as a turnkey single-node appliance, no Kubernetes.

The two forms, **Offline All-in-One** and **Online All-in-One**, differ only in how
usage is accounted for. The offline form records usage to a local
ledger and needs no network once the image is on the host. The online form authorises
each request against Fish Audio and charges the key that made it, so it needs outbound
access and a key with credit. The accounting is compiled in, so an image is one or the
other; the container logs which at startup.

<Note>
This page covers what the appliance is and what to plan for. The commands, tuning
Expand All @@ -19,12 +24,11 @@

## What it cannot do

The appliance runs one inference worker and one vocoder, a GPU each. It does not

Check warning on line 27 in developer-guide/self-hosting/all-in-one.mdx

View check run for this annotation

Mintlify / Mintlify Validation (hanabiaiinc) - vale-spellcheck

developer-guide/self-hosting/all-in-one.mdx#L27

Did you really mean 'vocoder'?
autoscale, does not shard across more GPUs or nodes, and does not ship the forced

Check warning on line 28 in developer-guide/self-hosting/all-in-one.mdx

View check run for this annotation

Mintlify / Mintlify Validation (hanabiaiinc) - vale-spellcheck

developer-guide/self-hosting/all-in-one.mdx#L28

Did you really mean 'autoscale'?

Check warning on line 28 in developer-guide/self-hosting/all-in-one.mdx

View check run for this annotation

Mintlify / Mintlify Validation (hanabiaiinc) - vale-spellcheck

developer-guide/self-hosting/all-in-one.mdx#L28

Did you really mean 'GPUs'?
aligner, so it returns no word or segment timings. It is offline-only: there is no
hosted-billing variant. For elastic or higher-throughput deployments, use the
[Kubernetes chart](/developer-guide/self-hosting/kubernetes), which scales replicas
across all GPUs and nodes.
aligner, so it returns no word or segment timings. For elastic or higher-throughput
deployments, use the [Kubernetes chart](/developer-guide/self-hosting/kubernetes), which
scales replicas across all GPUs and nodes.

Check warning on line 31 in developer-guide/self-hosting/all-in-one.mdx

View check run for this annotation

Mintlify / Mintlify Validation (hanabiaiinc) - vale-spellcheck

developer-guide/self-hosting/all-in-one.mdx#L31

Did you really mean 'GPUs'?

## What running it involves

Expand All @@ -32,32 +36,47 @@
[All-in-One host requirements](/developer-guide/self-hosting/requirements#all-in-one-container-host).
It needs:

- **Two GPUs.** The first runs the inference worker, the second the vocoder.

Check warning on line 39 in developer-guide/self-hosting/all-in-one.mdx

View check run for this annotation

Mintlify / Mintlify Validation (hanabiaiinc) - vale-spellcheck

developer-guide/self-hosting/all-in-one.mdx#L39

Did you really mean 'GPUs'?

Check warning on line 39 in developer-guide/self-hosting/all-in-one.mdx

View check run for this annotation

Mintlify / Mintlify Validation (hanabiaiinc) - vale-spellcheck

developer-guide/self-hosting/all-in-one.mdx#L39

Did you really mean 'vocoder'?
- **One exposed port** for the API.
- **One persistent volume.** Compile caches, the vocoder's built engine, reference

Check warning on line 41 in developer-guide/self-hosting/all-in-one.mdx

View check run for this annotation

Mintlify / Mintlify Validation (hanabiaiinc) - vale-spellcheck

developer-guide/self-hosting/all-in-one.mdx#L41

Did you really mean 'vocoder's'?
voice archives, and the usage ledger all live there. Model weights are in the image,
not on the volume.
voice archives, and — on the offline form — the usage ledger all live there. Model
weights are in the image, not on the volume.
- **The license bundle.** A host directory holding the certificate and key pair from
**Developer → Self Host**, mounted read-only. Without it the API never comes up; once
the certificate expires the container starts but refuses requests. Renewing is
replacing those files in place; only a bundle that carries a new key needs the
container restarted.

Everything inside the container runs as a non-root user, so a reused volume or a host
bind mount has to be writable by it.

**Plan for a slow first start.** The worker compiles its inference graphs and the
vocoder builds its engine before either serves, and the health endpoint verifies the

Check warning on line 54 in developer-guide/self-hosting/all-in-one.mdx

View check run for this annotation

Mintlify / Mintlify Validation (hanabiaiinc) - vale-spellcheck

developer-guide/self-hosting/all-in-one.mdx#L54

Did you really mean 'vocoder'?
speech backend end to end rather than reporting immediate liveness. Both artifacts are

Check warning on line 55 in developer-guide/self-hosting/all-in-one.mdx

View check run for this annotation

Mintlify / Mintlify Validation (hanabiaiinc) - vale-spellcheck

developer-guide/self-hosting/all-in-one.mdx#L55

Did you really mean 'liveness'?
cached on the volume, so later starts take minutes. The vocoder engine is specific to

Check warning on line 56 in developer-guide/self-hosting/all-in-one.mdx

View check run for this annotation

Mintlify / Mintlify Validation (hanabiaiinc) - vale-spellcheck

developer-guide/self-hosting/all-in-one.mdx#L56

Did you really mean 'vocoder'?
the GPU model, so moving to different cards rebuilds it once.

## Usage accounting and tenancy

This build records usage to a local, signed, append-only ledger on the volume instead
of calling a billing service. Two consequences worth designing around:
The form determines how usage is accounted for and what the bearer token has to be.

**Offline.** Usage goes to a local, signed, append-only ledger on the volume. Two
consequences worth designing around:

- **Any non-empty bearer token is accepted**; a missing or empty one is rejected. The
appliance has nothing to validate a token against.
- **The token is recorded verbatim as the billing identity.** Use a stable, distinct
token per tenant. Two tenants sharing a token are indistinguishable in the ledger.

See [Offline usage accounting](/developer-guide/self-hosting/air-gapped#offline-usage-accounting).
Backups and reconciliation: [Offline usage accounting](/developer-guide/self-hosting/air-gapped#offline-usage-accounting).

**Online.** Each request is authorised against Fish Audio and charged to the key that
made it. The token must be a [Fish Audio API key](/developer-guide/getting-started/api-key)
with credit — not the deploy token used to pull the image — and anything else is
rejected. No ledger is written; usage appears in your Fish Audio account instead.

Authorisation happens on every request, not only at startup, and the appliance has no
local fallback. Treat outbound reachability as a production dependency.

Check warning on line 79 in developer-guide/self-hosting/all-in-one.mdx

View check run for this annotation

Mintlify / Mintlify Validation (hanabiaiinc) - vale-spellcheck

developer-guide/self-hosting/all-in-one.mdx#L79

Did you really mean 'reachability'?

## Capacity

Expand Down
3 changes: 2 additions & 1 deletion developer-guide/self-hosting/enterprise-releases.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -16,13 +16,14 @@
| All-in-One | An image tag | The tag in the image reference |

Both Helm forms are one chart, so they share a version. Which form you deploy is
decided by the values file, not by the version.
decided by the values file, not by the version. The two All-in-One forms are separate
images with independent tags.

## Upgrading

A deployment stays on its version until you change it. Take the new version from
**Developer → Self Host**, then follow the upgrade
procedure in the deployment runbook for the Helm forms, or pull the new tag and

Check warning on line 26 in developer-guide/self-hosting/enterprise-releases.mdx

View check run for this annotation

Mintlify / Mintlify Validation (hanabiaiinc) - vale-spellcheck

developer-guide/self-hosting/enterprise-releases.mdx#L26

Did you really mean 'runbook'?
recreate the container against the same volume for
[All-in-One](/developer-guide/self-hosting/all-in-one). Air-gapped deployments
mirror the new version first. See
Expand Down
22 changes: 11 additions & 11 deletions developer-guide/self-hosting/introduction.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -16,25 +16,25 @@

| Reason | What it gives you |
| ----------------------- | --------------------------------------------------------------------------------------------------------------------------------------- |
| Colocation | Run inference in the same region, VPC, or rack as your application and remove public-internet round trips from time-to-first-audio. |

Check warning on line 19 in developer-guide/self-hosting/introduction.mdx

View check run for this annotation

Mintlify / Mintlify Validation (hanabiaiinc) - vale-spellcheck

developer-guide/self-hosting/introduction.mdx#L19

Did you really mean 'Colocation'?
| Single-tenant isolation | Dedicated GPUs and queues. Capacity is not shared with other tenants, and you decide when the deployment is upgraded. |

Check warning on line 20 in developer-guide/self-hosting/introduction.mdx

View check run for this annotation

Mintlify / Mintlify Validation (hanabiaiinc) - vale-spellcheck

developer-guide/self-hosting/introduction.mdx#L20

Did you really mean 'GPUs'?
| Security posture | Voice traffic never leaves your network. The offline delivery forms make no outbound calls at runtime and run on disconnected networks. |
| Data sovereignty | Input text, generated audio, and reference voices stay inside your boundary and under your own retention policy. |

## Delivery forms

The same engine ships in three forms. Your enterprise agreement determines which
The same engine ships in four forms. Your enterprise agreement determines which
ones your team is granted.

| | Online Helm | Offline Helm | Offline All-in-One |
| ---------------- | --------------------------------------------------- | ------------------------------------------------ | --------------------------------------------------- |
| Platform | Kubernetes | Kubernetes | A single container, `docker run` |
| Model assets | Served in-cluster from a bundled model warehouse | Same | Baked into the image |
| Usage accounting | Validated and billed against the Fish Audio service | Local signed usage ledger | Local signed usage ledger |
| Runtime egress | The billing endpoint, plus one more host if you enable timestamps | None | None |
| Air-gap capable | No | Yes, after mirroring images and charts | Yes |
| Scaling | Scales replicas across all GPUs and nodes | Same | Single node, single inference worker |
| Best for | Managed clusters with outbound access | Isolated or regulated production clusters | Evaluation, single-node appliances, strict air gaps |
| | Online Helm | Offline Helm | Offline All-in-One | Online All-in-One |
| ---------------- | --------------------------------------------------- | ------------------------------------------------ | --------------------------------------------------- | ------------------------------------------------ |
| Platform | Kubernetes | Kubernetes | A single container, `docker run` | A single container, `docker run` |
| Model assets | Served in-cluster from a bundled model warehouse | Same | Baked into the image | Baked into the image |
| Usage accounting | Validated and billed against the Fish Audio service | Local signed usage ledger | Local signed usage ledger | Validated and billed against the Fish Audio service |
| Runtime egress | The billing endpoint, plus one more host if you enable timestamps | None | None | The billing endpoint |
| Air-gap capable | No | Yes, after mirroring images and charts | Yes | No |
| Scaling | Scales replicas across all GPUs and nodes | Same | Single node, single inference worker | Single node, single inference worker |

Check warning on line 36 in developer-guide/self-hosting/introduction.mdx

View check run for this annotation

Mintlify / Mintlify Validation (hanabiaiinc) - vale-spellcheck

developer-guide/self-hosting/introduction.mdx#L36

Did you really mean 'GPUs'?
| Best for | Managed clusters with outbound access | Isolated or regulated production clusters | Evaluation, single-node appliances, strict air gaps | Single-node appliances billed like the hosted service |

Both Helm forms share one deployment procedure and differ only in a few values.
See [Kubernetes deployment](/developer-guide/self-hosting/kubernetes) for the
Expand All @@ -43,13 +43,13 @@

## What is included

| Capability | Online Helm | Offline Helm | All-in-One |
| Capability | Online Helm | Offline Helm | All-in-One (both forms) |
| ----------------------------------------- | ---------------------------- | ------------------------ | ------------------------ |
| Text to speech over `POST /v1/tts` | Included | Included | Included |
| WebSocket streaming | Included | Included | Included |
| Reference-voice requests (`reference_id`) | From pre-staged archives | From pre-staged archives | From pre-staged archives |
| Word and segment timestamps | Optional, needs an extra GPU | Not included | Not included |
| Horizontal scaling and autoscaling | Included | Included | Not included |

Check warning on line 52 in developer-guide/self-hosting/introduction.mdx

View check run for this annotation

Mintlify / Mintlify Validation (hanabiaiinc) - vale-spellcheck

developer-guide/self-hosting/introduction.mdx#L52

Did you really mean 'autoscaling'?
| Prometheus metrics | Included | Included | Container logs only |

<Note>
Expand All @@ -73,13 +73,13 @@
| Model API layer | Coordinates normalization, inference, and audio decoding. |
| Inference router | Distributes work across the GPU workers. |
| Inference workers | GPU-backed generation. |
| Vocoder | GPU-backed audio decoding. |

Check warning on line 76 in developer-guide/self-hosting/introduction.mdx

View check run for this annotation

Mintlify / Mintlify Validation (hanabiaiinc) - vale-spellcheck

developer-guide/self-hosting/introduction.mdx#L76

Did you really mean 'Vocoder'?
| Text normalizer | Text normalization ahead of inference. |
| Redis | Runtime state and cache. Installed by the chart into the release namespace. |

Check warning on line 78 in developer-guide/self-hosting/introduction.mdx

View check run for this annotation

Mintlify / Mintlify Validation (hanabiaiinc) - vale-spellcheck

developer-guide/self-hosting/introduction.mdx#L78

Did you really mean 'namespace'?
| Shared storage | Model cache, reference archives, and the usage ledger. Mounted on every node. |

Fish Audio delivers the deployment at the Kubernetes service level. Ingress, DNS,
TLS, external load balancing, and network allowlists are yours to choose and

Check warning on line 82 in developer-guide/self-hosting/introduction.mdx

View check run for this annotation

Mintlify / Mintlify Validation (hanabiaiinc) - vale-spellcheck

developer-guide/self-hosting/introduction.mdx#L82

Did you really mean 'allowlists'?
configure.

## Getting access
Expand Down
15 changes: 9 additions & 6 deletions developer-guide/self-hosting/kubernetes.mdx
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
---
title: "Kubernetes deployment"
description: "What the Helm delivery installs, what you decide before installing, and what to expect"
icon: "dharmachakra"

Check warning on line 4 in developer-guide/self-hosting/kubernetes.mdx

View check run for this annotation

Mintlify / Mintlify Validation (hanabiaiinc) - vale-spellcheck

developer-guide/self-hosting/kubernetes.mdx#L4

Did you really mean 'dharmachakra'?
---

Fish Audio Enterprise ships as a Helm chart that installs the whole speech stack into
Expand All @@ -10,9 +10,9 @@

<Note>
This page covers what the deployment involves and what you need to decide. The
commands, values, and troubleshooting are in the **deployment runbook**, which ships

Check warning on line 13 in developer-guide/self-hosting/kubernetes.mdx

View check run for this annotation

Mintlify / Mintlify Validation (hanabiaiinc) - vale-spellcheck

developer-guide/self-hosting/kubernetes.mdx#L13

Did you really mean 'runbook'?
in the documentation bundle alongside the chart version you install. See
[Releases](/developer-guide/self-hosting/enterprise-releases). The runbook is

Check warning on line 15 in developer-guide/self-hosting/kubernetes.mdx

View check run for this annotation

Mintlify / Mintlify Validation (hanabiaiinc) - vale-spellcheck

developer-guide/self-hosting/kubernetes.mdx#L15

Did you really mean 'runbook'?
versioned with the chart; this page is not, so follow the runbook when they differ.
</Note>

Expand All @@ -32,8 +32,8 @@
## What gets installed

The chart deploys the edge API, the model API layer, an inference router and worker,
a vocoder, a text normalizer, the model warehouse that serves the weights, and Redis.

Check warning on line 35 in developer-guide/self-hosting/kubernetes.mdx

View check run for this annotation

Mintlify / Mintlify Validation (hanabiaiinc) - vale-spellcheck

developer-guide/self-hosting/kubernetes.mdx#L35

Did you really mean 'vocoder'?
Everything lands in one namespace, in the single-worker shape the delivery was sized

Check warning on line 36 in developer-guide/self-hosting/kubernetes.mdx

View check run for this annotation

Mintlify / Mintlify Validation (hanabiaiinc) - vale-spellcheck

developer-guide/self-hosting/kubernetes.mdx#L36

Did you really mean 'namespace'?
against.

Ingress and TLS are off by default. Put your own ingress controller or load balancer
Expand All @@ -44,19 +44,22 @@

| Decision | Notes |
| --------------------------- | ------------------------------------------------------------------------------------------------------------------------- |
| Namespace | `fish-audio` is the supported default. A different one has to be set in two places; the runbook says where. |

Check warning on line 47 in developer-guide/self-hosting/kubernetes.mdx

View check run for this annotation

Mintlify / Mintlify Validation (hanabiaiinc) - vale-spellcheck

developer-guide/self-hosting/kubernetes.mdx#L47

Did you really mean 'Namespace'?

Check warning on line 47 in developer-guide/self-hosting/kubernetes.mdx

View check run for this annotation

Mintlify / Mintlify Validation (hanabiaiinc) - vale-spellcheck

developer-guide/self-hosting/kubernetes.mdx#L47

Did you really mean 'runbook'?
| Shared storage path | Mounted at the same path on **every** node. See [Requirements](/developer-guide/self-hosting/requirements#shared-storage). |
| GPU scheduling | Tolerations and node selectors, if your GPU nodes are tainted or you run more than one GPU model. |

Check warning on line 49 in developer-guide/self-hosting/kubernetes.mdx

View check run for this annotation

Mintlify / Mintlify Validation (hanabiaiinc) - vale-spellcheck

developer-guide/self-hosting/kubernetes.mdx#L49

Did you really mean 'Tolerations'?
| Metrics | The chart exposes Prometheus annotations and creates no ServiceMonitors, so an annotation-scraping Prometheus works as-is and kube-prometheus-stack needs a scrape config. |
| Reference voices | Requests carrying a `reference_id` resolve from a local archive, never from the network. One zip per voice, staged on shared storage before that traffic starts. |
| Replica counts | Talk to your account team first. The shipped shape is what the delivery was sized and validated against. |

Three Kubernetes Secrets have to exist before the install: registry credentials, a JWT
secret for the edge API, and one shared between the in-cluster model store and the
workers that read from it. That one authenticates nothing outside the cluster, and the
value is yours to choose. Generate the JWT secret once
and keep it stable, because changing it invalidates issued tokens. For production, prefer
External Secrets, Sealed Secrets, or your cloud secret manager over plain Secrets.
Three Kubernetes Secrets have to exist before the install: registry credentials, the
license bundle, and one shared between the in-cluster model store and the workers that
read from it. The license bundle is the certificate and key pair downloaded from
**Developer → Self Host**; the services that enforce it stay down until the Secret
exists, and refuse requests once the certificate lapses — renewal is described in
[Operations](/developer-guide/self-hosting/operations#license-renewal). The model-store
secret authenticates nothing outside the cluster, so the value is yours to choose. For
production, prefer External Secrets, Sealed Secrets, or your cloud secret manager over
plain Secrets.

## What to expect

Expand All @@ -64,11 +67,11 @@
rather than by editing tags.

The first install is slow: the inference worker compiles its GPU graphs and the
vocoder builds its inference engine before either becomes ready, which takes far longer

Check warning on line 70 in developer-guide/self-hosting/kubernetes.mdx

View check run for this annotation

Mintlify / Mintlify Validation (hanabiaiinc) - vale-spellcheck

developer-guide/self-hosting/kubernetes.mdx#L70

Did you really mean 'vocoder'?
than any later rollout. Both artifacts are cached on shared storage, so subsequent

Check warning on line 71 in developer-guide/self-hosting/kubernetes.mdx

View check run for this annotation

Mintlify / Mintlify Validation (hanabiaiinc) - vale-spellcheck

developer-guide/self-hosting/kubernetes.mdx#L71

Did you really mean 'rollout'?
starts are quick, as long as that storage persists.

Upgrades, rollback, and uninstall are all standard Helm operations. The runbook covers

Check warning on line 74 in developer-guide/self-hosting/kubernetes.mdx

View check run for this annotation

Mintlify / Mintlify Validation (hanabiaiinc) - vale-spellcheck

developer-guide/self-hosting/kubernetes.mdx#L74

Did you really mean 'runbook'?
the order to do them in and what to check at each step.

## Next steps
Expand Down
19 changes: 18 additions & 1 deletion developer-guide/self-hosting/operations.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -30,10 +30,12 @@
- CPU and memory usage against the configured requests and limits.
- Shared storage usage and latency.
- Redis availability and latency.
- License validity and time to expiry, exported as metrics; the services also log
warnings well before expiry.
- Image pull failures and Kubernetes events in the release namespace.

Check warning on line 35 in developer-guide/self-hosting/operations.mdx

View check run for this annotation

Mintlify / Mintlify Validation (hanabiaiinc) - vale-spellcheck

developer-guide/self-hosting/operations.mdx#L35

Did you really mean 'namespace'?

Centralize application logs and retain Kubernetes events long enough to
investigate a bad rollout. Redact secrets before sharing any logs outside your

Check warning on line 38 in developer-guide/self-hosting/operations.mdx

View check run for this annotation

Mintlify / Mintlify Validation (hanabiaiinc) - vale-spellcheck

developer-guide/self-hosting/operations.mdx#L38

Did you really mean 'rollout'?
environment.

## Scaling
Expand All @@ -41,7 +43,7 @@
Capacity is changed through the release configuration: API replicas, GPU worker
replicas, CPU and memory requests, GPU resource requests, and per-worker concurrency.
All of it takes effect on an upgrade. Scaling on load instead of by hand is included
and off by default; while it is on, the replica count is the autoscaler's rather than

Check warning on line 46 in developer-guide/self-hosting/operations.mdx

View check run for this annotation

Mintlify / Mintlify Validation (hanabiaiinc) - vale-spellcheck

developer-guide/self-hosting/operations.mdx#L46

Did you really mean 'autoscaler's'?
the file's. Your documentation bundle covers turning it on.

Add a GPU replica only when a GPU is actually free: a pod that requests one on a full
Expand All @@ -66,9 +68,22 @@
Compile and model caches on shared storage do not need backing up. Losing them
costs a slow first start, nothing more.

## License renewal

Update the license Secret in place with the renewed bundle; no restart is needed.
Kubernetes propagates the change to the running pods, and each service re-verifies on
its own short cycle, so expect up to a couple of minutes of mixed responses before
it settles. One exception: a bundle
that carries a new key rather than a re-signed certificate. The offline form's edge API
reads the ledger-signing key only at startup, so restart that deployment after
replacing a key.

Pods with an expired certificate keep running but refuse requests, then recover once
a renewal propagates.

## Upgrades

Commands are in the deployment runbook. Before running them, capture the release

Check warning on line 86 in developer-guide/self-hosting/operations.mdx

View check run for this annotation

Mintlify / Mintlify Validation (hanabiaiinc) - vale-spellcheck

developer-guide/self-hosting/operations.mdx#L86

Did you really mean 'runbook'?
history so you know the revision to roll back to, and keep that revision available
until the new one is accepted.

Expand All @@ -81,13 +96,15 @@
| Symptom | Likely causes |
| ------------------------------------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `ImagePullBackOff` or `ErrImagePull` | Missing pull secret, a deploy token that was rotated or deleted, a tag that is not in your mirror, or blocked egress to the registry. |
| Pods stay `Pending` | Not enough CPU, memory, or GPU capacity; GPU node taints without matching tolerations; a node selector that matches nothing. |

Check warning on line 99 in developer-guide/self-hosting/operations.mdx

View check run for this annotation

Mintlify / Mintlify Validation (hanabiaiinc) - vale-spellcheck

developer-guide/self-hosting/operations.mdx#L99

Did you really mean 'tolerations'?
| Pods stuck `ContainerCreating` | A Secret the pod mounts does not exist yet — most often the license bundle. The pods recover on their own once it is created. |
| GPU not available, or `nvidia-smi` fails in a pod | Driver missing or unhealthy, device plugin or GPU Operator not ready, container runtime not configured for NVIDIA, or an unsupported GPU for the driver stack. |
| hostPath volume errors, or missing model and cache paths | Shared storage is not mounted on that node, a newly added node never got the mount, the mount target is unreachable, or permissions prevent writes. |

Check warning on line 102 in developer-guide/self-hosting/operations.mdx

View check run for this annotation

Mintlify / Mintlify Validation (hanabiaiinc) - vale-spellcheck

developer-guide/self-hosting/operations.mdx#L102

Did you really mean 'hostPath'?
| Requests fail after reaching the edge API, with Redis errors in its logs | The Redis master or replica is not ready, the service is missing, or a network policy blocks it. |
| `Reference not found` | The archive was never staged, the filename does not exactly match the reference id, the zip is malformed, or the reference root was changed without mounting the new path into every edge API pod. |
| Metrics missing from Prometheus | Scrape annotations disabled in values, Prometheus not configured for annotation-based discovery, a port mismatch, or a network policy blocking scrapes. |
| `CrashLoopBackOff` | A missing secret or environment variable, model assets unreachable, or a path the container cannot write. |
| `CrashLoopBackOff` | A missing environment variable, a license file missing or unreadable inside the Secret, model assets unreachable, or a path the container cannot write. |
| Requests refused with a license error | The certificate expired, or a renewal has not finished propagating — see [License renewal](#license-renewal). |
| Time-to-first-audio high, GPU utilization low | Concurrency above what the current replicas can serve, cold caches after a restart, storage latency, or CPU saturation on the API nodes. |

## Escalating to Fish Audio
Expand Down
3 changes: 3 additions & 0 deletions developer-guide/self-hosting/registry-access.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,9 @@
page reports that self-host deployment is not enabled, contact your account manager.
- **The versions available to you**, and the documentation bundle for each. See
[Releases](/developer-guide/self-hosting/enterprise-releases).
- **Your license bundle**: the certificate and key the deployment needs before it will
serve. Some contracts bind it to specific GPUs — have the card UUIDs ready when you

Check warning on line 21 in developer-guide/self-hosting/registry-access.mdx

View check run for this annotation

Mintlify / Mintlify Validation (hanabiaiinc) - vale-spellcheck

developer-guide/self-hosting/registry-access.mdx#L21

Did you really mean 'GPUs'?

Check warning on line 21 in developer-guide/self-hosting/registry-access.mdx

View check run for this annotation

Mintlify / Mintlify Validation (hanabiaiinc) - vale-spellcheck

developer-guide/self-hosting/registry-access.mdx#L21

Did you really mean 'UUIDs'?
request it.
- **Install commands built for your team**, with the registry host, artifact references,
and the version you pick already filled in.

Expand Down Expand Up @@ -54,7 +57,7 @@

Authenticate with your account email as the username and the deploy token as the
password. Docker, Helm, and the cluster's image pull secret all use the same pair; the
exact commands are on the Self Host page and in the deployment runbook.

Check warning on line 60 in developer-guide/self-hosting/registry-access.mdx

View check run for this annotation

Mintlify / Mintlify Validation (hanabiaiinc) - vale-spellcheck

developer-guide/self-hosting/registry-access.mdx#L60

Did you really mean 'runbook'?

## Managing tokens

Expand Down
3 changes: 3 additions & 0 deletions developer-guide/self-hosting/requirements.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -16,8 +16,8 @@
| Node type | Minimum | Purpose |
| -------------------------------------- | ---------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| CPU-only control-plane or system nodes | 3 nodes, roughly 4 vCPU and 16 GB RAM each | Control-plane high availability on self-managed Kubernetes, or a system node group for platform add-ons on managed Kubernetes. |
| CPU workload capacity | Roughly 28 vCPU and 125 GiB of memory requests | Non-GPU workloads: Redis, the edge API, the text normalizer, the model API layer, and the inference router. Provide a CPU node group, or leave GPU nodes schedulable so these can land there. |

Check warning on line 19 in developer-guide/self-hosting/requirements.mdx

View check run for this annotation

Mintlify / Mintlify Validation (hanabiaiinc) - vale-spellcheck

developer-guide/self-hosting/requirements.mdx#L19

Did you really mean 'schedulable'?
| GPU capacity | 2 GPUs | One for the inference worker, one for the vocoder. Spread them across two nodes if the deployment also has to survive a node drain or failure. |

Check warning on line 20 in developer-guide/self-hosting/requirements.mdx

View check run for this annotation

Mintlify / Mintlify Validation (hanabiaiinc) - vale-spellcheck

developer-guide/self-hosting/requirements.mdx#L20

Did you really mean 'GPUs'?

Check warning on line 20 in developer-guide/self-hosting/requirements.mdx

View check run for this annotation

Mintlify / Mintlify Validation (hanabiaiinc) - vale-spellcheck

developer-guide/self-hosting/requirements.mdx#L20

Did you really mean 'vocoder'?
| Shared storage | 1 TB usable, mounted on every node | Model cache, reference archives, compile caches, and the offline usage ledger. |
| Network | Low-latency east-west networking | GPU workload stability, service-to-service calls, and shared storage access. |

Expand All @@ -25,11 +25,11 @@
profile applied (the chart defaults alone stop on a required value) and summing the
requests.

### GPUs

Check warning on line 28 in developer-guide/self-hosting/requirements.mdx

View check run for this annotation

Mintlify / Mintlify Validation (hanabiaiinc) - vale-spellcheck

developer-guide/self-hosting/requirements.mdx#L28

Did you really mean 'GPUs'?

| Requirement | Detail |
| ------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Count | 2 GPUs minimum: one inference worker, one vocoder. Enabling timestamp alignment adds a third. |

Check warning on line 32 in developer-guide/self-hosting/requirements.mdx

View check run for this annotation

Mintlify / Mintlify Validation (hanabiaiinc) - vale-spellcheck

developer-guide/self-hosting/requirements.mdx#L32

Did you really mean 'GPUs'?

Check warning on line 32 in developer-guide/self-hosting/requirements.mdx

View check run for this annotation

Mintlify / Mintlify Validation (hanabiaiinc) - vale-spellcheck

developer-guide/self-hosting/requirements.mdx#L32

Did you really mean 'vocoder'?
| Type | NVIDIA H100 or H200. H200 is the preferred target for its larger memory and bandwidth. |
| Memory | The worker pre-allocates most of its card for the KV cache, so a near-full card is expected, not a fault. A larger card raises the concurrency ceiling rather than leaving headroom. |
| Newer architectures | Contact Fish Audio before standardizing on a GPU generation that is not H100 or H200, so the image, CUDA stack, and driver combination can be confirmed. |
Expand All @@ -44,18 +44,18 @@
Provide at least 1 TB of usable shared storage backed by EFS, NFS, or an
equivalent service, mounted at the same path on every node that runs Fish Audio
workloads (`/mnt/share` by default, and configurable). The chart mounts it into containers with
hostPath volumes and creates no PersistentVolumeClaims.

Check warning on line 47 in developer-guide/self-hosting/requirements.mdx

View check run for this annotation

Mintlify / Mintlify Validation (hanabiaiinc) - vale-spellcheck

developer-guide/self-hosting/requirements.mdx#L47

Did you really mean 'hostPath'?

<Warning>
The hostPath mount is created with `DirectoryOrCreate`. A node that is missing

Check warning on line 50 in developer-guide/self-hosting/requirements.mdx

View check run for this annotation

Mintlify / Mintlify Validation (hanabiaiinc) - vale-spellcheck

developer-guide/self-hosting/requirements.mdx#L50

Did you really mean 'hostPath'?
the shared mount silently gets a local directory instead, and the deployment
looks healthy while data splits across nodes. Verify the mount, and that it is
writable, on **every** node before installing. The deployment runbook gives a check

Check warning on line 53 in developer-guide/self-hosting/requirements.mdx

View check run for this annotation

Mintlify / Mintlify Validation (hanabiaiinc) - vale-spellcheck

developer-guide/self-hosting/requirements.mdx#L53

Did you really mean 'runbook'?
that does this without node access.
</Warning>

Confirm the following with your storage team: how the mount is applied to newly
added nodes, the throughput mode, the backup policy, mount target reachability,

Check warning on line 58 in developer-guide/self-hosting/requirements.mdx

View check run for this annotation

Mintlify / Mintlify Validation (hanabiaiinc) - vale-spellcheck

developer-guide/self-hosting/requirements.mdx#L58

Did you really mean 'reachability'?
and expected growth of the model and cache data.

Redis is installed by the chart and does not use shared storage. Its append-only
Expand All @@ -68,17 +68,18 @@
| ----------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| Kubernetes | A currently supported minor version. Stay one minor behind the newest release if your GPU add-ons have not been validated on it yet. |
| Helm | A current maintained release supported by your platform. |
| Container runtime | The provider-managed containerd runtime where available. |

Check warning on line 71 in developer-guide/self-hosting/requirements.mdx

View check run for this annotation

Mintlify / Mintlify Validation (hanabiaiinc) - vale-spellcheck

developer-guide/self-hosting/requirements.mdx#L71

Did you really mean 'containerd'?
| GPU runtime | NVIDIA GPU Operator, or the provider-managed driver and device plugin stack. GPU nodes must expose `nvidia.com/gpu`. |
| GPU telemetry | DCGM exporter or the provider equivalent. |
| Storage | EFS, NFS, or an equivalent shared filesystem mounted on every node. |
| Object storage | Not required. Models are served from inside the cluster in every delivery form. Hosting the weights in your own bucket instead is possible; ask your account team. |
| Redis | Installed by the chart into the release namespace. Do not point the release at a shared Redis without discussing it first. |

Check warning on line 76 in developer-guide/self-hosting/requirements.mdx

View check run for this annotation

Mintlify / Mintlify Validation (hanabiaiinc) - vale-spellcheck

developer-guide/self-hosting/requirements.mdx#L76

Did you really mean 'namespace'?
| Metrics | Prometheus, scraping `prometheus.io/*` pod annotations. The chart creates no ServiceMonitors, so kube-prometheus-stack users must add an annotation-based scrape config. |
| Access layer | Yours to choose. Ingress controller, DNS, TLS, and load balancing are not part of the delivery. |
| License bundle | The certificate and key from **Developer → Self Host**, loaded into the cluster as a Secret before the install. |

The cluster also needs working in-cluster DNS, a dedicated namespace, node labels

Check warning on line 81 in developer-guide/self-hosting/requirements.mdx

View check run for this annotation

Mintlify / Mintlify Validation (hanabiaiinc) - vale-spellcheck

developer-guide/self-hosting/requirements.mdx#L81

Did you really mean 'namespace'?
and taints for GPU scheduling, and a Pod Security level in that namespace that

Check warning on line 82 in developer-guide/self-hosting/requirements.mdx

View check run for this annotation

Mintlify / Mintlify Validation (hanabiaiinc) - vale-spellcheck

developer-guide/self-hosting/requirements.mdx#L82

Did you really mean 'namespace'?
permits `hostPID` and `hostIPC`, which the inference workers require.

### Network
Expand All @@ -87,7 +88,7 @@
| ---------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| Install-time egress | Access to the Fish Audio registry to pull images and the chart. Mirror both into your own registry for air-gapped installs. |
| Runtime egress, offline form | None. No model download and no billing call. |
| Runtime egress, online form | HTTPS to the Fish Audio authorization and billing endpoint. Models are served from inside the cluster, so no object-store endpoint has to be reachable. Ask your account team for the billing hostname to allowlist. Enabling the timestamp aligner adds one more host. |

Check warning on line 91 in developer-guide/self-hosting/requirements.mdx

View check run for this annotation

Mintlify / Mintlify Validation (hanabiaiinc) - vale-spellcheck

developer-guide/self-hosting/requirements.mdx#L91

Did you really mean 'hostname'?

Check warning on line 91 in developer-guide/self-hosting/requirements.mdx

View check run for this annotation

Mintlify / Mintlify Validation (hanabiaiinc) - vale-spellcheck

developer-guide/self-hosting/requirements.mdx#L91

Did you really mean 'allowlist'?
| Ingress | Customer-approved ingress or a private endpoint, with DNS and TLS in place before production traffic. |

## All-in-One container host
Expand All @@ -95,13 +96,15 @@
| Requirement | Detail |
| ----------------- | -------------------------------------------------------------------------------------------------------------------------------- |
| OS | Linux x86-64. |
| GPUs | 2 GPUs, 32 GB of memory or larger. The first runs the inference worker, the second runs the vocoder. No NVLink required. |

Check warning on line 99 in developer-guide/self-hosting/requirements.mdx

View check run for this annotation

Mintlify / Mintlify Validation (hanabiaiinc) - vale-spellcheck

developer-guide/self-hosting/requirements.mdx#L99

Did you really mean 'GPUs'?

Check warning on line 99 in developer-guide/self-hosting/requirements.mdx

View check run for this annotation

Mintlify / Mintlify Validation (hanabiaiinc) - vale-spellcheck

developer-guide/self-hosting/requirements.mdx#L99

Did you really mean 'GPUs'?

Check warning on line 99 in developer-guide/self-hosting/requirements.mdx

View check run for this annotation

Mintlify / Mintlify Validation (hanabiaiinc) - vale-spellcheck

developer-guide/self-hosting/requirements.mdx#L99

Did you really mean 'vocoder'?

Check warning on line 99 in developer-guide/self-hosting/requirements.mdx

View check run for this annotation

Mintlify / Mintlify Validation (hanabiaiinc) - vale-spellcheck

developer-guide/self-hosting/requirements.mdx#L99

Did you really mean 'NVLink'?
| NVIDIA driver | Must support CUDA 13.x and your card's compute capability. This applies to the Kubernetes forms too: a driver capped at CUDA 12 reports a healthy GPU and then fails the workloads. |
| Docker | Docker Engine 24 or newer. |
| Container toolkit | NVIDIA Container Toolkit installed and the `nvidia` runtime registered, so `--gpus all` exposes GPUs. |

Check warning on line 102 in developer-guide/self-hosting/requirements.mdx

View check run for this annotation

Mintlify / Mintlify Validation (hanabiaiinc) - vale-spellcheck

developer-guide/self-hosting/requirements.mdx#L102

Did you really mean 'GPUs'?
| RAM | 128 GiB minimum, 192 GiB recommended. The container runs the whole stack in one process tree. |
| CPU | 32 vCPU minimum, 48 to 64 recommended. |
| Disk | 100 GB free if the host pulls the image itself; 120 GB if you move it as a file, since `docker load` needs the ~30 GB archive and the ~80 GB unpacked image at once. Plus the compile and engine caches. |
| Network | Offline form: none at runtime. Online form: outbound HTTPS to the Fish Audio authorization and billing endpoint, on every request. Ask your account team for the hostname to allowlist. |

Check warning on line 106 in developer-guide/self-hosting/requirements.mdx

View check run for this annotation

Mintlify / Mintlify Validation (hanabiaiinc) - vale-spellcheck

developer-guide/self-hosting/requirements.mdx#L106

Did you really mean 'hostname'?

Check warning on line 106 in developer-guide/self-hosting/requirements.mdx

View check run for this annotation

Mintlify / Mintlify Validation (hanabiaiinc) - vale-spellcheck

developer-guide/self-hosting/requirements.mdx#L106

Did you really mean 'allowlist'?
| License bundle | The certificate and key pair from **Developer → Self Host**, in a host directory readable by the container's non-root user. |

## Next step

Expand Down
Loading