Disaster recovery runbook

The single source of truth for "how do I get the platform back" — covering single-node loss, full-cluster loss, and credential rotation. Designed so that with this repo + the off-cluster artifacts listed below + ~30 minutes of manual control-plane work, prod can be reconstructed to a state indistinguishable from the day before the incident.

RPO target: 24 h (daily snapshots). RTO target: 4 h (mostly slack for manual Hetzner / Cloudflare clicks; the automated portion is < 15 minutes in CI).

Off-cluster artifacts you must keep safe

The repo + these items are the entire seed for a rebuild. Lose all of these simultaneously and you cannot recover.

Artifact	Where it lives	Recovery if lost
SOPS Age private keys (one per env)	Secure vault + offline backup	Re-encrypt all `*.enc.yaml` (below)
OpenBao unseal key + root token	`openbao-unseal` Secret (Velero-backed) + operator vault	Restore the `openbao-unseal` Secret from the most recent Velero backup; the paired raft snapshot lives on the `vault-snapshots` PVC and in the R2 `openbao-snapshots/` mirror, and the `vault-config` Job restores it automatically (openbao.md scenarios 2-3); only if every copy is gone, re-initialize OpenBao and re-seed KV — existing encrypted data is then unrecoverable
Cloudflare R2 access keys	Secure vault	Mint new in Cloudflare; SOPS-update
Hetzner Cloud API token	Secure vault	Mint new in Hetzner Cloud console
Cloudflare API token	Secure vault	Mint new in Cloudflare dashboard

Recommendation: store these in a shared vault accessible by at least one additional trusted operator, plus an offline copy in a second physical location. For the SOPS Age keys, a hardware-backed pair (two YubiKeys via age-plugin-yubikey) is the strongest configuration; see crypto-custody.md for the full design and per-artifact threat model.

CI deploy credentials — the KUBE_CONFIG and TALOS_CONFIG secrets in the GitHub prod environment — are deliberately not in the table above. They are derived from the cluster (regenerated by ksail cluster create), so losing them costs nothing permanent. But they go stale on every cluster rebuild (new Talos and Kubernetes PKI, so the stored credentials no longer authenticate) and must be refreshed, or the prod deploy pipeline cannot connect. See Scenario 9 below. Replacing a single control-plane node does not make them stale: the API address is a floating IP that outlives the node.

WireGuard server key — the WG_SERVER_PRIVATE_KEY secret in the prod environment feeds the control-plane wg0 listener (talos/control-planes/wireguard.yaml) via the deploy-prod composite and the DR rebuild. Unlike the derived credentials above it is not regenerated by ksail cluster create — it survives rebuilds unchanged, and losing it means generating a new pair (wg genkey | tee priv | wg pubkey) and updating the unifi tenant's cluster_wg_peer_public_key to the new public half.

Manual production lifecycle prerequisite: use scripts/run-ksail-prod-with-pull-auth.sh, never a bare ksail --config ksail.prod.yaml create/update/push/reconcile command. The wrapper injects the authoritative SOPS pull credential and its rotation marker. Export HCLOUD_TOKEN, WG_SERVER_PRIVATE_KEY, and a usable SOPS Age key for create/update; workload reconciliation also requires the SOPS key. Install yq v4 before invoking the wrapper or credential bridge; both fail fast before decryption or mutation when the compatible YAML query tool is unavailable. Publication additionally needs GHCR_TOKEN and GITHUB_ACTOR.

Scenario 1 — Single node loss

Expected behaviour: PDBs keep every multi-replica workload serving traffic. Re-scale workers or re-run ksail cluster update to replace the lost node.

# Inspect state
kubectl get nodes
kubectl get pods -A --field-selector=status.phase!=Running
kubectl get pdb -A    # all should show ALLOWED-DISRUPTIONS=1

# Replace the failed node (re-runs Hetzner provisioning for missing members)
./scripts/run-ksail-prod-with-pull-auth.sh cluster update

If any workload is stuck in Pending because all replicas were on the dead node and the PDB is blocking eviction on the new one, force a rollout:

kubectl -n <ns> rollout restart deployment/<name>

Scenario 2 — Planned rolling Talos / Kubernetes upgrade

Talos OS and Kubernetes upgrades are driven by the version pins, not by the ISO. Bump spec.cluster.talos.version (Renovate bumps this pin in ksail.prod.yaml; KSail derives the matching machine.install.image installer from it plus spec.cluster.talos.extensions) and/or spec.cluster.kubernetesVersion, then re-run ksail cluster update. KSail performs an in-place rolling upgrade of static nodes — one node at a time, workers first, rebooting each node into the new installer image (Kubernetes upgrades roll the static control-plane pods and kubelets). Existing autoscaler nodes are recycled on a version change, rather than upgraded in place: KSail refreshes the snapshot and worker config, then cordons and drains them one at a time using the eviction API, honoring PDBs, and deletes their provider servers. The autoscaler replaces still-needed capacity from the new snapshot. A no-op update leaves autoscaler nodes untouched; a blocked drain fails the update rather than forcing eviction. See Talos version upgrades. PDBs and maxUnavailable: 0 protect workload availability during the rollout.

The Hetzner iso field selects temporary snapshot-builder boot media, not the installed Talos version. Keep it set to an available Talos ISO so snapshot creation can run. Changing it alone does not upgrade existing nodes; autoscaler nodes boot from the managed snapshot, and KSail derives the installed image from the Talos version and extension pins.

# Pre-flight: confirm every multi-replica workload has a PDB
kubectl get pdb -A

# Pre-flight: confirm RollingUpdate strategy uses maxUnavailable: 0
kubectl get deploy -A -o jsonpath='{range .items[*]}{.metadata.namespace}/{.metadata.name}\t{.spec.strategy.rollingUpdate.maxUnavailable}{"\n"}{end}'

# Apply the upgrade (rolling static nodes; recycle autoscaler nodes on a version change)
./scripts/run-ksail-prod-with-pull-auth.sh cluster update

If anything reports maxUnavailable other than 0, that workload was either added without an HA configuration or has a chart limitation — fix before upgrading.

If a rolling upgrade is interrupted, re-run the production cluster update command above. The pinned KSail release resumes upgrades of lagging static nodes; a mixed version alone does not indicate that the rollout completed. Check both static-node convergence and autoscaler replacement before declaring recovery. If a static node is genuinely stuck and cannot recover through the normal update, upgrade it directly, one at a time, preserving etcd quorum:
# Read the installer image KSail derived (from spec.cluster.talos.version +
# .extensions) off a healthy, already-upgraded node, then reuse it:
IMAGE=$(talosctl --nodes <healthy-ip> get machineconfig -o jsonpath='{.spec.machine.install.image}')
talosctl --nodes <node-ip> upgrade --image "$IMAGE"

Scenario 3 — etcd corruption / control-plane loss

With Omni retired, there is no managed etcd snapshot. Recovery path is full cluster rebuild (Scenario 4) followed by Velero + CNPG restores. This is an accepted trade-off documented in the migration decision: workload state lives in R2-backed Velero and CNPG backups; the control plane is a cattle resource that ksail can re-provision in < 15 min.

Scenario 4 — Full cluster rebuild from zero

The "everything is gone" path. ~10 min of Hetzner provisioning + ~15 min of Flux reconciliation.

One-button path: run the DR - Rebuild Prod workflow (.github/workflows/dr-rebuild.yaml, workflow_dispatch, confirmation phrase REBUILD-PROD). It executes every step below from the CI runner — cluster create, Flux convergence, the Velero resource restore, and the OpenBao raft-snapshot recovery (openbao.md scenario 3) — and needs none of the (stale-after-rebuild) KUBE_CONFIG/TALOS_CONFIG secrets, because ksail cluster create writes fresh configs on the runner. The manual procedure below is the fallback when GitHub Actions itself is unavailable.

# 1. Set credentials locally
export HCLOUD_TOKEN=<hetzner-cloud-api-token>
export WG_SERVER_PRIVATE_KEY=<wireguard-server-private-key>
export GHCR_TOKEN=<ghcr-pat-with-packages-read>
export GITHUB_ACTOR=devantler
export SOPS_AGE_KEY_FILE=~/.config/sops/age/keys.txt  # points at the env's Age key
talosctl version --client  # must be installed; use the prod-pinned v1.13.7

# 2. Prove the Git/SOPS pull credential can read every private package before
#    creating infrastructure or publishing a mutable latest tag
./scripts/refresh-flux-ghcr-auth.sh --check-only

# 3. Boot a fresh cluster (ksail handles Talos boot, CCM, CSI, kubeconfig)
./scripts/run-ksail-prod-with-pull-auth.sh cluster create

# 4. Bootstrap Flux from the most recently evidenced latest artifact. The
#    Actions-unavailable fallback cannot mint the workflow OIDC evidence, so
#    it deliberately leaves latest untouched instead of publishing weaker
#    bytes. Use the one-button workflow for a new publication transaction.
./scripts/refresh-flux-ghcr-auth.sh --allow-incomplete-fanout
./scripts/guard-cilium-homogeneous-device-rollout.sh --before-publish
export DOCKER_CONFIG="$(mktemp -d)"
trap 'rm -rf "${DOCKER_CONFIG}"' EXIT
printf '%s' "${GHCR_TOKEN}" |
  docker login ghcr.io --username "${GITHUB_ACTOR}" --password-stdin
PLATFORM_MANIFEST_DIGEST="$(docker buildx imagetools inspect 'ghcr.io/devantler-tech/platform/manifests:latest' --format '{{.Manifest.Digest}}')"
[[ "${PLATFORM_MANIFEST_DIGEST}" =~ ^sha256:[0-9a-f]{64}$ ]] || {
  printf 'Invalid platform manifest digest: %s\n' "${PLATFORM_MANIFEST_DIGEST:-<empty>}" >&2
  exit 1
}
./scripts/refresh-flux-ghcr-auth.sh --check-only
./scripts/run-ksail-prod-with-pull-auth.sh workload reconcile
./scripts/wait-for-platform-flux-revision.sh "${PLATFORM_MANIFEST_DIGEST}"

# 5. Wait for Flux to settle
for k in bootstrap infrastructure-controllers infrastructure apps; do
  kubectl --context admin@prod -n flux-system wait "kustomization/${k}" \
    --for=condition=Ready --timeout=20m
done
./scripts/refresh-flux-ghcr-auth.sh  # prove completed fan-out + every stale node
CILIUM_ROLLOUT_REVISION_READY=true \
  ./scripts/guard-cilium-homogeneous-device-rollout.sh --after-deploy

# 6. ONLY if the OpenBao raft-snapshot recovery was impossible (no snapshot
#     in R2 — the vault came up fresh): re-feed the user-fed secrets that
#     SOPS deliberately does not seed (see the push-secret-seed-* files in
#     k8s/bases/infrastructure/vault-seed/). Until then,
#     cert-manager DNS01, external-dns, and fleetdm stay pending:
kubectl -n openbao exec openbao-0 -- \
  bao kv put secret/infrastructure/dns/cloudflare api_token=<cloudflare-token>
kubectl -n openbao exec openbao-0 -- \
  bao kv put secret/apps/fleetdm/license license-key=<fleet-license-jwt>

# 5. DNS — normally NO manual step: external-dns (hetzner overlay,
#    policy: sync, gateway-httproute source) repoints the Cloudflare
#    records at the new load balancer automatically once the HTTPRoutes
#    are Ready and its Cloudflare token has re-synced from the vault.
#    Verify, and only intervene if external-dns itself is broken:
kubectl -n external-dns logs deploy/external-dns | tail -20
kubectl -n kube-system get svc cilium-gateway-platform \
  -o jsonpath='{.status.loadBalancer.ingress[0].ip}'
# Fallback only: update A/AAAA records for ${domain} at your DNS provider.

# 6. Restore Velero backups (apps + PVCs)
kubectl -n velero create -f - <<EOF
apiVersion: velero.io/v1
kind: Restore
metadata:
  name: rebuild-$(date +%s)
  namespace: velero
spec:
  backupName: <pick-latest-from-velero-backup-get>
  includedNamespaces:
    - "*"
  excludedNamespaces:
    - kube-system
    - velero
EOF

# 7. (If any CNPG Cluster exists) restore from R2
kubectl cnpg restore <new-cluster-name> \
  --backup <backup-name> \
  --target-time '<RFC3339-timestamp-or-omit-for-latest>'

If this is the first time restoring after losing the SOPS keys, replace step 3 with the rotation flow in Scenario 6 first.

A rebuild also replaces the Gateway and its load balancer, so the Gateway receiver binding is stale. Prepare a replacement before relying on public routes or on a deploy: see Gateway receiver binding.

After a rebuild the cluster has a new Talos and Kubernetes PKI, and possibly a new API address (see Scenario 9), so the prod environment's KUBE_CONFIG / TALOS_CONFIG secrets are now stale. Refresh them per Scenario 9 before relying on the automated deploy pipeline, otherwise ksail cluster update in CI cannot reach the cluster.

Scenario 5 — Velero / CNPG restore (single namespace or app)

Quick path for "I deleted the wrong PVC" or "this Postgres database needs to roll back to last night".

# Find the relevant backup
kubectl -n velero get backups
velero backup get   # if velero CLI installed locally

# Namespace restore
kubectl -n velero create -f - <<EOF
apiVersion: velero.io/v1
kind: Restore
metadata:
  name: ns-restore-$(date +%s)
  namespace: velero
spec:
  backupName: daily-full-<date>
  includedNamespaces: ["<your-ns>"]
EOF

# CNPG point-in-time recovery (PITR is "free" once WAL archiving is on)
kubectl cnpg restore <new-cluster-name> \
  --source-cluster <old-cluster> \
  --target-time '2026-04-17T22:00:00Z'

Cross-provider / cross-distribution restore (StorageClass mapping). The backup data in R2 is storage-agnostic (Kopia repository), so there is no Longhorn/Hetzner dependency at the destination — this is what makes a GitOps migration to another distribution/provider possible. Two things to handle on the target cluster:
StorageClass names. Velero recreates each PVC with the same StorageClass name it had on the source (longhorn, hcloud). If the destination's classes are named differently (e.g. gp3, standard, local-path), those PVCs stay Pending. Map them before restoring with a velero.io/change-storage-class ConfigMap in the velero namespace:
apiVersion: v1
kind: ConfigMap
metadata:
  name: change-storage-class-config
  namespace: velero
  labels:
    velero.io/change-storage-class: RestoreItemAction
data:
  longhorn: <target-default-storageclass>
  hcloud: <target-default-storageclass>
Restore-side capabilities. The destination Velero needs the node-agent (DaemonSet) to rehydrate Kopia data from FSB and Longhorn CSI data-mover backups. For the data-mover backups it also needs features: EnableCSI; the target does not need Longhorn or any CSI-snapshot support of its own (the data is replayed into a fresh PVC by Kopia). OpenBao's hcloud-backed raft-snapshot PVC is outside that path: vault-snapshot-init restores its independently mirrored snapshot from R2 as documented in the OpenBao recovery scenario.

Scenario 6 — SOPS Age key rotation

# 1. Generate a new key
age-keygen -o new-key.txt
NEW_PUB=$(grep '^# public key' new-key.txt | cut -d: -f2 | tr -d ' ')

# 2. Add the new pub key as a recipient *before* removing the old one
#    (gives you a window where both keys can decrypt).
yq -i ".creation_rules[].age += \",\n$NEW_PUB\"" .sops.yaml

# 3. Re-encrypt every SOPS file with the new recipient list
find . -name '*.enc.yaml' -print0 | xargs -0 -n1 sops updatekeys --yes

# 4. Commit + merge. Verify Flux still decrypts (no errors in
#    flux-system pods).

# 5. Rotate the new key into your secret store, distribute to operators.

# 6. Once everyone is on the new key, drop the old one from .sops.yaml
#    and re-run sops updatekeys --yes one more time.

# 7. Securely destroy old-key.txt copies.

Scenario 7 — R2 / Cloudflare credential rotation

Choose the credential by bucket before editing it:

platform-backups uses k8s/clusters/prod/bootstrap/secret.enc.yaml and the seed-r2-credentials PushSecret.
wedding-db-backups uses k8s/clusters/prod/bootstrap/secret-wedding-db-backup-r2.enc.yaml and the seed-wedding-db-backup-r2 PushSecret.

They are independent identities. Wedding archives through its dedicated token; Umami, Coroot, and Velero use the shared platform token. Wedding's access to the shared token is retired (#3253): a dedicated-only isolated restore and the Verify Wedding Backup Denial access-denial proof both passed first, and the wedding-app namespace now holds only the dedicated credential. Rotating the dedicated Wedding token does not affect the shared consumers, and a shared-token rotation does not affect Wedding; verify the active shared consumers before revocation.

set -euo pipefail

# 1. Mint a new Object Read & Write R2 token scoped only to the selected bucket.
#    Keep the old token active until the final verification succeeds.

# 2. Select the bucket. This resolves only public paths and key names.
bucket=wedding-db-backups
case "$bucket" in
  platform-backups)
    secret_file=k8s/clusters/prod/bootstrap/secret.enc.yaml
    access_path='["stringData"]["r2_access_key_id"]'
    secret_path='["stringData"]["r2_secret_access_key"]'
    ;;
  wedding-db-backups)
    secret_file=k8s/clusters/prod/bootstrap/secret-wedding-db-backup-r2.enc.yaml
    access_path='["stringData"]["access_key_id"]'
    secret_path='["stringData"]["secret_access_key"]'
    ;;
  *) echo 'Unknown R2 bucket' >&2; exit 1 ;;
esac

# Read both values without terminal echo or command-line arguments. SOPS takes
# each JSON string on stdin and edits only the selected encrypted leaf.
IFS= read -rsp 'New R2 access key ID: ' new_access_key_id
printf '\n'
IFS= read -rsp 'New R2 secret access key: ' new_secret_access_key
printf '\n'
if [[ ! "$new_access_key_id" =~ ^[0-9a-f]{32}$ ]]; then echo 'Invalid R2 access key ID' >&2; exit 1; fi
if [[ ! "$new_secret_access_key" =~ ^[0-9a-f]{64}$ ]]; then echo 'Invalid R2 secret access key' >&2; exit 1; fi
test "${#new_secret_access_key}" -eq 64
printf '"%s"' "$new_access_key_id" |
  sops set --value-stdin "$secret_file" "$access_path"
printf '"%s"' "$new_secret_access_key" |
  sops set --value-stdin "$secret_file" "$secret_path"
unset new_access_key_id new_secret_access_key

# 3. For wedding-db-backups, calculate the two public source receipts. Replace
#    the corresponding constants in .github/actions/deploy-prod/action.yml.
shasum -a 256 k8s/clusters/prod/bootstrap/secret-wedding-db-backup-r2.enc.yaml
shasum -a 256 k8s/clusters/prod/bootstrap/kustomization.yaml

# 4. Open a PR and merge it. Flux reconciles the selected bootstrap Secret,
#    its PushSecret writes the matching OpenBao path, and the ExternalSecret
#    refreshes the namespace-local credential on its next reconciliation.

# 5. For wedding-db-backups, bind the non-printing projection verifier to the
#    current main commit, wait for that exact run, and require its three success
#    fields before continuing.
expected_sha="$(gh api repos/devantler-tech/platform/commits/main --jq .sha)"
run_url="$(gh workflow run cd.yaml --repo devantler-tech/platform --ref main \
  -f verify-wedding-backup-staging=true)"
run_id="${run_url##*/}"
case "${run_id}" in ''|*[!0-9]*) echo 'Could not identify the verifier run' >&2; exit 1;; esac
gh run watch "$run_id" --repo devantler-tech/platform --exit-status
gh run view "$run_id" --repo devantler-tech/platform \
  --json attempt,conclusion,event,headBranch,headSha |
  jq -e --arg expected "$expected_sha" \
    '.attempt == 1 and .conclusion == "success" and .event == "workflow_dispatch" and .headBranch == "main" and .headSha == $expected'
gh run view "$run_id" --repo devantler-tech/platform --log |
  grep -F '"verified":true' |
  grep -F '"projectionEqual":true' |
  grep -F '"liveSourceStable":true'

# 6. For wedding-db-backups, dispatch Verify Wedding Backup Cutover on main
#    with confirm=verify-wedding-backup-cutover. Bind its run to the main SHA
#    as above and require backupCompleted=true and clusterStable=true.
#    Record the backup ID and endWal receipt, then inspect the dedicated
#    catalogue for that completed backup and advancing WAL. A completed CNPG
#    Backup alone is not proof that the bucket contains a usable catalogue.
#
#    Wedding's shared access is retired (#3253), so there is no shared
#    catalogue to mirror or catch up. Do not revoke the shared credential as
#    part of a Wedding rotation: the other platform consumers still use it.

# 7. For wedding-db-backups, confirm in Cloudflare that the new token is
#    scoped to Object Read & Write on wedding-db-backups only. Verify Wedding
#    Backup Denial must then succeed on main with
#    confirm=verify-wedding-backup-denial. It runs beside the existing shared
#    platform credential using a temporary dedicated-key copy, and removes
#    that copy with a UID precondition. Until the workflow succeeds, the new
#    token's refusal by platform-backups is unproven. Never project the shared
#    credential back into the tenant namespace to run the proof.

# For a platform-backups credential rotation, observe a new successful Velero
# backup plus new backups and WAL archives from Umami and Coroot before
# revocation. Wedding no longer uses that credential.
kubectl -n velero get backups.velero.io -w

# 8. Revoke the old token only after the checks for its active bucket succeed.
#    A queued workflow, an old backup, or a healthy
#    unrelated consumer is not sufficient evidence.

The Cloudflare API token (DNS01 + external-dns) is user-fed, not in SOPS — rotate it with a single vault write; the consuming ExternalSecrets re-sync within their 1h refresh interval:

kubectl -n openbao exec openbao-0 -- \
  bao kv put secret/infrastructure/dns/cloudflare api_token=<new-token>

Encryption-at-rest verification

Run after any node replacement to confirm secrets are still ciphertext on disk.

# Pull a fresh etcd snapshot via talosctl
talosctl --nodes <cp-node> etcd snapshot /tmp/etcd.snapshot

# Inspect a Secret -- must NOT be plain text
etcdctl --endpoints unix:///tmp/etcd.snapshot \
  get --prefix /registry/secrets/ | head -c 200
# Expect bytes that look like cipher (binary garbage). If you see
# Kubernetes Secret YAML, the EncryptionConfiguration was lost.

This check is deliberately not part of the CI restore drill — Talos verifies the encryption key at install time, so a CI assertion would add complexity for a structurally-enforced property (see restore-drill.md for the full rationale).

Local clusters

Local clusters are ephemeral and reconstructed from this repo on every ksail cluster create. There is nothing meaningful to back up — the restore procedure for local is:

ksail cluster delete
ksail cluster create
ksail workload push && ksail workload reconcile

CI exercises this on every PR (.github/workflows/ci.yaml), and also exercises a Velero backup → restore against the in-cluster MinIO so the prod code path is regression-tested.

Scenario 8 — Cluster Autoscaler issues

Autoscaler not scaling up

# Check for pending pods
kubectl get pods -A --field-selector=status.phase=Pending

# Inspect autoscaler logs
kubectl -n kube-system logs -l app.kubernetes.io/name=cluster-autoscaler --tail=200

# Check status ConfigMap
kubectl -n kube-system get cm cluster-autoscaler-status -o yaml

Common causes:

Pool maxSize reached — increase max under the relevant pool in ksail.prod.yaml, then run ./scripts/run-ksail-prod-with-pull-auth.sh cluster update
HCLOUD_TOKEN expired — rotate in SOPS secrets and GitHub environment secrets

Orphaned autoscaler nodes after cluster delete

ksail cluster delete may not remove servers created by the Cluster Autoscaler. Clean up manually:

hcloud server list --selector hcloud/node-group
# Delete each orphaned server
hcloud server delete <server-id>

Stranded autoscaler nodes on a LIVE cluster (removed pool)

Removing a pool from ksail.prod.yaml does not delete servers already running in it. The autoscaler only ever considers nodes that belong to one of its configured node groups, so a node whose pool no longer exists becomes invisible to scale-down and survives indefinitely — however idle the cluster gets. It keeps consuming maxNodesTotal, which starves real bursts, and it is never rolled, so it drifts behind on Kubernetes and Talos.

Detect it by comparing live node names against the configured groups:

# pools the autoscaler actually manages — read BOTH command and args: the chart
# currently passes every flag in `command`, so an `args`-only query returns an
# empty stream and the grep matches nothing.
kubectl -n kube-system get deploy cluster-autoscaler-hetzner-cluster-autoscaler \
  -o jsonpath='{range .spec.template.spec.containers[0].command[*]}{@}{"\n"}{end}{range .spec.template.spec.containers[0].args[*]}{@}{"\n"}{end}' | grep '^--nodes='
# every autoscaler-provisioned node currently registered
kubectl get nodes -o name | grep '/autoscale-'

An empty result from the first command means the query is wrong, not that no pools are configured — check the container spec directly before concluding anything. It should print one --nodes= line per configured pool.

Any autoscale-<type>-* node whose <type> has no matching --nodes= group is stranded.

You should not have to find one by hand. The observability/stranded-autoscaler-node-alerter CronJob checks every 15 minutes and raises a StrandedAutoscalerNode alert in Slack, naming the node and its pool, for as long as such a node exists. It compares node names with its own copy of the pool list, and CI fails when that copy differs from the pools in ksail.prod.yaml, so adding or removing a pool means updating DECLARED_POOLS in k8s/providers/hetzner/infrastructure/coroot/cron-job-stranded-autoscaler-node-alerter.yaml in the same change.

Pre-flight — check these before draining anything

hcloud server delete neither cordons nor drains the node, and it does not wait for Longhorn to evict replicas. Deleting a server directly destroys any replica still on it, so the drain is what makes removal safe, and these checks are what make the drain safe.

# 1. Every storage node must actually be able to ATTACH volumes. A node whose
#    iSCSI database has an unparseable record accepts replicas but cannot start
#    an engine, and reports Ready/Schedulable while doing so (#3180). A node
#    with zero attachments while others have many is the tell.
kubectl -n longhorn-system get volumes.longhorn.io -o json |
  jq -r '[.items[]|select(.status.state=="attached")]|group_by(.status.currentNodeID)[]|"\(.[0].status.currentNodeID) attached=\(length)"'

# 2. Replica scheduling must have somewhere to go. With hard anti-affinity
#    (replica-soft-anti-affinity=false) and only three storage nodes, one node
#    below storage-minimal-available-percentage means degraded volumes can never
#    rebuild — they sit at ReplicaSchedulingFailure indefinitely.
#    Check EVERY disk; an unknown capacity or scheduling condition needs
#    investigation before draining. One healthy disk does not clear another.
kubectl -n longhorn-system get nodes.longhorn.io -o json |
  jq -r '
    .items[] | .metadata.name as $n |
    (.status.diskStatus // {} | to_entries |
      if length == 0 then [{key:"UNKNOWN",value:{}}] else . end)[] |
    .key as $disk | .value as $s |
    (if ($s.storageMaximum | type) == "number" and $s.storageMaximum > 0
        and ($s.storageAvailable | type) == "number" and $s.storageAvailable >= 0
      then "\(($s.storageAvailable * 100 / $s.storageMaximum) | floor)%"
      else "UNKNOWN" end) as $available |
    ([$s.conditions[]? | select(.type == "Schedulable") | .status] |
      if length == 0 then "UNKNOWN" else join(",") end) as $schedulable |
    "\($n) disk=\($disk) avail=\($available) Schedulable=\($schedulable)"'

# 3. Reclaim orphaned replica directories first — they are the usual reason a
#    node drifts under the threshold. Delete the listed orphans to free the
#    space (see #3180 for making this automatic).
kubectl -n longhorn-system get orphans.longhorn.io

# 4. Nothing already degraded.
kubectl -n longhorn-system get volumes.longhorn.io -o json |
  jq -r '.items[]|select(.status.robustness!="healthy" or .status.state!="attached")|"\(.metadata.name) \(.status.state)/\(.status.robustness)"'

Removal sequence

kubectl drain <node> --ignore-daemonsets --delete-emptydir-data --timeout=20m

The drain normally stalls at the end on the node's Longhorn instance-manager, which carries a PDB with zero allowed disruptions. That is expected: once the node is storage-idle the instance-manager is idle too, and Longhorn usually removes it on its own. Confirm idleness before forcing anything — both must return 0:

kubectl -n longhorn-system get engines.longhorn.io -o json | jq -r --arg n <node> '[.items[]|select(.spec.nodeID==$n)]|length'
kubectl -n longhorn-system get replicas.longhorn.io -o json | jq -r --arg n <node> '[.items[]|select(.spec.nodeID==$n)]|length'

Only then delete the server, and finally the now-orphaned Node object:

hcloud server delete <server-id>
kubectl delete node <node>

Do one node at a time and let volumes return to healthy in between. Deleting the server is also what frees a slot under maxNodesTotal, so the autoscaler cannot replace capacity until that step completes.

restrict-storage-to-baseline-workers and drain-autoscale-node-storage do cover these nodes, because they key on the node name rather than the ksail.io/autoscaled label — which a node predating ksail#5113 does not carry.

Whenever you remove a pool, check for its running nodes in the same change.

Autoscaler node not joining cluster

# Check if the server was created in Hetzner
hcloud server list

# If the server exists but node doesn't appear in kubectl:
# The worker machine config may be invalid or stale.
# Re-run cluster update to regenerate worker config and re-apply:
./scripts/run-ksail-prod-with-pull-auth.sh cluster update

Scenario 9 — Refresh CI deploy credentials after a cluster rebuild

The DR - Rebuild Prod workflow refreshes both secrets automatically at the end of a rebuild if a DR_GH_ADMIN_TOKEN secret (a fine-grained PAT with environment-secrets write on this repo) is configured; without it the workflow prints a warning and the manual procedure below applies.

The prod deploy pipeline (the merge-queue deploy-prod job in ci.yaml, and the manual .github/workflows/cd.yaml) authenticates to the cluster with two GitHub prod environment secrets:

Secret	Restored to	Used by
`KUBE_CONFIG`	`~/.kube/config`	`ksail cluster update` drift detection (kube API)
`TALOS_CONFIG`	`~/.talos/config`	`ksail cluster update` machine-config / secret sync

Endpoint selection during ordinary deployments

The shared deploy-prod action runs scripts/use-prod-stable-api-endpoint.sh immediately after restoring KUBE_CONFIG, before its first Kubernetes API call. The script resolves the production floating IP through the Hetzner API, requires exactly one match with KSail's ownership and cluster labels, and updates only the restored admin@prod cluster's server address. Missing, ambiguous, or foreign floating-IP state stops deployment before a node can be rolled.

A kubeconfig whose server still names a replaced control-plane node therefore does not need an environment-secret refresh solely to change that address. The deployment selects the stable address on each run, preserves the kubeconfig's credentials and CA, and checks /readyz through normal TLS validation. Check the Select stable prod API endpoint and Verify prod cluster is reachable steps when diagnosing endpoint failures.

Workflow logs are public, so on a runner the same script also asks it to redact the endpoint address and every address of every server the Hetzner token can list at that moment, before any later tool can print one. It stops the job when it cannot list those servers or take addresses from the answer. Such an address therefore shows as *** in a deploy log; read it from the Hetzner console when you need the value. A server created later in the same job is not covered.

TALOS_CONFIG retains direct node endpoints. Do not point the Talos API at the Kubernetes floating IP: recovery access must remain available when etcd or the Kubernetes API is unhealthy.

This automatic selection does not validate a control-plane failure drill by itself. A controlled recreate must separately demonstrate continued Kubernetes access without a secret refresh, with health and recovery checks recorded for that drill.

Start a drill only after scripts/cp-recreate-drill-preflight.sh reports ready=true. It is read-only and checks the target (an existing, Ready control-plane node), quorum (three Ready control-plane nodes and three voting etcd members with no learners) and the newest velero-daily-full backup (Completed and less than 26 hours old). It prints every failing guard, and it reports whether the target currently holds the floating IP:

kubectl get nodes -o json > nodes.json
kubectl get backups.velero.io -n velero -o json > backups.json
talosctl --nodes <control-plane-ip> etcd members > etcd-members.txt
# Count voters and learners from the LEARNER column; never type the healthy values in.
read -r etcd_voters etcd_learners < <(awk 'NR > 1 && $NF == "false" { v++ }
  NR > 1 && $NF == "true" { l++ } END { printf "%d %d\n", v, l }' etcd-members.txt)
# The floating IP's server, matched to its node through the node's hcloud providerID.
vip_server="$(hcloud floating-ip describe prod-floating-ip -o json | jq -r '.server // empty')"
vip_holder="$(jq -r --arg id "hcloud://${vip_server}" \
  '.items[] | select(.spec.providerID == $id) | .metadata.name' nodes.json)"
scripts/cp-recreate-drill-preflight.sh --target <node> \
  --nodes nodes.json --backups backups.json \
  --etcd-members "${etcd_voters}" --etcd-learners "${etcd_learners}" \
  --vip-holder "${vip_holder}"

An empty vip_holder (the floating IP is unassigned, or no node matches) reports target_is_vip_holder=unknown; resolve that before a drill meant to test failover.

Credential refresh after a full rebuild

After a full rebuild (Scenario 4) the Talos and Kubernetes PKI change, so both secrets are stale even when the API address is unchanged. Whether the address changes depends on the floating IP: ksail cluster create reuses the cluster's KSail-owned floating IP (prod-floating-ip) when it still exists in the Hetzner project, and creates a new one, with a new address, when it does not. ksail cluster delete releases that floating IP, so a rebuild after a delete always gets a new address. The deploy selects the current floating IP on every run either way, so the refresh below is needed for the new credentials, not for the address. Symptom in CI: the 🩺 Verify prod cluster is reachable preflight fails, or ksail cluster update reports a bogus "N configuration changes" plan (every component shows Default/Disabled/None) and then fails with connection refused / x509: certificate signed by unknown authority "talos".

# Run from a machine that can reach the rebuilt cluster (e.g. the homelab),
# after `ksail cluster create` has written fresh local configs.

# 1. Locate the fresh configs ksail produced during the rebuild:
#    kubeconfig  — ~/.kube/config   (context: admin@prod)
#    talosconfig — ~/.talos/config
#    Sanity-check they point at the new cluster:
kubectl --context admin@prod get --raw='/readyz'      # -> ok
talosctl --talosconfig ~/.talos/config version        # -> talks to the nodes

# 2. Push both into the GitHub `prod` environment secrets used by CI/CD.
#    (Drop `--env prod` if these are configured as repo-level secrets.)
gh secret set KUBE_CONFIG  --env prod --repo devantler-tech/platform < ~/.kube/config
gh secret set TALOS_CONFIG --env prod --repo devantler-tech/platform < ~/.talos/config

# 3. Re-trigger the deploy (push/re-tag, or re-queue the PR). The reachability
#    preflight should pass and `ksail cluster update` should report no spurious
#    component changes.

Upstream context: ksail currently warns (rather than fails) when the cluster is unreachable and then proposes a full reinstall from a default baseline — tracked in devantler-tech/ksail#4868 and #4869. The preflight step in the deploy workflows is the local guard against that behaviour until those land.

Scenario 10 — production GHCR pull credential rotation or denial

The authoritative Flux, tenant, Kyverno, and Talos-host pull credential is the SOPS-encrypted stringData.ghcr_dockerconfigjson value in k8s/bases/bootstrap/secret.enc.yaml. It is deliberately separate from the GitHub prod environment's GHCR_TOKEN, which only pushes/signs artifacts. Production KSail lifecycle commands run through scripts/run-ksail-prod-with-pull-auth.sh, which derives username/token from Git/SOPS and supplies a non-secret SOPS-ciphertext revision to the Talos machine/autoscaler template.

The production deploy closes the bootstrap loop in this order:

Decrypt only ghcr_dockerconfigjson and perform real OCI manifest reads for all eight private consumers: Platform manifests, both tenant manifest artifacts, the data-product-controller image, both tenant application images, and the KSail plus provider-upjet-unifi packages used by Kyverno verification. During DR, a read-only --check-only pass happens before infrastructure creation.
Before changing mutable latest, list every Kubernetes Node (including NotReady/autoscaled nodes). A changed verified credential revision takes the fenced Talos RegistryAuthConfig path: apply without reboot, then drain and reboot each node, workers before control planes. Containerd loads registry credentials at startup; a successful Talos image-API pull reads machine-config auth and does not prove that the running containerd adopted the new credential. After reboot, remove the exact incoming KSail image from the CRI cache, pull it again for a registry round-trip, then record both proof markers. If that credential proof is current and only the verified image differs, remove and pull a proof copy in Talos containerd's system namespace without cordoning or evicting the Kubernetes CRI image; publish the image marker only after a successful registry pull. A distinct desired revision in the committed Talos configuration refreshes future Cluster Autoscaler templates but never counts as proof.
Patch variables-base; force-sync seed-ghcr into OpenBao; force-sync the tenant/Kyverno ExternalSecrets; verify every materialised ghcr-auth payload matches Git/SOPS; and only then patch flux-system/ksail-registry-credentials. A fresh or partial DR bootstrap may have no ESO CRDs/resources yet; explicit --allow-incomplete-fanout mode stages variables-base and repairs root auth so the first reconcile can create the chain. Normal mode fails closed on any missing fan-out resource.
Push the artifact to a run-unique staging reference. Resolve its digest, cosign-sign it keyless, and attach its CycloneDX SBOM and SLSA provenance before promoting that exact digest to latest. Fulcio mints the certificate from the workflow's OIDC identity (dr-rebuild.yaml@refs/heads/main), and GHCR_TOKEN only pushes the artifact and its evidence. That identity must stay listed in ksail.prod.yaml's matchOIDCIdentity, or a recovery publishes an artifact a verifying cluster refuses. Any failure before promotion leaves the prior latest digest unchanged. Revalidate the promoted artifact with --check-only, and only then explicitly reconcile Flux. DR runs the full bridge again after every Flux Kustomization is Ready, proving that bootstrap mode completed the entire fan-out, and once more after an OpenBao raft restore so a snapshot cannot rematerialise an older credential.
Re-run the bridge after cluster update; it repairs any node left stale by a partial lifecycle operation and re-verifies the root plus downstream fan-out. Nodes carrying proof for both the current credential revision and declared image skip Talos API calls, so an ordinary deploy does not depend on every Talos endpoint.

For a normal rotation, update the encrypted value through a PR and let the merge queue deploy it. If the encrypted file was pushed directly to main, manually dispatch the CD workflow: direct pushes do not run the production deploy.

For an already-denied root source, run the same secret-safe bridge from a clean main checkout with the production Age key and admin@prod kube context, then reconcile the source:

./scripts/refresh-flux-ghcr-auth.sh
flux reconcile source oci flux-system -n flux-system --context admin@prod
flux reconcile kustomization flux-system -n flux-system --context admin@prod

If tenant sources still show DENIED after an older/manual recovery path, force the existing fan-out after variables-base has converged:

stamp=$(date +%s)
kubectl --context admin@prod -n flux-system annotate pushsecret seed-ghcr \
  force-sync="$stamp" --overwrite
for namespace in wedding-app ascoachingogvaner kyverno; do
  kubectl --context admin@prod -n "$namespace" annotate externalsecret ghcr-auth \
    force-sync="$stamp" --overwrite
done
flux get sources oci -A --context admin@prod

Do not replace this bridge with only a root ExternalSecret: Flux needs valid root auth to fetch the artifact that installs/updates External Secrets, while OpenBao receives this rotated value only through a downstream PushSecret. That circular dependency cannot recover a stale root credential by itself.

Scenario 11 — Recover an orphaned GHCR deploy fence

A deploy fails to start with one of:

Another GHCR synchronization transaction holds the synchronization lease

Another transaction already owns the image-verification policy handoff

refresh-flux-ghcr-auth.sh fences its transaction with a Lease and by suspending the infrastructure and flux-system Kustomizations. Those fences are released together at exit. A hard kill releases an arbitrary prefix and leaves the rest held. Protected deploy and heal jobs run --recover-fences: it releases policy fences and then the Lease only when every holder identifies a GitHub Actions run attempt that is confirmed terminal, the Lease heartbeat has stopped, no node fence is held, and every CAS-guarded patch still matches the reported state. Node fences remain a manual or full-bridge recovery because Talos machine-config writes expose no fencing token and need the bridge's revision and scheduling proofs.

Full-bridge cleanup treats a successfully confirmed deleted Node as having no scheduling fence left to release. It confirms absence through the Kubernetes API, including after a failed release patch. A denied or failed read, malformed response, replacement Node, or changed fence owner cannot authorize release. This cleanup result does not prove rollout convergence: losing a target still blocks the root cutover until the bridge's convergence checks pass.

A held policy fence is the more serious of the two: the deploy fails loudly, but the suspended Kustomization silently stops GitOps reconciliation for that layer until it is released.

1. List what is actually held. Read-only; needs no credential and works while a deploy is refusing to start:

./scripts/refresh-flux-ghcr-auth.sh --fences

It prints each held fence, its holder, liveness evidence, and the exact CAS-guarded release command. --fences is an alternative mode and cannot be combined with --check-only, --allow-incomplete-fanout, or either --*-runtime-proof flag; it reports and exits, so accepting the combination would let an automation step pass having skipped the work it was configured to do.

2. Prove the holder is dead — never skip this. A live deploy holds these fences legitimately.

If the Lease was renewed inside its duration, the holder is live. Stop.
The holder identity embeds the GitHub run: …-gh<run-id>.<attempt>-…. Confirm it is terminal:
```
gh run view <run-id> --repo devantler-tech/platform --attempt <attempt> --json status,conclusion
```
Anything other than status: completed means it is still running. Stop. --attempt is not optional: a rerun reuses the run id, so without it gh reports the newest attempt and an orphan left by a finished attempt reads as live — blocking a recovery that is valid.
An identity with no run reference predates that recording, or came from a local run. Establish liveness another way before continuing.

3. Release. For terminal GitHub-run policy fences plus the Lease, run ./scripts/refresh-flux-ghcr-auth.sh --recover-fences; it proves all holders before its first mutation and releases child, parent, then Lease. For a local or legacy holder that the API cannot prove terminal, use the commands from --fences only after establishing the process is dead. Each command is CAS-guarded on the holder and the resource's current state, so it fails safely if anything changed since the report.

A node fence may print NOT releasable and no command. That is a refusal, not a gap to work around by hand: the report only emits a release where it can prove the release is the same one the bridge would perform. It withholds one when the drain phase is anything but claimed (a Talos write may already be in flight, and an absent phase is never read as innocence), when another node under the same bootstrap owner is still quarantined (that quarantine is all-or-nothing), when the journal is release-ready (proving it covers the current credential revision needs the credential, which this mode deliberately never loads), or when the node's scheduling state has drifted from the intent its journal captured. In each case run the bridge — a normal CD dispatch — and let bootstrap recovery adjudicate the node. Never hand-clear a recovery journal: it is the only durable record of that state.

4. Confirm. Re-run --fences; it should report no fence held. Then dispatch CD on main.

Do not raise timeouts or add automatic expiry takeover to work around this — the absence of takeover is the property that makes the Talos write path safe.

Navigation Menu

Repository navigation

runbook.md

runbook.md

Disaster recovery runbook

Off-cluster artifacts you must keep safe

Scenario 1 — Single node loss

Scenario 2 — Planned rolling Talos / Kubernetes upgrade

Scenario 3 — etcd corruption / control-plane loss

Scenario 4 — Full cluster rebuild from zero

Scenario 5 — Velero / CNPG restore (single namespace or app)

Scenario 6 — SOPS Age key rotation

Scenario 7 — R2 / Cloudflare credential rotation

Encryption-at-rest verification

Local clusters

Scenario 8 — Cluster Autoscaler issues

Autoscaler not scaling up

Orphaned autoscaler nodes after cluster delete

Stranded autoscaler nodes on a LIVE cluster (removed pool)

Pre-flight — check these before draining anything

Removal sequence

Autoscaler node not joining cluster

Scenario 9 — Refresh CI deploy credentials after a cluster rebuild

Endpoint selection during ordinary deployments

Credential refresh after a full rebuild

Scenario 10 — production GHCR pull credential rotation or denial

Scenario 11 — Recover an orphaned GHCR deploy fence

Related documents

FilesExpand file tree

runbook.md

Latest commit

History

runbook.md

File metadata and controls

Disaster recovery runbook

Off-cluster artifacts you must keep safe

Scenario 1 — Single node loss

Scenario 2 — Planned rolling Talos / Kubernetes upgrade

Scenario 3 — etcd corruption / control-plane loss

Scenario 4 — Full cluster rebuild from zero

Scenario 5 — Velero / CNPG restore (single namespace or app)

Scenario 6 — SOPS Age key rotation

Scenario 7 — R2 / Cloudflare credential rotation

Encryption-at-rest verification

Local clusters

Scenario 8 — Cluster Autoscaler issues

Autoscaler not scaling up

Orphaned autoscaler nodes after cluster delete

Stranded autoscaler nodes on a LIVE cluster (removed pool)

Pre-flight — check these before draining anything

Removal sequence

Autoscaler node not joining cluster

Scenario 9 — Refresh CI deploy credentials after a cluster rebuild

Endpoint selection during ordinary deployments

Credential refresh after a full rebuild

Scenario 10 — production GHCR pull credential rotation or denial

Scenario 11 — Recover an orphaned GHCR deploy fence

Related documents