Skip to content

node-autoscaling.md

Latest commit

 

History

History
276 lines (230 loc) · 12.9 KB

node-autoscaling.md

File metadata and controls

276 lines (230 loc) · 12.9 KB
Raw

Node Autoscaling

Automatic horizontal and vertical scaling of Hetzner worker nodes via the Kubernetes Cluster Autoscaler with the Hetzner Cloud provider.

KSail natively installs and manages the Cluster Autoscaler when spec.cluster.autoscaler.node.enabled: true is set in the ksail config.

Architecture

KSail (static baseline)
├── 3 control planes (cx33, 4 vCPU / 8 GB, never autoscaled)
└── 3 static workers (cx43, 8 vCPU / 16 GB, guaranteed minimum, Longhorn storage nodes)

Cluster Autoscaler (dynamic workers, managed by KSail)
├── Pool: autoscale-cx43 → 0-4 × CX43 (8 vCPU, 16 GB)
├── Pool: autoscale-cx53 → 0-4 × CX53 (16 vCPU, 32 GB)
├── Pool: autoscale-arc-runners → 0-1 × CX53 (CI runners only: ci-runner label + NoSchedule taint)
├── capacityBuffers: true (CapacityBuffer warm headroom, see overprovisioning/)
├── maxNodesTotal: 9 (total cluster nodes incl. baseline: 6 static + 3 autoscaler — one server slot stays reserved, see Cost guardrails)
└── Expander: [LeastNodes, LeastWaste] (priority chain)
  • Horizontal scaling — autoscaler adds workers when pods are Pending due to insufficient resources, and removes underutilized workers after a configurable cooldown.
  • Vertical scaling — multiple node pools with different server types. The expander is an ordered priority chain, [LeastNodes, LeastWaste] (upstream --expander=least-nodes,least-waste). LeastNodes runs first and keeps the pools that satisfy the pending pods with the fewest total new nodes — preferring the largest adequate type so a burst consolidates onto fewer, bigger servers; LeastWaste then breaks any tie by least idle CPU/memory. (Price is not supported on Hetzner: the cluster-autoscaler hcloud provider implements no pricing API, so KSail rejects it and the autoscaler crashes on startup.) See cluster-autoscaler FAQ.
  • KSail integration — KSail installs the Cluster Autoscaler Helm chart, generates the worker config secret (cluster-autoscaler-config), and manages the Talos snapshot lifecycle. Node pool configuration lives in ksail.prod.yaml, not in Flux manifests.
  • Storage architecture — autoscaler nodes are compute-only (no Longhorn storage). Static KSail workers are the Longhorn storage nodes and keep replicas on their root filesystem under /var/lib/longhorn. Pods on autoscaler nodes access Longhorn PVCs via the CSI driver (network). The Hetzner Cluster Autoscaler does not support volume attachment.

How new nodes join

  1. Cluster Autoscaler detects Pending pods with unmet resource requests.
  2. It calls the Hetzner API to create a new server using:
    • HCLOUD_IMAGE — a Talos snapshot managed by KSail.
    • HCLOUD_CLOUD_INIT — base64-encoded Talos worker machine config generated by KSail.
  3. The server boots Talos, applies the machine config, and joins the cluster.
  4. Once the node is Ready, pending pods are scheduled.

Configuration

All autoscaler configuration lives in ksail.prod.yaml under spec.cluster.autoscaler.node:

spec:
  cluster:
    autoscaler:
      node:
        enabled: true
        expander: [LeastNodes, LeastWaste]
        maxNodesTotal: 9
        scaleDownUnneededTime: "10m"
        capacityBuffers: true
        pools:
          - name: autoscale-cx43
            serverType: cx43
            location: fsn1
            min: 0
            max: 4
          - name: autoscale-cx53
            serverType: cx53
            location: fsn1
            min: 0
            max: 4
          - name: autoscale-arc-runners
            serverType: cx53
            location: fsn1
            min: 0
            max: 1
            labels:
              platform.devantler.tech/ci-runner: enabled
            taints:
              - key: platform.devantler.tech/ci-runner
                value: enabled
                effect: NoSchedule
Field Default Description
enabled false Enable/disable node autoscaling
expander LeastWaste Node selection strategy: LeastWaste, LeastNodes, Random (Price is unsupported on Hetzner — no pricing API). Accepts a single value or an ordered priority chain, e.g. [LeastNodes, LeastWaste] (requires KSail expander-list support; scalar-only up to v7.57.0)
maxNodesTotal 0 (unlimited) Hard ceiling on total cluster nodes, including the static baseline (see ksail#5017)
scaleDownUnneededTime 10m Time before an underutilized node is eligible for removal
pools[].name — DNS-1123 pool identifier
pools[].serverType — Hetzner server type (e.g., cx23, cx33)
pools[].location — Hetzner datacenter (e.g., fsn1)
pools[].min — Minimum nodes in pool
pools[].max — Maximum nodes in pool
pools[].labels / pools[].taints — Node labels and taints for the pool's servers (the runner pool uses both to keep ordinary workloads off)
capacityBuffers false Install the CapacityBuffer CRD and controller so the platform's CapacityBuffer reserves warm scale-up headroom

Cost guardrails

  • Hard max per pool — pools[].max caps each pool independently. The two general pools are set to 4 so any single CX type can serve a full burst instead of forcing larger types, and the CI runner pool is capped at 1; the maxNodesTotal total ceiling still caps the autoscaler at 3 (9 − 6 baseline), so this changes only the type distribution, never the node count.
  • Total node ceiling — maxNodesTotal caps the total cluster node count, including the static baseline. Set to 9 (6 static + up to 3 autoscaler). It is passed straight to cluster-autoscaler's --max-nodes-total, so the runtime already enforces this total.
  • serverLimit (spec.provider.hetzner.serverLimit) — the Hetzner account hard cap (10). Keep maxNodesTotal strictly BELOW serverLimit — KSail's Talos snapshot build creates a TEMPORARY extra server, so a 10 ≤ 10 config leaves it no slot: every Talos image bump then fails at the snapshot-ensure step, evicting every merge-queue deploy and breaking Heal Prod until a slot is freed (the 2026-07 CD outage). KSail enforces maxNodesTotal < serverLimit for Talos+Hetzner since ksail#6172; the reserved slot here is that rule applied.
  • Expander — [LeastNodes, LeastWaste] (current) is an ordered priority chain. LeastNodes runs first and keeps the pools that scale up with the fewest total new nodes (preferring the largest adequate type to keep the node count down); LeastWaste then breaks any remaining tie by least idle CPU/memory. The list form needs KSail expander-list support — releases up to v7.57.0 are scalar-only and reject a list, so the pinned KSAIL_VERSION must be bumped to a release that ships it first. (Price is unsupported on Hetzner.)
  • Scale-down — underutilized nodes are removed after 10 minutes.

Adding more pools

Add a new entry to the pools list in ksail.prod.yaml and run ksail cluster update. KSail updates the Helm release automatically.

Pod rebalancing & consolidation (descheduler)

The Cluster Autoscaler only ever adds or removes nodes — it never moves a running pod. Over time placement degrades (replica clumps after a node failure, pods left on nodes that now violate their affinity, half-empty autoscale-cx* nodes the autoscaler can't drain because one pod is stuck there). On clouds with Karpenter that gap is closed by its consolidation feature, but Karpenter has no Hetzner provider — so the platform pairs the autoscaler with the Kubernetes SIG Descheduler instead: the descheduler evicts misplaced pods (the scheduler re-places them), and the autoscaler reclaims any node that empties out (scaleDownUnneededTime: 10m).

It runs as a HelmRelease in kube-system (k8s/providers/hetzner/infrastructure/controllers/descheduler/), as a two-replica Deployment on a 30-minute descheduling loop. Required pod anti-affinity keeps the replicas on different nodes, and leader election lets only the lease holder evict pods, so a standby survives a node drain. Enabled strategies:

  • Placement repair — RemovePodsViolatingNodeAffinity (required affinity), RemovePodsViolatingNodeTaints, RemovePodsViolatingInterPodAntiAffinity, RemovePodsViolatingTopologySpreadConstraint (hard constraints only), and RemoveDuplicates (re-spread replica clumps after node churn).
  • Consolidation — HighNodeUtilization drains nearly-empty nodes (< 15% requested CPU and memory) toward the rest of the cluster so the autoscaler can delete them. kube-scheduler scores nodes with MostAllocated (talos/cluster/pack-pods-onto-busy-nodes.yaml), so evicted pods are more likely to land on a busier node that still fits than back on an empty-ish one. On a cx43, the DaemonSets plus Longhorn's per-node instance manager already request about 19% CPU and 21% memory (measured 2026-09-24), so a cx43 never falls below 15%; a cx53's larger capacity puts the same requests at roughly half that. Raising the threshold above the cx43 floor is #4167.

Guardrails (all in the HelmRelease values):

  • Evictions go through the eviction API, so PodDisruptionBudgets are always honored.
  • PVC-backed pods are never evicted (PodsWithPVC protection): CNPG clusters, Longhorn-attached workloads and openbao. Stateful moves stay deliberate runbook operations (see docs/dr/).
  • All default protections stay on: DaemonSet, mirror/static, system-critical, terminating, local-storage, and failed bare pods are untouchable.
  • nodeFit: true — a pod is only evicted if it provably fits on another node.
  • Blast-radius caps: at most 2 evictions per node and 2 per namespace per 30-minute cycle.

Evictions surface as Kubernetes Events (visible in Coroot); watch for pods bouncing between nodes every cycle — that's the signal a threshold is fighting the scheduler and should be tuned down.

Troubleshooting

Autoscaler not scaling up

# Check autoscaler logs
kubectl -n kube-system logs -l app.kubernetes.io/name=cluster-autoscaler --tail=100

# Check for unschedulable pods
kubectl get pods -A --field-selector=status.phase=Pending

# Check autoscaler status ConfigMap
kubectl -n kube-system get cm cluster-autoscaler-status -o yaml

Autoscaler nodes not joining

# Check if the Hetzner server was created
hcloud server list --selector hcloud/node-group

# Check Talos bootstrap status (if server IP is known)
talosctl -n <node-ip> health

# Verify the machine config is valid
talosctl validate --config worker.yaml --mode cloud

Cluster rebuild

After a full cluster rebuild (ksail cluster delete + create):

  1. KSail regenerates the worker config secret automatically.
  2. KSail manages the Talos snapshot lifecycle — no manual snapshot creation is needed.

Maintenance

Talos version upgrades

When bumping the Talos (or Kubernetes) version in ksail.prod.yaml, the deploy's ksail cluster update brings both node classes onto the new baseline:

  1. KSail's snapshot lifecycle manager creates or updates the Talos snapshot automatically during cluster update, and the worker machine config is regenerated to match — so new autoscaler nodes boot the new version.
  2. The static control planes and workers are upgraded in place (rolling).
  3. Existing autoscaler nodes are recycled automatically so they follow the new baseline instead of drifting on the old version: after the refreshed cluster-autoscaler is ready, KSail cordons and drains each autoscaler node one at a time (via the Kubernetes eviction API, honoring PodDisruptionBudgets) and deletes its Hetzner server; the autoscaler then re-provisions any still-needed capacity from the new snapshot on demand. This runs only when the version actually changes — a no-op cluster update leaves autoscaler nodes untouched. See the KSail Autoscaler Node Upgrades docs.

A strict PodDisruptionBudget on a workload running on autoscaler nodes can slow or block the drain (the update fails rather than force-evicting), so keep PDBs realistic for compute-only/burstable workloads.

Hetzner server type changes

Hetzner periodically renames or retires server types. Check the Hetzner Cloud changelog and update the pools[].serverType values in ksail.prod.yaml.