Skip to content

chore(eks): release idle staging node group (ng_staging_min_size 1 → 0) - #1067

Closed
arielr-lt wants to merge 1 commit into
masterfrom
chore/scale-down-staging-nodegroup
Closed

arielr-lt wants to merge 1 commit into
masterfrom
chore/scale-down-staging-nodegroup

Conversation

@arielr-lt

Copy link
Copy Markdown
Collaborator

What

Set ng_staging_min_size = 0 (was 1) in terraform/environments/eks/terraform.tfvars for the ce-registry-eks cluster.

Why

The ce-registry-eks-ng-staging node group (1× t3.medium) is empty — there is no staging namespace on the cluster and nothing targets its env=staging taint. It was floored at min=1, so the cluster-autoscaler could not reclaim the idle node. Cost cleanup alongside the Argo/ES/Neo4j decommission node-group reclaims.

How it works

  • desired_size is under lifecycle.ignore_changes (autoscaler-managed): Terraform leaves it at 1, and the cluster-autoscaler drains desired 1 → 0 once min_size = 0 applies.
  • max_size stays at 6.

Safety (verified)

  • The sole node runs only DaemonSets (0 app pods), tainted env=staging.
  • No staging namespace; 0 pods anywhere target env=staging.
  • Only the staging node group is affected — prod (ng-prod-v2), sandbox, and private node groups are untouched (separate resources/variables).

Rollback

Raise ng_staging_min_size back to 1.

Fallback (if autoscaler is slow)

aws eks update-nodegroup-config \
  --region us-east-1 \
  --cluster-name ce-registry-eks \
  --nodegroup-name ce-registry-eks-ng-staging \
  --scaling-config minSize=0,maxSize=6,desiredSize=0 \
  --profile credreg-sso

The ce-registry-eks staging node group (ce-registry-eks-ng-staging, 1x
t3.medium) is empty: there is no staging namespace and nothing targets the
env=staging taint. It was floored at min=1, so the cluster-autoscaler could
not reclaim the idle node.

Set ng_staging_min_size to 0 so the autoscaler drains it. desired_size is under
lifecycle.ignore_changes (autoscaler-managed), left at 1 and driven to 0 by the
autoscaler; max_size stays at 6.

Only the staging node group is affected (dedicated env=staging taint); prod,
sandbox and private node groups are untouched. Reversible: raise min back to 1.
@rohit-joy

Copy link
Copy Markdown
Contributor

@arielr-lt Is this needed? If not, please close PR.

@arielr-lt arielr-lt closed this Sep 3, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants