Skip to content

scalability: tolerate one NotReady node from 500 nodes in performance runs - #18950

Closed
dims wants to merge 2 commits into
kubernetes:masterfrom
dims:amazon-scale-run-test
Closed

dims wants to merge 2 commits into
kubernetes:masterfrom
dims:amazon-scale-run-test

Conversation

@dims

@dims dims commented Oct 30, 2026

Copy link
Copy Markdown
Member

About one t3a.medium in a few thousand never joins (containerd can't find shim for sandbox, ci-kubernetes-e2e-ec2-scale-performance-golang-tip-500, 2026-10-27). The 5000-node runs already tolerate two. The correctness scenario keeps zero because the e2e suite waits for every node.

Second commit drops the dead ENABLE_RESTART_COUNT_CHECK: false line: experiments/enable_restart_count_check.yaml is passed later and wins, so the check has always been on for AWS performance runs. It caught the LB controller crash loop on 2026-10-23, so it stays on.

AI assisted with the triage and the patch.

dims added 2 commits October 30, 2026 10:45
… runs

About one t3a.medium in a few thousand never joins. The 5000-node runs already tolerate two; the correctness scenario keeps zero because the e2e suite waits for every node.

Signed-off-by: Davanum Srinivas <davanum@gmail.com>
experiments/enable_restart_count_check.yaml is passed after overrides.yaml and wins, so the check has always been on for AWS performance runs.

Signed-off-by: Davanum Srinivas <davanum@gmail.com>
@kubernetes-prow kubernetes-prow Bot added size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. labels Oct 30, 2026
@kubernetes-prow
kubernetes-prow Bot requested review from hakman and serathius October 30, 2026 14:49
@kubernetes-prow

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign dims for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@kubernetes-prow

Copy link
Copy Markdown
Contributor

@dims: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
pull-kops-kubernetes-e2e-ubuntu-ec2-build ff73e49 link false /test pull-kops-kubernetes-e2e-ubuntu-ec2-build

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

@dims dims closed this Oct 30, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. size/XS Denotes a PR that changes 0-9 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant