Skip to content

model-serving: fix flaky controller tests - #1973

Open
Abhayanthk wants to merge 2 commits into
volcano-sh:mainfrom
Abhayanthk:fix/model-serving-tests-ignore-events
Open

Abhayanthk wants to merge 2 commits into
volcano-sh:mainfrom
Abhayanthk:fix/model-serving-tests-ignore-events

Conversation

@Abhayanthk

@Abhayanthk Abhayanthk commented Nov 6, 2026 •

Copy link
Copy Markdown
Contributor

What type of PR is this?
/kind cleanup

What this PR does / why we need it:
While testing go test -race on main, I noticed some ModelServing controller tests failing now and then, even though I hadn't changed anything in the controller. This PR fixes four causes.

  1. Two tests in historical_plugin_context_test.go give the controller broken revision history and check that it makes no workload writes. The controller also sends a RevisionUnresolved warning Event, which is correct. But the event broadcaster writes that Event to the same fake client from its own goroutine. When the Event lands before the check, the test sees a create on events and fails. Both checks now skip the events resource, the same way revision_history_test.go already does.
  2. TestModelServingControllerModelServingLifecycle starts the controller with context.Background() and never stops it, so its worker keeps running after the test ends. NewModelServingController registers types in the global scheme.Scheme, and the leftover worker reads that scheme when it records an Event. With -race, a later test that creates a new controller (for example TestHandleDeletedPodNoneEnqueues) then fails with a data race. The test now stops the controller when it ends.
  3. The gang subtest in the same test checks the ServingGroups without syncing first, so it passes only when the background worker happens to finish in time. It now syncs first, like the other subtests do.
  4. The subtests of that test call syncModelServing directly while the controller's own worker is also running. The workqueue never syncs one ModelServing twice at the same time, but these direct calls skip it. So in ModelServingRoleReplicasScaleDown, the worker sometimes removes a role from the store while the test's sync is still scaling down, and the test's sync then fails with "role prefill-1 not found in roleName prefill". The worker and the direct calls now go through one lock, so they take turns.

Which issue(s) this PR fixes:
None

Bug evidence (required for bug-related PRs):
N/A

Special notes for your reviewer:
I used AI tools while working on this and verified everything myself.

Does this PR introduce a user-facing change?:

NONE

Signed-off-by: Abhayanth K <abhayanth2006@gmail.com>
Copilot AI balanced review requested due to automatic review settings November 6, 2026 10:38
@volcano-sh-bot

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign lizhencheng9527 for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.

Signed-off-by: Abhayanth K <abhayanth2006@gmail.com>
Copilot AI balanced review requested due to automatic review settings November 6, 2026 13:38

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.

@Abhayanthk Abhayanthk changed the title model-serving: fix false test failures caused by Event writes model-serving: fix flaky controller tests Nov 6, 2026
@Abhayanthk

Copy link
Copy Markdown
Contributor Author

/retest

@volcano-sh-bot

Copy link
Copy Markdown
Contributor

@Abhayanthk: Cannot trigger testing until a trusted user reviews the PR and leaves an /ok-to-test message.

Details

In response to this:

/retest

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

Status: Accepted

Development

Successfully merging this pull request may close these issues.

3 participants