Repository navigation
model-serving: fix flaky controller tests - #1973
Abhayanthk wants to merge 2 commits into
Conversation
Signed-off-by: Abhayanth K <abhayanth2006@gmail.com>
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
Signed-off-by: Abhayanth K <abhayanth2006@gmail.com>
|
/retest |
|
@Abhayanthk: Cannot trigger testing until a trusted user reviews the PR and leaves an DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
What type of PR is this?
/kind cleanup
What this PR does / why we need it:
While testing
go test -raceon main, I noticed some ModelServing controller tests failing now and then, even though I hadn't changed anything in the controller. This PR fixes four causes.historical_plugin_context_test.gogive the controller broken revision history and check that it makes no workload writes. The controller also sends aRevisionUnresolvedwarning Event, which is correct. But the event broadcaster writes that Event to the same fake client from its own goroutine. When the Event lands before the check, the test sees acreateoneventsand fails. Both checks now skip theeventsresource, the same wayrevision_history_test.goalready does.TestModelServingControllerModelServingLifecyclestarts the controller withcontext.Background()and never stops it, so its worker keeps running after the test ends.NewModelServingControllerregisters types in the globalscheme.Scheme, and the leftover worker reads that scheme when it records an Event. With-race, a later test that creates a new controller (for exampleTestHandleDeletedPodNoneEnqueues) then fails with a data race. The test now stops the controller when it ends.syncModelServingdirectly while the controller's own worker is also running. The workqueue never syncs one ModelServing twice at the same time, but these direct calls skip it. So inModelServingRoleReplicasScaleDown, the worker sometimes removes a role from the store while the test's sync is still scaling down, and the test's sync then fails with "role prefill-1 not found in roleName prefill". The worker and the direct calls now go through one lock, so they take turns.Which issue(s) this PR fixes:
None
Bug evidence (required for bug-related PRs):
N/A
Special notes for your reviewer:
I used AI tools while working on this and verified everything myself.
Does this PR introduce a user-facing change?: