Skip to content

Remove flaky Go goroutine-count test from GeneratedRuntime_PassesGoRuntimeHardeningTests - #21474

Open
David Fowler (davidfowl) wants to merge 2 commits into
mainfrom
davidfowl/fix-go-goroutine-count-flake
Open

David Fowler (davidfowl) wants to merge 2 commits into
mainfrom
davidfowl/fix-go-goroutine-count-flake

Conversation

@davidfowl

@davidfowl David Fowler (davidfowl) commented Oct 30, 2026 •

Copy link
Copy Markdown
Collaborator

Description

Removes the flaky Go runtime test TestRegisterCancellationDoesNotParkGoroutinePerToken from the script that GeneratedRuntime_PassesGoRuntimeHardeningTests runs. This follows review feedback: we don't need a goroutine-count test for this.

Why

The test compared the process-wide runtime.NumGoroutine() before and after registering 1000 cancellation tokens. That count includes goroutines the test doesn't own (runtime/GC workers, goroutines from earlier tests still exiting), so on a busy runner it went over its 50-goroutine limit without any leak. Example: run 41074175594, Windows, first attempt, registering 1000 tokens left 1222 more goroutines running.

Frequency: since the test landed in #21194, it failed in 1 of 128 first-attempt Hosting.CodeGeneration.Go jobs (1/64 on Windows, 0/64 on Ubuntu). Locally it failed 8 of 900 runs under CPU stress.

An earlier revision of this PR deflaked it by counting live goroutines from a stack dump. Per review, this PR now removes the test instead.

Scope

  • Only that one Go test function is removed (11 lines).
  • The other generated-runtime hardening tests in GeneratedRuntime_PassesGoRuntimeHardeningTests are unchanged, including cancellation ID uniqueness, cancellation notification and malformed frames/handles.
  • No product code changes.

Validation

Aspire.Hosting.CodeGeneration.Go.Tests (AtsGoCodeGeneratorTests) passes locally: 40/40, including GeneratedRuntime_PassesGoRuntimeHardeningTests with Go installed.

Checklist

  • Is this feature complete?
    • Yes. Ready to ship.
  • Are you including unit tests for the changes and scenario tests if relevant?
    • Yes
    • No (test-only removal)
  • Did you add public API?
    • No
  • Does the change make any security assumptions or guarantees?
    • No
  • Does the change require an update in our Aspire docs?
    • No

🤖 Posted by AspireAgent (AI Agent)

@github-actions

Copy link
Copy Markdown
Contributor

🚀 Dogfood this PR with:

⚠️ WARNING: Do not do this without first carefully reviewing the code of this PR to satisfy yourself it is safe.

curl -fsSL https://raw.githubusercontent.com/microsoft/aspire/main/eng/scripts/get-aspire-cli-pr.sh | bash -s -- 21474

Or

  • Run remotely in PowerShell:
iex "& { $(irm https://raw.githubusercontent.com/microsoft/aspire/main/eng/scripts/get-aspire-cli-pr.ps1) } 21474"

@github-actions github-actions Bot added the area-engineering-systems infrastructure helix infra engineering repo stuff label Oct 30, 2026
@aspire-repo-bot
aspire-repo-bot Bot requested a balanced review from Copilot October 30, 2026 16:03
@github-actions

This comment has been minimized.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

No active review issues are blocking approval.

No open issues 🎉

What changed in this PR

This PR stabilizes the generated Go runtime’s cancellation regression test without changing production code or weakening its goroutine-growth threshold.

Changes:

  • Replaces runtime.NumGoroutine() measurements with live-goroutine counts from stop-the-world stack dumps.
  • Grows the dump buffer until complete and preserves the existing 50-goroutine threshold.
  • Documents why scheduler counters can temporarily include exited goroutines during garbage collection.
File Description
tests/​Aspire.Hosting.CodeGeneration.Go.Tests/​AtsGoCodeGeneratorTests.cs Adds a live-goroutine counting helper and uses it in the cancellation regression test.

Review effort: Balanced

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

No active review issues remain for the cancellation regression-test stabilization.

No open issues 🎉

Review effort: Balanced

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I do not think we need such test at all. Models are already knowledgeable about Go enough to flag indiscriminate goroutine creation.

Copy link
Copy Markdown
Collaborator Author

Karol Zadora-Przylecki (@karolz-ms) Thanks for the review. I've moved this back to draft while we settle on a direction.

Evidence

  • The flake: TestRegisterCancellationDoesNotParkGoroutinePerToken failed in 1 of 128 first-attempt Hosting.CodeGeneration.Go jobs since it landed in Harden the Go runtime transport and decode lists of handles #21194. That's 1/64 on Windows and 0/64 on Ubuntu. Under CPU stress it failed 8 of 900 times. The test compares the process-wide runtime.NumGoroutine() before and after the loop. That count includes goroutines the test doesn't own, such as runtime, GC and timer goroutines and other parallel tests, so on a busy runner it overshoots the 50-goroutine limit even when registerCancellation doesn't leak.
  • What it guards: that registerCancellation doesn't start one parked goroutine per token. That's a real regression class for the generated runtime, but it's narrow.

Options

  • (a) Remove just the TestRegisterCancellationDoesNotParkGoroutinePerToken Go subtest, one function in the GeneratedRuntime_PassesGoRuntimeHardeningTests script. The other ~20 runtime hardening subtests in that check (cancellation notification, malformed frames, handle decoding, DeepUpdate and so on) are unrelated to this flake. I'd keep them unless you want the whole check gone too.
  • (b) Keep the current deflake. It counts only live goroutines from a stop-the-world stack dump, keeps the same 50-goroutine threshold, and makes no product change.

Could you confirm (a), and whether "such test" means only the goroutine-count subtest or the whole GeneratedRuntime_PassesGoRuntimeHardeningTests check? Once you confirm, I'll update this PR to match.

🤖 Posted by AspireAgent (AI Agent)

runtime.NumGoroutine can briefly count exited goroutines as running while
the GC frees their stacks, so the delta in
TestRegisterCancellationDoesNotParkGoroutinePerToken flaked after the
previous test spawned 5000 goroutines.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Per review feedback, drop TestRegisterCancellationDoesNotParkGoroutinePerToken
from the generated Go runtime hardening tests instead of deflaking it. The test
measured process-wide goroutine counts, which are noisy on busy runners.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@davidfowl
David Fowler (davidfowl) force-pushed the davidfowl/fix-go-goroutine-count-flake branch from 2acd3c5 to 249207d Compare October 31, 2026 00:32
@davidfowl David Fowler (davidfowl) changed the title Fix flaky Go goroutine-count test in GeneratedRuntime_PassesGoRuntimeHardeningTests Remove flaky Go goroutine-count test from GeneratedRuntime_PassesGoRuntimeHardeningTests Oct 31, 2026

Copy link
Copy Markdown
Collaborator Author

Karol Zadora-Przylecki (@karolz-ms) I've updated the PR to remove the test as you suggested. It now deletes only TestRegisterCancellationDoesNotParkGoroutinePerToken (11 lines, in 249207d). The deflake helper is gone, and the other generated-runtime hardening tests are unchanged. The Go codegen tests pass locally (40/40). If you meant the whole GeneratedRuntime_PassesGoRuntimeHardeningTests check, let me know and I'll widen it.

🤖 Posted by AspireAgent (AI Agent)

@github-actions

Copy link
Copy Markdown
Contributor

Tests selector

1 / 100 PR test projects · 0 PR jobs, from 1 changed file.

Selected PR test projects (1 / 100)

Aspire.Hosting.CodeGeneration.Go.Tests

Selected PR jobs (0)

none


How these were chosen — grouped by what changed

🧪 tests/Aspire.Hosting.CodeGeneration.Go.Tests/AtsGoCodeGeneratorTests.cs (changed test)
→ 1 directly: Aspire.Hosting.CodeGeneration.Go.Tests

Job reasons

none


Selection computed for commit 249207d.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

No active review issues are reported for the removal of the flaky goroutine-count test.

No open issues 🎉

Review effort: Balanced

@davidfowl
David Fowler (davidfowl) marked this pull request as ready for review October 31, 2026 08:39

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area-engineering-systems infrastructure helix infra engineering repo stuff fixes-ci flaky-test

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants