Skip to content

Fix flaky ExecuteCommandAsync_RebuildCommand_ReturnsBuildOutput build count - #21427

Merged
James Newton-King (JamesNK) merged 3 commits into
mainfrom
agent/fix-rebuild-command-build-count
Oct 31, 2026
Merged

James Newton-King (JamesNK) merged 3 commits into
mainfrom
agent/fix-rebuild-command-build-count

Conversation

@davidfowl

Copy link
Copy Markdown
Collaborator

Description

Fixes the intermittent ExecuteCommandAsync_RebuildCommand_ReturnsBuildOutput failure (Assert.Equal() Failure: Expected: 2, Actual: 3) on Hosting-2 (windows-latest).

Root cause. The test project appends to build-count.txt from an AfterTargets="Build" target and the test asserts that the second rebuild command adds exactly one line. But the project resource itself is launched with dotnet run (no --no-build), which also builds it. After the first rebuild restarts the project, the second rebuild has to stop that dotnet run process. On Windows the stop often takes several seconds (Executable ... is still running; trying again to stop it...), so the restarted dotnet run finishes its own build and writes an extra line. The test then sees 3 lines instead of 2. The native TRX output from run 39366740926 shows this sequence: a 6-second stop window between the restart and the second rebuild.

Fix. The hidden myProject-rebuilder resource gets an ASPIRE_TEST_REBUILD_COMMAND_BUILD=true environment variable, and the counting target only runs when that variable is set. Only builds started by the rebuild command are counted. The assertions are stricter than before: exactly 1 after the first rebuild and exactly 2 after the second. No product code changes.

Verification (local, Linux, repo SDK).

  • I used a temporary deterministic repro (not committed) that forces the CI interleaving by waiting for the restarted dotnet run to exit 0 before the second rebuild. With the original counting it failed 3/3 with Expected: 2, Actual: 3 (one run: 2→4). With the new counting it passed 3/3.
  • The updated ExecuteCommandAsync_RebuildCommand_ReturnsBuildOutput passed 5/5 runs.

Recurrence. On main CI between 2026-10-13 and 2026-10-30, 14 Hosting-2 (windows-latest) jobs failed out of about 120 sampled. 10 of those failures had TRX artifacts, and this test was the only failure in all 10. The other 4 had no artifacts. The most recent failure was on 2026-10-30 (run 40992075134).

Fixes #20662

Checklist

  • Is this feature complete?
    • Yes. Ready to ship.
    • No. Follow-up changes expected.
  • Are you including unit tests for the changes and scenario tests if relevant?
    • Yes
    • No
  • Did you add public API?
    • Yes
    • No
  • Does the change make any security assumptions or guarantees?
    • Yes
    • No

🤖 Posted by AspireAgent (AI Agent)

The project resource is launched with `dotnet run`, which also builds it.
When the rebuild command restarts the project and the next rebuild cannot
stop that process before its build finishes (common on Windows), the shared
build-count file gets an extra entry and the test sees 3 builds instead of 2.

Tag the hidden rebuilder's `dotnet build` with an environment variable and
only count builds that carry it, so the assertion measures exactly the
builds performed by the rebuild command.

Fixes #20662

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@github-actions

Copy link
Copy Markdown
Contributor

🚀 Dogfood this PR with:

⚠️ WARNING: Do not do this without first carefully reviewing the code of this PR to satisfy yourself it is safe.

curl -fsSL https://raw.githubusercontent.com/microsoft/aspire/main/eng/scripts/get-aspire-cli-pr.sh | bash -s -- 21427

Or

  • Run remotely in PowerShell:
iex "& { $(irm https://raw.githubusercontent.com/microsoft/aspire/main/eng/scripts/get-aspire-cli-pr.ps1) } 21427"

@aspire-repo-bot
aspire-repo-bot Bot requested a balanced review from Copilot October 30, 2026 04:27
@github-actions github-actions Bot added the area-engineering-systems infrastructure helix infra engineering repo stuff label Oct 30, 2026
@github-actions

This comment has been minimized.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

No active review issues are reported for the test-only change that isolates rebuild-command build counting.

No open issues 🎉

What changed in this PR

Addresses the flaky rebuild-command test in #20662 by excluding builds triggered by dotnet run from its build counter. Changes are limited to tests; product behavior is unchanged.

Changes:

  • Tags the hidden rebuilder resource with a test-only environment variable.
  • Counts builds only when that variable is set.
  • Asserts exactly one build after the first rebuild command and two after the second.
File Description
tests/​Aspire.Hosting.Tests/​ResourceCommandServiceTests.cs Isolates rebuild-command build counting and tightens assertions to avoid the Windows timing race.

Review effort: Balanced

@github-actions

Copy link
Copy Markdown
Contributor

Retrying the failed CI jobs for this pull request from the CI run attempt. The rerun is being tracked in the rerun attempt.

@github-actions

Copy link
Copy Markdown
Contributor

Retrying the failed CI jobs for this pull request from the CI run attempt. The rerun is being tracked in the rerun attempt.

@github-actions

Copy link
Copy Markdown
Contributor

Retrying the failed CI jobs for this pull request from the CI run attempt. The rerun is being tracked in the rerun attempt.

@aspire-repo-bot
aspire-repo-bot Bot requested a balanced review from Copilot October 30, 2026 05:55
@github-actions

This comment has been minimized.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

No active review issues remain for the test-only change that isolates rebuild-command counts from startup builds.

No open issues 🎉

Review effort: Balanced

@github-actions

Copy link
Copy Markdown
Contributor

Retrying the failed CI jobs for this pull request from the CI run attempt. The rerun is being tracked in the rerun attempt.

Copy link
Copy Markdown
Collaborator Author

New occurrence on an unrelated PR: #21461 run 41076125682, Hosting-2 (windows-latest), attempt 1, at ResourceCommandServiceTests.cs:1770.

🤖 Posted by AspireAgent (AI Agent)

# Conflicts:
#	tests/Aspire.Hosting.Tests/ResourceCommandServiceTests.cs
@github-actions

Copy link
Copy Markdown
Contributor

Tests selector

6 / 100 PR test projects · 0 PR jobs, from 1 changed file.

Selected PR test projects (6 / 100)

Aspire.Hosting.Azure.Kubernetes.Tests, Aspire.Hosting.Azure.Tests, Aspire.Hosting.Blazor.Tests, Aspire.Hosting.Dotnet.Tests, Aspire.Hosting.Radius.Tests, Aspire.Hosting.Tests

Selected PR jobs (0)

none


How these were chosen — grouped by what changed

🧪 tests/Aspire.Hosting.Tests/ResourceCommandServiceTests.cs (changed test)
→ 1 directly: Aspire.Hosting.Tests
→ 5 via the project graph: Aspire.Hosting.Azure.Kubernetes.Tests (2 hops), Aspire.Hosting.Azure.Tests, Aspire.Hosting.Blazor.Tests, Aspire.Hosting.Dotnet.Tests, Aspire.Hosting.Radius.Tests

Job reasons

none


Selection computed for commit fdfe4be.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

No active review issues are reported for the test-only change that isolates rebuild-command counting.

No open issues 🎉

Review effort: Balanced

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No issues found. Reviewed rebuilder environment isolation, conditional build counting, and sequential rebuild completion. The changed ExecuteCommandAsync_RebuildCommand_ReturnsBuildOutput regression test passed 3/3 local Windows runs with MTP-native filtering and quarantined/outerloop tests excluded.

@JamesNK
James Newton-King (JamesNK) merged commit 6a8f531 into main Oct 31, 2026
72 checks passed
@microsoft-github-policy-service microsoft-github-policy-service Bot added this to the 17.0 milestone Oct 31, 2026
@aspire-repo-bot

Copy link
Copy Markdown
Contributor

✅ No documentation update needed.

docs_optional → test_only. The only changed file is tests/Aspire.Hosting.Tests/ResourceCommandServiceTests.cs, which matches tests/**. The one triggered signal, pr_body_has_cli_flag_mention, is a false positive. It matched the PR body's mention of dotnet run --no-build while describing the flaky test fix, and no user-facing flag or behavior changed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area-engineering-systems infrastructure helix infra engineering repo stuff flaky-test

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[CI Failure] Flaky: ResourceCommandServiceTests.ExecuteCommandAsync_RebuildCommand_ReturnsBuildOutput build-count line mismatch on Windows

4 participants