[David Fowler (davidfowl)]

Description

Windows runner loss and hangs are the largest flake family in #21600: Windows extension E2E and .NET test jobs freeze at 47–59 minutes against a 3–9 minute baseline, then lose their runner (#21570). We can't currently answer whether the machines are overloaded on CPU, memory or disk, because nothing records machine load in a form that survives the job.

This PR adds a lightweight background resource sampler to:

• the Windows .NET test jobs (run-tests.yml)
• the Windows VS Code extension E2E shards (extension-e2e-tests.yml)

What it records (eng/scripts/ci-resource-monitor.ps1): a detached, below-normal-priority pwsh process starts right after checkout. Every 20s it writes:

• resources.csv: CPU %, processor queue length, available MB, commit charge % / committed GB / commit limit, paging pages/s, page file usage %, disk queue length, disk % time, free GB on the system and workspace drives, and process count.
• processes.log: the top 5 processes by CPU (since the previous sample) and by working set (with private bytes).

Where it goes: the sampler writes into directories that the existing if: always() uploads already collect, so there are no new artifacts:

• .NET tests: artifacts/log/test-logs/resource-monitor/ inside logs-<test>-windows-latest.
• Extension E2E: extension/.test-results/resource-monitor/ inside extension-e2e-diagnostics-win-x64-<shard>-attempt<N>.

A Stop resource monitor (Windows) step (if: always()) kills the sampler before the upload. It prints a peak/minimum table, including the largest gap between samples (a stall indicator), to the log and the job summary. This covers passing, failing, cancelled and timed-out jobs.

Safety:

• Every mode exits 0 and both steps are continue-on-error.
• Test commands, timeouts, retries and environment are unchanged.
• Linux and macOS are untouched.
• Overhead is one Get-Counter and one Get-Process call every 20s at below-normal priority.

Limitation: a runner that loses communication uploads nothing and keeps no logs. This telemetry can't capture the final minutes of a lost runner. It can show whether pressure builds on the slow jobs that survive and on the shards that typically get lost. That is still more than we have today: the existing console Heartbeat (60s, .NET tests only) vanishes with the log.

Contributes to #21570. It complements #21608 (Docker pin), which adds no telemetry.

Existing evidence (before this PR)

On main, the last 16 completed ci.yml push runs had 1,551 Windows jobs. All 20 jobs that ran 45 minutes or longer were runner-lost; job logs return BlobNotFound, so there's no Heartbeat data from any of them. Examples: extension E2E resource-debug-tools, 110m and Hosting-4, 59m.

Surviving Windows Hosting-6 jobs on main (example) show the following in Heartbeat:

• memory 3.1–3.7 / 16.0 GB (20–23%)
• CPU 60–80% during test execution
• C: about 29 GB free, D: about 140 GB free

None of this shows exhaustion, but it doesn't cover commit charge, page file or disk queue.

Validation

• tests/Infrastructure.Tests/PowerShellScripts/CiResourceMonitorTests.cs:
  • the summary math (peaks, minimums, averages, stall gap)
  • Stop with no samples
  • real Start → background sample → Stop that ends the sampler
• actionlint is clean on both workflows.
• This PR's own Windows CI run: results below.

Observed in this PR's CI

The data comes from three CI runs of this PR on Windows hosted runners: 41289502805, 41290605859 (both partly cancelled by later pushes) and 41291481771 (head d0358d42). It covers 331 Windows jobs with 2,923 samples: 312 .NET test jobs and 19 VS Code extension E2E shards. The sampler emitted no missing-counter warnings.

	.NET test jobs (312)	Extension E2E shards (19)
CPU: mean of per-job averages	54.8%	44.3%
CPU: share of samples at 90% or more	27.6%	6.5%
Jobs with at least half their samples at 90% or more CPU	44	0
Lowest available memory (of 16 GB)	9,891 MB	9,779 MB
Peak commit charge (limit about 18.9 GB)	33.4%	36.6%
Peak page file usage	22%	19.9%
Peak disk queue length	146.5 (one job; 4 jobs above 60)	28.7
Lowest free space on C:	25.6 GB	23.5 GB
Largest gap between samples	37 s	22 s

Examples:

• Extension E2E shard discovery-configuration: CPU 35% average (66% peak), at least 10.9 GB available, commit at most 26.7%, disk queue at most 11.6. The diagnostics artifact extension-e2e-diagnostics-win-x64-discovery-configuration-attempt1 contains resource-monitor/resources.csv and processes.log.
• Hosting.Dotnet: CPU 84.5% average with 10 of 16 samples at 90% or more. The logs-Hosting.Dotnet-windows-latest artifact contains resource-monitor/. Its processes.log shows VBCSCompiler, dotnet and MSBuild as the top CPU consumers, i.e. the test-driven project builds.
• Cancelled Hosting-4: Stop and upload still ran (CPU 82% average, disk queue at most 17.9).
• The sampler's own pwsh uses about 1% CPU in processes.log.

Runner loss was reproduced in this run, and those jobs stayed dark. On head d0358d42, three .NET jobs froze in Run tests (Windows) at about 09:15Z: Hosting.Java (2-minute baseline), Hosting.Browsers (3-minute baseline) and Hosting-4 (8-minute baseline). MTP's --timeout 20m never fired. All three were declared lost about 46 minutes after job start: "The hosted runner lost communication with the server", logs BlobNotFound, and no artifacts. The extension E2E winui-debug shard (5–6-minute baseline) froze the same way in Run extension E2E tests from 09:34Z and was lost about 47 minutes after job start. This matches main: in 9 of the last 16 ci.yml push runs, 1–3 Windows jobs per run were lost at 46–48 minutes. In this PR's two earlier runs the same Java and Browsers jobs, also running the sampler, finished in 1–3 minutes.

What this answers so far:

1. Memory and commit: not under pressure on any surviving Windows job. At least 9.8 GB was always available, commit charge stayed at or below 37%, and the page file at or below 22%.
2. Disk: space is fine (at least 23.5 GB free on C:, about 216 GB on D:). The disk queue spikes occasionally (above 60 in 4 of 331 jobs), during builds.
3. CPU: the 4-vCPU runners are often saturated during .NET test jobs, mostly by the builds that tests trigger. Extension E2E shards are moderate. No surviving job showed a stall: the largest gap between samples was 37s against a 20s interval.
4. Frozen jobs: they freeze within minutes of starting tests, and even the 2-minute Java suite froze. Telemetry can't see inside them, because a lost runner uploads nothing. So the data rules out gradual memory or disk exhaustion on the jobs we can observe, but can't confirm or exclude a sudden in-VM event. Capturing that would need an off-box sink, such as streaming samples to an external store. That is out of scope for this lightweight change.

Checklist

• Is this feature complete?
  • Yes. Ready to ship.
  • No. Follow-up changes expected.
• Are you including unit tests for the changes and scenario tests if relevant?
  • Yes
  • No
• Did you add public API?
  • Yes
  • No
• Does the change make any security assumptions or guarantees?
  • Yes
  • No

🤖 Posted by AspireAgent (AI Agent)

[Copilot]

🟢 Approval recommended

No active review issues are reported for the added Windows CI resource telemetry.

No open issues 🎉

_{Review effort: Balanced}

[github-actions]

Tests selector

Selects the full PR test matrix + all PR-gated jobs (ALL) — a rule matching '.github/workflows/run-tests.yml' selects ALL

---

Selection computed for commit d0358d4.

[github-actions]

🚀 Dogfood this PR with:

⚠️ WARNING: Do not do this without first carefully reviewing the code of this PR to satisfy yourself it is safe.

curl -fsSL https://raw.githubusercontent.com/microsoft/aspire/main/eng/scripts/get-aspire-cli-pr.sh | bash -s -- 21644

Or

• Run remotely in PowerShell:

iex "& { $(irm https://raw.githubusercontent.com/microsoft/aspire/main/eng/scripts/get-aspire-cli-pr.ps1) } 21644"

[Copilot]

🟡 Changes recommended

CPU rankings omit newly observed processes that may dominate an interval, and tests allow empty performance-counter fields without validating Windows counter collection, mapping, or unit conversions.

2 open issues

• Include newly started processes in interval CPU rankings · New
• Validate Windows counter samples and field mappings · New

What changed in this PR

Adds Windows resource telemetry to help investigate CI slowdowns and runner losses tracked in #21570, using existing diagnostic artifacts.

Changes:

• Adds a background PowerShell sampler for machine pressure and top-process usage, with a shutdown summary.
• Starts and stops monitoring in Windows .NET test jobs and extension E2E shards.
• Adds behavioral tests for summary calculations, empty output, and sampler lifecycle.

File	Description
tests/​Infrastructure.Tests/​PowerShellScripts/​CiResourceMonitorTests.cs	Tests summaries and sampler startup/shutdown.
eng/​scripts/​ci-resource-monitor.ps1	Collects resource metrics and generates diagnostic summaries.
.github/​workflows/​run-tests.yml	Adds Windows monitoring to .NET test jobs.
.github/​workflows/​extension-e2e-tests.yml	Adds Windows monitoring to extension E2E shards.

_{Review effort: Balanced}

[Copilot]

🟡 Changes recommended

Non-terminating Get-Counter errors are suppressed without being logged, leaving users with missing counter values and no underlying diagnostics in sampler.log.

1 open issue

• Log non-terminating Get-Counter errors · New

2 resolved since last review

• Validate Windows counter samples and field mappings
• Include newly started processes in interval CPU rankings

_{Review effort: Balanced}

[Copilot]

🟢 Approval recommended

No active review issues remain for the Windows resource-telemetry changes.

No open issues 🎉

1 resolved since last review

• Log non-terminating Get-Counter errors

_{Review effort: Balanced}

[github-actions]

Retrying the failed CI jobs for this pull request from the CI run attempt. The rerun is being tracked in the rerun attempt.

[David Fowler (davidfowl)]

CI attempt 1 on head d0358d4 (run 41291481771) had 8 failed jobs, all unrelated to this PR:

• Runner lost (Windows CI runners lose communication during test steps since the windows-2025-vs2026 20261007 image (not test hangs) #21570), frozen in tests, declared lost after 46-48 minutes: Windows Hosting.Java, Hosting.Browsers, Hosting-4 and extension E2E winui-debug.
• Hosting.JavaScript (Windows): Deno resource "should have valid address", known DCP/Deno issue ([CI Failure] Flaky: MySqlFunctionalTests.WithDataShouldPersistStateBetweenUsages times out because DCP state.sqlite3 is locked (SQLITE_BUSY) #20554).
• Hosting.MongoDB (Ubuntu): replica set rs0 failed to initialize, known (Wait for MongoDB single-member primary to accept writes before reporting healthy #21638).
• Hosting.Dotnet (Windows): UrlsAreMarkedAsInternalDependingOnDisplayLocation hit its 2-minute Polly timeout. The same test also failed on main (job 124467041214).
• Hosting-2 (Windows): ExecuteCommandAsync_RebuildCommand_ReturnsBuildOutput rebuild result flake (earlier history in Fix flaky ExecuteCommandAsync_RebuildCommand_ReturnsBuildOutput build count #21427).

The repository's auto-rerun workflow started attempt 2. That is the single rerun for this head; I won't trigger another.

🤖 Posted by AspireAgent (AI Agent)