Activity

1. David Fowler (davidfowl)

   added

   area-engineering-systemsinfrastructure helix infra engineering repo stuffinfrastructure helix infra engineering repo stuff

   ci-failure-cause

   on Oct 31, 2026
2. davidfowl commented on Oct 31, 2026

   

   David Fowler (davidfowl)

   on Oct 31, 2026 – with Copilot SWE Agent

   CollaboratorAuthor

   More actions

   Claiming JavaScriptPublishTests.AllPublishMethodsBuildDockerImages (Top unowned #5).

   From the CI logs, the aspire init timeouts are npm E404s during the TypeScript AppHost npm install. They hit tarballs published minutes earlier, for example @typescript-eslint/typescript-estree@8.72.1 and @types/estree@1.0.10. I'm working on a fix for the E2E environment and will link the PR here.

   🤖 Posted by AspireAgent (AI Agent)
3. davidfowl commented on Oct 31, 2026

   

   David Fowler (davidfowl)

   on Oct 31, 2026 – with Copilot SWE Agent

   CollaboratorAuthor

   More actions

   Extra evidence for the AllPublishMethodsBuildDockerImages claim above. Not all of the aspire init timeouts are E404s.

   Window: ci.yml check runs from 2026-10-24 20:00Z to 2026-10-31 20:00Z, excluding Dependabot npm branches, whose failures look deterministic. 8 attempts failed. 4 of them timed out in AllPublishMethodsBuildDockerImages at WaitUntilText("Created apphost.mts"), and every one of those 4 passed on retry:

   Job	Cause
   121723096663 (10-26)	npm E404 for typescript-estree-8.72.1.tgz
   123265152317 (10-29)	npm E404 for estree-1.0.10.tgz
   122678762903 (10-28)	npm install stall. The NuGet restore finished in about 4s (_aspire files 07:02:50 → 07:02:54). No package-lock.json had been written when the 120s timeout hit.
   124617913801 (10-31)	NuGet restore stall. In-process restore of Aspire.Hosting plus about 74 libraries took 96s (obj/ 18:20:14 → 18:21:50; passing runs take about 5s), followed by about 42s of npm. Created apphost.mts appeared at 148.6s.

   The other failures in the window are JavaScriptHostingApisRunFromTypeScriptAppHost (2 attempts), one aspire deploy pipeline timeout, and one PR-specific failure on nuget-cli-archive.

   A fix that only handles E404s will probably leave the two stall cases. Both are cold-cache registry fetches inside the E2E container.

   🤖 Posted by AspireAgent (AI Agent)
4. davidfowl commented on Oct 31, 2026

   

   David Fowler (davidfowl)

   on Oct 31, 2026 – with Copilot SWE Agent

   CollaboratorAuthor

   More actions

   Update on JavaScriptPublishTests.AllPublishMethodsBuildDockerImages: draft PR #21615 is open. It covers the npm fresh-release E404 race (3 of the 8 intermittent failures in the last 7 days are proven; 821 runs completed). The verify-runtime.sh 30s-budget failures are not covered by it.

   🤖 Posted by AspireAgent (AI Agent)
5. davidfowl commented on Oct 31, 2026

   

   David Fowler (davidfowl)

   on Oct 31, 2026 – with Copilot SWE Agent

   CollaboratorAuthor

   More actions

   Triage of the "Top unowned flakes" entries that had no claim. Neither one needs a new fix PR.

   #3 AddJavaContainerTests.AddJavaContainer_ExportsTelemetryToAspire: already fixed by #21432 (merged 2026-10-30 15:56Z; an empty DCP hostName now falls back to the runtime default).

   • Window: ci.yml attempt-1 runs from the Fall back to default container host name when DCP reports an empty one #21432 merge to about 2026-10-31 20:00Z.
   • Hosting.Java on Windows: 62 of 68 passed. 5 failures were runner loss (Windows CI runners lose communication during test steps since the windows-2025-vs2026 20261007 image (not test hangs) #21570). 1 was a deterministic failure specific to one PR (run 41194713391, "CliPath/DashboardPath required").
   • Hosting.Java on Ubuntu: 67 of 68 passed. The one failure is that same PR run.
   • main: 19 runs, 0 Java failures. The 4 main hits in the 7-day count all predate Fall back to default container host name when DCP reports an empty one #21432.

   #1 VS Code Aspire Node Functions debugger E2E: effectively fixed by #21327 (merged 2026-10-28 16:07Z).

   • All 5 main hits in the 7-day window predate the merge (latest: 10-28 02:40Z).
   • Check runs after the merge, excluding Dependabot: 573 succeeded, 14 failed, 4 cancelled.
     • 12 failures are on one branch (danegsta-nuget-mapping-regression). Every retry there failed too, so they're specific to that PR.
     • The other 2 passed on retry: job 124203484416 (aspireagent/fix-21465-dcp-session-folder, failed after 2m11s) and job 124390529919 (danegsta-browser-endpoint-resolution, failed 5s after the AppHost launch).
     • That's 2 of about 585 runs (about 0.3%), both on branches that change DCP or endpoint code.
   • Diagnostic gap for whoever investigates further: both failures report only E2E control command failed. getE2eErrorMessage in extension/src/testing/e2eStateFileBridge.ts keeps only messages that start with Aspire extension E2E . That drops the detailed Diagnostics: payload proveAppHostAndResourceDebugging builds (failed launch/attach responses, terminated sessions, stopped events). The teardown's stopDebugging control then overwrites extension-state.json, so the artifacts don't keep the reason either.

   The remaining entries are claimed: #2 and #4 by #21335, #5 by #21615.

   🤖 Posted by AspireAgent (AI Agent)
6. davidfowl commented on Oct 31, 2026

   

   David Fowler (davidfowl)

   on Oct 31, 2026 – with Copilot SWE Agent

   CollaboratorAuthor

   More actions

   Claiming the verify-runtime.sh failure mode of Aspire.Cli.EndToEnd.Tests.JavaScriptPublishTests.JavaScriptHostingApisRunFromTypeScriptAppHost. #21615 lists it as "not addressed here". That PR covers the npm E404 part of the same job; this work doesn't touch the files it changes.

   Evidence:

   • run 41207387696, job 124557722651
   • run 41214363995, job 124579017281

   These are on two different branches today. The recording shows aspire run > aspire-run.log 2>&1 & at t=68.08s and bash verify-runtime.sh at t=68.33s. The script then reports node-npm_FAIL at t=98.37s while aspire-run.log still ends at ⚙️ Preparing Aspire server.... The 30s per-resource polling budget starts before the AppHost server is even built, so it also covers AppHost preparation and startup.

   The fix gates on actual readiness instead of a longer budget: aspire start, which returns once the AppHost is running, then aspire wait <resource> --status up, then the existing endpoint checks.

   🤖 Posted by AspireAgent (AI Agent)
7. davidfowl commented on Oct 31, 2026

   

   David Fowler (davidfowl)

   on Oct 31, 2026 – with Copilot SWE Agent

   CollaboratorAuthor

   More actions

   Census of unclaimed flakes. Every intermittent test failure in the windows below already has an owner, a merged fix, or is PR-specific or a known external cause, so I'm not opening a new fix PR.

   Windows scanned:

   • main runs from 2026-10-28 16:00Z to about 20:40Z on 10-31: 51 commits, 336 failed check runs.
   • Non-Dependabot PR runs from 2026-10-29 20:00Z to now: 368 commits, 823 failed test jobs.
   • Failed tests were extracted from the job logs.

   Candidate (not on the skip list)	Hits	Disposition
   NativeCliProtocolTests.SilentAndPartialFramesExpire…	4 main commits	claimed: #21442
   DotNetCliRunnerTests.ComputeRunArgumentsPopulatesRunProperties…	2 main, 3 branches	claimed: #21502
   McpDocsE2ETests / RadiusDeployTests / StarterTemplateWithRedisCacheTests_Net11	3 outerloop main runs	#21450 / #21422 / #21423
   DashboardUrlTests.DashboardDisplaysResourceUpdatesWhileAppHostIsPausedAtBreakpoint	8 outerloop jobs	fails every outerloop run, so it's not intermittent (#18657)
   AgentTelemetryPersistenceTests.ForceFlush_LocalTelemetryReturnsBeforeUpload	2 branches	claimed: #21499
   DistributedApplicationPipelineTests.* (destroy access denied)	2 branches	claimed: #21453
   DotnetProjectBuildCoordinatorTests.FailedBuildResourceStartDeletesResponseFile	1 main, 5 branches	6 deterministic CliPath failures on one PR branch. The other 5 (4 PR branches plus the 1 main hit) are DCP SQLITE_BUSY state-store crashes (#20467, waiting on microsoft/dcp#326) or the StopAsync TaskCanceledException (#21528).
   DotnetProjectWithUrlsTests.UrlsAreMarkedAsInternal… ("should have valid address")	1 main	DCP SQLITE_BUSY crash (#20467)
   AuxiliaryBackchannelRpcTargetTests.GetResourcesAsync_MapsNonString…	1 main	StopAsync TaskCanceledException (#21528)
   *ReplCommandTests.ReplExecutesAuthenticatedQuery, the container functional tests	2–3 branches	only on the WSLC-on-Windows PR branch plus one branch with deterministic CliPath failures
   TestRunCoordinatorEndToEndTests.*	5 branches	the test exists only in feature draft PR #21527
   ResourcesLayoutTests, TerminalWebSocketTests, AnalyzeCiFailureWorkflowTests, IntegrationHostProcessTests	1–2	claimed: #21516, #21471, #21558, #21554

   The next highest-value step is landing the open, ready fixes above rather than opening new PRs.

   🤖 Posted by AspireAgent (AI Agent)
8. davidfowl commented on Oct 31, 2026

   

   David Fowler (davidfowl)

   on Oct 31, 2026 – with Copilot SWE Agent

   CollaboratorAuthor

   More actions

   Follow-up on the aspire init timeouts that #21615 did not cover (TypeScript AppHost WaitUntilText("Created apphost.mts"), 2-minute budget, with no E404). Conclusion: upstream registry latency into cold E2E containers. Not repo-fixable without changing test semantics, so I'm not opening a PR.

   Window: ci.yml, 2026-10-24 21:00Z to 2026-10-31 21:00Z, main and PRs, all attempts. 770 runs. I enumerated jobs for 387 of the 400 failed or retried runs: 40,325 completed Cli.EndToEnd-* jobs, 256 failed. Most green first-attempt runs skip CLI E2E (2 of 12 sampled ran it). That gives about 6.3k jobs in the 14 classes that run TypeScript aspire init.

   Classification of the 38 jobs that timed out waiting for Created apphost.mts:

   Cause	Jobs	Status
   npm E404/ETARGET on a freshly published version	29	fixed by #21615 (npm_config_before)
   Slow "Preparing Aspire server" (NuGet restore from nuget.org, cold /root/.nuget/packages): 99–106s instead of ~6s, then ~41s npm, finished at 141–149s	3	124521022816, 124617910212, 124617913801
   npm install still running after 144s (no lock file written)	1	122678762903
   "Preparing Aspire server" still running after 133s	1	124521019639
   Install phase, recording not attributable	3	122781136533, 124095196284, 124538123475
   "Preparing Aspire server", recording not attributable	1	124024646675

   Residual rate is 9 of about 6.3k TS-init jobs (about 0.14%). Every affected run passed on retry.

   Why this looks external:

   • The phases that stall are pure registry fetches: about 74 nuget.org packages, plus the npm dependency tree.
   • The slow cases cluster in time across concurrent jobs. Run 41194713391 had 2 classes stall at 11:08Z on 10-31; run 41227466919 had 3 classes at 18:16Z.
   • Passing baseline from 85 AllPublishMethodsBuildDockerImages recordings in sampled green runs:
     • restore + codegen: p50 6.4s, max 8.9s
     • npm install: p50 8.5s, p90 43s, max 78s

   Possible repo-side mitigations. These are not root-cause fixes and would need an infra decision:

   • Share or pre-warm the NuGet/npm caches across the E2E containers in a job. This changes the cold-restore experience the tests currently exercise.
   • Size a shared scaffold budget from data, like AspireRunStartupBudgetSeconds (Harden Daily CLI Smoke Tests: set an explicit AppHost startup budget for raw aspire run #18671). That would absorb the 141–149s slow tail but not the true stalls.

   I'm leaving both for an owner to decide, and I'm moving to another CLI E2E / install-infra flake.

   🤖 Posted by AspireAgent (AI Agent)
9. davidfowl commented on Oct 31, 2026

   

   David Fowler (davidfowl)

   on Oct 31, 2026 – with Copilot SWE Agent

   CollaboratorAuthor

   More actions

   Claiming the CLI Starter Validation flake: aspire-starter: aspire wait for resource apiservice failed with exit code 18 (WaitResourceFailed). apiservice and webfrontend go FailedToStart after the coordinated build __dotnet-project-build exits non-zero.

   Rate: 5 of 4,565 starter-validation jobs in failed or retried ci.yml runs from 10-24 to 10-31. The runs are 40937292091, 40958253916, 41167202562, 41197867795 and 41204248633, on Ubuntu ARM64 and macOS x64/ARM64. Each passed on rerun.

   Root cause (reproduced locally):

   1. cli-starter-validation.ps1 runs aspire run --detach, then aspire stop right away. That stop lands about 0.4 s after AppHost [main] Update dependencies from dotnet/arcade #1 started its coordinated dotnet build obj/.aspire/build/projects.<hash>.proj.
   2. aspire stop returns once the AppHost process exits. DCP cleans up afterwards, so the orphaned dotnet build keeps running. It survives SIGTERM: locally it ran to exit 0 after kill -TERM. I saw it alive in ps after stop returned.
   3. aspire start then launches AppHost Merging changes from internal repository #2's coordinated build of the same projects while the first build is still running. In CI run 41204248633, build [main] Update dependencies from dotnet/arcade #1 started at 13:28:30.3 and build Merging changes from internal repository #2 ran 13:28:35.5 → 40.5, then exited non-zero.
   4. Two overlapping builds of the same projects.<hash>.proj fail locally in 2 of 11 offset trials with error : The process cannot access the file 'Smoke.Web/obj/Debug/net10.0/compressed/<id>.gz' because it is being used by another process (StaticWebAssets compression).

   Fix direction: serialize coordinated builds across AppHost instances that share the same build directory. A new AppHost won't start its build while an earlier AppHost's build process for that directory is still alive. No sleeps or retries.

   #21621 (VS Code, Windows) is possibly related, but it has no stop/start, so I'm not claiming it.

   🤖 Posted by AspireAgent (AI Agent)
10. davidfowl commented on Nov 1, 2026

    

    David Fowler (davidfowl)

    on Nov 1, 2026 – with Copilot SWE Agent

    CollaboratorAuthor

    More actions

    ConfigSourceServiceGitTests.AcquireAndCheck_MatchRepositoryObjectFormat (Windows Cli; post-exit drain discards completed git output under thread-pool backlog; 2/307 Windows PR jobs, 0/56 main in 7d) | claimed: #21625, fix PR to follow.

    🤖 Posted by AspireAgent (AI Agent)
11. davidfowl commented on Nov 1, 2026

    

    David Fowler (davidfowl)

    on Nov 1, 2026 – with Copilot SWE Agent

    CollaboratorAuthor

    More actions

    Claiming [vscode] Aspire Node Functions debugger E2E > starts Core Tools, attaches pwa-node, hits an HTTP breakpoint and cleans up the worker (Top unowned #1; 24 occurrences on 17 branches, 5 on main in 7d). I found no open issue or PR that claims it. I'll post frequency, root cause and a fix PR here.

    Skipped higher on the 7d list: BeforeResourceStartedEvent_NotFiredForExplicitStartOnInitialCreation (#21026; fix #21360 merged and #21397 is open), MongoDB (#21594) and Azure Functions .NET (#21566).

    🤖 Posted by AspireAgent (AI Agent)
12. davidfowl commented on Nov 1, 2026

    

    David Fowler (davidfowl)

    on Nov 1, 2026 – with Copilot SWE Agent

    CollaboratorAuthor

    More actions

    Update on my earlier claim of [vscode] Aspire Node Functions debugger E2E > starts Core Tools, attaches pwa-node...: it is already fixed on main by #21327 (merged 10-28 16:07Z). Attempt-1 counts for the azure-functions-node shard:

    • main: 5 of 23 failed before the fix, 0 of 35 after.
    • PRs: 19 of 113 failed before, 5 of 260 after. All 5 post-fix PR failures are PR-specific (Stop persisting ambient NuGet policy into AppHost configs #21373 deterministic aspire new failure, a DCP-changing branch, Add browser-facing origin references for CORS and redirect configuration #21260).

    Releasing that claim.

    Other candidates skipped:

    • RunCommand_WhenAmbientDiscovery... and RepositoryTemplate_Resolves...: macOS test-host hang (Validate the stopped-child runtime fix and contain recurring macOS CLI test-host hangs #21259 / [Failing test] Cli macOS: multiple 30s timeouts (ProcessGuestLauncher, NewCommandLocalTemplate, RunCommand) #21335). No main hits since Restore macOS CLI tests with serialized collections #21389 restored the lane.
    • Java AddJavaContainer_ExportsTelemetryToAspire: fixed by Fall back to default container host name when DCP reports an empty one #21432.
    • TwoPassScanning_GeneratesWithEnvironmentOnTestRedisBuilder: deterministic snapshot mismatch on API-changing PR branches, not a flake.
    • ComputeRunArgumentsPopulatesRunPropertiesForSdkProject: open PR Fix flaky ComputeRunArgumentsPopulatesRunPropertiesForSdkProject (NuGet audit warning on stderr) #21502.
    • ProcessExecutionDetachedTests.StartAsync_OnWindows_ProvidesValidNullStdin: open PR Avoid Add-Type compile in detached Windows null-stdin CLI test #21549.

    Claiming GitCommandRunnerTests.ConfigureUnattendedSsh_VerifiesOpenSshAndAppliesBatchModeAfterUrlRewrites (Windows Cli). The test was added by #21388, and the first main run after that merge failed: job 124715227362.

    The TRX shows the first case failing after 5.43s, with its Git queries done in about 0.37s. That leaves roughly 5.0s, which matches the hard-coded 5s timeout on the ssh -V probe. A cold ssh.exe start then gets reported as "requires a verified OpenSSH executable" instead of relying on the 30s acquisition deadline. Fix PR coming.

    🤖 Posted by AspireAgent (AI Agent)
13. davidfowl commented on Nov 1, 2026

    

    David Fowler (davidfowl)

    on Nov 1, 2026 – with Copilot SWE Agent

    CollaboratorAuthor

    More actions

    Census of main attempt-1 failures (ci.yml push runs since 10-29 12:00Z), looking for an unowned flake:

    • Runs: 37 total. 25 failed, 10 cancelled, 1 succeeded, 1 still running.
    • Failed jobs: 97. 39 of them have a "runner lost communication" annotation (Pin Windows CI container runtime to Docker to stop DCP wslc probes (#21570) #21608/Windows CI runners lose communication during test steps since the windows-2025-vs2026 20261007 image (not test hangs) #21570).
    • Repeated test failures: all already owned.
      • TerminalHostSignalTests (3 hits): Keep terminal host shutdown connection open until the notification is delivered #21433
      • DotNetCliRunnerTests.ComputeRunArgumentsPopulatesRunPropertiesForSdkProject (2): Fix flaky ComputeRunArgumentsPopulatesRunPropertiesForSdkProject (NuGet audit warning on stderr) #21502
      • MauiBuildQueueTests.SingleResource_ShowsBuildingState (2): Fix MauiBuildQueueTests hang when a resource state is published after 5s #21492
      • ResourceCommandServiceTests: Treat DCP FailedToStart as stopped when stopping resources #21597
      • MetricsTests: Fix flaky MetricsTests exemplars dialog assertion #21596
      • GitCommandRunner ssh -V probe: already claimed
    • Single hits: each is owned, upstream, or infra.
      • DevTunnelMonitor ObjectDisposedException: Stop DevTunnel test monitor before disposing its host #21424
      • DotnetProjectBuildCoordinator and AuxiliaryBackchannel StopAsync cancellation: Don't fail AppHost stop on a no-op terminal teardown after the shutdown budget is exhausted #21528 family
      • DotnetProjectWithUrls "valid address": DCP SQLITE_BUSY, upstream
      • TerminalHostControl CancellationReachesServer: Fix lost cancellation signal in RPC cancellation tests #21430
      • Hosting-5 Windows exit -1073741502 (STATUS_DLL_INIT_FAILED): one occurrence, infra
      • NewCommandTemplateConfigPersistenceTests(aspire-java-starter) Windows 30s timeout: one occurrence. The log shows each embedded template file write taking 2–5s, i.e. runner I/O starvation. No code-level hang.

    Result: no unowned, repo-fixable main flake this pass. Not claiming anything.

    🤖 Posted by AspireAgent (AI Agent)
14. davidfowl commented on Nov 1, 2026

    

    David Fowler (davidfowl)

    on Nov 1, 2026 – with Copilot SWE Agent

    CollaboratorAuthor

    More actions

    ci-failure-cause backlog triage (ci.yml attempt-1, 2026-10-25 03:55Z → 2026-11-01 02:20Z: 516 decided runs, 59 main + 457 PR; 1,539 failed jobs, 1,070 with logs)

    • 168 open ci-failure-cause issues (excluding CI flaky-test burndown: driving flaky tests to 0 (live metrics) #21600).
    • Owned (skipped): 67 with an open cross-referencing PR or an assignee. Also owned: [CI Failure] Flaky: AzureFunctionsTests.AddAzureFunctionsProject_WorksWithMultipleProjects TaskCanceledException in DistributedApplication.StopAsync #21620→Wait for Functions host storage before stopping multi-project test #21445, [CI Failure] Flaky: TerminalHostSignalTests.TerminalHostControlShutdownUnlinksSockets throws TaskCanceledException on Windows #21409→Keep terminal host shutdown connection open until the notification is delivered #21433, [CI Failure] Flaky: AtsGoCodeGeneratorTests.GeneratedRuntime_PassesGoRuntimeHardeningTests goroutine count assertion on Windows #21619→Remove flaky Go goroutine-count test from GeneratedRuntime_PassesGoRuntimeHardeningTests #21474, the MongoDB family ([CI Failure] Flaky: MongoDbReplicaSetFunctionalTests.VerifyMongoDBMultiNodeReplicaWithDataShouldWorkAcrossUsages(topologyChange: None) fails to become healthy #20512), and VS Code shards ([CI Failure] VS Code extension E2E (Linux, apphost-lifecycle-tools) shard fails with long stalls and no Mocha failure details #21378/[CI Failure] Flaky: VS Code extension E2E (Windows, dynamic-debug-configuration) times out waiting for debug session to stop in duplicate-alias workspace scenario #20513/[CI Failure] Flaky: VS Code extension E2E (Linux, deno-debugger) times out waiting for Aspire AppHost debug startup #20839, mostly runner loss plus docs: latest Aspire standup report #21391/Give the Deno AppHost inspector attach the AppHost startup budget #21604).
    • 0 hits in the window: 59 unowned issues. 10 got close-candidate evidence comments this pass: [Main CI Failure] Main branch CI failure at a43c69baac2cc3f6e799a5d9813a5dd947c066e8 #21410, [CI Failure] Flaky: ChildProcessTests.Windows_GracefulTimeoutPreservesAppHostCleanupWorkerButReapsOwnedTrees throws when reading ExitCode on a process not started by the object #20871, [CI Failure] Flaky: TrayWatchStreamTests.StalledResourceWatchDoesNotBlockReconciliationOrQuit (removeFirst: False) asserts wrong reconciliation count #20772, [CI Failure] Flaky: TrayWatchStreamTests.StalledResourceWatchDoesNotBlockReconciliationOrQuit asserts wrong reconciliation count #20707, [CI Failure] Flaky: AzureServiceBusExtensionsTests.AzureServiceBusEmulatorResourceGeneratesConfigJsonOnlyChangedProperties times out during host shutdown (TaskCanceledException in DistributedApplication.StopAsync) #20227, [CI Failure] Flaky: DistributedApplicationPipelineTests.WithPipelineStepFactory_MultipleStepsAsyncOverload_ExecutesAllSteps fails with UnauthorizedAccessException during 'destroy' step on Windows #20078, [CI Failure] Flaky: Dashboard DashboardDataSourceTests.RunMode_DeletesOldestRunWhenLimitIsExceeded fails with UnauthorizedAccessException on Windows during metadata file move #19671, [CI Failure] Flaky: DashboardClientTests.SubscribeResources_ReplicaStartedAtChanged_EmitsParentChange sees duplicate resource change events #19515, [CI Failure] Flaky: AppHostSdkTargetsTests full-framework MSBuild test times out locating MSBuild via vswhere.exe #19517, [CI Failure] Flaky: ParentProcessLifetimeReusesResourcesAcrossAppRestartsAndStopsWhenParentExits fails due to DCP timestamp parsing BadRequest #18794.
    • Unmatchable: 14 generic "exit code N" issues.
    • Unowned and still recurring, by 7d job hits:
      • Runner loss: [CI Failure] Windows Package tests job fails: hosted runner lost communication with the server #20165 (197).
      • macOS Cli test-host hang family, tracked upstream in Validate the stopped-child runtime fix and contain recurring macOS CLI test-host hangs #21259: [CI Failure] Flaky: ChildProcessTests.Shutdown_UsesSharedGracefulBudgetAndEscalates(signalFailure: True) times out on macOS under CPU contention #21007/[CI Failure] Flaky: ChildProcessTests.Shutdown_UsesSharedGracefulBudgetAndEscalates times out on macOS under CPU contention #20938 (12), [CI Failure] Flaky: ProcessGuestLauncherTests.LaunchAsync_NoOptions_OnCancellation_ForceKillsProcessTreeAndReturns times out on macOS #20915 (2). 0 hits since 10-29.
      • Windows 0xC0000142 post-test: [CI Failure] Windows process initialization failure (exit code -1073741502 / 0xC0000142) in post-test steps #18720 (6; 3 main).
      • Network/feed infra: [CI Failure] NPM registry rate limiting (HTTP 429 Too Many Requests) from dnceng feed #18691 (5), [CI Failure] dotnet tool restore fails with 'Connection reset by peer' during arcade tool restore #19168, [CI Failure] dnceng NuGet feed returns HTTP 503 Service Unavailable during package restore #18994, [CI Failure] Set up .NET Core step fails with 'read ECONNRESET' network error #20028, [CI Failure] GitHub artifact upload fails with ETIMEDOUT network timeout #19876, [CI Failure] .NET install script download fails with SSL/TLS authentication error from builds.dotnet.microsoft.com on Windows runner #18821, [CI Failure] GitHub artifact upload fails with ENOTFOUND DNS resolution error #19252, [CI Failure] GitHub artifact upload fails with ECONNRESET during FinalizeArtifact #20982, [CI Failure] Polyglot SDK Validation: Java SDK Validation job fails building validation image with exit code 1, unrelated to PR changes; job logs unavailable for detailed diagnosis #19452, [CI Failure] Windows runner temp file command path inaccessible: 'A device which does not exist was specified' #19002 (1–4 each).
      • Linux SIGABRT post-test: [CI Failure] Azure.Search.Documents (ubuntu-latest) test run crashes with exit code 134 (SIGABRT), triggering hang dump check #20137 (1).
      • Upstream DCP: [CI Failure] Flaky: DenoFunctionalTests.VerifyDenoAppDirectExecutionWorks fixture InitializeAsync fails with 'denoscript' endpoint not allocated #20760 (11).
      • PR-caused only: [CI Failure] Flaky: TestingFactoryTests.HttpClientGetTest times out intermittently #19456 (4).
    • No unowned, recurring, repo-fixable item found.
      • [CI Failure] Windows process initialization failure (exit code -1073741502 / 0xC0000142) in post-test steps #18720 is the closest candidate. In every occurrence, "Run tests (Windows)" passes. Then every later process launch fails immediately with STATUS_DLL_INIT_FAILED, including the runner's own node actions (Post Checkout). That points to session-wide process-creation failure on the hosted runner, not test code.
      • The leftover Heartbeat process is a possible contributor but unproven. 3 of the 6 hits were on main.

    🤖 Posted by AspireAgent (AI Agent)
15. 5 remaining items

    Load more

16. davidfowl commented on Nov 1, 2026

    

    David Fowler (davidfowl)

    on Nov 1, 2026 – with Copilot SWE Agent

    CollaboratorAuthor

    More actions

    Claiming the Polyglot SDK Validation – Nuxt dev-mode health-wait hang (TypeScript SDK Validation, playground/NuxtApp/verify.mjs apphost).

    Signature: aspire start succeeds, then verify.mjs waits the full 240s and fails with "Nuxt and the backend did not become healthy within four minutes." It prints no resource state or logs, so the cause can't be proven from existing CI output. Examples: 124635347952 and 124524439865. In 40 of 40 sampled passing jobs the same wait took 32–46s, so this is a hang, not a slow tail.

    Plan: a diagnostics-only draft PR that dumps aspire describe --include-hidden state (including the hidden *-installer resources) and per-resource aspire logs when the wait times out, so the next occurrence is root-causable. No timeout bumps or retries. Related: #21146.

    🤖 Posted by AspireAgent (AI Agent)
17. davidfowl commented on Nov 1, 2026

    

    David Fowler (davidfowl)

    on Nov 1, 2026 – with Copilot SWE Agent

    CollaboratorAuthor

    More actions

    Claiming Windows CI resource telemetry for the runner-loss and hang family (#21570). This adds no fix to the tests themselves. It complements #21608 (Docker pin) and #21489 (extension E2E timing); neither of those adds machine telemetry.

    A draft PR is coming. It adds a below-normal-priority background sampler to Windows .NET test jobs and Windows extension E2E shards. Every 20s it records CPU, processor queue, available memory, commit charge, page file, paging, disk queue, free disk, and the top processes. The output goes into the existing if: always() artifacts, plus a step summary. It doesn't change test behavior, timeouts or retries.

    Baseline from today's census: on main, all 20 Windows jobs that ran 45 minutes or longer in the last 16 ci.yml runs (out of 1,551 Windows jobs) were runner-lost, with no logs (BlobNotFound). So the existing console Heartbeat never survives the jobs we care about. Surviving Windows Hosting jobs show 16 GB RAM at 20–23% used, CPU at 60–80%, and C: about 29 GB free.

    🤖 Posted by AspireAgent (AI Agent)
18. davidfowl commented on Nov 1, 2026

    

    David Fowler (davidfowl)

    on Nov 1, 2026 – with Copilot SWE Agent

    CollaboratorAuthor

    More actions

    Triage of two entries from the 08:28Z "top unowned" ranking. Neither needs a fix PR. Window: ci.yml attempt-1, 824 runs since 2026-10-25 09:00Z; skipped and cancelled jobs are excluded.

    [vscode] Aspire Node Functions debugger E2E > starts Core Tools, attaches pwa-node…: fixed by #21327 (merged 2026-10-28 16:07Z). Results for the VS Code extension E2E (Linux, azure-functions-node) job:

    	before #21327	after #21327
    main	5 / 23	0 / 37
    PR	19 / 113	5 / 277

    The 5 post-fix PR failures are not the old race:

    • 3 are on danegsta-nuget-mapping-regression: aspire new … --version 17.0.0-pr.21373… fails deterministically (123106717674).
    • 1 is on aspireagent/fix-21465-dcp-session-folder and 1 on danegsta-browser-endpoint-resolution. Both fail with "Failed to apply E2E control revision …: E2E control command failed" (124203484416, 124390529919). Both branches change DCP or endpoint resolution, and the shard passed on their other runs.

    ProcessInvocationOptionsTests.Clone_CopiesEveryProperty: deterministic and PR-specific, not a flake. All 23 failing attempt-1 jobs, across 7 runs, come from mitchdenny-aspire-test-lifetime. That branch adds ProcessInvocationOptions.DisableExtensionLaunch but doesn't give it a non-default value in the reflection test, so it fails with "Property 'DisableExtensionLaunch' was not set to a non-default value in this test" (124794863774). It has 0 hits on main or any other branch.

    DotNetCliRunnerTests.ComputeRunArgumentsPopulatesRunPropertiesForSdkProject is already claimed by open PR #21502.

    🤖 Posted by AspireAgent (AI Agent)
19. davidfowl commented on Nov 1, 2026

    

    David Fowler (davidfowl)

    on Nov 1, 2026 – with Copilot SWE Agent

    CollaboratorAuthor

    More actions

    Claim: #19992, the Infrastructure.Tests pwsh crash (FileLoadException: The given assembly name was invalid … System.Collections.Specialized, exit 134).

    • 7-day ci.yml attempt-1 hits: main 11-01 (run 41266341036); PR 10-28 (40615677324) and 10-30 (41019908161, same signal-6 createdump crash). They span 3 unrelated branches. Infrastructure (ubuntu) jobs over the window: main 3/63 failed, PR 6/161.
    • Root cause (from the CI crash dump and a local repro): concurrent pwsh processes share ~/.cache/powershell/StartupProfileData-NonInteractive (the multicore-JIT profile; upstream Parallel Powershell Sessions Lead to Corruption of the StartUpProfileData Cache PowerShell/PowerShell#26528, Parallel Powershell sessions lead to corruption of the StartUpProfileData cache created by ProfileOptimization.StartProfile dotnet/runtime#121977, both still open). Local pwsh 7.6.9 (same runtime commit as CI) with 8 parallel processes reproduced the exact crash in 24 of 408 runs.
    • I'm working on a test-side fix in Infrastructure.Tests now.

    🤖 Posted by AspireAgent (AI Agent)
20. davidfowl commented on Nov 1, 2026

    

    David Fowler (davidfowl)

    on Nov 1, 2026 – with Copilot SWE Agent

    CollaboratorAuthor

    More actions

    Claiming the quarantined test KubernetesDeployWithNatsTests.DeployK8sWithNats (#15789) for the quarantined-tests scope.

    • Quarantine lane (tests-quarantine.yml, Cli.EndToEnd ubuntu, last 14 days): it failed in 155 of 155 scheduled runs (0% pass). Every failure is Step 'prepare-deployment-targets-env' failed: Command line args must be strings.
    • Root cause: KubernetesResource.ProcessArgumentsAsync resolves container args in non-embedded mode. A top-level ParameterResource, or a single-value ReferenceExpression such as NATS --user {UserNameReference} / --pass {PasswordParameter}, resolves to a HelmValue instead of a string, so the publisher throws.
    • Plan: render these args as Helm value expressions, using the same embedded-parameter path already used for multi-value args such as --cs "Url={{ .Values.config... }}". Then add publisher tests and remove the quarantine attribute. Draft PR to follow.

    🤖 Posted by AspireAgent (AI Agent)
21. davidfowl commented on Nov 1, 2026

    

    David Fowler (davidfowl)

    on Nov 1, 2026 – with Copilot SWE Agent

    CollaboratorAuthor

    More actions

    Claiming an unquarantine of 3 quarantined tests that have passed 100% in the quarantine lane for 14 days (quarantined-tests scope):

    • LogLevelTests.LogLevelTrace_ProducesTraceEntriesInCliLogFile (Flaky: ResourceCommand_FailsWhenInteractionServiceIsRequired times out on aspire stop with DCP container still running #17485, already closed as fixed by Fix flaky ResourceCommand_FailsWhenInteractionServiceIsRequired test #18396): 162/162 passed.
    • ResourceCommandTests.ResourceCommand_FailsWhenInteractionServiceIsRequired (Flaky: ResourceCommand_FailsWhenInteractionServiceIsRequired times out on aspire stop with DCP container still running #17485): 162/162 passed.
    • SlimTestProgramTests.TestPortOnEndpointAnnotationAndAllocatedEndpointAnnotationMatchForReplicatedServices (Flaky test: Aspire.Hosting.Tests.SlimTestProgramTests.TestPortOnEndpointAnnotationAndAllocatedEndpointAnnotationMatchForReplicatedServices #9673): 316/316 passed (Linux and Windows).

    Source: tests-quarantine.yml scheduled runs from 2026-10-18 to 2026-11-01, using each run's Final Results table.

    I'm not touching #15789 (owned by #21648) or any assigned or open-PR quarantine issues. The remaining unowned failing quarantined tests are ExecutableExitsImmediately (#16189, every failure is the DCP state-store SQLITE_BUSY error, which is in the excluded DCP family) and #19517 (already a closure candidate).

    🤖 Posted by AspireAgent (AI Agent)
22. davidfowl commented on Nov 1, 2026

    

    David Fowler (davidfowl)

    on Nov 1, 2026 – with Copilot SWE Agent

    CollaboratorAuthor

    More actions

    Claim (AspireAgent, worker for David Fowler (@davidfowl)): SecretDotNetAppHost secret set silent hang. Tracking issue: #21650 (2 of 284 PR jobs over 7 days, 2 unrelated branches; main doesn't run this lane).

    The root cause can't be proven from the current artifacts, because the hung-command failure path captures no CLI logs. I'm opening a draft diagnostics PR so that hung CLI E2E commands capture a process snapshot and ~/.aspire/logs. It's disjoint from DCP, TerminalHost, runner loss, K8s NATS, MongoDB, pwsh and AzureFunctions.

    🤖 Posted by AspireAgent (AI Agent)
23. davidfowl commented on Nov 1, 2026

    

    David Fowler (davidfowl)

    on Nov 1, 2026 – with Copilot SWE Agent

    CollaboratorAuthor

    More actions

    Claiming KubernetesDeployTypeScriptWithPersistentVolumeTests.DeployTypeScriptK8sWithPersistentVolumeSurvivesPodRestart, tracked as #21652. Over 7 days of ci.yml attempt-1 runs it failed in 0 of 62 main runs and 4 of 482 PR runs.

    • 1 real flake (sebros/dashboard-modern-layout): kubectl wait --for=condition=Ready pod app-statefulset-0 runs before the StatefulSet controller has recreated the pod, so it fails immediately with NotFound. The two C# PV durability tests have the same race. A fix PR is coming: wait for creation (kubectl wait --for=create) before waiting for Ready.
    • 3 PR-caused (dependabot npm_and_yarn branches): the dependabot lockfile for the ts-starter frontend resolves 17 packages to the internal dnceng npm feed, and the Docker npm ci then gets a 401. Details are in Flaky: KubernetesDeployTypeScriptWithPersistentVolumeTests fails with NotFound when waiting for the recreated StatefulSet pod #21652.

    🤖 Posted by AspireAgent (AI Agent)
24. davidfowl commented on Nov 1, 2026

    

    David Fowler (davidfowl)

    on Nov 1, 2026 – with Copilot SWE Agent

    CollaboratorAuthor

    More actions

    Claiming unquarantine batch 2. It's separate from #21649 and #21648. Each test passed 100% in tests-quarantine.yml scheduled runs from 2026-10-18 to 2026-11-01 (at least 100 results each):

    • SlimTestProgramTests.TestProjectStartsAndStopsCleanly (Flaky test: Aspire.Hosting.Tests.SlimTestProgramTests.TestProjectStartsAndStopsCleanly #9672): 316/316 on Linux and Windows.
    • StopNonInteractiveTests.StopAllAppHostsFromUnrelatedDirectory (Flaky: StopNonInteractiveTests E2E timeouts #16643): 162/162.
    • StopNonInteractiveTests.StopNonInteractiveMultipleAppHostsShowsError (Flaky: StopNonInteractiveTests E2E timeouts #16643): 162/162.

    I'm leaving the other candidates alone:

    • [Failing test]: Aspire.Cli.EndToEnd.Tests.PersistentContainerEndToEndTests.PersistentContainersPreserveDataAcrossAppHostRuns #17995 also qualifies (162/162) but is over the 3-test cap.
    • [Failing test]: Aspire.Cli.EndToEnd.Tests.StartStopTests.CreateStartAndStopAspireProject #16191, Flaky test: Aspire.Hosting.Tests.DistributedApplicationTests.ProxylessEndpointWorks #8728, [Failing test]: Aspire.Hosting.Tests.DistributedApplicationTests.TestServicesWithMultipleReplicas #9340 and Flaky test: VerifyYarpResourceExtensionsConfig - backend resource fails to become healthy on ubuntu-latest (DCP managedFields error) #17020 are assigned.
    • Flaky test: Aspire.Hosting.Tests.SlimTestProgramTests.TestPortOnEndpointAnnotationAndAllocatedEndpointAnnotationMatch #9671 is assigned and mentioned in open PR feat(ci): investigate flaky-test issues from an issue URL #21253.

    🤖 Posted by AspireAgent (AI Agent)
25. davidfowl commented on Nov 1, 2026

    

    David Fowler (davidfowl)

    on Nov 1, 2026 – with Copilot SWE Agent

    CollaboratorAuthor

    More actions

    Claim: Aspire.Hosting.Testing.Tests.DashboardUrlTests.DashboardDisplaysResourceUpdatesWhileAppHostIsPausedAtBreakpoint (outerloop)

    • Recurrence: this test failed in 2 of the 14 scheduled Outerloop runs on main in the last 14 days (41158473015 on 10-31 and 41263841334 on 11-01, per Test failures: Outerloop Tests #18657). Those are the only 2 runs after Modernize the dashboard experience #21412 ("Modernize the dashboard experience", merged 2026-10-30 04:47Z). The 12 runs before Modernize the dashboard experience #21412 all passed.
    • Cause (from logs): Playwright times out waiting for tr.resource-row. The aria snapshot shows the new default Cards layout, and the card already shows dashboard-test-resource … State Dashboard test initial. The product behaves correctly; the test's grid-row locator no longer matches the default UI.
    • Owner: none. This test isn't mentioned in any issue or PR other than the Test failures: Outerloop Tests #18657 failure log, and none of the excluded families cover it.

    I'm taking this one: reproduce locally, update the locator to the current dashboard markup without weakening the state assertions, then open a draft PR.

    🤖 Posted by AspireAgent (AI Agent)
26. davidfowl commented on Nov 1, 2026

    

    David Fowler (davidfowl)

    on Nov 1, 2026 – with Copilot SWE Agent

    CollaboratorAuthor

    More actions

    Claiming the dnceng npm 429 during corepack yarn install infra flake in Run VS Code extension unit tests (Windows), tracked as #18691 (open, unassigned, no linked PR).

    • Window: ci.yml attempt-1, 2026-10-25T13:15Z → 2026-11-01T13:23Z (840 runs).
    • Hits: 5 failed jobs contain Request failed "429 Too Many Requests" from pkgs.dev.azure.com/dnceng/.../dotnet-public-npm.
      • 3 in Run VS Code extension unit tests (Windows), all at the bare Install dependencies step. That's 3 of 315 decided jobs (main 62, PR 313) and 3 of the lane's 12 failures.
      • 1 in the bootstrap job, which deliberately runs on a cold cache. I'm leaving that one as is.
      • 1 in the E2E job, where the existing bounded 3-attempt install retry recovered: attempt 2 finished with Done in 6.76s, and that job failed later, at the test step.
    • Plan: give the unit-test Install dependencies step the same bounded install retry that extension-e2e-tests.yml already uses. Yarn 1.22 retries only 408 and 5xx, not 429. Tests are not retried.

    🤖 Posted by AspireAgent (AI Agent)
27. davidfowl commented on Nov 1, 2026

    

    David Fowler (davidfowl)

    on Nov 1, 2026 – with Copilot SWE Agent

    CollaboratorAuthor

    More actions

    Claiming the Dependabot npm lockfile E401 cause: #21659.

    Dependabot npm_and_yarn PRs inherit the repo-root dnceng .npmrc, because Dependabot looks for .npmrc in parent directories. They write dnceng resolved URLs into template and E2E-fixture lockfiles that must use the public registry.

    Over 7 days, 4 of 5 Dependabot npm PRs failed attempt 1, with 7–18 jobs failed per run: Cli.Tests on all OSes, Starter Validation, K8s TS / PV E2E, and JavaScriptPublish. This accounts for 3 of the 4 PV-test failures in #21652.

    A fix PR (per-directory .npmrc pinning npmjs, excluded from template payloads) is coming as a draft. It doesn't overlap with #18691 or #21657 (429 retry).

    🤖 Posted by AspireAgent (AI Agent)
28. davidfowl commented on Nov 1, 2026

    

    David Fowler (davidfowl)

    on Nov 1, 2026 – with Copilot SWE Agent

    CollaboratorAuthor

    More actions

    Claim (AspireAgent, worker for David Fowler (@davidfowl)): GitHub artifact service network failures are never auto-rerun (#19876 ETIMEDOUT, #19252 ENOTFOUND).

    7-day ci.yml attempt-1 census (2026-10-25 13:15Z → 2026-11-01 11:50Z; 840 runs, 80 main + 759 PR): 7 runs out of 840 failed only because an actions/upload-artifact / download-artifact step hit Failed to {Create,Finalize,List}Artifact: Unable to make request: ETIMEDOUT|ENOTFOUND|ECONNRESET. That's 5 PR runs on 5 unrelated branches out of 759, and 2 main runs out of 80. Affected steps: Upload RID-specific packages, Upload CLI npm packages, Upload starter validation diagnostics, Download RID-specific NuGet packages and Download VSIX.

    Root cause (repo-side): classifyFailedJob in .github/workflows/auto-rerun-transient-ci-failures.js inspects annotations only for steps listed in retryableWithAnnotationStepPatterns. None of these artifact steps are listed, so all 7 jobs return retryable: false ("not covered by the retry-safe rerun rules"). I reproduced this by running the main classifier on the real job and annotation payloads. ETIMEDOUT also isn't in transientAnnotationPatterns.

    Planned fix: a draft PR that makes artifact-service transport failures rerunnable only when every failed step is an Upload/Download artifact step. Test steps are never rerun or skipped, and no assertions change. Regression tests use the real annotation strings.

    🤖 Posted by AspireAgent (AI Agent)
29. davidfowl commented on Nov 1, 2026

    

    David Fowler (davidfowl)

    on Nov 1, 2026 – with Copilot SWE Agent

    CollaboratorAuthor

    More actions

    Claiming #17995 (PersistentContainerEndToEndTests.PersistentContainersPreserveDataAcrossAppHostRuns) for an unquarantine PR. The 12:49Z batch-2 claim above left it out because of the 3-test cap.

    Quarantine-lane evidence (tests-quarantine.yml, scheduled on main, 2026-10-18 → 2026-11-01): 151/151 Cli.EndToEnd (ubuntu-latest) jobs ran it. There were 0 failures and 0 skips; each run produced a .cast recording. I parsed the per-job MTP failed … lines, not the Final Results table.

    Why nothing else qualified (7-day scheduled Outerloop + 14-day quarantine-lane per-test log scan of 2,421 jobs):

    • Outerloop: Templates Redis 6/6 ([automated] test(templates): Fix child-project builds and endpoint discovery #21423), Radius 6/6 (MongoDB), McpDocs 4/6 (Build the CLI before MCP initialization in McpDocsE2ETests #21450), DashboardUrl 2/6 (Target dashboard resource cards in DashboardUrlTests breakpoint test #21656), Blazor gateway 1/6 (not recurring).
    • Quarantine lane:
      • ProcessGuestLauncherTests…BlockingSignaler…: ~100% failure until 10-31 14:02Z. Already fixed by Cancel blocked graceful signaler once the child process exits #21581.
      • RpcCancellationReachesServer ([CI Failure] Flaky: BackchannelJsonSerializerContextTests.RpcCancellationReachesServer times out after 30s on Linux #20666), OpenAI ([Failing test]: DependentResourceWaitsForOpenAIModelResourceWithHealthCheckToBeHealthy #10977) and StartResourceForcesStart ([Failing test]: Aspire.Hosting.Tests.DistributedApplicationTests.StartResourceForcesStart #15777) are assigned.
      • ExecutableExitsImmediately ([Failing test]: Aspire.Hosting.Tests.ExecutableResourceFailureLoggingTests.ExecutableExitsImmediately #16189): 8/148 Windows, all DCP SQLITE_BUSY ([CI Failure] Flaky: MySqlFunctionalTests.WithDataShouldPersistStateBetweenUsages times out because DCP state.sqlite3 is locked (SQLITE_BUSY) #20554).
      • AppHostSdkTargetsTests vswhere ([CI Failure] Flaky: AppHostSdkTargetsTests full-framework MSBuild test times out locating MSBuild via vswhere.exe #19517): 5/148 Windows, last on 10-30. It's Windows full-framework MSBuild only, so it isn't reproducible here.

    Heads-up for #21649 / #21653 (SlimTestProgramTests #9673 / #9672): these are not 100% in the lane. All three SlimTestProgram tests failed together in 5 of about 299 Hosting jobs. Four were Windows (148 jobs): runs 39720421729, 40591739820, 40804236902 and 41271567232 (2026-11-01 04:37Z). One was ubuntu (151 jobs): run 39208460905. The signature is Resource '__dotnet-project-build' failed to meet the predicate condition before the operation was cancelled. Quarantine-lane jobs report success even when tests fail, and the Final Results table appears to miss these.

    🤖 Posted by AspireAgent (AI Agent)
30. David Fowler (davidfowl)

    added a commit that references this issue on Nov 1, 2026

    Give RemoteHost tests thread-pool headroom above xunit's synchronous …

    ...

    9ea245a