Skip to content

ci(mutation): Stryker.NET 5 with the MTP runner, shards sized by mutant count, zero-kill and missing-shard guards - #1713

Draft
dlrivada wants to merge 38 commits into
mainfrom
ci/stryker5-mtp-1441
Draft

dlrivada wants to merge 38 commits into
mainfrom
ci/stryker5-mtp-1441

Conversation

@dlrivada

@dlrivada dlrivada commented Oct 3, 2026 •

Copy link
Copy Markdown
Owner

Part of #1441 (phase 1), #1440 and #1682. The closing keywords, and the knowledge records for #1440 and #1682, are added in phase 2, once the CI validation run proves that mutants are killed and that a missing shard is reported.

Summary

Under xUnit v3 the VsTest runner of Stryker.NET 4.x killed 0 mutants (#1440), per-folder test-case-filter was the only thing keeping shards inside their timeouts, and shard 14 could lose its runner and upload nothing while the run stayed green (#1682). The Mutation Tests workflow now runs Stryker.NET 5.0.0 with the Microsoft Testing Platform (MTP) runner and fails loudly when a shard proves nothing.

Verification

  • Local smoke run (one file, SequentialDispatchStrategy.cs): 22,069 tests found, 4 mutants tested, 4 killed, score 100 %; 4 mutants in 92 s at concurrency 2.
  • Exclusion-glob probe: **/Dispatchers/Strategies/*.cs plus ! entries selected only the remaining file's 4 mutants (the remainder-shard design works).
  • actionlint clean on mutation-tests.yml; run-stryker.cs builds with no warnings.
  • Self-review (adversarial-reviewer): simulated the 20 shards against every file in src/Encina: no gap or overlap against the old 17 folders; counts and timeouts match. Four minors fixed (step-log readings, zero-kill guard only in matrix mode, configuration message, a docs claim).
  • Pending (phase 2, before merge): calibration run 37130291929 (custom_scope=**/Dispatchers/Strategies/*.cs) measures the CI time per mutant (local figure is from 32 cores; runners have 4 vCPU). Then the full 20-shard validation run, re-sized TIMEOUTS (max(60, ceil(minutes × 1.5))), and the record and docs updated with the observed figures.

Cross-cutting checklist (ADR-018)

Not applicable: CI and test tooling only; no entity, store, behavior, service or external integration in src/.

…de from Encina.UnitTests (#1441)

Bump dotnet-stryker to 5.0.0, set test-runner mtp and project Encina.csproj in stryker-config.json (no solution, test-projects or test-case-filter keys), run Stryker from tests/Encina.UnitTests, and stop passing --log-to-file by default because the MTP runner's trace JSON-RPC logs grow by about 40 MB per test run.
…d missing and zero-kill shards (#1441, #1440, #1682)

Replace FOLDERS/FILTERS with SHARDS of ';'-separated mutate globs (20 shards of at most about 150 tested mutants, large folders split by file with an exclusion-based remainder shard), set provisional TIMEOUTS from the #1395 formula, drop the inert test-case-filter patching, build only Encina.UnitTests, log runner memory and disk, upload reports even when a step fails, fail a shard and the aggregate when a report tested mutants but killed none, and warn in the summary when a shard report is missing.
…l only in matrix mode, forward --configuration only when given (#1441)
…er-resources step log and optional --configuration (#1441)
Copilot AI balanced review requested due to automatic review settings October 3, 2026 14:41

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@coderabbitai

coderabbitai Bot commented Oct 3, 2026

Copy link
Copy Markdown
Contributor

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

  • Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true
  • Autopilot · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

Copilot AI balanced review requested due to automatic review settings October 3, 2026 14:46

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@dlrivada

dlrivada commented Oct 3, 2026

Copy link
Copy Markdown
Owner Author

Review of PR #1713 (phase 1): merge after fixes and the validation run

Multi-dimension review with adversarial verification: each finding was checked by two independent skeptics. CONFIRMED = both found it real; PLAUSIBLE = one did. Refuted findings are listed at the end.

Confirmed (5)

Plausible (3)

  • minor [workflow] .github/workflows/mutation-tests.yml:22: Workflow-level permissions block (pre-existing on origin/main) contradicts AGENTS.md section 10. A top-level permissions: contents: read exists. It is not new: git show origin/main:.github/workflows/mutation-tests.yml has the same block. AGENTS.md section 10 says permissions are declared per job, never at workflow level. The PR rewrites most of this file and leaves the block in place. No job here needs more than contents: read, so the move is mechanical. Fix: Move permissions: contents: read into each of the select-matrix, test-baseline, run-mutation-tests and aggregate jobs, or record it as a known deviation.
  • nit [stryker] .github/scripts/run-stryker.cs:57: Pass-through arguments with relative paths now resolve against tests/Encina.UnitTests, undocumented. Stryker is now started with working directory tests/Encina.UnitTests (run-stryker.cs:57-59), but the script still resets its own cwd to the repo root. A relative path in a pass-through argument (for example --output, --reporter file paths, --baseline paths) resolves against the test project directory, not the repo root. --mutate globs are unaffected (relative to the mutated project) and the script passes absolute paths for --config-file and --output. The docs say to run the command from the repository root, which could suggest repo-root-relative paths. No current call site is affected. Fix: Add one sentence to MUTATION_TESTING.md and to the script comment saying that extra path arguments are relative to tests/Encina.UnitTests, or tell users to pass absolute paths.
  • minor [docs] docs/testing/mutation-measurement-methodology.md:100: Timing and kill figures from a local smoke run are hand-typed and unbacked; the PR body and docs disagree on its duration. The methodology page (Smoke check section) and MUTATION_TESTING.md state 'tested 4 mutants, killed all 4, in 8 min 41 s (initial test run about 4 min 40 s)'. The PR body says '4 mutants in 92 s at concurrency 2'. 8:41 minus 4:40 leaves about 241 s for the mutant phase, not 92 s. The figures are not reproducible from anything in the repo or CI (no artifact, no mutref marker), and 'killed 4 of 4' is a mutation result typed by hand (SPEC-001). The '22,069 tests' figure is also unbacked, and differs from the 21,091 of spike [SPIKE] Evaluate Stryker.NET 5.0.0 MTP runner with perTest coverage on one mutation shard #1087 without saying why. The page also states ubuntu-latest has '4 vCPU and 16 GB' with no source. Fix: Reconcile the two durations (state what 92 s measures) or drop it from the PR body. Label the figures as a dated local observation, with no score wording, or replace them with the calibration run's numbers once run 37130291929 completes. Cite a GitHub doc URL for the runner size.

Refuted (11)

  • [workflow] Provisional TIMEOUTS rest on a local figure that the repo's own data says is optimistic for CI (Class C)
  • [workflow] Resource monitor keeps the step's stdout open after the trap kills it
  • [workflow] Zero-kill guard is blind to a shard whose mutants are all RuntimeError, and is skipped in custom and diff mode
  • [shards] TIMEOUTS rest on a per-mutant figure that the spike itself marks as inflated by flaky kills, and on a 20-minute setup guess measured on a 32-core machine
  • [shards] The 20 shards still cover only 125 of the 285 mutable files of src/Encina; the PR text does not say so
  • [shards] Comment format for combined shards (5, 6, 11) is a sum expression, not a single count
  • [stryker] Dropping --log-to-file leaves CI with only info-level console output for failed shards
  • [docs] Knowledge record fails knowledge-records --check (11 errors); the required knowledge-records CI job is already red on this PR
  • [docs] Fixes #1440 and Fixes #1682 are not yet justified; the evidence is a pending run
  • [docs] Guide tells readers to 'run from the repository root' because the script runs Stryker from tests/Encina.UnitTests; the script already resolves the root itself
  • [docs] Timeouts section is provisional but the PR body cites 20 shards 'of about 150 tested mutants or fewer' while 3 shards exceed it

Copilot AI balanced review requested due to automatic review settings October 3, 2026 14:53

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

Stryker 5.0.0 under-reports kills with the MTP runner above concurrency 1 (stryker-net#3832), and each slot keeps a multi-GB test host alive. The value lives in the config because run-stryker.cs reads -c as --configuration.
…tant merge for span-split files (#1441)

The Run mutation tests step caps the managed heap at 4 GiB (DOTNET_GCHeapHardLimit), raises its own OOM score and runs Stryker under nice so a memory or CPU squeeze takes Stryker or a test host instead of the runner agent, and tees Stryker's console to stryker-console.log with pipefail. A monitor publishes runner readings every minute (every PUBLISH_EVERY minutes on the GitHub API, sized to the shard count) to a 'mutation telemetry (shard N)' check run that survives a lost runner; a final step and the aggregate job complete it.

SHARDS can split one file with Stryker's character-offset span suffix; Build matrix rejects a span glob without its twin, and the aggregate merges reports mutant by mutant, drops empty and 'Removed by mutate filter' placeholders, and fails when two reports hold the same mutant. SHARDS and TIMEOUTS stay marked pending the phase-2 measurement.
… splits and per-mutant merge of the mutation workflow (#1441)
…nd slower telemetry updates (#1441)

A shard whose Stryker console reports an OutOfMemoryException fails and does not upload its report, since a test failing on the heap cap may count as a kill; the visible-failure effect of the cap is marked as a hypothesis for the calibration runs. The check-run token no longer reaches Stryker or the test hosts, every gh call times out, check-run updates stay near 360 per hour for the whole matrix, a failed oom_score_adj write only warns, and the readings also count processes naming Encina.UnitTests.
…ration; document the OOM guard, publish rate, token handling and process counts (#1441)
Copilot AI balanced review requested due to automatic review settings October 3, 2026 19:51
Copilot AI balanced review requested due to automatic review settings October 5, 2026 10:50

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

…erver (#1441 phase 2f)

Phase 2e measured that glibc's dynamic mmap threshold kept the native memory freed after each run of the ABAC EEL tests, so the reused test server grew ~2.5 GB per run. Fixed 128 KiB thresholds cut the first-run peak from 7.6 GB to 4.9 GB and the growth to ~230 MB per run with no slowdown.
…during Stryker runs (#1441 phase 2f)

Adds an MTP TestingPlatformBuilderHook to Encina.UnitTests that registers an
ITestSessionLifetimeHandler only when STRYKER_MUTANT_FILE is set. At the start
of a test session after the first one in the process, it kills the test
server when its private memory is above ENCINA_MTP_RECYCLE_MB (default
6144 MB) and writes one [encina-mtp-recycle] line to stderr. Stryker 5.0.0
treats the lost connection as a crashed host, discards the server and reruns
the same mutant on a fresh one (RunAssemblyTestsInternalAsync, two attempts),
so the verdict comes from the second attempt. The first session of a process
never recycles, so the retry cannot fail because of the hook.
#1441 phase 2f)

Review fixes for the MTP test server recycler:
- Stryker 5.0.0 discards the test server's stderr unless --log-to-file is set,
  so the recycle line is also appended to the file named by
  ENCINA_MTP_RECYCLE_LOG when that variable is set.
- ENCINA_MTP_RECYCLE_MB values too large to express in bytes are capped
  instead of overflowing into a negative threshold that recycles every session.
- The builder hook moves to its own file and takes the environment reader as
  a parameter, so tests prove it registers nothing without STRYKER_MUTANT_FILE.
Copilot AI balanced review requested due to automatic review settings October 5, 2026 20:39

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@dlrivada

dlrivada commented Oct 5, 2026

Copy link
Copy Markdown
Owner Author

#1441 phase 2f, ready for verification. Branch head 28dbfad8.

Changes:

  • MALLOC_MMAP_THRESHOLD_ and MALLOC_TRIM_THRESHOLD_ are fixed at 128 KiB.
  • New test-server recycler, StrykerServerRecycler:
    • it is active only when STRYKER_MUTANT_FILE is set;
    • at the start of a mutant's session it kills its own process if private memory is above ENCINA_MTP_RECYCLE_MB (default 6144). It never does this on a process's first session;
    • Stryker 5.0.0 then retries the mutant on a fresh server (MicrosoftTestingPlatformRunner.cs#L1184-L1244);
    • 27 tests cover it.
  • Every recycle is logged to ENCINA_MTP_RECYCLE_LOG, uploaded with the stryker-logs artifact, and counted in the shard summary.

Local Windows runs with and without a recycle gave identical verdicts.

Verification run 37371127806 uses custom_scope **/Dispatchers/Strategies/*.cs, the same scope as the phase 2c/2d OOM runs. It is accepted when all of these hold:

  • no RuntimeError mutants;
  • every "failed on attempt 1/2" line is followed by a verdict;
  • the cgroup peak stays under 12 GiB;
  • the kill set matches a run without recycles.

Related: #1858 / PR #1860 (EEL tests share a static compiler) and #1859 (collectible ALC in src).

@codecov

codecov Bot commented Oct 5, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

Copilot AI balanced review requested due to automatic review settings October 6, 2026 11:07

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

Copilot AI balanced review requested due to automatic review settings October 6, 2026 13:17

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@dlrivada

dlrivada commented Oct 6, 2026

Copy link
Copy Markdown
Owner Author

Follow-up for the rewritten docs/testing/mutation-measurement-methodology.md: #1900 adds the job-graph diagram once this PR merges (the page still has no Mermaid block).

Copilot AI balanced review requested due to automatic review settings October 6, 2026 15:46

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants