Umbrella: test-strength, gate-tightening, and code-observatory tracks

[thymikee]

Why

Agents now write most code and most tests here. Two consequences drive this effort:

1. Test strength: agent-written tests are example-based and optimized to pass — we need tests-of-tests (properties, fuzz, mutation score) and deliberately provoked failure modes (races, compat breaks) so weak tests can't hide.
2. Observatory, not dashboard: we run 24+ CI checks whose signals are scattered. feat(scripts): interactive dependency-graph viewer, with when-to-use guidance #1409 (graph viewer) was closed because a rendered picture drew no conclusions; feat(scripts): dependency-graph report over the layering gate's model #1410 (analysis over the gate's own model) survived. The observatory is therefore agent-queryable data + quiet PR deltas, never a UI.

Design principles for every chunk (the patterns that already work in this repo):

• Something enumerates N — coverage claims derive from a registry (PUBLIC_COMMANDS, help topics, error codes), never a hand-picked list.
• Ratchets over absolutes — new metrics start non-gating, then only tighten.
• Quiet by default — PR surfacing shows only thresholded deltas in one sticky comment (evolve scripts/size-report.mjs, no new bots).
• Guards explain the fix — every new failure message names the command or file that resolves it.

Descope (2026-07-28)

A scope review split these tracks into two kinds of gate: PR-time, deterministic, self-explanatory failures (they replace the reviewer and make agents productive) and scheduled telemetry someone must read (it consumes the one maintainer). The first kind stays; the second is cut or deferred until a concrete consumer exists. Decisions:

• SkillGym is dropped. Two parallel LLM-judge suites answered one question; the help-conformance bench is now the single non-gating small-model oracle. test/skillgym/ and the check-affected lane are removed.
• Track C (observatory) is deferred, not staffed. obs: evolve the size-report PR comment into a quiet quality-delta comment + metric history on main #1424 is closed not-planned and its PR (obs: quiet quality-delta PR comment + repo-health metric history on main #1477) closed. The obs: repo-health snapshot command aggregating existing analyzers into one JSON #1423 repo-health snapshot command (landed in obs: repo-health snapshot command aggregating existing analyzers into one JSON #1471) is removed with the descope: its only gating assertion duplicated the Layering Guard job and its consumer was obs: evolve the size-report PR comment into a quiet quality-delta comment + metric history on main #1424. The depgraph/layering models remain the queryable data. Revisit when someone reaches for a metric twice and it isn't there.
• obs: scheduled-lane health, freshness telemetry, and standard artifact envelope #1430 (scheduled-lane health) is closed not-planned. It monitored the scheduled lanes; with the paid bench still ungated and mutation's scale undecided, a monitoring stack for the test stack is premature. The minimal lane-envelope that already shipped stays as-is.
• gate-of-gates: derived manifest proving every check is owned, wired, and reachable #1429 proceeds in minimal form only: the derived workflow→script→suite manifest and reachability check, deterministic and offline. No telemetry, no envelope stack. The branch-protection audit stays a separate scheduled read-only check.
• The mutation lane needs a commit-or-downscope decision (gate(mutation): ratchet graduates on stability alone — a 28.75% score can lock in as the permanently gated floor #1457) before any new mutation infrastructure. Either expand the baseline until the 2.3k-LOC harness is amortized, or shrink to changed-line mutation over the enumerated kernels and delete the ratchet machinery. The middle state (full infra, one module group) is the worst of both.
• gate: released public-surface compatibility (dts, CLI/MCP schemas, daemon RPC, persisted formats) #1432 is promoted: released public-surface compatibility includes the persisted .ad format, and replay is the product. This is the most product-critical open item in the umbrella.
• test: deterministic boundary fault-injection matrix for daemon HTTP and subprocess seams #1431 is product testing (daemon HTTP + subprocess seams) wearing a Track E label — sequence it against feature work, not gate work.

Track A — test strength

• test: property-based testing foundation (fast-check) over parsing/geometry kernels #1413 property-based testing foundation (fast-check) over parsing/geometry kernels
• test: nightly fuzz lane — parser inputs must fail as typed AppErrors, never hang #1414 nightly fuzz lane — parser inputs fail as typed AppErrors, never hang
• test: weekly mutation-score ratchet over decision kernels (Stryker, non-gating first) #1415 weekly mutation-score ratchet over decision kernels (non-gating first) — paused for the gate(mutation): ratchet graduates on stability alone — a 28.75% score can lock in as the permanently gated floor #1457 commit-or-downscope decision before it grows
• test: seeded concurrency torture lane for session/lease/lock invariants #1416 seeded concurrency torture lane for session/lease/lock invariants
• test: frozen replay-compat corpus with expected verdicts #1417 frozen replay-compat corpus with expected verdicts

Track B — gate tightening

• ci: changed-line coverage gate on PRs #1418 changed-line coverage gate on PRs
• ci: single-retry policy for enumerated contention-flaky files (timeouts only) #1419 single-retry policy for enumerated contention-flaky files (timeouts only) — in review (test(ci): single-retry policy for enumerated contention-flaky files (timeouts only) #1448)
• gate: website command docs enumerated against the command registry #1420 website command docs enumerated against the command registry
• gate: error-code recovery enumeration (retriable codes get a bench recovery quiz) #1421 error-code recovery enumeration (retriable codes get a bench quiz)
• chore: dependency hygiene — Renovate + corepack packageManager enforcement #1422 dependency hygiene: Renovate + corepack packageManager enforcement

Track C — observatory (descoped 2026-07-28)

• obs: repo-health snapshot command aggregating existing analyzers into one JSON #1423 repo-health snapshot command — landed (obs: repo-health snapshot command aggregating existing analyzers into one JSON #1471), then removed with the descope (see above)
• obs: evolve the size-report PR comment into a quiet quality-delta comment + metric history on main #1424 quiet quality-delta comment + metric history on main — closed not-planned; PR obs: quiet quality-delta PR comment + repo-health metric history on main #1477 closed
• obs: blast-radius query — file/symbol to dependents, owning gates, and live-coverage owners #1425 blast-radius query: file/symbol → dependents, owning gates, live-coverage owners — landed (feat(scripts): blast-radius query — dependents, owning gates, live-coverage owners (#1425) #1470) and kept: agent-queryable data contributors actually use

Track D — parity + scheduled benches

• test(android): command-coverage manifest parity with iOS #1426 Android command-coverage manifest parity with iOS
• ci: weekly scheduled model-backed help-bench run with pass-rate deltas #1427 weekly scheduled model-backed help-bench run with pass-rate deltas — now targets the single remaining suite (help-conformance); still gated on one manual paid bench run first (feat(bench): renderer-pinned samples, topic-coverage gate, error-recovery quizzes; trim skillgym to agentic checks #1411)

Worker coordination rules (added after the first PR wave, 2026-08-01)

The first wave (#1436, #1437 merged, #1438, #1439, #1441) validated the direction — real Stryker, real fast-check, real producers, and two real production bugs already found (#1433 excess positionals; the replay quoted-token SyntaxError leak). It also produced one systemic failure worth a standing rule:

1. One issue per PR. Three PRs each shipped their own parallel implementation of obs: scheduled-lane health, freshness telemetry, and standard artifact envelope #1430's health watcher as a rider (three directory names, two identical workflow filenames, two undisclosed issues: write grants). If your lane needs a sibling issue's deliverable, implement only the minimal interface your lane consumes and link the sibling — never the sibling's feature.
2. Rebase before building. Two branches were cut minutes before test: property-based testing foundation (fast-check) over parse/print and geometry kernels #1437 merged and duplicated its fast-check foundation (parallel PRNGs/hazard lists, lost shrinking). Workers start from current main and reuse src/__tests__/test-utils/property-arbitraries.ts and scripts/lib/lane-envelope.ts.
3. CI-scope changes are decisions, not side effects. Anything that adds work to the PR gate (a fast sweep, a path-triggered mutation run) is disclosed in the PR body with its measured cost.
4. Subprocess-spawning tests go in the subprocess-stub project. Never unit-core.

Track E — gate integrity (added from external review, 2026-07-27; narrowed 2026-07-28)

The original tracks assumed the gates keep running. Nothing proved it — the most dangerous failure mode is a perfectly designed gate that silently stops running (verified instances: CHECK_CATALOG.ciJobs are unchecked manual strings; ci.yml ignores website/**, unreachable for the docs gate as first scoped).

• gate-of-gates: derived manifest proving every check is owned, wired, and reachable #1429 gate-of-gates, minimal form: derived, transitive workflow → action → script → suite manifest proving every check is owned, wired, and reachable; deterministic/offline on PRs, with branch-protection contexts audited separately by a scheduled read-only online check
• obs: scheduled-lane health, freshness telemetry, and standard artifact envelope #1430 scheduled-lane health, freshness telemetry, and standard artifact envelope — closed not-planned (descope)
• test: deterministic boundary fault-injection matrix for daemon HTTP and subprocess seams #1431 deterministic boundary fault-injection matrix (daemon HTTP + subprocess seams) — product testing; sequence with feature work
• gate: released public-surface compatibility (dts, CLI/MCP schemas, daemon RPC, persisted formats) #1432 released public-surface compatibility (dts, CLI/MCP schemas, daemon RPC, persisted formats) — promoted: protects the .ad format replay customers depend on

The remaining platform manifests (macOS, web, Linux, tvOS, visionOS) are tracked as checkboxes on #1426 rather than as stub issues.

Architecture direction (written down)

The rule → gate → ledger mapping behind these tracks lives in CONTEXT.md § "Principles and their gates" (#1428): Dependency Rule → layering R1–R3/spine + the R6 TYPE_INVERSION_BASELINE ratchet (long-term target: zero), acyclicity → R4 + depgraph's tolerated-cycle report, policy×detail → the two registries + leak guard + transport seam, information hiding → R7, and the boundaries-are-earned norm (#1409 evidence). Component metrics (instability/abstractness/main-sequence per zone) were headed for #1423 as observatory data; with the descope they stay out until a consumer exists, and must never become CI thresholds. Standing direction alongside these tracks: shrink R6, then the deferred client-types narrowing and legacy alias drops (CONTEXT.md § Deferred).

Sequencing notes

(2026-07-28: entries referencing descoped issues — the #1423 → #1424 → #1425 chain and "#1430 before the scheduled lanes" — no longer apply.)

• ci: changed-line coverage gate on PRs #1418 feeds rows into future PR surfacing; it landed first, so the data exists if Track C is ever revived.
• ci: weekly scheduled model-backed help-bench run with pass-rate deltas #1427 needs one manual paid bench run first (matcher validation, see feat(bench): renderer-pinned samples, topic-coverage gate, error-recovery quizzes; trim skillgym to agentic checks #1411).
• Everything in Track A is independent and parallelizable.
• gate-of-gates: derived manifest proving every check is owned, wired, and reachable #1429 (gate manifest) is cheap and catches real wiring holes today — land it early in its minimal form; its PR gate must remain deterministic and network-free, prove transitive execution ownership, and fail closed on unresolved dynamic edges. gate: website command docs enumerated against the command registry #1420 depends on the reachability fix or must self-fix the website/** exclusion.
• Child issues test: weekly mutation-score ratchet over decision kernels (Stryker, non-gating first) #1415–gate: website command docs enumerated against the command registry #1420, chore: dependency hygiene — Renovate + corepack packageManager enforcement #1422–obs: evolve the size-report PR comment into a quiet quality-delta comment + metric history on main #1424, test(android): command-coverage manifest parity with iOS #1426 carry "Review amendments (2026-07-27)" sections — read them as part of the spec.

Pointers: docs/agents/testing.md (gate matrix + traps), scripts/check-affected/ (gate router), scripts/layering/ (R1–R8 + ratchets), scripts/depgraph/ (#1410), test/integration/ios-simulator-e2e/coverage-manifest.ts (#1408), scripts/help-conformance-*.mjs + scripts/tests/ (renderer-pinned bench gates, #1411).