Repository navigation
Description
Summary
- Goal: a PR flow that is deterministic, gated, and trusted by every maintainer, so an approved PR can be merged safely.
- How: deterministic checks plus one human approval on the exact head commit decide every merge.
- AI review speeds people up, but it never decides.
- Every step below is a small, separate PR. Maintainers can accept or decline each one on its own. Nothing here is decided. The plan has been revised per the reviews in the comments below.
This is the broader "PR assurance flow" suggested in a comment on #464. #464 becomes its AI-review layer (gate 3 below).
The layering follows the approach the Claude Code team describes in How the Claude Code team does code review: several independent layers, each catching what slips past the one before, with people kept for direction, risk and accountability.
Why now
All of these come from public data: the workflow files and rulesets on main, the Actions run history, and the PR and commit history.
- The required status checks are the four CodeQL legs (from
ci.yml). The ruleset also blocks on CodeQL alerts and needs one approval with code-owner review. Unit tests, lint and the scanners don't block. - CI is slow. The
CIworkflow took a median of 41 minutes over 63 completed runs (2026-11-15 to 11-16 UTC). The full Python suite runs 3 times per push: Unit Tests 3.11, Unit Tests 3.12, and the backend-coverage step of CAO MCP Apps. - Flaky tests force re-runs. Between 2026-11-15 and 11-16, 19 of 66
CIruns were re-runs, and 15 ended green. tedswinyar's review found that about half of those re-runs look like fork-PR approvals, not failures, so the window classifies red jobs rather than counting re-runs. Every failed job failed on one Four tests fail intermittently in CI on unrelated PRs (memory BM25 default sort, two terminal-service timing tests, workflow run view) #1341 test. Four tests fail intermittently in CI on unrelated PRs (memory BM25 default sort, two terminal-service timing tests, workflow run view) #1341 lists four flaky tests. Two tests fail intermittently on pristine main, roughly 1 in 3 and 1 in 4, reproducible in isolation #704 lists two more and has been open since 2026-08-28. - CHANGELOG conflicts force repeated main merges. 29 of the 77 first-parent commits to
mainfrom 2026-11-05 to 2026-11-16 touchedCHANGELOG.md. Any two open PRs that add a bullet conflict, and each main merge starts a fresh 40-minute CI run. - The release step regenerates
CHANGELOG.mdwith git-cliff and overwrites the hand-written notes (it did so at 2.5.0). Tracked in its own issue, [Bug] The release job's changelog step replaces the hand-written CHANGELOG with commit subjects #1353, and fixed first. - The up-to-date rule adds more cycles. On 2026-11-16, at least 12 of 47 open PRs were BEHIND
main. - Stacked PRs get no CI.
ci.yml, Secret Scan, cargo-deny and zizmor trigger only onpull_request: branches: [main]. On fix(memory): stop writing memory into project files by default (#1335) #1339, which is stacked on fix(memory): injected memory blocks no longer go stale or skip whole scopes (#1329 P1b) #1333, the reviewer had to run the tests locally.
The target flow
PR ─► 1 ci-gate: tests, lint, scanners; registered flakes get one automatic retry (deterministic, required)
─► 2 risk label from the changed paths; high-risk paths need their code owners (deterministic)
─► 3 AI review in Actions (#464), posted as an advisory comment (advisory only, never required)
─► 4 human approval on the exact head commit (dismissed on push, as today)
─► 5 merge queue: tests the real merged result, then merges (deterministic)
Terms used here:
- ci-gate: one aggregate job in
ci.yml. It runs withif: always(). It passes only if every gated job succeeded, and a skip passes only where it's allowed (for exampledependency-reviewonpushandmerge_group). It includesDependency Security, asSECURITY.mdasks. Rulesets can then require one check instead of 20. - Flake registry: a small checked-in list of known flaky tests. Each entry has an open issue, an owner and an expiry (at most 30 days). A registered test that fails is retried in the same job, and the job summary says so. An unregistered failure stays red. An expired entry fails CI. Built only if the measured window shows recurring flakes.
- Changelog fragments: one small file per PR (for example
changelog.d/<slug>.md). They are folded intoCHANGELOG.mdat release time, so PRs stop conflicting on the same lines. - Merge queue: GitHub's queue that tests each PR against the latest
mainplus the PRs ahead of it, then merges. PRs no longer need "update branch" loops. It would use squash, to match the squash-only and linear-history rules, and every required workflow getsmerge_group:.
The phases
Each step is a separate, small PR.
| Phase | Step | Status |
|---|---|---|
| 1. Make CI trustworthy | 1. Fix the known flaky tests at their root. | In flight: #1346 and #1349 merged. Next: #1341 tests 2–3, then #704. |
1a. PR-only concurrency with cancel-in-progress (never on main pushes, which the release preflight reads). |
proposed | |
| 1b. The coverage ratchet reads the 3.12 leg (3 suite runs become 2). It saves runner time and one chance to flake per push, not wall time: the suite jobs already run in parallel. | proposed | |
| 1c. A nightly Python 3.14 job that isn't required, so 3.14-only bugs show up without slowing PRs. | proposed | |
1d. Keep the suite away from the real ~/.aws and tmux. |
in place (#1231, #1245); kept as a rule | |
1e. Sweep the short wall-clock limits in tests (.wait() and wait_for timeouts of 5 seconds or less): triage the list first, then fix the risky ones in small PRs. It doesn't hold up the window. |
proposed | |
1f. An order and isolation canary: a nightly job that isn't required, with the test order shuffled (the seed printed, through a small conftest.py hook, so no new dependency), a fresh HOME and no CAO_* variables. A failure gets an issue and a root fix, never a retry entry. Its own PR, after 1c. |
proposed | |
| 2. Measure 1–2 weeks after tests 2–3; add a flake registry (one automatic retry, an expiry, an issue link) only if flakes recur. | proposed | |
3. CI on stacked PRs: plain pull_request: with no branch filter, in the four workflows above. Safe while "up to date" stays on; drop it only together with a merge queue (step 8). |
proposed | |
| 4. After the changelog fix (#1353): fragments or generated notes (Q3). | proposed | |
5. The ci-gate job, advisory at first. |
proposed | |
5b. Bring the seven path-filtered workflows (the four provider tests, the two robotics examples and the docs site) under ci-gate: move their jobs into ci.yml behind a changed-paths job, so ci-gate counts them and allows a skip only when no matching path changed. Until then, they sit outside the gate. |
proposed; maintainers' choice | |
| 2. Switch the gates on | Before 6. The bypass policy (Q8): who stays on the bypass list of each main ruleset, and admin bypass as break-glass only, with a stated reason. Otherwise every required check below can be skipped. |
maintainers' choice |
6. Require ci-gate and the scanners (gitleaks, zizmor, cargo-deny), next to CodeQL, with a named owner and an expected turnaround for new-advisory failures. Trivy (Security Scan) and cargo-deny can take an ignore entry with a reason and an expiry. Dependency Security takes no ignore file and blocks advisories with no published fix, so its owner needs either the authority to make a product call (as in #1119), or an exemption path that the script and SECURITY.md allow. |
after Phase 1 shows CI is stable | |
| 7. A deterministic risk-label script from the changed paths, seeded from gutosantos82's tested path rules. Extend CODEOWNERS (workflows and actions are already owned) to the release scripts, trust-boundary code and the scanners' policy files; land this before step 6. | proposed | |
8. A merge queue (squash; merge_group triggers), the recommended choice. Non-strict mode (drop "branch must be up to date") leaves stacked PRs untested against main. |
maintainers' choice | |
| 3. Standardize the AI review | 9. A checked-in review rubric (for example REVIEW.md) and a machine-readable findings format. |
later |
| 10. The #464 reviewer in Actions, advisory; started by hand first, then automated. A single agent. It records which review angles ran and marks a partial review as partial; cancels stale runs and keeps one comment updated in place; and checks its pinned model first, ending as a neutral check instead of hanging. | later | |
| 11. A reviewer eval: 12–30 labelled cases from past PRs, offered by gutosantos82 (reports tied to exact head commits, with the maintainers' verdicts). It can score any reviewer, so it can start before steps 9–10. | Accepted; case format proposed in the comments | |
| 12. Copilot code review: make it post or remove the ruleset. If it posts, compare it with step 10 in the shadow run. | later | |
| 4. Measure and tune | A scorecard: human wait time, rounds per PR, re-runs, conflict merges, escaped defects, and whether approval is becoming a formality (time from AI report to approval, approvals with no human comment). It shows what to tighten or relax. | ongoing |
Order matters. Requiring ci-gate before the flaky tests are fixed (and the registry is in place, if it's needed) would turn today's flakes into hard merge blocks.
What stays human
- The approve click on the head commit.
- Scope and design calls, including "should we do this at all?".
- Changes to high-risk paths (CI, release, security and trust-boundary code), through CODEOWNERS.
- Every policy switch: required checks, the merge queue, the registry's entries, the risk tiers, and the AI lane.
Security notes for the AI lane (#464)
This plan builds on #464's security boundary, with one difference. The AI lane stays advisory. It is never a required or blocking check, because model output isn't deterministic. #464 proposes blocking on validated, high-confidence Must Fix findings. This plan doesn't adopt that: an AI finding never blocks a merge on its own. When a finding is real, the human reviewer acts on it.
The suggested shape:
- One workflow, two jobs. The review job reads the key and has read-only permissions. The post job writes and has no key.
- The key lives in an environment whose branch rule allows only
main, so apull_requestrun (whose workflow file comes from the PR) can never receive it. - The first trigger is a manual dispatch from
main. Any later automatic trigger uses a workflow defined onmain, never checks out or runs PR code, and is recorded in the zizmor config with a reason. - One agent, read-only tools. No shell, no MCP servers, no web access, and the CLI's sandbox mode on.
- Base-branch config. The job checks out the base commit. The reviewer's agent config comes from the base branch. The PR's diff and files are plain data in a folder outside the working directory. No PR code runs.
- Schema-checked advisory output. The post job validates the output against a fixed schema, scans it for secrets, and renders it with a fixed template. It keeps one comment, updated in place for each new head commit, and a neutral check that is never required.
Questions for maintainers
- Which checks should become required, and when:
ci-gate, the scanners, or both? Who owns new-advisory failures, and what turnaround do they commit to? ForDependency Security, should its owner have the authority to make a product call when an advisory has no fix, or should the script andSECURITY.mdgain an exemption path? - A merge queue, or non-strict mode?
- Changelog fragments, or generated notes from commit subjects (
git-cliff), or both? Which fragment tool? - Which paths should CODEOWNERS cover as high-risk, and who owns them?
- The flake registry policy: how many retries, how long an entry may live, and the maximum number of entries?
- For the AI lane: who owns the Kiro API key and the subscription that pays for it?
- Copilot code review: fix it or remove it?
- The bypass policy: who stays on the bypass list of each
mainruleset, and should admin bypass become break-glass only, with a stated reason?
Related: #464, #1341, #704, #1346, #1349, #1353, #1231, #1245
Activity
fanhongy commentedon Nov 16, 2026
@haofeif tell me what do you think.
haofeif commentedon Nov 16, 2026
Thanks @fanhongy. I checked each claim against
mainat6bf6c68e, the three rulesets, the Actions history of 11-15 and 11-16 (UTC), and the GitHub, zizmor and git-cliff docs. The direction is good, and the order is correct: fix the flaky tests first, then make the checks required. Below are the confirmed facts, new data, some corrections and the trade-offs I see. Nothing here stops the plan.Confirmed
protect-mainhasstrict_required_status_checks_policy: true.CIruns.ci.yml:70runs it in bothUnit Testslegs, andci.yml:310runs it again inCAO MCP Apps.CIruns were re-runs, and 15 of them ended green.mainci.yml,secret-scan.yml,cargo-deny.ymlandzizmor.yml. #1339's base is #1333's branch.New data for Phase 1
I read the logs of the 15 jobs that failed before a re-run in those two days. Each job failed on exactly one test, and each of these tests is in #1341:
The two #704 tests did not fail in this window. From this data:
mainremove 11 of the 15 failures. Tests 2 and 3 caused the other 4.Unit Tests (3.12)leg, each push runs the suite one time less.ci.ymlhas noconcurrencygroup, so a new push to a PR does not stop the old run.cancel-in-progressforpull_requestevents is a small change.Corrections
protect-mainalso has acode_scanningrule: CodeQL alerts aterrors, or security alerts athigh_or_higher, block the merge. It also requires one approval, dismisses stale approvals on push, and hasrequire_code_owner_review: true..github/workflows/,.github/actions/,.github/CODEOWNERS,patches/,vendor/, the threebraces-compat/folders,scripts/dependency_security.pyandscripts/test-braces-security.cjs. The owner is@awslabs/multiq. So step 7 extends the coverage. Candidates:scripts/bump_version.py,scripts/release_preflight.py,scripts/release_commit.pyand the trust-boundary code undersrc/.Copilot Auto Reviewruleset hasreview_on_push: trueon all branches (not drafts). Gate 3 adds a second AI reviewer. Please say if [Feat] Add autonomous headless PR reviewer to CI #464 replaces Copilot review or runs next to it. The Phase 4 scorecard can compare them.ci.yml. They are itscodeqljob (ci.yml:1068-1087), not default setup. This helps step 8: amerge_group:trigger inci.ymlcovers them.SECURITY.md:151-153already asks forDependency Securityas a required check in themainruleset, next to CodeQL.protect-maindoes not require it. Ifci-gateincludes thedependency-securityjob, step 6 also closes this gap.branch protectionruleset allows only squash, andprotect-mainrequires linear history. A merge queue should use squash to match.Trade-offs by step
Step 3 (CI on stacked PRs). Each push to a stacked PR starts a full run of about 41 minutes. Also, when the parent merges, GitHub deletes its branch (
delete_branch_on_mergeis on) and retargets the child tomain. That is apull_requesteditedevent. By default, a workflow runs only onopened,synchronizeandreopened, so no new run starts. Strict mode makes the author update the child, and a merge queue tests the merged result. Both re-test the child againstmain. With non-strict mode, nothing re-tests it before the merge. So decide step 3 together with step 8.Steps 5 and 6 (
ci-gateand required scanners).if: ${{ always() }}on the gate, and fail it when aneeds.*.resultisfailureorcancelled. Withoutalways(), a failed job makes the gate skip, and GitHub counts a skipped check as a pass (troubleshooting required status checks).skippedmeans.dependency-reviewskips onpushby design (ci.yml:1056), and it will also skip onmerge_group. Any other skip should fail the gate.SECURITY.md:94-96documents this forDependency Security.cargo deny checkruns its advisories check on each PR, andtui/deny.tomlalso denies yanked crates. When these checks are required, a new advisory blocks all merges until a lock bump or an ignore entry lands. This is the documented intent, but the plan needs an owner for that fast path. The flake registry does not cover it.Step 8 (merge queue or non-strict mode).
merge_group:trigger (managing a merge queue). This includessecret-scan.yml,cargo-deny.ymlandzizmor.ymlif step 6 requires them. Each queue group runs the full CI, and a failed check removes the PR from the queue. So Phase 1 is as important for the queue as for required checks. If you turn off the queue setting "Only merge non-failing pull requests", a later group that passes can absorb some intermittent failures.maintogether, and a stacked PR can merge on checks from its old base (step 3). Thepushrun onmainfinds the break only after the merge.Step 4 and Q3 (changelog).
scripts/bump_version.py:141runsgit-cliff --tag vX -o CHANGELOG.md. With-o, git-cliff writes a new file from the git history; its docs use--prependto add to an existing file. So the release step does not read the hand-written[Unreleased]notes, and it will not read fragments.scripts/release_commit.py:43then commits that file.release.ymlrun was on 2026-08-28 (v2.5.0). Norelease.ymlrun cut v2.5.1 to v2.5.4. So all notes since 2.5.0, about 1,850 lines, are under[Unreleased].generate_changelog(test/scripts/test_bump_version.py:181,:269), so no test catches this.generate_changelogfirst. This can be its own issue, because the weekly scheduled release has the problem without this plan.release.ymlalso creates each GitHub Release withgenerate_release_notes: true. So notes generated from PR titles already exist there.Phase 3 (AI lane security).
pull_requestworkflow in the context of the PR merge commit (the events docs compare this withpull_request_target). So the workflow file comes from the PR. A same-repo branch can change the review job and read the model key before any review. "Base-branch config" protects the agent config, but not the workflow file.workflow_runas a Highdangerous-triggersfinding (audit docs). The repo gate fails at medium and higher (.github/workflows/zizmor.yml:42-44), and.github/zizmor.ymlrequires a written reason for each ignore.contents: read. The post job haspull-requests: writeand no secret. So the first stage may not needworkflow_run. If you keep it, treat the artifact as untrusted, because the PR run can upload any file with that name. The schema check and the fixed template already cover the rendering.The questions
Q4 and Q6 need owners. The maintainers must decide them, and I have no data to add. For the others, my view from the data:
ci-gateas advisory first, as proposed. Make it required after Four tests fail intermittently in CI on unrelated PRs (memory BM25 default sort, two terminal-service timing tests, workflow run view) #1341 tests 2 and 3 are fixed and a measured window is clean. Add the scanners at the same time, with an owner for advisory failures.generate_changelogfirst. Then use fragments if the hand-written notes are the product, or generated notes if PR titles are sufficient.🤖 Reviewed with CAO review workflows
fanhongy commentedon Nov 16, 2026
Thanks @haofeif, this is very useful, and you're right on almost every point. I checked them against
main, the rulesets and the docs. Here's where I land.What I accept
concurrency. Cancel in progress forpull_requestonly. Pushes tomainmust never be cancelled, because the release preflight needsci.ymlgreen at the exact release SHA. Merge-queue runs aren't cancelled either.coverage.json, a small ratchet job reads it together with the vitest summary, and the third suite run goes away. That leg also runs the Trivy-gated tests, so the figure can only rise against the floors..github/zizmor.yml,.gitleaks.toml,tui/deny.toml,.coverage-baseline.json,mypy-baseline.json,.github/dependabot.yml). Once the scanners are required, those should be owned before the switch, or an ignore entry becomes a one-approval way around a required check.ci-gate. It getsif: always(). It passes only if every gated job succeeded, and a skip passes only from an allowlist (dependency-reviewonpushandmerge_group). It includesDependency Security, which closes theSECURITY.mdgap, and the required check is pinned to the Actions app. No path filters.securityjob has the same property as Dependency Security and cargo-deny. The fast path would be an ignore entry with a reason and an expiry, reviewed by a code owner. The flake registry never covers these.merge_group:on every required workflow. One detail:secret-scan.ymlonly scans onpull_requestandschedule, so onmerge_groupthegitleaksjob would run no scan and still pass. It needs a branch that scans the group's range.editedtype, since every title edit would start a full run.generate_changelog. Confirmed, and it has already happened once: the v2.5.0 release commit replaced the hand-written[Unreleased]notes with commit subjects, including the full GHSA-6m35-gcf5-xm75 write-up. The weekly run currently refuses because the next bump is a major, so the step would fire on the manually dispatched 3.0.0 release. I'll open it as its own issue with a small fix: promote[Unreleased]to the version heading deterministically, drop git-cliff from the release job (the GitHub Release already has generated notes), and test the real function instead of a stub.--prependalone wouldn't fix it.Your direct questions
Copilot and #464. In a sample of 17 recent PRs (15 with review data), the Copilot ruleset posted no reviews and no comments. So it's configured, but it isn't producing anything. I'd suggest the maintainers first either make it post (it may be a billing or policy setting) or remove the ruleset. If it does post, run it next to the #464 reviewer in shadow mode on the same 20–30 PRs, compare them in the Phase 4 scorecard, and then keep one automatic lane. Both stay advisory, and neither approves. If Copilot works, it has a real advantage: there's no key to protect.
The key model. You're right. Under
pull_request, the job that holds the key is the PR's own workflow file, so "base-branch config" didn't protect it. I also think an environment with required reviewers doesn't fully fix it: the approver approves a job the PR wrote, once per push. The proposal I'd make instead:main(norefs/pull/*), with admin bypass disallowed. GitHub matches that rule against the run'sGITHUB_REF, so apull_requestrun can never receive the key, whatever its workflow file says;contents: read, and the post job haspull-requests: writeand no key. The model's output is still treated as untrusted (schema, secret scan, fixed template);workflow_dispatchfrommainfor shadow mode. That avoidsworkflow_run, and zizmor stays clean;pull_request_targetwith a base-only checkout and a reasoned zizmor entry, or a small no-secret PR job that dispatches themain-defined workflow.What I'd like to discuss
Next steps
concurrency, the coverage de-dup and an advisoryci-gate.I'll update the issue body with these corrections.
haofeif commentedon Nov 16, 2026
@plauzy @tedswinyar @gutosantos82 please have a look at this proposal
gutosantos82 commentedon Nov 16, 2026
Thanks @fanhongy, and @haofeif for the data. I support the plan and the order. I've been running a multi-agent CAO review pipeline (
examples/pr-review/on my fork) on this repo's PRs since August, about 1,000 reports. Below is what that adds, mostly for Phase 3.Where our runs back the plan
mainon fix(fleet): bound terminal status read timeout to prevent send overrun #1217 committed a stray>>>>>>> upstream/mainmarker atCHANGELOG.md:72. Missing[Unreleased]entries were our most common conventions nit. That supports fragments (Q3).Unit Tests (3.12)andDependency Securitywere red onmainand on every PR, and nothing blocked. That supportsci-gateand the owner for new advisories.Gaps for the AI lane (Phase 3)
cao-mcp-server. Under kiro's sandbox mode, everyassignfailed silently from September until fix(orchestration): load an unreadable child profile from cao-server #1213. If [Feat] Add autonomous headless PR reviewer to CI #464 is a CAO team, it needscao-mcp-serverallowed (and fix(orchestration): load an unreadable child profile from cao-server #1213). If it's a single agent, the "no MCP servers" rule is fine. It would help to state which.pull_requestruns get no secrets. This affects both the queue's speed and the choice of trigger for the AI lane.Two cautions
Path.exists()hides a permission error), and CI runs only 3.11 and 3.12. A nightly 3.14 job that isn't required would cover it without slowing PRs. Test isolation (fix(status): agy init reaches idle; run the test suite against a throwaway HOME #1231, test: run the test suite against a private tmux server #1245: no access to the real~/.awsor tmux) also belongs with Phase 1.Offer for step 11 (reviewer eval)
I have a labelled set: reports tied to exact head commits, what was posted, and other maintainers' verdicts at the same heads, including disagreements (e.g. #886, #1150). I can pull 12–30 cases into a small eval if that helps.
On the questions: Q2, a merge queue. Q3, fragments. Q4 and Q6 are for the maintainers.
fanhongy commentedon Nov 17, 2026
Thanks @gutosantos82, this is very useful data. I'd like to take you up on the step 11 offer.
Step 11: the reviewer eval. Yes, please pull the 12–30 labelled cases. To make them reusable by any reviewer we try (the #464 lane, Copilot code review, or your pipeline), a format like this would work:
Partial reviews and missed findings matter most here, since those are what a scorecard should catch. A small JSON or Markdown file per case is fine. Where should they live: in your fork for now, or as a small
test/review_eval/folder in a PR, so the eval runs against the same cases every time the rubric or the runner changes?The rest of your comment. I'll fold your points into the plan and the issue body:
On your question: the #464 lane as proposed is a single agent with read-only tools and no MCP servers, so the sandbox
assignlimit doesn't apply. If it ever becomes a CAO team, it will needcao-mcp-serverallowed and the #1213 fix, and I'll note that in the plan.🤖 Generated with Claude Code
fanhongy commentedon Nov 17, 2026
Step 2: the measuring window (rule v2, revised 2026-11-17 after plauzy's review)
Changed from v1: the floor is 150 finished runs, not 40; every red job is classified, with a limit on unclassified jobs; timeouts are classed by the step they hit; one cause counts once per run;
pushruns onmainare counted; and a fail now says what happens to the clock.So the result can't be argued afterwards, here is how we'll measure CI before deciding on the flake registry and on making
ci-gaterequired:pull_requestruns ofCI. A finished run ended in success or failure. Cancelled runs don't count toward the 150. If both aren't met within 14 days, we report what we have and extend.pushruns onmain: counted too, on their own row, and their flaky failures count toward the limit.main.At the end, we'll post the counts per run, per job and per test, the outage list and the unclassified jobs, with links to the runs.
🤖 Generated with Claude Code
plauzy commentedon Nov 17, 2026
Thanks @fanhongy. I support the plan and the order, and writing the rule down before the window starts is right.
Everything below is new to this thread. I agree with the rest of the plan and with the earlier reviews, and I haven't repeated them.
Legend: 🆕 new feedback · ❓ new open question · ✏️ correction to something already accepted
pushruns onmainDependency SecurityThe step 2 rule
1 · 🆕 NEW · Classify every red job, and report what nobody classified
The definition needs a re-run of the same commit. When an author pushes a new commit instead, that failure is never counted, and 1a (cancel-in-progress) will make that more common.
A re-run on its own isn't the signal either. From #1349's merge to 2026-11-17 05:24Z there were 38 completed PR runs, and 5 went green after a re-run. Only 2 of those have a failed first attempt to read:
[False]case in Unit Tests (3.11), run 43988559088 (this corrects my earlier update on Four tests fail intermittently in CI on unrelated PRs (memory BM25 default sort, two terminal-service timing tests, workflow run view) #1341; see the correction);The other 3 list no first-attempt jobs. So I'd add three things to the rule:
unclassifiedcount with its own limit;timeout-minutescounts as flaky, not as an outage.2 · 🆕 NEW · Raise the 40-run floor
At 40 runs, "at most 1" is consistent with a flaky rate of up to 11% of runs (95% upper bound). At 150 runs it is about 3%. A normal week gives 200 or more, so this only matters in the 14-day fallback, which is where the result is weakest.
3 · ❓ NEW QUESTION · What does a fail do to the clock?
Does the window restart, or continue once the fix lands?
4 · ❓ NEW QUESTION · One cause can count twice
Counting per job means one cause can fail two jobs in one run. Test 2 did that on #1343. Is that one failure or two?
5 · 🆕 NEW · Count the
pushruns onmainThey are the closest thing we have to a merge-queue run.
What the window won't measure
6 · 🆕 NEW · Outages need their own limit for step 6
Listing outages separately is right for the registry decision. For step 6 it isn't enough, because a required
ci-gateis blocked by a mirror outage exactly as it is by a flaky test. On 2026-11-11 an apt outage turned five jobs red onmain(#1303). I'd give the outage list its own limit in the step 6 decision. The known remaining exposure is tracked in #1375.7 · ✏️ CORRECTION · The advisory fast path doesn't exist for
Dependency SecurityThe plan's fast path is an ignore entry with a reason and an expiry. That works for the scanners that read an ignore file.
Dependency Securitycan't take one today:--ignorefile /dev/null;SECURITY.mdsays an advisory with no published fix blocks.#1118 is the case to design for. The advisory had no fix on Python 3.10, so the fix was to drop 3.10 (#1119). That was a product decision, not a lock bump. Once the check is required, the owner needs either the authority to make that call, or an exemption path that the script and
SECURITY.mdallow. That is the maintainers' decision.Optional
8 · 🆕 NEW · Sweep the short wall-clock limits before the window
Three of the five known flaky tests are short wall-clock limits that fail under load. A rough grep of
test/onmainfinds about 100.wait()calls of 5 seconds or less in 32 files. Many are harmless, but a sweep with the #1357 pattern may be cheaper than finding the rest one window at a time.fanhongy commentedon Nov 17, 2026
Thanks @plauzy. These are fair, and most of them tighten the rule before it starts, which is why we wrote it down. I checked the ones I could against
main.The step 2 rule
I've folded points 1–5 into the rule and updated the rule comment in place to the revised version. The window hasn't started, so nothing is measured under the old one.
1. Classify every red job. Agreed. A re-run isn't a reliable signal, and 1a will make same-commit re-runs rarer. So:
unclassifiedgets its own limit: at most 2. Above that, the window is inconclusive and extends. Missing evidence shouldn't count as a pass, but it isn't a flake either.ci.yml, fix: stop orphaned pipe-pane writers and harden suite isolation (tmux gate, non-blocking FIFO opens) #1251), so hitting it is a test problem. A timeout in an install or download step is an outage, though: on 2026-11-11 the apt jobs ran 11 to 64 minutes before failing (ci: take apt off the browser jobs' critical path #1302). So a timeout is classed by the step it hit.2. The floor. Agreed. I get the same bounds: about 11% at 40 runs and about 3% at 150. The floor is now 150. It counts finished runs only, not cancelled ones. Once 1a lands, some runs will be cancelled, so the floor may matter outside the 14-day fallback too.
3. A fail and the clock. The window still runs to its planned end, so the report has the full per-test counts for Q5. Then a fresh window, under the same rule, starts once the root fixes for the new tests merge. Step 6 waits for a clean window. Continuing across a fix would mix the before and after data.
4. One cause, two jobs. One. For the limit, one cause in one run counts once, because a required
ci-gateblocks a run once, however many jobs fail. The report still shows the per-job counts, which stay comparable if 1b lands during the window.5.
pushruns onmain. Agreed. They go into the totals and the limit, on their own row. With "up to date" on, a redpushrun right after a green PR run on the same tree is strong flake evidence.What the window won't measure
6. Outages and step 6. Agreed. The report will give outages their own count and rate, by source. I'd suggest the step 6 decision use the same bar for outages as for flaky failures, since both block a required gate the same way. The limit itself is yours to set.
7. The
Dependency Securityfast path. You're right, and this corrects what I wrote earlier. Onmain:scripts/dependency_security.pyruns Trivy with--ignorefile /dev/nulland--ignore-unfixed=false;SECURITY.mdsays the gate blocks advisories with no published fix, and uses no repository ignore files.So an ignore entry only fits the other scanners:
cargo-deny(throughtui/deny.toml, which has no ignores today) and the Trivy gate inSecurity Scan(it already skips unfixed findings, and there's no.trivyignoretoday). I've corrected step 6 and Q1 in the description. ForDependency Security, the owner needs either the authority to make a product call like #1119, or an exemption path that the script andSECURITY.mdallow. Q1 now asks the maintainers to choose.Optional
8. The short wall-clock sweep. Worth doing, and it's now step 1e. I get a similar count: 98
.wait()calls of 5 seconds or less in 32 files undertest/. The same grep misses about 30 more written aswait_for(..., timeout=...). We'll triage the list first. Then we'll send small PRs for the ones that guard a handoff between threads or tasks under load, using the #1357 pattern (an explicit barrier and a generous deadlock bound). The window won't wait for them.🤖 Generated with Claude Code
fix(fifo): support descriptors above FD_SETSIZE (#1350) (#1376)
tedswinyar commentedon Nov 17, 2026
Thanks for writing this up, and for the review rounds already on it. I agree with the diagnosis and the order: fix CI, measure, then require it. Fixing
generate_changelogfirst (#1353) is the right call too. Three things I don't see covered yet.1. "Required" does not block here today. The admin role is a bypass actor on both
mainrulesets, and nearly every collaborator is an admin. As of yesterday the rule-suites API showed 134 of 249 pushes tomainin the previous 30 days recorded as bypasses (38 of 100 since 11-03, so it is improving). The up-to-date rule was skipped 110 times and the review rule 36 times, 8 of those with changes requested standing. My own #1036 is one of them. As written,ci-gate, the scanners, CODEOWNERS and the queue can all be skipped the same way. Before step 6, can we decide who stays on the bypass list (the release deploy key has to), make admin bypass break-glass with a stated reason, and put bypasses per rule on the Phase 4 scorecard? The audit trail already exists:2. One failure class passes every proposed gate. While merging main into #1069 I hit two tests in
test/security/test_origin_matrix.pythat fail on a clean checkout of main when run after themain()tests, which leavehttp://localhost:9889in the liveCORS_ORIGINSlist. CI is green because the order is fixed, and a retry registry would hide it. It still reproduces on main5a8dc10f; the fix is #1381. A small canary job (random order, fresh HOME, noCAO_*in the environment) would catch this kind of thing before it becomes "re-run it".3. On Q1 (who owns the scanner fast path), one data point. @plauzy's correction is right that Dependency Security has no ignore route. The last time it went red for everyone (11-14, a new advisory), the fix went in without an approving review, by bypass. So today's fast path is the bypass. Whoever takes Q1, I'd write the expected turnaround next to the name, and count that path on the scorecard too.
Smaller points:
v*tags, and release runs use the workflow file at the tag's commit, so none of these gates apply to a release. v2.5.4 is still a draft.ci-gate.I'd support requiring
ci-gateonce Phase 1 measures clean, running CI on stacked PRs, and keeping the AI lane advisory. For that lane, I'd add one measure to the scorecard: how often a PR is approved with no human comment after the AI report lands. If that number is high, the lane is replacing review, not advising it.fanhongy commentedon Nov 17, 2026
Thanks @tedswinyar. These are real gaps, and the bypass point matters most: a required check that most people can skip isn't really a gate. I checked what I could against
main(5a8dc10f). I haven't re-run the rule-suites query, so I've taken those numbers as given.Where things stand: the measuring window started at 13:20Z today, when #1368 merged. 1a and 1b are open as #1386 and #1387. I've updated the description with the changes below.
1. Bypass
Agreed. Our own sample points the same way: of 60 merges from 11-05 to 11-16, 7 ended without an approval, and 4 of those 7 were ours. So this applies to us too.
I've added a step before step 6, and a new question for the maintainers, Q8:
mainruleset;I support making admin bypass break-glass, including for our own urgent fixes. The list itself is yours to decide. For Phase 4, the scorecard will show bypasses per rule, read from the rule-suites endpoint you gave. Your 110 up-to-date skips also support the merge queue (step 8): admins are skipping the very update loop the queue removes.
2. Order-dependent failures
Thanks for #1381. On
mainit reads as you describe:TestMainEntryPointrestoresos.environbut not theCORS_ORIGINSlist thatadd_local_cors_origins()appends to in place.On why CI stays green, one likely reason, from reading the code only (I haven't run it): CI runs the suite in path order, so
test/api/comes beforetest/security/. In between,test/providers/test_cursor_cli_unit.pycallsimportlib.reload()oncli_agent_orchestrator.constants, which rebuildsCORS_ORIGINSfrom scratch. So the leak is reset before the origin rows run.You're right that the window can't see this class. In a fixed order it never fails, and in a random order a retry would hide it. So we'll add a canary, now step 1f in the description:
conftest.pydoes the shuffle, so there's no new dependency;HOME, and noCAO_*variables in the environment;It will be its own small PR, after the 3.14 job.
3. Q1 and the scanner fast path
Agreed. That 11-14 fix was ours, #1326. Q1 now asks for an owner and an expected turnaround. The scorecard will count each advisory fix: the time from red to merged, and whether it merged with an approval or by bypass.
There's a live case right now. Since the vulnerability database updated around 21:39Z today,
Dependency Securityhas been red on every PR, and it will be onmain's next run too:cao_mcp_apps,docusaurus,webandag-ui-operator-dashboardlockfiles.aws-cdk-lib(examples/decision-model-host/aws/cdk). Evenaws-cdk-lib2.277.0, the latest, bundles minimatch^10.2.6, so no bump on our side clears it.So neither one can be fixed from this repo today. That's exactly the Q1 choice: owner authority for a product call, or an exemption path. The choice stays with the maintainers.
Smaller points
v*tag ruleset is a maintainers' setting, so I'd track it apart from this issue rather than fold it in.ci-gatecan only depend on jobs inci.yml, and a required path-filtered workflow stays pending. So the firstci-gatePR will list them as outside the gate. I've added step 5b to the description: move their jobs intoci.ymlbehind a changed-paths job, soci-gatecounts them and allows a skip only when no matching path changed. That's a biggerci.ymlchange, so it's yours to accept or decline.The AI lane
Thanks for the support on the order, stacked-PR CI and the advisory lane. The scorecard already lists approvals with no human comment. I'll count them the way you put it: approvals with no human comment after the AI report lands.
🤖 Generated with Claude Code
fanhongy commentedon Nov 17, 2026
Thank you, and where this lands
Thanks to everyone who reviewed this. Each review changed the plan:
mainand read the logs of every failed job. That review found the changelog bug (fixed first, in fix: preserve hand-written changelog during releases #1354) and corrected what blocks a merge today. It also reviewed the fixes that merged today.Dependency Securityfast path.The design in one picture
Three principles:
The implementation plan
Statuses: Done, In review, Built (opens next), Measuring, Proposed, Maintainers' call.
~/.awsand tmuxHOME, noCAO_*ci-gate, advisory firstci-gateci-gateand the scannersExpected outcomes
These are targets, not promises. Each one is measured, and the scorecard reports it.
CIruns were re-runs (11-15 and 11-16; about half may be fork-PR approvals, not failures)main)main(11-05 to 11-16) touchedCHANGELOG.mdmain(11-16), and 110 up-to-date skips by bypass in 30 daysmainin 30 days were bypasses (38 of 100 since 11-03)ci-gateand the scanners block, but only after a clean windowWhat we need from maintainers
ci-gateand the scanners become required? Who owns new-advisory failures, and with what turnaround? ForDependency Security: owner authority for a product call, or an exemption path?mainruleset's bypass list, and does admin bypass become break-glass only, with a stated reason?🤖 Generated with Claude Code
plauzy commentedon Nov 18, 2026
The Dependency Security gate is failing 14 open PRs today (#1397, #1395, #1394, #1392, #1391, #1389, #1388, #1387, #1386, #1340, #1333, #1331, #1330, #1022). None of them touch picomatch. The cause is CVE-2026-112366 (GHSA-fc5q-39xc-vjpf), which affects every picomatch release through 4.0.7, and no fix has been released. Main is green only because its last scan ran before the advisory reached the Trivy database. The same code went red at 22:44Z.
This is the determinism problem this proposal describes. I propose keeping the #884 bar but moving where it fails:
.trivyignore.yaml: each entry needs an advisory link, a reachability statement, and anexpired_atdate at most 14 days out. After that date the check fails again.The picomatch case would be the first exception. CAO pulls it in only through Vite, Vitest, Docusaurus and patch-package. All its glob patterns come from config files in this repo, so the ReDoS input (the pattern itself) is not attacker-controlled.
I can open the PR once maintainers agree. The braces adapter from #884 stays as it is.
fanhongy commentedon Nov 18, 2026
Thanks @plauzy, +1 from me.
This answers the
Dependency Securitypart of Q1: an exemption path with an advisory link, a reachability statement and an expiry of at most 14 days, while the full scan keeps running on main and on the weekly schedule. The split in #1398, between what CAO ships and the build toolchain, fits the plan too, and the fail-closed rule for an empty shipped set is a good guard.When it lands, step 6 can treat
Dependency Securitylike the other scanners: required throughci-gate, with.trivyignore.yamlas its exemption path. I'll update the plan in the description when your PR is up.🤖 Generated with Claude Code
ci: cancel superseded pull_request runs (#1350) (#1386)
plauzy commentedon Nov 18, 2026
Follow-up to my earlier comment (#1350 (comment)): the Dependency Security part is now tracked in #1398 and implemented in #1400, and it took a different shape from what I proposed there.
uv export --no-dev, the packages bundled into the web, MCP Apps and docs builds, and the Rust runtime tree. Findings in build and test tooling are reported, and the weekly run keeps a tracking issue for them. Fixable toolchain findings that a PR adds still block. Malware always blocks..trivyignore.yaml. Each one needs a package or path scope and expires after 14 days, so there is no permanent blanket exemption. This replaces my earlier points 2 and 3, which read as contradicting each other.Review is welcome on #1400.