Skip to content

[Proposal] A deterministic, gated PR pipeline: trustworthy CI first, then required gates, with AI review as an advisory layer #1350

Open

Description

@fanhongy

Summary

  • Goal: a PR flow that is deterministic, gated, and trusted by every maintainer, so an approved PR can be merged safely.
  • How: deterministic checks plus one human approval on the exact head commit decide every merge.
  • AI review speeds people up, but it never decides.
  • Every step below is a small, separate PR. Maintainers can accept or decline each one on its own. Nothing here is decided. The plan has been revised per the reviews in the comments below.

This is the broader "PR assurance flow" suggested in a comment on #464. #464 becomes its AI-review layer (gate 3 below).

The layering follows the approach the Claude Code team describes in How the Claude Code team does code review: several independent layers, each catching what slips past the one before, with people kept for direction, risk and accountability.

Why now

All of these come from public data: the workflow files and rulesets on main, the Actions run history, and the PR and commit history.

The target flow

PR ─► 1 ci-gate: tests, lint, scanners; registered flakes get one automatic retry  (deterministic, required)
   ─► 2 risk label from the changed paths; high-risk paths need their code owners  (deterministic)
   ─► 3 AI review in Actions (#464), posted as an advisory comment                 (advisory only, never required)
   ─► 4 human approval on the exact head commit (dismissed on push, as today)
   ─► 5 merge queue: tests the real merged result, then merges                      (deterministic)

Terms used here:

  • ci-gate: one aggregate job in ci.yml. It runs with if: always(). It passes only if every gated job succeeded, and a skip passes only where it's allowed (for example dependency-review on push and merge_group). It includes Dependency Security, as SECURITY.md asks. Rulesets can then require one check instead of 20.
  • Flake registry: a small checked-in list of known flaky tests. Each entry has an open issue, an owner and an expiry (at most 30 days). A registered test that fails is retried in the same job, and the job summary says so. An unregistered failure stays red. An expired entry fails CI. Built only if the measured window shows recurring flakes.
  • Changelog fragments: one small file per PR (for example changelog.d/<slug>.md). They are folded into CHANGELOG.md at release time, so PRs stop conflicting on the same lines.
  • Merge queue: GitHub's queue that tests each PR against the latest main plus the PRs ahead of it, then merges. PRs no longer need "update branch" loops. It would use squash, to match the squash-only and linear-history rules, and every required workflow gets merge_group:.

The phases

Each step is a separate, small PR.

Phase Step Status
1. Make CI trustworthy 1. Fix the known flaky tests at their root. In flight: #1346 and #1349 merged. Next: #1341 tests 2–3, then #704.
1a. PR-only concurrency with cancel-in-progress (never on main pushes, which the release preflight reads). proposed
1b. The coverage ratchet reads the 3.12 leg (3 suite runs become 2). It saves runner time and one chance to flake per push, not wall time: the suite jobs already run in parallel. proposed
1c. A nightly Python 3.14 job that isn't required, so 3.14-only bugs show up without slowing PRs. proposed
1d. Keep the suite away from the real ~/.aws and tmux. in place (#1231, #1245); kept as a rule
1e. Sweep the short wall-clock limits in tests (.wait() and wait_for timeouts of 5 seconds or less): triage the list first, then fix the risky ones in small PRs. It doesn't hold up the window. proposed
1f. An order and isolation canary: a nightly job that isn't required, with the test order shuffled (the seed printed, through a small conftest.py hook, so no new dependency), a fresh HOME and no CAO_* variables. A failure gets an issue and a root fix, never a retry entry. Its own PR, after 1c. proposed
2. Measure 1–2 weeks after tests 2–3; add a flake registry (one automatic retry, an expiry, an issue link) only if flakes recur. proposed
3. CI on stacked PRs: plain pull_request: with no branch filter, in the four workflows above. Safe while "up to date" stays on; drop it only together with a merge queue (step 8). proposed
4. After the changelog fix (#1353): fragments or generated notes (Q3). proposed
5. The ci-gate job, advisory at first. proposed
5b. Bring the seven path-filtered workflows (the four provider tests, the two robotics examples and the docs site) under ci-gate: move their jobs into ci.yml behind a changed-paths job, so ci-gate counts them and allows a skip only when no matching path changed. Until then, they sit outside the gate. proposed; maintainers' choice
2. Switch the gates on Before 6. The bypass policy (Q8): who stays on the bypass list of each main ruleset, and admin bypass as break-glass only, with a stated reason. Otherwise every required check below can be skipped. maintainers' choice
6. Require ci-gate and the scanners (gitleaks, zizmor, cargo-deny), next to CodeQL, with a named owner and an expected turnaround for new-advisory failures. Trivy (Security Scan) and cargo-deny can take an ignore entry with a reason and an expiry. Dependency Security takes no ignore file and blocks advisories with no published fix, so its owner needs either the authority to make a product call (as in #1119), or an exemption path that the script and SECURITY.md allow. after Phase 1 shows CI is stable
7. A deterministic risk-label script from the changed paths, seeded from gutosantos82's tested path rules. Extend CODEOWNERS (workflows and actions are already owned) to the release scripts, trust-boundary code and the scanners' policy files; land this before step 6. proposed
8. A merge queue (squash; merge_group triggers), the recommended choice. Non-strict mode (drop "branch must be up to date") leaves stacked PRs untested against main. maintainers' choice
3. Standardize the AI review 9. A checked-in review rubric (for example REVIEW.md) and a machine-readable findings format. later
10. The #464 reviewer in Actions, advisory; started by hand first, then automated. A single agent. It records which review angles ran and marks a partial review as partial; cancels stale runs and keeps one comment updated in place; and checks its pinned model first, ending as a neutral check instead of hanging. later
11. A reviewer eval: 12–30 labelled cases from past PRs, offered by gutosantos82 (reports tied to exact head commits, with the maintainers' verdicts). It can score any reviewer, so it can start before steps 9–10. Accepted; case format proposed in the comments
12. Copilot code review: make it post or remove the ruleset. If it posts, compare it with step 10 in the shadow run. later
4. Measure and tune A scorecard: human wait time, rounds per PR, re-runs, conflict merges, escaped defects, and whether approval is becoming a formality (time from AI report to approval, approvals with no human comment). It shows what to tighten or relax. ongoing

Order matters. Requiring ci-gate before the flaky tests are fixed (and the registry is in place, if it's needed) would turn today's flakes into hard merge blocks.

What stays human

  • The approve click on the head commit.
  • Scope and design calls, including "should we do this at all?".
  • Changes to high-risk paths (CI, release, security and trust-boundary code), through CODEOWNERS.
  • Every policy switch: required checks, the merge queue, the registry's entries, the risk tiers, and the AI lane.

Security notes for the AI lane (#464)

This plan builds on #464's security boundary, with one difference. The AI lane stays advisory. It is never a required or blocking check, because model output isn't deterministic. #464 proposes blocking on validated, high-confidence Must Fix findings. This plan doesn't adopt that: an AI finding never blocks a merge on its own. When a finding is real, the human reviewer acts on it.

The suggested shape:

  • One workflow, two jobs. The review job reads the key and has read-only permissions. The post job writes and has no key.
  • The key lives in an environment whose branch rule allows only main, so a pull_request run (whose workflow file comes from the PR) can never receive it.
  • The first trigger is a manual dispatch from main. Any later automatic trigger uses a workflow defined on main, never checks out or runs PR code, and is recorded in the zizmor config with a reason.
  • One agent, read-only tools. No shell, no MCP servers, no web access, and the CLI's sandbox mode on.
  • Base-branch config. The job checks out the base commit. The reviewer's agent config comes from the base branch. The PR's diff and files are plain data in a folder outside the working directory. No PR code runs.
  • Schema-checked advisory output. The post job validates the output against a fixed schema, scans it for secrets, and renders it with a fixed template. It keeps one comment, updated in place for each new head commit, and a neutral check that is never required.

Questions for maintainers

  1. Which checks should become required, and when: ci-gate, the scanners, or both? Who owns new-advisory failures, and what turnaround do they commit to? For Dependency Security, should its owner have the authority to make a product call when an advisory has no fix, or should the script and SECURITY.md gain an exemption path?
  2. A merge queue, or non-strict mode?
  3. Changelog fragments, or generated notes from commit subjects (git-cliff), or both? Which fragment tool?
  4. Which paths should CODEOWNERS cover as high-risk, and who owns them?
  5. The flake registry policy: how many retries, how long an entry may live, and the maximum number of entries?
  6. For the AI lane: who owns the Kiro API key and the subscription that pays for it?
  7. Copilot code review: fix it or remove it?
  8. The bypass policy: who stays on the bypass list of each main ruleset, and should admin bypass become break-glass only, with a stated reason?

Related: #464, #1341, #704, #1346, #1349, #1353, #1231, #1245

Activity

  1. fanhongy commented on Nov 16, 2026

    @fanhongy
    CollaboratorAuthor

    @haofeif tell me what do you think.

  2. haofeif commented on Nov 16, 2026

    @haofeif
    Collaborator

    Thanks @fanhongy. I checked each claim against main at 6bf6c68e, the three rulesets, the Actions history of 11-15 and 11-16 (UTC), and the GitHub, zizmor and git-cliff docs. The direction is good, and the order is correct: fix the flaky tests first, then make the checks required. Below are the confirmed facts, new data, some corrections and the trade-offs I see. Nothing here stops the plan.

    Confirmed

    Claim What I found
    Only the four CodeQL checks are required, with "up to date" Correct. protect-main has strict_required_status_checks_policy: true.
    CI median is about 40 minutes Correct. The median is 41.1 min over 63 completed CI runs.
    The Python suite runs 3 times per push Correct. ci.yml:70 runs it in both Unit Tests legs, and ci.yml:310 runs it again in CAO MCP Apps.
    Re-runs often go green Correct. Over the two days, 19 of 66 CI runs were re-runs, and 15 of them ended green.
    CHANGELOG churn Correct. I count 29 of 77 first-parent commits from 11-05 to 11-16.
    Four workflows run only on PRs to main Correct for ci.yml, secret-scan.yml, cargo-deny.yml and zizmor.yml. #1339's base is #1333's branch.
    One check instead of 20 A PR run has 21 jobs. Four of them are CodeQL.
    12 of 47 open PRs were BEHIND A point-in-time count. I did not check it again.

    New data for Phase 1

    I read the logs of the 15 jobs that failed before a re-run in those two days. Each job failed on exactly one test, and each of these tests is in #1341:

    #1341 test Failed jobs Status
    1. BM25 default sort 6 Fixed by #1346 (merged 11-16)
    4. Run-view event order 5 Fixed by #1349 (merged 11-16)
    2. Replay of a deferred cleanup 3 Open
    3. Early-completion mask 1 Open

    The two #704 tests did not fail in this window. From this data:

    • The phase table is out of date: test(workflow): wait for the terminal event in the live run-view test (#1341) #1349 is merged.
    • The two fixes on main remove 11 of the 15 failures. Tests 2 and 3 caused the other 4.
    • After tests 2 and 3 are fixed, measure one or two weeks of CI. Then decide if the flake registry is necessary.
    • Each run of the suite gives a flaky test one more chance to fail a push. The failures came from all three jobs (3.11: 4, 3.12: 6, MCP Apps: 5). If the coverage ratchet reads the coverage report of the Unit Tests (3.12) leg, each push runs the suite one time less.
    • ci.yml has no concurrency group, so a new push to a PR does not stop the old run. cancel-in-progress for pull_request events is a small change.

    Corrections

    • More than CodeQL blocks a merge. protect-main also has a code_scanning rule: CodeQL alerts at errors, or security alerts at high_or_higher, block the merge. It also requires one approval, dismisses stale approvals on push, and has require_code_owner_review: true.
    • CODEOWNERS exists, and code-owner review is already required. It covers .github/workflows/, .github/actions/, .github/CODEOWNERS, patches/, vendor/, the three braces-compat/ folders, scripts/dependency_security.py and scripts/test-braces-security.cjs. The owner is @awslabs/multiq. So step 7 extends the coverage. Candidates: scripts/bump_version.py, scripts/release_preflight.py, scripts/release_commit.py and the trust-boundary code under src/.
    • An AI reviewer already runs on each push. The Copilot Auto Review ruleset has review_on_push: true on all branches (not drafts). Gate 3 adds a second AI reviewer. Please say if [Feat] Add autonomous headless PR reviewer to CI #464 replaces Copilot review or runs next to it. The Phase 4 scorecard can compare them.
    • The required CodeQL checks come from ci.yml. They are its codeql job (ci.yml:1068-1087), not default setup. This helps step 8: a merge_group: trigger in ci.yml covers them.
    • SECURITY.md:151-153 already asks for Dependency Security as a required check in the main ruleset, next to CodeQL. protect-main does not require it. If ci-gate includes the dependency-security job, step 6 also closes this gap.
    • Merges are squash-only. The branch protection ruleset allows only squash, and protect-main requires linear history. A merge queue should use squash to match.

    Trade-offs by step

    Step 3 (CI on stacked PRs). Each push to a stacked PR starts a full run of about 41 minutes. Also, when the parent merges, GitHub deletes its branch (delete_branch_on_merge is on) and retargets the child to main. That is a pull_request edited event. By default, a workflow runs only on opened, synchronize and reopened, so no new run starts. Strict mode makes the author update the child, and a merge queue tests the merged result. Both re-test the child against main. With non-strict mode, nothing re-tests it before the merge. So decide step 3 together with step 8.

    Steps 5 and 6 (ci-gate and required scanners).

    • Put if: ${{ always() }} on the gate, and fail it when a needs.*.result is failure or cancelled. Without always(), a failed job makes the gate skip, and GitHub counts a skipped check as a pass (troubleshooting required status checks).
    • Decide what skipped means. dependency-review skips on push by design (ci.yml:1056), and it will also skip on merge_group. Any other skip should fail the gate.
    • Do not add a path filter to a required workflow. A required workflow that a path filter skips stays "Pending" and blocks the merge (same page). The three scanners have no path filters today.
    • Some scanners fail on new advisories in dependencies that a PR does not change. SECURITY.md:94-96 documents this for Dependency Security. cargo deny check runs its advisories check on each PR, and tui/deny.toml also denies yanked crates. When these checks are required, a new advisory blocks all merges until a lock bump or an ignore entry lands. This is the documented intent, but the plan needs an owner for that fast path. The flake registry does not cover it.

    Step 8 (merge queue or non-strict mode).

    • The queue tests the real merged result and removes the update loop. Each required workflow needs a merge_group: trigger (managing a merge queue). This includes secret-scan.yml, cargo-deny.yml and zizmor.yml if step 6 requires them. Each queue group runs the full CI, and a failed check removes the PR from the queue. So Phase 1 is as important for the queue as for required checks. If you turn off the queue setting "Only merge non-failing pull requests", a later group that passes can absorb some intermittent failures.
    • Non-strict mode adds no CI runs. But two PRs that pass alone can break main together, and a stacked PR can merge on checks from its old base (step 3). The push run on main finds the break only after the merge.

    Step 4 and Q3 (changelog). scripts/bump_version.py:141 runs git-cliff --tag vX -o CHANGELOG.md. With -o, git-cliff writes a new file from the git history; its docs use --prepend to add to an existing file. So the release step does not read the hand-written [Unreleased] notes, and it will not read fragments. scripts/release_commit.py:43 then commits that file.

    • The last successful release.yml run was on 2026-08-28 (v2.5.0). No release.yml run cut v2.5.1 to v2.5.4. So all notes since 2.5.0, about 1,850 lines, are under [Unreleased].
    • The next successful run replaces them with generated sections, one bullet per commit subject. The tests stub generate_changelog (test/scripts/test_bump_version.py:181, :269), so no test catches this.
    • Whatever Q3 decides, fix generate_changelog first. This can be its own issue, because the weekly scheduled release has the problem without this plan.
    • release.yml also creates each GitHub Release with generate_release_notes: true. So notes generated from PR titles already exist there.

    Phase 3 (AI lane security).

    • GitHub runs a pull_request workflow in the context of the PR merge commit (the events docs compare this with pull_request_target). So the workflow file comes from the PR. A same-repo branch can change the review job and read the model key before any review. "Base-branch config" protects the agent config, but not the workflow file.
    • If each user who can push a branch may have the key, write that in the threat model. If not, put the key in a GitHub environment with required reviewers. Environment secrets are available only after its protection rules pass (environments). The cost is one approval per run.
    • zizmor reports workflow_run as a High dangerous-triggers finding (audit docs). The repo gate fails at medium and higher (.github/workflows/zizmor.yml:42-44), and .github/zizmor.yml requires a written reason for each ignore.
    • For same-repo PRs and manual runs, two jobs in one workflow give the same split. The review job has the key and contents: read. The post job has pull-requests: write and no secret. So the first stage may not need workflow_run. If you keep it, treat the artifact as untrusted, because the PR run can upload any file with that name. The schema check and the fixed template already cover the rendering.

    The questions

    Q4 and Q6 need owners. The maintainers must decide them, and I have no data to add. For the others, my view from the data:

    1. Q1: add ci-gate as advisory first, as proposed. Make it required after Four tests fail intermittently in CI on unrelated PRs (memory BM25 default sort, two terminal-service timing tests, workflow run view) #1341 tests 2 and 3 are fixed and a measured window is clean. Add the scanners at the same time, with an owner for advisory failures.
    2. Q2: the merge queue, after Phase 1. Non-strict mode is weaker for stacked PRs (step 3).
    3. Q3: fix generate_changelog first. Then use fragments if the hand-written notes are the product, or generated notes if PR titles are sufficient.
    4. Q5: set the retry count, the expiry and the cap from the measured window. These two days had 4 distinct flaky tests.

    🤖 Reviewed with CAO review workflows

  3. fanhongy commented on Nov 16, 2026

    @fanhongy
    CollaboratorAuthor

    Thanks @haofeif, this is very useful, and you're right on almost every point. I checked them against main, the rulesets and the docs. Here's where I land.

    What I accept

    • The phase table and the numbers. test(workflow): wait for the terminal event in the live run-view test (#1341) #1349 is merged, and Four tests fail intermittently in CI on unrelated PRs (memory BM25 default sort, two terminal-service timing tests, workflow run view) #1341 tests 2 and 3 are next. I'll use your larger sample (41 min over 63 runs; 19 of 66 re-runs, 15 green; 29 of 77 commits). "12 of 47 BEHIND" was a lower bound: 29 PRs hadn't been computed yet.
    • Measure before building the flake registry. After tests 2 and 3, we measure one to two weeks against a rule written in advance, and build the registry only if flakes recur.
    • concurrency. Cancel in progress for pull_request only. Pushes to main must never be cancelled, because the release preflight needs ci.yml green at the exact release SHA. Merge-queue runs aren't cancelled either.
    • Coverage de-dup. The 3.12 leg writes coverage.json, a small ratchet job reads it together with the vitest summary, and the third suite run goes away. That leg also runs the Trivy-gated tests, so the figure can only rise against the floors.
    • More than CodeQL blocks a merge, and CODEOWNERS already exists. I'll fix both statements. Step 7 extends CODEOWNERS rather than adding it. One addition to your list: the gates' own policy files aren't owned yet (.github/zizmor.yml, .gitleaks.toml, tui/deny.toml, .coverage-baseline.json, mypy-baseline.json, .github/dependabot.yml). Once the scanners are required, those should be owned before the switch, or an ignore entry becomes a one-approval way around a required check.
    • ci-gate. It gets if: always(). It passes only if every gated job succeeded, and a skip passes only from an allowlist (dependency-review on push and merge_group). It includes Dependency Security, which closes the SECURITY.md gap, and the required check is pinned to the Actions app. No path filters.
    • Scanners need an owner for new advisories. Agreed. The Trivy gate in the security job has the same property as Dependency Security and cargo-deny. The fast path would be an ignore entry with a reason and an expiry, reviewed by a code owner. The flake registry never covers these.
    • Merge queue. Squash, with merge_group: on every required workflow. One detail: secret-scan.yml only scans on pull_request and schedule, so on merge_group the gitleaks job would run no scan and still pass. It needs a branch that scans the group's range.
    • Stacked PRs (step 3) and step 8. Agreed, they go together. Step 3 is safe while "up to date" stays on, and "up to date" should only be dropped together with the queue. I'd rather not add the edited type, since every title edit would start a full run.
    • generate_changelog. Confirmed, and it has already happened once: the v2.5.0 release commit replaced the hand-written [Unreleased] notes with commit subjects, including the full GHSA-6m35-gcf5-xm75 write-up. The weekly run currently refuses because the next bump is a major, so the step would fire on the manually dispatched 3.0.0 release. I'll open it as its own issue with a small fix: promote [Unreleased] to the version heading deterministically, drop git-cliff from the release job (the GitHub Release already has generated notes), and test the real function instead of a stub. --prepend alone wouldn't fix it.

    Your direct questions

    Copilot and #464. In a sample of 17 recent PRs (15 with review data), the Copilot ruleset posted no reviews and no comments. So it's configured, but it isn't producing anything. I'd suggest the maintainers first either make it post (it may be a billing or policy setting) or remove the ruleset. If it does post, run it next to the #464 reviewer in shadow mode on the same 20–30 PRs, compare them in the Phase 4 scorecard, and then keep one automatic lane. Both stay advisory, and neither approves. If Copilot works, it has a real advantage: there's no key to protect.

    The key model. You're right. Under pull_request, the job that holds the key is the PR's own workflow file, so "base-branch config" didn't protect it. I also think an environment with required reviewers doesn't fully fix it: the approver approves a job the PR wrote, once per push. The proposal I'd make instead:

    • the key lives in an environment whose deployment-branch rule allows only main (no refs/pull/*), with admin bypass disallowed. GitHub matches that rule against the run's GITHUB_REF, so a pull_request run can never receive the key, whatever its workflow file says;
    • one workflow, two jobs, as you suggested: the review job has the key and contents: read, and the post job has pull-requests: write and no key. The model's output is still treated as untrusted (schema, secret scan, fixed template);
    • the first trigger is a manual workflow_dispatch from main for shadow mode. That avoids workflow_run, and zizmor stays clean;
    • automation comes later and is a choice for you: pull_request_target with a base-only checkout and a reasoned zizmor entry, or a small no-secret PR job that dispatches the main-defined workflow.

    What I'd like to discuss

    • The merge-queue setting "Only merge non-failing pull requests": I'd keep it on while we measure, so flakes stay visible. Turning it off can make sense afterwards.
    • Q4 and Q6 (CODEOWNERS owners and the key) are yours to decide, as you said.

    Next steps

    1. Open the changelog issue.
    2. Fix Four tests fail intermittently in CI on unrelated PRs (memory BM25 default sort, two terminal-service timing tests, workflow run view) #1341 tests 2 and 3.
    3. Then small PRs for concurrency, the coverage de-dup and an advisory ci-gate.
    4. Measure, then come back with the numbers for Q1 and Q5.

    I'll update the issue body with these corrections.

  4. haofeif commented on Nov 16, 2026

    @haofeif
    Collaborator

    @plauzy @tedswinyar @gutosantos82 please have a look at this proposal

  5. gutosantos82 commented on Nov 16, 2026

    @gutosantos82
    Contributor

    Thanks @fanhongy, and @haofeif for the data. I support the plan and the order. I've been running a multi-agent CAO review pipeline (examples/pr-review/ on my fork) on this repo's PRs since August, about 1,000 reports. Below is what that adds, mostly for Phase 3.

    Where our runs back the plan

    Gaps for the AI lane (Phase 3)

    1. Record how much of the review ran. 484 of our 1,066 reports say at least one review angle didn't return. In the worst cases (fix: keep handoff's worker reachable when provider startup outlasts the create timeout #773 and feat(agent-plugins): creator and audit skills for Agent Plugin packages, shipped as a third package #1200 on 11-03), the whole fan-out failed and the supervisor still wrote a confident "Approve with nits" on its own. The step 9 format should list the angles that ran and say whether dynamic verification ran. A partial review should be marked as partial, or not posted. The angle that went missing most often was dynamic verification.
    2. The tool limits may not fit a CAO reviewer. A CAO multi-agent reviewer fans out through cao-mcp-server. Under kiro's sandbox mode, every assign failed silently from September until fix(orchestration): load an unreadable child profile from cao-server #1213. If [Feat] Add autonomous headless PR reviewer to CI #464 is a CAO team, it needs cao-mcp-server allowed (and fix(orchestration): load an unreadable child profile from cao-server #1213). If it's a single agent, the "no MCP servers" rule is fine. It would help to state which.
    3. Heads move during a review. Our driver logged 445 "no report at current head" outcomes against 887 completed reviews, because authors push mid-review. Use step 1a's cancel-in-progress for the AI workflow as well. Also prefer one comment updated in place to one comment per head: feat(decisions): pick helper CLIs and models locally from task features (off by default) #1202 went through four rounds.
    4. Fork PRs. Most of the PRs we saw come from forks. Their CI waits for a maintainer's approval, and their pull_request runs get no secrets. This affects both the queue's speed and the choice of trigger for the AI lane.
    5. Pinned models get retired. A retired model once left our review sessions stuck. Check the model before the run, and end as a neutral check, not a hang.
    6. Seed for step 7. Our publish script already decides "sensitive" from a path regex (provider status detection, auth and security, the tool-permission surface, release and CI) plus size limits. It could be a tested starting point for the risk-label script.

    Two cautions

    Offer for step 11 (reviewer eval)

    I have a labelled set: reports tied to exact head commits, what was posted, and other maintainers' verdicts at the same heads, including disagreements (e.g. #886, #1150). I can pull 12–30 cases into a small eval if that helps.

    On the questions: Q2, a merge queue. Q3, fragments. Q4 and Q6 are for the maintainers.

  6. fanhongy commented on Nov 17, 2026

    @fanhongy
    CollaboratorAuthor

    Thanks @gutosantos82, this is very useful data. I'd like to take you up on the step 11 offer.

    Step 11: the reviewer eval. Yes, please pull the 12–30 labelled cases. To make them reusable by any reviewer we try (the #464 lane, Copilot code review, or your pipeline), a format like this would work:

    • the PR number and the exact head SHA that was reviewed;
    • what the reviewer posted (the findings and the verdict);
    • the maintainers' verdict at the same head, including where they disagreed (your feat: add Cohesivity backend provisioning example #886 and feat: add optional Cooper Email inbox example #1150 examples);
    • for each finding: whether it was real, a false positive, or a miss (something a maintainer found that the reviewer didn't), with its severity;
    • whether the review was complete or partial (which angles ran, and whether dynamic verification ran).

    Partial reviews and missed findings matter most here, since those are what a scorecard should catch. A small JSON or Markdown file per case is fine. Where should they live: in your fork for now, or as a small test/review_eval/ folder in a PR, so the eval runs against the same cases every time the rubric or the runner changes?

    The rest of your comment. I'll fold your points into the plan and the issue body:

    On your question: the #464 lane as proposed is a single agent with read-only tools and no MCP servers, so the sandbox assign limit doesn't apply. If it ever becomes a CAO team, it will need cao-mcp-server allowed and the #1213 fix, and I'll note that in the plan.

    🤖 Generated with Claude Code

  7. fanhongy commented on Nov 17, 2026

    @fanhongy
    CollaboratorAuthor

    Step 2: the measuring window (rule v2, revised 2026-11-17 after plauzy's review)

    Changed from v1: the floor is 150 finished runs, not 40; every red job is classified, with a limit on unclassified jobs; timeouts are classed by the step they hit; one cause counts once per run; push runs on main are counted; and a fail now says what happens to the clock.

    So the result can't be argued afterwards, here is how we'll measure CI before deciding on the flake registry and on making ci-gate required:

    • Start: when test(terminal): stabilize the deferred-cleanup replay test (#1341) #1357, test(terminal): assert early-completion ordering instead of elapsed time (#1341) #1358 and test(vault): make the projection re-entry test load-safe (#1341) #1368 merge. Those are the open root fixes for Four tests fail intermittently in CI on unrelated PRs (memory BM25 default sort, two terminal-service timing tests, workflow run view) #1341 tests 2 and 3 and for the vault re-entry test.
    • Length: at least 7 days, and at least 150 finished pull_request runs of CI. A finished run ended in success or failure. Cancelled runs don't count toward the 150. If both aren't met within 14 days, we report what we have and extend.
    • push runs on main: counted too, on their own row, and their flaky failures count toward the limit.
    • Every red job gets a class: flaky, real, outage, or unclassified. That includes jobs that failed inside a run that was later cancelled. We classify each one from its job log, with the run link and the failing line, and any maintainer can overrule a class. The measuring script saves each failed job's log when it happens.
    • Flaky: a failed job whose failing test then passes on a re-run of the same commit, or that fails on a PR whose change can't affect it while the same test passes on main.
    • Outage: a download, mirror or runner failure. Outages aren't counted as flaky. We list them separately, with their own count by source.
    • Timeouts: classed by the step they hit. A timeout in a test step counts as flaky (or real, if the PR's change can cause the hang). A timeout in an install or download step is an outage.
    • Unclassified: at most 2. With more, the window is inconclusive and extends. It neither passes nor fails.
    • Counting: one cause counts once per run for the limit, even if it fails several jobs in that run. The report also keeps the per-job counts, so the numbers stay comparable if a change like 1b (one fewer suite run per push) lands during the window.
    • Pass: at most 1 flaky failure. Then step 6 can go ahead, and no flake registry is needed.
    • Fail: 2 or more. Each newly found flaky test gets an issue and a root fix, and we revisit the registry (Q5). The window still runs to its planned end, so the report has the full counts. A fresh window, under this rule, starts once those root fixes merge.
    • The two tests in Two tests fail intermittently on pristine main, roughly 1 in 3 and 1 in 4, reproducible in isolation #704 aren't fixed yet. Their failures count in the window; it doesn't wait for them.

    At the end, we'll post the counts per run, per job and per test, the outage list and the unclassified jobs, with links to the runs.

    🤖 Generated with Claude Code

  8. plauzy commented on Nov 17, 2026

    @plauzy
    Collaborator

    Thanks @fanhongy. I support the plan and the order, and writing the rule down before the window starts is right.

    Everything below is new to this thread. I agree with the rest of the plan and with the earlier reviews, and I haven't repeated them.

    Legend: 🆕 new feedback · ❓ new open question · ✏️ correction to something already accepted

    # Item Type Needs a decision
    1 Classify every red job, and report what nobody classified 🆕 NEW · change to the step 2 rule Before the window starts
    2 Raise the 40-run floor 🆕 NEW · change to the step 2 rule Before the window starts
    3 What a fail does to the clock ❓ NEW QUESTION Before the window starts
    4 One cause can count twice ❓ NEW QUESTION Before the window starts
    5 Count the push runs on main 🆕 NEW · small addition Before the window starts
    6 Outages need their own limit 🆕 NEW · extends "outages listed separately" Before step 6
    7 The advisory fast path doesn't exist for Dependency Security ✏️ CORRECTION to the fast path in the first and second reviews Before step 6
    8 Sweep the short wall-clock limits 🆕 NEW · optional Any time

    The step 2 rule

    1 · 🆕 NEW · Classify every red job, and report what nobody classified

    The definition needs a re-run of the same commit. When an author pushes a new commit instead, that failure is never counted, and 1a (cancel-in-progress) will make that more common.

    A re-run on its own isn't the signal either. From #1349's merge to 2026-11-17 05:24Z there were 38 completed PR runs, and 5 went green after a re-run. Only 2 of those have a failed first attempt to read:

    The other 3 list no first-attempt jobs. So I'd add three things to the rule:

    • an unclassified count with its own limit;
    • who classifies;
    • a job that hits its timeout-minutes counts as flaky, not as an outage.

    2 · 🆕 NEW · Raise the 40-run floor

    At 40 runs, "at most 1" is consistent with a flaky rate of up to 11% of runs (95% upper bound). At 150 runs it is about 3%. A normal week gives 200 or more, so this only matters in the 14-day fallback, which is where the result is weakest.

    3 · ❓ NEW QUESTION · What does a fail do to the clock?

    Does the window restart, or continue once the fix lands?

    4 · ❓ NEW QUESTION · One cause can count twice

    Counting per job means one cause can fail two jobs in one run. Test 2 did that on #1343. Is that one failure or two?

    5 · 🆕 NEW · Count the push runs on main

    They are the closest thing we have to a merge-queue run.

    What the window won't measure

    6 · 🆕 NEW · Outages need their own limit for step 6

    Listing outages separately is right for the registry decision. For step 6 it isn't enough, because a required ci-gate is blocked by a mirror outage exactly as it is by a flaky test. On 2026-11-11 an apt outage turned five jobs red on main (#1303). I'd give the outage list its own limit in the step 6 decision. The known remaining exposure is tracked in #1375.

    7 · ✏️ CORRECTION · The advisory fast path doesn't exist for Dependency Security

    The plan's fast path is an ignore entry with a reason and an expiry. That works for the scanners that read an ignore file. Dependency Security can't take one today:

    • it runs Trivy with --ignorefile /dev/null;
    • SECURITY.md says an advisory with no published fix blocks.

    #1118 is the case to design for. The advisory had no fix on Python 3.10, so the fix was to drop 3.10 (#1119). That was a product decision, not a lock bump. Once the check is required, the owner needs either the authority to make that call, or an exemption path that the script and SECURITY.md allow. That is the maintainers' decision.

    Optional

    8 · 🆕 NEW · Sweep the short wall-clock limits before the window

    Three of the five known flaky tests are short wall-clock limits that fail under load. A rough grep of test/ on main finds about 100 .wait() calls of 5 seconds or less in 32 files. Many are harmless, but a sweep with the #1357 pattern may be cheaper than finding the rest one window at a time.

  9. fanhongy commented on Nov 17, 2026

    @fanhongy
    CollaboratorAuthor

    Thanks @plauzy. These are fair, and most of them tighten the rule before it starts, which is why we wrote it down. I checked the ones I could against main.

    The step 2 rule

    I've folded points 1–5 into the rule and updated the rule comment in place to the revised version. The window hasn't started, so nothing is measured under the old one.

    1. Classify every red job. Agreed. A re-run isn't a reliable signal, and 1a will make same-commit re-runs rarer. So:

    • Every red job in the window gets a class: flaky, real, outage, or unclassified. That includes jobs that failed inside a run that 1a later cancelled. Those jobs still show as failed.
    • I'll classify each one from its job log, with the run link and the failing line. Any maintainer can overrule a class.
    • The measuring script saves each failed job's log when it happens, so fewer end up unreadable.
    • unclassified gets its own limit: at most 2. Above that, the window is inconclusive and extends. Missing evidence shouldn't count as a pass, but it isn't a flake either.
    • Timeouts: agreed for test steps. The Unit Tests limit is there to bound a hung test (ci.yml, fix: stop orphaned pipe-pane writers and harden suite isolation (tmux gate, non-blocking FIFO opens) #1251), so hitting it is a test problem. A timeout in an install or download step is an outage, though: on 2026-11-11 the apt jobs ran 11 to 64 minutes before failing (ci: take apt off the browser jobs' critical path #1302). So a timeout is classed by the step it hit.

    2. The floor. Agreed. I get the same bounds: about 11% at 40 runs and about 3% at 150. The floor is now 150. It counts finished runs only, not cancelled ones. Once 1a lands, some runs will be cancelled, so the floor may matter outside the 14-day fallback too.

    3. A fail and the clock. The window still runs to its planned end, so the report has the full per-test counts for Q5. Then a fresh window, under the same rule, starts once the root fixes for the new tests merge. Step 6 waits for a clean window. Continuing across a fix would mix the before and after data.

    4. One cause, two jobs. One. For the limit, one cause in one run counts once, because a required ci-gate blocks a run once, however many jobs fail. The report still shows the per-job counts, which stay comparable if 1b lands during the window.

    5. push runs on main. Agreed. They go into the totals and the limit, on their own row. With "up to date" on, a red push run right after a green PR run on the same tree is strong flake evidence.

    What the window won't measure

    6. Outages and step 6. Agreed. The report will give outages their own count and rate, by source. I'd suggest the step 6 decision use the same bar for outages as for flaky failures, since both block a required gate the same way. The limit itself is yours to set.

    7. The Dependency Security fast path. You're right, and this corrects what I wrote earlier. On main:

    • scripts/dependency_security.py runs Trivy with --ignorefile /dev/null and --ignore-unfixed=false;
    • SECURITY.md says the gate blocks advisories with no published fix, and uses no repository ignore files.

    So an ignore entry only fits the other scanners: cargo-deny (through tui/deny.toml, which has no ignores today) and the Trivy gate in Security Scan (it already skips unfixed findings, and there's no .trivyignore today). I've corrected step 6 and Q1 in the description. For Dependency Security, the owner needs either the authority to make a product call like #1119, or an exemption path that the script and SECURITY.md allow. Q1 now asks the maintainers to choose.

    Optional

    8. The short wall-clock sweep. Worth doing, and it's now step 1e. I get a similar count: 98 .wait() calls of 5 seconds or less in 32 files under test/. The same grep misses about 30 more written as wait_for(..., timeout=...). We'll triage the list first. Then we'll send small PRs for the ones that guard a handoff between threads or tasks under load, using the #1357 pattern (an explicit barrier and a generous deadlock bound). The window won't wait for them.

    🤖 Generated with Claude Code

  10. tedswinyar commented on Nov 17, 2026

    @tedswinyar
    Member

    Thanks for writing this up, and for the review rounds already on it. I agree with the diagnosis and the order: fix CI, measure, then require it. Fixing generate_changelog first (#1353) is the right call too. Three things I don't see covered yet.

    1. "Required" does not block here today. The admin role is a bypass actor on both main rulesets, and nearly every collaborator is an admin. As of yesterday the rule-suites API showed 134 of 249 pushes to main in the previous 30 days recorded as bypasses (38 of 100 since 11-03, so it is improving). The up-to-date rule was skipped 110 times and the review rule 36 times, 8 of those with changes requested standing. My own #1036 is one of them. As written, ci-gate, the scanners, CODEOWNERS and the queue can all be skipped the same way. Before step 6, can we decide who stays on the bypass list (the release deploy key has to), make admin bypass break-glass with a stated reason, and put bypasses per rule on the Phase 4 scorecard? The audit trail already exists:

    gh api 'repos/awslabs/cli-agent-orchestrator/rulesets/rule-suites?ref=refs/heads/main&time_period=month'
    

    2. One failure class passes every proposed gate. While merging main into #1069 I hit two tests in test/security/test_origin_matrix.py that fail on a clean checkout of main when run after the main() tests, which leave http://localhost:9889 in the live CORS_ORIGINS list. CI is green because the order is fixed, and a retry registry would hide it. It still reproduces on main 5a8dc10f; the fix is #1381. A small canary job (random order, fresh HOME, no CAO_* in the environment) would catch this kind of thing before it becomes "re-run it".

    3. On Q1 (who owns the scanner fast path), one data point. @plauzy's correction is right that Dependency Security has no ignore route. The last time it went red for everyone (11-14, a new advisory), the fix went in without an approving review, by bypass. So today's fast path is the bypass. Whoever takes Q1, I'd write the expected turnaround next to the name, and count that path on the scorecard too.

    Smaller points:

    • Step 1b saves compute, not wall-clock: the three suite jobs already run in parallel at about 33 to 36 minutes each.
    • About half of the re-runs counted on 11-15 and 11-16 look like fork-PR approvals, not failures. Worth leaving those out of the re-run count now that every red job gets a class.
    • No ruleset covers v* tags, and release runs use the workflow file at the tag's commit, so none of these gates apply to a release. v2.5.4 is still a draft.
    • Seven path-filtered workflows that the contributing skill calls blocking sit outside ci-gate.

    I'd support requiring ci-gate once Phase 1 measures clean, running CI on stacked PRs, and keeping the AI lane advisory. For that lane, I'd add one measure to the scorecard: how often a PR is approved with no human comment after the AI report lands. If that number is high, the lane is replacing review, not advising it.

  11. fanhongy commented on Nov 17, 2026

    @fanhongy
    CollaboratorAuthor

    Thanks @tedswinyar. These are real gaps, and the bypass point matters most: a required check that most people can skip isn't really a gate. I checked what I could against main (5a8dc10f). I haven't re-run the rule-suites query, so I've taken those numbers as given.

    Where things stand: the measuring window started at 13:20Z today, when #1368 merged. 1a and 1b are open as #1386 and #1387. I've updated the description with the changes below.

    1. Bypass

    Agreed. Our own sample points the same way: of 60 merges from 11-05 to 11-16, 7 ended without an approval, and 4 of those 7 were ours. So this applies to us too.

    I've added a step before step 6, and a new question for the maintainers, Q8:

    • who stays on the bypass list for each main ruleset;
    • admin bypass as break-glass only, with a stated reason.

    I support making admin bypass break-glass, including for our own urgent fixes. The list itself is yours to decide. For Phase 4, the scorecard will show bypasses per rule, read from the rule-suites endpoint you gave. Your 110 up-to-date skips also support the merge queue (step 8): admins are skipping the very update loop the queue removes.

    2. Order-dependent failures

    Thanks for #1381. On main it reads as you describe: TestMainEntryPoint restores os.environ but not the CORS_ORIGINS list that add_local_cors_origins() appends to in place.

    On why CI stays green, one likely reason, from reading the code only (I haven't run it): CI runs the suite in path order, so test/api/ comes before test/security/. In between, test/providers/test_cursor_cli_unit.py calls importlib.reload() on cli_agent_orchestrator.constants, which rebuilds CORS_ORIGINS from scratch. So the leak is reset before the origin rows run.

    You're right that the window can't see this class. In a fixed order it never fails, and in a random order a retry would hide it. So we'll add a canary, now step 1f in the description:

    • a nightly job that isn't required, next to the 3.14 one;
    • the test order shuffled, with the seed printed so a failure can be reproduced. A small hook in conftest.py does the shuffle, so there's no new dependency;
    • a fresh HOME, and no CAO_* variables in the environment;
    • a failure gets an issue and a root fix, never a retry entry.

    It will be its own small PR, after the 3.14 job.

    3. Q1 and the scanner fast path

    Agreed. That 11-14 fix was ours, #1326. Q1 now asks for an owner and an expected turnaround. The scorecard will count each advisory fix: the time from red to merged, and whether it merged with an approval or by bypass.

    There's a live case right now. Since the vulnerability database updated around 21:39Z today, Dependency Security has been red on every PR, and it will be on main's next run too:

    • picomatch CVE-2026-112366 (GHSA-fc5q-39xc-vjpf, HIGH, affects <= 4.0.7) has no published fix. It's in the cao_mcp_apps, docusaurus, web and ag-ui-operator-dashboard lockfiles.
    • minimatch CVE-2026-118764 (GHSA-g3hv-75wg-m249, MEDIUM) is fixed only in 11.0.1. The one vulnerable copy is bundled inside aws-cdk-lib (examples/decision-model-host/aws/cdk). Even aws-cdk-lib 2.277.0, the latest, bundles minimatch ^10.2.6, so no bump on our side clears it.

    So neither one can be fixed from this repo today. That's exactly the Q1 choice: owner authority for a product call, or an exemption path. The choice stays with the maintainers.

    Smaller points

    • 1b. Right, it saves compute, not wall time. The suite jobs already run in parallel, and the new ratchet job waits for them. ci: the coverage ratchet reads the Unit Tests 3.12 report (#1350) #1387 says wall time stays about the same, and the description now says 1b saves runner time and one chance to flake per push.
    • Re-runs. The window doesn't count re-runs. It classifies every red job, so an approval with no failed job never enters it. On the scorecard, a re-run will only count when an earlier attempt had a failed job. I've added that caveat next to the 19-of-66 figure in the description.
    • Tags and releases. Agreed, the release path is outside these gates. A v* tag ruleset is a maintainers' setting, so I'd track it apart from this issue rather than fold it in.
    • Path-filtered workflows. Confirmed: seven of them (the four provider test workflows, the two robotics examples and the docs site). ci-gate can only depend on jobs in ci.yml, and a required path-filtered workflow stays pending. So the first ci-gate PR will list them as outside the gate. I've added step 5b to the description: move their jobs into ci.yml behind a changed-paths job, so ci-gate counts them and allows a skip only when no matching path changed. That's a bigger ci.yml change, so it's yours to accept or decline.

    The AI lane

    Thanks for the support on the order, stacked-PR CI and the advisory lane. The scorecard already lists approvals with no human comment. I'll count them the way you put it: approvals with no human comment after the AI report lands.

    🤖 Generated with Claude Code

  12. fanhongy commented on Nov 17, 2026

    @fanhongy
    CollaboratorAuthor

    Thank you, and where this lands

    Thanks to everyone who reviewed this. Each review changed the plan:

    The design in one picture

    PR ─► deterministic gates      ci-gate (every CI job + the scanners), behind a bypass policy   required
       ─► risk label + code owners high-risk paths need their code owners                           deterministic
       ─► AI review                one advisory comment per head commit, partial reviews marked     never required
       ─► human approval           on the exact head commit (dismissed on push)                     required
       ─► merge queue              tests the real merged result, then merges                        deterministic
    

    Three principles:

    1. Deterministic checks plus a human decide every merge.
    2. AI review only speeds people up. It never approves, blocks or decides.
    3. Measure before requiring. A check becomes required only after a clean measured window.

    The implementation plan

    Statuses: Done, In review, Built (opens next), Measuring, Proposed, Maintainers' call.

    Phase Step Status Link
    1. Make CI trustworthy 0. Keep the hand-written changelog at release Done; a follow-up is in review #1353, #1354, #1388
    1. Root-fix the known flaky tests Done #1346, #1349, #1357, #1358, #1368
    1. The two #704 tests Proposed (ours, next) #704
    1a. Cancel superseded PR runs In review #1386
    1b. The suite runs 2 times per push, not 3 In review #1387
    1c. A nightly Python 3.14 job, not required Built (opens next), with the suite fd-leak fix #1376 (done), #1389 (in review)
    1d. Keep the suite away from the real ~/.aws and tmux Done; kept as a rule #1231, #1245
    1e. Sweep the short wall-clock limits in tests Built (opens next), as small PRs —
    1f. An order and isolation canary: nightly, shuffled order, fresh HOME, no CAO_* Built (opens next), after 1c #1381
    2. Measure CI for 7–14 days Measuring since 2026-11-17 13:20Z the rule
    2. A flake registry Proposed, only if the window fails Q5
    3. CI on stacked PRs Built (opens next), with the Q2 answer Q2
    4. Changelog fragments or generated notes Maintainers' call Q3
    5. ci-gate, advisory first Built (opens next), after 1b —
    5b. Bring the seven path-filtered workflows under ci-gate Proposed; maintainers' choice —
    2. Switch the gates on Before 6. The bypass policy: who stays on the list; admin bypass as break-glass with a reason Maintainers' call Q8
    6. Require ci-gate and the scanners Maintainers' call. Trigger: 5 merged, a clean window, 7a landed, the bypass policy set, an advisory owner with a turnaround, and an outage limit Q1
    7a. Extend CODEOWNERS to the release scripts, trust-boundary code and the gates' policy files Maintainers' call, before 6 Q4
    7b. A deterministic risk-label script Proposed, after Q4 —
    8. A merge queue (squash), or non-strict mode Maintainers' call, after Phase 1 Q2
    3. Standardize the AI review 9. A review rubric and findings format Proposed (later) —
    10. The #464 reviewer in Actions, advisory Proposed (later), after 9 and Q6 #464, Q6
    11. A reviewer eval on 12–30 labelled cases Proposed (accepted); waiting for the cases —
    12. Copilot code review: fix it or remove it Maintainers' call Q7
    4. Measure and tune The scorecard Proposed; it starts from the window's measuring script —

    Expected outcomes

    These are targets, not promises. Each one is measured, and the scorecard reports it.

    Today Target Source
    19 of 66 CI runs were re-runs (11-15 and 11-16; about half may be fork-PR approvals, not failures) at most 1 flaky failure in at least 150 finished PR runs the window rule
    The full suite runs 3 times per push 2 times. This saves runner time (about 30 runner-minutes per push) and one chance to flake, not wall time 1b, #1387
    A new push leaves the old run going superseded PR runs are cancelled (never on main) 1a, #1386
    29 of 77 commits to main (11-05 to 11-16) touched CHANGELOG.md 0 changelog conflicts between PRs step 4, Q3
    At least 12 of 47 open PRs were behind main (11-16), and 110 up-to-date skips by bypass in 30 days 0 update-branch loops merge queue, Q2
    134 of 249 pushes to main in 30 days were bypasses (38 of 100 since 11-03) every bypass visible per rule, and break-glass only, with a reason Q8
    Only the four CodeQL legs are required; red tests and scanners don't block ci-gate and the scanners block, but only after a clean window step 6, Q1
    In one reviewer pipeline, 484 of 1,066 AI reports missed at least one review angle the AI review stays advisory, marks partial reviews, and is scored on a 12–30-case eval steps 10–11
    No shared numbers on review health a scorecard: human wait time, rounds per PR, re-runs, conflict merges, escaped defects, bypasses per rule, advisory-fix turnaround, and approvals with no human comment after the AI report Phase 4

    What we need from maintainers

    1. Q1. When do ci-gate and the scanners become required? Who owns new-advisory failures, and with what turnaround? For Dependency Security: owner authority for a product call, or an exemption path?
    2. Q2. A merge queue, or non-strict mode?
    3. Q3. Changelog fragments, generated notes, or both?
    4. Q4. Which paths do CODEOWNERS cover as high-risk, and who owns them?
    5. Q5. If the window fails: the registry's retries, expiry and cap?
    6. Q6. Who owns the AI lane's API key and its subscription?
    7. Q7. Copilot code review: fix it or remove it?
    8. Q8. Who stays on each main ruleset's bypass list, and does admin bypass become break-glass only, with a stated reason?

    🤖 Generated with Claude Code

  13. plauzy commented on Nov 18, 2026

    @plauzy
    Collaborator

    The Dependency Security gate is failing 14 open PRs today (#1397, #1395, #1394, #1392, #1391, #1389, #1388, #1387, #1386, #1340, #1333, #1331, #1330, #1022). None of them touch picomatch. The cause is CVE-2026-112366 (GHSA-fc5q-39xc-vjpf), which affects every picomatch release through 4.0.7, and no fix has been released. Main is green only because its last scan ran before the advisory reached the Trivy database. The same code went red at 22:44Z.

    This is the determinism problem this proposal describes. I propose keeping the #884 bar but moving where it fails:

    1. PR gate: scan the base and head lockfiles. Fail only on HIGH/CRITICAL findings the PR adds. Findings already on the base are listed in the summary as warnings.
    2. Main pushes and the weekly schedule: keep the full scan (dev dependencies and unfixed advisories, no exemptions). A failure opens or updates one tracking issue.
    3. .trivyignore.yaml: each entry needs an advisory link, a reachability statement, and an expired_at date at most 14 days out. After that date the check fails again.

    The picomatch case would be the first exception. CAO pulls it in only through Vite, Vitest, Docusaurus and patch-package. All its glob patterns come from config files in this repo, so the ReDoS input (the pattern itself) is not attacker-controlled.

    I can open the PR once maintainers agree. The braces adapter from #884 stays as it is.

  14. fanhongy commented on Nov 18, 2026

    @fanhongy
    CollaboratorAuthor

    Thanks @plauzy, +1 from me.

    This answers the Dependency Security part of Q1: an exemption path with an advisory link, a reachability statement and an expiry of at most 14 days, while the full scan keeps running on main and on the weekly schedule. The split in #1398, between what CAO ships and the build toolchain, fits the plan too, and the fail-closed rule for an empty shipped set is a good guard.

    When it lands, step 6 can treat Dependency Security like the other scanners: required through ci-gate, with .trivyignore.yaml as its exemption path. I'll update the plan in the description when your PR is up.

    🤖 Generated with Claude Code

  15. plauzy commented on Nov 18, 2026

    @plauzy
    Collaborator

    Follow-up to my earlier comment (#1350 (comment)): the Dependency Security part is now tracked in #1398 and implemented in #1400, and it took a different shape from what I proposed there.

    Review is welcome on #1400.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

No labels
No labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions