Skip to content

ci(security): block on shipped dependencies, report toolchain findings (#1398) - #1400

Open
plauzy wants to merge 1 commit into
awslabs:mainfrom
plauzy:ci/1398-depsec-shipped-gate
Open

plauzy wants to merge 1 commit into
awslabs:mainfrom
plauzy:ci/1398-depsec-shipped-gate

Conversation

@plauzy

@plauzy plauzy commented Nov 18, 2026 •

Copy link
Copy Markdown
Collaborator

Closes #1398

What this does

Dependency Security now blocks only on findings in code CAO ships. Everything else is still scanned in full (dev dependencies and unfixed advisories included), listed in the step summary and in a SARIF file, and tracked by one weekly issue. It does not fail the pull request.

  • Shipped set, measured from the build, not from package.json labels.
    • uv.lock: uv export --locked --no-dev --all-extras (the wheel's runtime closure plus the optional agui and otel extras, which users install; without --all-extras the set is 87 packages, with it 97).
    • web/ and cao_mcp_apps/: a Vite plugin (scripts/shipped-packages.mjs) records the npm packages each build bundled to <project>/.shipped/bundled-packages.<config>.json.
    • docusaurus/: a webpack plugin records the client bundle's packages, not the server render bundle's.
    • tui/Cargo.lock: cargo tree --locked -e normal --target all.
    • examples/**: ships nothing.
  • scripts/dependency_security.py classifies each finding shipped or toolchain. It exits 1 only on a shipped HIGH/CRITICAL not covered by a valid, unexpired entry in .github/dependency-security/trivyignore.yaml. Exit 2 (fail closed) if a producer fails or returns an empty set, if a tracked lockfile outside examples/ has no producer, or if the ignore file is invalid. It never reads "nothing shipped" as a pass.
  • Ignore file, .github/dependency-security/trivyignore.yaml (a deliberate deviation from the issue's literal .trivyignore.yaml): a .trivyignore.yaml at the repository root is auto-loaded by Trivy into the separate security job, and test_security_scan_steps.py::check_no_autoloaded_trivy_config rejects it. The file is read only by scripts/dependency_security.py; Trivy still runs with --ignorefile /dev/null and one call, so every fix(security): enforce full dependency checks and consolidate toolchain fixes #884 argument assertion holds. Each entry needs id, an https url, a statement, added and expired_at (at most 14 days after added, and added may not be in the future, so the limit cannot be stretched by back-dating). An entry without paths covers every lockfile and package carrying that advisory; the schema is unchanged, and the gate emits a warning in the summary for each such entry. An expired entry covers nothing and the finding blocks again.
  • Schedule workflow: a separate toolchain-tracking job opens or updates one issue (label dependency-toolchain) while toolchain HIGH/CRITICAL findings remain. issues: write is held only by that job; the scan job keeps contents: read.
  • Docs: SECURITY.md, DEVELOPMENT.md and the contributor skill's gate map (and its packaged mirror and the cao-contributor package copy) describe the split.

The #884 braces mitigation checks (npm run test:dependencies, the braces-compat adapters) are untouched and still run.

Before / after (same command, today's lockfiles, Trivy 0.70.0)

Command: python3 scripts/dependency_security.py --output-dir out (head also needs the builds, below).

main (5a8dc10):

Status: blocked
Blocking findings: 8. Total findings: 9. Distinct advisories: 2.
HIGH CVE-2026-112366 picomatch 2.3.2 Not published cao_mcp_apps/package-lock.json
HIGH CVE-2026-112366 picomatch 4.0.4 Not published cao_mcp_apps/package-lock.json
HIGH CVE-2026-112366 picomatch 4.0.7 Not published cao_mcp_apps/package-lock.json
HIGH CVE-2026-112366 picomatch 2.3.2 Not published docusaurus/package-lock.json
HIGH CVE-2026-112366 picomatch 4.0.7 Not published examples/ag-ui/ag-ui-operator-dashboard/package-lock.json
HIGH CVE-2026-112366 picomatch 2.3.2 Not published web/package-lock.json
HIGH CVE-2026-112366 picomatch 4.0.4 Not published web/package-lock.json
HIGH CVE-2026-112366 picomatch 4.0.7 Not published web/package-lock.json
MEDIUM CVE-2026-118764 minimatch 10.2.5 11.0.1 examples/decision-model-host/aws/cdk/package-lock.json

head:

Status: passed
Blocking findings: 0. Shipped findings: 0. Toolchain findings: 9.
Distinct advisories: 2.
Shipped sets: uv.lock 97, web 13, cao_mcp_apps 9, docusaurus 37, tui/Cargo.lock 80 packages
Toolchain only (not blocking), each with its dependency path:
HIGH picomatch 2.3.2  web/package-lock.json  patch-package > find-yarn-workspace-root > micromatch > picomatch
HIGH picomatch 4.0.4  web/package-lock.json  @testing-library/jest-dom > vitest > tinyglobby > picomatch; @vitejs/plugin-react > vite > tinyglobby > picomatch; ...
HIGH picomatch 4.0.7  web/package-lock.json  @testing-library/jest-dom > vitest > picomatch; @vitejs/plugin-react > vite > picomatch; ...
HIGH picomatch 2.3.2  cao_mcp_apps/package-lock.json  patch-package > find-yarn-workspace-root > micromatch > picomatch
HIGH picomatch 4.0.4 / 4.0.7  cao_mcp_apps/package-lock.json  vite, tinyglobby, lint-staged, vite-plugin-singlefile paths
HIGH picomatch 2.3.2  docusaurus/package-lock.json  patch-package ...; @docusaurus/core > @docusaurus/utils > micromatch > picomatch; ...
HIGH picomatch 4.0.7  examples/ag-ui/ag-ui-operator-dashboard/package-lock.json  (examples ship nothing)
MEDIUM minimatch 10.2.5  examples/decision-model-host/aws/cdk/package-lock.json

picomatch is in none of the five shipped sets, as #1398 predicted. patch-package, which package.json lists as a dependency, is correctly classed toolchain because no build bundles it.

Fixture checks (in test/test_dependency_security_shipped.py): a HIGH finding bundled into web/ fails (exit 1); the same finding only in a devDependency passes with a warning; an empty or failed producer, a lockfile with no producer, and an invalid ignore file exit 2; missing statement, an expiry over 14 days and an expired entry are each covered.

Job time (local measurement, NOT GitHub Actions)

Sequence timed on one laptop, warm npm and Trivy caches, from a clean node_modules:

main head
npm ci + test:dependencies for docusaurus, web, cao_mcp_apps 26.1 s 21.4 s
uv sync --frozen --no-dev --no-install-project n/a 0.7 s
builds (web 3 s, MCP apps 7 s, docusaurus 12 s measured separately; 26.4 s together) n/a 26.4 s
gate script (Trivy; head adds uv export and cargo tree) 0.7 s 3.1 s
total 26.8 s 51.5 s

About 25 s more locally. Actions runners are slower and cargo tree there starts with a cold registry cache, so expect more; I have not measured Actions.

What we gain / what we lose

Gain

  • Pull requests are blocked only by risk that reaches users. Today 14 open PRs fail on a picomatch advisory that does not reach the wheel, the web UI, the MCP App bundles, the docs pages or the TUI.
  • Unfixed toolchain advisories become one tracking issue, not N red PRs.
  • The shipped set is measured from the build, which also corrects misplaced labels such as patch-package.

Lose

  • A compromise of the build toolchain, or a ReDoS in a tool that runs on CI runners, no longer blocks merges. It is reported and tracked, not gated.
  • The shipped-set producers are new code and can be wrong. Mitigations: the gate fails closed on a failed or empty producer, and the fixture tests above. A bundler plugin that silently misses a package would classify a real shipped finding as toolchain; the packages it records are visible in the shipped-set.json artifact.
  • The job is slower because it builds the bundles (about 25 s locally, see above).
  • The Vite plugin reads each output chunk's modules, so a package that contributes only CSS or other assets and no JavaScript module would be missed and classed toolchain. None of today's bundles has one; this is documented in SECURITY.md.
  • Maintainers must triage the tracking issue; if nobody does, toolchain debt grows unseen.

Cargo features

The release build is cargo build --release --locked (scripts/build_tui.py), with no --features or --no-default-features, and tui/Cargo.toml defines no [features]. cargo tree --locked -e normal --target all therefore matches the release build's default-feature normal edges, and --target all is a superset of any single cross target.

Verification and what I could not run

Run locally: the new and existing dependency tests, the gate-map and packaging tests, sync_skills.py --check, make check-agent-plugin, actionlint, zizmor 1.30.1 (offline mode, medium+: no findings), black/isort/ruff, mypy_ratchet.py, the markdown link check, and the web npm test and MCP Apps and docs type checks.

Not run: the composite action and the weekly toolchain-tracking job on GitHub Actions. They are covered by actionlint, zizmor and the existing workflow-shape tests only. The action's PyYAML comes from uv sync --frozen --no-dev --no-install-project; cargo tree --locked needs network on the runner. Please check the first CI run of this PR for Dependency Security.

Existing test changed

test_ci_and_scheduled_dependency_gates_are_identical_and_unconditional is unchanged and still green: the gate job stays byte-identical in ci.yml and the schedule workflow with contents: read. The tracking issue lives in a separate toolchain-tracking job, and a new test, test_issues_write_exists_only_on_the_schedule_tracking_job, asserts that issues: write is held only by that job, never by anything in ci.yml, and that the gate steps are equal. Red check: putting issues: write on the gate job fails both tests. The repository fixture in test_dependency_security.py now injects a shipped set (the #884 tests run on generic lockfiles with no build). The uv calls use --locked and the schedule workflow sets UV_LOCKED=1, as test_workflow_pins.py requires. patch-package stays in dependencies; the build-derived shipped set makes the label irrelevant.

Real bundle check

From actual builds on this head: web bundles 13 packages (@xterm/addon-fit, @xterm/xterm, events, graphology, graphology-layout, graphology-utils, lucide-react, pandemonium, react, react-dom, scheduler, sigma, zustand); the four MCP App views bundle 3, 3, 3 and 9; the Docusaurus client bundle 37. picomatch, micromatch and patch-package are in none of them (grep -c picomatch on every bundled-packages.*.json is 0).

Co-authored-by: Kiro Agent 244629292+kiro-agent@users.noreply.github.com

@plauzy
plauzy force-pushed the ci/1398-depsec-shipped-gate branch from 3b8c3a7 to c2ee227 Compare November 18, 2026 04:23
@fanhongy

Copy link
Copy Markdown
Collaborator

This PR is still a draft; reviewing now because it unblocks the repo-wide red Dependency Security check (#1398).

P1

1. npm aliases make a bundled package look like toolchain (scripts/shipped-packages.mjs, owningPackage)
The plugin names a package by its node_modules/<dir> folder. For an npm alias the folder and the package name differ, and Trivy reports the real name (parse.go L109-112 at v0.70.0). classify_findings matches by name, so a HIGH/CRITICAL on such a package is classed toolchain and does not block.
This is live in the docs client bundle today: react-helmet-async is @slorber/react-helmet-async and react-loadable is @docusaurus/react-loadable in docusaurus/package-lock.json. The html_writers pattern react-helmet-async misses it too.
Real npm run build in docusaurus/ on c2ee227:

client 37 ['react-loadable', 'react-helmet-async']   # names in .shipped/bundled-packages.client.json
node_modules/react-helmet-async/package.json name: @slorber/react-helmet-async
node_modules/react-loadable/package.json     name: @docusaurus/react-loadable

Gate with a fixture Trivy report (CRITICAL on @slorber/react-helmet-async@1.3.0, shipped set as the plugin records it):

exit 0 status passed class toolchain - not in the shipped set (bundled-packages.json)

Suggest recording the name from the package's own package.json (already read in readVersion), or both names, plus a fixture with an aliased bundled package.

P2

1. A failed weekly scan marks the tracking issue as clear (.github/workflows/dependency-security.yml, toolchain-tracking)
The job runs on !cancelled(), so it also runs after the gate exits 2. An error run writes no toolchain-issue.md, and the step reads that as "no findings". Running the gate with a forced error, then the step's own run: block with a fake gh and open issue 42:

exit 2 status error files ['report.json', 'summary.md']
gh issue edit 42 --body-file report/none.md
No toolchain-only HIGH/CRITICAL findings remain at <sha>. This issue can be closed.

A transient Trivy DB or cargo tree failure would wipe the only place toolchain findings are now tracked. Suggest taking the "none remain" branch only when report.json status is not error.

P3

  1. .github/dependency-security/trivyignore.yaml L16 says paths is optional and "omitted means every lockfile", and the PR body says the same. The code rejects an entry with neither paths nor purl (SECURITY.md is right). The template could show purl too.
  2. fail-on-scopes: runtime, development, unknown makes Dependency Review fail on dev dependencies, including advisories with no fix. A lockfile refresh that moves picomatch between two vulnerable versions would go red again on a toolchain-only advisory. Is that intended? (Not run: needs GitHub's dependency graph.)

Checked by running: the four test_dependency_security* files plus the workflow, pin, skill-parity and must-run guards (612 passed, 18 skipped); node --test scripts/test-shipped-packages.mjs (8/8); gate exit codes on fixture Trivy reports (clean 0, shipped HIGH 1, expired entry 1, missing Results / missing lockfile / schema 3 / future added / over 14 days / http url 2); web and cao_mcp_apps built on 5a8dc10 and c2ee227 give byte-identical output (diff -r empty), so the Vite plugin only records. Not run: a real Trivy scan (fixture reports only), the action and the weekly job on Actions.

independently reviewed by Kiro Crew

@fanhongy

Copy link
Copy Markdown
Collaborator

Re-review at 7c08de2

Our P1, our P2 and both P3s are fixed. No new P1 or P2. Suggest approval once CI is green and the PR leaves draft. Dependency Security passes on this head.

  • P1 (npm aliases): collectPackages now records each bundled package under its folder name and, for an alias, under the name in its own package.json, on every path (externals, virtual modules, file: links, normal modules). policy.json html_writers now has *react-helmet-async and @docusaurus/react-loadable. node --test scripts/test-shipped-packages.mjs: 9 passed (was 8).
  • P2 (weekly issue on error): toolchain-tracking now reads report.json status first. On error or a missing report, it comments the error on the open issue and exits, and never takes the "none remain" branch.
  • P3s: the trivyignore.yaml template now says an entry needs paths or purl and shows a purl example. fail-on-scopes is removed from Dependency Review.

Checked by running: the four test_dependency_security* files plus test_security_scan_steps.py, test_cao_contributing_skill_accuracy.py, test_skill_packaging_parity.py and test_workflow_pins.py: 547 passed, 17 skipped. Not run: a real Trivy scan, a fresh docusaurus build on this head, and the weekly job on Actions.

independently reviewed by Kiro Crew

@codecov-commenter

codecov-commenter commented Nov 18, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
⚠️ Please upload report for BASE (main@b2b3666). Learn more about missing BASE report.

Additional details and impacted files
@@           Coverage Diff           @@
##             main    #1400   +/-   ##
=======================================
  Coverage        ?   91.96%           
=======================================
  Files           ?      296           
  Lines           ?    57085           
  Branches        ?     9397           
=======================================
  Hits            ?    52499           
  Misses          ?     3218           
  Partials        ?     1368           
Flag Coverage Δ
unittests 91.96% <ø> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@plauzy

plauzy commented Nov 18, 2026

Copy link
Copy Markdown
Collaborator Author

Updated the PR to 7c08de2.

P1, npm aliases: fixed at 7c08de2. scripts/shipped-packages.mjs now records each bundled package under its node_modules folder name and, when its own package.json has a different name, under that real name too, which is what Trivy reports. A real npm run build in docusaurus/ now records react-helmet-async and @slorber/react-helmet-async, react-loadable and @docusaurus/react-loadable, in both the client and server records. The html_writers patterns now match the real names (*react-helmet-async, @docusaurus/react-loadable). Tests: an npm alias is recorded under its folder name AND its real package name (node --test scripts/test-shipped-packages.mjs, red before the fix), test_aliased_bundled_package_blocks (CRITICAL on @slorber/react-helmet-async@1.3.0 with both names recorded exits 1) with the control test_alias_folder_name_alone_does_not_match, and test_checked_in_html_writer_patterns_match_aliased_real_names (red before the fix).

P2, failed weekly scan clears the tracking issue: fixed at 7c08de2. The tracking step reads report.json first; when status is error or there is no report, it comments the error on the open tracking issue, leaves it open, and stops. Only a run with a verdict reaches the "none remain" branch. Tests run the step's own run: block with a fake gh and open issue 42: test_tracking_issue_is_not_cleared_after_a_gate_error (red before the fix), with the controls test_tracking_issue_is_cleared_after_a_clean_run and test_tracking_issue_is_updated_when_findings_remain.

P3.1, template comment: fixed at 7c08de2. The trivyignore.yaml template now says an entry must have paths or purl, and shows one entry of each. test_template_example_is_accepted_and_shows_both_scopes loads the uncommented example through the real loader (two valid entries, one per scope); test_template_example_without_a_scope_is_rejected is the control. The PR description is corrected too.

P3.2, fail-on-scopes: reverted at 7c08de2; Dependency Review keeps its default scopes, as on main. The gate already blocks a toolchain HIGH/CRITICAL that has a fixed version when the PR adds or changes that package version (compared with the base lockfile), so a PR that brings in a fixable vulnerable dev dependency still fails. Widening Dependency Review's scopes would re-block an unfixed advisory such as picomatch whenever a lockfile refresh moves it between two vulnerable versions, which is what this PR sets out to stop. test_j_dependency_review_keeps_its_base_scopes asserts the input is absent.

CI red on c2ee227: the Unit Tests (3.12) and CAO MCP Apps failures (42 tests across the four test_dependency_security* files) all came from GITHUB_EVENT_NAME=pull_request on the runner leaking into the tests, which put the gate in pull-request mode with no base revision (exit 2). The test files now clear that variable and CAO_DEPSEC_BASE_REV with an autouse fixture, and test_pull_request_mode_without_a_base_exits_2 (with the control test_push_mode_without_a_base_runs_the_gate) sets the mode explicitly. Locally the same files gave 42 failed with GITHUB_EVENT_NAME=pull_request before the change, and 254 passed with it set and unset after.

@plauzy
plauzy marked this pull request as ready for review November 18, 2026 08:44
@plauzy
plauzy requested a review from a team as a code owner November 18, 2026 08:44
@plauzy
plauzy requested a review from fanhongy November 18, 2026 08:44
@plauzy
plauzy force-pushed the ci/1398-depsec-shipped-gate branch from 7c08de2 to a229b23 Compare November 18, 2026 08:45
@plauzy
plauzy enabled auto-merge (squash) November 18, 2026 09:14
@fanhongy

Copy link
Copy Markdown
Collaborator

Suggest approval at a229b23

All CI checks pass (28 passed; only the push-only deploy job is skipped), including Dependency Security. The PR is out of draft. a229b23 is a rebase on main with the same changes as 7c08de2, where we confirmed our P1, our P2 and both P3s are fixed. No open P1 or P2 from us. A maintainer needs to approve.

independently reviewed by Kiro Crew

@plauzy
plauzy disabled auto-merge November 18, 2026 09:41
@plauzy

plauzy commented Nov 18, 2026

Copy link
Copy Markdown
Collaborator Author

This suite is unbearably slow even though this change is an improvement ... looking closer I realized the MCP Apps CI job is unnecessarily duplicating 45 minutes of unit tests that the Unit Tests (3.12 already runs - https://github.com/awslabs/cli-agent-orchestrator/actions/runs/44269686813/job/134682882135?pr=1400#step:14:887

I'm going to fix that as well as some other minor fixes to attempt to get unit tests run running in parallel ... we'll see if it makes a dent on the github hosted runners

@fanhongy

Copy link
Copy Markdown
Collaborator

@plauzy, before you change the MCP Apps job: #1387 already removes that duplicate Python run. It moves the coverage ratchet into its own coverage-ratchet job, which reads the Unit Tests (3.12) report. On that PR, CAO MCP Apps took 55 s instead of about 33 min, and its timeout went from 60 to 30 min. #1387 is waiting for a maintainer to approve it again after a P3 fix. Building on it, or leaving that part out of #1400, avoids two PRs editing the same job.

Our Suggest approval above stands for a229b23. We will re-check the next push.

independently reviewed by Kiro Crew

@plauzy

plauzy commented Nov 18, 2026

Copy link
Copy Markdown
Collaborator Author

Thanks, @fanhongy. We missed #1387. #1400 will not touch the MCP Apps job or the coverage ratchet; it stays at a229b23 with the scope you reviewed, and nothing was pushed after your note.

Merging #1400 first should also clear #1387's only red check (Dependency Security). I'll point #1408 at #1387 for the duplicate run. Any parallel-test work there will wait for #1387 and your #1350 test-stability series (#1389, #1394, #1401, #1402, #1405, #1406), so it doesn't edit the same job or the same tests.

awslabs#1398)

Dependency Security failed every pull request on any HIGH or CRITICAL finding in any
tracked lockfile, including the build, test and lint toolchain. When
CVE-2026-112366 (picomatch, no fix) reached the Trivy database, it failed open PRs
that do not touch picomatch, although every path to it is toolchain-only.

The gate now blocks on risk that reaches users, at the same strength as before for
shipped code, and reports the rest.

- Measure the shipped set from the build: `uv export --locked --no-dev --all-extras` for
  the wheel and the optional extras users install; a Vite plugin that records what
  web/ and every cao_mcp_apps view bundled (modules, externals and injected helpers,
  plus the emitted files), each package under its folder name and, for an npm alias,
  its real package.json name; a webpack plugin for both Docusaurus passes, plus the
  build-time HTML writers; `cargo tree -e normal` for the TUI. examples/ ships nothing.
- Match shipped membership by package name at any version (over-block is the safe
  direction).
- Fail closed (exit 2) on a missing, empty or incomplete build record, a shipped file
  no record lists, a Trivy inventory without its sentinel packages, a lockfile with no
  producer, a pull-request run without its base, an invalid ignore file, or any
  unexpected error.
- Block (exit 1) on a shipped HIGH/CRITICAL not covered by a valid entry, on malware
  advisories anywhere, and on a toolchain HIGH/CRITICAL with a fix that the change adds.
  Other toolchain findings, with dependency paths, go to the summary and a SARIF file.
- Validate .github/dependency-security/trivyignore.yaml, read only by the script (Trivy
  still runs with its ignore file disabled): each entry needs a paths or purl scope and
  must match a current shipped finding, and its 14-day window is checked against the
  YAML date and the date git first shows it on the base.
- List packages that left the shipped set compared with the base's baseline.
- Test the shipped-set recorder with node --test; toolchain.sarif stays an artifact.
- The weekly workflow opens or updates one tracking issue for toolchain HIGH/CRITICAL
  findings, in a separate job that alone holds issues: write; a run that ends in an
  error comments on the issue and never marks it clear.
- Update the contributor gate map, its packaged mirror and SECURITY.md.

Co-authored-by: Kiro Agent <244629292+kiro-agent@users.noreply.github.com>
@plauzy
plauzy force-pushed the ci/1398-depsec-shipped-gate branch from a229b23 to b490ebc Compare November 19, 2026 05:11

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

CI: Dependency Security should block on shipped dependencies, not the build toolchain

3 participants