Skip to content

feat: per-package file allow-lists for cargo soothfast test - #260

Merged
Verdenroz merged 3 commits into
masterfrom
feat/test-allow-list
Oct 21, 2026
Merged

Verdenroz merged 3 commits into
masterfrom
feat/test-allow-list

Conversation

@Verdenroz

Copy link
Copy Markdown
Owner

What changed

A package at the workspace root owns every file outside the member crates, so keeping cargo soothfast test from re-running it meant an ignore entry for every directory no test reads. Any new root file selected the package and invalidated its stored passes until someone added another entry. soothfast.toml can now say which files a package's tests depend on instead, and everything else the package owns stops counting.

[test.packages.mylib]
files = ["src/**", "tests/**", "benches/**", "examples/**", "build.rs", "docs/**"]
  • Added [test.packages.NAME] tables with a files list of globs relative to the package's manifest directory. A file the package owns that matches none of them is ignored for selection and left out of the package's key.
  • Kept the existing precedence. inputs still wins over a list, ignore still subtracts from it, and the build-wide files (Cargo.toml, Cargo.lock, the toolchain and cargo config) still select every package.
  • Counted a package's own Cargo.toml for it whatever its list says. A member's manifest is not a build-wide file, so leaving it out would have hidden dependency and feature changes.
  • Kept a nested member's files with the member. A root package's list never applies to them.
  • Added errors, each naming the table as written: a table for a package the workspace does not have, a table with no files, a repeated table, an unknown key, an unbalanced quote in the name, and any other [test.*] table.
  • Documented the lists in docs/testing.md. That includes what to list (the generated tests/soothfast_docs/ and examples/soothfast_capture/ targets among them), that the list is about what tests read rather than what cargo package ships, and that reads by relative path outside the package are invisible to it.

Why

A deny-list goes stale every time a file is added at the root. An allow-list only changes when the tests start reading something new.

How was this tested

  • Selection and the key run through the same select::classify, from select() and from package_inputs and workspace_inputs in test/key.rs.
  • Added config tests for bare and quoted table names, every error case, and the missing-package check.
  • Added selection tests:
    • a root package with a list ignores new root files and still selects itself and its dependents for a listed file;
    • a nested member's files stay the member's whatever the root lists;
    • inputs, ignore and the build-wide files keep their precedence;
    • a member's list is relative to its own directory, and its own Cargo.toml still counts.
  • Added a key test: with a root list, new unlisted files move neither the package nor the workspace digest, and a listed file moves only the package digest.
  • Removing the list check fails exactly the tests that depend on it. Removing the manifest exception fails the manifest case.
  • Added a quoted [test.packages."..."] table to the site config test that checks foreign tables are skipped.
  • cargo test -p cargo-soothfast --bin cargo-soothfast -- test::config test::select test::key: 43 passed. cargo test -p soothfast-site config: 10 passed.
  • cargo soothfast docs build: 20 pages built, so the site renderer accepts the new section.
  • make check: passed (test: 2 of 10 packages selected, 4 binaries run, 0 skipped (cached), doc tests in 1 packages), at the commit before the manifest fix. The manifest fix was covered by the scoped tests above and the pre-commit hook's fmt and clippy.
  • make gate BASE=master was not run. This changes no measured code.

Checklist

  • make check passes (fmt, clippy -D warnings, cargo test --workspace)
  • make gate BASE=master passes, or any intentional cost change is explained above
  • Tests added/updated for new behavior (not required for docs/CI-only changes)
  • Public API items have /// doc comments
  • No new dependency, or its justification is included above (see Dependency Policy)
  • Docs (README.md, docs/, soothfast:bind/soothfast:claim markers) updated if behavior changed

- A root package owns every file outside the members, so a deny-list
  goes stale with each new root file and re-runs every root test
- A file its package's list leaves out is ignored, after inputs, ignore
  and build-wide files, which keep their precedence
- A member's Cargo.toml is not a build-wide file, so a list without it
  hid dependency and feature changes and served a stale pass
@Verdenroz
Verdenroz deployed to soothfast-bot October 21, 2026 03:48 — with GitHub Actions Active
@Verdenroz
Verdenroz deployed to soothfast-bot October 21, 2026 03:48 — with GitHub Actions Active
@Verdenroz
Verdenroz deployed to soothfast-bot October 21, 2026 03:48 — with GitHub Actions Active
@Verdenroz
Verdenroz deployed to soothfast-bot October 21, 2026 03:48 — with GitHub Actions Active
@Verdenroz
Verdenroz deployed to soothfast-bot October 21, 2026 03:48 — with GitHub Actions Active
@Verdenroz
Verdenroz deployed to soothfast-bot October 21, 2026 03:48 — with GitHub Actions Active
@Verdenroz
Verdenroz deployed to soothfast-bot October 21, 2026 03:48 — with GitHub Actions Active
@soothfast-bot

soothfast-bot Bot commented Oct 21, 2026

Copy link
Copy Markdown
Contributor

soothfast gate: soothfast-measure

gate: run cache keyed on callgrind (guest 80eb05cd5d172b67 glibc 2.43 8fc174992c9ed915 valgrind-3.26.0)
gate: measuring merge-base of origin/master in worktree (interleaved rounds)
gate: bench binaries identical (code and data match) — no measurable change possible
gate: gating backend = walltime
gate: build=26cf1b noise_floor=1.66% thresholds: instructions +5% ir +5% walltime +10.0% alloc/size +5% polls/wakes +5%
ok    soothfast_measure::bench_summarize walltime_median_ns 342974.2 -> 342974.2 (+0.0%)
ok    soothfast_measure::bench_summarize allocs 4 -> 4 (allowed <= 4)
ok    soothfast_measure::bench_summarize alloc_bytes 524288 -> 524288 (allowed <= 550502)
ok    soothfast_measure::bench_sweep_evaluate walltime_median_ns 12.9 -> 12.9 (+0.0%)
ok    soothfast_measure::bench_sweep_evaluate allocs 0 -> 0 (allowed <= 0)
ok    soothfast_measure::bench_sweep_evaluate alloc_bytes 0 -> 0 (allowed <= 0)
ok    soothfast_measure::bench_summarize assert alloc: allocs 4 <= 8
ok    soothfast_measure::bench_summarize assert complexity: claimed O(n log n); growth drift x1.12 over sizes [1024, 4096, 16384] (limit x2.5)
ok    soothfast_measure::bench_sweep_evaluate assert alloc: allocs 0 <= 0
gate: passed (2 item(s))

@soothfast-bot

soothfast-bot Bot commented Oct 21, 2026

Copy link
Copy Markdown
Contributor

soothfast gate: soothfast-docs

gate: run cache keyed on callgrind (guest 80eb05cd5d172b67 glibc 2.43 8fc174992c9ed915 valgrind-3.26.0)
gate: measuring merge-base of origin/master in worktree (interleaved rounds)
gate: bench binaries identical (code and data match) — no measurable change possible
gate: gating backend = walltime
gate: build=26cf1b noise_floor=0.05% thresholds: instructions +5% ir +5% walltime +10.0% alloc/size +5% polls/wakes +5%
ok    soothfast_docs::bench_claim_parse walltime_median_ns 212.1 -> 212.1 (+0.0%)
ok    soothfast_docs::bench_claim_parse allocs 6 -> 6 (allowed <= 6)
ok    soothfast_docs::bench_claim_parse alloc_bytes 170 -> 170 (allowed <= 178)
ok    soothfast_docs::bench_markdown_scan walltime_median_ns 418180.8 -> 418180.8 (+0.0%)
ok    soothfast_docs::bench_markdown_scan allocs 4114 -> 4114 (allowed <= 4319)
ok    soothfast_docs::bench_markdown_scan alloc_bytes 338400 -> 338400 (allowed <= 355320)
ok    soothfast_docs::bench_claim_parse assert p99: p99 224ns <= 50000ns
ok    soothfast_docs::bench_markdown_scan assert complexity: claimed O(n); growth drift x0.95 over sizes [512, 2048, 8192] (limit x2.5)
gate: passed (2 item(s))

@soothfast-bot

soothfast-bot Bot commented Oct 21, 2026

Copy link
Copy Markdown
Contributor

soothfast gate: soothfast-registry

gate: run cache keyed on callgrind (guest 80eb05cd5d172b67 glibc 2.43 8fc174992c9ed915 valgrind-3.26.0)
gate: measuring merge-base of origin/master in worktree (interleaved rounds)
gate: bench binaries identical (code and data match) — no measurable change possible
gate: gating backend = walltime
gate: build=26cf1b noise_floor=0.16% thresholds: instructions +5% ir +5% walltime +10.0% alloc/size +5% polls/wakes +5%
ok    soothfast_registry::bench_fnv1a walltime_median_ns 71472.6 -> 71472.6 (+0.0%)
ok    soothfast_registry::bench_fnv1a allocs 0 -> 0 (allowed <= 0)
ok    soothfast_registry::bench_fnv1a alloc_bytes 0 -> 0 (allowed <= 0)
ok    soothfast_registry::bench_yield_chain walltime_median_ns 121.5 -> 121.5 (+0.0%)
ok    soothfast_registry::bench_yield_chain allocs 1 -> 1 (allowed <= 1)
ok    soothfast_registry::bench_yield_chain alloc_bytes 24 -> 24 (allowed <= 25)
ok    soothfast_registry::bench_yield_chain polls 9 -> 9 (allowed <= 9)
ok    soothfast_registry::bench_yield_chain wakes 8 -> 8 (allowed <= 8)
ok    soothfast_registry::bench_fnv1a assert alloc: allocs 0 <= 0
ok    soothfast_registry::bench_fnv1a assert complexity: claimed O(n); growth drift x1.01 over sizes [1024, 8192, 65536] (limit x2.5)
ok    soothfast_registry::bench_yield_chain assert alloc: allocs 1 <= 1
gate: passed (2 item(s))

@soothfast-bot

soothfast-bot Bot commented Oct 21, 2026

Copy link
Copy Markdown
Contributor

soothfast gate: soothfast-report

gate: run cache keyed on callgrind (guest 80eb05cd5d172b67 glibc 2.43 8fc174992c9ed915 valgrind-3.26.0)
gate: measuring merge-base of origin/master in worktree (interleaved rounds)
gate: bench binaries identical (code and data match) — no measurable change possible
gate: gating backend = walltime
gate: build=26cf1b noise_floor=0.20% thresholds: instructions +5% ir +5% walltime +10.0% alloc/size +5% polls/wakes +5%
ok    soothfast_report::bench_llms_render walltime_median_ns 315449.6 -> 315449.6 (+0.0%)
ok    soothfast_report::bench_llms_render allocs 7191 -> 7191 (allowed <= 7550)
ok    soothfast_report::bench_llms_render alloc_bytes 1197537 -> 1197537 (allowed <= 1257413)
ok    soothfast_report::bench_perf_table walltime_median_ns 828308.5 -> 828308.5 (+0.0%)
ok    soothfast_report::bench_perf_table allocs 8204 -> 8204 (allowed <= 8614)
ok    soothfast_report::bench_perf_table alloc_bytes 465747 -> 465747 (allowed <= 489034)
ok    soothfast_report::bench_llms_render assert complexity: claimed O(n); growth drift x0.98 over sizes [64, 256, 1024] (limit x2.5)
ok    soothfast_report::bench_perf_table assert complexity: claimed O(n); growth drift x0.95 over sizes [64, 256, 1024] (limit x2.5)
gate: passed (2 item(s))

@soothfast-bot

soothfast-bot Bot commented Oct 21, 2026

Copy link
Copy Markdown
Contributor

soothfast gate: soothfast-site

gate: run cache keyed on callgrind (guest 80eb05cd5d172b67 glibc 2.43 8fc174992c9ed915 valgrind-3.26.0)
gate: measuring merge-base of origin/master in worktree (interleaved rounds)
gate: bench binaries identical (code and data match) — no measurable change possible
gate: gating backend = walltime
gate: build=26cf1b noise_floor=0.22% thresholds: instructions +5% ir +5% walltime +10.0% alloc/size +5% polls/wakes +5%
ok    soothfast_site::bench_highlight walltime_median_ns 1960418.0 -> 1960418.0 (+0.0%)
ok    soothfast_site::bench_highlight allocs 88069 -> 88069 (allowed <= 92472)
ok    soothfast_site::bench_highlight alloc_bytes 2079336 -> 2079336 (allowed <= 2183302)
ok    soothfast_site::bench_md_render walltime_median_ns 2280388.0 -> 2280388.0 (+0.0%)
ok    soothfast_site::bench_md_render allocs 68118 -> 68118 (allowed <= 71523)
ok    soothfast_site::bench_md_render alloc_bytes 3502526 -> 3502526 (allowed <= 3677652)
ok    soothfast_site::bench_highlight assert complexity: claimed O(n); growth drift x1.14 over sizes [128, 512, 2048] (limit x2.5)
ok    soothfast_site::bench_md_render assert complexity: claimed O(n); growth drift x1.12 over sizes [256, 1024, 4096] (limit x2.5)
gate: passed (2 item(s))

@soothfast-bot

soothfast-bot Bot commented Oct 21, 2026

Copy link
Copy Markdown
Contributor

soothfast gate: soothfast-sdk

gate: run cache keyed on callgrind (guest 80eb05cd5d172b67 glibc 2.43 8fc174992c9ed915 valgrind-3.26.0)
gate: measuring merge-base of origin/master in worktree (interleaved rounds)
gate: bench binaries identical (code and data match) — no measurable change possible
gate: gating backend = walltime
gate: build=26cf1b noise_floor=0.04% thresholds: instructions +5% ir +5% walltime +10.0% alloc/size +5% polls/wakes +5%
ok    soothfast_sdk::bench_emit_typescript walltime_median_ns 1571626.0 -> 1571626.0 (+0.0%)
ok    soothfast_sdk::bench_emit_typescript allocs 42930 -> 42930 (allowed <= 45076)
ok    soothfast_sdk::bench_emit_typescript alloc_bytes 2938940 -> 2938940 (allowed <= 3085887)
ok    soothfast_sdk::bench_lower walltime_median_ns 911388.5 -> 911388.5 (+0.0%)
ok    soothfast_sdk::bench_lower allocs 23907 -> 23907 (allowed <= 25102)
ok    soothfast_sdk::bench_lower alloc_bytes 2064788 -> 2064788 (allowed <= 2168027)
ok    soothfast_sdk::bench_emit_typescript assert complexity: claimed O(n); growth drift x0.97 over sizes [16, 64, 256] (limit x2.5)
ok    soothfast_sdk::bench_lower assert complexity: claimed O(n); growth drift x1.05 over sizes [16, 64, 256] (limit x2.5)
gate: passed (2 item(s))

@soothfast-bot

soothfast-bot Bot commented Oct 21, 2026

Copy link
Copy Markdown
Contributor

soothfast gate: soothfast-spec

gate: run cache keyed on callgrind (guest 80eb05cd5d172b67 glibc 2.43 8fc174992c9ed915 valgrind-3.26.0)
gate: measuring merge-base of origin/master in worktree (interleaved rounds)
gate: bench binaries identical (code and data match) — no measurable change possible
gate: gating backend = walltime
gate: build=26cf1b noise_floor=0.20% thresholds: instructions +5% ir +5% walltime +10.0% alloc/size +5% polls/wakes +5%
ok    soothfast_spec::bench_openapi_diff walltime_median_ns 4329198.0 -> 4329198.0 (+0.0%)
ok    soothfast_spec::bench_openapi_diff allocs 77722 -> 77722 (allowed <= 81608)
ok    soothfast_spec::bench_openapi_diff alloc_bytes 10815487 -> 10815487 (allowed <= 11356261)
ok    soothfast_spec::bench_openapi_document walltime_median_ns 1522905.0 -> 1522905.0 (+0.0%)
ok    soothfast_spec::bench_openapi_document allocs 30152 -> 30152 (allowed <= 31659)
ok    soothfast_spec::bench_openapi_document alloc_bytes 4715065 -> 4715065 (allowed <= 4950818)
ok    soothfast_spec::bench_serialize_yaml walltime_median_ns 5847966.0 -> 5847966.0 (+0.0%)
ok    soothfast_spec::bench_serialize_yaml allocs 74277 -> 74277 (allowed <= 77990)
ok    soothfast_spec::bench_serialize_yaml alloc_bytes 8533615 -> 8533615 (allowed <= 8960295)
ok    soothfast_spec::bench_openapi_diff assert complexity: claimed O(n); growth drift x1.00 over sizes [16, 64, 256] (limit x2.5)
ok    soothfast_spec::bench_openapi_document assert complexity: claimed O(n); growth drift x1.05 over sizes [16, 64, 256] (limit x2.5)
ok    soothfast_spec::bench_serialize_yaml assert complexity: claimed O(n); growth drift x1.00 over sizes [16, 64, 256] (limit x2.5)
gate: passed (3 item(s))

@Verdenroz
Verdenroz merged commit d127fbb into master Oct 21, 2026
19 checks passed
@Verdenroz
Verdenroz deleted the feat/test-allow-list branch October 21, 2026 03:53
@Verdenroz Verdenroz mentioned this pull request Oct 21, 2026
4 of 6 tasks

This branch was successfully deployed

1 active deployment
soothfast-bot — 9b7d7399 Deployed Oct 21, 2026 by Verdenroz via gate (soothfast-report) / gate #588
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant