Skip to content

feat: skip timing-only gate rounds under advisory walltime - #268

Merged
Verdenroz merged 3 commits into
masterfrom
feat/advisory-skip-timing-rounds
Oct 22, 2026
Merged

Verdenroz merged 3 commits into
masterfrom
feat/advisory-skip-timing-rounds

Conversation

@Verdenroz

Copy link
Copy Markdown
Owner

What changed

With [gate] walltime = "advisory", gate --against-ref now measures each side once, so it runs two rounds instead of four. Rounds 3 and 4 skip the gating counters and only add a second walltime sample per side. Advisory walltime never fails, so that second sample can't change the verdict. On a real project's PR gate the measure phase took 5.2 of 15.6 minutes over four rounds. Now that the base build is seeded, measuring is the longest phase of that gate.

  • Passed walltime_advisory from gate and gate accept through resolve into measure_ref_interleaved. Under advisory the timing-only rounds 3 and 4 are not run.
  • Changed the round line to show the real total: gate: round 1/2 (head) under advisory, round n/4 otherwise.
  • Kept combine_rounds unchanged. Its caller uses each side's first round directly when the timing-only rounds were skipped.
  • Documented in docs/gating.md that advisory mode measures each side once, head first. A baseline it saves, and a run it stores in the run cache, then holds one walltime sample per item instead of the minimum of two.

Instructions, Ir, allocations, size and a bench's own p99 claim come from the first round, so they gate exactly as before. The default, walltime = "hard", still runs four rounds.

Why

The measure phase is now the largest part of an adopter's PR gate. Half of it bought a walltime sample that advisory mode reports as SOFT at most.

How was this tested

  • Ran a live gate on a real project's clone (--against-ref HEAD~5, one item, --no-reuse-base --no-item-cache). With walltime = "advisory" it printed round 1/2 (head) and round 2/2 (base) and passed with exit 0. Without the setting it printed rounds 1/4 to 4/4 and passed with exit 0. With one filtered item each round printed 0s, so this run shows the round count, not the time saved.
  • make check: exit 0, 449, 6 and 4 tests passed (test: 1 of 10 packages selected, 3 binaries run).
  • cargo clippy -p cargo-soothfast --all-targets -- -D warnings: clean.
  • No unit test covers the skip itself. It lives inside the merge-base worktree flow, which the live run exercised.
  • make gate BASE=master was not run. No measured code changed.

Checklist

  • make check passes (fmt, clippy -D warnings, cargo test --workspace)
  • make gate BASE=master passes, or any intentional cost change is explained above
  • Tests added/updated for new behavior (not required for docs/CI-only changes)
  • Public API items have /// doc comments
  • No new dependency, or its justification is included above (see Dependency Policy)
  • Docs (README.md, docs/, soothfast:bind/soothfast:claim markers) updated if behavior changed

- Rounds 3 and 4 only add a second walltime sample per side, which
  advisory walltime never fails on
- Measure was the largest gate phase once the base build was seeded
@Verdenroz
Verdenroz deployed to soothfast-bot October 22, 2026 00:56 — with GitHub Actions Active
@Verdenroz
Verdenroz deployed to soothfast-bot October 22, 2026 00:56 — with GitHub Actions Active
@Verdenroz
Verdenroz deployed to soothfast-bot October 22, 2026 00:56 — with GitHub Actions Active
@Verdenroz
Verdenroz deployed to soothfast-bot October 22, 2026 00:56 — with GitHub Actions Active
@Verdenroz
Verdenroz deployed to soothfast-bot October 22, 2026 00:56 — with GitHub Actions Active
@Verdenroz
Verdenroz deployed to soothfast-bot October 22, 2026 00:56 — with GitHub Actions Active
@Verdenroz
Verdenroz deployed to soothfast-bot October 22, 2026 00:56 — with GitHub Actions Active
@soothfast-bot

soothfast-bot Bot commented Oct 22, 2026

Copy link
Copy Markdown
Contributor

soothfast gate: soothfast-registry

gate: run cache keyed on callgrind (guest 80eb05cd5d172b67 glibc 2.43 8fc174992c9ed915 valgrind-3.26.0)
gate: measuring merge-base of origin/master in worktree (interleaved rounds)
gate: seeded 6 dependency units (24.2 MiB) from HEAD's target dir in 0.1s
gate: bench binaries identical (code and data match) — no measurable change possible
gate: gating backend = walltime
gate: build=26cf1b noise_floor=0.13% thresholds: instructions +5% ir +5% walltime +10.0% alloc/size +5% polls/wakes +5%
ok    soothfast_registry::bench_fnv1a walltime_median_ns 81531.3 -> 81531.3 (+0.0%)
ok    soothfast_registry::bench_fnv1a allocs 0 -> 0 (allowed <= 0)
ok    soothfast_registry::bench_fnv1a alloc_bytes 0 -> 0 (allowed <= 0)
ok    soothfast_registry::bench_yield_chain walltime_median_ns 139.4 -> 139.4 (+0.0%)
ok    soothfast_registry::bench_yield_chain allocs 1 -> 1 (allowed <= 1)
ok    soothfast_registry::bench_yield_chain alloc_bytes 24 -> 24 (allowed <= 25)
ok    soothfast_registry::bench_yield_chain polls 9 -> 9 (allowed <= 9)
ok    soothfast_registry::bench_yield_chain wakes 8 -> 8 (allowed <= 8)
ok    soothfast_registry::bench_fnv1a assert alloc: allocs 0 <= 0
ok    soothfast_registry::bench_fnv1a assert complexity: claimed O(n); growth drift x1.01 over sizes [1024, 8192, 65536] (limit x2.5)
ok    soothfast_registry::bench_yield_chain assert alloc: allocs 1 <= 1
gate: passed (2 item(s))

@soothfast-bot

soothfast-bot Bot commented Oct 22, 2026

Copy link
Copy Markdown
Contributor

soothfast gate: soothfast-measure

gate: run cache keyed on callgrind (guest 80eb05cd5d172b67 glibc 2.43 8fc174992c9ed915 valgrind-3.26.0)
gate: measuring merge-base of origin/master in worktree (interleaved rounds)
gate: seeded 6 dependency units (24.2 MiB) from HEAD's target dir in 0.1s
gate: bench binaries identical (code and data match) — no measurable change possible
gate: gating backend = walltime
gate: build=26cf1b noise_floor=0.20% thresholds: instructions +5% ir +5% walltime +10.0% alloc/size +5% polls/wakes +5%
ok    soothfast_measure::bench_summarize walltime_median_ns 608763.7 -> 608763.7 (+0.0%)
ok    soothfast_measure::bench_summarize allocs 4 -> 4 (allowed <= 4)
ok    soothfast_measure::bench_summarize alloc_bytes 524288 -> 524288 (allowed <= 550502)
ok    soothfast_measure::bench_sweep_evaluate walltime_median_ns 24.5 -> 24.5 (+0.0%)
ok    soothfast_measure::bench_sweep_evaluate allocs 0 -> 0 (allowed <= 0)
ok    soothfast_measure::bench_sweep_evaluate alloc_bytes 0 -> 0 (allowed <= 0)
ok    soothfast_measure::bench_summarize assert alloc: allocs 4 <= 8
ok    soothfast_measure::bench_summarize assert complexity: claimed O(n log n); growth drift x1.32 over sizes [1024, 4096, 16384] (limit x2.5)
ok    soothfast_measure::bench_sweep_evaluate assert alloc: allocs 0 <= 0
gate: passed (2 item(s))

@soothfast-bot

soothfast-bot Bot commented Oct 22, 2026

Copy link
Copy Markdown
Contributor

soothfast gate: soothfast-docs

gate: run cache keyed on callgrind (guest 80eb05cd5d172b67 glibc 2.43 8fc174992c9ed915 valgrind-3.26.0)
gate: measuring merge-base of origin/master in worktree (interleaved rounds)
gate: seeded 17 dependency units (44.9 MiB) from HEAD's target dir in 0.1s
gate: bench binaries identical (code and data match) — no measurable change possible
gate: gating backend = walltime
gate: build=26cf1b noise_floor=0.11% thresholds: instructions +5% ir +5% walltime +10.0% alloc/size +5% polls/wakes +5%
ok    soothfast_docs::bench_claim_parse walltime_median_ns 213.8 -> 213.8 (+0.0%)
ok    soothfast_docs::bench_claim_parse allocs 6 -> 6 (allowed <= 6)
ok    soothfast_docs::bench_claim_parse alloc_bytes 170 -> 170 (allowed <= 178)
ok    soothfast_docs::bench_markdown_scan walltime_median_ns 413968.0 -> 413968.0 (+0.0%)
ok    soothfast_docs::bench_markdown_scan allocs 4114 -> 4114 (allowed <= 4319)
ok    soothfast_docs::bench_markdown_scan alloc_bytes 338400 -> 338400 (allowed <= 355320)
ok    soothfast_docs::bench_claim_parse assert p99: p99 242ns <= 50000ns
ok    soothfast_docs::bench_markdown_scan assert complexity: claimed O(n); growth drift x0.95 over sizes [512, 2048, 8192] (limit x2.5)
gate: passed (2 item(s))

@soothfast-bot

soothfast-bot Bot commented Oct 22, 2026

Copy link
Copy Markdown
Contributor

soothfast gate: soothfast-report

gate: run cache keyed on callgrind (guest 80eb05cd5d172b67 glibc 2.43 8fc174992c9ed915 valgrind-3.26.0)
gate: measuring merge-base of origin/master in worktree (interleaved rounds)
gate: seeded 17 dependency units (44.9 MiB) from HEAD's target dir in 0.1s
gate: bench binaries identical (code and data match) — no measurable change possible
gate: gating backend = walltime
gate: build=26cf1b noise_floor=0.05% thresholds: instructions +5% ir +5% walltime +10.0% alloc/size +5% polls/wakes +5%
ok    soothfast_report::bench_llms_render walltime_median_ns 309078.2 -> 309078.2 (+0.0%)
ok    soothfast_report::bench_llms_render allocs 7191 -> 7191 (allowed <= 7550)
ok    soothfast_report::bench_llms_render alloc_bytes 1197537 -> 1197537 (allowed <= 1257413)
ok    soothfast_report::bench_perf_table walltime_median_ns 821631.0 -> 821631.0 (+0.0%)
ok    soothfast_report::bench_perf_table allocs 8204 -> 8204 (allowed <= 8614)
ok    soothfast_report::bench_perf_table alloc_bytes 465747 -> 465747 (allowed <= 489034)
ok    soothfast_report::bench_llms_render assert complexity: claimed O(n); growth drift x0.99 over sizes [64, 256, 1024] (limit x2.5)
ok    soothfast_report::bench_perf_table assert complexity: claimed O(n); growth drift x0.92 over sizes [64, 256, 1024] (limit x2.5)
gate: passed (2 item(s))

@soothfast-bot

soothfast-bot Bot commented Oct 22, 2026

Copy link
Copy Markdown
Contributor

soothfast gate: soothfast-site

gate: run cache keyed on callgrind (guest 80eb05cd5d172b67 glibc 2.43 8fc174992c9ed915 valgrind-3.26.0)
gate: measuring merge-base of origin/master in worktree (interleaved rounds)
gate: seeded 17 dependency units (44.9 MiB) from HEAD's target dir in 0.1s
gate: bench binaries identical (code and data match) — no measurable change possible
gate: gating backend = walltime
gate: build=26cf1b noise_floor=0.06% thresholds: instructions +5% ir +5% walltime +10.0% alloc/size +5% polls/wakes +5%
ok    soothfast_site::bench_highlight walltime_median_ns 2845675.0 -> 2845675.0 (+0.0%)
ok    soothfast_site::bench_highlight allocs 88069 -> 88069 (allowed <= 92472)
ok    soothfast_site::bench_highlight alloc_bytes 2079336 -> 2079336 (allowed <= 2183302)
ok    soothfast_site::bench_md_render walltime_median_ns 3345469.0 -> 3345469.0 (+0.0%)
ok    soothfast_site::bench_md_render allocs 68118 -> 68118 (allowed <= 71523)
ok    soothfast_site::bench_md_render alloc_bytes 3502526 -> 3502526 (allowed <= 3677652)
ok    soothfast_site::bench_highlight assert complexity: claimed O(n); growth drift x1.22 over sizes [128, 512, 2048] (limit x2.5)
ok    soothfast_site::bench_md_render assert complexity: claimed O(n); growth drift x1.11 over sizes [256, 1024, 4096] (limit x2.5)
gate: passed (2 item(s))

@soothfast-bot

soothfast-bot Bot commented Oct 22, 2026

Copy link
Copy Markdown
Contributor

soothfast gate: soothfast-spec

gate: run cache keyed on callgrind (guest 80eb05cd5d172b67 glibc 2.43 8fc174992c9ed915 valgrind-3.26.0)
gate: measuring merge-base of origin/master in worktree (interleaved rounds)
gate: seeded 24 dependency units (54.7 MiB) from HEAD's target dir in 0.1s
gate: bench binaries identical (code and data match) — no measurable change possible
gate: gating backend = walltime
gate: build=26cf1b noise_floor=0.12% thresholds: instructions +5% ir +5% walltime +10.0% alloc/size +5% polls/wakes +5%
ok    soothfast_spec::bench_openapi_diff walltime_median_ns 3311996.0 -> 3311996.0 (+0.0%)
ok    soothfast_spec::bench_openapi_diff allocs 77722 -> 77722 (allowed <= 81608)
ok    soothfast_spec::bench_openapi_diff alloc_bytes 10815487 -> 10815487 (allowed <= 11356261)
ok    soothfast_spec::bench_openapi_document walltime_median_ns 1241679.0 -> 1241679.0 (+0.0%)
ok    soothfast_spec::bench_openapi_document allocs 30152 -> 30152 (allowed <= 31659)
ok    soothfast_spec::bench_openapi_document alloc_bytes 4715065 -> 4715065 (allowed <= 4950818)
ok    soothfast_spec::bench_serialize_yaml walltime_median_ns 4215125.0 -> 4215125.0 (+0.0%)
ok    soothfast_spec::bench_serialize_yaml allocs 74277 -> 74277 (allowed <= 77990)
ok    soothfast_spec::bench_serialize_yaml alloc_bytes 8533615 -> 8533615 (allowed <= 8960295)
ok    soothfast_spec::bench_openapi_diff assert complexity: claimed O(n); growth drift x1.03 over sizes [16, 64, 256] (limit x2.5)
ok    soothfast_spec::bench_openapi_document assert complexity: claimed O(n); growth drift x1.08 over sizes [16, 64, 256] (limit x2.5)
ok    soothfast_spec::bench_serialize_yaml assert complexity: claimed O(n); growth drift x0.99 over sizes [16, 64, 256] (limit x2.5)
gate: passed (3 item(s))

@soothfast-bot

soothfast-bot Bot commented Oct 22, 2026

Copy link
Copy Markdown
Contributor

soothfast gate: soothfast-sdk

gate: run cache keyed on callgrind (guest 80eb05cd5d172b67 glibc 2.43 8fc174992c9ed915 valgrind-3.26.0)
gate: measuring merge-base of origin/master in worktree (interleaved rounds)
gate: seeded 24 dependency units (54.7 MiB) from HEAD's target dir in 0.1s
gate: bench binaries identical (code and data match) — no measurable change possible
gate: gating backend = walltime
gate: build=26cf1b noise_floor=0.11% thresholds: instructions +5% ir +5% walltime +10.0% alloc/size +5% polls/wakes +5%
ok    soothfast_sdk::bench_emit_typescript walltime_median_ns 2289496.0 -> 2289496.0 (+0.0%)
ok    soothfast_sdk::bench_emit_typescript allocs 42930 -> 42930 (allowed <= 45076)
ok    soothfast_sdk::bench_emit_typescript alloc_bytes 2938940 -> 2938940 (allowed <= 3085887)
ok    soothfast_sdk::bench_lower walltime_median_ns 1296542.0 -> 1296542.0 (+0.0%)
ok    soothfast_sdk::bench_lower allocs 23907 -> 23907 (allowed <= 25102)
ok    soothfast_sdk::bench_lower alloc_bytes 2064788 -> 2064788 (allowed <= 2168027)
ok    soothfast_sdk::bench_emit_typescript assert complexity: claimed O(n); growth drift x1.00 over sizes [16, 64, 256] (limit x2.5)
ok    soothfast_sdk::bench_lower assert complexity: claimed O(n); growth drift x1.05 over sizes [16, 64, 256] (limit x2.5)
gate: passed (2 item(s))

@Verdenroz
Verdenroz merged commit 0bf3e9c into master Oct 22, 2026
19 checks passed
@Verdenroz
Verdenroz deleted the feat/advisory-skip-timing-rounds branch October 22, 2026 03:11

This branch was successfully deployed

1 active deployment
soothfast-bot — 6e8bf11c Deployed Oct 22, 2026 by Verdenroz via gate (soothfast-docs) / gate #604
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant