Skip to content

Publish the performance suite report as a docs speed page - #1497

Merged
thekevinbot merged 4 commits into
mainfrom
bot/1470-speed-page-1011
Oct 14, 2026
Merged

thekevinbot merged 4 commits into
mainfrom
bot/1470-speed-page-1011

Conversation

@thekevinbot

@thekevinbot thekevinbot commented Oct 12, 2026 •

Copy link
Copy Markdown
Collaborator

Publishes the performance suite's report on the docs site as a speed page (/speed, under Reference).

  • docs/speed.json is the committed report (source of truth); docs/speed.md is render_markdown of it. A hermetic test asserts the two agree and that the sidebar links the page, so the page cannot drift from the JSON. Refresh both from just bench-suite.
  • The page shows commit, date, machine, versions, pairs, the 30-day/hot-path cadence, per-case ratios vs the fastest native and every native, and spread. Case names are code-formatted so globs like ./**/{a,b}*.md render literally.
  • No CI wiring beyond the existing docs build.

Report

  • 100 pairs per case, all 24 cases (including watch, and the three ./**/*.md variants under distinct names).
  • Measured at main 462b3c6, which carries bench-suite: hand the watch case a Natives #1505 (watch case fix) and bench-suite: distinct names for the glob cases #1506 (distinct case names); the suite needs both to produce a full report. main was not merged into this branch: the suite ran from a detached checkout of main, and the page was rendered with this branch's render_markdown:
    # in a detached worktree at origin/main (462b3c6a)
    cd packages/python && uv run maturin develop --release
    cargo build --release -p dirsql --features cli
    cd packages/python && uv run python -m tests.e2e.bench_suite --pairs 100 --out /tmp/dirsql-bench-1470
    # in this branch
    cp /tmp/dirsql-bench-1470/report.json docs/speed.json
    cd packages/python && uv run python -c "import json,pathlib; from tests.e2e.bench_report import render_markdown; d=pathlib.Path('../../docs'); d.joinpath('speed.md').write_text(render_markdown(json.loads(d.joinpath('speed.json').read_text())))"
    
    After merge the same thing is just bench-suite, followed by copying report.json and report.md into docs/.
  • Runtime 3h15m (24 passed in 11697.80s).
  • Host load: started at a 1-minute load of 0.16 (load average: 0.16, 2.65, 17.18, 18:34). Ended at 33.12, 30.93, 21.75 (21:49). Sampled every minute during the run: min 0.52, median 5.4, p75 10.7, max 44.8 (32 CPUs). Other workloads on the shared host caused the rise. An earlier attempt was aborted when the load hit 160. Pairs are interleaved, so load mostly widens the spread, not the median ratio, but treat the wide spreads as noise.

Headline: dirsql is at or under 1.0x the fastest native on 7 of 24 cases (./**/*.md (symlinks) 0.12x, SELECT path, size, mtime 0.32x, join 0.42x, transcripts 0.53x, dossier 0.72x, notebook 0.87x, ./**/.* 0.92x). The other globs run 1.3-2.6x behind fd or rg. Frontmatter is 2.41x, jsonl 1.73x, and watch 1.15x.

Fixes #1470

E2E Verification

  • Build: maturin develop --release, cargo build --release -p dirsql --features cli
  • uv run python -m pytest tests/e2e/glob_speed_recursive_test.py tests/e2e/stat_speed_test.py -x -q: 3 passed; receipt packages/python/e2e-attestations/bot-1470-speed-page-1011.json
  • Full suite at 100 pairs (above): 24 passed
  • Hermetic speed_page_test.py + bench_report_test.py: 19 passed
  • cd docs && pnpm build: succeeds; dist/speed.html renders the summary table and 24 per-case tables.
  • Gated preflight (rule 4): 0 failing pairs.
  • Red: run 38187341258 failed bench_report_test.py::...it_lists_each_case_with_its_ratio_to_the_fastest_native (names not code-formatted).

@thekevinbot
thekevinbot merged commit 654c4f8 into main Oct 14, 2026
43 checks passed
@thekevinbot
thekevinbot deleted the bot/1470-speed-page-1011 branch October 14, 2026 14:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Performance suite: a benchmarking tool whose reports publish to the docs site

1 participant