Skip to content

perf: judge a wide directory in parts the walk schedules like directories - #1535

Merged
thekevinbot merged 8 commits into
mainfrom
bot/1477-onelevel-speed-1016
Oct 19, 2026
Merged

thekevinbot merged 8 commits into
mainfrom
bot/1477-onelevel-speed-1016

Conversation

@thekevinbot

@thekevinbot thekevinbot commented Oct 16, 2026 •

Copy link
Copy Markdown
Collaborator

Fixes #1477

./*/*.md over the depth-bound tree: 32 directories, each tens of thousands of entries wide. A directory's rows reached the SQLite consumer only once its whole listing had been judged. On this tree the consumer waited about 130-190ms for its first chunk, then serialized about 2.7M rows mostly after the walk had already finished.

A directory of more than PART_ENTRIES (8192) entries is now returned from Place::explore as parts. Each part is a Step::Dir holding the shared Arc<Listing> and an entry range. walk_in_order schedules those parts like any directory: workers judge them ahead, and the walk thread hands their rows out in order. Rows start flowing after the first part, and consumption overlaps the walk.

That parallelism replaces judge_all, idle_cores and tree_walk::busy_workers (added in #1522), which are removed. tree_walk.rs goes back to its form before #1522.

Output is unchanged: byte-identical against main for ./*/*.md, ./* and ./**/*.md on the 3.2M-item tree, plus a mixed wide tree.

Measurements

These are paired medians vs fdfind, net of startup, on a host with other agents running.

case main this PR
depth-bound ./*/*.md, e2e test (SPEED fastest, 15 pairs) 1.124 0.861
depth-bound ./*/*.md, CLI capture (15 pairs) 1.122 0.910

The capture bench gave these for other shapes: flat ./*.md 0.88 -> 0.71, flat ./** 0.83 -> 0.69, mixed 0.90 -> 0.75, one-level ./* 0.81 -> 0.67.

Expected conflict: #1521 keeps busy_workers in tree_walk.rs, which this PR deletes. The parts also work on top of #1521's ordered heap: about 0.89x fd on that combination.

E2E Verification

  • Ran e2e suites locally for every affected SDK
  • Python SDK e2e: pass
  • TypeScript SDK e2e: pass
  • Rust core e2e (if applicable): pass
  • packages/python/e2e-attestations/<branch>.json receipt written if packages/python changed (just e2e-attest-python)
  • packages/ts/e2e-attestations/<branch>.json receipt written if packages/ts changed (just e2e-attest-ts)
  • Command(s) run:
    • cargo test --workspace --features cli
    • cd packages/python && uv run maturin develop --release
    • testing-conventions e2e attest 'uv run python -m pytest tests/e2e/glob_speed_{depth_bound,one_level,recursive,matchers,dotfiles,gitignore,node_modules,symlink}_test.py tests/e2e/query_contract_test.py tests/e2e/fanout_test.py tests/e2e/use_case_join_test.py -q -s'
    • cd packages/ts && testing-conventions e2e attest 'pnpm test:e2e'
  • Result summary:
    • Rust: all pass.
    • Python e2e: 28 passed.
    • TS e2e: 18 passed.

Changelog / Migrations

  • Changelog fragment added under <root>/<pkg>/changelog.d/ for each changed package (or: skip-changelog trailer on a commit with reason)
  • Migration fragment with the same slug added under <root>/<pkg>/migrations.d/ for each changed package (every subsection but Summary _None._ when nothing breaks)

A directory of more than 8192 entries yields its parts as directories to enter, so its rows reach the consumer as each part is judged instead of once the whole listing is. The parts spread across the walk's workers, which replaces judge_all's split and the busy-worker count it needed.
…ed-1016

# Conflicts:
#	packages/rust/src/scanner.rs
#	packages/rust/src/tree_walk.rs
@thekevinbot
thekevinbot merged commit 976c6b7 into main Oct 19, 2026
89 checks passed
@thekevinbot
thekevinbot deleted the bot/1477-onelevel-speed-1016 branch October 19, 2026 11:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Speed: ./*/*.md is 4.2x the fastest native (fd)

1 participant