Repository navigation
perf: judge a wide directory in parts the walk schedules like directories - #1535
Merged
Merged
Conversation
A directory of more than 8192 entries yields its parts as directories to enter, so its rows reach the consumer as each part is judged instead of once the whole listing is. The parts spread across the walk's workers, which replaces judge_all's split and the busy-worker count it needed.
…outside --gitignore
…ed-1016 # Conflicts: # packages/rust/src/scanner.rs # packages/rust/src/tree_walk.rs
7 tasks done
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #1477
./*/*.mdover the depth-bound tree: 32 directories, each tens of thousands of entries wide. A directory's rows reached the SQLite consumer only once its whole listing had been judged. On this tree the consumer waited about 130-190ms for its first chunk, then serialized about 2.7M rows mostly after the walk had already finished.A directory of more than
PART_ENTRIES(8192) entries is now returned fromPlace::exploreas parts. Each part is aStep::Dirholding the sharedArc<Listing>and an entry range.walk_in_orderschedules those parts like any directory: workers judge them ahead, and the walk thread hands their rows out in order. Rows start flowing after the first part, and consumption overlaps the walk.That parallelism replaces
judge_all,idle_coresandtree_walk::busy_workers(added in #1522), which are removed.tree_walk.rsgoes back to its form before #1522.Output is unchanged: byte-identical against main for
./*/*.md,./*and./**/*.mdon the 3.2M-item tree, plus a mixed wide tree.Measurements
These are paired medians vs
fdfind, net of startup, on a host with other agents running../*/*.md, e2e test (SPEED fastest, 15 pairs)./*/*.md, CLI capture (15 pairs)The capture bench gave these for other shapes: flat
./*.md0.88 -> 0.71, flat./**0.83 -> 0.69, mixed 0.90 -> 0.75, one-level./*0.81 -> 0.67.Expected conflict: #1521 keeps
busy_workersintree_walk.rs, which this PR deletes. The parts also work on top of #1521's ordered heap: about 0.89x fd on that combination.E2E Verification
packages/python/e2e-attestations/<branch>.jsonreceipt written ifpackages/pythonchanged (just e2e-attest-python)packages/ts/e2e-attestations/<branch>.jsonreceipt written ifpackages/tschanged (just e2e-attest-ts)cargo test --workspace --features clicd packages/python && uv run maturin develop --releasetesting-conventions e2e attest 'uv run python -m pytest tests/e2e/glob_speed_{depth_bound,one_level,recursive,matchers,dotfiles,gitignore,node_modules,symlink}_test.py tests/e2e/query_contract_test.py tests/e2e/fanout_test.py tests/e2e/use_case_join_test.py -q -s'cd packages/ts && testing-conventions e2e attest 'pnpm test:e2e'Changelog / Migrations
<root>/<pkg>/changelog.d/for each changed package (or:skip-changelogtrailer on a commit with reason)<root>/<pkg>/migrations.d/for each changed package (every subsection but Summary_None._when nothing breaks)