Repository navigation
Per-file hooks and the persist refill update keyed rows in place - #1545
Open
thekevinbot wants to merge 5 commits into
Open
thekevinbot wants to merge 5 commits into
thekevinbot wants to merge 5 commits into
Conversation
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #1544
Extends the keyed refresh to the two paths that still deleted and reinserted a keyed table's rows:
handle_upsert): rows are matched against the rows that file owns by the table's key (thePRIMARY KEY, else the first unique, non-partial index the rows fill non-NULL). A changed row isUPDATEd by rowid with only the emitted columns written, a vanished key is deleted, a new key inserted; the file then owns its rows by their new index. Events aredelete/update/insertfromdiff_keyed, and anupdatecarries the kept columns (the generatedid). An unchanged file writes nothing. A table with no such key keeps the old delete-and-reinsert path.--persistrefill (stream_table_rows): when a config table already holds rows from the cache, the command's rows are gathered and then matched in place on a keyed table (reconcile_batch_rows), or deleted and reinserted on an unkeyed one. Nothing is written if the hook or a row fails. A cold build still streams.Reuse: the matching and writing core of
reconcile_batch_rowsmoved intoDb::reconcile_rows, shared byreconcile_batch_rowsand the newreconcile_file_rows;row_keyanddiff_keyedare reused as is.Since foreign-key checks are now deferred to commit, an edited keyed parent no longer fires
ON DELETE CASCADEon its children (test intests/foreign_keys.rs;docs/reference/config.mdupdated).Not in this PR: the per-file
--persistwarm re-parse of a changed file (lib.rs, theper_fileloop of the build) still deletes and reinserts.Red commit
e55ed1f5failed in CI (Linux and Windows) for the three asserted reasons: per-file update'snew_row["id"]was NULL, the cascading child was gone, and persist refill ids moved from 1,2 to 2,3.E2E Verification
cargo test -p dirsql --features cli)packages/python/e2e-attestations/<branch>.jsonreceipt written ifpackages/pythonchanged (just e2e-attest-python)packages/ts/e2e-attestations/<branch>.jsonreceipt written ifpackages/tschanged (just e2e-attest-ts)testing-conventions e2e attest 'uv run --no-sync python -m pytest tests/e2e/watch_speed_test.py tests/e2e/on_file_query_test.py tests/e2e/fanout_test.py -q -p no:cacheprovider'(packages/python);testing-conventions e2e attest 'pnpm test:e2e'(packages/ts);cargo test -p dirsql --features climain: branch medians 0.91-1.21 (median of medians 1.10),main0.97-1.11 (1.09); no difference beyond noise, andmainalso crosses the 1.1x bar at this load. Manual: Python SDK per-file tableexperiments(id INTEGER PRIMARY KEY, folder_name TEXT UNIQUE, name TEXT)in watch; renamingaemitted oneupdatewithid1 kept on both sides, ids unchanged.Changelog / Migrations
<root>/<pkg>/changelog.d/for each changed package (or:skip-changelogtrailer on a commit with reason)<root>/<pkg>/migrations.d/for each changed package (every subsection but Summary_None._when nothing breaks)