Skip to content

Per-file hooks and the persist refill update keyed rows in place - #1545

Open
thekevinbot wants to merge 5 commits into
mainfrom
bot/1544-perfile-update-1020
Open

thekevinbot wants to merge 5 commits into
mainfrom
bot/1544-perfile-update-1020

Conversation

@thekevinbot

@thekevinbot thekevinbot commented Oct 20, 2026 •

Copy link
Copy Markdown
Collaborator

Fixes #1544

Extends the keyed refresh to the two paths that still deleted and reinserted a keyed table's rows:

  • Per-file hooks (handle_upsert): rows are matched against the rows that file owns by the table's key (the PRIMARY KEY, else the first unique, non-partial index the rows fill non-NULL). A changed row is UPDATEd by rowid with only the emitted columns written, a vanished key is deleted, a new key inserted; the file then owns its rows by their new index. Events are delete/update/insert from diff_keyed, and an update carries the kept columns (the generated id). An unchanged file writes nothing. A table with no such key keeps the old delete-and-reinsert path.
  • --persist refill (stream_table_rows): when a config table already holds rows from the cache, the command's rows are gathered and then matched in place on a keyed table (reconcile_batch_rows), or deleted and reinserted on an unkeyed one. Nothing is written if the hook or a row fails. A cold build still streams.

Reuse: the matching and writing core of reconcile_batch_rows moved into Db::reconcile_rows, shared by reconcile_batch_rows and the new reconcile_file_rows; row_key and diff_keyed are reused as is.

Since foreign-key checks are now deferred to commit, an edited keyed parent no longer fires ON DELETE CASCADE on its children (test in tests/foreign_keys.rs; docs/reference/config.md updated).

Not in this PR: the per-file --persist warm re-parse of a changed file (lib.rs, the per_file loop of the build) still deletes and reinserts.

Red commit e55ed1f5 failed in CI (Linux and Windows) for the three asserted reasons: per-file update's new_row["id"] was NULL, the cascading child was gone, and persist refill ids moved from 1,2 to 2,3.

E2E Verification

  • Ran e2e suites locally for every affected SDK
  • Python SDK e2e: pass
  • TypeScript SDK e2e: pass
  • Rust core e2e (if applicable): pass (cargo test -p dirsql --features cli)
  • packages/python/e2e-attestations/<branch>.json receipt written if packages/python changed (just e2e-attest-python)
  • packages/ts/e2e-attestations/<branch>.json receipt written if packages/ts changed (just e2e-attest-ts)
  • Command(s) run: testing-conventions e2e attest 'uv run --no-sync python -m pytest tests/e2e/watch_speed_test.py tests/e2e/on_file_query_test.py tests/e2e/fanout_test.py -q -p no:cacheprovider' (packages/python); testing-conventions e2e attest 'pnpm test:e2e' (packages/ts); cargo test -p dirsql --features cli
  • Result summary: all pass. Watch speed vs inotifywait under load 7-17, six paired runs each, branch vs main: branch medians 0.91-1.21 (median of medians 1.10), main 0.97-1.11 (1.09); no difference beyond noise, and main also crosses the 1.1x bar at this load. Manual: Python SDK per-file table experiments(id INTEGER PRIMARY KEY, folder_name TEXT UNIQUE, name TEXT) in watch; renaming a emitted one update with id 1 kept on both sides, ids unchanged.

Changelog / Migrations

  • Changelog fragment added under <root>/<pkg>/changelog.d/ for each changed package (or: skip-changelog trailer on a commit with reason)
  • Migration fragment with the same slug added under <root>/<pkg>/migrations.d/ for each changed package (every subsection but Summary _None._ when nothing breaks)

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Per-file hooks and the persist refill update keyed rows in place

1 participant