Pin silver input tables to fixed Delta versions per report run - #90
Merged
Conversation
…s lazy evaluation (#87) - Add `MeasurementDB.pin_versions()` to resolve the current Delta version of every configured silver table once at run start. - Update `MeasurementDB._read_table()` to read with `versionAsOf` when a pin exists, so all lazy reads in a run observe the same snapshot even if the table changes mid-run. - Call `self.db.pin_versions()` from `Report` before solving. - Exempt debug mode from pinning; skip non-Delta/unresolvable tables with a warning and continue reading latest. - Add unit tests covering snapshot freezing, debug-mode exemption, and graceful degradation for unresolvable tables.
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #90 +/- ##
==========================================
+ Coverage 89.37% 89.50% +0.12%
==========================================
Files 62 62
Lines 5544 5591 +47
Branches 680 686 +6
==========================================
+ Hits 4955 5004 +49
Misses 466 466
+ Partials 123 121 -2
Flags with carried forward coverage won't be shown. Click here to find out more.
🚀 New features to boost your workflow:
|
- Add a "Snapshot consistency" section to the query engine reference docs. - Clarify in `MeasurementDB.pin_versions()` docstring that pinning is opt-in for direct query-engine use, automatic in `Report`, and persists until cleared.
Relocate `_configured_table_uris` from `MeasurementDB` to a public `configured_table_uris()` method on `MeasurementDBConfig`, and update `pin_versions()` to call it via `self.config`.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Report DataFrames are lazy, so every silver input read through
MeasurementDB._read_table(plainspark.read.table(...)in UC mode, orspark.read.format("delta").load(...)in path mode) previously pinned no Deltaversion. If an input table changed between when a run started and when a given
lazy op materialized, different stages of the same run could read different
snapshots, producing inconsistent results. This is easy to hit when reports run
while upstream ingestion is still writing.
This PR resolves each configured silver table's current Delta version once at
run start and reads every table with
versionAsOf, so the whole run observes asingle consistent snapshot. Fixes #87.
Changes
MeasurementDBConfig.pinned_versions(URI to version; empty means "readlatest", preserving existing behavior).
MeasurementDB.pin_versions(spark), which resolves the current Deltaversion of every configured silver table once via
DeltaTable.history(1).MeasurementDB._read_tableto apply.option("versionAsOf", v)forboth UC and path modes when a version is pinned.
self.db.pin_versions(self.spark)at the start ofReport.determine_report(), before any solving.tables are skipped with a warning and keep reading latest, so deployments that
use them are unaffected.
Testing
tests/impulse_query_engine/unit/measurement_db_versioning_test.py:snapshot freezing (pin, append rows, assert the read still sees the pinned
snapshot), debug-mode exemption, and graceful skip plus warning for
unresolvable tables.
the complete
determine_reportpath both pass.make lintis clean.Test Plan
Checklist