Skip to content

Pin silver input tables to fixed Delta versions per report run - #90

Merged
tombonfert merged 5 commits into
mainfrom
feature/silver_delta_version_pinning
Sep 9, 2026
Merged

Pin silver input tables to fixed Delta versions per report run#90
tombonfert merged 5 commits into
mainfrom
feature/silver_delta_version_pinning

Conversation

@tombonfert

Copy link
Copy Markdown
Collaborator

Summary

Report DataFrames are lazy, so every silver input read through
MeasurementDB._read_table (plain spark.read.table(...) in UC mode, or
spark.read.format("delta").load(...) in path mode) previously pinned no Delta
version. If an input table changed between when a run started and when a given
lazy op materialized, different stages of the same run could read different
snapshots, producing inconsistent results. This is easy to hit when reports run
while upstream ingestion is still writing.

This PR resolves each configured silver table's current Delta version once at
run start and reads every table with versionAsOf, so the whole run observes a
single consistent snapshot. Fixes #87.

Changes

  • Add MeasurementDBConfig.pinned_versions (URI to version; empty means "read
    latest", preserving existing behavior).
  • Add MeasurementDB.pin_versions(spark), which resolves the current Delta
    version of every configured silver table once via DeltaTable.history(1).
  • Update MeasurementDB._read_table to apply .option("versionAsOf", v) for
    both UC and path modes when a version is pinned.
  • Call self.db.pin_versions(self.spark) at the start of
    Report.determine_report(), before any solving.
  • Exempt debug (in-memory) mode. Non-Delta inputs, views, and unresolvable
    tables are skipped with a warning and keep reading latest, so deployments that
    use them are unaffected.

Testing

  • New tests/impulse_query_engine/unit/measurement_db_versioning_test.py:
    snapshot freezing (pin, append rows, assert the read still sees the pinned
    snapshot), debug-mode exemption, and graceful skip plus warning for
    unresolvable tables.
  • Full query engine suite (935 passed) and a reporting integration test through
    the complete determine_report path both pass. make lint is clean.

Test Plan

  • Unit tests added/updated
  • Manual testing completed
  • Documentation updated (if applicable)

Checklist

  • Code follows project style guidelines
  • Self-review completed
  • No new linter warnings introduced

…s lazy evaluation (#87)

- Add `MeasurementDB.pin_versions()` to resolve the current Delta version of
  every configured silver table once at run start.
- Update `MeasurementDB._read_table()` to read with `versionAsOf` when a pin
  exists, so all lazy reads in a run observe the same snapshot even if the
  table changes mid-run.
- Call `self.db.pin_versions()` from `Report` before solving.
- Exempt debug mode from pinning; skip non-Delta/unresolvable tables with a
  warning and continue reading latest.
- Add unit tests covering snapshot freezing, debug-mode exemption, and
  graceful degradation for unresolvable tables.
@tombonfert
tombonfert requested a review from a team as a code owner September 3, 2026 09:14
@codecov

codecov Bot commented Sep 3, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 89.50%. Comparing base (dfbf0a1) to head (533f425).
⚠️ Report is 4 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main      #90      +/-   ##
==========================================
+ Coverage   89.37%   89.50%   +0.12%     
==========================================
  Files          62       62              
  Lines        5544     5591      +47     
  Branches      680      686       +6     
==========================================
+ Hits         4955     5004      +49     
  Misses        466      466              
+ Partials      123      121       -2     
Flag Coverage Δ
query_engine 86.10% <100.00%> (+0.26%) ⬆️
reporting 94.25% <100.00%> (+<0.01%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines Coverage Δ
src/impulse_query_engine/measurement_db.py 87.85% <100.00%> (+8.58%) ⬆️
src/impulse_reporting/core/report.py 92.08% <100.00%> (+0.02%) ⬆️

... and 1 file with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

- Add a "Snapshot consistency" section to the query engine reference docs.
- Clarify in `MeasurementDB.pin_versions()` docstring that pinning is opt-in for direct query-engine use, automatic in `Report`, and persists until cleared.
Relocate `_configured_table_uris` from `MeasurementDB` to a public `configured_table_uris()` method on `MeasurementDBConfig`, and update `pin_versions()` to call it via `self.config`.
@tombonfert
tombonfert merged commit c06c3ef into main Sep 9, 2026
6 checks passed
@tombonfert
tombonfert deleted the feature/silver_delta_version_pinning branch September 9, 2026 07:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Pin silver input tables to fixed Delta versions for a report run

1 participant