From a3331db6e98940d67069df67ef643874f2420244 Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Thu, 27 Aug 2026 07:53:27 +0200 Subject: [PATCH 01/49] Start stacked release gate alignment journal --- PROGRESS.md | 32 +++++++++++++++++++++++++++++++- 1 file changed, 31 insertions(+), 1 deletion(-) diff --git a/PROGRESS.md b/PROGRESS.md index a0394ddf0..6df8432a8 100644 --- a/PROGRESS.md +++ b/PROGRESS.md @@ -1,4 +1,34 @@ -# Gate-failed base-pool release lane +# Weeksgate: stacked release gates and integer-week provenance + +## State + +In progress on 2026-08-27. The branch and clean `origin/main` base have been +verified, repository instructions have been read, and the supplied release-log +symptoms are ready for direct source/artifact tracing. No network access, +artifact build, publication, push, pool build, or release build is in scope. + +## Done + +- Read `CLAUDE.md` and the GitNexus debugging workflow. +- Confirmed branch `stacked-release-gate-alignment` is clean at `4f453746`. +- Confirmed the local GitNexus CLI is installed but the repository is not yet + indexed; an offline local index may be generated for call-path tracing. +- Recorded the four requested workstreams: fractional-week provenance and PUF + misclassification; integer-support calibration repair; stacked/legacy weeks + gate alignment; and the full release-side gate archaeology sweep. +- Recorded the required verification boundary: repository Ruff plus one pytest + process per shard, with no pool/release builds. + +## Next + +- Trace the release-gate roster and weeks/calibration call paths. +- Read the supplied pool and manifests to classify every noninteger week row + and diagnose the 5,218-row channel error. +- Implement and test only evidence-supported source and architecture fixes. +- Audit all release-side gates, run the complete prescribed verification, and + write the final evidence and judgment calls to `out.md`. + +# Historical: gate-failed base-pool release lane ## State From 9979d1017917888bc5c98d766fe4d06686222ab6 Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Thu, 27 Aug 2026 08:00:11 +0200 Subject: [PATCH 02/49] Record fractional weeks provenance --- PROGRESS.md | 42 +++++++++++++++++++++++++++++++++--------- 1 file changed, 33 insertions(+), 9 deletions(-) diff --git a/PROGRESS.md b/PROGRESS.md index 6df8432a8..999360081 100644 --- a/PROGRESS.md +++ b/PROGRESS.md @@ -2,29 +2,53 @@ ## State -In progress on 2026-08-27. The branch and clean `origin/main` base have been -verified, repository instructions have been read, and the supplied release-log -symptoms are ready for direct source/artifact tracing. No network access, -artifact build, publication, push, pool build, or release build is in scope. +In progress on 2026-08-27. Real-pool provenance has refuted the proposed +post-transfer amount-mapping mechanism: every fractional week is an ACS-origin +non-native clone prediction outside the calibration's clone-0 recipient scope. +The source codec and release-gate architecture fixes are now being designed. +No network access, artifact build, publication, push, pool build, or release +build is in scope. ## Done - Read `CLAUDE.md` and the GitNexus debugging workflow. - Confirmed branch `stacked-release-gate-alignment` is clean at `4f453746`. - Confirmed the local GitNexus CLI is installed but the repository is not yet - indexed; an offline local index may be generated for call-path tracing. + indexed. Its offline analyzer parsed the repository but could not register + the index because the sandbox forbids writes to `~/.gitnexus`; the generated + local index was moved out of the worktree to `/private/tmp`. - Recorded the four requested workstreams: fractional-week provenance and PUF misclassification; integer-support calibration repair; stacked/legacy weeks gate alignment; and the full release-side gate archaeology sweep. - Recorded the required verification boundary: repository Ruff plus one pytest process per shard, with no pool/release builds. +- Read the fixed-format HDF5 blocks directly and classified all 369 noninteger + `weeks_unemployed` rows: 360 are ACS clone 1 (355 UC=0, 5 UC>0) and 9 are ACS + clone 2 (all UC=0); all are positive, all 369 values are distinct, and the + exact range is 1.0003521955067698--37.796501228614694. +- Confirmed zero nonintegers on ASEC rows and ACS clone 0. The receipted + calibration covers exactly the 856,626 ACS clone-0 rows, maps 8,419 carrier + amounts onto observed ASEC support with zero donor-support violations, and + records QED 0.5882352941176471 to 0.0. +- Reproduced the 5,218-row false "PUF" classification: the legacy role helper + calls every clone index above zero `puf_tax_detail`, regardless of raw source + channel. The rows are all ACS-origin clones: 4,733 integer clone-1 rows, 355 + fractional clone-1 rows, 121 integer clone-2 rows, and 9 fractional clone-2 + rows with nonzero weeks while UC is nonpositive. +- Traced the actual fractional mechanism to the ACS transfer target codec: + PolicyEngine-US declares `weeks_unemployed` as physical `float`, so the + generic QRF path treats it as continuous even though its reviewed source + contract is integer-supported. The later calibration repairs clone 0 only. ## Next -- Trace the release-gate roster and weeks/calibration call paths. -- Read the supplied pool and manifests to classify every noninteger week row - and diagnose the 5,218-row channel error. -- Implement and test only evidence-supported source and architecture fixes. +- Bind `weeks_unemployed` to observed integer donor support in the actual ACS + transfer codec and ensure the authority/receipt contract records that policy. +- Modernize the weeks gate to use raw assembled channels when present, scope + LKWEEKS reconciliation to ASEC-source rows, and scope UC consistency to the + rows for which the source constraint is defined. +- Complete the full release-gate roster audit and implement only unambiguous + stacked-awareness repairs. - Audit all release-side gates, run the complete prescribed verification, and write the final evidence and judgment calls to `out.md`. From d7ad753e1342750a0f5b8dede72692f7319d14c8 Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Thu, 27 Aug 2026 08:05:25 +0200 Subject: [PATCH 03/49] Preserve integer support for transferred weeks --- PROGRESS.md | 20 +++++++-- .../build/us_runtime/acs_transfer.py | 9 ++-- .../tests/test_us_acs_transfer.py | 42 +++++++++++++++++++ 3 files changed, 64 insertions(+), 7 deletions(-) diff --git a/PROGRESS.md b/PROGRESS.md index 999360081..c3e2ff4c3 100644 --- a/PROGRESS.md +++ b/PROGRESS.md @@ -5,7 +5,8 @@ In progress on 2026-08-27. Real-pool provenance has refuted the proposed post-transfer amount-mapping mechanism: every fractional week is an ACS-origin non-native clone prediction outside the calibration's clone-0 recipient scope. -The source codec and release-gate architecture fixes are now being designed. +The source codec fix is implemented and focused-tested; release-gate +architecture fixes and the broader audit are now in progress. No network access, artifact build, publication, push, pool build, or release build is in scope. @@ -39,11 +40,24 @@ build is in scope. PolicyEngine-US declares `weeks_unemployed` as physical `float`, so the generic QRF path treats it as continuous even though its reviewed source contract is integer-supported. The later calibration repairs clone 0 only. +- Bound every final fractional value bit-for-bit to the late-transfer target + bank. Its raw QRF draw contains 711 nonintegers: 342 ACS clone 0, 360 clone 1, + and 9 clone 2. Post-transfer calibration changes 13,417 clone-0 rows and + eliminates all 342 clone-0 fractions; the 369 out-of-scope clone fractions + pass through unchanged. +- Added `weeks_unemployed` to the ACS transfer's authority-bound discrete + numeric target registry. The existing codec now snaps every prediction to + actual observed ASEC donor support, and the execution-contract identity + receipts the policy alongside the two mortgage-year targets. +- Added a focused ACS-transfer regression proving weeks predictions retain + observed integer support and the execution contract declares the target. +- Passed the complete ACS-transfer test file (65 tests), the complete + post-transfer calibration receipt-contract file (47 tests), focused Ruff, + and `git diff --check` using the prebuilt `.venv` directly. `uv run --no-sync` + is unavailable because this sandbox forbids uv's read of `~/.cache/uv`. ## Next -- Bind `weeks_unemployed` to observed integer donor support in the actual ACS - transfer codec and ensure the authority/receipt contract records that policy. - Modernize the weeks gate to use raw assembled channels when present, scope LKWEEKS reconciliation to ASEC-source rows, and scope UC consistency to the rows for which the source constraint is defined. diff --git a/packages/microcosm-build/src/microcosm/build/us_runtime/acs_transfer.py b/packages/microcosm-build/src/microcosm/build/us_runtime/acs_transfer.py index f791823a0..8688d8468 100644 --- a/packages/microcosm-build/src/microcosm/build/us_runtime/acs_transfer.py +++ b/packages/microcosm-build/src/microcosm/build/us_runtime/acs_transfer.py @@ -126,14 +126,15 @@ } ) -# These inputs are numeric in current dense artifacts, but their domain is a -# finite set of years. QRF quantile interpolation is valid for continuous -# amounts and invalid for a year code, so predictions are snapped to observed -# donor support exactly as the existing PUF support stage does. +# These inputs are numeric in current dense artifacts, but their domain has +# finite integer support. QRF quantile interpolation is valid for continuous +# amounts and invalid for year codes or counts, so predictions are snapped to +# observed donor support exactly as the existing PUF support stage does. _DISCRETE_NUMERIC_TARGETS = frozenset( { "first_home_mortgage_origination_year", "second_home_mortgage_origination_year", + "weeks_unemployed", } ) diff --git a/packages/microcosm-build/tests/test_us_acs_transfer.py b/packages/microcosm-build/tests/test_us_acs_transfer.py index a436f50a8..f08c84c30 100644 --- a/packages/microcosm-build/tests/test_us_acs_transfer.py +++ b/packages/microcosm-build/tests/test_us_acs_transfer.py @@ -1388,6 +1388,48 @@ def test_discrete_year_predictions_snap_to_observed_donor_support( assert pd.api.types.is_integer_dtype(values.dtype) +def test_integer_supported_weeks_predictions_snap_to_observed_donor_support( + monkeypatch: pytest.MonkeyPatch, +) -> None: + donor_support = {0.0, 1.0, 3.0, 8.0, 13.0} + donor = _with_columns( + _donor_frame(), + "person", + { + "weeks_unemployed": [ + 0.0, + 0.0, + 0.0, + 1.0, + 3.0, + 3.0, + 8.0, + 13.0, + ] + }, + ) + monkeypatch.setattr(acs_transfer_module, "QRF", _MeanQRF) + _MeanQRF.calls = [] + + result = transfer_acs_inputs( + _recipient_frame(), + donor, + target_families={ + "person": {"weeks": ("weeks_unemployed",)}, + }, + seed=2, + n_estimators=1, + ) + + values = result.frame.table("person")["weeks_unemployed"] + assert set(values).issubset(donor_support) + assert pd.api.types.is_integer_dtype(values.dtype) + contract = acs_transfer_module.acs_transfer_execution_contract_identity( + targets=("weeks_unemployed",), + ) + assert "weeks_unemployed" in contract["discrete_numeric_targets"] + + def test_engine_boolean_metadata_restores_primary_qrf_float_h5_donor( monkeypatch: pytest.MonkeyPatch, tmp_path: Path, From 0c5b05a8ec4e6bf3d5a375b615f45c3205dd9bac Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Thu, 27 Aug 2026 08:11:38 +0200 Subject: [PATCH 04/49] Receipt integer weeks transfer support --- PROGRESS.md | 5 +++++ .../src/microcosm/build/us/spec/imputation.yaml | 1 + 2 files changed, 6 insertions(+) diff --git a/PROGRESS.md b/PROGRESS.md index c3e2ff4c3..077a7fdd4 100644 --- a/PROGRESS.md +++ b/PROGRESS.md @@ -51,6 +51,11 @@ build is in scope. receipts the policy alongside the two mortgage-year targets. - Added a focused ACS-transfer regression proving weeks predictions retain observed integer support and the execution contract declares the target. +- Regenerated the checked-in US imputation authority so + `transfer_execution.discrete_numeric_targets` includes `weeks_unemployed`. + The generator's compile and byte-staleness checks pass at bundle spec SHA + `821d5838da3ac368170e61e017f1a72648f93e8a011aa40e33b8c2b4b14511f3`; + the spec-bundle and imputation-semantics suites pass. - Passed the complete ACS-transfer test file (65 tests), the complete post-transfer calibration receipt-contract file (47 tests), focused Ruff, and `git diff --check` using the prebuilt `.venv` directly. `uv run --no-sync` diff --git a/packages/microcosm-build/src/microcosm/build/us/spec/imputation.yaml b/packages/microcosm-build/src/microcosm/build/us/spec/imputation.yaml index f0a7ed69c..99d853a9a 100644 --- a/packages/microcosm-build/src/microcosm/build/us/spec/imputation.yaml +++ b/packages/microcosm-build/src/microcosm/build/us/spec/imputation.yaml @@ -408,6 +408,7 @@ transfer_execution: discrete_numeric_targets: - first_home_mortgage_origination_year - second_home_mortgage_origination_year + - weeks_unemployed predictor_bindings: person_required: acs_person_required person_optional: From c22a67994fd15840322b2bb7e9bedf4641f3a70c Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Thu, 27 Aug 2026 08:16:12 +0200 Subject: [PATCH 05/49] Align weeks gate with stacked source roles --- PROGRESS.md | 20 ++- .../build/us_runtime/weeks_unemployed.py | 130 ++++++++++++++---- .../tests/test_us_weeks_unemployed.py | 108 +++++++++++++++ 3 files changed, 230 insertions(+), 28 deletions(-) diff --git a/PROGRESS.md b/PROGRESS.md index 077a7fdd4..21a1e6477 100644 --- a/PROGRESS.md +++ b/PROGRESS.md @@ -5,8 +5,8 @@ In progress on 2026-08-27. Real-pool provenance has refuted the proposed post-transfer amount-mapping mechanism: every fractional week is an ACS-origin non-native clone prediction outside the calibration's clone-0 recipient scope. -The source codec fix is implemented and focused-tested; release-gate -architecture fixes and the broader audit are now in progress. +The source codec and weeks-gate architecture fixes are implemented and +focused-tested; the broader release-gate audit and repairs are now in progress. No network access, artifact build, publication, push, pool build, or release build is in scope. @@ -60,12 +60,22 @@ build is in scope. post-transfer calibration receipt-contract file (47 tests), focused Ruff, and `git diff --check` using the prebuilt `.venv` directly. `uv run --no-sync` is unavailable because this sandbox forbids uv's read of `~/.cache/uv`. +- Modernized the weeks summary/gate to derive its roster from actual assembled + source channels, while retaining the legacy ASEC/PUF role path. The ASEC + source-validity scope, direct native-clone reconciliation scope, and reviewed + UC-constraint scope are now distinct and explicitly receipted in details. +- Kept all four plausibility-band tuples and every numeric threshold unchanged; + non-ASEC assembled channels use the unchanged legacy recipient band. +- Added stacked ASEC+ACS and legacy ASEC+PUF fixtures covering roster detection, + raw-source scoping, native reconciliation, and UC constraint ownership. The + complete weeks file passes (26 passed, 1 skipped) with focused Ruff. +- Replayed the updated gate over the supplied pool's exact live arrays and + weights. It now reports 234,133 valid ASEC source rows, 108,073 exact native + reconciliation rows, 982,686 UC-constrained rows, zero source/UC failures, + both unchanged channel bands passing, and only the genuine 369 nonintegers. ## Next -- Modernize the weeks gate to use raw assembled channels when present, scope - LKWEEKS reconciliation to ASEC-source rows, and scope UC consistency to the - rows for which the source constraint is defined. - Complete the full release-gate roster audit and implement only unambiguous stacked-awareness repairs. - Audit all release-side gates, run the complete prescribed verification, and diff --git a/packages/microcosm-build/src/microcosm/build/us_runtime/weeks_unemployed.py b/packages/microcosm-build/src/microcosm/build/us_runtime/weeks_unemployed.py index bd156ecbc..84085120f 100644 --- a/packages/microcosm-build/src/microcosm/build/us_runtime/weeks_unemployed.py +++ b/packages/microcosm-build/src/microcosm/build/us_runtime/weeks_unemployed.py @@ -42,6 +42,9 @@ ) from microcosm.build.us_runtime.support_provenance import ( has_support_role_metadata, + spine_source_id_column, + support_channel_column, + support_clone_index_column, support_role_series, ) from microcosm.frame import Frame @@ -1220,14 +1223,18 @@ def us_weeks_unemployed_summary(frame: Frame) -> dict[str, object]: in_range = integer & (values >= 0.0) & (values <= 52.0) positive = in_range & (values > 0.0) - if has_support_role_metadata(person, entity="person"): - channel = support_role_series(person, entity="person").to_numpy() - else: - channel = np.full(len(person), _ASEC_CHANNEL, dtype=object) + ( + channel, + channel_roster, + source_rows, + source_reconciliation_rows, + uc_constraint_rows, + ) = _weeks_unemployed_gate_scopes(person) channels: dict[str, dict[str, float | int]] = {} - for name in (_ASEC_CHANNEL, _PUF_CHANNEL): + for name in channel_roster: mask = channel == name channel_weight = float(weights[mask].sum()) + plausibility_role = _ASEC_CHANNEL if name == _ASEC_CHANNEL else _PUF_CHANNEL channels[name] = { "rows": int(np.count_nonzero(mask)), "positive_rows": int(np.count_nonzero(mask & positive)), @@ -1243,8 +1250,12 @@ def us_weeks_unemployed_summary(frame: Frame) -> dict[str, object]: if channel_weight > 0.0 else 0.0 ), - "positive_share_band": list(_CHANNEL_POSITIVE_SHARE_BANDS[name]), - "weighted_mean_weeks_band": list(_CHANNEL_WEIGHTED_MEAN_WEEKS_BANDS[name]), + "positive_share_band": list( + _CHANNEL_POSITIVE_SHARE_BANDS[plausibility_role] + ), + "weighted_mean_weeks_band": list( + _CHANNEL_WEIGHTED_MEAN_WEEKS_BANDS[plausibility_role] + ), } source_missing = _SOURCE not in person @@ -1253,25 +1264,32 @@ def us_weeks_unemployed_summary(frame: Frame) -> dict[str, object]: source_raw = pd.to_numeric(person[_SOURCE], errors="coerce").to_numpy( dtype=np.float64 ) - asec_mask = channel == _ASEC_CHANNEL source_valid = np.isfinite(source_raw) & (source_raw == np.floor(source_raw)) source_valid &= (source_raw == -1.0) | ( (source_raw >= 0.0) & (source_raw <= 52.0) ) - source_invalid = int(np.count_nonzero(asec_mask & ~source_valid)) + source_invalid = int(np.count_nonzero(source_rows & ~source_valid)) expected = np.where(source_raw == -1.0, 0.0, source_raw) source_mismatch = int( - np.count_nonzero(asec_mask & source_valid & finite & (values != expected)) + np.count_nonzero( + source_reconciliation_rows + & source_valid + & finite + & (values != expected) + ) ) - puf_uc_zero_mismatch = 0 + uc_constraint_mismatch = 0 if _OPTIONAL_UC_PREDICTOR in person: uc = pd.to_numeric(person[_OPTIONAL_UC_PREDICTOR], errors="coerce").to_numpy( dtype=np.float64 ) - puf_mask = channel == _PUF_CHANNEL - puf_uc_zero_mismatch = int( + uc_constraint_mismatch = int( np.count_nonzero( - puf_mask & np.isfinite(uc) & (uc <= 0.0) & finite & (values != 0.0) + uc_constraint_rows + & np.isfinite(uc) + & (uc <= 0.0) + & finite + & (values != 0.0) ) ) @@ -1284,15 +1302,84 @@ def us_weeks_unemployed_summary(frame: Frame) -> dict[str, object]: "positive_share": float(weights[positive].sum() / weights.sum()), "weighted_weeks": float(np.dot(weights, np.nan_to_num(values))), "source_missing": source_missing, + "source_rows": int(np.count_nonzero(source_rows)), + "source_reconciliation_rows": int( + np.count_nonzero(source_reconciliation_rows) + ), "source_invalid": source_invalid, "source_mismatch_count": source_mismatch, - "puf_uc_zero_mismatch_count": puf_uc_zero_mismatch, + "uc_constraint_rows": int(np.count_nonzero(uc_constraint_rows)), + "uc_constraint_mismatch_count": uc_constraint_mismatch, + "channel_roster": list(channel_roster), "channels": channels, } +def _weeks_unemployed_gate_scopes( + person: pd.DataFrame, +) -> tuple[np.ndarray, tuple[str, ...], np.ndarray, np.ndarray, np.ndarray]: + """Resolve source channels and the two reviewed constraint scopes.""" + + rows = len(person) + if not has_support_role_metadata(person, entity="person"): + all_rows = np.ones(rows, dtype=bool) + return ( + np.full(rows, _ASEC_CHANNEL, dtype=object), + (_ASEC_CHANNEL,), + all_rows, + all_rows.copy(), + np.zeros(rows, dtype=bool), + ) + + # Validate complete channel/clone metadata even when an assembled frame's + # raw source channels, rather than its compatibility roles, own the gate. + roles = support_role_series(person, entity="person") + assembled = spine_source_id_column("person") in person + if not assembled: + channel = roles.to_numpy(dtype=object, copy=True) + roster = tuple( + name for name in (_ASEC_CHANNEL, _PUF_CHANNEL) if np.any(channel == name) + ) + source_rows = channel == _ASEC_CHANNEL + if not np.any(source_rows): + raise ValueError("US weeks-unemployed support has no ASEC source rows.") + return ( + channel, + roster, + source_rows, + source_rows.copy(), + channel == _PUF_CHANNEL, + ) + + channel_column = support_channel_column("person") + clone_column = support_clone_index_column("person") + source_channel = person[channel_column].to_numpy(dtype=object, copy=True) + observed = {str(value) for value in source_channel} + if _ASEC_CHANNEL not in observed: + raise ValueError("US weeks-unemployed support has no ASEC source rows.") + roster = (_ASEC_CHANNEL, *sorted(observed - {_ASEC_CHANNEL})) + clone_index = pd.to_numeric(person[clone_column], errors="raise").to_numpy( + dtype=np.int64 + ) + native = clone_index == 0 + source_rows = source_channel == _ASEC_CHANNEL + # Native ASEC rows are the direct LKWEEKS carry. Non-native ASEC rows are + # the reviewed PUF-detail imputation; native non-ASEC rows are the reviewed + # post-transfer calibration scope. Other non-native source rows have no UC + # zero rule in either producer contract. + source_reconciliation_rows = source_rows & native + uc_constraint_rows = (source_rows & ~native) | (~source_rows & native) + return ( + source_channel, + roster, + source_rows, + source_reconciliation_rows, + uc_constraint_rows, + ) + + def us_weeks_unemployed_signal_gate(frame: Frame) -> GateResult: - """Require exact ASEC carry and integer, nondefault signal on both halves.""" + """Require exact ASEC carry and integer signal on every support channel.""" person = frame.table("person") if _OUTPUT not in person: @@ -1330,15 +1417,12 @@ def us_weeks_unemployed_signal_gate(frame: Frame) -> GateResult: f"{_OUTPUT} has {summary['source_mismatch_count']} ASEC source " "reconciliation mismatch(es)." ) - if int(summary["puf_uc_zero_mismatch_count"]): + if int(summary["uc_constraint_mismatch_count"]): failures.append( - f"{_OUTPUT} has {summary['puf_uc_zero_mismatch_count']} PUF row(s) " - "positive without unemployment compensation." + f"{_OUTPUT} has {summary['uc_constraint_mismatch_count']} row(s) " + "positive outside the unemployment-compensation constraint." ) - has_support_roles = has_support_role_metadata(person, entity="person") - required_channels = ( - (_ASEC_CHANNEL, _PUF_CHANNEL) if has_support_roles else (_ASEC_CHANNEL,) - ) + required_channels = tuple(summary["channel_roster"]) channels = summary["channels"] for name in required_channels: channel = channels[name] diff --git a/packages/microcosm-build/tests/test_us_weeks_unemployed.py b/packages/microcosm-build/tests/test_us_weeks_unemployed.py index bd6e33419..479b11490 100644 --- a/packages/microcosm-build/tests/test_us_weeks_unemployed.py +++ b/packages/microcosm-build/tests/test_us_weeks_unemployed.py @@ -243,6 +243,46 @@ def _gate_frame() -> Frame: ) +def _stacked_gate_frame() -> Frame: + frame = _gate_frame() + person = frame.table("person").copy() + asec_rows = 1_000 + asec_native_rows = 500 + acs_native_rows = 1_000 + person["person_support_channel"] = ["asec"] * asec_rows + ["acs"] * 2_000 + person["person_support_clone_index"] = np.concatenate( + [ + np.zeros(asec_native_rows, dtype=np.int64), + np.ones(asec_rows - asec_native_rows, dtype=np.int64), + np.zeros(acs_native_rows, dtype=np.int64), + np.ones(2_000 - acs_native_rows, dtype=np.int64), + ] + ) + person["person_spine_source_id"] = np.concatenate( + [ + np.arange(asec_native_rows, dtype=np.int64), + np.arange(asec_native_rows, dtype=np.int64), + np.arange(10_000, 10_000 + acs_native_rows, dtype=np.int64), + np.arange(10_000, 10_000 + acs_native_rows, dtype=np.int64), + ] + ) + weeks = np.zeros(len(person), dtype=np.float64) + weeks[:18] = 17.0 + weeks[asec_native_rows : asec_native_rows + 12] = 17.0 + weeks[asec_rows + acs_native_rows : asec_rows + acs_native_rows + 12] = 16.0 + person[_OUTPUT] = weeks + source = np.full(len(person), np.nan, dtype=np.float64) + source[:asec_rows] = 0.0 + source[:18] = 17.0 + person["LKWEEKS"] = source + unemployment_compensation = np.zeros(len(person), dtype=np.float64) + unemployment_compensation[ + asec_native_rows : asec_native_rows + 12 + ] = 100.0 + person["unemployment_compensation"] = unemployment_compensation + return module._replace_person_table(frame, person) + + def test_public_stage_contract_is_exactly_manifest_pinned() -> None: spec = us_weeks_unemployed_stage_spec() @@ -665,6 +705,74 @@ def test_signal_gate_requires_exact_asec_and_nondefault_integer_both_channels() assert any("puf_tax_detail" in failure for failure in gate.failures) +def test_signal_gate_derives_legacy_asec_puf_roster_and_constraint_scope() -> None: + frame = _gate_frame() + summary = module.us_weeks_unemployed_summary(frame) + + assert summary["channel_roster"] == ["asec", "puf_tax_detail"] + assert summary["source_rows"] == 1_000 + assert summary["source_reconciliation_rows"] == 1_000 + assert summary["uc_constraint_rows"] == 2_000 + assert summary["uc_constraint_mismatch_count"] == 0 + + person = frame.table("person").copy() + first_puf_carrier = person.index[ + person["person_support_channel"].eq("puf_tax_detail") + & person[_OUTPUT].gt(0.0) + ][0] + person.loc[first_puf_carrier, "unemployment_compensation"] = 0.0 + gate = us_weeks_unemployed_signal_gate(module._replace_person_table(frame, person)) + + assert not gate.passed + assert any("unemployment-compensation constraint" in item for item in gate.failures) + + +def test_signal_gate_derives_stacked_asec_acs_roster_and_reviewed_scopes() -> None: + frame = _stacked_gate_frame() + gate = us_weeks_unemployed_signal_gate(frame) + summary = gate.details + + assert gate.passed + assert summary["channel_roster"] == ["asec", "acs"] + assert set(summary["channels"]) == {"asec", "acs"} + assert summary["source_rows"] == 1_000 + assert summary["source_reconciliation_rows"] == 500 + assert summary["source_invalid"] == 0 + assert summary["source_mismatch_count"] == 0 + assert summary["uc_constraint_rows"] == 1_500 + assert summary["uc_constraint_mismatch_count"] == 0 + + +def test_signal_gate_stacked_source_and_uc_checks_ignore_unowned_rows() -> None: + frame = _stacked_gate_frame() + person = frame.table("person").copy() + acs_clone = person["person_support_channel"].eq("acs") + person.loc[acs_clone, "LKWEEKS"] = 99.0 + ignored = module._replace_person_table(frame, person) + assert us_weeks_unemployed_signal_gate(ignored).passed + + asec_clone_one = person["person_support_channel"].eq("asec") & person[ + "person_support_clone_index" + ].eq(1) + person.loc[person.index[asec_clone_one][0], "LKWEEKS"] = 99.0 + invalid_source = us_weeks_unemployed_signal_gate( + module._replace_person_table(frame, person) + ) + assert not invalid_source.passed + assert invalid_source.details["source_invalid"] == 1 + + person = frame.table("person").copy() + acs_native = person["person_support_channel"].eq("acs") & person[ + "person_support_clone_index" + ].eq(0) + person.loc[person.index[acs_native][0], _OUTPUT] = 1.0 + uc_mismatch = us_weeks_unemployed_signal_gate( + module._replace_person_table(frame, person) + ) + assert not uc_mismatch.passed + assert uc_mismatch.details["uc_constraint_mismatch_count"] == 1 + + def test_signal_gate_rejects_collapsed_puf_share_and_weighted_weeks() -> None: frame = _gate_frame() person = frame.table("person").copy() From 9bbd6dfe02662e3e10e0c66336b13e61c6b87d1d Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Thu, 27 Aug 2026 08:21:58 +0200 Subject: [PATCH 06/49] Centralize stacked source channel scopes --- PROGRESS.md | 24 ++++++++-- .../microcosm/build/us_runtime/__init__.py | 4 ++ .../microcosm/build/us_runtime/puf_support.py | 4 ++ .../build/us_runtime/support_provenance.py | 47 +++++++++++++++++++ .../build/us_runtime/weeks_unemployed.py | 12 +++-- .../tests/test_us_multispine_puf_clone.py | 14 ++++++ 6 files changed, 95 insertions(+), 10 deletions(-) diff --git a/PROGRESS.md b/PROGRESS.md index 21a1e6477..bd0c7d6c4 100644 --- a/PROGRESS.md +++ b/PROGRESS.md @@ -6,7 +6,9 @@ In progress on 2026-08-27. Real-pool provenance has refuted the proposed post-transfer amount-mapping mechanism: every fractional week is an ACS-origin non-native clone prediction outside the calibration's clone-0 recipient scope. The source codec and weeks-gate architecture fixes are implemented and -focused-tested; the broader release-gate audit and repairs are now in progress. +focused-tested. The complete release-call roster is classified; the +unambiguous source-scope and clone-layout repairs are now in progress, while +three archived-model input assumptions require owner rulings. No network access, artifact build, publication, push, pool build, or release build is in scope. @@ -73,13 +75,25 @@ build is in scope. weights. It now reports 234,133 valid ASEC source rows, 108,073 exact native reconciliation rows, 982,686 UC-constrained rows, zero source/UC failures, both unchanged channel bands passing, and only the genuine 369 nonintegers. +- Added a centralized provenance-owner API that distinguishes validated + physical source channels from legacy clone-operator roles. The weeks gate + now consumes that API rather than reading provenance columns directly, and + both repository source-blindness tripwires pass alongside the stacked and + legacy provenance/weekly-signal suites. +- Completed the release-tool call-roster audit. It found unambiguous physical + ASEC scoping repairs in SSI take-up, workers' compensation, alimony, + retirement contributions/distributions, and Medicare; clone-2 layout fixes + in Head Start, voluntary filing, and prior-year income; and a stable-key fix + in WIC. Archived ASEC-only predictor assumptions in SSI disability, SCF + wealth, SCF auto loans, and SIPP vehicles require explicit model-owner + rulings and will be reported rather than guessed. ## Next -- Complete the full release-gate roster audit and implement only unambiguous - stacked-awareness repairs. -- Audit all release-side gates, run the complete prescribed verification, and - write the final evidence and judgment calls to `out.md`. +- Implement and focused-test the unambiguous source-scope, clone-layout, and + stable-key repairs. +- Run the complete prescribed verification and write the final provenance, + audit evidence, and judgment calls to `out.md`. # Historical: gate-failed base-pool release lane diff --git a/packages/microcosm-build/src/microcosm/build/us_runtime/__init__.py b/packages/microcosm-build/src/microcosm/build/us_runtime/__init__.py index e9e756473..ba7adc9d8 100644 --- a/packages/microcosm-build/src/microcosm/build/us_runtime/__init__.py +++ b/packages/microcosm-build/src/microcosm/build/us_runtime/__init__.py @@ -593,6 +593,7 @@ PufTaxDetailChainInputs, clone_us_frame_for_puf_support, finalize_us_puf_tax_detail_predictions, + has_assembled_support_metadata, has_support_role_metadata, impute_us_puf_tax_detail_support, prepare_us_puf_tax_detail_chain_inputs, @@ -602,6 +603,7 @@ support_channel_column, support_clone_index_column, support_role_series, + support_source_channel_series, support_source_id_column, ) from microcosm.build.us_runtime.puma_ladder import ( @@ -1953,12 +1955,14 @@ "validate_spine_agreement_registry", "assemble_spines", "PUF_TAX_DETAIL_CLONE_INDEX", + "has_assembled_support_metadata", "has_support_role_metadata", "puf_tax_detail_clone_mask", "spine_source_id_column", "support_channel_column", "support_clone_index_column", "support_role_series", + "support_source_channel_series", "support_source_id_column", "transfer_puf_capital_gains_tail", "build_puf_e01000_reconciliation_basis", diff --git a/packages/microcosm-build/src/microcosm/build/us_runtime/puf_support.py b/packages/microcosm-build/src/microcosm/build/us_runtime/puf_support.py index c359eaeed..deabb214f 100644 --- a/packages/microcosm-build/src/microcosm/build/us_runtime/puf_support.py +++ b/packages/microcosm-build/src/microcosm/build/us_runtime/puf_support.py @@ -41,12 +41,14 @@ BASE_ASEC_SUPPORT_CHANNEL, PUF_TAX_DETAIL_CLONE_INDEX, PUF_TAX_DETAIL_SUPPORT_CHANNEL, + has_assembled_support_metadata, has_support_role_metadata, puf_tax_detail_clone_mask, spine_source_id_column, support_channel_column, support_clone_index_column, support_role_series, + support_source_channel_series, support_source_id_column, validate_assembly_provenance, ) @@ -73,6 +75,7 @@ "bind_puf_clone_attachment_tail_descendant", "clone_us_frame_for_puf_support", "finalize_us_puf_tax_detail_predictions", + "has_assembled_support_metadata", "has_support_role_metadata", "impute_us_puf_tax_detail_support", "puf_tax_detail_clone_mask", @@ -85,6 +88,7 @@ "support_channel_column", "support_clone_index_column", "support_role_series", + "support_source_channel_series", "support_source_id_column", "validate_puf_clone_attachment", ] diff --git a/packages/microcosm-build/src/microcosm/build/us_runtime/support_provenance.py b/packages/microcosm-build/src/microcosm/build/us_runtime/support_provenance.py index da5bf9e08..ade62a0e9 100644 --- a/packages/microcosm-build/src/microcosm/build/us_runtime/support_provenance.py +++ b/packages/microcosm-build/src/microcosm/build/us_runtime/support_provenance.py @@ -16,6 +16,7 @@ "PUF_TAX_DETAIL_CLONE_INDEX", "PUF_TAX_DETAIL_SUPPORT_CHANNEL", "SPINE_ASSEMBLY_MANIFEST_KEY", + "has_assembled_support_metadata", "has_support_role_metadata", "puf_tax_detail_clone_mask", "spine_assembly_manifest", @@ -25,6 +26,7 @@ "support_channel_column", "support_clone_index_column", "support_role_series", + "support_source_channel_series", "support_source_id_column", "us_reported_coverage_vintage_signal_gate", "validate_assembly_provenance", @@ -326,6 +328,21 @@ def has_support_role_metadata( ) +def has_assembled_support_metadata( + table: pd.DataFrame, + *, + entity: str, +) -> bool: + """Return whether support metadata belongs to a multispine assembly. + + The raw spine-record ID is the assembly discriminator. Historical frames + used the support-channel column for the two operator roles instead, so the + presence of a channel or clone index alone is not sufficient. + """ + + return spine_source_id_column(entity) in table + + def spine_source_id_column(entity: str) -> str: """Return the entity-prefixed raw spine-record ID metadata column.""" @@ -476,6 +493,36 @@ def support_role_series( ) +def support_source_channel_series( + table: pd.DataFrame, + *, + entity: str, +) -> pd.Series: + """Return validated physical source channels with a legacy fallback. + + Assembled frames retain their receipt-declared physical channels (for + example, ``asec`` and ``acs``). Historical frames have no raw spine ID and + use the exact ``asec``/``puf_tax_detail`` operator roles as their channels. + Keeping this resolution in the provenance owner lets reporting and release + gates inspect source coverage without teaching population operators how to + read provenance columns directly. + """ + + roles = support_role_series(table, entity=entity) + if not has_assembled_support_metadata(table, entity=entity): + return pd.Series( + roles.to_numpy(dtype=object, copy=True), + index=table.index, + name=f"{entity}_source_support_channel", + ) + channels = table[support_channel_column(entity)] + return pd.Series( + channels.to_numpy(dtype=object, copy=True), + index=table.index, + name=f"{entity}_source_support_channel", + ) + + def puf_tax_detail_clone_mask( table: pd.DataFrame, *, diff --git a/packages/microcosm-build/src/microcosm/build/us_runtime/weeks_unemployed.py b/packages/microcosm-build/src/microcosm/build/us_runtime/weeks_unemployed.py index 84085120f..e6899d7fb 100644 --- a/packages/microcosm-build/src/microcosm/build/us_runtime/weeks_unemployed.py +++ b/packages/microcosm-build/src/microcosm/build/us_runtime/weeks_unemployed.py @@ -41,11 +41,11 @@ run_source_stage, ) from microcosm.build.us_runtime.support_provenance import ( + has_assembled_support_metadata, has_support_role_metadata, - spine_source_id_column, - support_channel_column, support_clone_index_column, support_role_series, + support_source_channel_series, ) from microcosm.frame import Frame from microcosm.frame.units import US_SCHEMA @@ -1334,7 +1334,7 @@ def _weeks_unemployed_gate_scopes( # Validate complete channel/clone metadata even when an assembled frame's # raw source channels, rather than its compatibility roles, own the gate. roles = support_role_series(person, entity="person") - assembled = spine_source_id_column("person") in person + assembled = has_assembled_support_metadata(person, entity="person") if not assembled: channel = roles.to_numpy(dtype=object, copy=True) roster = tuple( @@ -1351,9 +1351,11 @@ def _weeks_unemployed_gate_scopes( channel == _PUF_CHANNEL, ) - channel_column = support_channel_column("person") clone_column = support_clone_index_column("person") - source_channel = person[channel_column].to_numpy(dtype=object, copy=True) + source_channel = support_source_channel_series( + person, + entity="person", + ).to_numpy(dtype=object, copy=True) observed = {str(value) for value in source_channel} if _ASEC_CHANNEL not in observed: raise ValueError("US weeks-unemployed support has no ASEC source rows.") diff --git a/packages/microcosm-build/tests/test_us_multispine_puf_clone.py b/packages/microcosm-build/tests/test_us_multispine_puf_clone.py index 2e0663c05..ea1828efd 100644 --- a/packages/microcosm-build/tests/test_us_multispine_puf_clone.py +++ b/packages/microcosm-build/tests/test_us_multispine_puf_clone.py @@ -10,6 +10,7 @@ BASE_ASEC_SUPPORT_CHANNEL, PUF_TAX_DETAIL_SUPPORT_CHANNEL, clone_us_frame_for_puf_support, + has_assembled_support_metadata, has_support_role_metadata, prepare_us_puf_tax_detail_chain_inputs, puf_tax_detail_clone_mask, @@ -17,6 +18,7 @@ support_channel_column, support_clone_index_column, support_role_series, + support_source_channel_series, support_source_id_column, ) from microcosm.build.us_runtime.support_provenance import spine_assembly_manifest @@ -111,6 +113,13 @@ def test_puf_clone_preserves_source_spines_and_routes_by_clone_index() -> None: PUF_TAX_DETAIL_SUPPORT_CHANNEL, PUF_TAX_DETAIL_SUPPORT_CHANNEL, ] + assert has_assembled_support_metadata(table, entity=entity) + assert support_source_channel_series(table, entity=entity).tolist() == [ + "asec", + "acs", + "asec", + "acs", + ] assert puf_tax_detail_clone_mask(table, entity=entity).tolist() == [ False, False, @@ -176,6 +185,11 @@ def test_support_role_legacy_fallback_is_closed_to_known_roles() -> None: ) assert has_support_role_metadata(legacy, entity="person") + assert not has_assembled_support_metadata(legacy, entity="person") + assert support_source_channel_series(legacy, entity="person").tolist() == [ + BASE_ASEC_SUPPORT_CHANNEL, + PUF_TAX_DETAIL_SUPPORT_CHANNEL, + ] assert support_role_series(legacy, entity="person").tolist() == [ BASE_ASEC_SUPPORT_CHANNEL, PUF_TAX_DETAIL_SUPPORT_CHANNEL, From 49052e6ed904d4b6be23739dbfb90217e7c28244 Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Thu, 27 Aug 2026 08:26:04 +0200 Subject: [PATCH 07/49] Scope alimony source checks to physical ASEC --- .../src/microcosm/build/us_runtime/alimony.py | 4 +- .../microcosm-build/tests/test_us_alimony.py | 70 +++++++++++++++++++ 2 files changed, 72 insertions(+), 2 deletions(-) diff --git a/packages/microcosm-build/src/microcosm/build/us_runtime/alimony.py b/packages/microcosm-build/src/microcosm/build/us_runtime/alimony.py index 5ab6c1995..756756b9a 100644 --- a/packages/microcosm-build/src/microcosm/build/us_runtime/alimony.py +++ b/packages/microcosm-build/src/microcosm/build/us_runtime/alimony.py @@ -24,7 +24,7 @@ from microcosm.build.us_runtime.support_provenance import ( BASE_ASEC_SUPPORT_CHANNEL, has_support_role_metadata, - support_role_series, + support_source_channel_series, ) from microcosm.frame import Frame @@ -293,7 +293,7 @@ def us_alimony_signal_gate(frame: Frame) -> GateResult: source_mask = np.ones(len(person), dtype=bool) if has_support_role_metadata(person, entity="person"): source_mask = ( - support_role_series(person, entity="person") + support_source_channel_series(person, entity="person") .eq(BASE_ASEC_SUPPORT_CHANNEL) .to_numpy() ) diff --git a/packages/microcosm-build/tests/test_us_alimony.py b/packages/microcosm-build/tests/test_us_alimony.py index d52ab7be9..2026a1b92 100644 --- a/packages/microcosm-build/tests/test_us_alimony.py +++ b/packages/microcosm-build/tests/test_us_alimony.py @@ -374,6 +374,76 @@ def test_signal_gate_rejects_discarded_asec_strike_benefits() -> None: assert any("does not conserve OI_VAL" in failure for failure in result.failures) +def _stacked_alimony_person() -> pd.DataFrame: + source_count = 500 + source_numbers = np.repeat(np.arange(source_count), 2) + clone_indices = np.tile([0, 1], source_count) + source_channels = np.where(source_numbers < 250, "asec", "acs") + asec_source = source_channels == "asec" + + codes = np.full(len(source_numbers), np.nan) + amounts = np.full(len(source_numbers), np.nan) + codes[asec_source] = 0.0 + amounts[asec_source] = 0.0 + reported_alimony = source_numbers == 10 + reported_strike = source_numbers == 20 + codes[reported_alimony] = 20.0 + amounts[reported_alimony] = 2_000.0 + codes[reported_strike] = 12.0 + amounts[reported_strike] = 700.0 + + alimony_income = np.where(reported_alimony, amounts, 0.0) + alimony_expense = np.where(source_numbers == 300, 3_000.0, 0.0) + strike_benefits = np.where(reported_strike, amounts, 0.0) + return pd.DataFrame( + { + "person_spine_source_id": source_numbers, + "person_support_channel": source_channels, + "person_support_clone_index": clone_indices, + "OI_OFF": codes, + "OI_VAL": amounts, + "alimony_income": alimony_income, + "alimony_expense": alimony_expense, + "strike_benefits": strike_benefits, + "miscellaneous_income": np.zeros(len(source_numbers)), + } + ) + + +def test_signal_gate_reconciles_physical_asec_rows_in_stacked_pool() -> None: + person = _stacked_alimony_person() + + result = us_alimony_signal_gate(_PersonFrame(person)) # type: ignore[arg-type] + + assert result.passed, result.failures + + asec_clone = ( + person["person_support_channel"].eq("asec") + & person["person_support_clone_index"].eq(1) + ) + person.loc[asec_clone.idxmax(), "miscellaneous_income"] = 1.0 + mismatch = us_alimony_signal_gate( # type: ignore[arg-type] + _PersonFrame(person) + ) + assert not mismatch.passed + assert any("does not conserve OI_VAL" in failure for failure in mismatch.failures) + + +def test_signal_gate_preserves_legacy_asec_puf_source_scope() -> None: + person = _stacked_alimony_person().drop(columns=["person_spine_source_id"]) + person["person_support_channel"] = np.where( + person["person_support_clone_index"].eq(0), + "asec", + "puf_tax_detail", + ) + puf = person["person_support_channel"].eq("puf_tax_detail") + person.loc[puf, ["OI_OFF", "OI_VAL"]] = np.nan + + result = us_alimony_signal_gate(_PersonFrame(person)) # type: ignore[arg-type] + + assert result.passed, result.failures + + @pytest.mark.parametrize( "person", [ From 17ed9a0be74b00789f6086b1065d07dcf241142e Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Thu, 27 Aug 2026 08:26:16 +0200 Subject: [PATCH 08/49] Key WIC draws by stacked person identity --- PROGRESS.md | 5 +++ .../microcosm/build/us_runtime/wic_claim.py | 21 ++++++++++ .../tests/test_us_wic_claim.py | 38 +++++++++++++++++++ 3 files changed, 64 insertions(+) diff --git a/PROGRESS.md b/PROGRESS.md index bd0c7d6c4..6c723892d 100644 --- a/PROGRESS.md +++ b/PROGRESS.md @@ -80,6 +80,11 @@ build is in scope. now consumes that API rather than reading provenance columns directly, and both repository source-blindness tripwires pass alongside the stacked and legacy provenance/weekly-signal suites. +- Made WIC's deterministic draw key prefer the assembly-unique + `person_source_id` on multispine frames, before the source-local raw identity + triple. Distinct ASEC/ACS records can no longer collide when their vintages + align, while all support clones of one assembled person retain the same draw + and the legacy key order remains unchanged. - Completed the release-tool call-roster audit. It found unambiguous physical ASEC scoping repairs in SSI take-up, workers' compensation, alimony, retirement contributions/distributions, and Medicare; clone-2 layout fixes diff --git a/packages/microcosm-build/src/microcosm/build/us_runtime/wic_claim.py b/packages/microcosm-build/src/microcosm/build/us_runtime/wic_claim.py index 48accbcef..f6434a741 100644 --- a/packages/microcosm-build/src/microcosm/build/us_runtime/wic_claim.py +++ b/packages/microcosm-build/src/microcosm/build/us_runtime/wic_claim.py @@ -46,6 +46,7 @@ run_source_stage, ) from microcosm.build.us_runtime.support_provenance import ( + has_assembled_support_metadata, has_support_role_metadata, support_role_series, ) @@ -108,6 +109,7 @@ # protocol attests this historical salt separately from the output column. _DRAW_SALT = "would_claim_wic" _PERSON_WEIGHT_COLUMN = "person_weight" +_PERSON_SOURCE_ID_COLUMN = "person_source_id" _PERSON_SUPPORT_SOURCE_ID_COLUMN = "person_support_source_id" _SOURCE_IDENTITY_COLUMNS = ( "source_year", @@ -344,6 +346,25 @@ def _wic_categories(person: pd.DataFrame) -> np.ndarray: def _stable_person_keys(person: pd.DataFrame) -> pd.Series: + # Multispine assembly makes this ID unique across physical source + # channels while retaining it across every support clone. Prefer it before + # the historical source-local triple, whose namespaces can overlap once + # ASEC and ACS vintages align. + assembled = has_assembled_support_metadata(person, entity="person") + if assembled and _PERSON_SOURCE_ID_COLUMN not in person.columns: + raise SourceRuntimeError( + "US WIC claim multispine identity requires person_source_id." + ) + if assembled: + source_id = person[_PERSON_SOURCE_ID_COLUMN] + if source_id.isna().any(): + rows = np.flatnonzero(source_id.isna().to_numpy())[:5].tolist() + raise SourceRuntimeError( + "US WIC claim assembly source identity contains missing values at " + f"row(s): {rows}." + ) + return "source:" + source_id.astype(str) + present = [column in person.columns for column in _SOURCE_IDENTITY_COLUMNS] if any(present) and not all(present): missing = [ diff --git a/packages/microcosm-build/tests/test_us_wic_claim.py b/packages/microcosm-build/tests/test_us_wic_claim.py index 1375153fe..3b307e063 100644 --- a/packages/microcosm-build/tests/test_us_wic_claim.py +++ b/packages/microcosm-build/tests/test_us_wic_claim.py @@ -47,6 +47,7 @@ RESTORED_REFERENCE_ECPS_REQUIRED_INPUTS, ) from microcosm.build.us_runtime.source_runtime import us_source_operation_handlers +from microcosm.build.us_runtime.wic_claim import _stable_person_keys from microcosm.frame import US_SCHEMA, EntitySchema, Frame, WeightKind, Weights policyengine_us_installed = importlib.util.find_spec("policyengine_us") is not None @@ -333,6 +334,43 @@ def test_draws_are_reproducible_and_keyed_by_source_identity(self) -> None: _derive(many, seed=2)[_OUTPUT].to_numpy(), ) + def test_multispine_identity_precedes_source_local_identity(self) -> None: + person = _frame( + [ + { + "person_source_id": 10, + "person_spine_source_id": 1, + "source_year": 2024, + "source_household_id": 50, + "source_person_id": 7, + }, + { + "person_source_id": 20, + "person_spine_source_id": 2, + "source_year": 2024, + "source_household_id": 50, + "source_person_id": 7, + }, + { + "person_source_id": 10, + "person_spine_source_id": 1, + "source_year": 2024, + "source_household_id": 50, + "source_person_id": 7, + }, + ] + ).table("person") + + assert _stable_person_keys(person).tolist() == [ + "source:10", + "source:20", + "source:10", + ] + + person.loc[person.index[0], "person_source_id"] = np.nan + with pytest.raises(SourceRuntimeError, match="assembly source identity"): + _stable_person_keys(person) + @pytest.mark.parametrize("column", US_WIC_CLAIM_REQUIRED_SOURCE_COLUMNS) def test_missing_source_columns_fail_closed(self, column: str) -> None: person = _frame([{}]).table("person").drop(columns=[column]) From c7dd53ad91ef4252a6460cd222569774f9589ef9 Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Thu, 27 Aug 2026 08:27:18 +0200 Subject: [PATCH 09/49] Support stacked Head Start clone layouts --- .../build/us_runtime/sipp_head_start.py | 68 +++++++++++++------ .../tests/test_us_sipp_head_start.py | 36 ++++++++++ 2 files changed, 85 insertions(+), 19 deletions(-) diff --git a/packages/microcosm-build/src/microcosm/build/us_runtime/sipp_head_start.py b/packages/microcosm-build/src/microcosm/build/us_runtime/sipp_head_start.py index e7c61f01a..5d45b4764 100644 --- a/packages/microcosm-build/src/microcosm/build/us_runtime/sipp_head_start.py +++ b/packages/microcosm-build/src/microcosm/build/us_runtime/sipp_head_start.py @@ -35,7 +35,9 @@ from microcosm.build.gates import GateResult from microcosm.build.source_manifest import SourceStageSpec, load_source_manifest from microcosm.build.us_runtime.support_provenance import ( + has_assembled_support_metadata, has_support_role_metadata, + support_clone_index_column, support_role_series, ) from microcosm.build.us_runtime.voluntary_filing import ( @@ -481,7 +483,7 @@ def _support_group_keys( person: pd.DataFrame, source_id: pd.Series, ) -> tuple[pd.Series, pd.Series]: - """Return clone-pair keys without consulting source-spine channels.""" + """Return source-person keys and validated legacy-compatible roles.""" if not has_support_role_metadata(person, entity="person"): return ( @@ -571,30 +573,58 @@ def _recipient_predictors(frame: Frame) -> tuple[pd.DataFrame, pd.Series, np.nda "US SIPP Head Start source clones disagree on age for " f"person_source_id(s): {inconsistent[:5]}." ) - role_rows = pd.DataFrame({"source_id": source_id, "role": roles}) - duplicate_roles = role_rows.duplicated( - ["source_id", "role"], - keep=False, - ) - if duplicate_roles.any(): - bad = ( - role_rows.loc[duplicate_roles, ["source_id", "role"]] - .drop_duplicates() - .itertuples(index=False, name=None) - ) - raise ValueError( - "US SIPP Head Start source units carry duplicated same-role rows; " - f"invalid source role(s): {list(bad)[:5]}." - ) - order = pd.DataFrame(index=person.index) order["source_id"] = source_id order["source_key"] = source_key - order["role_priority"] = roles.map({_ASEC_CHANNEL: 0, _PUF_CHANNEL: 1}) + if has_assembled_support_metadata(person, entity="person"): + clone_column = support_clone_index_column("person") + clone_indices = pd.to_numeric(person[clone_column], errors="raise").astype( + "int64" + ) + clone_rows = pd.DataFrame( + {"source_id": source_id, "clone_index": clone_indices} + ) + duplicate_clones = clone_rows.duplicated( + ["source_id", "clone_index"], + keep=False, + ) + if duplicate_clones.any(): + bad = ( + clone_rows.loc[ + duplicate_clones, + ["source_id", "clone_index"], + ] + .drop_duplicates() + .itertuples(index=False, name=None) + ) + raise ValueError( + "US SIPP Head Start assembled source units carry duplicated " + f"clone-index rows; invalid source clone(s): {list(bad)[:5]}." + ) + order["clone_priority"] = clone_indices + priority_column = "clone_priority" + else: + role_rows = pd.DataFrame({"source_id": source_id, "role": roles}) + duplicate_roles = role_rows.duplicated( + ["source_id", "role"], + keep=False, + ) + if duplicate_roles.any(): + bad = ( + role_rows.loc[duplicate_roles, ["source_id", "role"]] + .drop_duplicates() + .itertuples(index=False, name=None) + ) + raise ValueError( + "US SIPP Head Start source units carry duplicated same-role " + f"rows; invalid source role(s): {list(bad)[:5]}." + ) + order["role_priority"] = roles.map({_ASEC_CHANNEL: 0, _PUF_CHANNEL: 1}) + priority_column = "role_priority" order["person_key"] = person["person_id"].astype(str) canonical_index = ( order.sort_values( - ["source_id", "role_priority", "person_key"], + ["source_id", priority_column, "person_key"], kind="mergesort", ) .drop_duplicates("source_key", keep="first") diff --git a/packages/microcosm-build/tests/test_us_sipp_head_start.py b/packages/microcosm-build/tests/test_us_sipp_head_start.py index 010a1d2f3..52f13aa02 100644 --- a/packages/microcosm-build/tests/test_us_sipp_head_start.py +++ b/packages/microcosm-build/tests/test_us_sipp_head_start.py @@ -429,6 +429,42 @@ def test_imputer_rejects_identical_duplicate_same_role_source_rows( impute_us_sipp_head_start(duplicate, _donor(), seed=0) +def test_assembled_clone_two_uses_lowest_clone_and_fans_to_every_clone( + monkeypatch: pytest.MonkeyPatch, +) -> None: + monkeypatch.setattr(module, "QRF", _FakeQRF) + frame = _frame( + [10, 10, 10, 20, 20], + ages=[4, 4, 4, 5, 5], + female=[True, False, False, False, True], + channels=["acs"] * 5, + ) + person = frame.table("person").copy() + person["person_spine_source_id"] = [100, 100, 100, 200, 200] + person["person_support_clone_index"] = [0, 1, 2, 1, 2] + assembled = _replace_person(frame, person) + + predicted = impute_us_sipp_head_start(assembled, _donor(), seed=7) + by_source = pd.DataFrame( + { + "source": person["person_source_id"], + "value": predicted, + } + ).groupby("source")["value"] + assert (by_source.nunique() == 1).all() + assert by_source.first().to_dict() == {10: True, 20: False} + + materialized = person.copy() + materialized[_OUTPUT] = predicted.to_numpy() + summary = us_sipp_head_start_summary(_replace_person(frame, materialized)) + assert summary["clone_group_count"] == 2 + assert summary["clone_mismatch_count"] == 0 + + materialized.loc[materialized["person_support_clone_index"].eq(2), _OUTPUT] ^= True + mismatch = us_sipp_head_start_summary(_replace_person(frame, materialized)) + assert mismatch["clone_mismatch_count"] == 2 + + def test_wrapper_heals_stale_output_and_is_exactly_idempotent( monkeypatch: pytest.MonkeyPatch, ) -> None: From 3f9346887a936a9b35169d9079551e54af771df6 Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Thu, 27 Aug 2026 08:27:20 +0200 Subject: [PATCH 10/49] Scope Medicare source gate to stacked ASEC --- .../build/us_runtime/medicare_take_up.py | 45 ++++++++++++++++-- .../tests/test_us_medicare_take_up.py | 47 +++++++++++++++++++ 2 files changed, 87 insertions(+), 5 deletions(-) diff --git a/packages/microcosm-build/src/microcosm/build/us_runtime/medicare_take_up.py b/packages/microcosm-build/src/microcosm/build/us_runtime/medicare_take_up.py index d93ae5066..fb7167733 100644 --- a/packages/microcosm-build/src/microcosm/build/us_runtime/medicare_take_up.py +++ b/packages/microcosm-build/src/microcosm/build/us_runtime/medicare_take_up.py @@ -27,8 +27,11 @@ run_source_stage, ) from microcosm.build.us_runtime.support_provenance import ( + BASE_ASEC_SUPPORT_CHANNEL, + has_assembled_support_metadata, has_support_role_metadata, support_role_series, + support_source_channel_series, ) from microcosm.frame import Frame from microcosm.frame.units import US_SCHEMA @@ -128,6 +131,20 @@ def _source_codes(person: pd.DataFrame, source: str) -> np.ndarray: return numeric.astype(np.int8) +def _asec_source_mask(person: pd.DataFrame) -> np.ndarray: + """Select physical ASEC rows without changing legacy source semantics.""" + + if not has_assembled_support_metadata(person, entity="person"): + return np.ones(len(person), dtype=bool) + source_channels = support_source_channel_series(person, entity="person") + mask = source_channels.eq(BASE_ASEC_SUPPORT_CHANNEL).to_numpy() + if not mask.any(): + raise SourceRuntimeError( + "US Medicare take-up support has no physical ASEC source rows." + ) + return mask + + def derive_us_medicare_take_up_from_manifest( frame: pd.DataFrame | None, operation: SourceOperationSpec, @@ -151,7 +168,16 @@ def derive_us_medicare_take_up_from_manifest( f"expected {_EXPECTED_PARAMETERS}, got {parameters}." ) result = frame.copy(deep=True) - result[_OUTPUT] = _source_codes(result, _SOURCE) == _ENROLLED_CODE + source_mask = _asec_source_mask(result) + source_values = ( + _source_codes(result.loc[source_mask], _SOURCE) == _ENROLLED_CODE + ) + if source_mask.all(): + result[_OUTPUT] = source_values + else: + if _OUTPUT not in result: + result[_OUTPUT] = pd.Series(pd.NA, index=result.index, dtype="boolean") + result.loc[source_mask, _OUTPUT] = source_values return result @@ -162,10 +188,15 @@ def _surface_matches_source(frame: Frame) -> bool: if _SOURCE not in person: return True try: - expected = _source_codes(person, _SOURCE) == _ENROLLED_CODE + source_mask = _asec_source_mask(person) + expected = ( + _source_codes(person.loc[source_mask], _SOURCE) == _ENROLLED_CODE + ) except SourceRuntimeError: return False - observed = person[_OUTPUT].fillna(False).astype(bool).to_numpy() + observed = ( + person.loc[source_mask, _OUTPUT].fillna(False).astype(bool).to_numpy() + ) return bool(np.array_equal(observed, expected)) @@ -228,10 +259,14 @@ def us_medicare_take_up_summary(frame: Frame) -> dict[str, object]: "missing_count": int(person[_OUTPUT].isna().sum()), } if _SOURCE in person: - source_values = _source_codes(person, _SOURCE) == _ENROLLED_CODE + source_mask = _asec_source_mask(person) + source_values = ( + _source_codes(person.loc[source_mask], _SOURCE) == _ENROLLED_CODE + ) summary["source_mismatch_count"] = int( - np.count_nonzero(values != source_values) + np.count_nonzero(values[source_mask] != source_values) ) + summary["source_rows"] = int(np.count_nonzero(source_mask)) if has_support_role_metadata(person, entity="person"): channel_shares: dict[str, float] = {} channels = support_role_series(person, entity="person").to_numpy() diff --git a/packages/microcosm-build/tests/test_us_medicare_take_up.py b/packages/microcosm-build/tests/test_us_medicare_take_up.py index 81182315b..929a1d77f 100644 --- a/packages/microcosm-build/tests/test_us_medicare_take_up.py +++ b/packages/microcosm-build/tests/test_us_medicare_take_up.py @@ -109,6 +109,21 @@ def _operation() -> SourceOperationSpec: ) +def _stacked_frame() -> Frame: + derived = with_us_medicare_take_up_input(_frame(), seed=0, time_period=2024) + stacked = clone_us_frame_for_puf_support(derived) + person = stacked.table("person") + source_record = np.tile(np.arange(5, dtype=np.int64), 2) + person["person_spine_source_id"] = source_record + person["person_support_channel"] = np.where( + source_record < 3, + "asec", + "acs", + ) + person.loc[person["person_support_channel"].eq("acs"), _SOURCE] = np.nan + return stacked + + def _derive(person: pd.DataFrame) -> pd.DataFrame: return derive_us_medicare_take_up_from_manifest(person, _operation(), None) @@ -261,6 +276,38 @@ def test_support_cloning_preserves_measured_values_on_both_channels(self) -> Non } assert us_medicare_take_up_signal_gate(cloned).passed + def test_stacked_gate_reconciles_only_physical_asec_source_rows(self) -> None: + stacked = _stacked_frame() + + gate = us_medicare_take_up_signal_gate(stacked) + + assert gate.passed, gate.failures + assert gate.details["source_rows"] == 6 + assert gate.details["source_mismatch_count"] == 0 + assert gate.details["channel_weighted_enrolled_shares"] == { + "asec": pytest.approx(0.2), + PUF_TAX_DETAIL_SUPPORT_CHANNEL: pytest.approx(0.2), + } + + person = stacked.table("person") + asec = person["person_support_channel"].eq("asec") + person.loc[person.index[asec][0], _SOURCE] = np.nan + failed = us_medicare_take_up_signal_gate(stacked) + assert not failed.passed + assert any("MCARE" in failure for failure in failed.failures) + + def test_stacked_derivation_preserves_non_source_outputs(self) -> None: + person = _stacked_frame().table("person").copy() + acs = person["person_support_channel"].eq("acs") + person.loc[acs, _OUTPUT] = np.asarray([True, False, True, False]) + before = person.loc[acs, _OUTPUT].copy() + + result = _derive(person) + + pd.testing.assert_series_equal(result.loc[acs, _OUTPUT], before) + expected = person.loc[~acs, _SOURCE].to_numpy() == 1 + np.testing.assert_array_equal(result.loc[~acs, _OUTPUT], expected) + def test_gate_rejects_missing_constant_bad_share_and_mismatch(self) -> None: missing = _frame() assert not us_medicare_take_up_signal_gate(missing).passed From 5852f4ba72f614487b8d7074feb104b4fd17897d Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Thu, 27 Aug 2026 08:29:16 +0200 Subject: [PATCH 11/49] Scope retirement contribution source diagnostics --- .../us_runtime/retirement_contributions.py | 86 ++++++++++++++----- .../tests/test_us_retirement_contributions.py | 64 ++++++++++++++ 2 files changed, 129 insertions(+), 21 deletions(-) diff --git a/packages/microcosm-build/src/microcosm/build/us_runtime/retirement_contributions.py b/packages/microcosm-build/src/microcosm/build/us_runtime/retirement_contributions.py index 753110d3e..29725c12d 100644 --- a/packages/microcosm-build/src/microcosm/build/us_runtime/retirement_contributions.py +++ b/packages/microcosm-build/src/microcosm/build/us_runtime/retirement_contributions.py @@ -38,8 +38,10 @@ run_source_stage, ) from microcosm.build.us_runtime.support_provenance import ( + has_assembled_support_metadata, has_support_role_metadata, support_role_series, + support_source_channel_series, ) from microcosm.frame import Frame from microcosm.frame.units import US_SCHEMA @@ -167,6 +169,30 @@ def _numeric_source(frame: pd.DataFrame, column: str) -> np.ndarray: return values +def _asec_source_mask(frame: pd.DataFrame) -> np.ndarray: + """Select physical ASEC rows while retaining the legacy all-row source.""" + + if not has_assembled_support_metadata(frame, entity="person"): + return np.ones(len(frame), dtype=bool) + source_channels = support_source_channel_series(frame, entity="person") + mask = source_channels.eq(_BASE_ASEC_SUPPORT_CHANNEL).to_numpy() + if not mask.any(): + raise SourceRuntimeError( + "US retirement-contribution support has no physical ASEC source rows." + ) + return mask + + +def _source_reconciliation_mask(frame: pd.DataFrame) -> np.ndarray: + """Select direct ASEC operator rows whose allocation remains source-exact.""" + + source_mask = _asec_source_mask(frame) + if not has_assembled_support_metadata(frame, entity="person"): + return source_mask + roles = support_role_series(frame, entity="person").to_numpy() + return source_mask & (roles == _BASE_ASEC_SUPPORT_CHANNEL) + + def derive_us_retirement_contributions_from_manifest( frame: pd.DataFrame | None, operation: SourceOperationSpec, @@ -197,15 +223,17 @@ def derive_us_retirement_contributions_from_manifest( shares = _share_parameters(operation) result = frame.copy(deep=True) - retirement_contributions = _numeric_source(result, "RETCB_VAL") + source_mask = _asec_source_mask(result) + source = result.loc[source_mask] + retirement_contributions = _numeric_source(source, "RETCB_VAL") negative_source = int(np.count_nonzero(retirement_contributions < 0)) if negative_source: raise SourceRuntimeError( "US retirement-contribution source 'RETCB_VAL' contains " f"{negative_source} negative value(s)." ) - has_wages = _numeric_source(result, "WSAL_VAL") > 0 - has_self_employment = _numeric_source(result, "SEMP_VAL") > 0 + has_wages = _numeric_source(source, "WSAL_VAL") > 0 + has_self_employment = _numeric_source(source, "SEMP_VAL") > 0 has_earned_income = has_wages | has_self_employment self_employed = np.where( @@ -221,17 +249,24 @@ def derive_us_retirement_contributions_from_manifest( ) ira_pool = np.where(has_earned_income, remaining - dc_pool, 0.0) - result["traditional_401k_contributions_desired"] = dc_pool * ( - 1.0 - shares["roth_dc_share"] - ) - result["roth_401k_contributions_desired"] = dc_pool * shares["roth_dc_share"] - result["traditional_ira_contributions_desired"] = ( - ira_pool * shares["traditional_ira_share"] - ) - result["roth_ira_contributions_desired"] = ira_pool * ( - 1.0 - shares["traditional_ira_share"] - ) - result["self_employed_pension_contributions_desired"] = self_employed + derived = { + "traditional_401k_contributions_desired": dc_pool + * (1.0 - shares["roth_dc_share"]), + "roth_401k_contributions_desired": dc_pool * shares["roth_dc_share"], + "traditional_ira_contributions_desired": ira_pool + * shares["traditional_ira_share"], + "roth_ira_contributions_desired": ira_pool + * (1.0 - shares["traditional_ira_share"]), + "self_employed_pension_contributions_desired": self_employed, + } + if source_mask.all(): + for column, values in derived.items(): + result[column] = values + else: + for column, values in derived.items(): + if column not in result: + result[column] = np.nan + result.loc[source_mask, column] = values return result @@ -569,11 +604,16 @@ def us_retirement_contributions_summary(frame: Frame) -> dict[str, object]: ) for column in US_RETIREMENT_CONTRIBUTION_OUTPUT_COLUMNS } - source = ( - _numeric_source(person, "RETCB_VAL") - if "RETCB_VAL" in person - else np.zeros(len(person), dtype=np.float64) - ) + source = np.full(len(person), np.nan, dtype=np.float64) + source_mask = np.zeros(len(person), dtype=bool) + reconciliation_mask = np.zeros(len(person), dtype=bool) + if "RETCB_VAL" in person: + source_mask = _asec_source_mask(person) + source[source_mask] = _numeric_source( + person.loc[source_mask], + "RETCB_VAL", + ) + reconciliation_mask = _source_reconciliation_mask(person) combined = np.sum(np.column_stack(tuple(contributions.values())), axis=1) def _share(values: np.ndarray) -> float: @@ -590,7 +630,7 @@ def _share(values: np.ndarray) -> float: column: int(np.count_nonzero(values < 0)) for column, values in contributions.items() } - source_positive = source > 0 + source_positive = reconciliation_mask & (source > 0) allocation_mismatch = source_positive & ~np.isclose( combined, source, @@ -608,9 +648,13 @@ def _share(values: np.ndarray) -> float: "nonzero_share_band": list(_NONZERO_SHARE_BAND), "nonfinite": nonfinite, "negative": negative, + "source_rows": int(np.count_nonzero(source_mask)), + "source_reconciliation_rows": int(np.count_nonzero(reconciliation_mask)), "source_positive_rows": int(np.count_nonzero(source_positive)), "allocation_mismatch_rows": int(np.count_nonzero(allocation_mismatch)), - "source_total": float(np.sum(source * weights)), + "source_total": float( + np.sum(np.where(reconciliation_mask, source, 0.0) * weights) + ), "allocated_total": float(np.sum(np.nan_to_num(combined) * weights)), } diff --git a/packages/microcosm-build/tests/test_us_retirement_contributions.py b/packages/microcosm-build/tests/test_us_retirement_contributions.py index db8c97d5a..1167c70a8 100644 --- a/packages/microcosm-build/tests/test_us_retirement_contributions.py +++ b/packages/microcosm-build/tests/test_us_retirement_contributions.py @@ -14,6 +14,7 @@ derive_us_retirement_contributions_from_manifest, us_retirement_contributions_signal_gate, us_retirement_contributions_stage_spec, + us_retirement_contributions_summary, with_us_retirement_contribution_inputs, ) from microcosm.frame import US_SCHEMA, Frame, WeightKind, Weights @@ -87,6 +88,26 @@ def _derive(frame: pd.DataFrame) -> pd.DataFrame: return derive_us_retirement_contributions_from_manifest(frame, operation, None) +def _stacked_frame() -> Frame: + direct = with_us_retirement_contribution_inputs( + _frame(), + seed=0, + time_period=2024, + ) + stacked = clone_us_frame_for_puf_support(direct) + person = stacked.table("person") + source_record = np.tile(np.arange(4, dtype=np.int64), 2) + person["person_spine_source_id"] = source_record + person["person_support_channel"] = np.where( + source_record < 2, + "asec", + "acs", + ) + acs = person["person_support_channel"].eq("acs") + person.loc[acs, ["RETCB_VAL", "WSAL_VAL", "SEMP_VAL"]] = np.nan + return stacked + + def test_stage_manifest_pins_sources_operations_and_five_desired_leaves() -> None: spec = us_retirement_contributions_stage_spec() @@ -160,6 +181,49 @@ def test_with_inputs_materializes_signal_and_preserves_reported_total() -> None: assert allocated.iloc[3] == 0.0 +def test_stacked_gate_validates_physical_source_and_reconciles_direct_role() -> None: + stacked = _stacked_frame() + + gate = us_retirement_contributions_signal_gate(stacked) + + assert gate.passed, gate.failures + assert gate.details["source_rows"] == 4 + assert gate.details["source_reconciliation_rows"] == 2 + assert gate.details["allocation_mismatch_rows"] == 0 + + person = stacked.table("person") + asec_puf_role = person["person_support_channel"].eq("asec") & person[ + "person_support_clone_index" + ].eq(1) + person.loc[person.index[asec_puf_role][0], "RETCB_VAL"] = np.nan + with pytest.raises(SourceRuntimeError, match="RETCB_VAL"): + us_retirement_contributions_summary(stacked) + + +def test_stacked_derivation_preserves_non_source_recipient_outputs() -> None: + person = _stacked_frame().table("person").copy() + acs = person["person_support_channel"].eq("acs") + sentinels = np.arange( + np.count_nonzero(acs) * len(US_RETIREMENT_CONTRIBUTION_OUTPUT_COLUMNS), + dtype=float, + ).reshape(np.count_nonzero(acs), -1) + person.loc[acs, list(US_RETIREMENT_CONTRIBUTION_OUTPUT_COLUMNS)] = sentinels + before = person.loc[ + acs, + list(US_RETIREMENT_CONTRIBUTION_OUTPUT_COLUMNS), + ].copy() + + result = _derive(person) + + pd.testing.assert_frame_equal( + result.loc[acs, list(US_RETIREMENT_CONTRIBUTION_OUTPUT_COLUMNS)], + before, + ) + assert np.isfinite( + result.loc[~acs, list(US_RETIREMENT_CONTRIBUTION_OUTPUT_COLUMNS)] + ).all(axis=None) + + def test_puf_half_uses_qrf_predictions_and_applies_income_constraints( monkeypatch: pytest.MonkeyPatch, ) -> None: From e40d5db72613731a9312024fbc0064b9d1a210e6 Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Thu, 27 Aug 2026 08:29:21 +0200 Subject: [PATCH 12/49] Support stacked voluntary filing clones --- .../build/us_runtime/voluntary_filing.py | 125 +++++++++++------- .../tests/test_us_voluntary_filing.py | 49 +++++++ 2 files changed, 126 insertions(+), 48 deletions(-) diff --git a/packages/microcosm-build/src/microcosm/build/us_runtime/voluntary_filing.py b/packages/microcosm-build/src/microcosm/build/us_runtime/voluntary_filing.py index e814cb476..b19d32dda 100644 --- a/packages/microcosm-build/src/microcosm/build/us_runtime/voluntary_filing.py +++ b/packages/microcosm-build/src/microcosm/build/us_runtime/voluntary_filing.py @@ -38,7 +38,9 @@ from microcosm.build.gates import GateResult from microcosm.build.source_manifest import SourceStageSpec, load_source_manifest from microcosm.build.us_runtime.support_provenance import ( + has_assembled_support_metadata, has_support_role_metadata, + support_clone_index_column, support_role_series, ) from microcosm.frame import Frame @@ -742,53 +744,77 @@ def _source_receiver_rows( rows["_support_role"] = support_role_series( tax_unit, entity="tax_unit" ).to_numpy() - role_counts = rows.groupby( - ["_source_id", "_support_role"], - sort=False, - ).size() - duplicated_roles = role_counts[role_counts > 1] - if not duplicated_roles.empty: - bad = duplicated_roles.index.tolist() - raise ValueError( - "US voluntary-filing support source units carry duplicated " - f"same-role rows; invalid source role(s) {bad[:5]}." - ) rows["_source_key"] = rows["_source_id"] - asec_counts = ( - rows["_support_role"] - .eq(_BASE_ASEC_SUPPORT_CHANNEL) - .groupby(rows["_source_key"]) - .sum() - ) - if asec_counts.gt(1).any(): - bad = asec_counts.index[asec_counts.gt(1)].tolist() - raise ValueError( - "US voluntary-filing support source units carry duplicated " - f"ASEC rows; invalid source unit(s) {bad[:5]}." - ) - # Prefer each unit's ASEC row, but a frozen-support selection may - # legitimately keep only a unit's PUF clone (the L0-survivor case the - # SSI reporter lineage also handles — Build M's certified 57,240 - # selection does exactly this). Clones carry the unit's source - # predictors, so the surviving row predicts identically; pick it - # deterministically by channel then tax-unit id. ordered_rows = rows.copy() - ordered_rows["_asec_rank"] = ( - ~ordered_rows["_support_role"].eq(_BASE_ASEC_SUPPORT_CHANNEL) - ).astype(int) - source_rows = ( - ordered_rows.sort_values( - [ - "_source_id", - "_asec_rank", - "_support_role", - "_tax_unit_id", - ], - kind="stable", + if has_assembled_support_metadata(tax_unit, entity="tax_unit"): + clone_column = support_clone_index_column("tax_unit") + ordered_rows["_clone_index"] = pd.to_numeric( + tax_unit[clone_column], errors="raise" + ).to_numpy(dtype=np.int64) + clone_counts = ordered_rows.groupby( + ["_source_id", "_clone_index"], + sort=False, + ).size() + duplicated_clones = clone_counts[clone_counts > 1] + if not duplicated_clones.empty: + bad = duplicated_clones.index.tolist() + raise ValueError( + "US voluntary-filing assembled source units carry " + "duplicated clone-index rows; invalid source clone(s) " + f"{bad[:5]}." + ) + source_rows = ( + ordered_rows.sort_values( + ["_source_id", "_clone_index", "_tax_unit_id"], + kind="stable", + ) + .drop_duplicates("_source_key", keep="first") + .drop(columns="_clone_index") + ) + else: + role_counts = rows.groupby( + ["_source_id", "_support_role"], + sort=False, + ).size() + duplicated_roles = role_counts[role_counts > 1] + if not duplicated_roles.empty: + bad = duplicated_roles.index.tolist() + raise ValueError( + "US voluntary-filing support source units carry duplicated " + f"same-role rows; invalid source role(s) {bad[:5]}." + ) + asec_counts = ( + rows["_support_role"] + .eq(_BASE_ASEC_SUPPORT_CHANNEL) + .groupby(rows["_source_key"]) + .sum() + ) + if asec_counts.gt(1).any(): + bad = asec_counts.index[asec_counts.gt(1)].tolist() + raise ValueError( + "US voluntary-filing support source units carry duplicated " + f"ASEC rows; invalid source unit(s) {bad[:5]}." + ) + # Prefer each unit's ASEC row, but a frozen-support selection may + # legitimately keep only a unit's PUF clone. Clones carry the + # unit's source predictors, so the surviving row predicts + # identically; pick it deterministically by role then tax-unit id. + ordered_rows["_asec_rank"] = ( + ~ordered_rows["_support_role"].eq(_BASE_ASEC_SUPPORT_CHANNEL) + ).astype(int) + source_rows = ( + ordered_rows.sort_values( + [ + "_source_id", + "_asec_rank", + "_support_role", + "_tax_unit_id", + ], + kind="stable", + ) + .drop_duplicates("_source_key", keep="first") + .drop(columns="_asec_rank") ) - .drop_duplicates("_source_key", keep="first") - .drop(columns="_asec_rank") - ) else: if rows["_source_id"].duplicated().any(): duplicates = rows.loc[ @@ -1015,10 +1041,13 @@ def us_voluntary_filing_summary(frame: Frame) -> dict[str, object]: "value": values, } ) - clone_table["source_occurrence"] = clone_table.groupby( - ["source_id", "role"], sort=False - ).cumcount() - clone_groups = ["source_id", "source_occurrence"] + if has_assembled_support_metadata(tax_unit, entity="tax_unit"): + clone_groups = ["source_id"] + else: + clone_table["source_occurrence"] = clone_table.groupby( + ["source_id", "role"], sort=False + ).cumcount() + clone_groups = ["source_id", "source_occurrence"] sizes = clone_table.groupby(clone_groups, sort=False).size() clone_source_units = int((sizes > 1).sum()) unique = clone_table.groupby(clone_groups, sort=False)["value"].nunique( diff --git a/packages/microcosm-build/tests/test_us_voluntary_filing.py b/packages/microcosm-build/tests/test_us_voluntary_filing.py index b3460662b..304f4d63d 100644 --- a/packages/microcosm-build/tests/test_us_voluntary_filing.py +++ b/packages/microcosm-build/tests/test_us_voluntary_filing.py @@ -652,6 +652,55 @@ def test_duplicate_same_role_source_rows_fail_closed() -> None: impute_us_voluntary_filing(expanded, _donor(), seed=17) +def test_assembled_clone_two_uses_explicit_index_and_checks_every_clone() -> None: + tax_unit = pd.DataFrame( + { + "tax_unit_id": [100, 101, 102, 200, 201], + "tax_unit_source_id": [10, 10, 10, 20, 20], + "tax_unit_spine_source_id": [1, 1, 1, 2, 2], + "tax_unit_support_channel": ["acs"] * 5, + "tax_unit_support_clone_index": [0, 1, 2, 1, 2], + } + ) + receiver = pd.DataFrame( + { + predictor: np.arange(5, dtype=np.float64) + offset + for offset, predictor in enumerate( + SIPP_VOLUNTARY_FILING_MODEL_PREDICTORS + ) + }, + index=tax_unit["tax_unit_id"], + ) + + class TaxUnitFrame: + def table(self, entity: str) -> pd.DataFrame: + assert entity == "tax_unit" + return tax_unit + + prediction_rows, fan_keys = module._source_receiver_rows( + TaxUnitFrame(), receiver + ) + assert prediction_rows.index.tolist() == ["10", "20"] + assert fan_keys.tolist() == ["10", "10", "10", "20", "20"] + # Source 10 prefers clone 0; source 20 has no native survivor and picks + # the lowest surviving clone index, clone 1. + assert prediction_rows.iloc[:, 0].tolist() == [0.0, 3.0] + + summary_frame = _replace_tax_unit( + _frame(3), + **{ + _OUTPUT: np.asarray([False, False, True]), + "tax_unit_source_id": np.asarray([7, 7, 7]), + "tax_unit_spine_source_id": np.asarray([70, 70, 70]), + "tax_unit_support_channel": np.asarray(["acs", "acs", "acs"]), + "tax_unit_support_clone_index": np.asarray([0, 1, 2]), + }, + ) + summary = us_voluntary_filing_summary(summary_frame) + assert summary["clone_source_units"] == 1 + assert summary["clone_mismatch_source_units"] == 1 + + def test_real_qrf_recomputation_is_deterministic() -> None: frame = _frame(14) donor = _donor(120) From 148881c6319e18a140b77c3948ff10c44a284d8f Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Thu, 27 Aug 2026 08:30:48 +0200 Subject: [PATCH 13/49] Check all stacked prior-year income clones --- .../build/us_runtime/prior_year_income.py | 5 ++- .../tests/test_us_prior_year_income.py | 36 +++++++++++++++++++ 2 files changed, 40 insertions(+), 1 deletion(-) diff --git a/packages/microcosm-build/src/microcosm/build/us_runtime/prior_year_income.py b/packages/microcosm-build/src/microcosm/build/us_runtime/prior_year_income.py index 254181ca5..b03340266 100644 --- a/packages/microcosm-build/src/microcosm/build/us_runtime/prior_year_income.py +++ b/packages/microcosm-build/src/microcosm/build/us_runtime/prior_year_income.py @@ -39,6 +39,7 @@ run_source_stage, ) from microcosm.build.us_runtime.support_provenance import ( + has_assembled_support_metadata, has_support_role_metadata, support_role_series, without_support_role_metadata, @@ -750,7 +751,9 @@ def _share(mask: np.ndarray) -> float: } ) group_columns = ["_source_id"] - if has_support_role_metadata(person, entity="person"): + if has_support_role_metadata( + person, entity="person" + ) and not has_assembled_support_metadata(person, entity="person"): clone_work["_role"] = support_role_series( person, entity="person" ).to_numpy() diff --git a/packages/microcosm-build/tests/test_us_prior_year_income.py b/packages/microcosm-build/tests/test_us_prior_year_income.py index eb20c37eb..c4fdbcd3a 100644 --- a/packages/microcosm-build/tests/test_us_prior_year_income.py +++ b/packages/microcosm-build/tests/test_us_prior_year_income.py @@ -410,6 +410,42 @@ def test_signal_gate_accepts_signed_source_signal_and_rejects_defaults() -> None assert "availability" in " ".join(failing.failures) +def test_clone_availability_checks_all_assembled_clones_and_legacy_pairs() -> None: + assembled = _frame( + pd.DataFrame( + { + "person_source_id": [10, 10, 10, 20, 20, 20], + "person_spine_source_id": [1, 1, 1, 2, 2, 2], + "person_support_channel": ["acs"] * 6, + "person_support_clone_index": [0, 1, 2, 0, 1, 2], + "self_employment_income_last_year": [10, 10, 10, -5, -5, -5], + "previous_year_income_available": [True, True, False] + [False] * 3, + } + ) + ) + assembled_summary = module.us_prior_year_income_summary(assembled) + assert assembled_summary["clone_availability_mismatches"] == 1 + assembled_gate = us_prior_year_income_signal_gate(assembled) + assert any("1 source person" in failure for failure in assembled_gate.failures) + + legacy = _frame( + pd.DataFrame( + { + "person_source_id": [10, 10, 20, 20], + "person_support_channel": [ + BASE_ASEC_SUPPORT_CHANNEL, + PUF_TAX_DETAIL_SUPPORT_CHANNEL, + ] + * 2, + "self_employment_income_last_year": [10, 10, -5, -5], + "previous_year_income_available": [True, True, False, False], + } + ) + ) + legacy_summary = module.us_prior_year_income_summary(legacy) + assert legacy_summary["clone_availability_mismatches"] == 0 + + def test_source_reconciliation_detects_plausible_but_wrong_asec_carry() -> None: derived = with_us_prior_year_income_inputs( _source_frame(), seed=0, time_period=2024 From 52b27f78a42e1f0f519c5e44ffc39dbb6a1a8e7d Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Thu, 27 Aug 2026 08:31:06 +0200 Subject: [PATCH 14/49] Separate alimony source validation and carry scopes --- .../src/microcosm/build/us_runtime/alimony.py | 42 +++++++++++++++---- .../microcosm-build/tests/test_us_alimony.py | 22 +++++++++- 2 files changed, 56 insertions(+), 8 deletions(-) diff --git a/packages/microcosm-build/src/microcosm/build/us_runtime/alimony.py b/packages/microcosm-build/src/microcosm/build/us_runtime/alimony.py index 756756b9a..49c724d9f 100644 --- a/packages/microcosm-build/src/microcosm/build/us_runtime/alimony.py +++ b/packages/microcosm-build/src/microcosm/build/us_runtime/alimony.py @@ -23,7 +23,9 @@ from microcosm.build.source_manifest import SourceStageSpec, load_source_manifest from microcosm.build.us_runtime.support_provenance import ( BASE_ASEC_SUPPORT_CHANNEL, + has_assembled_support_metadata, has_support_role_metadata, + support_role_series, support_source_channel_series, ) from microcosm.frame import Frame @@ -291,18 +293,41 @@ def us_alimony_signal_gate(frame: Frame) -> GateResult: details=summary, ) source_mask = np.ones(len(person), dtype=bool) + source_reconciliation_mask = source_mask.copy() if has_support_role_metadata(person, entity="person"): source_mask = ( support_source_channel_series(person, entity="person") .eq(BASE_ASEC_SUPPORT_CHANNEL) .to_numpy() ) - amounts = pd.to_numeric(person["OI_VAL"], errors="coerce").to_numpy( + source_reconciliation_mask = source_mask.copy() + if has_assembled_support_metadata(person, entity="person"): + source_reconciliation_mask &= ( + support_role_series(person, entity="person") + .eq(BASE_ASEC_SUPPORT_CHANNEL) + .to_numpy() + ) + raw_amounts = pd.to_numeric(person["OI_VAL"], errors="coerce").to_numpy( dtype=np.float64 - )[source_mask] - codes = pd.to_numeric(person["OI_OFF"], errors="coerce").to_numpy( + ) + raw_codes = pd.to_numeric(person["OI_OFF"], errors="coerce").to_numpy( dtype=np.float64 - )[source_mask] + ) + source_invalid = source_mask & ( + ~np.isfinite(raw_amounts) | ~np.isfinite(raw_codes) + ) + summary["asec_source_rows"] = int(np.count_nonzero(source_mask)) + summary["asec_source_invalid"] = int(np.count_nonzero(source_invalid)) + summary["asec_source_reconciliation_rows"] = int( + np.count_nonzero(source_reconciliation_mask) + ) + if bool(source_invalid.any()): + failures.append( + "ASEC OI_VAL/OI_OFF contain nonfinite source values on " + f"{int(np.count_nonzero(source_invalid))} row(s)." + ) + amounts = raw_amounts[source_reconciliation_mask] + codes = raw_codes[source_reconciliation_mask] if bool(np.isfinite(amounts).all() and np.isfinite(codes).all()): integer_codes = codes.astype(np.int64) expected_alimony = np.where( @@ -321,13 +346,16 @@ def us_alimony_signal_gate(frame: Frame) -> GateResult: amounts, ) actual_alimony = pd.to_numeric( - person.loc[source_mask, "alimony_income"], errors="coerce" + person.loc[source_reconciliation_mask, "alimony_income"], + errors="coerce", ).to_numpy(dtype=np.float64) actual_strike_benefits = pd.to_numeric( - person.loc[source_mask, "strike_benefits"], errors="coerce" + person.loc[source_reconciliation_mask, "strike_benefits"], + errors="coerce", ).to_numpy(dtype=np.float64) actual_miscellaneous = pd.to_numeric( - person.loc[source_mask, "miscellaneous_income"], errors="coerce" + person.loc[source_reconciliation_mask, "miscellaneous_income"], + errors="coerce", ).to_numpy(dtype=np.float64) alimony_mismatch = ~np.isclose(actual_alimony, expected_alimony) strike_benefits_mismatch = ~np.isclose( diff --git a/packages/microcosm-build/tests/test_us_alimony.py b/packages/microcosm-build/tests/test_us_alimony.py index 2026a1b92..25aefd15c 100644 --- a/packages/microcosm-build/tests/test_us_alimony.py +++ b/packages/microcosm-build/tests/test_us_alimony.py @@ -422,12 +422,28 @@ def test_signal_gate_reconciles_physical_asec_rows_in_stacked_pool() -> None: & person["person_support_clone_index"].eq(1) ) person.loc[asec_clone.idxmax(), "miscellaneous_income"] = 1.0 - mismatch = us_alimony_signal_gate( # type: ignore[arg-type] + transferred_clone = us_alimony_signal_gate( # type: ignore[arg-type] _PersonFrame(person) ) + assert transferred_clone.passed, transferred_clone.failures + + asec_native = ( + person["person_support_channel"].eq("asec") + & person["person_support_clone_index"].eq(0) + ) + person.loc[asec_native.idxmax(), "miscellaneous_income"] = 1.0 + mismatch = us_alimony_signal_gate(_PersonFrame(person)) # type: ignore[arg-type] assert not mismatch.passed assert any("does not conserve OI_VAL" in failure for failure in mismatch.failures) + person.loc[asec_native.idxmax(), "miscellaneous_income"] = 0.0 + person.loc[asec_clone.idxmax(), "OI_VAL"] = np.nan + invalid_source = us_alimony_signal_gate( # type: ignore[arg-type] + _PersonFrame(person) + ) + assert not invalid_source.passed + assert invalid_source.details["asec_source_invalid"] == 1 + def test_signal_gate_preserves_legacy_asec_puf_source_scope() -> None: person = _stacked_alimony_person().drop(columns=["person_spine_source_id"]) @@ -436,6 +452,10 @@ def test_signal_gate_preserves_legacy_asec_puf_source_scope() -> None: "asec", "puf_tax_detail", ) + asec = person["person_support_channel"].eq("asec") + person.loc[asec, ["OI_OFF", "OI_VAL"]] = person.loc[ + asec, ["OI_OFF", "OI_VAL"] + ].fillna(0.0) puf = person["person_support_channel"].eq("puf_tax_detail") person.loc[puf, ["OI_OFF", "OI_VAL"]] = np.nan From 058f27b5319c8f3c1ffaaeaf39b47c5e662b4931 Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Thu, 27 Aug 2026 08:31:44 +0200 Subject: [PATCH 15/49] Record stacked clone-layout repairs --- PROGRESS.md | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/PROGRESS.md b/PROGRESS.md index 6c723892d..a4f9ca499 100644 --- a/PROGRESS.md +++ b/PROGRESS.md @@ -92,6 +92,15 @@ build is in scope. in WIC. Archived ASEC-only predictor assumptions in SSI disability, SCF wealth, SCF auto loans, and SIPP vehicles require explicit model-owner rulings and will be reported rather than guessed. +- Reworked the assembled Head Start and voluntary-filing receiver layouts to + key by assembly-unique source ID plus explicit clone index, accept clone 2 + and later, reject duplicate source/clone rows, choose clone 0 (or the lowest + surviving clone) deterministically, and fan one source-level decision to all + clones. Their legacy role-only duplicate contract remains unchanged. +- Replaced occurrence-pair clone diagnostics in voluntary filing and + prior-year income with all-clone grouping on assembled source IDs, so a + clone-2-only divergence is now detected. Focused tests and Ruff passed for + all three module/test pairs. ## Next From 97de22e7785e6168090d1d1d70f010b6a9c3a444 Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Thu, 27 Aug 2026 08:32:12 +0200 Subject: [PATCH 16/49] Scope workers compensation sources in stacked pools --- .../build/us_runtime/workers_compensation.py | 17 ++++- .../tests/test_us_workers_compensation.py | 71 +++++++++++++++++++ 2 files changed, 86 insertions(+), 2 deletions(-) diff --git a/packages/microcosm-build/src/microcosm/build/us_runtime/workers_compensation.py b/packages/microcosm-build/src/microcosm/build/us_runtime/workers_compensation.py index b571a1226..d531d009a 100644 --- a/packages/microcosm-build/src/microcosm/build/us_runtime/workers_compensation.py +++ b/packages/microcosm-build/src/microcosm/build/us_runtime/workers_compensation.py @@ -28,8 +28,10 @@ run_source_stage, ) from microcosm.build.us_runtime.support_provenance import ( + has_assembled_support_metadata, has_support_role_metadata, support_role_series, + support_source_channel_series, ) from microcosm.frame import Frame from microcosm.frame.units import US_SCHEMA @@ -570,16 +572,27 @@ def us_workers_compensation_summary(frame: Frame) -> dict[str, object]: dtype=np.float64 ) source_mask = np.ones(len(person), dtype=bool) + source_reconciliation_mask = source_mask.copy() if has_support_role_metadata(person, entity="person"): source_mask = ( - support_role_series(person, entity="person").to_numpy() + support_source_channel_series(person, entity="person").to_numpy() == _BASE_ASEC_SUPPORT_CHANNEL ) + source_reconciliation_mask = source_mask.copy() + if has_assembled_support_metadata(person, entity="person"): + source_reconciliation_mask &= ( + support_role_series(person, entity="person").to_numpy() + == _BASE_ASEC_SUPPORT_CHANNEL + ) source_valid = np.isfinite(source) & (source >= 0.0) + summary["source_rows"] = int(np.count_nonzero(source_mask)) + summary["source_reconciliation_rows"] = int( + np.count_nonzero(source_reconciliation_mask) + ) summary["source_invalid"] = int(np.count_nonzero(source_mask & ~source_valid)) summary["source_mismatch_count"] = int( np.count_nonzero( - source_mask + source_reconciliation_mask & source_valid & finite & ~np.isclose(values, source, rtol=0.0, atol=0.0) diff --git a/packages/microcosm-build/tests/test_us_workers_compensation.py b/packages/microcosm-build/tests/test_us_workers_compensation.py index 1e0fdabe9..786e28614 100644 --- a/packages/microcosm-build/tests/test_us_workers_compensation.py +++ b/packages/microcosm-build/tests/test_us_workers_compensation.py @@ -532,6 +532,77 @@ def test_signal_gate_rejects_either_dead_support_channel(dead_channel: str) -> N assert any(dead_channel in failure for failure in gate.failures) +def _stacked_workers_compensation_frame() -> Frame: + direct = with_us_workers_compensation(_frame(), seed=0, time_period=2024) + expanded = clone_us_frame_for_puf_support(direct) + tables = {entity: expanded.table(entity).copy() for entity in expanded.entities} + person = tables["person"] + physical_asec = person["person_source_id"].le(50) + person["person_spine_source_id"] = person["person_source_id"] + person["person_support_channel"] = np.where(physical_asec, "asec", "acs") + person.loc[~physical_asec, "WC_VAL"] = np.nan + return Frame( + tables, + expanded.schema, + { + entity: expanded.weights_for(entity) + for entity in expanded.weighted_entities + }, + expanded.strata, + mass_log=expanded.mass_log, + metadata=expanded.metadata, + ) + + +def test_signal_gate_reconciles_physical_asec_rows_in_stacked_pool() -> None: + frame = _stacked_workers_compensation_frame() + + gate = us_workers_compensation_signal_gate(frame) + + assert gate.passed, gate.failures + assert gate.details["source_invalid"] == 0 + assert gate.details["source_mismatch_count"] == 0 + + person = frame.table("person") + asec_clone = person["person_support_channel"].eq("asec") & person[ + "person_support_clone_index" + ].eq(1) + person.loc[asec_clone.idxmax(), _OUTPUT] = 1.0 + transferred_clone = us_workers_compensation_signal_gate(frame) + assert transferred_clone.passed, transferred_clone.failures + assert transferred_clone.details["source_mismatch_count"] == 0 + + asec_native = person["person_support_channel"].eq("asec") & person[ + "person_support_clone_index" + ].eq(0) + person.loc[asec_native.idxmax(), _OUTPUT] = 1.0 + mismatch = us_workers_compensation_signal_gate(frame) + assert not mismatch.passed + assert mismatch.details["source_mismatch_count"] == 1 + + person.loc[asec_native.idxmax(), _OUTPUT] = person.loc[ + asec_native.idxmax(), "WC_VAL" + ] + person.loc[asec_clone.idxmax(), "WC_VAL"] = np.nan + invalid_source = us_workers_compensation_signal_gate(frame) + assert not invalid_source.passed + assert invalid_source.details["source_invalid"] == 1 + + +def test_signal_gate_preserves_legacy_asec_puf_source_scope() -> None: + direct = with_us_workers_compensation(_frame(), seed=0, time_period=2024) + legacy = clone_us_frame_for_puf_support(direct) + person = legacy.table("person") + puf = person["person_support_channel"].eq("puf_tax_detail") + person.loc[puf, "WC_VAL"] = np.nan + + gate = us_workers_compensation_signal_gate(legacy) + + assert gate.passed, gate.failures + assert gate.details["source_invalid"] == 0 + assert gate.details["source_mismatch_count"] == 0 + + @requires_us def test_policyengine_us_1_819_0_contract_and_positive_annual_behavior() -> None: from policyengine_us import CountryTaxBenefitSystem, Simulation From 74e961875a6f41395369098159c19da2c169fc55 Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Thu, 27 Aug 2026 08:32:15 +0200 Subject: [PATCH 17/49] Scope retirement distribution source gates --- .../us_runtime/retirement_distributions.py | 61 +++++++++++++++--- .../tests/test_us_retirement_distributions.py | 62 +++++++++++++++++++ 2 files changed, 114 insertions(+), 9 deletions(-) diff --git a/packages/microcosm-build/src/microcosm/build/us_runtime/retirement_distributions.py b/packages/microcosm-build/src/microcosm/build/us_runtime/retirement_distributions.py index 7f8d1427c..e0c5e3455 100644 --- a/packages/microcosm-build/src/microcosm/build/us_runtime/retirement_distributions.py +++ b/packages/microcosm-build/src/microcosm/build/us_runtime/retirement_distributions.py @@ -43,8 +43,10 @@ run_source_stage, ) from microcosm.build.us_runtime.support_provenance import ( + has_assembled_support_metadata, has_support_role_metadata, support_role_series, + support_source_channel_series, ) from microcosm.frame import Frame from microcosm.frame.units import US_SCHEMA @@ -269,6 +271,30 @@ def _derived_outputs( return outputs +def _asec_source_mask(frame: pd.DataFrame) -> np.ndarray: + """Select physical ASEC rows while retaining the legacy all-row source.""" + + if not has_assembled_support_metadata(frame, entity="person"): + return np.ones(len(frame), dtype=bool) + source_channels = support_source_channel_series(frame, entity="person") + mask = source_channels.eq(_BASE_ASEC_SUPPORT_CHANNEL).to_numpy() + if not mask.any(): + raise SourceRuntimeError( + "US retirement-distribution support has no physical ASEC source rows." + ) + return mask + + +def _source_reconciliation_mask(frame: pd.DataFrame) -> np.ndarray: + """Select direct operator rows whose outputs remain measured-source exact.""" + + compare = _asec_source_mask(frame).copy() + if has_support_role_metadata(frame, entity="person"): + roles = support_role_series(frame, entity="person").to_numpy() + compare &= roles == _BASE_ASEC_SUPPORT_CHANNEL + return compare + + def derive_us_retirement_distributions_from_manifest( frame: pd.DataFrame | None, operation: SourceOperationSpec, @@ -298,7 +324,8 @@ def derive_us_retirement_distributions_from_manifest( ) output_by_code = _manifest_output_by_code(operation) result = frame.copy(deep=True) - derived = _derived_outputs(result, output_by_code) + source_mask = _asec_source_mask(result) + derived = _derived_outputs(result.loc[source_mask], output_by_code) preserved_puf_taxable_ira: np.ndarray | None = None puf_mask: np.ndarray | None = None if has_support_role_metadata(result, entity="person"): @@ -326,7 +353,12 @@ def derive_us_retirement_distributions_from_manifest( ) for output, values in derived.items(): - result[output] = values + if source_mask.all(): + result[output] = values + else: + if output not in result: + result[output] = np.nan + result.loc[source_mask, output] = values if output == "taxable_ira_distributions" and puf_mask is not None: assert preserved_puf_taxable_ira is not None result.loc[puf_mask, output] = preserved_puf_taxable_ira[puf_mask] @@ -674,17 +706,26 @@ def us_retirement_distributions_summary(frame: Frame) -> dict[str, object]: } source_mismatches: dict[str, int] = {} + source_rows = 0 + source_reconciliation_rows = 0 if all( column in person for column in US_RETIREMENT_DISTRIBUTION_REQUIRED_SOURCE_COLUMNS ): - expected = _derived_outputs(person, _EXPECTED_OUTPUT_BY_ACCOUNT_CODE) - compare = np.ones(len(person), dtype=bool) - if has_support_role_metadata(person, entity="person"): - compare = ( - support_role_series(person, entity="person").to_numpy() - == _BASE_ASEC_SUPPORT_CHANNEL - ) + source_mask = _asec_source_mask(person) + source_expected = _derived_outputs( + person.loc[source_mask], + _EXPECTED_OUTPUT_BY_ACCOUNT_CODE, + ) + expected = { + column: np.full(len(person), np.nan, dtype=np.float64) + for column in US_RETIREMENT_DISTRIBUTION_OUTPUT_COLUMNS + } + for column, source_values in source_expected.items(): + expected[column][source_mask] = source_values + compare = _source_reconciliation_mask(person) + source_rows = int(np.count_nonzero(source_mask)) + source_reconciliation_rows = int(np.count_nonzero(compare)) source_mismatches = { column: int( np.count_nonzero( @@ -722,6 +763,8 @@ def _share(array: np.ndarray) -> float: "negative": { column: int(np.count_nonzero(array < 0)) for column, array in values.items() }, + "source_rows": source_rows, + "source_reconciliation_rows": source_reconciliation_rows, "source_mismatches": source_mismatches, } diff --git a/packages/microcosm-build/tests/test_us_retirement_distributions.py b/packages/microcosm-build/tests/test_us_retirement_distributions.py index 5ac1b0495..0382f3dd4 100644 --- a/packages/microcosm-build/tests/test_us_retirement_distributions.py +++ b/packages/microcosm-build/tests/test_us_retirement_distributions.py @@ -28,6 +28,7 @@ derive_us_retirement_distributions_from_manifest, us_retirement_distributions_signal_gate, us_retirement_distributions_stage_spec, + us_retirement_distributions_summary, with_us_retirement_distribution_inputs, ) from microcosm.build.us_runtime.source_runtime import us_source_operation_handlers @@ -125,6 +126,29 @@ def _derive(frame: pd.DataFrame) -> pd.DataFrame: ) +def _stacked_frame() -> Frame: + direct = with_us_retirement_distribution_inputs( + _frame(), + seed=0, + time_period=2024, + ) + stacked = clone_us_frame_for_puf_support(direct) + person = stacked.table("person") + source_record = np.tile(np.arange(8, dtype=np.int64), 2) + person["person_spine_source_id"] = source_record + person["person_support_channel"] = np.where( + source_record < 4, + "asec", + "acs", + ) + acs = person["person_support_channel"].eq("acs") + person.loc[ + acs, + list(US_RETIREMENT_DISTRIBUTION_REQUIRED_SOURCE_COLUMNS), + ] = np.nan + return stacked + + def _sha256(path: Path) -> str: digest = sha256() with path.open("rb") as stream: @@ -274,6 +298,44 @@ def test_frame_integration_gate_and_idempotence() -> None: ) +def test_stacked_gate_validates_physical_source_and_reconciles_direct_role() -> None: + stacked = _stacked_frame() + + gate = us_retirement_distributions_signal_gate(stacked) + + assert gate.passed, gate.failures + assert gate.details["source_rows"] == 8 + assert gate.details["source_reconciliation_rows"] == 4 + assert all(value == 0 for value in gate.details["source_mismatches"].values()) + + person = stacked.table("person") + asec_puf_role = person["person_support_channel"].eq("asec") & person[ + "person_support_clone_index" + ].eq(1) + person.loc[person.index[asec_puf_role][0], "DST_SC1"] = np.nan + with pytest.raises(SourceRuntimeError, match="account codes"): + us_retirement_distributions_summary(stacked) + + +def test_stacked_derivation_preserves_non_source_recipient_outputs() -> None: + person = _stacked_frame().table("person").copy() + acs = person["person_support_channel"].eq("acs") + sentinels = ( + np.arange( + np.count_nonzero(acs) * len(_OUTPUTS), + dtype=float, + ).reshape(np.count_nonzero(acs), -1) + + 1_000.0 + ) + person.loc[acs, list(_OUTPUTS)] = sentinels + before = person.loc[acs, list(_OUTPUTS)].copy() + + result = _derive(person) + + pd.testing.assert_frame_equal(result.loc[acs, list(_OUTPUTS)], before) + assert np.isfinite(result.loc[~acs, list(_OUTPUTS)]).all(axis=None) + + def test_puf_half_uses_qrf_and_asec_half_remains_exact( monkeypatch: pytest.MonkeyPatch, ) -> None: From 41007992b752f426534a99d786e17ab6877b3adf Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Thu, 27 Aug 2026 08:35:03 +0200 Subject: [PATCH 18/49] Anchor SSI reporters to physical ASEC sources --- .../microcosm/build/us_runtime/ssi_take_up.py | 29 ++++++--- .../tests/test_us_ssi_take_up.py | 62 +++++++++++++++++++ 2 files changed, 81 insertions(+), 10 deletions(-) diff --git a/packages/microcosm-build/src/microcosm/build/us_runtime/ssi_take_up.py b/packages/microcosm-build/src/microcosm/build/us_runtime/ssi_take_up.py index 0b8b035cf..b97f4f62c 100644 --- a/packages/microcosm-build/src/microcosm/build/us_runtime/ssi_take_up.py +++ b/packages/microcosm-build/src/microcosm/build/us_runtime/ssi_take_up.py @@ -78,6 +78,7 @@ has_support_role_metadata, support_clone_index_column, support_role_series, + support_source_channel_series, ) from microcosm.frame import Frame from microcosm.frame.units import US_SCHEMA @@ -530,17 +531,20 @@ def us_ssi_take_up_reporter_source_ids(frame: Frame) -> frozenset[str]: if person[_SOURCE_ID].isna().any(): raise ValueError("US SSI take-up reporter lineage requires provenance.") source_ids = _decoded_strings(person[_SOURCE_ID]) - roles = support_role_series(person, entity="person") + source_channels = support_source_channel_series(person, entity="person") + asec_source = source_channels.eq(_ASEC_CHANNEL).to_numpy() reported = pd.to_numeric(person[US_SSI_TAKE_UP_ANCHOR], errors="coerce").to_numpy( dtype=np.float64 ) - if source_ids.str.strip().eq("").any() or not np.isfinite(reported).all(): + if source_ids.str.strip().eq("").any() or not np.isfinite( + reported[asec_source] + ).all(): raise ValueError( "US SSI take-up reporter lineage requires nonblank identities and " - "finite SSI_VAL values." + "finite SSI_VAL values on physical ASEC source rows." ) reporter_ids = frozenset( - source_ids[roles.eq(_ASEC_CHANNEL).to_numpy() & (reported > 0.0)] + source_ids[asec_source & (reported > 0.0)] ) if not reporter_ids: raise ValueError("US SSI take-up found no direct ASEC SSI reporters.") @@ -576,8 +580,6 @@ def _source_table( weights = np.asarray(frame.resolve_weights("person").values, dtype=np.float64) if not np.isfinite(age).all() or (age < 0).any(): raise ValueError("US SSI take-up ages must be finite and nonnegative.") - if not np.isfinite(reported).all(): - raise ValueError("US SSI take-up SSI_VAL anchors must be finite.") if not np.isfinite(potential).all(): raise ValueError("US SSI take-up uncapped_ssi values must be finite.") if not (np.isfinite(weights) & (weights >= 0)).all() or weights.sum() <= 0: @@ -592,6 +594,7 @@ def _source_table( source_ids = _decoded_strings(person[_SOURCE_ID]) channels = support_role_series(person, entity="person") + source_channels = support_source_channel_series(person, entity="person") if source_ids.str.strip().eq("").any(): raise ValueError("US SSI take-up source identities must be nonblank.") observed_channels = set(channels.unique()) @@ -602,7 +605,13 @@ def _source_table( f"unsupported {sorted(observed_channels - _KNOWN_CHANNELS)}." ) - direct_anchor = (reported > 0.0) & channels.eq(_ASEC_CHANNEL).to_numpy() + asec_source = source_channels.eq(_ASEC_CHANNEL).to_numpy() + if not np.isfinite(reported[asec_source]).all(): + raise ValueError( + "US SSI take-up SSI_VAL anchors must be finite on physical ASEC " + "source rows." + ) + direct_anchor = (reported > 0.0) & asec_source if reporter_source_ids is None: anchored_source_ids = frozenset(source_ids[direct_anchor]) else: @@ -627,9 +636,9 @@ def _source_table( "weight": weights, "candidate": potential > 0.0, # Capture lineage on the full support before L0. When pruning keeps - # only a PUF clone, the explicit source-ID set still preserves the - # underlying direct ASEC measurement without promoting PUF-only - # SSI_VAL copies into independent anchors. + # only a donor-role clone, the explicit source-ID set still + # preserves the underlying physical ASEC measurement without + # promoting non-ASEC SSI_VAL values into independent anchors. "anchor": source_ids.isin(anchored_source_ids).to_numpy(), }, index=person.index, diff --git a/packages/microcosm-build/tests/test_us_ssi_take_up.py b/packages/microcosm-build/tests/test_us_ssi_take_up.py index 8ae286b8c..5819e6302 100644 --- a/packages/microcosm-build/tests/test_us_ssi_take_up.py +++ b/packages/microcosm-build/tests/test_us_ssi_take_up.py @@ -150,6 +150,29 @@ def _frame(*, stale_output: bool = False) -> tuple[Frame, np.ndarray]: return frame, np.asarray(potential, dtype=np.float64) +def _stacked_frame() -> tuple[Frame, np.ndarray]: + """Convert the legacy operator fixture to physical ASEC/ACS channels.""" + + frame, potential = _frame() + person = frame.table("person").copy() + source_number = person["person_source_id"].str.rsplit(":").str[-1].astype(int) + legacy_roles = person["person_support_channel"].copy() + person["person_spine_source_id"] = person["person_source_id"] + person["person_support_clone_index"] = np.where( + legacy_roles.eq("asec"), + 0, + 1, + ) + person["person_support_channel"] = np.where( + source_number.le(6), + "asec", + "acs", + ) + acs_source = person["person_support_channel"].eq("acs") + person.loc[acs_source, US_SSI_TAKE_UP_ANCHOR] = np.nan + return _replace_person(frame, person), potential + + def _replace_person(frame: Frame, person: pd.DataFrame) -> Frame: tables = {entity: frame.table(entity).copy() for entity in frame.entities} tables["person"] = person @@ -309,6 +332,45 @@ def test_puf_only_ssi_value_is_not_promoted_to_reporter_anchor() -> None: assert diagnostics["age_bands"] == baseline_diagnostics["age_bands"] +def test_stacked_asec_sources_own_reporters_across_operator_clones() -> None: + frame, potential = _stacked_frame() + + reporter_source_ids = us_ssi_take_up_reporter_source_ids(frame) + expected = frozenset( + f"{band}:{source_number}" + for band in _AGES + for source_number in (0, 6) + ) + assert reporter_source_ids == expected + + result, diagnostics = with_us_ssi_take_up( + frame, + uncapped_ssi=potential, + seed=17, + targets=_TARGETS, + ) + assert diagnostics["reporter_anchor_lost_count"] == 0 + assert result.table("person").loc[ + result.table("person")["person_source_id"].isin(expected), _OUTPUT + ].all() + + person = frame.table("person").copy() + asec_clone = person["person_support_channel"].eq("asec") & person[ + "person_support_clone_index" + ].eq(1) + person.loc[asec_clone.idxmax(), US_SSI_TAKE_UP_ANCHOR] = np.nan + invalid = _replace_person(frame, person) + with pytest.raises(ValueError, match="physical ASEC source rows"): + us_ssi_take_up_reporter_source_ids(invalid) + with pytest.raises(ValueError, match="physical ASEC source rows"): + with_us_ssi_take_up( + invalid, + uncapped_ssi=potential, + seed=17, + targets=_TARGETS, + ) + + def test_reporter_lineage_survives_when_l0_keeps_only_the_puf_clone() -> None: full, potential = _frame() reporter_source_ids = us_ssi_take_up_reporter_source_ids(full) From eb5cd4b66bba470c6274a06d5898266f55f408e7 Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Thu, 27 Aug 2026 08:37:12 +0200 Subject: [PATCH 19/49] Keep SSI assignment source blind --- .../microcosm/build/us_runtime/ssi_take_up.py | 21 ++++++------------- 1 file changed, 6 insertions(+), 15 deletions(-) diff --git a/packages/microcosm-build/src/microcosm/build/us_runtime/ssi_take_up.py b/packages/microcosm-build/src/microcosm/build/us_runtime/ssi_take_up.py index b97f4f62c..47b2b9fd0 100644 --- a/packages/microcosm-build/src/microcosm/build/us_runtime/ssi_take_up.py +++ b/packages/microcosm-build/src/microcosm/build/us_runtime/ssi_take_up.py @@ -594,7 +594,6 @@ def _source_table( source_ids = _decoded_strings(person[_SOURCE_ID]) channels = support_role_series(person, entity="person") - source_channels = support_source_channel_series(person, entity="person") if source_ids.str.strip().eq("").any(): raise ValueError("US SSI take-up source identities must be nonblank.") observed_channels = set(channels.unique()) @@ -605,27 +604,19 @@ def _source_table( f"unsupported {sorted(observed_channels - _KNOWN_CHANNELS)}." ) - asec_source = source_channels.eq(_ASEC_CHANNEL).to_numpy() - if not np.isfinite(reported[asec_source]).all(): - raise ValueError( - "US SSI take-up SSI_VAL anchors must be finite on physical ASEC " - "source rows." - ) - direct_anchor = (reported > 0.0) & asec_source if reporter_source_ids is None: - anchored_source_ids = frozenset(source_ids[direct_anchor]) + anchored_source_ids = us_ssi_take_up_reporter_source_ids(frame) else: anchored_source_ids = frozenset(str(value) for value in reporter_source_ids) if not anchored_source_ids: raise ValueError("US SSI take-up reporter lineage cannot be empty.") if any(not value.strip() for value in anchored_source_ids): raise ValueError("US SSI take-up reporter source identities are nonblank.") - omitted_direct = sorted(set(source_ids[direct_anchor]) - anchored_source_ids) - if omitted_direct: - raise ValueError( - "US SSI take-up reporter lineage omitted direct ASEC anchors; " - f"examples {omitted_direct[:5]}." - ) + anchored_rows = source_ids.isin(anchored_source_ids).to_numpy() + if not np.isfinite(reported[anchored_rows]).all(): + raise ValueError( + "US SSI take-up SSI_VAL anchors must be finite on anchored source rows." + ) rows = pd.DataFrame( { From 3dc5df514a56c9b07a8dad80004a7edd4b0d508a Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Thu, 27 Aug 2026 08:37:37 +0200 Subject: [PATCH 20/49] Keep stacked source repairs gate scoped --- PROGRESS.md | 7 ++++ .../build/us_runtime/medicare_take_up.py | 20 ++-------- .../us_runtime/retirement_contributions.py | 37 +++++++------------ .../us_runtime/retirement_distributions.py | 10 +---- .../tests/test_us_medicare_take_up.py | 12 ------ .../tests/test_us_retirement_contributions.py | 24 ------------ .../tests/test_us_retirement_distributions.py | 19 ---------- 7 files changed, 26 insertions(+), 103 deletions(-) diff --git a/PROGRESS.md b/PROGRESS.md index a4f9ca499..17fce6178 100644 --- a/PROGRESS.md +++ b/PROGRESS.md @@ -101,6 +101,13 @@ build is in scope. prior-year income with all-clone grouping on assembled source IDs, so a clone-2-only divergence is now detected. Focused tests and Ruff passed for all three module/test pairs. +- Scoped Medicare and retirement release diagnostics to physical ASEC source + rows, separating raw-source validity from native direct-carry reconciliation + where transferred clones intentionally differ. Kept the producer kernels + origin-blind: the authenticated-pool release path skips those producers, and + indirect physical-source routing would violate the repository's population- + operator boundary. Their 60 focused module/source-blindness tests and Ruff + pass after that review correction. ## Next diff --git a/packages/microcosm-build/src/microcosm/build/us_runtime/medicare_take_up.py b/packages/microcosm-build/src/microcosm/build/us_runtime/medicare_take_up.py index fb7167733..217193417 100644 --- a/packages/microcosm-build/src/microcosm/build/us_runtime/medicare_take_up.py +++ b/packages/microcosm-build/src/microcosm/build/us_runtime/medicare_take_up.py @@ -168,16 +168,7 @@ def derive_us_medicare_take_up_from_manifest( f"expected {_EXPECTED_PARAMETERS}, got {parameters}." ) result = frame.copy(deep=True) - source_mask = _asec_source_mask(result) - source_values = ( - _source_codes(result.loc[source_mask], _SOURCE) == _ENROLLED_CODE - ) - if source_mask.all(): - result[_OUTPUT] = source_values - else: - if _OUTPUT not in result: - result[_OUTPUT] = pd.Series(pd.NA, index=result.index, dtype="boolean") - result.loc[source_mask, _OUTPUT] = source_values + result[_OUTPUT] = _source_codes(result, _SOURCE) == _ENROLLED_CODE return result @@ -188,15 +179,10 @@ def _surface_matches_source(frame: Frame) -> bool: if _SOURCE not in person: return True try: - source_mask = _asec_source_mask(person) - expected = ( - _source_codes(person.loc[source_mask], _SOURCE) == _ENROLLED_CODE - ) + expected = _source_codes(person, _SOURCE) == _ENROLLED_CODE except SourceRuntimeError: return False - observed = ( - person.loc[source_mask, _OUTPUT].fillna(False).astype(bool).to_numpy() - ) + observed = person[_OUTPUT].fillna(False).astype(bool).to_numpy() return bool(np.array_equal(observed, expected)) diff --git a/packages/microcosm-build/src/microcosm/build/us_runtime/retirement_contributions.py b/packages/microcosm-build/src/microcosm/build/us_runtime/retirement_contributions.py index 29725c12d..fd0f2a14d 100644 --- a/packages/microcosm-build/src/microcosm/build/us_runtime/retirement_contributions.py +++ b/packages/microcosm-build/src/microcosm/build/us_runtime/retirement_contributions.py @@ -223,17 +223,15 @@ def derive_us_retirement_contributions_from_manifest( shares = _share_parameters(operation) result = frame.copy(deep=True) - source_mask = _asec_source_mask(result) - source = result.loc[source_mask] - retirement_contributions = _numeric_source(source, "RETCB_VAL") + retirement_contributions = _numeric_source(result, "RETCB_VAL") negative_source = int(np.count_nonzero(retirement_contributions < 0)) if negative_source: raise SourceRuntimeError( "US retirement-contribution source 'RETCB_VAL' contains " f"{negative_source} negative value(s)." ) - has_wages = _numeric_source(source, "WSAL_VAL") > 0 - has_self_employment = _numeric_source(source, "SEMP_VAL") > 0 + has_wages = _numeric_source(result, "WSAL_VAL") > 0 + has_self_employment = _numeric_source(result, "SEMP_VAL") > 0 has_earned_income = has_wages | has_self_employment self_employed = np.where( @@ -249,24 +247,17 @@ def derive_us_retirement_contributions_from_manifest( ) ira_pool = np.where(has_earned_income, remaining - dc_pool, 0.0) - derived = { - "traditional_401k_contributions_desired": dc_pool - * (1.0 - shares["roth_dc_share"]), - "roth_401k_contributions_desired": dc_pool * shares["roth_dc_share"], - "traditional_ira_contributions_desired": ira_pool - * shares["traditional_ira_share"], - "roth_ira_contributions_desired": ira_pool - * (1.0 - shares["traditional_ira_share"]), - "self_employed_pension_contributions_desired": self_employed, - } - if source_mask.all(): - for column, values in derived.items(): - result[column] = values - else: - for column, values in derived.items(): - if column not in result: - result[column] = np.nan - result.loc[source_mask, column] = values + result["traditional_401k_contributions_desired"] = dc_pool * ( + 1.0 - shares["roth_dc_share"] + ) + result["roth_401k_contributions_desired"] = dc_pool * shares["roth_dc_share"] + result["traditional_ira_contributions_desired"] = ( + ira_pool * shares["traditional_ira_share"] + ) + result["roth_ira_contributions_desired"] = ira_pool * ( + 1.0 - shares["traditional_ira_share"] + ) + result["self_employed_pension_contributions_desired"] = self_employed return result diff --git a/packages/microcosm-build/src/microcosm/build/us_runtime/retirement_distributions.py b/packages/microcosm-build/src/microcosm/build/us_runtime/retirement_distributions.py index e0c5e3455..b98f7df52 100644 --- a/packages/microcosm-build/src/microcosm/build/us_runtime/retirement_distributions.py +++ b/packages/microcosm-build/src/microcosm/build/us_runtime/retirement_distributions.py @@ -324,8 +324,7 @@ def derive_us_retirement_distributions_from_manifest( ) output_by_code = _manifest_output_by_code(operation) result = frame.copy(deep=True) - source_mask = _asec_source_mask(result) - derived = _derived_outputs(result.loc[source_mask], output_by_code) + derived = _derived_outputs(result, output_by_code) preserved_puf_taxable_ira: np.ndarray | None = None puf_mask: np.ndarray | None = None if has_support_role_metadata(result, entity="person"): @@ -353,12 +352,7 @@ def derive_us_retirement_distributions_from_manifest( ) for output, values in derived.items(): - if source_mask.all(): - result[output] = values - else: - if output not in result: - result[output] = np.nan - result.loc[source_mask, output] = values + result[output] = values if output == "taxable_ira_distributions" and puf_mask is not None: assert preserved_puf_taxable_ira is not None result.loc[puf_mask, output] = preserved_puf_taxable_ira[puf_mask] diff --git a/packages/microcosm-build/tests/test_us_medicare_take_up.py b/packages/microcosm-build/tests/test_us_medicare_take_up.py index 929a1d77f..0ab8d2202 100644 --- a/packages/microcosm-build/tests/test_us_medicare_take_up.py +++ b/packages/microcosm-build/tests/test_us_medicare_take_up.py @@ -296,18 +296,6 @@ def test_stacked_gate_reconciles_only_physical_asec_source_rows(self) -> None: assert not failed.passed assert any("MCARE" in failure for failure in failed.failures) - def test_stacked_derivation_preserves_non_source_outputs(self) -> None: - person = _stacked_frame().table("person").copy() - acs = person["person_support_channel"].eq("acs") - person.loc[acs, _OUTPUT] = np.asarray([True, False, True, False]) - before = person.loc[acs, _OUTPUT].copy() - - result = _derive(person) - - pd.testing.assert_series_equal(result.loc[acs, _OUTPUT], before) - expected = person.loc[~acs, _SOURCE].to_numpy() == 1 - np.testing.assert_array_equal(result.loc[~acs, _OUTPUT], expected) - def test_gate_rejects_missing_constant_bad_share_and_mismatch(self) -> None: missing = _frame() assert not us_medicare_take_up_signal_gate(missing).passed diff --git a/packages/microcosm-build/tests/test_us_retirement_contributions.py b/packages/microcosm-build/tests/test_us_retirement_contributions.py index 1167c70a8..c40f22d66 100644 --- a/packages/microcosm-build/tests/test_us_retirement_contributions.py +++ b/packages/microcosm-build/tests/test_us_retirement_contributions.py @@ -200,30 +200,6 @@ def test_stacked_gate_validates_physical_source_and_reconciles_direct_role() -> us_retirement_contributions_summary(stacked) -def test_stacked_derivation_preserves_non_source_recipient_outputs() -> None: - person = _stacked_frame().table("person").copy() - acs = person["person_support_channel"].eq("acs") - sentinels = np.arange( - np.count_nonzero(acs) * len(US_RETIREMENT_CONTRIBUTION_OUTPUT_COLUMNS), - dtype=float, - ).reshape(np.count_nonzero(acs), -1) - person.loc[acs, list(US_RETIREMENT_CONTRIBUTION_OUTPUT_COLUMNS)] = sentinels - before = person.loc[ - acs, - list(US_RETIREMENT_CONTRIBUTION_OUTPUT_COLUMNS), - ].copy() - - result = _derive(person) - - pd.testing.assert_frame_equal( - result.loc[acs, list(US_RETIREMENT_CONTRIBUTION_OUTPUT_COLUMNS)], - before, - ) - assert np.isfinite( - result.loc[~acs, list(US_RETIREMENT_CONTRIBUTION_OUTPUT_COLUMNS)] - ).all(axis=None) - - def test_puf_half_uses_qrf_predictions_and_applies_income_constraints( monkeypatch: pytest.MonkeyPatch, ) -> None: diff --git a/packages/microcosm-build/tests/test_us_retirement_distributions.py b/packages/microcosm-build/tests/test_us_retirement_distributions.py index 0382f3dd4..8921b521e 100644 --- a/packages/microcosm-build/tests/test_us_retirement_distributions.py +++ b/packages/microcosm-build/tests/test_us_retirement_distributions.py @@ -317,25 +317,6 @@ def test_stacked_gate_validates_physical_source_and_reconciles_direct_role() -> us_retirement_distributions_summary(stacked) -def test_stacked_derivation_preserves_non_source_recipient_outputs() -> None: - person = _stacked_frame().table("person").copy() - acs = person["person_support_channel"].eq("acs") - sentinels = ( - np.arange( - np.count_nonzero(acs) * len(_OUTPUTS), - dtype=float, - ).reshape(np.count_nonzero(acs), -1) - + 1_000.0 - ) - person.loc[acs, list(_OUTPUTS)] = sentinels - before = person.loc[acs, list(_OUTPUTS)].copy() - - result = _derive(person) - - pd.testing.assert_frame_equal(result.loc[acs, list(_OUTPUTS)], before) - assert np.isfinite(result.loc[~acs, list(_OUTPUTS)]).all(axis=None) - - def test_puf_half_uses_qrf_and_asec_half_remains_exact( monkeypatch: pytest.MonkeyPatch, ) -> None: From 9b36720bbf8f63286a7fe2872d9a32227fd6a71e Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Thu, 27 Aug 2026 08:38:17 +0200 Subject: [PATCH 21/49] Record stacked raw-source gate repairs --- PROGRESS.md | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/PROGRESS.md b/PROGRESS.md index 17fce6178..7f66bf7f2 100644 --- a/PROGRESS.md +++ b/PROGRESS.md @@ -108,6 +108,15 @@ build is in scope. indirect physical-source routing would violate the repository's population- operator boundary. Their 60 focused module/source-blindness tests and Ruff pass after that review correction. +- Scoped alimony and workers' compensation raw validity to every physical ASEC + clone and exact source-carry checks to physical ASEC native rows, leaving all + clone-operator plausibility bands unchanged. Stacked ASEC+ACS and legacy + ASEC+PUF fixtures pass (28 alimony and 21 workers' compensation tests). +- Changed SSI reporter-lineage capture to validate `SSI_VAL` only on physical + ASEC rows and accept null ACS raw-source cells. Assignment remains source- + blind: it consumes the source-ID set captured before L0, or obtains that set + through the reporter helper when no explicit set is supplied. The complete + SSI take-up file passes (71 tests). ## Next From 9520c03f7530d930294f7b89a1802f5153cfa72b Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Thu, 27 Aug 2026 08:41:50 +0200 Subject: [PATCH 22/49] Report divergence across all SSI disability clones --- PROGRESS.md | 5 +++++ .../build/us_runtime/ssi_disability_criteria.py | 17 +++++++++++------ .../tests/test_us_ssi_disability_criteria.py | 17 +++++++++++++++++ 3 files changed, 33 insertions(+), 6 deletions(-) diff --git a/PROGRESS.md b/PROGRESS.md index 7f66bf7f2..6fbf77736 100644 --- a/PROGRESS.md +++ b/PROGRESS.md @@ -117,6 +117,11 @@ build is in scope. blind: it consumes the source-ID set captured before L0, or obtains that set through the reporter helper when no explicit set is supplied. The complete SSI take-up file passes (71 tests). +- Fixed SSI-disability's non-fatal clone-divergence diagnostic to group every + assembled clone by source person, so clone-2-only divergence is reported. + The existing decision not to make divergence gate-fatal remains unchanged + for an owner ruling; the complete focused file and source-blindness checks + pass. ## Next diff --git a/packages/microcosm-build/src/microcosm/build/us_runtime/ssi_disability_criteria.py b/packages/microcosm-build/src/microcosm/build/us_runtime/ssi_disability_criteria.py index c90d8169b..4a6b56611 100644 --- a/packages/microcosm-build/src/microcosm/build/us_runtime/ssi_disability_criteria.py +++ b/packages/microcosm-build/src/microcosm/build/us_runtime/ssi_disability_criteria.py @@ -42,6 +42,7 @@ from microcosm.build.gates import GateResult from microcosm.build.source_manifest import SourceStageSpec, load_source_manifest from microcosm.build.us_runtime.support_provenance import ( + has_assembled_support_metadata, has_support_role_metadata, support_role_series, ) @@ -1137,12 +1138,16 @@ def us_ssi_disability_criteria_summary(frame: Frame) -> dict[str, object]: "value": values, } ) - clone_table["source_occurrence"] = clone_table.groupby( - ["source_id", "role"], sort=False - ).cumcount() - unique = clone_table.groupby( - ["source_id", "source_occurrence"], sort=False - )["value"].nunique(dropna=False) + if has_assembled_support_metadata(person, entity="person"): + clone_groups = ["source_id"] + else: + clone_table["source_occurrence"] = clone_table.groupby( + ["source_id", "role"], sort=False + ).cumcount() + clone_groups = ["source_id", "source_occurrence"] + unique = clone_table.groupby(clone_groups, sort=False)["value"].nunique( + dropna=False + ) clone_divergence_source_people = int((unique > 1).sum()) reporter_mismatches = 0 diff --git a/packages/microcosm-build/tests/test_us_ssi_disability_criteria.py b/packages/microcosm-build/tests/test_us_ssi_disability_criteria.py index 5c52c49af..adf0e19ca 100644 --- a/packages/microcosm-build/tests/test_us_ssi_disability_criteria.py +++ b/packages/microcosm-build/tests/test_us_ssi_disability_criteria.py @@ -584,6 +584,23 @@ def test_signal_gate_requires_each_channel_but_allows_clone_divergence( ) +def test_stacked_clone_divergence_diagnostic_checks_clone_two() -> None: + stacked = _replace_person( + _frame(3), + **{ + "person_source_id": np.asarray([10, 10, 10]), + "person_spine_source_id": np.asarray([1, 1, 1]), + "person_support_channel": np.asarray(["acs", "acs", "acs"]), + "person_support_clone_index": np.asarray([0, 1, 2]), + _OUTPUT: np.asarray([False, False, True]), + }, + ) + + summary = us_ssi_disability_criteria_summary(stacked) + + assert summary["clone_divergence_source_people"] == 1 + + def test_gate_requires_complete_support_provenance( monkeypatch: pytest.MonkeyPatch, ) -> None: From b8e04e9c8b1ca61af46484e1f501a787af03f5d0 Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Thu, 27 Aug 2026 08:49:57 +0200 Subject: [PATCH 23/49] Confine physical source access to release gates --- PROGRESS.md | 5 + .../microcosm/build/us_runtime/__init__.py | 2 - .../src/microcosm/build/us_runtime/alimony.py | 4 +- .../build/us_runtime/medicare_take_up.py | 4 +- .../microcosm/build/us_runtime/puf_support.py | 2 - .../us_runtime/retirement_contributions.py | 4 +- .../us_runtime/retirement_distributions.py | 4 +- .../microcosm/build/us_runtime/ssi_take_up.py | 4 +- .../build/us_runtime/support_provenance.py | 13 +- .../build/us_runtime/weeks_unemployed.py | 4 +- .../build/us_runtime/workers_compensation.py | 6 +- .../tests/test_us_multispine_puf_clone.py | 10 +- .../tests/test_us_spine_blindness.py | 112 ++++++++++++++++++ 13 files changed, 146 insertions(+), 28 deletions(-) diff --git a/PROGRESS.md b/PROGRESS.md index 6fbf77736..78067381e 100644 --- a/PROGRESS.md +++ b/PROGRESS.md @@ -122,6 +122,11 @@ build is in scope. The existing decision not to make divergence gate-fatal remains unchanged for an owner ruling; the complete focused file and source-blindness checks pass. +- Narrowed physical-channel resolution to a gate/reporter-only provenance API, + removed its general runtime/PUF-support re-exports, and added a static exact- + caller contract (including internal mask-helper callers). A future derive, + impute, or wrapper use now fails the source-blindness suite instead of + passing through indirection. ## Next diff --git a/packages/microcosm-build/src/microcosm/build/us_runtime/__init__.py b/packages/microcosm-build/src/microcosm/build/us_runtime/__init__.py index ba7adc9d8..518baacb6 100644 --- a/packages/microcosm-build/src/microcosm/build/us_runtime/__init__.py +++ b/packages/microcosm-build/src/microcosm/build/us_runtime/__init__.py @@ -603,7 +603,6 @@ support_channel_column, support_clone_index_column, support_role_series, - support_source_channel_series, support_source_id_column, ) from microcosm.build.us_runtime.puma_ladder import ( @@ -1962,7 +1961,6 @@ "support_channel_column", "support_clone_index_column", "support_role_series", - "support_source_channel_series", "support_source_id_column", "transfer_puf_capital_gains_tail", "build_puf_e01000_reconciliation_basis", diff --git a/packages/microcosm-build/src/microcosm/build/us_runtime/alimony.py b/packages/microcosm-build/src/microcosm/build/us_runtime/alimony.py index 49c724d9f..8ae7fe834 100644 --- a/packages/microcosm-build/src/microcosm/build/us_runtime/alimony.py +++ b/packages/microcosm-build/src/microcosm/build/us_runtime/alimony.py @@ -25,8 +25,8 @@ BASE_ASEC_SUPPORT_CHANNEL, has_assembled_support_metadata, has_support_role_metadata, + support_gate_source_channel_series, support_role_series, - support_source_channel_series, ) from microcosm.frame import Frame @@ -296,7 +296,7 @@ def us_alimony_signal_gate(frame: Frame) -> GateResult: source_reconciliation_mask = source_mask.copy() if has_support_role_metadata(person, entity="person"): source_mask = ( - support_source_channel_series(person, entity="person") + support_gate_source_channel_series(person, entity="person") .eq(BASE_ASEC_SUPPORT_CHANNEL) .to_numpy() ) diff --git a/packages/microcosm-build/src/microcosm/build/us_runtime/medicare_take_up.py b/packages/microcosm-build/src/microcosm/build/us_runtime/medicare_take_up.py index 217193417..ba40e479b 100644 --- a/packages/microcosm-build/src/microcosm/build/us_runtime/medicare_take_up.py +++ b/packages/microcosm-build/src/microcosm/build/us_runtime/medicare_take_up.py @@ -30,8 +30,8 @@ BASE_ASEC_SUPPORT_CHANNEL, has_assembled_support_metadata, has_support_role_metadata, + support_gate_source_channel_series, support_role_series, - support_source_channel_series, ) from microcosm.frame import Frame from microcosm.frame.units import US_SCHEMA @@ -136,7 +136,7 @@ def _asec_source_mask(person: pd.DataFrame) -> np.ndarray: if not has_assembled_support_metadata(person, entity="person"): return np.ones(len(person), dtype=bool) - source_channels = support_source_channel_series(person, entity="person") + source_channels = support_gate_source_channel_series(person, entity="person") mask = source_channels.eq(BASE_ASEC_SUPPORT_CHANNEL).to_numpy() if not mask.any(): raise SourceRuntimeError( diff --git a/packages/microcosm-build/src/microcosm/build/us_runtime/puf_support.py b/packages/microcosm-build/src/microcosm/build/us_runtime/puf_support.py index deabb214f..d78c70a23 100644 --- a/packages/microcosm-build/src/microcosm/build/us_runtime/puf_support.py +++ b/packages/microcosm-build/src/microcosm/build/us_runtime/puf_support.py @@ -48,7 +48,6 @@ support_channel_column, support_clone_index_column, support_role_series, - support_source_channel_series, support_source_id_column, validate_assembly_provenance, ) @@ -88,7 +87,6 @@ "support_channel_column", "support_clone_index_column", "support_role_series", - "support_source_channel_series", "support_source_id_column", "validate_puf_clone_attachment", ] diff --git a/packages/microcosm-build/src/microcosm/build/us_runtime/retirement_contributions.py b/packages/microcosm-build/src/microcosm/build/us_runtime/retirement_contributions.py index fd0f2a14d..3265139ab 100644 --- a/packages/microcosm-build/src/microcosm/build/us_runtime/retirement_contributions.py +++ b/packages/microcosm-build/src/microcosm/build/us_runtime/retirement_contributions.py @@ -40,8 +40,8 @@ from microcosm.build.us_runtime.support_provenance import ( has_assembled_support_metadata, has_support_role_metadata, + support_gate_source_channel_series, support_role_series, - support_source_channel_series, ) from microcosm.frame import Frame from microcosm.frame.units import US_SCHEMA @@ -174,7 +174,7 @@ def _asec_source_mask(frame: pd.DataFrame) -> np.ndarray: if not has_assembled_support_metadata(frame, entity="person"): return np.ones(len(frame), dtype=bool) - source_channels = support_source_channel_series(frame, entity="person") + source_channels = support_gate_source_channel_series(frame, entity="person") mask = source_channels.eq(_BASE_ASEC_SUPPORT_CHANNEL).to_numpy() if not mask.any(): raise SourceRuntimeError( diff --git a/packages/microcosm-build/src/microcosm/build/us_runtime/retirement_distributions.py b/packages/microcosm-build/src/microcosm/build/us_runtime/retirement_distributions.py index b98f7df52..7d9659af0 100644 --- a/packages/microcosm-build/src/microcosm/build/us_runtime/retirement_distributions.py +++ b/packages/microcosm-build/src/microcosm/build/us_runtime/retirement_distributions.py @@ -45,8 +45,8 @@ from microcosm.build.us_runtime.support_provenance import ( has_assembled_support_metadata, has_support_role_metadata, + support_gate_source_channel_series, support_role_series, - support_source_channel_series, ) from microcosm.frame import Frame from microcosm.frame.units import US_SCHEMA @@ -276,7 +276,7 @@ def _asec_source_mask(frame: pd.DataFrame) -> np.ndarray: if not has_assembled_support_metadata(frame, entity="person"): return np.ones(len(frame), dtype=bool) - source_channels = support_source_channel_series(frame, entity="person") + source_channels = support_gate_source_channel_series(frame, entity="person") mask = source_channels.eq(_BASE_ASEC_SUPPORT_CHANNEL).to_numpy() if not mask.any(): raise SourceRuntimeError( diff --git a/packages/microcosm-build/src/microcosm/build/us_runtime/ssi_take_up.py b/packages/microcosm-build/src/microcosm/build/us_runtime/ssi_take_up.py index 47b2b9fd0..86f208960 100644 --- a/packages/microcosm-build/src/microcosm/build/us_runtime/ssi_take_up.py +++ b/packages/microcosm-build/src/microcosm/build/us_runtime/ssi_take_up.py @@ -77,8 +77,8 @@ PUF_TAX_DETAIL_SUPPORT_CHANNEL, has_support_role_metadata, support_clone_index_column, + support_gate_source_channel_series, support_role_series, - support_source_channel_series, ) from microcosm.frame import Frame from microcosm.frame.units import US_SCHEMA @@ -531,7 +531,7 @@ def us_ssi_take_up_reporter_source_ids(frame: Frame) -> frozenset[str]: if person[_SOURCE_ID].isna().any(): raise ValueError("US SSI take-up reporter lineage requires provenance.") source_ids = _decoded_strings(person[_SOURCE_ID]) - source_channels = support_source_channel_series(person, entity="person") + source_channels = support_gate_source_channel_series(person, entity="person") asec_source = source_channels.eq(_ASEC_CHANNEL).to_numpy() reported = pd.to_numeric(person[US_SSI_TAKE_UP_ANCHOR], errors="coerce").to_numpy( dtype=np.float64 diff --git a/packages/microcosm-build/src/microcosm/build/us_runtime/support_provenance.py b/packages/microcosm-build/src/microcosm/build/us_runtime/support_provenance.py index ade62a0e9..ec5a51eaf 100644 --- a/packages/microcosm-build/src/microcosm/build/us_runtime/support_provenance.py +++ b/packages/microcosm-build/src/microcosm/build/us_runtime/support_provenance.py @@ -26,7 +26,7 @@ "support_channel_column", "support_clone_index_column", "support_role_series", - "support_source_channel_series", + "support_gate_source_channel_series", "support_source_id_column", "us_reported_coverage_vintage_signal_gate", "validate_assembly_provenance", @@ -493,19 +493,20 @@ def support_role_series( ) -def support_source_channel_series( +def support_gate_source_channel_series( table: pd.DataFrame, *, entity: str, ) -> pd.Series: - """Return validated physical source channels with a legacy fallback. + """Return physical source channels for read-only gates and reporters. Assembled frames retain their receipt-declared physical channels (for example, ``asec`` and ``acs``). Historical frames have no raw spine ID and use the exact ``asec``/``puf_tax_detail`` operator roles as their channels. - Keeping this resolution in the provenance owner lets reporting and release - gates inspect source coverage without teaching population operators how to - read provenance columns directly. + This deliberately narrow accessor must not route population treatments. + Keeping resolution in the provenance owner lets reporting and release gates + inspect source coverage without teaching operators how to read provenance + columns directly. A static call-site contract pins its reviewed consumers. """ roles = support_role_series(table, entity=entity) diff --git a/packages/microcosm-build/src/microcosm/build/us_runtime/weeks_unemployed.py b/packages/microcosm-build/src/microcosm/build/us_runtime/weeks_unemployed.py index e6899d7fb..dd7e36641 100644 --- a/packages/microcosm-build/src/microcosm/build/us_runtime/weeks_unemployed.py +++ b/packages/microcosm-build/src/microcosm/build/us_runtime/weeks_unemployed.py @@ -44,8 +44,8 @@ has_assembled_support_metadata, has_support_role_metadata, support_clone_index_column, + support_gate_source_channel_series, support_role_series, - support_source_channel_series, ) from microcosm.frame import Frame from microcosm.frame.units import US_SCHEMA @@ -1352,7 +1352,7 @@ def _weeks_unemployed_gate_scopes( ) clone_column = support_clone_index_column("person") - source_channel = support_source_channel_series( + source_channel = support_gate_source_channel_series( person, entity="person", ).to_numpy(dtype=object, copy=True) diff --git a/packages/microcosm-build/src/microcosm/build/us_runtime/workers_compensation.py b/packages/microcosm-build/src/microcosm/build/us_runtime/workers_compensation.py index d531d009a..80426c5e2 100644 --- a/packages/microcosm-build/src/microcosm/build/us_runtime/workers_compensation.py +++ b/packages/microcosm-build/src/microcosm/build/us_runtime/workers_compensation.py @@ -30,8 +30,8 @@ from microcosm.build.us_runtime.support_provenance import ( has_assembled_support_metadata, has_support_role_metadata, + support_gate_source_channel_series, support_role_series, - support_source_channel_series, ) from microcosm.frame import Frame from microcosm.frame.units import US_SCHEMA @@ -575,7 +575,9 @@ def us_workers_compensation_summary(frame: Frame) -> dict[str, object]: source_reconciliation_mask = source_mask.copy() if has_support_role_metadata(person, entity="person"): source_mask = ( - support_source_channel_series(person, entity="person").to_numpy() + support_gate_source_channel_series( + person, entity="person" + ).to_numpy() == _BASE_ASEC_SUPPORT_CHANNEL ) source_reconciliation_mask = source_mask.copy() diff --git a/packages/microcosm-build/tests/test_us_multispine_puf_clone.py b/packages/microcosm-build/tests/test_us_multispine_puf_clone.py index ea1828efd..d9b3dc7b1 100644 --- a/packages/microcosm-build/tests/test_us_multispine_puf_clone.py +++ b/packages/microcosm-build/tests/test_us_multispine_puf_clone.py @@ -18,10 +18,12 @@ support_channel_column, support_clone_index_column, support_role_series, - support_source_channel_series, support_source_id_column, ) -from microcosm.build.us_runtime.support_provenance import spine_assembly_manifest +from microcosm.build.us_runtime.support_provenance import ( + spine_assembly_manifest, + support_gate_source_channel_series, +) from microcosm.frame import US_SCHEMA, Frame, WeightKind, Weights @@ -114,7 +116,7 @@ def test_puf_clone_preserves_source_spines_and_routes_by_clone_index() -> None: PUF_TAX_DETAIL_SUPPORT_CHANNEL, ] assert has_assembled_support_metadata(table, entity=entity) - assert support_source_channel_series(table, entity=entity).tolist() == [ + assert support_gate_source_channel_series(table, entity=entity).tolist() == [ "asec", "acs", "asec", @@ -186,7 +188,7 @@ def test_support_role_legacy_fallback_is_closed_to_known_roles() -> None: assert has_support_role_metadata(legacy, entity="person") assert not has_assembled_support_metadata(legacy, entity="person") - assert support_source_channel_series(legacy, entity="person").tolist() == [ + assert support_gate_source_channel_series(legacy, entity="person").tolist() == [ BASE_ASEC_SUPPORT_CHANNEL, PUF_TAX_DETAIL_SUPPORT_CHANNEL, ] diff --git a/packages/microcosm-build/tests/test_us_spine_blindness.py b/packages/microcosm-build/tests/test_us_spine_blindness.py index aa6b80162..98c59af52 100644 --- a/packages/microcosm-build/tests/test_us_spine_blindness.py +++ b/packages/microcosm-build/tests/test_us_spine_blindness.py @@ -174,6 +174,36 @@ "workers_compensation.py", ) +_GATE_SOURCE_CHANNEL_ACCESSOR = "support_gate_source_channel_series" +_GATE_SOURCE_CHANNEL_CALLERS = { + "alimony.py": frozenset({"us_alimony_signal_gate"}), + "medicare_take_up.py": frozenset({"_asec_source_mask"}), + "retirement_contributions.py": frozenset({"_asec_source_mask"}), + "retirement_distributions.py": frozenset({"_asec_source_mask"}), + "ssi_take_up.py": frozenset({"us_ssi_take_up_reporter_source_ids"}), + "weeks_unemployed.py": frozenset({"_weeks_unemployed_gate_scopes"}), + "workers_compensation.py": frozenset({"us_workers_compensation_summary"}), +} +_GATE_SOURCE_SCOPE_HELPER_CALLERS = { + ("medicare_take_up.py", "_asec_source_mask"): frozenset( + {"us_medicare_take_up_summary"} + ), + ("retirement_contributions.py", "_asec_source_mask"): frozenset( + {"_source_reconciliation_mask", "us_retirement_contributions_summary"} + ), + ( + "retirement_contributions.py", + "_source_reconciliation_mask", + ): frozenset({"us_retirement_contributions_summary"}), + ("retirement_distributions.py", "_asec_source_mask"): frozenset( + {"_source_reconciliation_mask", "us_retirement_distributions_summary"} + ), + ( + "retirement_distributions.py", + "_source_reconciliation_mask", + ): frozenset({"us_retirement_distributions_summary"}), +} + # Every runtime module must be deliberately classified. This allowlist does # not exempt a module from the all-runtime AST scan below; it only records # modules outside the migrated population-treatment registry. Keeping the @@ -3071,6 +3101,46 @@ def _called_function_names(source: str) -> set[str]: } +def _function_callers(source: str, callee: str) -> tuple[tuple[str, int], ...]: + """Return function names and lines that call one imported or local name.""" + + tree = ast.parse(source) + aliases = {callee} + for node in ast.walk(tree): + if not isinstance(node, ast.ImportFrom): + continue + aliases.update( + alias.asname or alias.name + for alias in node.names + if alias.name == callee + ) + + class CallerVisitor(ast.NodeVisitor): + def __init__(self) -> None: + self.function_stack: list[str] = [] + self.callers: list[tuple[str, int]] = [] + + def visit_FunctionDef(self, node: ast.FunctionDef) -> None: + self.function_stack.append(node.name) + self.generic_visit(node) + self.function_stack.pop() + + def visit_AsyncFunctionDef(self, node: ast.AsyncFunctionDef) -> None: + self.function_stack.append(node.name) + self.generic_visit(node) + self.function_stack.pop() + + def visit_Call(self, node: ast.Call) -> None: + if _call_name(node) in aliases: + caller = self.function_stack[-1] if self.function_stack else "" + self.callers.append((caller, node.lineno)) + self.generic_visit(node) + + visitor = CallerVisitor() + visitor.visit(tree) + return tuple(visitor.callers) + + def _imported_us_runtime_modules(source: str) -> tuple[str, ...]: """Return statically imported, flat ``us_runtime`` module filenames.""" @@ -3267,6 +3337,48 @@ def test_registered_population_operators_do_not_read_any_source_channel() -> Non ) +def test_physical_source_accessor_is_confined_to_reviewed_gates() -> None: + """Origin-aware diagnostics must not become population treatments.""" + + actual: dict[str, frozenset[str]] = {} + call_details: dict[str, tuple[tuple[str, int], ...]] = {} + for path in sorted(_US_RUNTIME.glob("*.py")): + if path.name == "support_provenance.py": + continue + callers = _function_callers(path.read_text(), _GATE_SOURCE_CHANNEL_ACCESSOR) + if callers: + actual[path.name] = frozenset(caller for caller, _line in callers) + call_details[path.name] = callers + assert actual == _GATE_SOURCE_CHANNEL_CALLERS, ( + "The physical support-channel accessor is restricted to reviewed " + "release gates and reporter capture; population derivation, imputation, " + f"or wrapper use is forbidden. Found callers: {call_details}" + ) + + for (module_name, helper), expected_callers in ( + _GATE_SOURCE_SCOPE_HELPER_CALLERS.items() + ): + callers = _function_callers((_US_RUNTIME / module_name).read_text(), helper) + actual_callers = frozenset(caller for caller, _line in callers) + assert actual_callers == expected_callers, ( + f"Physical source-scope helper {module_name}:{helper} may be called " + "only by its reviewed gate-summary chain; " + f"expected={sorted(expected_callers)}, found={callers}." + ) + + +def test_physical_source_accessor_guard_rejects_a_derivation_call() -> None: + source = """ +from owner import support_gate_source_channel_series as channels + +def derive_population(frame): + return channels(frame, entity=\"person\") +""" + assert _function_callers(source, _GATE_SOURCE_CHANNEL_ACCESSOR) == ( + ("derive_population", 5), + ) + + def test_pool_build_tool_import_graph_is_source_spine_blind() -> None: """The wired CLI and every runtime operator it reaches remain blind.""" From b2de92b973431b96677b6c71bf14161e3c9b0f1d Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Thu, 27 Aug 2026 09:00:04 +0200 Subject: [PATCH 24/49] Document stacked release gate alignment --- PROGRESS.md | 33 ++++++++++++++----- .../stacked-release-gate-alignment.fixed.md | 1 + 2 files changed, 26 insertions(+), 8 deletions(-) create mode 100644 changelog.d/stacked-release-gate-alignment.fixed.md diff --git a/PROGRESS.md b/PROGRESS.md index 78067381e..e5e232040 100644 --- a/PROGRESS.md +++ b/PROGRESS.md @@ -5,10 +5,12 @@ In progress on 2026-08-27. Real-pool provenance has refuted the proposed post-transfer amount-mapping mechanism: every fractional week is an ACS-origin non-native clone prediction outside the calibration's clone-0 recipient scope. -The source codec and weeks-gate architecture fixes are implemented and -focused-tested. The complete release-call roster is classified; the -unambiguous source-scope and clone-layout repairs are now in progress, while -three archived-model input assumptions require owner rulings. +The source codec, weeks-gate architecture, source-scope, clone-layout, and +stable-identity repairs are implemented and focused-tested. The complete +release-call roster is classified; six archived-model input assumptions require +owner rulings and are deliberately reported instead of guessed. Repository-wide +Ruff, the CI inventory verifier, and four of five full pytest shards pass; the +build shard is still running. No network access, artifact build, publication, push, pool build, or release build is in scope. @@ -127,13 +129,28 @@ build is in scope. caller contract (including internal mask-helper callers). A future derive, impute, or wrapper use now fails the source-blindness suite instead of passing through indirection. +- Replayed the repaired release gates read-only against the supplied pool. + Alimony, Medicare, retirement contributions/distributions, workers' + compensation, and SSI reporter capture pass; the weeks gate now fails only + on the genuine 369 fractional values. Prior-year income remains outside its + unchanged availability band and WIC finds pregnant nonfemale rows, both + genuine data/spec outcomes rather than stacked-layout archaeology. +- Completed an adversarial review of the repaired code and focused tests with + no additional implementation defect found. It confirmed six owner-ruling + items: SSI disability criteria, SCF wealth, SCF auto loans, SIPP vehicles, + SIPP tips, and ORG wages/FLSA all consume ASEC-only archived predictors on a + frame whose 1,736,840 physical ACS rows carry null source cells. ORG is + guaranteed to fail its unchanged race/occupation bands; SIPP tips can + silently pass its global band while the ACS channel is dead. +- Passed repository-wide Ruff and the CI test-group inventory verifier. The + calibrate, data, fit, and frame shards pass in four independent pytest + processes; the build shard remains in progress. ## Next -- Implement and focused-test the unambiguous source-scope, clone-layout, and - stable-key repairs. -- Run the complete prescribed verification and write the final provenance, - audit evidence, and judgment calls to `out.md`. +- Finish the full build-shard pytest process. +- Write the final provenance, audit evidence, verification receipts, and + judgment calls to `out.md`, then leave a clean committed worktree. # Historical: gate-failed base-pool release lane diff --git a/changelog.d/stacked-release-gate-alignment.fixed.md b/changelog.d/stacked-release-gate-alignment.fixed.md new file mode 100644 index 000000000..4fe48f469 --- /dev/null +++ b/changelog.d/stacked-release-gate-alignment.fixed.md @@ -0,0 +1 @@ +Preserve integer support for transferred ``weeks_unemployed`` values and align US release-side source, clone, and deterministic-identity checks with stacked ASEC+ACS pools while retaining legacy ASEC+PUF behavior and every existing plausibility threshold. From cd8c55a7d9ca917904352b7a33f80eff9e20229b Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Thu, 27 Aug 2026 09:38:11 +0200 Subject: [PATCH 25/49] Repin spec engine for integer weeks --- PROGRESS.md | 30 +++++++++-- docs/evidence/spec-engine/us-f0-coverage.json | 54 +++++++++---------- .../build/spec_engine/field_usage.py | 14 ++--- .../build/spec_engine/inventory_coverage.py | 8 +-- .../src/microcosm/build/spec_engine/seeds.py | 1 + .../tests/test_spec_engine_country_bundles.py | 4 +- .../tests/test_spec_engine_coverage_tool.py | 16 +++--- .../tests/test_spec_engine_field_usage.py | 16 +++--- .../tests/test_spec_engine_loader.py | 2 +- .../tests/test_spec_engine_seeds.py | 10 ++++ tools/spec_engine_coverage.py | 2 +- 11 files changed, 94 insertions(+), 63 deletions(-) diff --git a/PROGRESS.md b/PROGRESS.md index e5e232040..d277fb22d 100644 --- a/PROGRESS.md +++ b/PROGRESS.md @@ -10,7 +10,8 @@ stable-identity repairs are implemented and focused-tested. The complete release-call roster is classified; six archived-model input assumptions require owner rulings and are deliberately reported instead of guessed. Repository-wide Ruff, the CI inventory verifier, and four of five full pytest shards pass; the -build shard is still running. +build shard is pending restart after its fail-closed spec-engine receipts were +coherently re-pinned. No network access, artifact build, publication, push, pool build, or release build is in scope. @@ -58,8 +59,10 @@ build is in scope. - Regenerated the checked-in US imputation authority so `transfer_execution.discrete_numeric_targets` includes `weeks_unemployed`. The generator's compile and byte-staleness checks pass at bundle spec SHA - `821d5838da3ac368170e61e017f1a72648f93e8a011aa40e33b8c2b4b14511f3`; - the spec-bundle and imputation-semantics suites pass. + `821d5838da3ac368170e61e017f1a72648f93e8a011aa40e33b8c2b4b14511f3` + at that source-fix step; the later runtime/seed identity repin below + supersedes this intermediate digest. The spec-bundle and imputation- + semantics suites pass. - Passed the complete ACS-transfer test file (65 tests), the complete post-transfer calibration receipt-contract file (47 tests), focused Ruff, and `git diff --check` using the prebuilt `.venv` directly. `uv run --no-sync` @@ -144,11 +147,28 @@ build is in scope. silently pass its global band while the ACS channel is dead. - Passed repository-wide Ruff and the CI test-group inventory verifier. The calibrate, data, fit, and frame shards pass in four independent pytest - processes; the build shard remains in progress. + processes. +- Corrected the reviewed WIC seed protocol to match the implemented assembled- + multispine key precedence: assembly-unique `person_source_id` first, then the + unchanged legacy raw/support/person fallbacks. An exact seed-grammar test now + binds that order. +- Re-pinned the fail-closed spec-engine proof after adding one authored + transfer-execution field and one resolved seed-protocol field: 42,122 total + fields (32,352 authored and 9,770 resolved), complete exact-pointer claims, + and all 41 inventory checks. Regenerated the committed coverage report and + validated the final US spec SHA + `5f44d96d45e9aabcea2d565ef063d68bfc0652df1b38b08aa31ce6896d15f371`. +- Verified in a detached `origin/main` worktree, using the same prebuilt venv, + that the old BE, UK, and minimal-spec golden vectors still pass there. Their + current repins therefore reflect this branch's attested runtime and seed- + protocol changes rather than environment drift. All 102 tests in the eight + directly affected spec-engine files pass; generated-bundle and coverage- + report byte checks, focused Ruff, and `git diff --check` also pass. ## Next -- Finish the full build-shard pytest process. +- Restart and finish the full build-shard pytest process after the committed + spec-engine repin. - Write the final provenance, audit evidence, verification receipts, and judgment calls to `out.md`, then leave a clean committed worktree. diff --git a/docs/evidence/spec-engine/us-f0-coverage.json b/docs/evidence/spec-engine/us-f0-coverage.json index 68f70290f..c3f6bd0a9 100644 --- a/docs/evidence/spec-engine/us-f0-coverage.json +++ b/docs/evidence/spec-engine/us-f0-coverage.json @@ -6,7 +6,7 @@ "country": "us", "documentation_sha256": "4b39450dbdb8dafb83c3b627123b8026c6f82c660b66fe76f341a67c4f37c77b", "field_usage": { - "authored_normative_field_count": 32351, + "authored_normative_field_count": 32352, "claim_count": 49, "claims": [ { @@ -220,8 +220,8 @@ ], "mode": "legacy_behavior", "pointer_class": "all", - "pointer_count": 79, - "pointer_sha256": "1575513251c72be60edc75f378221fa878ea76e009512f1352a111586d0a60f8", + "pointer_count": 80, + "pointer_sha256": "10bd811f86dae7941035533cfe0d51d81cd632f5995c1adb80234bc7ecbf6565", "rationale": null, "relative_sink_prefix": null, "source_prefix": "/authored/spec~1imputation.yaml/transfer_execution", @@ -736,8 +736,8 @@ "legacy_sinks": [], "mode": "compiler_semantic", "pointer_class": "all", - "pointer_count": 823, - "pointer_sha256": "c05ebb88538195798e9365f325381c14d92b74007f846f1fef50a08268e6d27d", + "pointer_count": 824, + "pointer_sha256": "7537385c3fd399a2dbb7dcd8ed7cf1ff2481ed336db621eafcbfd741d5792f40", "rationale": null, "relative_sink_prefix": null, "source_prefix": "/resolved/seed_protocol", @@ -772,21 +772,21 @@ "verifier": "vintages" } ], - "configuration_field_count": 42120, - "consumed_field_count": 42120, + "configuration_field_count": 42122, + "consumed_field_count": 42122, "generation0_effect_counts": { - "legacy_behavior": 38443, - "no_generation0_effect": 3677 + "legacy_behavior": 38444, + "no_generation0_effect": 3678 }, "mode_counts": { - "compiler_semantic": 27699, + "compiler_semantic": 27700, "front_end_validation": 348, "identity_only": 103, - "legacy_behavior": 13970 + "legacy_behavior": 13971 }, "multiple_primary_use_field_count": 0, - "pointer_inventory_sha256": "bc4a948ab632191954600da8474c5b011f977a65e24c399d126f3dc4a79f23e5", - "resolved_binding_field_count": 9769, + "pointer_inventory_sha256": "ae7881c07c6137037f43a5992b3e8fffbbdbd5ddf2e1b70956986cb9df043f71", + "resolved_binding_field_count": 9770, "unused_field_count": 0 }, "inventory_coverage": { @@ -1361,12 +1361,12 @@ ], "expected": { "producer_count": 38, - "sha256": "3850554eb804fde5e4f86a34ac1bb8a7a07aafff7e8b48396a3d5fca844798e8" + "sha256": "ef491637933945e28645479da70646d9c50366213b6f70477de445ab9e4bb1cf" }, "failures": [], "observed": { "producer_count": 38, - "sha256": "3850554eb804fde5e4f86a34ac1bb8a7a07aafff7e8b48396a3d5fca844798e8" + "sha256": "ef491637933945e28645479da70646d9c50366213b6f70477de445ab9e4bb1cf" }, "status": "covered" }, @@ -1656,13 +1656,13 @@ "compiler_ir.node_slices" ], "expected": { - "map_sha256": "36a9d819ef196c312888591936d49c025b0407df9928440cceefabc5458f72af", - "protocol_sha256": "15840b380329410a7094f60b0f1dad453457fc785859f0c372f2c8e2d59b0246" + "map_sha256": "ee1ad8d5196ca48d70657675237294c2dd925a520900d4d476d256a1b514b283", + "protocol_sha256": "c7df307cba6b20cc26f4aeeb1ee5eae246362318cef885cd747c64349e08cc41" }, "failures": [], "observed": { - "map_sha256": "36a9d819ef196c312888591936d49c025b0407df9928440cceefabc5458f72af", - "protocol_sha256": "15840b380329410a7094f60b0f1dad453457fc785859f0c372f2c8e2d59b0246" + "map_sha256": "ee1ad8d5196ca48d70657675237294c2dd925a520900d4d476d256a1b514b283", + "protocol_sha256": "c7df307cba6b20cc26f4aeeb1ee5eae246362318cef885cd747c64349e08cc41" }, "status": "covered" }, @@ -1677,7 +1677,7 @@ "compiler_ir.seed_stream_map" ], "expected": { - "implementation_sha256": "15840b380329410a7094f60b0f1dad453457fc785859f0c372f2c8e2d59b0246", + "implementation_sha256": "c7df307cba6b20cc26f4aeeb1ee5eae246362318cef885cd747c64349e08cc41", "protocol": "legacy-v1", "streams": [ "build_model", @@ -1698,7 +1698,7 @@ }, "failures": [], "observed": { - "implementation_sha256": "15840b380329410a7094f60b0f1dad453457fc785859f0c372f2c8e2d59b0246", + "implementation_sha256": "c7df307cba6b20cc26f4aeeb1ee5eae246362318cef885cd747c64349e08cc41", "protocol": "legacy-v1", "streams": [ "build_model", @@ -1736,7 +1736,7 @@ }, "failures": [], "observed": { - "sha256": "99710bf212c3801cfd0434d56d2d1d2e4902a2a958978cdbd91cd9acce81d2fb", + "sha256": "a0366cd518f42a240b589af3ef6580aac44ad7ed70969bcc669b365d315146ab", "sites": 53 }, "status": "covered" @@ -1887,7 +1887,7 @@ "alpha", "zeta" ], - "sha256": "a128a85f877fb32def9382b841b8b340f974e8a9148ac029c1f04becdc956c18" + "sha256": "1fa30ae9213c19607168df3cf0035478b9c96f815e1541cc97f4e1b8e7246935" }, "failures": [], "observed": { @@ -1911,7 +1911,7 @@ "alpha", "zeta" ], - "sha256": "a128a85f877fb32def9382b841b8b340f974e8a9148ac029c1f04becdc956c18" + "sha256": "1fa30ae9213c19607168df3cf0035478b9c96f815e1541cc97f4e1b8e7246935" }, "status": "covered" }, @@ -1975,7 +1975,7 @@ "take_up_contract", "us_qbi_reconciliation_contract" ], - "sha256": "91c65a9ff36839d575036264c4bf57ffde6457e8fd180a0856f9be712ada371d" + "sha256": "001a2bdc700615cd96771fc258abf5f769d67ea41687efd8cc2ad30361619057" }, "status": "covered" }, @@ -2599,7 +2599,7 @@ "country": "us", "schema_id": "country_spec", "schema_version": 1, - "spec_sha256": "5378bb9189aec96f50da22aac71e5bd2c3d919e9795f6ef2147e0bc9c739dd8e" + "spec_sha256": "5f44d96d45e9aabcea2d565ef063d68bfc0652df1b38b08aa31ce6896d15f371" } }, "report_schema_version": 3, @@ -2609,7 +2609,7 @@ "country": "us", "schema_id": "country_spec", "schema_version": 1, - "spec_sha256": "5378bb9189aec96f50da22aac71e5bd2c3d919e9795f6ef2147e0bc9c739dd8e" + "spec_sha256": "5f44d96d45e9aabcea2d565ef063d68bfc0652df1b38b08aa31ce6896d15f371" }, "status": "pass" } diff --git a/packages/microcosm-build/src/microcosm/build/spec_engine/field_usage.py b/packages/microcosm-build/src/microcosm/build/spec_engine/field_usage.py index a5dd282ab..ad66f291a 100644 --- a/packages/microcosm-build/src/microcosm/build/spec_engine/field_usage.py +++ b/packages/microcosm-build/src/microcosm/build/spec_engine/field_usage.py @@ -26,9 +26,9 @@ ) from .schemas import load_schema_registry -EXPECTED_AUTHORED_FIELD_COUNT = 32_351 -EXPECTED_RESOLVED_BINDING_FIELD_COUNT = 9_769 -EXPECTED_CONFIGURATION_FIELD_COUNT = 42_120 +EXPECTED_AUTHORED_FIELD_COUNT = 32_352 +EXPECTED_RESOLVED_BINDING_FIELD_COUNT = 9_770 +EXPECTED_CONFIGURATION_FIELD_COUNT = 42_122 class FieldUsageError(AssertionError): @@ -429,8 +429,8 @@ def _path_inventory(rows: Sequence[tuple[str, object]]) -> tuple[int, str]: "1b9f7ffb88d31d2bd365c092d04a96038b2d5a2b03f4f2a79f42e95472b213d6", ), "imputation_transfer_execution": ( - 79, - "1575513251c72be60edc75f378221fa878ea76e009512f1352a111586d0a60f8", + 80, + "10bd811f86dae7941035533cfe0d51d81cd632f5995c1adb80234bc7ecbf6565", ), "imputation_waiver_records": ( 70, @@ -457,8 +457,8 @@ def _path_inventory(rows: Sequence[tuple[str, object]]) -> tuple[int, str]: "6a781915fd491d2c4b16d2b7d482f69cf362c904130093c59f9629f7a319269b", ), "resolved_seed_protocol": ( - 823, - "c05ebb88538195798e9365f325381c14d92b74007f846f1fef50a08268e6d27d", + 824, + "7537385c3fd399a2dbb7dcd8ed7cf1ff2481ed336db621eafcbfd741d5792f40", ), "resolved_seed_site_bindings": ( 277, diff --git a/packages/microcosm-build/src/microcosm/build/spec_engine/inventory_coverage.py b/packages/microcosm-build/src/microcosm/build/spec_engine/inventory_coverage.py index b48afbf58..f0dd2529c 100644 --- a/packages/microcosm-build/src/microcosm/build/spec_engine/inventory_coverage.py +++ b/packages/microcosm-build/src/microcosm/build/spec_engine/inventory_coverage.py @@ -350,17 +350,17 @@ "acs_person_predictors": "878c788a6f037d7aca12b3586ea034eff04f3034ffa11935a736493042551f25", "authority": "3a980927227704d0589f246eef9cd825c2ae84f3a4134ac835e0e5ed39a563ac", "early_families": "4aa9f736fd76e83955477ad1667e58f48f264783f05bdc7f0102cd32d61323bd", - "full_checkpoint": "a128a85f877fb32def9382b841b8b340f974e8a9148ac029c1f04becdc956c18", + "full_checkpoint": "1fa30ae9213c19607168df3cf0035478b9c96f815e1541cc97f4e1b8e7246935", "gap_fill_schedule": "1c31f9868f7884347cc19cf1ff65da43f950b9114941a715bab168246db414a7", "graph_nodes": "7125ad28ae2c69f22094a574bbf6ed2ddf1682a2c2c3b416f8f49304b7016ce7", "geography_assignment": "f49425ca8734ac559c73cf44f6458d86d3162a48956b98a27e6e758959361585", "late_families": "d91f9ff0eb52f43e7b6eed3d5c58c37abe1620c3a11021da15dae9c10e16d382", - "late_resource_semantics": "3850554eb804fde5e4f86a34ac1bb8a7a07aafff7e8b48396a3d5fca844798e8", + "late_resource_semantics": "ef491637933945e28645479da70646d9c50366213b6f70477de445ab9e4bb1cf", "late_schedule": "dcf3c6d2eade3449836c49a1dc4d3b8cd395aab9142db700c3c60598fa9c1c79", "ownership": "5f64f0aac49e2313177564f71876bffc8c81b3ded4df701e70930e60e9c98356", "primary_tuples": "987b501c695e31f45521c4a178528f75ab3df22c09bc407b182213b2de99ee57", - "seed_map": "36a9d819ef196c312888591936d49c025b0407df9928440cceefabc5458f72af", - "seed_protocol": "15840b380329410a7094f60b0f1dad453457fc785859f0c372f2c8e2d59b0246", + "seed_map": "ee1ad8d5196ca48d70657675237294c2dd925a520900d4d476d256a1b514b283", + "seed_protocol": "c7df307cba6b20cc26f4aeeb1ee5eae246362318cef885cd747c64349e08cc41", "source_manifest": "cd5ba8924d64da5425ee14cca82a774e3f4b2bb5aabe06df291cc3cc457287a9", "take_up": "fa186daea0f8dd641cc470e41d1a2953f887d45282ec990201298f47bedf8d4d", "tail": "ac92829c88a1a4fb6460d61190918d5d99c6c377fc8dd8f62f02b332d09bf59c", diff --git a/packages/microcosm-build/src/microcosm/build/spec_engine/seeds.py b/packages/microcosm-build/src/microcosm/build/spec_engine/seeds.py index 36e6c27e8..04fb87ad8 100644 --- a/packages/microcosm-build/src/microcosm/build/spec_engine/seeds.py +++ b/packages/microcosm-build/src/microcosm/build/spec_engine/seeds.py @@ -726,6 +726,7 @@ def _stable_site( "wic_claim_assignment", salt="would_claim_wic", key_grammar=( + "source:person_source_id_if_assembled_multispine", "source_year:source_household_id:source_person_id_if_complete", "else_support:person_support_source_id", "else_person:person_id", diff --git a/packages/microcosm-build/tests/test_spec_engine_country_bundles.py b/packages/microcosm-build/tests/test_spec_engine_country_bundles.py index cc672a159..bcac668ec 100644 --- a/packages/microcosm-build/tests/test_spec_engine_country_bundles.py +++ b/packages/microcosm-build/tests/test_spec_engine_country_bundles.py @@ -32,7 +32,7 @@ [ ( "be", - "86143c1c3f98980e34490c75706fe7dbf72e96e8accb8d7d4bbd2c1ae1a29b65", + "f7fd69bf4964faa2eeea1094323ce3a5ace180d76acfff0c30ed0d592ebc129e", { "household.household_id", "person.person_id", @@ -42,7 +42,7 @@ ), ( "uk", - "0c85845b4d463638ae3e5c5a25e17de8b720794e3653c5991dc4f069d95762d3", + "e040368ecf8fb89da14ff0b8f2b1f1c8f6845a18423dfe9792c0c2abc094bbbf", { "benunit.benunit_id", "household.household_id", diff --git a/packages/microcosm-build/tests/test_spec_engine_coverage_tool.py b/packages/microcosm-build/tests/test_spec_engine_coverage_tool.py index 7a4e443bf..0bd070285 100644 --- a/packages/microcosm-build/tests/test_spec_engine_coverage_tool.py +++ b/packages/microcosm-build/tests/test_spec_engine_coverage_tool.py @@ -52,22 +52,22 @@ def test_us_coverage_is_exact_complete_and_honest( assert_coverage_complete(coverage_report) assert coverage_report["status"] == "pass" fields = coverage_report["field_usage"] - assert fields["configuration_field_count"] == 42_120 - assert fields["authored_normative_field_count"] == 32_351 - assert fields["resolved_binding_field_count"] == 9_769 - assert fields["consumed_field_count"] == 42_120 + assert fields["configuration_field_count"] == 42_122 + assert fields["authored_normative_field_count"] == 32_352 + assert fields["resolved_binding_field_count"] == 9_770 + assert fields["consumed_field_count"] == 42_122 assert fields["unused_field_count"] == 0 assert fields["multiple_primary_use_field_count"] == 0 assert fields["claim_count"] == 49 assert fields["mode_counts"] == { - "legacy_behavior": 13_970, - "compiler_semantic": 27_699, + "legacy_behavior": 13_971, + "compiler_semantic": 27_700, "front_end_validation": 348, "identity_only": 103, } assert fields["generation0_effect_counts"] == { - "legacy_behavior": 38_443, - "no_generation0_effect": 3_677, + "legacy_behavior": 38_444, + "no_generation0_effect": 3_678, } inventory = coverage_report["inventory_coverage"] diff --git a/packages/microcosm-build/tests/test_spec_engine_field_usage.py b/packages/microcosm-build/tests/test_spec_engine_field_usage.py index 77fad4942..fb6b525b4 100644 --- a/packages/microcosm-build/tests/test_spec_engine_field_usage.py +++ b/packages/microcosm-build/tests/test_spec_engine_field_usage.py @@ -92,22 +92,22 @@ def _mutated_bundle( def test_exact_complete_ledger_has_one_primary_mode_per_pointer(field_ledger) -> None: - assert len(field_ledger.fields) == EXPECTED_CONFIGURATION_FIELD_COUNT == 42_120 + assert len(field_ledger.fields) == EXPECTED_CONFIGURATION_FIELD_COUNT == 42_122 assert field_ledger.source_counts == { - "authored": 32_351, - "resolved_bindings": 9_769, + "authored": 32_352, + "resolved_bindings": 9_770, } assert field_ledger.mode_counts == { - "legacy_behavior": 13_970, - "compiler_semantic": 27_699, + "legacy_behavior": 13_971, + "compiler_semantic": 27_700, "front_end_validation": 348, "identity_only": 103, } assert field_ledger.generation0_effect_counts == { - "legacy_behavior": 38_443, - "no_generation0_effect": 3_677, + "legacy_behavior": 38_444, + "no_generation0_effect": 3_678, } - assert len({field.pointer for field in field_ledger.fields}) == 42_120 + assert len({field.pointer for field in field_ledger.fields}) == 42_122 def test_eligibility_concepts_are_validation_not_generation0_behavior( diff --git a/packages/microcosm-build/tests/test_spec_engine_loader.py b/packages/microcosm-build/tests/test_spec_engine_loader.py index a52518ee6..8a3e0e232 100644 --- a/packages/microcosm-build/tests/test_spec_engine_loader.py +++ b/packages/microcosm-build/tests/test_spec_engine_loader.py @@ -235,7 +235,7 @@ def test_semantic_hash_has_golden_vector_and_surface_separation(tmp_path) -> Non # Pin the domain separator, normalization rules, schema-set receipt, and # exact normative projection as one reviewable golden vector. assert first.spec_sha256 == ( - "1ff676072985f104b9d80e3e5fa6e2078778969246246cf1371e982c772550cc" + "2473f29cdd4aef89cb7998e7effeeb14272f4aaaa91f277f48163d8f5137d1de" ) second_root = _rich_minimal(tmp_path / "xy", note="second", store="local:b") diff --git a/packages/microcosm-build/tests/test_spec_engine_seeds.py b/packages/microcosm-build/tests/test_spec_engine_seeds.py index ec1a50ab1..c0e181a93 100644 --- a/packages/microcosm-build/tests/test_spec_engine_seeds.py +++ b/packages/microcosm-build/tests/test_spec_engine_seeds.py @@ -292,6 +292,16 @@ def test_exact_blake2b_salts_keys_candidates_and_absence_conditions() -> None: assert count.draw_condition == "only_when_declared_draw_column_is_absent" assert joint.draw_condition == "only_when_declared_draw_column_is_absent" + wic = LEGACY_V1_PROTOCOL.site("wic_claim_assignment") + assert wic.seed_material == ( + "build_model_seed", + "literal_salt=would_claim_wic", + "source:person_source_id_if_assembled_multispine", + "source_year:source_household_id:source_person_id_if_complete", + "else_support:person_support_source_id", + "else_person:person_id", + ) + vectors = ( (0, "snap_take_up", "2024:10:2", 9_193_979_365_434_741_258), (0, "immigration:ead_workers", "2024:2", 3_846_788_339_087_460_008), diff --git a/tools/spec_engine_coverage.py b/tools/spec_engine_coverage.py index 7a7a1786b..5bd0d07d3 100644 --- a/tools/spec_engine_coverage.py +++ b/tools/spec_engine_coverage.py @@ -41,7 +41,7 @@ REPORT_SCHEMA_VERSION = 3 EXPECTED_POINTER_INVENTORY_SHA256 = ( - "bc4a948ab632191954600da8474c5b011f977a65e24c399d126f3dc4a79f23e5" + "ae7881c07c6137037f43a5992b3e8fffbbdbd5ddf2e1b70956986cb9df043f71" ) DEFAULT_REPORT_PATH = ( Path(__file__).resolve().parents[1] From 12a918ed32e54829508395d12e73fdee35d0ddc7 Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Thu, 27 Aug 2026 11:06:41 +0200 Subject: [PATCH 26/49] Repin multispine live spec fixture --- PROGRESS.md | 25 ++++++++++++------- .../tests/test_us_multispine_pool_tool.py | 2 +- 2 files changed, 17 insertions(+), 10 deletions(-) diff --git a/PROGRESS.md b/PROGRESS.md index d277fb22d..eb715841d 100644 --- a/PROGRESS.md +++ b/PROGRESS.md @@ -9,9 +9,10 @@ The source codec, weeks-gate architecture, source-scope, clone-layout, and stable-identity repairs are implemented and focused-tested. The complete release-call roster is classified; six archived-model input assumptions require owner rulings and are deliberately reported instead of guessed. Repository-wide -Ruff, the CI inventory verifier, and four of five full pytest shards pass; the -build shard is pending restart after its fail-closed spec-engine receipts were -coherently re-pinned. +Ruff, the CI inventory verifier, and four of five full pytest shards pass. The +first post-repin build-shard run reached 100% with one stale live-spec SHA +expectation and no other failures; that expectation is corrected, its complete +test file is running, and a clean full-shard rerun follows. No network access, artifact build, publication, push, pool build, or release build is in scope. @@ -65,8 +66,9 @@ build is in scope. semantics suites pass. - Passed the complete ACS-transfer test file (65 tests), the complete post-transfer calibration receipt-contract file (47 tests), focused Ruff, - and `git diff --check` using the prebuilt `.venv` directly. `uv run --no-sync` - is unavailable because this sandbox forbids uv's read of `~/.cache/uv`. + and `git diff --check` using the prebuilt `.venv` directly. A task-local + `UV_CACHE_DIR` later allowed the required `uv run --no-sync` commands to run + against that same prebuilt environment without accessing `~/.cache/uv`. - Modernized the weeks summary/gate to derive its roster from actual assembled source channels, while retaining the legacy ASEC/PUF role path. The ASEC source-validity scope, direct native-clone reconciliation scope, and reviewed @@ -143,8 +145,8 @@ build is in scope. items: SSI disability criteria, SCF wealth, SCF auto loans, SIPP vehicles, SIPP tips, and ORG wages/FLSA all consume ASEC-only archived predictors on a frame whose 1,736,840 physical ACS rows carry null source cells. ORG is - guaranteed to fail its unchanged race/occupation bands; SIPP tips can - silently pass its global band while the ACS channel is dead. + guaranteed to fail its unchanged race/occupation bands; SIPP tips' unchanged + tipped-occupation band passes while the ACS channel is dead. - Passed repository-wide Ruff and the CI test-group inventory verifier. The calibrate, data, fit, and frame shards pass in four independent pytest processes. @@ -164,11 +166,16 @@ build is in scope. protocol changes rather than environment drift. All 102 tests in the eight directly affected spec-engine files pass; generated-bundle and coverage- report byte checks, focused Ruff, and `git diff --check` also pass. +- Ran the full build shard after that coherent spec repin. It reached 100% with + exactly one failure and no errors: the multispine constants-adapter fixture + still expected the former live US spec SHA. Updated only that live-binding + expectation to the regenerated final SHA; the separate arbitrary checkpoint + identity fixture remains deliberately unchanged. ## Next -- Restart and finish the full build-shard pytest process after the committed - spec-engine repin. +- Finish the complete multispine-tool test file, commit its live-spec fixture + repin, then rerun the full build-shard pytest process from zero. - Write the final provenance, audit evidence, verification receipts, and judgment calls to `out.md`, then leave a clean committed worktree. diff --git a/packages/microcosm-build/tests/test_us_multispine_pool_tool.py b/packages/microcosm-build/tests/test_us_multispine_pool_tool.py index d17303b3b..473daf364 100644 --- a/packages/microcosm-build/tests/test_us_multispine_pool_tool.py +++ b/packages/microcosm-build/tests/test_us_multispine_pool_tool.py @@ -2465,7 +2465,7 @@ def capture_equality(expected: object, actual: object) -> None: "country": "us", "schema_id": "country_spec", "schema_version": 1, - "spec_sha256": "5378bb9189aec96f50da22aac71e5bd2c3d919e9795f6ef2147e0bc9c739dd8e", + "spec_sha256": "5f44d96d45e9aabcea2d565ef063d68bfc0652df1b38b08aa31ce6896d15f371", }, } From 606cbd699dd048fe3439106018fbd4c770e75b8e Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Thu, 27 Aug 2026 12:13:51 +0200 Subject: [PATCH 27/49] Finalize weeksgate verification report --- PROGRESS.md | 26 +- out.md | 757 +++++++++++++++++++++++++++++++++++++--------------- 2 files changed, 559 insertions(+), 224 deletions(-) diff --git a/PROGRESS.md b/PROGRESS.md index eb715841d..d87e1c806 100644 --- a/PROGRESS.md +++ b/PROGRESS.md @@ -2,17 +2,16 @@ ## State -In progress on 2026-08-27. Real-pool provenance has refuted the proposed +Complete on 2026-08-27. Real-pool provenance has refuted the proposed post-transfer amount-mapping mechanism: every fractional week is an ACS-origin non-native clone prediction outside the calibration's clone-0 recipient scope. The source codec, weeks-gate architecture, source-scope, clone-layout, and stable-identity repairs are implemented and focused-tested. The complete release-call roster is classified; six archived-model input assumptions require owner rulings and are deliberately reported instead of guessed. Repository-wide -Ruff, the CI inventory verifier, and four of five full pytest shards pass. The -first post-repin build-shard run reached 100% with one stale live-spec SHA -expectation and no other failures; that expectation is corrected, its complete -test file is running, and a clean full-shard rerun follows. +Ruff, the CI inventory verifier, and all five full pytest shards pass in their +required independent processes. The completed provenance, audit, verification, +judgment calls, and host-owned checkpoint-rerun consequence are in `out.md`. No network access, artifact build, publication, push, pool build, or release build is in scope. @@ -171,13 +170,22 @@ build is in scope. still expected the former live US spec SHA. Updated only that live-binding expectation to the regenerated final SHA; the separate arbitrary checkpoint identity fixture remains deliberately unchanged. +- Passed the complete multispine-pool-tool file after that correction and + committed the coherent fixture repin as `12a918ed`. +- Reran the entire build shard from zero in one process: 6,608 tests collected, + 100% reached, and pytest exited 0 with expected skips only. All five full + package shards, repository-wide Ruff, generated-artifact checks, the spec + coverage proof, the CI test inventory, and `git diff --check` are green. +- Wrote the final provenance tables, mechanism verdict, per-file rationale, + exhaustive release-gate audit, owner-ruling list, verification evidence, and + judgment calls to `out.md`. ## Next -- Finish the complete multispine-tool test file, commit its live-spec fixture - repin, then rerun the full build-shard pytest process from zero. -- Write the final provenance, audit evidence, verification receipts, and - judgment calls to `out.md`, then leave a clean committed worktree. +- Host session: rerun `late_transfer -> simulated -> terminal-gates` from the + candidate checkpoints because the discrete weeks codec changes pool content. +- Review the six archived-model owner rulings in `out.md`; do not reinterpret + their missing ACS predictors through a gate-only threshold/scope change. # Historical: gate-failed base-pool release lane diff --git a/out.md b/out.md index df65be158..6b9d8437a 100644 --- a/out.md +++ b/out.md @@ -1,233 +1,560 @@ -# Gate-failed base-pool release lane: final report +# Weeksgate: stacked release gates and fractional-weeks final report -Date: 2026-08-26 +Date: 2026-08-27 -Branch: `release-from-gate-failed-pool` +Branch: `stacked-release-gate-alignment` -Base: `origin/main` at `2263df36` +Lane base: `4f453746` (`origin/main` at kickoff; includes #786) ## Outcome -The legacy `--base-h5` release and preflight paths now fail closed when the H5 -identifies as a US multispine pool. They require and authenticate the canonical -sibling pool manifest, bind it to the exact requested H5, and reject a red -terminal agreement battery by default. The new explicit opt-in is: - -`--allow-gate-failed-base-pool` - -The flag is valid only with `--base-h5` and only for an authenticated current -stacked pool with `status=gate_failed` and `simulation_ready=false`. It is -rejected for generic H5 inputs, green pools, and the exact-k manifest arm. A -release built with the flag carries the full authenticated red verdict so a -reader can see `battery: red, N failures` without fetching the pool. - -Publication preflight authenticates the same pool and displays the red receipt -as prominent human-review evidence. The red battery alone does not change the -existing preflight PASS/AT_RISK/FAIL calculation or exit code. Publication -remains a separate human-gated operation. - -No network access, pool build, release build, publication, push, battery -threshold change, or gate-logic change was performed. - -## Diff summary and rationale - -- `CLAUDE.md`: documents the authenticated `--base-h5` boundary, explicit red - opt-in, verdict carriage, strict exact-k arm, and separate publication step. -- `PROGRESS.md`: maintains the required state/done/next journal from kickoff - through the final verified handoff while preserving the prior lane history. -- `changelog.d/gate-failed-pool-release.fixed.md`: records the closed receipt - bypass and explicit opt-in under the repository changelog convention. -- `packages/microcosm-build/src/microcosm/build/us_runtime/h5_io.py`: adds the - shared pool classifier, exact-H5 binding check, release-specific authenticated - loader, and normalized self-contained release receipt. The existing strict - simulation-ready and scoring loader contracts were not weakened. -- `packages/microcosm-build/src/microcosm/build/us_runtime/release_gate_preflight.py`: - authenticates identified base pools through the shared seam, records their - receipt in `PreflightReport`, and renders authenticated red evidence before - the ordinary check table without including it in exit-code calculation. -- `tools/build_us_fiscal_refresh_release.py`: adds - `--allow-gate-failed-base-pool`, authenticates pool-like legacy base H5s before - the generic loader can run, uses the authenticated frame/H5 identity, and - passes the receipt into both generated manifests. The exact-k arm remains - simulation-ready-only. -- `tools/preflight_us_release_gates.py`: adds the same explicit opt-in and an - optional `--release-manifest`; validates the carried full verdict, requires - its receipt to exactly equal the pool authenticated from `--base-h5`, emits a - large red/human-review banner, and adds machine-readable carried evidence. - Existing required `--base-h5` and `--selection-source-manifest` arguments and - all other preflight behavior remain intact. -- `packages/microcosm-build/tests/test_us_multispine_pool_h5_io.py`: exercises - real stamped H5 classification, strict/default refusal, explicit red - acceptance, redundant-green refusal, receipt contents, identity binding, and - malformed aggregate/nested verdict refusal. PyTables-dependent cases use the - sibling-test `importorskip` idiom. -- `packages/microcosm-build/tests/test_us_fiscal_refresh_builder.py`: covers the - old bare-H5 bypass directly, red-manifest default refusal, flag routing, - generic/non-pool refusal when the flag is present, both-manifest carriage, - exact-k isolation, and AST-level ordinary base-pool wiring guards. -- `packages/microcosm-build/tests/test_us_release_gate_preflight.py`: covers - bare/missing/red receipt authentication, prominent text and JSON carriage, - nonblocking PASS/AT_RISK/FAIL exit semantics, required static inputs, and - exact mismatch refusal across manifest/H5/publication/gates/full-verdict - identities. -- `out.md`: this final implementation, audit, verification, and handoff report. - -## Flag and manifest contract - -The exact flag is `--allow-gate-failed-base-pool`. The release builder permits -it only on its legacy `--base-h5` arm. Publication preflight uses the same flag -to authenticate and inspect the same red pool; neither invocation is an -authorization to publish. - -The build manifest stores the receipt at top-level `base_pool`. The release -manifest stores the same receipt at `build.base_pool`: - -- `artifact_kind` -- `status` -- `simulation_ready` -- `manifest_sha256` -- `publication_run_id` -- `pool_h5_sha256` -- `pool_h5_size_bytes` -- `allow_gate_failed_base_pool` -- `agreement_gate_reference.battery_status` -- `agreement_gate_reference.passed` -- `agreement_gate_reference.gates_json_sha256` -- `agreement_gate_reference.failure_count` -- `agreement_gate_reference.failures`, a flattened list of `{gate, message}` -- `agreement_gate_reference.verdict`, the complete agreement-gate verdict - -The receipt accepts only coherent pairs: simulation-ready/green/passed or -gate-failed/red/failed. Every nested gate must carry a boolean `passed` and a -coherent failure list; the nested aggregate must match the terminal status. -For a red pool the failure list must be nonempty. SHA-256, publication run ID, -H5 size, full verdict, and all other receipt fields are bound through exact -preflight equality. - -The preflight report also exposes the authenticated receipt as `base_pool`. -When `--release-manifest` supplies a red carried receipt, JSON adds -`carried_base_pool_agreement_battery`, including `battery_status=red`, the -failure count/list, gates digest, full agreement reference, -`publication_decision=human_review_required`, and `affects_exit_code=false`. - -## §4 consumer audit - -The requested exhaustive source audit used -`simulation_ready|gate_failed|load_simulation_ready` across `tools/` and -`packages/`, followed by source tracing of every non-test match. - -- `h5_io.py` owns the authentication boundary. The new release wrapper selects - the unchanged strict loader by default and the existing private terminal - loader only when its required explicit boolean is true. The public scoring - and strict simulation-ready contracts remain unchanged. -- `tools/build_us_exact_k_ladder_release.py` remains deliberately strict: it - uses the simulation-ready loader, validates config/release identities, checks - `agreement_gate.passed`, and never forwards the new flag. -- `tools/score_us_release_head_to_head.py` retains its existing authenticated - terminal-evidence exception. It requires a pool manifest and terminal gates; - it cannot authorize a release or accept a naked pool H5. -- `tools/build_us_multispine_pool.py` and - `us_runtime/multispine_pool.py` are producers of the status, readiness, H5 - stamp, manifest, diagnostics, and terminal verdict rather than downstream - release consumers. -- `tools/_legacy/build_us_acs_multispine_base.py` writes its own pre-calibration - `simulation_ready=false` state; it is not a current terminal-pool consumer. -- `tools/build_us_acs_local_release.py` uses its own calibrated-release - readiness summary. A derivative keeps donor release/revision provenance but - does not project the nested `build.base_pool` verdict into its own manifest. -- The microcosm-data release contract, loader, TRACE conversion, and publisher - tolerate but otherwise ignore the additive `build.base_pool` object. The - designated publication preflight is therefore the prominent human-facing - red-verdict surface. - -Report-only interactions left unchanged, as directed: - -1. `tools/build_us_multispine_pool.py::_stacked_manifest_payload` still labels - `calibration.consumer` as `k-ladder` and says - `requires_manifest_simulation_ready=true`. That metadata is now incomplete - for the explicit legacy base-H5 red-pool route, but it is informational and - unenforced. -2. A red pool producer writes its H5, manifest, gates evidence, and failed - Logbook row, then returns status 1. Existing `set -e` candidate chains stop - there, so an operator must deliberately start the separate release command - to use the new opt-in. -3. ACS-local derivative manifests retain fetchable donor provenance but do not - self-contain the nested red receipt. -4. Generic data/release consumers accept the additive receipt but do not - surface it. This is not a publication bypass because the publication tool's - designated preflight now authenticates and displays it. - -No audit-only consumer was modified. +The 369 fractional `weeks_unemployed` values are genuine pool-content +defects, but the proposed post-transfer-calibration mechanism is refuted. They +were emitted earlier by the ACS-transfer QRF because PolicyEngine-US physically +types this integer-supported variable as `float`. The calibration maps ACS +clone 0 onto actual observed ASEC support and removes all 342 fractions in that +scope; it never touches the 360 ACS clone-1 and 9 ACS clone-2 fractions that +survive into the pool. + +The source fix registers `weeks_unemployed` as a discrete numeric ACS-transfer +target. Predictions now snap deterministically to actual donor support, with +lower-support tie breaking, and are returned on integer support. The transfer +execution contract and generated imputation authority receipt the policy. +The strict post-transfer calibration receipt and validator remain unchanged +because that kernel did not cause the defect; their complete contract suite +still passes. + +The weeks release gate now distinguishes physical source channel from +clone-operator role. It derives an assembled roster (`asec+acs`) or retains a +legacy roster (`asec+puf_tax_detail`), validates raw `LKWEEKS` only where an +ASEC source exists, reconciles direct carries only on native ASEC rows, and +checks the UC rule only on rows owned by that constraint. Every plausibility +band and numeric threshold is unchanged. + +The full release-side sweep fixed all unambiguous physical-source, +clone-layout, and stable-identity archaeology. Six archived-model predictor +contracts remain owner decisions and are reported precisely below. In +particular, ORG wages/FLSA is guaranteed to fail on the stacked pool, while +SIPP tips' global tipped-occupation component passes despite a dead ACS +channel. Those are not safe gate-only edits. + +No network access, pool build, release build, artifact publication, push, +plausibility-band change, or by-origin battery change was performed. Issue +#782's weeks-incidence-band adjudication was not touched. + +## Task 1: provenance of all 369 noninteger weeks + +### Evidence read + +The analysis read the supplied fixed-format pool directly: + +`/Users/maxghenis/PolicyEngine/_buildo-runtime/out/candidate-25/pool/pool.h5` + +It contains 1,970,973 person rows: 234,133 physical ASEC rows and 1,736,840 +physical ACS rows. It also read the exact pre-calibration late-transfer target +bank: + +`/Users/maxghenis/PolicyEngine/_buildo-runtime/out/candidate-25/pool/checkpoints/stacked/3847d30e1488ae18891780a7a5de6a1b75d1329f2491796e35639fc9e802c26e/acs-transfer/0914d1b858dff5171f7743f0e535349be4459fb99b3386bbc2e3be72d40dd5fc/late_producer_dag/person/source_operator_weeks_unemployed/targets/000__weeks_unemployed.h5` + +The raw-draw value SHA-256 is +`6ea20782bcb53fa730bb7e7045c7d79ab096dfe69bc5ceb540158d93f6672db4`. +The final fractional-row-index SHA-256 is +`3431befe89f007c353296b0c59792bc31e45a1031e62a6f9c99e003df286305f`; +the sorted 369-value SHA-256 is +`314ba4459bb57169eaf492dd28c8ea04af460104b962bb29285104e70eee976c`. + +### Counts by physical channel, clone, and UC + +| Physical channel | Clone index | All rows | Fractional, UC=0 | Fractional, UC>0 | +|---|---:|---:|---:|---:| +| ASEC | 0 | 108,073 | 0 | 0 | +| ASEC | 1 | 108,073 | 0 | 0 | +| ASEC | 2 | 17,987 | 0 | 0 | +| ACS | 0 | 856,626 | 0 | 0 | +| ACS | 1 | 856,626 | 355 | 5 | +| ACS | 2 | 23,588 | 9 | 0 | +| **Total** | | **1,970,973** | **364** | **5** | + +All 369 values are positive, have null ASEC `LKWEEKS`, and are distinct at +their exact IEEE-754 representation. Every fractional row maps one-to-one by +`person_source_id` to a unique ACS clone-0 sibling, and +`unemployment_compensation` is identical across that sibling group. Of the +364 UC-zero rows, the clone-0 sibling has zero weeks. The five UC-positive +rows have clone-0 integer weeks in `{2, 4, 40, 48, 50}`. + +The five fractional value / annual-UC pairs are: + +| Fractional weeks | Unemployment compensation | +|---:|---:| +| 3.3297787140375217 | 10,800 | +| 3.948788299342264 | 20,000 | +| 17.23699569220113 | 7,200 | +| 18.64447048256536 | 10,800 | +| 21.97077954067273 | 4,400 | + +### Value distribution + +| Statistic | Value | +|---|---:| +| Distinct values | 369 | +| Minimum | 1.0003521955067698 | +| p10 | 2.306610095577396 | +| p25 | 6.226837695984924 | +| Median | 12.319331262236448 | +| Mean | 14.643462963265243 | +| p75 | 23.618846787450813 | +| p90 | 27.706623369972394 | +| Maximum | 37.796501228614694 | + +| Weeks interval | All | ACS clone 1, UC=0 | ACS clone 1, UC>0 | ACS clone 2, UC=0 | +|---|---:|---:|---:|---:| +| [1, 5) | 74 | 70 | 2 | 2 | +| [5, 10) | 85 | 84 | 0 | 1 | +| [10, 15) | 45 | 44 | 0 | 1 | +| [15, 20) | 40 | 36 | 2 | 2 | +| [20, 25) | 60 | 58 | 1 | 1 | +| [25, 30) | 39 | 39 | 0 | 0 | +| [30, 35) | 16 | 15 | 0 | 1 | +| [35, 40) | 10 | 9 | 0 | 1 | + +The nine clone-2 values are +`1.3112775468533615`, `3.155651627690991`, +`8.163243626548894`, `13.03279351472897`, +`15.40528737169489`, `15.477036717133474`, +`24.60208367015022`, `31.658750512333974`, and +`37.10384792802646`. + +### Mechanism verdict: calibration hypothesis refuted + +The late-transfer target bank contains 711 fractional predictions: + +| ACS clone | Fractional before calibration | Fractional in final pool | Rows changed by calibration | +|---:|---:|---:|---:| +| 0 | 342 | 0 | 13,417 | +| 1 | 360 | 360 | 0 | +| 2 | 9 | 9 | 0 | + +Every surviving fractional bit is identical between the target bank and final +pool. The stacked calibration explicitly defines ASEC clone 0 as reference and +ACS clone 0 as recipient in +`packages/microcosm-build/src/microcosm/build/us_runtime/stacked_spine.py:8975-8985`. +Its amount mapping selects values from the sorted observed donor array rather +than interpolating in +`packages/microcosm-build/src/microcosm/build/us_runtime/post_transfer_calibration.py:577-680`. + +The pool receipt confirms the calibration ran: + +- 8,419 allowed-carrier/addition-candidate rows within 856,626 mutable ACS + clone-0 rows; +- 13,417 changed clone-0 rows, including 4,998 cleared and 8,290 added; +- `capacity_limited=true`; +- 8,419 mapped positive amounts; +- reference quantiles `[2, 6, 12, 26, 36]`; +- recipient quantiles `[2, 8, 22, 24, 32]` before and + `[2, 6, 12, 26, 36]` after; +- amount QED `0.5882352941176471 -> 0.0`; +- zero donor-support violations. + +Weeks' declared post-transfer calibration spec is at +`packages/microcosm-build/src/microcosm/build/us_runtime/post_transfer_calibration.py:237-263`. +The actual cause precedes calibration. Before this branch, +`_target_encoding` treated numeric `weeks_unemployed` as continuous because +the engine metadata says `float`; a quantile-regression-forest prediction may +therefore interpolate between integer observations. The relevant encoding and +decoding seam is +`packages/microcosm-build/src/microcosm/build/us_runtime/acs_transfer.py:3037-3120` +and `:3240-3270`. Calibration then repaired only clone 0 by design. + +### Why the gate said “5,218 PUF rows” + +There is no physical PUF channel in this pool. The compatibility helper +`support_role_series` deliberately labels clone 0 as the ASEC-compatible +operator role and every clone above zero as the PUF-tax-detail operator role, +independent of physical source. See +`packages/microcosm-build/src/microcosm/build/us_runtime/support_provenance.py:390-489`. +The old summary confused that operator role with physical channel. + +The 5,218 rows are all ACS-origin, UC-zero, positive-week non-native clones: + +| Actual rows | Integer weeks | Fractional weeks | Total | +|---|---:|---:|---:| +| ACS clone 1 | 4,733 | 355 | 5,088 | +| ACS clone 2 | 121 | 9 | 130 | +| **Total** | **4,854** | **364** | **5,218** | + +The UC-zero constraint is not defined for those ACS non-native QRF clones. +The updated gate preserves the compatibility role for operator plausibility +but uses physical source and clone provenance to own source/UC checks. + +## Task 2: source fix and receipt contract + +`weeks_unemployed` is now in `_DISCRETE_NUMERIC_TARGETS` at +`packages/microcosm-build/src/microcosm/build/us_runtime/acs_transfer.py:133-141`. +The existing discrete codec: + +1. rejects nonintegral donor support; +2. snaps finite predictions to the nearest value in actual observed support; +3. resolves exact-distance ties to the lower support value; +4. emits integer/nullable-integer output. + +The snap is at +`packages/microcosm-build/src/microcosm/build/us_runtime/acs_transfer.py:3327-3339`. +The sorted discrete-target roster is part of the transfer execution identity at +`:271`; the generated authority records it at +`packages/microcosm-build/src/microcosm/build/us/spec/imputation.yaml:408-411`. +The final regenerated bundle spec SHA-256 is +`5f44d96d45e9aabcea2d565ef063d68bfc0652df1b38b08aa31ce6896d15f371`. +This invalidates affected target-bank identities instead of silently changing +content under an old receipt. + +The release-gate and WIC edits also change exact source bytes attested by the +shared legacy seed protocol. Its reviewed WIC grammar now explicitly records +assembled `person_source_id` before the unchanged legacy fallbacks at +`packages/microcosm-build/src/microcosm/build/spec_engine/seeds.py:720-737`. +The fail-closed spec proof was coherently re-pinned to 42,122/42,122 claimed +fields (32,352 authored and 9,770 resolved) and 41/41 inventory checks. The +committed coverage report, producer-semantics digest, seed protocol/map, +checkpoint identity, pointer inventory, and affected country-envelope goldens +all move together. A detached `origin/main` comparison confirmed that the +BE/UK/minimal-envelope changes come from this branch's attested source and seed +grammar, not environment drift. + +The regression at +`packages/microcosm-build/tests/test_us_acs_transfer.py:1391-1430` sends a +fraction-producing mean QRF through the real target codec, verifies that output +lies on observed integer donor support, and verifies that the execution +contract declares `weeks_unemployed`. + +Because the hypothesis was refuted, no change was made to +`post_transfer_calibration.py` or its strict validator. Its observed-support +amount mapping, before/after margin fidelity, hashes, anchors, capacity status, +and donor-support validation remain mandatory; the complete 47-test receipt +contract passes. + +**Artifact consequence:** pool content changes. The host session must rerun +`late_transfer -> simulated -> terminal-gates` from checkpoints for the +candidate pool. This lane did not run any pool or release build. + +## Task 3: stacked-aware weeks release gate + +The gate now uses the narrow, read-only physical-source accessor +`support_gate_source_channel_series` at +`packages/microcosm-build/src/microcosm/build/us_runtime/support_provenance.py:496-524`. +That accessor is restricted to reviewed gates/reporters by the static +source-blindness contract. + +| Frame layout | Plausibility roster | Raw LKWEEKS validity | Exact LKWEEKS carry | UC-zero consistency | +|---|---|---|---|---| +| No provenance | ASEC | all rows | all rows | none, matching legacy behavior | +| Legacy ASEC+PUF roles | ASEC + PUF | ASEC role | ASEC role | PUF role | +| Assembled ASEC+ACS | actual physical channels | every physical ASEC clone | physical ASEC clone 0 | physical ASEC non-native clones + non-ASEC clone 0 | + +The assembled scope logic is at +`packages/microcosm-build/src/microcosm/build/us_runtime/weeks_unemployed.py:1320-1380`; +summary and failure construction are at `:1204-1317` and `:1383-1435`. +The ASEC plausibility bands and the legacy recipient/PUF bands remain +byte-identical, and every non-ASEC physical channel uses the unchanged +recipient band. + +A read-only replay on the supplied pool reports: + +- roster `asec, acs`; +- 234,133 raw ASEC source rows, zero invalid; +- 108,073 direct native reconciliation rows, zero mismatches; +- 982,686 UC-constrained rows, zero mismatches; +- ASEC positive share `0.01956717285126748`, weighted mean + `0.32416674108944377`, weighted weeks `52,743,981.18128455`; +- ACS positive share `0.00800370929739283`, weighted mean + `0.1376988711272782`, weighted weeks `21,987,101.082573153`; +- the unchanged channel bands pass; +- the gate fails only on the genuine 369 noninteger values. + +Stacked ASEC+ACS and legacy ASEC+PUF fixtures cover roster derivation, raw +source scoping, native reconciliation, and UC ownership. The complete focused +weeks file passed: 26 passed, 1 skipped. + +## Task 4: complete release-side archaeology audit + +The audit started from every release call in +`tools/build_us_fiscal_refresh_release.py:8690-11726`, traced each gate and +its immediately preceding producer/wrapper, and classified physical-source, +operator-role, clone-layout, entity-layout, and raw-column assumptions. The +table groups gates only where they share the same architectural conclusion; +every release gate in the roster is named. + +| Gate/stage | Stacked assumption found | Disposition | +|---|---|---| +| Validation input coverage; register consistency; release target parity; target-profile coverage; base-population scale | Manifest, registry, target, or aggregate-weight contracts; no legacy per-row source roster | Already compatible; no change | +| PUF capital-gains-tail presence and post-selection preservation | Authenticated tail-column presence/preservation checks, not a demand for a physical PUF channel (`build_us_fiscal_refresh_release.py:8837-8841,8937-8941`) | Already compatible; no change | +| Exact-k PUF tail support; exact-k frozen-register fit; fiscal-target materialization/skip/zero-support; critical/SOI fit | Calibrated diagnostics and authenticated exact-k inputs, not physical source labels | Already compatible; no change | +| Weeks input and post-selection weeks input | Used clone operator roles as physical ASEC/PUF channels and applied raw/UC checks to the wrong rows | **Fixed** with physical roster and separate validity/reconciliation/UC scopes | +| QBI input | Already has an explicit stacked ACS path | Already compatible; no change | +| Workers' compensation; alimony; Medicare; retirement contributions; retirement distributions | Raw ASEC-only columns were validated/reconciled through legacy operator roles | **Fixed**: all physical ASEC clones own raw validity; native physical ASEC rows own exact direct-carry reconciliation where applicable | +| SSI reporter capture/take-up | Strict `SSI_VAL` validation treated null ACS raw-source cells as invalid | **Fixed**: capture reporters from physical ASEC; assignment remains source-blind and consumes the captured ID set | +| SIPP Head Start; voluntary filing | Assumed exactly an occurrence pair / two clones | **Fixed**: explicit assembly clone index, arbitrary clone count, duplicate source+clone refusal, clone-0-or-lowest canonical decision, fanout to every clone | +| Prior-year income clone diagnostic | Compared only the first two occurrences | **Fixed**: group all assembled clones; existing availability band unchanged | +| SSI disability clone-divergence diagnostic | Compared only a pair and missed clone-2-only divergence | **Fixed** for all assembled clones; whether divergence becomes fatal still requires owner ruling | +| WIC deterministic draw | Raw source-local identity could collide across physical ASEC and ACS | **Fixed**: assembled frames prefer unique `person_source_id`; clones remain draw-stable and legacy precedence is unchanged | +| Farm business; domestic production; child support; disability benefits; educator expenses; Form 4952; SALT refund; capital-gain details; energy subsidy; housing; other health insurance | These use clone-operator roles for reviewed producer/plausibility semantics, not as claims about physical source | Intentionally unchanged; stacked-compatible role diagnostics | +| Childcare; casualty loss; miscellaneous itemized; immigration; generic take-up; hours; SNAP take-up; relationship; eligibility; education; pregnancy; reported-coverage vintage; SNAP discretionary exemption | Output/input signal checks do not assume a physical PUF channel or require raw ASEC cells on all rows | Already compatible; no change | +| Local health input; Medicaid take-up; SNAP state take-up; SSI final/delivery | Output or diagnostics-based checks | Already compatible after SSI reporter fix | +| Input-mass reference; degenerate input; eCPS parity; release input coverage; export input mass; QRF tail concentration; reform-coverage smoke; source coverage | Export-wide schema/mass/tail/simulation contracts, not legacy source rosters | Already compatible; no change | +| Export count-calibrated take-up staleness | Export consistency against recorded count-calibration receipts (`build_us_fiscal_refresh_release.py:11702-11726`), independent of physical source roster | Already compatible; no change | +| SCF wealth | Archived predictors silently map ACS-null CPS race/Hispanic source cells to `Other` | **Owner ruling required**, detailed below | +| SSI disability criteria | Archived receiver strictly requires six ASEC disability predictors and `SSI_VAL` on every row | **Owner ruling required**, detailed below | +| SCF auto loans | Imports the SCF wealth CPS-race mapping, silently mapping ACS recipients to `Other` | **Owner ruling required**, detailed below | +| SIPP vehicles | Missing ACS `SPM_TENMORTSTATUS` defaults every ACS recipient to tenant code 3 | **Owner ruling required**, detailed below | +| SIPP tips | Missing ACS `PEIOOCC` defaults to non-tipped; global band can conceal a dead ACS channel | **Owner ruling required**, detailed below | +| ORG wages/FLSA | Missing ACS race/ethnicity/occupation defaults to zero; unchanged global gates are guaranteed to fail | **Owner ruling required**, detailed below | + +### Unambiguous repairs + +- `support_provenance.py:331-354,496-524` centralizes detection of assembled + metadata and physical channels. General derive/imputation code cannot call + the physical accessor: `test_us_spine_blindness.py` pins the exact reviewed + call graph. +- `alimony.py:282-327` and `workers_compensation.py:560-604` split physical + source validity from direct-carry reconciliation while preserving all role + bands. +- `medicare_take_up.py:134-143,248-270`, + `retirement_contributions.py:172-199,602-628`, and + `retirement_distributions.py:274-301,709-739` scope release diagnostics + without routing their population operators by physical origin. +- `ssi_take_up.py:515-550` captures reporters from physical ASEC and accepts + ACS nulls; the assignment consumes captured lineage rather than reading + origin. +- `sipp_head_start.py:579-621` and + `voluntary_filing.py:749-783` use explicit clone indices on assembled + frames. `prior_year_income.py:744-765` and + `ssi_disability_criteria.py:1119-1152` inspect all clones. +- `wic_claim.py:337-376` prefers the assembly-unique identity only when + assembled metadata exists. + +### Owner rulings required + +1. **SSI disability criteria.** Release call + `tools/build_us_fiscal_refresh_release.py:9840-9847`. The receiver + requires and strictly evaluates all six `PEDIS*` predictors on every row + at + `packages/microcosm-build/src/microcosm/build/us_runtime/ssi_disability_criteria.py:755-859`, + then strictly evaluates `SSI_VAL` at the same file's `:1023-1034`; the + support-role prediction loop is `:965-1005`. Each input is finite on all + 234,133 physical ASEC rows and null on all 1,736,840 ACS rows, so the stage + fails before its gate. + Required decision: transfer/map canonical ACS disability inputs and + reporter-anchor semantics, or revise/retrain the archived recipient model. + +2. **SCF wealth.** Release call + `tools/build_us_fiscal_refresh_release.py:9798-9808`. + `packages/microcosm-build/src/microcosm/build/us_runtime/scf_wealth.py:654-667` + requires `PRDTRACE`, `PRDTHSP`, `A_MARITL`, `PEPAR1/2`, `PH_SEQ`, and + `A_LINENO`; `:670-691` selects the archived reference person, and + `:694-740` builds race/marriage/children predictors. Only + `PRDTRACE/PRDTHSP` are ACS-null; `:611-620` maps those to zero/`Other`. + The other structural fields are finite and mapped on every ACS row. All + three asset leaves are all-null and `net_worth` is absent, so the wrapper's + recompute branch at `:1109-1159` is certain and every ACS recipient is + silently forced to `Other`. Required decision: authoritative ACS + race/Hispanic mapping or transfer, or a reviewed fallback/retrained model. + +3. **SCF auto loans.** Release call + `tools/build_us_fiscal_refresh_release.py:9990-9996`. + `packages/microcosm-build/src/microcosm/build/us_runtime/scf_auto_loans.py:125-137` + requires the archived layout; `:320-362` selects the reference person and + `:365-417` supplies race/marriage/children inputs. `A_LINENO` is required + by the selector's presence check but is not subsequently used; the mapped + structural fields are finite. The actual defect is the imported SCF-wealth + CPS-race mapping of ACS-null `PRDTRACE/PRDTHSP` to `Other`. All three + auto-loan outputs are absent, so `:475-513` certainly takes the QRF path. + Required decision: the same authoritative mapping or reviewed model change + as SCF wealth. + +4. **SIPP vehicles.** Release call + `tools/build_us_fiscal_refresh_release.py:10026-10033`. + `packages/microcosm-build/src/microcosm/build/us_runtime/sipp_vehicles.py:150-157` + declares the archived layout; `:577-622` reads person + `SPM_TENMORTSTATUS` and fills missing values to tenant code 3, while + `:649-763` uses marriage, `A_LINENO` reference selection, and tenure. + The pool's mapped `A_LINENO` is finite, but person + `SPM_TENMORTSTATUS` is ACS-null and the household field is absent. Both + vehicle outputs are absent, so `:872-917` certainly recomputes and + classifies ACS tenure as tenant. Required decision: canonical ACS tenure + mapping/transfer and whether assembled `A_LINENO` preserves the archived + reference-person semantics, or model revision/retraining. + +5. **SIPP tips.** Release call + `tools/build_us_fiscal_refresh_release.py:10103-10109`. + `packages/microcosm-build/src/microcosm/build/us_runtime/sipp_tips.py:249-263` + maps null `PEIOOCC` to tipped code 0; recipient use is `:363-396`, wrapper + branching is `:450-473`, and the gate is `:484-549`. The pool lacks both + tip output columns, so the wrapper takes its imputation branch. + `PEIOOCC` is ASEC-finite and ACS-null. The ASEC-conditional tipped-code + share is `0.0700204513`; ACS is exactly zero; the global share is + nevertheless `0.0353393976`, inside the unchanged `[0.02, 0.15]` band. + This proves that gate component conceals the dead ACS channel, not that the + separate tip-income component necessarily passes. Required decision: + map/transfer ACS occupation or select a reviewed alternate model, then add + channel-aware diagnostics once those semantics are owned. + +6. **ORG wages/FLSA.** Release call + `tools/build_us_fiscal_refresh_release.py:10144-10150`. + `packages/microcosm-build/src/microcosm/build/us_runtime/org_wages.py:535-559` + fills null `PRDTRACE/PRDTHSP/POCCU2` to zero; these enter features at + `:581-625`; wrapper `:901-945` always recomputes; gate `:959-1034`. All + three raw fields are ASEC-finite and ACS-null. On real weights, CPS-race + nonzero share is `0.5047010838` versus unchanged minimum `0.95`, and + detailed-occupation nonzero share is `0.4155259611` versus minimum `0.65`. + The next release is guaranteed to fail here. Required decision: + authoritative ACS race/ethnicity/occupation mapping or a revised model + contract, not threshold weakening. + +### Read-only real-pool replay after the fixes + +The repaired alimony, Medicare, retirement-contribution, +retirement-distribution, workers'-compensation, and SSI-reporter checks pass. +Their physical ASEC source count is 234,133; direct-carry gates use 108,073 +native ASEC rows where applicable. The weeks gate fails only on 369 fractional +values. + +Two other observed failures are genuine data/spec outcomes, not row-label +archaeology: + +- prior-year-income weighted availability is `0.042839`, outside the + unchanged `[0.05, 0.50]` band; +- WIC finds `is_pregnant=true` on nonfemale rows, with example row positions + `129405, 167076, 171133, 192443, 195546`. + +No threshold was changed to hide either outcome. + +## Diff summary by file + +| File | Rationale | +|---|---| +| `PROGRESS.md` | Maintains the required state/done/next journal from kickoff through verified handoff while preserving historical lanes | +| `changelog.d/stacked-release-gate-alignment.fixed.md` | Records the user-visible integer-support and stacked-gate fix under repository convention | +| `out.md` | This provenance, audit, verification, and handoff report | +| `us/spec/imputation.yaml` | Generated authority now receipts `weeks_unemployed` in the discrete numeric transfer roster | +| `us_runtime/acs_transfer.py` | Declares weeks integer-supported through the existing deterministic observed-support codec | +| `us_runtime/support_provenance.py` | Adds assembled-metadata detection and a narrow read-only physical-source accessor | +| `us_runtime/__init__.py`, `us_runtime/puf_support.py` | Re-export assembled-layout detection while keeping the physical accessor confined to its owner | +| `us_runtime/weeks_unemployed.py` | Derives stacked/legacy rosters and separates source validity, native reconciliation, and UC scopes | +| `us_runtime/alimony.py`, `us_runtime/workers_compensation.py` | Scope raw-source and exact-carry checks to their physical/native owners | +| `us_runtime/medicare_take_up.py`, `us_runtime/retirement_contributions.py`, `us_runtime/retirement_distributions.py` | Make release summaries source-aware while retaining source-blind producer kernels | +| `us_runtime/ssi_take_up.py` | Captures SSI reporters from physical ASEC and permits null ACS source cells without origin-routing assignment | +| `us_runtime/sipp_head_start.py`, `us_runtime/voluntary_filing.py` | Replace pair/occurrence layouts with explicit arbitrary-clone assembled layouts | +| `us_runtime/prior_year_income.py`, `us_runtime/ssi_disability_criteria.py` | Detect divergence across all assembled clones | +| `us_runtime/wic_claim.py` | Uses assembly-unique person identity for cross-origin deterministic draws | +| `spec_engine/seeds.py` | Receipts assembled `person_source_id` as WIC's first seed-key source | +| `spec_engine/field_usage.py`, `spec_engine/inventory_coverage.py`, `tools/spec_engine_coverage.py` | Re-pin the exact two-field expansion and downstream producer/seed/checkpoint/pointer identities | +| `docs/evidence/spec-engine/us-f0-coverage.json` | Regenerates the closed 42,122-field, 41-item coverage attestation | +| `tests/test_us_acs_transfer.py` | Proves fractional QRF output is snapped to observed integer weeks support and receipted | +| `tests/test_us_weeks_unemployed.py` | Covers stacked ASEC+ACS and legacy ASEC+PUF roster/scope contracts | +| `tests/test_us_alimony.py`, `test_us_workers_compensation.py`, `test_us_medicare_take_up.py`, `test_us_retirement_contributions.py`, `test_us_retirement_distributions.py`, `test_us_ssi_take_up.py` | Cover physical ASEC source validity, native reconciliation, legacy behavior, and SSI reporter lineage | +| `tests/test_us_sipp_head_start.py`, `test_us_voluntary_filing.py`, `test_us_prior_year_income.py`, `test_us_ssi_disability_criteria.py` | Cover clone 2+, duplicate clone refusal, deterministic canonical rows/fanout, and all-clone divergence | +| `tests/test_us_wic_claim.py` | Covers cross-origin identity collision avoidance and clone-stable draws | +| `tests/test_us_multispine_puf_clone.py`, `test_us_spine_blindness.py` | Cover assembled provenance and pin physical-source access to the reviewed gate/reporter call graph | +| `tests/test_spec_engine_seeds.py` | Pins WIC's assembled-first seed grammar | +| `tests/test_spec_engine_field_usage.py`, `test_spec_engine_coverage_tool.py`, `test_spec_engine_country_bundles.py`, `test_spec_engine_loader.py` | Pin the reviewed field totals, report identities, and source-attested envelope goldens | +| `tests/test_us_multispine_pool_tool.py` | Re-pins the constants adapter's live US spec identity after regeneration while retaining the separate arbitrary checkpoint identity fixture | + +All source paths in the table are under +`packages/microcosm-build/src/microcosm/build/`; all abbreviated test paths +are under `packages/microcosm-build/tests/`. ## Judgment calls -- Pool detection uses either a canonical sibling manifest whose artifact kind - is the pool kind or the H5's own artifact-metadata stamp. Either positive - identity requires the sidecar; sidecar existence alone is never trusted. -- A dedicated release loader with a required - `allow_terminal_gate_failure` argument keeps the strict and scoring APIs - semantically stable. -- The opt-in is rejected for a green pool rather than silently accepted, so - every recorded flag use has one unambiguous meaning. -- Preflight requires exact equality between the release-carried receipt and the - freshly authenticated base receipt. This prevents displaying release A's red - evidence while checking release B's pool. -- The earlier salvage branch was used as a reference and selected commits were - replayed only after line-by-line review. The final tree corrected its - manifest-only preflight expansion, loader naming/contract ambiguity, - redundant-green behavior, receipt binding, nested-verdict coherence, and an - inaccurate exact-k diagnostic. Full verification ran on the corrected tree. +- The post-transfer calibration was not modified merely because its receipt is + visible near the symptom. Checkpoint bits prove the surviving values predate + it, and its mapping already uses actual donor support. +- The integer contract is explicit by target rather than inferred from the + current PolicyEngine physical dtype. That dtype is `float` and caused the + bug; reviewed domain semantics are the authority. +- Snapping to observed donor support was preferred over generic rounding. It + cannot invent an unsupported week count, is deterministic, and reuses an + existing receipted codec. +- Physical source identity is exposed only to read-only gates/reporters. + Medicare and retirement producer changes considered during review were + reverted: routing population treatment by origin would violate the + source-blind operator boundary. The authenticated release path consumes + already-produced pool surfaces. +- Raw validity covers all physical ASEC clones because they carry the raw + source. Exact reconciliation covers only native ASEC where transferred + non-native clones are intentionally allowed to differ. +- Operator roles remain appropriate for producer/plausibility bands. They are + not aliases for physical ASEC/ACS source, which was the old weeks bug. +- Clone divergence in SSI disability remains diagnostic rather than newly + fatal; changing that policy needs an owner ruling. +- The six archived-model cases were not “fixed” with zero fills, gate + rescoping, or weaker thresholds. Each requires substantive decisions about + ACS predictors and model semantics. ## Verification evidence -All commands ran offline against the pre-built environment with -`uv run --no-sync` (and a task-local UV cache where required). Each pytest +All commands ran offline against the prebuilt environment, with +`UV_CACHE_DIR=/private/tmp/microcosm-weeksgate-uv-cache` so the required +`uv run --no-sync` command could operate inside the sandbox. Each pytest shard ran in one independent process. - `uv run --no-sync ruff check .`: PASS, `All checks passed!` -- `uv run --no-sync pytest -q packages/microcosm-build/tests`: PASS, 6,545 - passed, 45 skipped, 2,351 warnings in 3,326.74 seconds. -- `uv run --no-sync pytest -q packages/microcosm-calibrate/tests`: PASS, 203 - passed, 2 warnings in 11.39 seconds. -- `uv run --no-sync pytest -q packages/microcosm-data/tests`: PASS, 318 passed, - 2 skipped in 9.11 seconds. -- `uv run --no-sync pytest -q packages/microcosm-fit/tests`: PASS, 93 passed, - 1 warning in 30.27 seconds. -- `uv run --no-sync pytest -q packages/microcosm-frame/tests`: PASS, 295 - passed, 36 skipped, 1 warning in 89.30 seconds. -- Full aggregate: 7,454 passed, 83 skipped. -- `.venv/bin/python tools/ci_test_groups.py --verify`: PASS, - `tracked_test_files=309`, `verification=ok`. -- Focused changed-file suites plus exact-k E2E/launcher and data - contract/release/publish-guard regressions: PASS; the full shard runs above - subsequently covered the same tests on the final code tree. -- `git diff --check origin/main...HEAD`: PASS. - -The build-shard warnings are expected numerical, pandas chained-assignment, -PolicyEngine division, and intentionally fragmented-frame test warnings; no -warning is a test failure and none originates in the new receipt path. +- `uv run --no-sync python tools/ci_test_groups.py --verify`: PASS, + `tracked_test_files=309`, `verification=ok` +- Eight directly affected spec/receipt files: PASS, 102 tests +- `tools/generate_us_bundle_from_constants.py --check`: PASS, final US spec + SHA-256 + `5f44d96d45e9aabcea2d565ef063d68bfc0652df1b38b08aa31ce6896d15f371` +- `tools/spec_engine_coverage.py --check`: PASS, 42,122/42,122 fields and + 41/41 inventory checks +- `uv run --no-sync pytest packages/microcosm-calibrate/tests -q`: PASS, + 203 passed +- `uv run --no-sync pytest packages/microcosm-data/tests -q`: PASS, + 318 passed, 2 skipped +- `uv run --no-sync pytest packages/microcosm-fit/tests -q`: PASS, + 93 passed +- `uv run --no-sync pytest packages/microcosm-frame/tests -q`: PASS, + 295 passed, 36 skipped +- `uv run --no-sync pytest packages/microcosm-build/tests -q`: PASS, + 6,608 tests collected, 100% reached, exit code 0, expected skips only +- Complete ACS-transfer file: PASS, 65 tests +- Complete post-transfer calibration receipt-contract file: PASS, 47 tests +- Complete weeks file: PASS, 26 passed, 1 skipped +- Complete alimony/workers'/SSI files: PASS, 28 / 21 / 71 tests +- Medicare + retirement contribution/distribution focused files: PASS, 65 tests +- Repository `git diff --check`: PASS + +The shard warnings observed are existing numerical, sparse-tensor, +joblib core-detection, and PolicyEngine divide warnings; none is a failure and +none originates in the new transfer/source-scope paths. ## Commit inventory -Implementation and journal commits before this final report carrier: - -1. `34375fe6` Document gate-failed release lane kickoff -2. `5be9e49a` Preserve prior progress journal history -3. `c5a24c0f` Record release containment design review -4. `72274910` feat: allow explicit gate-failed pool release inputs -5. `e3d847d6` feat: surface carried red pool verdict in preflight -6. `46811fdb` fix: preserve default exact-k receipt fixtures -7. `dd1ad19a` Add shared multispine base-pool authentication seam -8. `271ee1fe` Close legacy base-H5 pool receipt bypass -9. `92134322` Authenticate and surface gate-failed pools in preflight -10. `cd39f756` Document gate-failed base-pool release boundary -11. `9db4694a` Tighten the authenticated red-pool release opt-in -12. `72c4c7a1` Bind carried pool verdicts to preflight inputs -13. `59c5759d` Record completed release containment work -14. `af20d3f5` Lock ordinary base-pool release wiring in tests - -Commits after the list carry this report, the final `PROGRESS.md` state, and -their handoff cleanup. A report cannot embed the hash of the commit containing -its own final bytes, so `git log --oneline origin/main..HEAD` is the -authoritative complete inventory. At final handoff the branch is 17 commits -ahead of `origin/main` with no uncommitted paths. +Implementation and evidence commits preceding the final report carrier: + +1. `a3331db6` Start stacked release gate alignment journal +2. `9979d101` Record fractional weeks provenance +3. `d7ad753e` Preserve integer support for transferred weeks +4. `0c5b05a8` Receipt integer weeks transfer support +5. `c22a6799` Align weeks gate with stacked source roles +6. `9bbd6dfe` Centralize stacked source channel scopes +7. `49052e6e` Scope alimony source checks to physical ASEC +8. `17ed9a0b` Key WIC draws by stacked person identity +9. `c7dd53ad` Support stacked Head Start clone layouts +10. `3f934688` Scope Medicare source gate to stacked ASEC +11. `5852f4ba` Scope retirement contribution source diagnostics +12. `e40d5db7` Support stacked voluntary filing clones +13. `148881c6` Check all stacked prior-year income clones +14. `52b27f78` Separate alimony source validation and carry scopes +15. `058f27b5` Record stacked clone-layout repairs +16. `97de22e7` Scope workers compensation sources in stacked pools +17. `74e96187` Scope retirement distribution source gates +18. `41007992` Anchor SSI reporters to physical ASEC sources +19. `eb5cd4b6` Keep SSI assignment source blind +20. `3dc5df51` Keep stacked source repairs gate scoped +21. `9b36720b` Record stacked raw-source gate repairs +22. `9520c03f` Report divergence across all SSI disability clones +23. `b8e04e9c` Confine physical source access to release gates +24. `b2de92b9` Document stacked release gate alignment +25. `cd8c55a7` Repin spec engine for integer weeks +26. `12a918ed` Repin multispine live spec fixture + +The final report/journal carrier follows this inventory and is necessarily +self-excluded; the clean-worktree handoff lists it in the final response. From 2aa14e84ce897dd3e8c9eace7d971f8f0dd06643 Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Thu, 27 Aug 2026 12:19:58 +0200 Subject: [PATCH 28/49] Start ACS predictor release join journal --- PROGRESS.md | 34 ++++++++++++++++++++++++++++++++++ 1 file changed, 34 insertions(+) diff --git a/PROGRESS.md b/PROGRESS.md index d87e1c806..40e6e95eb 100644 --- a/PROGRESS.md +++ b/PROGRESS.md @@ -1,3 +1,37 @@ +# ACS predictor release join + +## State + +In progress on 2026-08-27. Implementing the owner-approved release-time join +from stacked-pool ACS person source IDs to the sha-pinned 2024 one-year ACS +person/household CSV zips. Scope is limited to populating the six archived +donor models' CPS-named predictors through reviewed native-ACS crosswalks, +strict join/receipt contracts, unchanged model selection logic, and unchanged +gate thresholds. No network access, pool build, release build, publication, +push, retraining, threshold change, or launcher-contract edit is in scope. + +## Done + +- Read `CLAUDE.md` and the prior weeksgate report's six owner-ruling items with + their release-call and model-consumer evidence. +- Confirmed the requested branch `acs-predictor-release-join` is clean at + `606cbd69`, based on `stacked-release-gate-alignment`. +- Read the GitNexus exploration and impact-analysis workflows. This workspace + exposes neither GitNexus repository resources nor query tools, so the same + call/dependency analysis will be performed directly from source and tests. +- Recorded the required source zips and SHA-256 pins, strict exact/total join + contract, explicit crosswalk and receipt requirements, and verification + boundary. + +## Next + +- Prove the `person_source_id` encoding from assembly source and bind it to + `SERIALNO`/`SPORDER`, including collision and totality invariants. +- Establish each ACS/CPS code semantic from repository evidence; implement + only mappings supported by that evidence and report any unresolved field. +- Add pinned CLI inputs, the release join, receipts, crosswalk/join tests, + changelog fragment, full shard verification, and the final `out.md` report. + # Weeksgate: stacked release gates and integer-week provenance ## State From b7be9c35c581be92aed749f869d770d779e9b17a Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Thu, 27 Aug 2026 12:20:44 +0200 Subject: [PATCH 29/49] Start pregnancy and prior-year defect journal --- PROGRESS.md | 27 +++++++++++++++++++++++++++ 1 file changed, 27 insertions(+) diff --git a/PROGRESS.md b/PROGRESS.md index d87e1c806..cb5e454d3 100644 --- a/PROGRESS.md +++ b/PROGRESS.md @@ -1,3 +1,30 @@ +# Pregnancy / prior-year defect lane (issues #798 and #799) + +## State + +In progress on 2026-08-27 on branch `pregnancy-prioryear-defects`, based on +`stacked-release-gate-alignment` at `606cbd69`. This headless lane is fixing the +nonfemale-pregnancy producer defect and diagnosing the prior-year-availability +shortfall against the supplied 25% pool. It will not change bands, thresholds, +seeds, batteries, or publish/build pool artifacts. + +## Done + +- Read `CLAUDE.md` and the GitNexus debugging workflow. +- Confirmed the assigned branch starts clean at `606cbd69`. +- Recorded the required source fix, structural refusal guard and receipt, + checkpoint-identity review, focused regressions, real-pool decompositions, + changelog, repository Ruff, and independent full-shard pytest boundary. +- Recorded Task 2 as diagnosis-only unless the evidence proves an unambiguous + transfer defect. + +## Next + +- Query the local code graph and trace the pregnancy producer/transfer path. +- Inspect all affected real-pool rows by channel, clone, sex, and age. +- Decompose prior-year availability by physical channel and clone, then compare + physical ASEC alone with the assembled population and intended ACS behavior. + # Weeksgate: stacked release gates and integer-week provenance ## State From 364f45250fac601d3106670f176bdaa75a7a5aaa Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Thu, 27 Aug 2026 12:33:05 +0200 Subject: [PATCH 30/49] Record pregnancy and prior-year root causes --- PROGRESS.md | 32 ++++++++++++++++++++++++++++---- 1 file changed, 28 insertions(+), 4 deletions(-) diff --git a/PROGRESS.md b/PROGRESS.md index cb5e454d3..eb8be6caf 100644 --- a/PROGRESS.md +++ b/PROGRESS.md @@ -17,13 +17,37 @@ seeds, batteries, or publish/build pool artifacts. changelog, repository Ruff, and independent full-shard pytest boundary. - Recorded Task 2 as diagnosis-only unless the evidence proves an unambiguous transfer defect. +- Rebuilt a transient local GitNexus graph offline and queried pregnancy/WIC + execution paths. Registration alone failed because the sandbox forbids the + CLI's global `~/.gitnexus` registry; the usable worktree-local index remains + untracked and will be removed before handoff. +- Inspected all 1,970,973 person rows in the supplied 25% pool. There are 108 + `is_pregnant=true` nonfemale rows, all on physical ACS records: 45 clone 0, + 61 clone 1, and 2 clone 2. ASEC has zero; sex/channel/clone assembly is + internally consistent. +- Localized the pregnancy defect to the ACS QRF path. The ASEC producer hard- + conditions its stable draw on female ages 15--44, while ACS transfer treats + sex and age only as soft predictors, models physical clone rows separately, + and has no pregnancy-domain postcondition. The pool has 11,287 ACS source + people whose clones disagree on pregnancy and zero ASEC disagreements. +- Proved the prior-year shortfall is not ACS dilution or an ACS transfer hole. + Physical ASEC and ACS are both about 4.3% available because assembly samples + each raw ASEC year independently before the adjacent-year `PERIDNUM` join. + Of 18,518 sampled current rows that match the intact full predecessor files, + only 4,724 retain a predecessor after sampled-to-sampled joining: weighted + match survival is 25.4117%, the expected 25% rung effect. Full pooled ASEC + availability is 16.9147%, and selected current rows joined to intact prior + files are 16.9541%. ## Next -- Query the local code graph and trace the pregnancy producer/transfer path. -- Inspect all affected real-pool rows by channel, clone, sex, and age. -- Decompose prior-year availability by physical channel and clone, then compare - physical ASEC alone with the assembled population and intended ACS behavior. +- Implement hard pregnancy eligibility before QRF, one source-person result + fanned across clones, refusing pre-existing/final domain violations and + recording structural counts. +- Bind the pregnancy structural policy into checkpointed transfer execution + identity and add transfer/source/gate regressions. +- Finalize Task 2 as a sampling-order verdict with no band or implementation + change absent a reviewed ownership decision about pre-sampling joins. # Weeksgate: stacked release gates and integer-week provenance From f8b9a75c7bbac8f2dcc31d695abaae9ef35ebe67 Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Thu, 27 Aug 2026 14:09:38 +0200 Subject: [PATCH 31/49] Reschedule CI (Actions events lost for prior head) From 6d3351fd43d4e72967fc5b92c088d381630ed485 Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Thu, 27 Aug 2026 14:13:08 +0200 Subject: [PATCH 32/49] Record prior-year release-gate ruling --- PROGRESS.md | 24 +++++++++++++++++------- 1 file changed, 17 insertions(+), 7 deletions(-) diff --git a/PROGRESS.md b/PROGRESS.md index eb8be6caf..43e3a25cd 100644 --- a/PROGRESS.md +++ b/PROGRESS.md @@ -2,11 +2,13 @@ ## State -In progress on 2026-08-27 on branch `pregnancy-prioryear-defects`, based on -`stacked-release-gate-alignment` at `606cbd69`. This headless lane is fixing the -nonfemale-pregnancy producer defect and diagnosing the prior-year-availability -shortfall against the supplied 25% pool. It will not change bands, thresholds, -seeds, batteries, or publish/build pool artifacts. +Resumed on 2026-08-27 on branch `pregnancy-prioryear-defects`, based on +`stacked-release-gate-alignment` at `606cbd69`. The root-cause findings are +adopted and implementation is in progress for the nonfemale-pregnancy producer +defect and the owner-approved rung-aware prior-year availability release floor. +The authored 0.05 floor, upper bound, all other bands, thresholds, seeds, and +batteries remain unchanged. This lane will not build, publish, or push pool or +release artifacts. ## Done @@ -38,6 +40,11 @@ seeds, batteries, or publish/build pool artifacts. match survival is 25.4117%, the expected 25% rung effect. Full pooled ASEC availability is 16.9147%, and selected current rows joined to intact prior files are 16.9541%. +- Received the owner ruling that assembly-before-join sampling is the accepted + mechanism verdict. The prior-year release gate may scale only its availability + floor by the assembly's sampled-to-sampled match-survival factor, recorded in + or derived from the pool manifest. Rung 1.0 must remain byte-identical to the + existing gate; the authored 0.05 constant and upper bound do not change. ## Next @@ -46,8 +53,11 @@ seeds, batteries, or publish/build pool artifacts. recording structural counts. - Bind the pregnancy structural policy into checkpointed transfer execution identity and add transfer/source/gate regressions. -- Finalize Task 2 as a sampling-order verdict with no band or implementation - change absent a reviewed ownership decision about pre-sampling joins. +- Implement and receipt the ruling-approved sampling-rung scaling of only the + prior-year availability floor, with exact rung-1 compatibility and focused + release-gate regressions. +- Run the charter's repository Ruff and independent full-shard pytest boundary, + then write the complete handoff to `out.md`. # Weeksgate: stacked release gates and integer-week provenance From c1a41ccf830759261029043dd87475cff4163fc7 Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Thu, 27 Aug 2026 14:37:36 +0200 Subject: [PATCH 33/49] Record ACS predictor join contracts --- PROGRESS.md | 49 +++++++++++++++++++++++++++++++++++++++++++------ 1 file changed, 43 insertions(+), 6 deletions(-) diff --git a/PROGRESS.md b/PROGRESS.md index 40e6e95eb..63d15b908 100644 --- a/PROGRESS.md +++ b/PROGRESS.md @@ -22,15 +22,52 @@ push, retraining, threshold change, or launcher-contract edit is in scope. - Recorded the required source zips and SHA-256 pins, strict exact/total join contract, explicit crosswalk and receipt requirements, and verification boundary. +- Proved that `person_source_id` is not a reversible ACS key: ACS people are + sorted by `(SERIALNO, SPORDER)`, receive a zero-based raw spine ID, and then + receive a collision-dependent assembly offset. The pool retains the raw + spine ID, `source_row_id`, `source_person_id`, household `SERIALNO`, and clone + metadata, so the release join will use the retained semantic + `(SERIALNO, integral SPORDER)` key and treat `person_source_id` only as the + one-to-many clone fan-out identity. +- Audited the supplied candidate pool read-only: 856,626 distinct ACS source + people expand to 1,736,840 rows (856,626 clone 0, 856,626 clone 1, and + 23,588 clone 2), with no duplicate `(person_source_id, clone_index)` pair. + Every ACS row agrees with its raw spine/source lineage, and all selected + people match the pinned raw person archive exactly. +- Verified both local archives against the charter pins. The person archive + has 3,422,888 unique `(SERIALNO, SPORDER)` rows and no household orphans; + the household archive has 1,631,969 unique serials, including 1,531,614 + occupied records. Both contain every requested native predictor. +- Established the disability universes from the pinned archive and the + archived repository mapping: DEAR/DEYE are complete at every age; + DREM/DPHY/DDRS are asked from age 5; DOUT from age 15; native code 1 is the + consumer's difficulty bin and code 2 (plus an age-valid universe blank) is + its non-difficulty bin. +- Established the consumed race/Hispanic bins: both SCF models distinguish + White, Black, Asian, Hispanic, and Other; ORG distinguishes Hispanic, + non-Hispanic White, non-Hispanic Black, and Other. `RAC1P`/`HISP` can map + exactly to those bins without inventing detailed CPS combinations. +- Recovered the complete 2024 Census detailed-occupation-to-`POCCU2` consumed + grouping from the native ASEC relationship and confirmed that ACS `OCCP` + uses the same detailed codes. `PEIOOCC` is therefore a direct carry, while + `POCCU2` will use an explicit reviewed 53-bin table; blank out-of-universe + occupation maps to code 0, military to 52, and code 9920 to 53. +- Confirmed ACS `TEN` maps to the SPM vehicle model's three consumed tenure + bins (mortgaged owner, outright owner, non-owner); no-cash-rent and verified + group-quarters blanks belong to the non-owner bin. Confirmed the SSI model's + `SSI_VAL` use is only the `> 0` reporter anchor and that native ACS `SSIP` is + already carried as harmonized `ssi_reported`, observed exactly from age 15. ## Next -- Prove the `person_source_id` encoding from assembly source and bind it to - `SERIALNO`/`SPORDER`, including collision and totality invariants. -- Establish each ACS/CPS code semantic from repository evidence; implement - only mappings supported by that evidence and report any unresolved field. -- Add pinned CLI inputs, the release join, receipts, crosswalk/join tests, - changelog fragment, full shard verification, and the final `out.md` report. +- Implement the SHA-pinned streaming loaders, explicit crosswalk identity, + strict semantic-key join, clone fan-out, `ssi_reported` anchor bridge, and + per-model/predictor receipt with focused refusal/totality tests. +- Add the paired release CLI inputs and carry the receipt into both build and + release manifests before the six archived model stages, without changing + their feature/selection logic or any gate threshold. +- Add the changelog fragment, run focused then full shard verification, and + write the final `out.md` report. # Weeksgate: stacked release gates and integer-week provenance From f5284a07bdcbdcd8b179347cd23cb82415b6afb6 Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Thu, 27 Aug 2026 14:51:25 +0200 Subject: [PATCH 34/49] Scale prior-year availability gate by sampling rung --- .../src/microcosm/build/us_runtime/h5_io.py | 159 ++++++++++++++++++ .../build/us_runtime/prior_year_income.py | 57 ++++++- .../tests/test_us_multispine_pool_h5_io.py | 141 +++++++++++++++- .../tests/test_us_prior_year_income.py | 130 ++++++++++++++ 4 files changed, 485 insertions(+), 2 deletions(-) diff --git a/packages/microcosm-build/src/microcosm/build/us_runtime/h5_io.py b/packages/microcosm-build/src/microcosm/build/us_runtime/h5_io.py index 516dead3c..af546794b 100644 --- a/packages/microcosm-build/src/microcosm/build/us_runtime/h5_io.py +++ b/packages/microcosm-build/src/microcosm/build/us_runtime/h5_io.py @@ -16,6 +16,7 @@ import shutil import uuid from collections.abc import Mapping +from copy import deepcopy from dataclasses import dataclass from pathlib import Path @@ -94,6 +95,15 @@ _TIME_PERIOD_KEY = "_time_period" _LOWERCASE_SHA256 = re.compile(r"[0-9a-f]{64}") _STACKED_PIPELINE = "us-stacked-pool" +_STACKED_SPINE_MANIFEST_VERSION = 4 +_STACKED_SURVEY_CHANNELS = ("asec", "acs") +_STACKED_SAMPLE_RUNG_TOKENS: Mapping[float, str] = { + 0.01: "f001", + 0.04: "f004", + 0.10: "f010", + 0.25: "f025", + 1.00: "f100", +} US_STACKED_POOL_OPERATOR_ORDER = ( "assemble_stacked_spine", "assign_us_puma_ladder", @@ -139,6 +149,8 @@ { "pipeline", "operator_order", + "sampling", + "stack_manifest", "geography_assignment", "stage_receipts", } @@ -255,6 +267,139 @@ def _validated_pool_manifest_envelope( ) +def _validated_stacked_sampling_manifest_binding( + manifest: Mapping[str, object], + *, + manifest_path: Path, +) -> Mapping[str, object] | None: + """Authenticate the production-wide survey rung carried by a stack. + + The adjacent-year ASEC join runs after the two survey arms are sampled. + Release gates may therefore consume the configured production rung only + after the pool manifest proves that its top-level sampling receipt, frozen + stack manifest, and both per-arm sample receipts all name the same value. + Legacy two-spine manifests have no production-wide stack receipt and return + ``None``. + """ + + if manifest.get("pipeline") != _STACKED_PIPELINE: + return None + + label = f"US stacked pool manifest {manifest_path}" + sampling = _mapping(manifest.get("sampling"), label=f"{label}.sampling") + stack_manifest = _mapping( + manifest.get("stack_manifest"), + label=f"{label}.stack_manifest", + ) + if stack_manifest.get("version") != _STACKED_SPINE_MANIFEST_VERSION: + raise ValueError( + f"{label} stack manifest must have production version " + f"{_STACKED_SPINE_MANIFEST_VERSION}." + ) + + sampling_fraction = sampling.get("sample_fraction") + stack_fraction = stack_manifest.get("sample_fraction") + for location, value in ( + ("sampling.sample_fraction", sampling_fraction), + ("stack_manifest.sample_fraction", stack_fraction), + ): + if ( + type(value) is not float + or not np.isfinite(value) + or not 0.0 < value <= 1.0 + ): + raise ValueError( + f"{label} {location} must be a finite float in (0, 1]." + ) + if sampling_fraction != stack_fraction: + raise ValueError( + f"{label} sampling.sample_fraction differs from " + "stack_manifest.sample_fraction." + ) + expected_token = _STACKED_SAMPLE_RUNG_TOKENS.get(sampling_fraction) + if expected_token is None or sampling.get("fraction_token") != expected_token: + raise ValueError( + f"{label} sampling fraction/token pair is not an approved stacked rung." + ) + + sampling_seed = sampling.get("sample_seed") + stack_seed = stack_manifest.get("sample_seed") + if ( + isinstance(sampling_seed, bool) + or not isinstance(sampling_seed, int) + or sampling_seed < 0 + or stack_seed != sampling_seed + ): + raise ValueError( + f"{label} sampling.sample_seed and stack_manifest.sample_seed must " + "be the same non-negative integer." + ) + + survey_samples = _mapping( + stack_manifest.get("survey_samples"), + label=f"{label}.stack_manifest.survey_samples", + ) + if set(survey_samples) != set(_STACKED_SURVEY_CHANNELS): + raise ValueError( + f"{label} stack survey samples must exactly cover " + f"{list(_STACKED_SURVEY_CHANNELS)}." + ) + realized_households = _mapping( + sampling.get("realized_households"), + label=f"{label}.sampling.realized_households", + ) + if set(realized_households) != set(_STACKED_SURVEY_CHANNELS): + raise ValueError( + f"{label} realized-household counts must exactly cover " + f"{list(_STACKED_SURVEY_CHANNELS)}." + ) + for channel in _STACKED_SURVEY_CHANNELS: + sample = _mapping( + survey_samples[channel], + label=f"{label}.stack_manifest.survey_samples.{channel}", + ) + if sample.get("fraction") != sampling_fraction: + raise ValueError( + f"{label} {channel} survey-sample fraction differs from the " + "production sampling rung." + ) + if sample.get("seed") != sampling_seed: + raise ValueError( + f"{label} {channel} survey-sample seed differs from the " + "production sample seed." + ) + realized = sample.get("realized_household_count") + if ( + isinstance(realized, bool) + or not isinstance(realized, int) + or realized < 1 + or realized_households[channel] != realized + ): + raise ValueError( + f"{label} {channel} realized-household count is malformed or " + "inconsistent." + ) + + expected_stack_sha256 = sampling.get("stack_manifest_sha256") + if ( + not isinstance(expected_stack_sha256, str) + or _LOWERCASE_SHA256.fullmatch(expected_stack_sha256) is None + ): + raise ValueError(f"{label} sampling stack-manifest SHA-256 is malformed.") + canonical_stack = json.dumps( + stack_manifest, + allow_nan=False, + ensure_ascii=False, + separators=(",", ":"), + sort_keys=True, + ).encode("utf-8") + if hashlib.sha256(canonical_stack).hexdigest() != expected_stack_sha256: + raise ValueError( + f"{label} sampling stack-manifest SHA-256 does not match its receipt." + ) + return stack_manifest + + class AuthenticatedPoolH5MismatchError(RuntimeError): """A pool H5 no longer matches the bytes authenticated by its manifest.""" @@ -617,6 +762,10 @@ def _load_authenticated_us_multispine_pool_manifest( manifest, manifest_path=manifest_path, ) + _validated_stacked_sampling_manifest_binding( + manifest, + manifest_path=manifest_path, + ) expected_schema_version = ( US_MULTISPINE_POOL_MANIFEST_SCHEMA_VERSION if envelope == "stacked" @@ -1587,6 +1736,16 @@ def _load_us_multispine_pool( manifest_path=manifest_path, ) frame_metadata: dict[str, object] = {} + stack_manifest = _validated_stacked_sampling_manifest_binding( + manifest, + manifest_path=manifest_path, + ) + if stack_manifest is not None: + from microcosm.build.us_runtime.stacked_spine import ( + STACKED_SPINE_MANIFEST_KEY, + ) + + frame_metadata[STACKED_SPINE_MANIFEST_KEY] = deepcopy(stack_manifest) if late_transition is not None: _dag, transition_authority, _transition_authority_sha256 = late_transition from microcosm.build.us_runtime.stacked_spine import ( diff --git a/packages/microcosm-build/src/microcosm/build/us_runtime/prior_year_income.py b/packages/microcosm-build/src/microcosm/build/us_runtime/prior_year_income.py index b03340266..ab3e56d6f 100644 --- a/packages/microcosm-build/src/microcosm/build/us_runtime/prior_year_income.py +++ b/packages/microcosm-build/src/microcosm/build/us_runtime/prior_year_income.py @@ -20,6 +20,7 @@ from __future__ import annotations +from collections.abc import Mapping from importlib.resources import files from typing import Any @@ -181,6 +182,47 @@ ) _PREVIOUS_YEAR_AVAILABLE_SHARE_BAND = (0.05, 0.50) _SELF_EMPLOYMENT_NONZERO_SHARE_BAND = (0.01, 0.25) +_STACKED_SPINE_MANIFEST_METADATA_KEY = "us_stacked_spine_manifest" +_PRODUCTION_STACKED_SPINE_MANIFEST_VERSION = 4 + + +def _previous_year_availability_match_survival_factor(frame: Frame) -> float: + """Return the production-wide ASEC sampled-match survival factor. + + Production stacked assembly samples every raw ASEC year independently + before the adjacent-year join, so the configured all-survey rung scales + the join's expected availability floor. The version-1 pilot sampled ACS + only and left ASEC whole; unstacked and legacy frames therefore retain the + authored floor exactly. + """ + + manifest = frame.metadata.get(_STACKED_SPINE_MANIFEST_METADATA_KEY) + if manifest is None: + return 1.0 + if not isinstance(manifest, Mapping): + raise ValueError( + "US prior-year-income availability requires the stacked spine " + "manifest metadata to be an object." + ) + version = manifest.get("version") + if version == 1: + return 1.0 + if version != _PRODUCTION_STACKED_SPINE_MANIFEST_VERSION: + raise ValueError( + "US prior-year-income availability received an unsupported stacked " + f"spine manifest version {version!r}." + ) + factor = manifest.get("sample_fraction") + if ( + type(factor) is not float + or not np.isfinite(factor) + or not 0.0 < factor <= 1.0 + ): + raise ValueError( + "US prior-year-income availability requires a finite production " + "stacked sample_fraction in (0, 1]." + ) + return factor def us_prior_year_income_stage_spec() -> SourceStageSpec: @@ -844,12 +886,25 @@ def us_prior_year_income_signal_gate(frame: Frame) -> GateResult: details={"nonfinite_rows": rows}, ) + match_survival_factor = _previous_year_availability_match_survival_factor(frame) summary = us_prior_year_income_summary(frame) + availability_band_key = "previous_year_income_available_share_band" + if match_survival_factor != 1.0: + authored_lower, authored_upper = summary[availability_band_key] + applied_floor = float(authored_lower) * match_survival_factor + availability_band_key = ( + "previous_year_income_available_applied_share_band" + ) + summary[ + "previous_year_income_available_sampled_match_survival_factor" + ] = match_survival_factor + summary["previous_year_income_available_applied_floor"] = applied_floor + summary[availability_band_key] = [applied_floor, authored_upper] failures: list[str] = [] checks = ( ( "previous_year_income_available_share", - "previous_year_income_available_share_band", + availability_band_key, "previous-year availability weighted share", ), ( diff --git a/packages/microcosm-build/tests/test_us_multispine_pool_h5_io.py b/packages/microcosm-build/tests/test_us_multispine_pool_h5_io.py index 5544e92a5..6eb1f5b68 100644 --- a/packages/microcosm-build/tests/test_us_multispine_pool_h5_io.py +++ b/packages/microcosm-build/tests/test_us_multispine_pool_h5_io.py @@ -54,6 +54,17 @@ def _sha256(path: Path) -> str: return hashlib.sha256(path.read_bytes()).hexdigest() +def _json_sha256(value: object) -> str: + payload = json.dumps( + value, + allow_nan=False, + ensure_ascii=False, + separators=(",", ":"), + sort_keys=True, + ).encode("utf-8") + return hashlib.sha256(payload).hexdigest() + + def _pool_frame() -> Frame: ids = np.asarray([10, 20, 30], dtype=np.int64) person = pd.DataFrame( @@ -580,7 +591,46 @@ def _fixture_geography_assignment( } -def _write_ready_pool(tmp_path: Path, *, stacked: bool = False) -> Path: +def _fixture_stacked_sampling( + sample_fraction: float, +) -> tuple[dict[str, object], dict[str, object]]: + fraction_token = { + 0.01: "f001", + 0.04: "f004", + 0.10: "f010", + 0.25: "f025", + 1.00: "f100", + }[sample_fraction] + realized = {"asec": 2, "acs": 1} + stack_manifest: dict[str, object] = { + "version": 4, + "sample_fraction": sample_fraction, + "sample_seed": 578, + "survey_samples": { + channel: { + "fraction": sample_fraction, + "seed": 578, + "realized_household_count": count, + } + for channel, count in realized.items() + }, + } + sampling = { + "sample_fraction": sample_fraction, + "fraction_token": fraction_token, + "sample_seed": 578, + "realized_households": realized, + "stack_manifest_sha256": _json_sha256(stack_manifest), + } + return sampling, stack_manifest + + +def _write_ready_pool( + tmp_path: Path, + *, + stacked: bool = False, + sample_fraction: float = 1.0, +) -> Path: run_id = "fixture-publication" pool_path = tmp_path / "pool.h5" diagnostics_path = tmp_path / "pool.agreement.json" @@ -691,6 +741,7 @@ def _write_ready_pool(tmp_path: Path, *, stacked: bool = False) -> Path: if stacked: dag = _canonical_stacked_late_dag_receipt() assert geography_assignment is not None + sampling, stack_manifest = _fixture_stacked_sampling(sample_fraction) transition_authority = ( stacked_spine_module._late_producer_transition_authority_receipt(dag) ) @@ -698,6 +749,8 @@ def _write_ready_pool(tmp_path: Path, *, stacked: bool = False) -> Path: { "pipeline": "us-stacked-pool", "random_seed": 0, + "sampling": sampling, + "stack_manifest": stack_manifest, "geography_assignment": geography_assignment, "provenance_pins": { role: { @@ -1358,6 +1411,92 @@ def test_ready_stacked_pool_loader_binds_terminal_gate_aliases( transition_authority["sha256"] == manifest["late_producer_transition_authority_sha256"] ) + assert stacked_spine_module._json_ready( + frame.metadata[stacked_spine_module.STACKED_SPINE_MANIFEST_KEY] + ) == ( + manifest["stack_manifest"] + ) + + +def test_ready_stacked_pool_loader_restores_sampled_rung_manifest( + tmp_path: Path, +) -> None: + pytest.importorskip("tables") + manifest_path = _write_ready_pool( + tmp_path, + stacked=True, + sample_fraction=0.25, + ) + + frame, manifest, _ = load_simulation_ready_us_multispine_pool(manifest_path) + + stack_manifest = frame.metadata[ + stacked_spine_module.STACKED_SPINE_MANIFEST_KEY + ] + assert stacked_spine_module._json_ready(stack_manifest) == manifest[ + "stack_manifest" + ] + assert stack_manifest["version"] == 4 + assert stack_manifest["sample_fraction"] == 0.25 + + +def test_ready_stacked_pool_loader_rejects_inconsistent_sampling_factor( + tmp_path: Path, +) -> None: + pytest.importorskip("tables") + manifest_path = _write_ready_pool( + tmp_path, + stacked=True, + sample_fraction=0.25, + ) + manifest = json.loads(manifest_path.read_text(encoding="utf-8")) + manifest["sampling"]["sample_fraction"] = 0.10 + manifest["sampling"]["fraction_token"] = "f010" + manifest_path.write_text(json.dumps(manifest), encoding="utf-8") + + with pytest.raises(ValueError, match="sample_fraction differs"): + load_simulation_ready_us_multispine_pool(manifest_path) + + +def test_ready_stacked_pool_loader_rejects_inconsistent_arm_sampling( + tmp_path: Path, +) -> None: + pytest.importorskip("tables") + manifest_path = _write_ready_pool( + tmp_path, + stacked=True, + sample_fraction=0.25, + ) + manifest = json.loads(manifest_path.read_text(encoding="utf-8")) + manifest["stack_manifest"]["survey_samples"]["asec"]["fraction"] = 0.10 + manifest["sampling"]["stack_manifest_sha256"] = _json_sha256( + manifest["stack_manifest"] + ) + manifest_path.write_text(json.dumps(manifest), encoding="utf-8") + + with pytest.raises(ValueError, match="asec survey-sample fraction differs"): + load_simulation_ready_us_multispine_pool(manifest_path) + + +@pytest.mark.parametrize("sample_fraction", [True, 0, 0.25]) +def test_ready_stacked_pool_loader_rejects_malformed_sampling_receipt( + tmp_path: Path, + sample_fraction: object, +) -> None: + pytest.importorskip("tables") + manifest_path = _write_ready_pool(tmp_path, stacked=True) + manifest = json.loads(manifest_path.read_text(encoding="utf-8")) + if sample_fraction == 0.25: + del manifest["stack_manifest"]["sample_fraction"] + manifest["sampling"]["stack_manifest_sha256"] = _json_sha256( + manifest["stack_manifest"] + ) + else: + manifest["sampling"]["sample_fraction"] = sample_fraction + manifest_path.write_text(json.dumps(manifest), encoding="utf-8") + + with pytest.raises(ValueError, match="sample_fraction.*finite float"): + load_simulation_ready_us_multispine_pool(manifest_path) def test_ready_stacked_pool_loader_binds_h5_cd_vintage_attrs( diff --git a/packages/microcosm-build/tests/test_us_prior_year_income.py b/packages/microcosm-build/tests/test_us_prior_year_income.py index c4fdbcd3a..8cd06d53d 100644 --- a/packages/microcosm-build/tests/test_us_prior_year_income.py +++ b/packages/microcosm-build/tests/test_us_prior_year_income.py @@ -3,6 +3,7 @@ from __future__ import annotations import importlib.util +import json from pathlib import Path import numpy as np @@ -10,6 +11,7 @@ import pytest import microcosm.build.us_runtime.prior_year_income as module +from microcosm.build.gates import GateReport from microcosm.build.source_runtime import SourceRuntimeError from microcosm.build.us_runtime.l0_refit_export import ( US_RELEASE_REQUIRED_PERSON_SOURCE_COLUMNS, @@ -395,6 +397,30 @@ def _signal_frame() -> Frame: ) +def _signal_frame_with_availability_rows(rows: int) -> Frame: + frame = _signal_frame() + person = frame.table("person").copy() + person["previous_year_income_available"] = np.arange(len(person)) < rows + return module._replace_person_table(frame, person) + + +def _with_stack_manifest( + frame: Frame, + manifest: object, +) -> Frame: + return Frame( + {entity: frame.table(entity).copy() for entity in frame.entities}, + frame.schema, + { + entity: frame.weights_for(entity) + for entity in frame.weighted_entities + }, + frame.strata, + mass_log=frame.mass_log, + metadata={"us_stacked_spine_manifest": manifest}, + ) + + def test_signal_gate_accepts_signed_source_signal_and_rejects_defaults() -> None: passing = us_prior_year_income_signal_gate(_signal_frame()) assert passing.passed, passing.failures @@ -410,6 +436,110 @@ def test_signal_gate_accepts_signed_source_signal_and_rejects_defaults() -> None assert "availability" in " ".join(failing.failures) +def test_sampled_rung_scales_only_prior_year_availability_floor() -> None: + frame = _with_stack_manifest( + _signal_frame_with_availability_rows(6), + {"version": 4, "sample_fraction": 0.25}, + ) + + gate = us_prior_year_income_signal_gate(frame) + + assert gate.passed, gate.failures + assert gate.details["previous_year_income_available_share"] == pytest.approx( + 0.04101010101010102 + ) + assert gate.details["previous_year_income_available_share_band"] == [0.05, 0.50] + assert gate.details[ + "previous_year_income_available_sampled_match_survival_factor" + ] == pytest.approx(0.25) + assert gate.details["previous_year_income_available_applied_floor"] == pytest.approx( + 0.0125 + ) + assert gate.details["previous_year_income_available_applied_share_band"] == [ + 0.0125, + 0.50, + ] + assert gate.details["self_employment_income_last_year_nonzero_share_band"] == [ + 0.01, + 0.25, + ] + + +def test_sampled_rung_preserves_applied_floor_and_authored_upper_bound() -> None: + below_floor = us_prior_year_income_signal_gate( + _with_stack_manifest( + _signal_frame_with_availability_rows(1), + {"version": 4, "sample_fraction": 0.25}, + ) + ) + above_upper = us_prior_year_income_signal_gate( + _with_stack_manifest( + _signal_frame_with_availability_rows(60), + {"version": 4, "sample_fraction": 0.25}, + ) + ) + + assert any("outside [0.012500, 0.500000]" in row for row in below_floor.failures) + assert any("outside [0.012500, 0.500000]" in row for row in above_upper.failures) + + +def test_full_rung_gate_manifest_is_byte_identical_to_legacy_gate() -> None: + frame = _signal_frame_with_availability_rows(6) + legacy = us_prior_year_income_signal_gate(frame) + full_rung = us_prior_year_income_signal_gate( + _with_stack_manifest( + frame, + {"version": 4, "sample_fraction": 1.0}, + ) + ) + + def manifest_bytes(gate) -> bytes: + return json.dumps( + GateReport((gate,)).to_manifest(), + allow_nan=False, + ensure_ascii=False, + separators=(",", ":"), + sort_keys=True, + ).encode("utf-8") + + assert full_rung == legacy + assert manifest_bytes(full_rung) == manifest_bytes(legacy) + assert not any("applied" in key or "survival_factor" in key for key in full_rung.details) + + +def test_legacy_acs_only_sampling_does_not_scale_asec_match_floor() -> None: + frame = _signal_frame_with_availability_rows(6) + legacy = us_prior_year_income_signal_gate(frame) + pilot = us_prior_year_income_signal_gate( + _with_stack_manifest( + frame, + {"version": 1, "acs_sample_fraction": 0.25}, + ) + ) + + assert not legacy.passed + assert pilot == legacy + + +@pytest.mark.parametrize( + "manifest", + [ + {"version": 4}, + {"version": 4, "sample_fraction": True}, + {"version": 4, "sample_fraction": 0.0}, + {"version": 3, "sample_fraction": 0.25}, + "malformed", + ], +) +def test_signal_gate_rejects_malformed_stacked_sampling_metadata( + manifest: object, +) -> None: + with pytest.raises(ValueError, match="prior-year-income availability"): + us_prior_year_income_signal_gate( + _with_stack_manifest(_signal_frame(), manifest) + ) + + def test_clone_availability_checks_all_assembled_clones_and_legacy_pairs() -> None: assembled = _frame( pd.DataFrame( From 4aa6269a7d5e74e5d0837ad47ebc692661330a0c Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Thu, 27 Aug 2026 14:52:33 +0200 Subject: [PATCH 35/49] Record prior-year gate implementation --- PROGRESS.md | 13 ++++++++++--- 1 file changed, 10 insertions(+), 3 deletions(-) diff --git a/PROGRESS.md b/PROGRESS.md index 43e3a25cd..77d835003 100644 --- a/PROGRESS.md +++ b/PROGRESS.md @@ -45,6 +45,14 @@ release artifacts. floor by the assembly's sampled-to-sampled match-survival factor, recorded in or derived from the pool manifest. Rung 1.0 must remain byte-identical to the existing gate; the authored 0.05 constant and upper bound do not change. +- Implemented the owner-approved prior-year gate policy in `f5284a07`. An + authenticated production stacked manifest now restores its version-4 sample + receipt to the loaded frame; the availability gate scales only the authored + lower floor by that rung, conditionally receipts the factor/applied floor, + and leaves full-rung output byte-identical. Legacy/no-rung frames retain the + original gate. The prior-year and H5 focused suites pass (23 and 60 tests), + focused Ruff and `git diff --check` pass, and the real candidate-25 manifest + validates at factor 0.25. ## Next @@ -53,9 +61,8 @@ release artifacts. recording structural counts. - Bind the pregnancy structural policy into checkpointed transfer execution identity and add transfer/source/gate regressions. -- Implement and receipt the ruling-approved sampling-rung scaling of only the - prior-year availability floor, with exact rung-1 compatibility and focused - release-gate regressions. +- Complete and focused-test the pregnancy source/transfer/gate structural + policy, receipt authentication, and checkpoint execution identity. - Run the charter's repository Ruff and independent full-shard pytest boundary, then write the complete handoff to `out.md`. From ecea55a20acd1752aebea4b417985d6037f27f35 Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Thu, 27 Aug 2026 14:55:37 +0200 Subject: [PATCH 36/49] Add strict ACS release predictor join --- PROGRESS.md | 34 +- .../us_runtime/acs_release_predictors.py | 1584 +++++++++++++++++ .../us_runtime/ssi_disability_criteria.py | 84 +- .../tests/test_us_acs_release_predictors.py | 374 ++++ .../tests/test_us_spine_blindness.py | 14 +- .../tests/test_us_ssi_disability_criteria.py | 29 + 6 files changed, 2091 insertions(+), 28 deletions(-) create mode 100644 packages/microcosm-build/src/microcosm/build/us_runtime/acs_release_predictors.py create mode 100644 packages/microcosm-build/tests/test_us_acs_release_predictors.py diff --git a/PROGRESS.md b/PROGRESS.md index 63d15b908..22356a698 100644 --- a/PROGRESS.md +++ b/PROGRESS.md @@ -57,17 +57,41 @@ push, retraining, threshold change, or launcher-contract edit is in scope. group-quarters blanks belong to the non-owner bin. Confirmed the SSI model's `SSI_VAL` use is only the `> 0` reporter anchor and that native ACS `SSIP` is already carried as harmonized `ssi_reported`, observed exactly from age 15. +- Added the dedicated `acs_release_predictors` release boundary. It verifies + the two canonical archive pins before opening either zip, streams only + selected households, validates exact archive members and headers, rejects + raw/person/clone collisions, binds retained pool lineage to + `(SERIALNO, SPORDER)`, requires total one-to-one source-person matching, and + fans mapped values to clones only through `person_source_id`. +- Added explicit disability, race/Hispanic, 530-code occupation, and tenure + tables. A canonical crosswalk payload is pinned at SHA-256 + `cf21e20831dd15479e8f5704743dc5e22e5b8a8b78546107ba5024f22d8f3f1b` + and rides the JSON-ready join receipt with per-model/per-predictor + ASEC-native, ACS-joined, and still-null counts. +- Preserved CPS disability universe semantics (`-1` below the question age) + and the POCCU2 age universe (0 below 15, consumed no-occupation code 53 for + older ACS OCCP blanks). The explicit occupation table covers every one of + the 530 codes in the pinned ACS person archive and every consumed POCCU2 bin. +- Changed the SSI-disability reporter read, without source routing, to + row-wise coalesce measured ASEC `SSI_VAL` with harmonized native ACS + `ssi_reported`. Adult blanks and conflicting dual reporters fail; genuine + below-age-15 ACS blanks remain null in the frame and become false only for + the consumer's `> 0` predicate. +- Added focused tests for crosswalk identity/all consumed bins, exact join and + clone invariance, ASEC byte preservation, receipt contents, missing joins, + raw and source-identity collisions, hash refusal, no-ACS identity, and SSI + coalescing/universe refusal. The new join file passes all 7 tests; the SSI + and source-blindness suites passed alongside it before the final join-only + merge-indicator repair, and focused Ruff is green. ## Next -- Implement the SHA-pinned streaming loaders, explicit crosswalk identity, - strict semantic-key join, clone fan-out, `ssi_reported` anchor bridge, and - per-model/predictor receipt with focused refusal/totality tests. - Add the paired release CLI inputs and carry the receipt into both build and release manifests before the six archived model stages, without changing their feature/selection logic or any gate threshold. -- Add the changelog fragment, run focused then full shard verification, and - write the final `out.md` report. +- Re-run the complete focused SSI/source-blindness group, exercise the join + read-only against the supplied pool and canonical zips, add the changelog + fragment, then run full shard verification and write `out.md`. # Weeksgate: stacked release gates and integer-week provenance diff --git a/packages/microcosm-build/src/microcosm/build/us_runtime/acs_release_predictors.py b/packages/microcosm-build/src/microcosm/build/us_runtime/acs_release_predictors.py new file mode 100644 index 000000000..7fcd0210c --- /dev/null +++ b/packages/microcosm-build/src/microcosm/build/us_runtime/acs_release_predictors.py @@ -0,0 +1,1584 @@ +"""Release-time ACS crosswalks for CPS-named archived-model predictors. + +The stacked pool deliberately preserves native source columns rather than +pretending that ACS fields are CPS fields. Six release-stage archived donor +models nevertheless consume CPS-named predictors. Until a cold pool rebuild +can carry the reviewed harmonization, this module is the single origin-aware +release boundary that joins selected ACS people back to the exact 2024 one-year +PUMS archives and materializes only the bins those models consume. + +``person_source_id`` is an assembly identity, not a Census key. Assembly can +offset it to avoid cross-spine collisions. The semantic join therefore uses +the raw lineage retained by the pool: parent-household ``SERIALNO`` plus +integral person ``SPORDER``. ``person_source_id`` is used only after the +one-to-one raw join, to fan one source person's values to all support clones. + +The mappings below cite their executable consumers rather than inventing CPS +detail the models never read: + +* ``ssi_disability_criteria._ASEC_DIFFICULTY_SOURCE_COLUMNS`` consumes each + disability field only as ``== 1``; +* ``scf_wealth._recipient_cps_race`` and ``org_wages._derive_wbho`` consume + White, Black, Asian, Hispanic, and residual Other bins; +* ``sipp_tips.CENSUS_OCCUPATION_CODE_TO_TTOC`` consumes detailed Census + occupation codes directly; +* ``org_wages.FLSA_OVERTIME_OCCUPATION_CODES`` and its EAP set consume the + 53-category CPS detailed occupation recode; and +* ``sipp_vehicles._household_tenure_status`` consumes only mortgaged owner, + outright owner, and non-owner tenure codes. + +Every mapping is explicit and included in a canonical digest carried by the +release receipt. Changing one code or universe boundary requires a deliberate +digest repin and focused contract-test change. +""" + +from __future__ import annotations + +import hashlib +import json +import re +from collections.abc import Mapping, Sequence +from dataclasses import dataclass +from pathlib import Path +from typing import Any +from zipfile import ZipFile + +import numpy as np +import pandas as pd + +from microcosm.build.us_runtime.support_provenance import ( + support_channel_column, + support_clone_index_column, + support_source_id_column, + validate_assembly_provenance, +) +from microcosm.frame import US_SCHEMA, Frame + +__all__ = [ + "ACS_2024_HOUSEHOLD_ZIP_SHA256", + "ACS_2024_PERSON_ZIP_SHA256", + "ACS_DIFFICULTY_TO_CPS", + "ACS_OCCP_TO_POCCU2", + "ACS_RAC1P_TO_CONSUMED_PRDTRACE", + "ACS_RELEASE_PREDICTOR_CROSSWALK_SHA256", + "ACS_TEN_TO_SPM_TENMORTSTATUS", + "AcsReleasePredictorJoinResult", + "acs_release_predictor_crosswalk_payload", + "join_acs_release_predictors", +] + +ACS_2024_PERSON_ZIP_SHA256 = ( + "afdc6d90c6e2f0bab365ed32d95ba4c4d8ac651162f46ac7861295b2dc469894" +) +ACS_2024_HOUSEHOLD_ZIP_SHA256 = ( + "8281008e53de98f0ef81e7a2ee5a8725991dda1ecfd2713ead73246425e515d0" +) +ACS_RELEASE_PREDICTOR_CROSSWALK_VERSION = 1 +ACS_RELEASE_PREDICTOR_CROSSWALK_SHA256 = ( + "cf21e20831dd15479e8f5704743dc5e22e5b8a8b78546107ba5024f22d8f3f1b" +) + +_SHA256_PATTERN = re.compile(r"[0-9a-f]{64}") +_DEFAULT_CHUNKSIZE = 250_000 +_ACS_CHANNEL = "acs" +_ASEC_CHANNEL = "asec" +_ACS_VINTAGE = 2024 + +_PERSON_MEMBERS = ("psam_pusa.csv", "psam_pusb.csv") +_HOUSEHOLD_MEMBERS = ("psam_husa.csv", "psam_husb.csv") +_PERSON_RAW_COLUMNS = ( + "SERIALNO", + "SPORDER", + "AGEP", + "DEAR", + "DEYE", + "DREM", + "DPHY", + "DDRS", + "DOUT", + "RAC1P", + "HISP", + "OCCP", + "ESR", +) +_HOUSEHOLD_RAW_COLUMNS = ("SERIALNO", "NP", "TYPEHUGQ", "TEN") + +ACS_DIFFICULTY_TO_CPS: Mapping[str, str] = { + "DDRS": "PEDISDRS", + "DEAR": "PEDISEAR", + "DEYE": "PEDISEYE", + "DOUT": "PEDISOUT", + "DPHY": "PEDISPHY", + "DREM": "PEDISREM", +} +_ACS_DIFFICULTY_MIN_AGE: Mapping[str, int] = { + "DEAR": 0, + "DEYE": 0, + "DREM": 5, + "DPHY": 5, + "DDRS": 5, + "DOUT": 15, +} + +# The consumers distinguish White (1), Black (2), Asian (4), and residual +# Other. ACS codes 3--5 are American Indian / Alaska Native detail, 7 is +# Native Hawaiian / Pacific Islander, 8 is another race, and 9 is multiracial; +# none has a separate consumed model bin, so code 3 is the canonical CPS Other +# representative rather than a fabricated detailed multiracial code. +ACS_RAC1P_TO_CONSUMED_PRDTRACE: Mapping[int, int] = { + 1: 1, + 2: 2, + 3: 3, + 4: 3, + 5: 3, + 6: 4, + 7: 3, + 8: 3, + 9: 3, +} + +# ACS HISP=1 is non-Hispanic. Codes 2--24 are Hispanic-origin detail; the +# consumers test only zero versus positive, so 1 is the canonical positive +# CPS representative. +_ACS_HISP_TO_CONSUMED_PRDTHSP: Mapping[int, int] = { + 1: 0, + **{code: 1 for code in range(2, 25)}, +} + +# 2024 Census detailed occupation codes mapped to the 53 POCCU2 categories +# used by the archived ORG/FLSA consumer. The table is intentionally explicit: +# the canonical archive pin and crosswalk digest jointly refuse a new Census +# code until its consumed category is reviewed. Blank OCCP is handled +# separately: PEIOOCC uses 0, while POCCU2 preserves its age universe. +ACS_OCCP_TO_POCCU2: Mapping[int, int] = { + 10: 1, + 20: 1, + 40: 1, + 51: 1, + 52: 1, + 60: 1, + 101: 1, + 102: 1, + 110: 1, + 120: 1, + 135: 2, + 136: 2, + 137: 2, + 140: 2, + 150: 2, + 160: 2, + 205: 2, + 220: 2, + 230: 3, + 300: 3, + 310: 3, + 335: 3, + 340: 3, + 350: 3, + 360: 3, + 410: 3, + 420: 3, + 425: 3, + 440: 3, + 500: 4, + 510: 5, + 520: 5, + 530: 5, + 540: 5, + 565: 5, + 600: 5, + 630: 5, + 640: 5, + 650: 5, + 700: 5, + 705: 5, + 710: 5, + 725: 5, + 726: 5, + 735: 5, + 750: 5, + 800: 6, + 810: 7, + 820: 7, + 830: 7, + 845: 7, + 850: 7, + 860: 7, + 900: 7, + 910: 7, + 930: 7, + 940: 7, + 960: 7, + 1005: 8, + 1006: 8, + 1007: 8, + 1010: 8, + 1021: 8, + 1022: 8, + 1031: 8, + 1032: 8, + 1050: 8, + 1065: 8, + 1105: 8, + 1106: 8, + 1108: 8, + 1200: 9, + 1220: 9, + 1240: 9, + 1305: 10, + 1306: 10, + 1310: 11, + 1320: 12, + 1340: 12, + 1350: 12, + 1360: 12, + 1400: 12, + 1410: 12, + 1420: 12, + 1430: 12, + 1440: 12, + 1450: 12, + 1460: 12, + 1520: 12, + 1530: 12, + 1541: 12, + 1545: 12, + 1551: 12, + 1555: 12, + 1560: 12, + 1600: 13, + 1610: 13, + 1640: 13, + 1650: 13, + 1700: 13, + 1710: 13, + 1720: 13, + 1745: 13, + 1750: 13, + 1760: 13, + 1800: 14, + 1821: 15, + 1822: 15, + 1825: 15, + 1840: 15, + 1860: 15, + 1900: 16, + 1910: 16, + 1920: 16, + 1935: 16, + 1970: 16, + 1980: 16, + 2001: 17, + 2002: 17, + 2003: 17, + 2004: 17, + 2005: 17, + 2006: 17, + 2011: 17, + 2012: 17, + 2013: 17, + 2014: 17, + 2015: 17, + 2016: 17, + 2025: 17, + 2040: 17, + 2050: 17, + 2060: 17, + 2100: 18, + 2105: 18, + 2145: 19, + 2170: 19, + 2180: 19, + 2205: 20, + 2300: 21, + 2310: 21, + 2320: 21, + 2330: 21, + 2350: 21, + 2360: 21, + 2400: 22, + 2435: 22, + 2440: 22, + 2545: 22, + 2555: 22, + 2600: 23, + 2631: 23, + 2632: 23, + 2633: 23, + 2634: 23, + 2635: 23, + 2636: 23, + 2640: 23, + 2700: 23, + 2710: 23, + 2721: 23, + 2722: 23, + 2723: 23, + 2740: 23, + 2751: 23, + 2752: 23, + 2755: 23, + 2770: 23, + 2805: 23, + 2810: 23, + 2825: 23, + 2830: 23, + 2840: 23, + 2850: 23, + 2861: 23, + 2862: 23, + 2865: 23, + 2905: 23, + 2910: 23, + 2920: 23, + 3000: 24, + 3010: 24, + 3030: 24, + 3040: 24, + 3050: 24, + 3090: 24, + 3100: 24, + 3110: 24, + 3120: 24, + 3140: 25, + 3150: 25, + 3160: 25, + 3200: 25, + 3210: 25, + 3220: 25, + 3230: 25, + 3245: 25, + 3250: 26, + 3255: 25, + 3256: 25, + 3258: 25, + 3261: 27, + 3270: 27, + 3300: 27, + 3310: 27, + 3321: 27, + 3322: 27, + 3323: 27, + 3324: 27, + 3330: 27, + 3401: 27, + 3402: 27, + 3421: 27, + 3422: 27, + 3423: 27, + 3424: 27, + 3430: 27, + 3500: 27, + 3515: 27, + 3520: 27, + 3545: 27, + 3550: 27, + 3601: 28, + 3602: 28, + 3603: 28, + 3605: 28, + 3610: 28, + 3620: 28, + 3630: 28, + 3640: 28, + 3645: 28, + 3646: 28, + 3647: 28, + 3648: 28, + 3649: 28, + 3655: 28, + 3700: 29, + 3710: 29, + 3720: 29, + 3725: 29, + 3740: 30, + 3750: 30, + 3801: 30, + 3802: 30, + 3820: 30, + 3840: 30, + 3870: 30, + 3900: 31, + 3910: 31, + 3930: 31, + 3940: 31, + 3945: 31, + 3946: 31, + 3960: 31, + 4000: 32, + 4010: 32, + 4020: 32, + 4030: 33, + 4040: 33, + 4055: 33, + 4110: 33, + 4120: 33, + 4130: 33, + 4140: 33, + 4150: 33, + 4160: 33, + 4200: 34, + 4210: 34, + 4220: 35, + 4230: 35, + 4240: 35, + 4251: 35, + 4252: 35, + 4255: 35, + 4330: 36, + 4340: 37, + 4350: 37, + 4400: 37, + 4420: 37, + 4435: 37, + 4461: 37, + 4465: 37, + 4500: 37, + 4510: 37, + 4521: 37, + 4522: 37, + 4525: 37, + 4530: 37, + 4540: 37, + 4600: 37, + 4621: 37, + 4622: 37, + 4640: 37, + 4655: 37, + 4700: 38, + 4710: 38, + 4720: 39, + 4740: 39, + 4750: 39, + 4760: 39, + 4800: 39, + 4810: 39, + 4820: 39, + 4830: 39, + 4840: 39, + 4850: 39, + 4900: 39, + 4920: 39, + 4930: 39, + 4940: 39, + 4950: 39, + 4965: 39, + 5000: 40, + 5010: 40, + 5020: 40, + 5040: 40, + 5100: 40, + 5110: 40, + 5120: 40, + 5140: 40, + 5150: 40, + 5160: 40, + 5165: 40, + 5220: 40, + 5230: 40, + 5240: 40, + 5250: 40, + 5260: 40, + 5300: 40, + 5310: 40, + 5320: 40, + 5330: 40, + 5340: 40, + 5350: 40, + 5360: 40, + 5400: 40, + 5410: 40, + 5420: 40, + 5500: 40, + 5510: 40, + 5521: 40, + 5522: 40, + 5530: 40, + 5540: 40, + 5550: 40, + 5560: 40, + 5600: 40, + 5610: 40, + 5630: 40, + 5710: 40, + 5720: 40, + 5730: 40, + 5740: 40, + 5810: 40, + 5820: 40, + 5840: 40, + 5850: 40, + 5860: 40, + 5900: 40, + 5910: 40, + 5920: 40, + 5940: 40, + 6005: 41, + 6010: 41, + 6040: 41, + 6050: 41, + 6115: 41, + 6120: 41, + 6130: 41, + 6200: 42, + 6210: 42, + 6220: 42, + 6230: 43, + 6240: 44, + 6250: 44, + 6260: 44, + 6305: 44, + 6330: 44, + 6355: 45, + 6360: 46, + 6400: 46, + 6410: 46, + 6441: 46, + 6442: 46, + 6460: 46, + 6515: 46, + 6520: 46, + 6530: 46, + 6540: 46, + 6600: 46, + 6660: 46, + 6700: 46, + 6710: 46, + 6720: 46, + 6730: 46, + 6740: 46, + 6765: 46, + 6800: 47, + 6825: 47, + 6835: 47, + 6850: 47, + 6950: 47, + 7000: 48, + 7010: 48, + 7020: 48, + 7030: 48, + 7040: 48, + 7100: 48, + 7120: 48, + 7130: 48, + 7140: 48, + 7150: 48, + 7160: 48, + 7200: 48, + 7210: 48, + 7220: 48, + 7240: 48, + 7260: 48, + 7300: 48, + 7315: 48, + 7320: 48, + 7330: 48, + 7340: 48, + 7350: 48, + 7360: 48, + 7410: 48, + 7420: 48, + 7430: 48, + 7510: 48, + 7540: 48, + 7560: 48, + 7610: 48, + 7640: 48, + 7700: 49, + 7720: 49, + 7730: 49, + 7740: 49, + 7750: 49, + 7800: 49, + 7810: 49, + 7830: 49, + 7840: 49, + 7850: 49, + 7855: 49, + 7905: 49, + 7925: 49, + 7950: 49, + 8000: 49, + 8025: 49, + 8030: 49, + 8040: 49, + 8100: 49, + 8130: 49, + 8140: 49, + 8225: 49, + 8250: 49, + 8255: 49, + 8256: 49, + 8300: 49, + 8310: 49, + 8320: 49, + 8335: 49, + 8350: 49, + 8365: 49, + 8450: 49, + 8465: 49, + 8500: 49, + 8510: 49, + 8530: 49, + 8540: 49, + 8555: 49, + 8600: 49, + 8610: 49, + 8620: 49, + 8630: 49, + 8640: 49, + 8650: 49, + 8710: 49, + 8720: 49, + 8730: 49, + 8740: 49, + 8750: 49, + 8760: 49, + 8800: 49, + 8810: 49, + 8830: 49, + 8850: 49, + 8910: 49, + 8920: 49, + 8930: 49, + 8940: 49, + 8950: 49, + 8990: 49, + 9005: 50, + 9030: 50, + 9040: 50, + 9050: 50, + 9110: 51, + 9121: 51, + 9122: 51, + 9130: 51, + 9141: 51, + 9142: 51, + 9150: 51, + 9210: 51, + 9240: 51, + 9265: 51, + 9300: 51, + 9310: 51, + 9350: 51, + 9365: 51, + 9410: 51, + 9415: 51, + 9430: 51, + 9510: 51, + 9570: 51, + 9600: 51, + 9610: 51, + 9620: 51, + 9630: 51, + 9640: 51, + 9645: 51, + 9650: 51, + 9720: 51, + 9760: 51, + 9800: 52, + 9810: 52, + 9825: 52, + 9830: 52, + 9920: 53, +} + +ACS_TEN_TO_SPM_TENMORTSTATUS: Mapping[int, int] = { + 1: 1, # owned with a mortgage or loan + 2: 2, # owned free and clear + 3: 3, # rented for cash + 4: 3, # occupied without cash rent: non-owner consumed bin +} + +_MODEL_PREDICTORS: Mapping[str, tuple[str, ...]] = { + "ssi_disability_criteria": ( + "PEDISDRS", + "PEDISEAR", + "PEDISEYE", + "PEDISOUT", + "PEDISPHY", + "PEDISREM", + ), + "scf_wealth": ("PRDTRACE", "PRDTHSP"), + "scf_auto_loans": ("PRDTRACE", "PRDTHSP"), + "sipp_vehicles": ("SPM_TENMORTSTATUS",), + "sipp_tips": ("PEIOOCC",), + "org_wages": ("PRDTRACE", "PRDTHSP", "POCCU2"), +} +_OUTPUT_COLUMNS = tuple( + dict.fromkeys( + column for columns in _MODEL_PREDICTORS.values() for column in columns + ) +) + + +@dataclass(frozen=True) +class AcsReleasePredictorJoinResult: + """A predictor-enriched frame and JSON-ready release receipt.""" + + frame: Frame + receipt: Mapping[str, Any] + + +def acs_release_predictor_crosswalk_payload() -> dict[str, Any]: + """Return the canonical, JSON-ready crosswalk specification.""" + + return { + "version": ACS_RELEASE_PREDICTOR_CROSSWALK_VERSION, + "disability": { + source: { + "target": ACS_DIFFICULTY_TO_CPS[source], + "minimum_question_age": _ACS_DIFFICULTY_MIN_AGE[source], + "codes": {"1": 1, "2": 2, "below_universe_blank": -1}, + } + for source in ACS_DIFFICULTY_TO_CPS + }, + "race": { + "RAC1P_to_consumed_PRDTRACE": { + str(key): value for key, value in ACS_RAC1P_TO_CONSUMED_PRDTRACE.items() + }, + "HISP_to_consumed_PRDTHSP": { + str(key): value for key, value in _ACS_HISP_TO_CONSUMED_PRDTHSP.items() + }, + }, + "occupation": { + "OCCP_to_PEIOOCC": "identity; blank out-of-universe to 0", + "OCCP_to_POCCU2": { + str(key): value for key, value in ACS_OCCP_TO_POCCU2.items() + }, + "blank_OCCP_to_POCCU2": {"age_below_15": 0, "age_15_plus": 53}, + }, + "tenure": { + "TEN_to_SPM_TENMORTSTATUS": { + str(key): value for key, value in ACS_TEN_TO_SPM_TENMORTSTATUS.items() + }, + "group_quarters_blank": 3, + }, + "ssi_reporter_anchor": { + "source": "person.ssi_reported (native adjusted ACS SSIP)", + "target": "receiver-coalesced reported SSI anchor", + "consumer_semantic": "> 0", + "below_age_15_blank": "preserved", + }, + "model_predictors": { + model: list(columns) for model, columns in _MODEL_PREDICTORS.items() + }, + } + + +def _computed_crosswalk_sha256() -> str: + payload = json.dumps( + acs_release_predictor_crosswalk_payload(), + sort_keys=True, + separators=(",", ":"), + allow_nan=False, + ).encode() + return hashlib.sha256(payload).hexdigest() + + +def join_acs_release_predictors( + frame: Frame, + *, + person_zip: str | Path | None, + person_sha256: str | None, + household_zip: str | Path | None, + household_sha256: str | None, + chunksize: int = _DEFAULT_CHUNKSIZE, +) -> AcsReleasePredictorJoinResult: + """Populate CPS-named predictors for every physical ACS support row. + + Frames without an assembled ACS channel pass through by identity and do + not require archive options. An assembled frame with any ACS row requires + all four explicit CLI values and the two canonical 2024 archive pins. + """ + + if frame.schema != US_SCHEMA: + raise ValueError("ACS release predictor join requires the US schema.") + if chunksize <= 0: + raise ValueError("ACS release predictor join chunksize must be positive.") + person = frame.table("person") + channel_column = support_channel_column("person") + if ( + channel_column not in person + or not person[channel_column].eq(_ACS_CHANNEL).any() + ): + provided = [person_zip, person_sha256, household_zip, household_sha256] + if any(value is not None for value in provided): + raise ValueError( + "ACS release archive options were provided for a frame with no " + "physical ACS source rows." + ) + return AcsReleasePredictorJoinResult( + frame=frame, + receipt={"enabled": False, "reason": "no physical ACS source rows"}, + ) + + missing_options = [ + name + for name, value in ( + ("person_zip", person_zip), + ("person_sha256", person_sha256), + ("household_zip", household_zip), + ("household_sha256", household_sha256), + ) + if value is None + ] + if missing_options: + raise ValueError( + "Physical ACS rows require all pinned release archive options; " + f"missing {missing_options}." + ) + assert person_zip is not None + assert person_sha256 is not None + assert household_zip is not None + assert household_sha256 is not None + + actual_crosswalk_sha256 = _computed_crosswalk_sha256() + if actual_crosswalk_sha256 != ACS_RELEASE_PREDICTOR_CROSSWALK_SHA256: + raise RuntimeError( + "ACS release predictor crosswalk identity is stale: expected " + f"{ACS_RELEASE_PREDICTOR_CROSSWALK_SHA256}, got " + f"{actual_crosswalk_sha256}." + ) + person_identity = _verify_archive( + person_zip, + expected_sha256=person_sha256, + canonical_sha256=ACS_2024_PERSON_ZIP_SHA256, + label="ACS 2024 person zip", + ) + household_identity = _verify_archive( + household_zip, + expected_sha256=household_sha256, + canonical_sha256=ACS_2024_HOUSEHOLD_ZIP_SHA256, + label="ACS 2024 household zip", + ) + + canonical, acs_rows, clone_counts = _canonical_pool_acs_people(frame) + selected_serials = frozenset(canonical["SERIALNO"].astype(str)) + raw_household = _read_filtered_archive( + Path(household_zip), + expected_members=_HOUSEHOLD_MEMBERS, + columns=_HOUSEHOLD_RAW_COLUMNS, + selected_serials=selected_serials, + chunksize=chunksize, + label="ACS household", + ) + tenure_by_serial = _validated_tenure_by_serial(canonical, raw_household) + + raw_person = _read_filtered_archive( + Path(person_zip), + expected_members=_PERSON_MEMBERS, + columns=_PERSON_RAW_COLUMNS, + selected_serials=selected_serials, + chunksize=chunksize, + label="ACS person", + ) + raw_person["SPORDER"] = _required_integral( + raw_person["SPORDER"], label="raw ACS SPORDER", minimum=1 + ) + duplicate_raw_people = raw_person.duplicated(["SERIALNO", "SPORDER"], keep=False) + if duplicate_raw_people.any(): + examples = ( + raw_person.loc[duplicate_raw_people, ["SERIALNO", "SPORDER"]] + .head() + .to_dict("records") + ) + raise ValueError(f"ACS raw person key collision(s): {examples}.") + + joined = canonical.merge( + raw_person, + on=["SERIALNO", "SPORDER"], + how="left", + validate="one_to_one", + indicator=True, + sort=False, + ) + unmatched = joined["_merge"].ne("both") + if unmatched.any(): + examples = ( + joined.loc[unmatched, ["person_source_id", "SERIALNO", "SPORDER"]] + .head() + .to_dict("records") + ) + raise ValueError( + "ACS release person join is not total over pool source people; " + f"unmatched={int(unmatched.sum())}, examples={examples}." + ) + joined = joined.drop(columns="_merge") + if len(joined) != len(canonical): # pragma: no cover - merge validation guard + raise AssertionError("ACS release person join changed canonical row count.") + + mapped = _crosswalk_people(joined) + mapped["SPM_TENMORTSTATUS"] = joined["SERIALNO"].map(tenure_by_serial).to_numpy() + _canonical_ssi_reporter_values(frame, canonical) + if mapped.loc[:, list(_OUTPUT_COLUMNS)].isna().any().any(): + missing = { + column: int(mapped[column].isna().sum()) + for column in _OUTPUT_COLUMNS + if mapped[column].isna().any() + } + raise ValueError( + "ACS release crosswalk must populate every consumed predictor; " + f"missing={missing}." + ) + + tables = {entity: frame.table(entity).copy() for entity in frame.entities} + output_person = tables["person"] + _require_asec_native_predictors(output_person) + by_source = mapped.set_index("person_source_id") + acs_mask = output_person[channel_column].eq(_ACS_CHANNEL) + source_ids = _required_integral( + output_person.loc[acs_mask, support_source_id_column("person")], + label="ACS person_source_id", + minimum=0, + ) + for column in _OUTPUT_COLUMNS: + values = source_ids.map(by_source[column]) + if values.isna().any(): # pragma: no cover - canonical totality guard + raise AssertionError(f"ACS clone fan-out lost {column!r} value(s).") + if column in output_person: + current = pd.to_numeric( + output_person.loc[acs_mask, column], errors="coerce" + ) + observed = current.notna() + if observed.any() and not np.array_equal( + current.loc[observed].to_numpy(dtype=np.float64), + values.loc[observed].to_numpy(dtype=np.float64), + ): + raise ValueError( + "ACS release predictor join refuses to overwrite conflicting " + f"pre-existing ACS {column!r} values." + ) + else: + output_person[column] = np.nan + output_person.loc[acs_mask, column] = values.to_numpy() + + enriched = Frame( + tables, + frame.schema, + {entity: frame.weights_for(entity) for entity in frame.weighted_entities}, + frame.strata, + mass_log=frame.mass_log, + metadata=frame.metadata, + ) + receipt = _receipt( + enriched, + person_identity=person_identity, + household_identity=household_identity, + canonical=canonical, + raw_person_rows=len(raw_person), + raw_household_rows=len(raw_household), + acs_rows=acs_rows, + clone_counts=clone_counts, + ) + return AcsReleasePredictorJoinResult(frame=enriched, receipt=receipt) + + +def _verify_archive( + path: str | Path, + *, + expected_sha256: str, + canonical_sha256: str, + label: str, +) -> dict[str, Any]: + if not isinstance(expected_sha256, str) or not _SHA256_PATTERN.fullmatch( + expected_sha256 + ): + raise ValueError(f"{label} SHA-256 must be 64 lowercase hex characters.") + if expected_sha256 != canonical_sha256: + raise ValueError( + f"{label} pin must be the reviewed {canonical_sha256}; got " + f"{expected_sha256}." + ) + archive = Path(path) + if not archive.is_file(): + raise FileNotFoundError(f"{label} not found: {archive}") + digest = hashlib.sha256() + size = 0 + with archive.open("rb") as stream: + for chunk in iter(lambda: stream.read(1024 * 1024), b""): + digest.update(chunk) + size += len(chunk) + actual = digest.hexdigest() + if actual != expected_sha256: + raise ValueError( + f"{label} SHA-256 mismatch: expected {expected_sha256}, got {actual}." + ) + return {"path": str(archive), "sha256": actual, "size_bytes": size} + + +def _read_filtered_archive( + path: Path, + *, + expected_members: Sequence[str], + columns: Sequence[str], + selected_serials: frozenset[str], + chunksize: int, + label: str, +) -> pd.DataFrame: + pieces: list[pd.DataFrame] = [] + with ZipFile(path) as archive: + csv_members = { + Path(name).name.lower(): name + for name in archive.namelist() + if name.lower().endswith(".csv") + and Path(name) + .name.lower() + .startswith("psam_pus" if "person" in label else "psam_hus") + } + expected = tuple(name.lower() for name in expected_members) + if tuple(sorted(csv_members)) != tuple(sorted(expected)): + raise ValueError( + f"{label} archive members differ: expected {list(expected)}, " + f"got {sorted(csv_members)}." + ) + for basename in expected: + member_name = csv_members[basename] + with archive.open(member_name) as member: + header = pd.read_csv(member, nrows=0).columns.tolist() + missing = sorted(set(columns) - set(header)) + if missing: + raise ValueError( + f"{label} member {member_name!r} missing column(s): {missing}." + ) + with archive.open(member_name) as member: + for chunk in pd.read_csv( + member, + usecols=list(columns), + dtype={"SERIALNO": "string"}, + chunksize=chunksize, + low_memory=False, + ): + retained = chunk.loc[chunk["SERIALNO"].isin(selected_serials)] + if not retained.empty: + pieces.append(retained) + if not pieces: + raise ValueError(f"{label} archive contains no selected pool records.") + result = pd.concat(pieces, ignore_index=True) + if result["SERIALNO"].isna().any(): + raise ValueError(f"{label} selected SERIALNO values must not be blank.") + result["SERIALNO"] = result["SERIALNO"].astype(str) + return result + + +def _canonical_pool_acs_people( + frame: Frame, +) -> tuple[pd.DataFrame, int, dict[str, int]]: + validate_assembly_provenance( + frame, + boundary="ACS release predictor join", + ) + person = frame.table("person") + household = frame.table("household") + person_channel = support_channel_column("person") + household_channel = support_channel_column("household") + person_clone = support_clone_index_column("person") + household_clone = support_clone_index_column("household") + person_source = support_source_id_column("person") + required_person = { + "person_household_id", + "person_spine_source_id", + person_source, + person_channel, + person_clone, + "source_row_id", + "source_year", + "source_household_id", + "source_person_id", + "SPORDER", + "age", + "ssi_reported", + } + required_household = { + "household_id", + "household_spine_source_id", + "household_source_id", + household_channel, + household_clone, + "SERIALNO", + "TEN", + } + missing_person = sorted(required_person - set(person.columns)) + missing_household = sorted(required_household - set(household.columns)) + if missing_person or missing_household: + raise ValueError( + "ACS release predictor join requires complete pool lineage; " + f"missing person={missing_person}, household={missing_household}." + ) + observed_channels = set(person[person_channel].astype(str).unique()) + if observed_channels != {_ASEC_CHANNEL, _ACS_CHANNEL}: + raise ValueError( + "ACS release predictor receipts require exact ASEC/ACS physical " + f"channels; got {sorted(observed_channels)}." + ) + acs_mask = person[person_channel].eq(_ACS_CHANNEL) + acs_rows = int(acs_mask.sum()) + clone_index = _required_integral( + person.loc[acs_mask, person_clone], + label="ACS person clone index", + minimum=0, + ) + source_id = _required_integral( + person.loc[acs_mask, person_source], + label="ACS person_source_id", + minimum=0, + ) + duplicate_clone = pd.DataFrame( + {"person_source_id": source_id, "clone_index": clone_index} + ).duplicated(keep=False) + if duplicate_clone.any(): + examples = ( + pd.DataFrame({"person_source_id": source_id, "clone_index": clone_index}) + .loc[duplicate_clone] + .head() + .to_dict("records") + ) + raise ValueError( + "ACS pool has duplicate (person_source_id, clone_index) " + f"collision(s): {examples}." + ) + + selected = person.loc[ + acs_mask, + [ + "person_household_id", + "person_spine_source_id", + person_source, + person_clone, + "source_row_id", + "source_year", + "source_household_id", + "source_person_id", + "SPORDER", + ], + ].copy() + selected[person_source] = source_id.to_numpy() + selected[person_clone] = clone_index.to_numpy() + selected["_pool_row"] = selected.index + linked = selected.merge( + household.loc[ + :, + [ + "household_id", + "household_spine_source_id", + "household_source_id", + household_channel, + household_clone, + "SERIALNO", + "TEN", + ], + ], + left_on="person_household_id", + right_on="household_id", + how="left", + validate="many_to_one", + indicator=True, + sort=False, + ) + if linked["_merge"].ne("both").any(): + raise ValueError("ACS pool person rows contain orphan household links.") + linked = linked.drop(columns="_merge") + if not linked[household_channel].eq(_ACS_CHANNEL).all(): + raise ValueError("ACS pool person/household physical channels disagree.") + household_clone_values = _required_integral( + linked[household_clone], label="ACS household clone index", minimum=0 + ) + if not np.array_equal( + linked[person_clone].to_numpy(dtype=np.int64), + household_clone_values.to_numpy(dtype=np.int64), + ): + raise ValueError("ACS pool person/household clone indices disagree.") + if linked["SERIALNO"].isna().any(): + raise ValueError("ACS pool household SERIALNO values must be complete.") + linked["SERIALNO"] = linked["SERIALNO"].astype(str) + source_person = _required_integral( + linked["source_person_id"], label="ACS source_person_id", minimum=1 + ) + sporder = _required_integral(linked["SPORDER"], label="ACS pool SPORDER", minimum=1) + if not np.array_equal(source_person.to_numpy(), sporder.to_numpy()): + raise ValueError("ACS pool source_person_id does not equal integral SPORDER.") + source_year = _required_integral( + linked["source_year"], label="ACS source_year", minimum=_ACS_VINTAGE + ) + if not source_year.eq(_ACS_VINTAGE).all(): + raise ValueError("ACS release predictor join is pinned to source_year=2024.") + spine_id = _required_integral( + linked["person_spine_source_id"], + label="ACS person_spine_source_id", + minimum=0, + ) + row_id = _required_integral( + linked["source_row_id"], label="ACS source_row_id", minimum=0 + ) + if not np.array_equal(spine_id.to_numpy(), row_id.to_numpy()): + raise ValueError( + "ACS pool raw ordinal contract failed: person_spine_source_id must " + "equal source_row_id." + ) + source_household = _required_integral( + linked["source_household_id"], + label="ACS source_household_id", + minimum=1, + ) + household_spine = _required_integral( + linked["household_spine_source_id"], + label="ACS household_spine_source_id", + minimum=1, + ) + if not np.array_equal(source_household.to_numpy(), household_spine.to_numpy()): + raise ValueError( + "ACS pool household lineage failed: source_household_id must equal " + "the linked household_spine_source_id." + ) + + invariant_columns = [ + "person_spine_source_id", + "source_row_id", + "source_household_id", + "source_person_id", + "SPORDER", + "SERIALNO", + ] + conflicting_sources = [] + for column in invariant_columns: + counts = linked.groupby(person_source, sort=False)[column].nunique(dropna=False) + if counts.gt(1).any(): + conflicting_sources.extend(counts.index[counts.gt(1)].tolist()[:5]) + if conflicting_sources: + raise ValueError( + "ACS person_source_id maps to conflicting raw identities: " + f"{sorted(set(map(int, conflicting_sources)))[:5]}." + ) + native_counts = linked.loc[linked[person_clone].eq(0), person_source].value_counts() + all_sources = pd.Index(linked[person_source].unique()) + invalid_native = native_counts.reindex(all_sources, fill_value=0).ne(1) + if invalid_native.any(): + raise ValueError( + "Every ACS person_source_id must have exactly one clone-index-zero row." + ) + + canonical = linked.loc[linked[person_clone].eq(0)].copy() + canonical["person_source_id"] = canonical[person_source].astype("int64") + canonical["SPORDER"] = sporder.loc[canonical.index].to_numpy(dtype=np.int64) + duplicate_semantic = canonical.duplicated(["SERIALNO", "SPORDER"], keep=False) + if duplicate_semantic.any(): + examples = ( + canonical.loc[duplicate_semantic, ["SERIALNO", "SPORDER"]] + .head() + .to_dict("records") + ) + raise ValueError(f"ACS pool semantic person key collision(s): {examples}.") + clone_counts = { + str(int(index)): int(count) + for index, count in clone_index.value_counts().sort_index().items() + } + return canonical, acs_rows, clone_counts + + +def _validated_tenure_by_serial( + canonical: pd.DataFrame, + raw_household: pd.DataFrame, +) -> pd.Series: + duplicate = raw_household["SERIALNO"].duplicated(keep=False) + if duplicate.any(): + examples = raw_household.loc[duplicate, "SERIALNO"].head().tolist() + raise ValueError(f"ACS raw household SERIALNO collision(s): {examples}.") + expected_serials = set(canonical["SERIALNO"].astype(str)) + observed_serials = set(raw_household["SERIALNO"].astype(str)) + if observed_serials != expected_serials: + raise ValueError( + "ACS raw household join is not exact over selected serials; " + f"missing={sorted(expected_serials - observed_serials)[:5]}, " + f"extra={sorted(observed_serials - expected_serials)[:5]}." + ) + people = _required_integral(raw_household["NP"], label="ACS NP", minimum=1) + if people.le(0).any(): # pragma: no cover - minimum guard + raise ValueError("Selected ACS households must be occupied.") + kind = _required_integral( + raw_household["TYPEHUGQ"], label="ACS TYPEHUGQ", minimum=1 + ) + if not kind.isin([1, 2, 3]).all(): + bad = sorted(kind.loc[~kind.isin([1, 2, 3])].unique().tolist()) + raise ValueError(f"ACS TYPEHUGQ contains unsupported code(s): {bad}.") + raw_tenure = pd.to_numeric(raw_household["TEN"], errors="coerce") + housing_unit = kind.eq(1) + invalid_hu = housing_unit & ~raw_tenure.isin(ACS_TEN_TO_SPM_TENMORTSTATUS) + invalid_gq = ~housing_unit & raw_tenure.notna() + if invalid_hu.any() or invalid_gq.any(): + raise ValueError( + "ACS TEN/TYPEHUGQ universe mismatch: housing units require TEN 1--4 " + "and group quarters require blank TEN." + ) + + pool = canonical.loc[:, ["SERIALNO", "TEN"]].drop_duplicates("SERIALNO") + if pool["SERIALNO"].duplicated().any(): # pragma: no cover - drop guard + raise AssertionError("Canonical pool serial deduplication failed.") + comparison = pool.merge( + raw_household.loc[:, ["SERIALNO", "TEN"]], + on="SERIALNO", + how="left", + validate="one_to_one", + suffixes=("_pool", "_raw"), + ) + pool_tenure = pd.to_numeric(comparison["TEN_pool"], errors="coerce") + raw_tenure_aligned = pd.to_numeric(comparison["TEN_raw"], errors="coerce") + equal = (pool_tenure.isna() & raw_tenure_aligned.isna()) | pool_tenure.eq( + raw_tenure_aligned + ) + if not equal.all(): + examples = comparison.loc[~equal, ["SERIALNO", "TEN_pool", "TEN_raw"]] + raise ValueError( + "Pool household TEN disagrees with pinned ACS archive; examples=" + f"{examples.head().to_dict('records')}." + ) + + mapped = raw_tenure.map(ACS_TEN_TO_SPM_TENMORTSTATUS) + mapped.loc[~housing_unit] = 3 + if mapped.isna().any(): # pragma: no cover - universe guards above + raise AssertionError("ACS tenure crosswalk produced missing values.") + return pd.Series( + mapped.to_numpy(dtype=np.int16), + index=raw_household["SERIALNO"].astype(str), + ) + + +def _crosswalk_people(joined: pd.DataFrame) -> pd.DataFrame: + result = pd.DataFrame( + {"person_source_id": joined["person_source_id"].to_numpy()}, + index=joined.index, + ) + age = _required_integral(joined["AGEP"], label="ACS AGEP", minimum=0) + for source, target in ACS_DIFFICULTY_TO_CPS.items(): + values = pd.to_numeric(joined[source], errors="coerce") + in_universe = age.ge(_ACS_DIFFICULTY_MIN_AGE[source]) + invalid = (in_universe & ~values.isin([1, 2])) | (~in_universe & values.notna()) + if invalid.any(): + bad = joined.loc[invalid, ["SERIALNO", "SPORDER", "AGEP", source]] + raise ValueError( + f"ACS {source} code/universe mismatch; examples=" + f"{bad.head().to_dict('records')}." + ) + result[target] = np.select( + [values.eq(1), values.eq(2)], + [1, 2], + default=-1, + ).astype(np.int16) + + race = _required_integral(joined["RAC1P"], label="ACS RAC1P", minimum=1) + unknown_race = sorted(set(race) - set(ACS_RAC1P_TO_CONSUMED_PRDTRACE)) + if unknown_race: + raise ValueError(f"ACS RAC1P contains unsupported code(s): {unknown_race}.") + result["PRDTRACE"] = race.map(ACS_RAC1P_TO_CONSUMED_PRDTRACE).to_numpy( + dtype=np.int16 + ) + hisp = _required_integral(joined["HISP"], label="ACS HISP", minimum=1) + unknown_hisp = sorted(set(hisp) - set(_ACS_HISP_TO_CONSUMED_PRDTHSP)) + if unknown_hisp: + raise ValueError(f"ACS HISP contains unsupported code(s): {unknown_hisp}.") + result["PRDTHSP"] = hisp.map(_ACS_HISP_TO_CONSUMED_PRDTHSP).to_numpy(dtype=np.int16) + + occupation = pd.to_numeric(joined["OCCP"], errors="coerce") + employment = pd.to_numeric(joined["ESR"], errors="coerce") + invalid_blank = occupation.isna() & ~(employment.isna() | employment.eq(6)) + if invalid_blank.any(): + bad = joined.loc[invalid_blank, ["SERIALNO", "SPORDER", "OCCP", "ESR"]] + raise ValueError( + "ACS OCCP is blank inside its observed employment universe; examples=" + f"{bad.head().to_dict('records')}." + ) + observed = occupation.notna() + observed_values = occupation.loc[observed].to_numpy(dtype=np.float64) + if not np.equal(observed_values, np.floor(observed_values)).all(): + raise ValueError("ACS OCCP contains non-integer code(s).") + occupation_codes = occupation.fillna(0).astype(np.int64) + unknown_occupation = sorted( + set(occupation_codes.loc[observed]) - set(ACS_OCCP_TO_POCCU2) + ) + if unknown_occupation: + raise ValueError( + f"ACS OCCP contains unsupported code(s): {unknown_occupation}." + ) + result["PEIOOCC"] = occupation_codes.to_numpy(dtype=np.int16) + poccu2 = occupation_codes.map(ACS_OCCP_TO_POCCU2) + # CPS POCCU2 is in universe from age 15 and uses 53 for the no-occupation / + # never-worked consumed bin. ACS OCCP starts at age 16, so age-15 blanks + # also belong to 53; younger children retain the CPS out-of-universe 0. + poccu2.loc[occupation.isna()] = np.where(age.loc[occupation.isna()].ge(15), 53, 0) + result["POCCU2"] = poccu2.to_numpy(dtype=np.int16) + return result + + +def _canonical_ssi_reporter_values( + frame: Frame, + canonical: pd.DataFrame, +) -> np.ndarray: + person = frame.table("person") + source_id_column = support_source_id_column("person") + channel_column = support_channel_column("person") + clone_column = support_clone_index_column("person") + native = person.loc[ + person[channel_column].eq(_ACS_CHANNEL) + & pd.to_numeric(person[clone_column], errors="coerce").eq(0), + [source_id_column, "age", "ssi_reported"], + ].copy() + if native[source_id_column].duplicated().any(): + raise ValueError("ACS native SSI reporter rows collide by person_source_id.") + age = pd.to_numeric(native["age"], errors="coerce") + reported = pd.to_numeric(native["ssi_reported"], errors="coerce") + invalid_blank = reported.isna() & age.ge(15) + invalid_observed = reported.notna() & ( + ~np.isfinite(reported.to_numpy(dtype=np.float64)) | reported.lt(0) + ) + invalid_child = reported.notna() & age.lt(15) + if invalid_blank.any() or invalid_observed.any() or invalid_child.any(): + raise ValueError( + "ACS native ssi_reported violates its age-15 amount universe or " + "finite nonnegative contract." + ) + canonical_ids = set(canonical["person_source_id"].astype(int)) + native_ids = set( + _required_integral( + native[source_id_column], + label="ACS SSI person_source_id", + minimum=0, + ) + ) + if canonical_ids != native_ids: + raise ValueError( + "ACS native ssi_reported source identities do not exactly cover the " + "canonical raw join." + ) + by_source = pd.Series(reported.to_numpy(), index=native[source_id_column]) + aligned = canonical["person_source_id"].map(by_source) + return aligned.to_numpy(dtype=np.float64) + + +def _require_asec_native_predictors(person: pd.DataFrame) -> None: + channel_column = support_channel_column("person") + asec = person[channel_column].eq(_ASEC_CHANNEL) + missing = [column for column in _OUTPUT_COLUMNS if column not in person] + if missing: + raise ValueError( + f"ACS release join requires native ASEC predictor column(s): {missing}." + ) + null_counts = { + column: int(person.loc[asec, column].isna().sum()) + for column in _OUTPUT_COLUMNS + if person.loc[asec, column].isna().any() + } + if null_counts: + raise ValueError( + "ACS release predictor receipt requires complete native ASEC inputs; " + f"null_counts={null_counts}." + ) + + +def _receipt( + frame: Frame, + *, + person_identity: Mapping[str, Any], + household_identity: Mapping[str, Any], + canonical: pd.DataFrame, + raw_person_rows: int, + raw_household_rows: int, + acs_rows: int, + clone_counts: Mapping[str, int], +) -> dict[str, Any]: + person = frame.table("person") + channel = person[support_channel_column("person")].astype(str) + models: dict[str, Any] = {} + for model, predictors in _MODEL_PREDICTORS.items(): + models[model] = { + "predictors": { + predictor: { + "asec_native": int( + (channel.eq(_ASEC_CHANNEL) & person[predictor].notna()).sum() + ), + "acs_joined": int( + (channel.eq(_ACS_CHANNEL) & person[predictor].notna()).sum() + ), + "still_null": int(person[predictor].isna().sum()), + } + for predictor in predictors + } + } + if "SSI_VAL" not in person or "ssi_reported" not in person: + raise AssertionError("Reported SSI receipt columns unexpectedly absent.") + asec_ssi = pd.to_numeric(person["SSI_VAL"], errors="coerce") + acs_ssi = pd.to_numeric(person["ssi_reported"], errors="coerce") + reported_anchor = pd.Series( + np.where(channel.eq(_ASEC_CHANNEL), asec_ssi, acs_ssi), + index=person.index, + ) + models["ssi_disability_criteria"]["predictors"]["reported_ssi_anchor"] = { + "source_columns": { + "asec_native": "SSI_VAL", + "acs_joined": "ssi_reported (native adjusted ACS SSIP)", + }, + "asec_native": int((channel.eq(_ASEC_CHANNEL) & reported_anchor.notna()).sum()), + "acs_joined": int((channel.eq(_ACS_CHANNEL) & reported_anchor.notna()).sum()), + "still_null": int(reported_anchor.isna().sum()), + "null_semantic": ( + "below-age-15 ACS SSIP universe; the receiver's > 0 predicate " + "treats it as false without rewriting the source value" + ), + } + semantic_keys = canonical.loc[:, ["SERIALNO", "SPORDER"]].sort_values( + ["SERIALNO", "SPORDER"], kind="stable" + ) + key_digest = hashlib.sha256( + "".join( + f"{serial}:{int(sporder)}\n" + for serial, sporder in semantic_keys.itertuples(index=False, name=None) + ).encode() + ).hexdigest() + return { + "enabled": True, + "version": 1, + "artifacts": { + "person": dict(person_identity), + "household": dict(household_identity), + }, + "crosswalk": { + "version": ACS_RELEASE_PREDICTOR_CROSSWALK_VERSION, + "sha256": ACS_RELEASE_PREDICTOR_CROSSWALK_SHA256, + }, + "join": { + "semantic_key": ["household.SERIALNO", "person.SPORDER"], + "clone_fanout_key": "person_source_id", + "acs_source_people": int(len(canonical)), + "acs_support_rows": acs_rows, + "acs_support_rows_by_clone_index": dict(clone_counts), + "selected_raw_person_rows": int(raw_person_rows), + "selected_raw_household_rows": int(raw_household_rows), + "unmatched_pool_source_people": 0, + "source_identity_collisions": 0, + "semantic_key_sha256": key_digest, + }, + "count_semantics": { + "asec_native": "physical ASEC rows with an observed predictor", + "acs_joined": "physical ACS rows populated by this exact join", + "still_null": "all remaining rows with a null predictor", + }, + "models": models, + } + + +def _required_integral( + values: pd.Series, + *, + label: str, + minimum: int, +) -> pd.Series: + numeric = pd.to_numeric(values, errors="coerce") + array = numeric.to_numpy(dtype=np.float64) + if ( + numeric.isna().any() + or not np.isfinite(array).all() + or not np.equal(array, np.floor(array)).all() + or (array < minimum).any() + ): + raise ValueError( + f"{label} must contain finite integers greater than or equal to {minimum}." + ) + return pd.Series(array.astype(np.int64), index=values.index) diff --git a/packages/microcosm-build/src/microcosm/build/us_runtime/ssi_disability_criteria.py b/packages/microcosm-build/src/microcosm/build/us_runtime/ssi_disability_criteria.py index 4a6b56611..a939d2b64 100644 --- a/packages/microcosm-build/src/microcosm/build/us_runtime/ssi_disability_criteria.py +++ b/packages/microcosm-build/src/microcosm/build/us_runtime/ssi_disability_criteria.py @@ -19,9 +19,12 @@ The extended-CPS pipeline predicted its ASEC and PUF-support people separately, because the latter carried separately imputed income and asset predictors. We do the same. Direct under-65 ASEC ``SSI_VAL`` reporters are then preserved as -positive anchors; that anchor is not copied onto the PUF channel. An arbitrary -pre-existing criterion column is never trusted: every run recomputes the full -source-backed surface and uses an equality check only for idempotent return. +positive anchors; that anchor is not copied onto the PUF channel. A stacked +ACS row instead contributes its harmonized native ``ssi_reported`` value to the +same ``> 0`` predicate. The row-wise coalesce is source-blind and preserves +the real below-age-15 ACS amount-universe blank. An arbitrary pre-existing +criterion column is never trusted: every run recomputes the full source-backed +surface and uses an equality check only for idempotent return. The full 2023 SIPP public-use file is the same immutable 3.73 GB artifact already pinned by the vehicle and voluntary-filing stages. It contains 39,513 @@ -752,6 +755,64 @@ def _strict_person_aggregate( ) +def _reported_ssi_anchor(person: pd.DataFrame, *, age: np.ndarray) -> np.ndarray: + """Coalesce CPS and harmonized ACS reporter amounts for the ``> 0`` test. + + ``SSI_VAL`` is the measured CPS ASEC amount. ``ssi_reported`` is the + adjusted native ACS SSIP amount produced by ``map_acs_native_inputs``. + Stacked rows carry exactly one of the two. ACS SSIP is out of universe + below age 15, so that genuine blank is interpreted only at predicate time + and is never rewritten into a fake measured zero. + """ + + available = [column for column in ("SSI_VAL", "ssi_reported") if column in person] + if not available: + raise ValueError( + "US SSI disability receiver requires measured SSI_VAL or harmonized " + "ssi_reported for the under-65 reporter anchor." + ) + + numeric: dict[str, pd.Series] = {} + for column in available: + raw = person[column] + values = pd.to_numeric(raw, errors="coerce") + invalid = raw.notna() & values.isna() + finite = np.isfinite(values.fillna(0.0).to_numpy(dtype=np.float64)) + if invalid.any() or not finite.all(): + raise ValueError( + "US SSI disability receiver reported SSI source " + f"{column!r} contains nonnumeric or nonfinite values." + ) + numeric[column] = values + + combined = pd.Series(np.nan, index=person.index, dtype=np.float64) + if "SSI_VAL" in numeric: + combined = numeric["SSI_VAL"].copy() + if "ssi_reported" in numeric: + if "SSI_VAL" in numeric: + both = numeric["SSI_VAL"].notna() & numeric["ssi_reported"].notna() + positivity_conflict = both & numeric["SSI_VAL"].gt(0).ne( + numeric["ssi_reported"].gt(0) + ) + if positivity_conflict.any(): + raise ValueError( + "US SSI disability receiver SSI_VAL and ssi_reported " + "disagree on reporter status." + ) + combined = combined.combine_first(numeric["ssi_reported"]) + + age_values = np.asarray(age, dtype=np.float64) + if len(age_values) != len(combined) or not np.isfinite(age_values).all(): + raise ValueError("US SSI disability receiver age must be finite.") + invalid_blank = combined.isna().to_numpy() & (age_values >= 15.0) + if invalid_blank.any(): + raise ValueError( + "US SSI disability receiver reported SSI amount may be blank only " + "below the ACS age-15 universe." + ) + return combined.fillna(0.0).to_numpy(dtype=np.float64) + + def _person_ssi_disability_predictors(frame: Frame) -> pd.DataFrame: """Build the exact nineteen predictors on every recipient support row.""" @@ -1020,20 +1081,9 @@ def impute_us_ssi_disability_criteria( # The archived direct-CPS pass preserves measured SSI reporters. Its PUF # clone override does not, even though raw ASEC columns were duplicated. - if "SSI_VAL" not in person: - raise ValueError( - "US SSI disability receiver requires measured ASEC SSI_VAL for the " - "under-65 reporter anchor." - ) - reported_ssi = ( - _strict_person_numeric( - person, - ("SSI_VAL",), - label="reported SSI", - ) - > 0.0 - ) - under_65 = receiver["age"].to_numpy(dtype=np.float64) < 65.0 + receiver_age = receiver["age"].to_numpy(dtype=np.float64) + reported_ssi = _reported_ssi_anchor(person, age=receiver_age) > 0.0 + under_65 = receiver_age < 65.0 if has_support_role_metadata(person, entity="person"): channels = support_role_series(person, entity="person") asec = channels.eq(_BASE_ASEC_SUPPORT_CHANNEL).to_numpy() diff --git a/packages/microcosm-build/tests/test_us_acs_release_predictors.py b/packages/microcosm-build/tests/test_us_acs_release_predictors.py new file mode 100644 index 000000000..43d5b9a71 --- /dev/null +++ b/packages/microcosm-build/tests/test_us_acs_release_predictors.py @@ -0,0 +1,374 @@ +from __future__ import annotations + +import hashlib +import io +from pathlib import Path +from zipfile import ZIP_DEFLATED, ZipFile + +import numpy as np +import pandas as pd +import pytest +from pandas.testing import assert_frame_equal + +import microcosm.build.us_runtime.acs_release_predictors as module +from microcosm.build.us_runtime.acs_release_predictors import ( + ACS_OCCP_TO_POCCU2, + ACS_RELEASE_PREDICTOR_CROSSWALK_SHA256, + acs_release_predictor_crosswalk_payload, + join_acs_release_predictors, +) +from microcosm.build.us_runtime.puf_support import clone_us_frame_for_puf_support +from microcosm.build.us_runtime.spine_assembly import assemble_spines +from microcosm.frame import US_SCHEMA, Frame, WeightKind, Weights + +_CPS_PREDICTORS = ( + "PEDISDRS", + "PEDISEAR", + "PEDISEYE", + "PEDISOUT", + "PEDISPHY", + "PEDISREM", + "SSI_VAL", + "PRDTRACE", + "PRDTHSP", + "PEIOOCC", + "POCCU2", + "SPM_TENMORTSTATUS", +) + + +def _source_frame(*, acs: bool) -> Frame: + if acs: + household_ids = np.asarray([1, 2], dtype=np.int64) + person_ids = np.asarray([0, 1, 2], dtype=np.int64) + memberships = np.asarray([1, 1, 2], dtype=np.int64) + ages = np.asarray([4.0, 40.0, 30.0]) + else: + household_ids = np.asarray([1], dtype=np.int64) + person_ids = np.asarray([0], dtype=np.int64) + memberships = np.asarray([1], dtype=np.int64) + ages = np.asarray([35.0]) + + offsets = { + "tax_unit": 100, + "spm_unit": 200, + "family": 300, + "marital_unit": 400, + } + person = pd.DataFrame( + { + "person_id": person_ids, + "person_household_id": memberships, + "age": ages, + } + ) + for entity, offset in offsets.items(): + person[f"person_{entity}_id"] = memberships + offset + + household = pd.DataFrame({"household_id": household_ids}) + tables: dict[str, pd.DataFrame] = {"person": person, "household": household} + for entity, offset in offsets.items(): + tables[entity] = pd.DataFrame({f"{entity}_id": household_ids + offset}) + + if acs: + person["source_row_id"] = person_ids + person["source_year"] = 2024 + person["source_household_id"] = memberships + person["source_person_id"] = ["1", "2", "1"] + person["SPORDER"] = [1, 2, 1] + person["ssi_reported"] = [np.nan, 900.0, 0.0] + household["SERIALNO"] = ["2024HU0000001", "2024GQ0000002"] + household["TEN"] = [1.0, np.nan] + else: + for column in _CPS_PREDICTORS: + person[column] = 2.0 + person["SSI_VAL"] = 0.0 + person["PRDTRACE"] = 1.0 + person["PRDTHSP"] = 0.0 + person["PEIOOCC"] = 1005.0 + person["POCCU2"] = 8.0 + person["SPM_TENMORTSTATUS"] = 1.0 + + return Frame( + tables, + US_SCHEMA, + { + "household": Weights( + np.ones(len(household_ids), dtype=np.float64), + WeightKind.DESIGN, + ) + }, + pd.Series(["acs_2024_1yr" if acs else "asec_2024"] * len(person)), + ) + + +def _stacked_frame() -> Frame: + assembled = assemble_spines( + {"asec": _source_frame(acs=False), "acs": _source_frame(acs=True)}, + household_mass_shares={"asec": 0.5, "acs": 0.5}, + ) + return clone_us_frame_for_puf_support(assembled) + + +def _raw_person() -> pd.DataFrame: + return pd.DataFrame( + { + "SERIALNO": ["2024HU0000001", "2024HU0000001", "2024GQ0000002"], + "SPORDER": [1, 2, 1], + "AGEP": [4, 40, 30], + "DEAR": [2, 1, 2], + "DEYE": [2, 2, 2], + "DREM": [np.nan, 2, 2], + "DPHY": [np.nan, 2, 2], + "DDRS": [np.nan, 2, 2], + "DOUT": [np.nan, 2, 2], + "RAC1P": [1, 6, 2], + "HISP": [1, 1, 2], + "OCCP": [np.nan, 1005, 9800], + "ESR": [np.nan, 1, 4], + } + ) + + +def _raw_household() -> pd.DataFrame: + return pd.DataFrame( + { + "SERIALNO": ["2024HU0000001", "2024GQ0000002"], + "NP": [2, 1], + "TYPEHUGQ": [1, 2], + "TEN": [1.0, np.nan], + } + ) + + +def _write_zip(path: Path, members: dict[str, pd.DataFrame]) -> str: + with ZipFile(path, "w", compression=ZIP_DEFLATED) as archive: + for member, table in members.items(): + buffer = io.StringIO() + table.to_csv(buffer, index=False) + archive.writestr(member, buffer.getvalue()) + return hashlib.sha256(path.read_bytes()).hexdigest() + + +def _archives( + tmp_path: Path, + *, + person: pd.DataFrame | None = None, + household: pd.DataFrame | None = None, +) -> tuple[Path, str, Path, str]: + tmp_path.mkdir(parents=True, exist_ok=True) + raw_person = _raw_person() if person is None else person + raw_household = _raw_household() if household is None else household + person_path = tmp_path / "csv_pus.zip" + household_path = tmp_path / "csv_hus.zip" + person_sha = _write_zip( + person_path, + { + "psam_pusa.csv": raw_person.iloc[:2], + "psam_pusb.csv": raw_person.iloc[2:], + }, + ) + household_sha = _write_zip( + household_path, + { + "psam_husa.csv": raw_household.iloc[:1], + "psam_husb.csv": raw_household.iloc[1:], + }, + ) + return person_path, person_sha, household_path, household_sha + + +def _join( + frame: Frame, + archives: tuple[Path, str, Path, str], + monkeypatch: pytest.MonkeyPatch, +): + person_path, person_sha, household_path, household_sha = archives + monkeypatch.setattr(module, "ACS_2024_PERSON_ZIP_SHA256", person_sha) + monkeypatch.setattr(module, "ACS_2024_HOUSEHOLD_ZIP_SHA256", household_sha) + return join_acs_release_predictors( + frame, + person_zip=person_path, + person_sha256=person_sha, + household_zip=household_path, + household_sha256=household_sha, + chunksize=1, + ) + + +def test_crosswalk_digest_and_every_consumed_occupation_bin_are_pinned() -> None: + assert module._computed_crosswalk_sha256() == ( + ACS_RELEASE_PREDICTOR_CROSSWALK_SHA256 + ) + assert len(ACS_OCCP_TO_POCCU2) == 530 + assert set(ACS_OCCP_TO_POCCU2.values()) == set(range(1, 54)) + assert ACS_OCCP_TO_POCCU2[3250] == 26 + assert ACS_OCCP_TO_POCCU2[3255] == 25 + assert ACS_OCCP_TO_POCCU2[1005] == 8 + assert ACS_OCCP_TO_POCCU2[6005] == 41 + assert ACS_OCCP_TO_POCCU2[9800] == 52 + assert ACS_OCCP_TO_POCCU2[9920] == 53 + + payload = acs_release_predictor_crosswalk_payload() + assert payload["disability"]["DREM"]["minimum_question_age"] == 5 + assert payload["disability"]["DOUT"]["minimum_question_age"] == 15 + assert payload["disability"]["DOUT"]["codes"]["below_universe_blank"] == -1 + assert payload["race"]["RAC1P_to_consumed_PRDTRACE"] == { + "1": 1, + "2": 2, + "3": 3, + "4": 3, + "5": 3, + "6": 4, + "7": 3, + "8": 3, + "9": 3, + } + + +def test_release_join_is_exact_total_clone_stable_and_receipted( + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, +) -> None: + frame = _stacked_frame() + before = frame.table("person").copy(deep=True) + + result = _join(frame, _archives(tmp_path), monkeypatch) + person = result.frame.table("person") + asec = person["person_support_channel"].eq("asec") + acs = person["person_support_channel"].eq("acs") + + assert_frame_equal( + person.loc[asec, list(_CPS_PREDICTORS)], + before.loc[asec, list(_CPS_PREDICTORS)], + ) + assert person.loc[acs, list(module._OUTPUT_COLUMNS)].notna().all().all() + assert person.loc[acs, "SSI_VAL"].isna().all() + assert person.loc[acs, "ssi_reported"].isna().sum() == 2 + + by_source = person.loc[ + acs, + [ + "person_source_id", + "PEDISDRS", + "PRDTRACE", + "PRDTHSP", + "PEIOOCC", + "POCCU2", + "SPM_TENMORTSTATUS", + ], + ].drop_duplicates("person_source_id") + assert sorted(by_source["PEDISDRS"].tolist()) == [-1.0, 2.0, 2.0] + assert sorted(by_source["PRDTRACE"].tolist()) == [1.0, 2.0, 4.0] + assert sorted(by_source["PRDTHSP"].tolist()) == [0.0, 0.0, 1.0] + assert sorted(by_source["PEIOOCC"].tolist()) == [0.0, 1005.0, 9800.0] + assert sorted(by_source["POCCU2"].tolist()) == [0.0, 8.0, 52.0] + assert sorted(by_source["SPM_TENMORTSTATUS"].tolist()) == [1.0, 1.0, 3.0] + + for _, clones in person.loc[acs].groupby("person_source_id"): + for column in module._OUTPUT_COLUMNS: + assert clones[column].nunique(dropna=False) == 1 + + receipt = result.receipt + assert receipt["crosswalk"]["sha256"] == ACS_RELEASE_PREDICTOR_CROSSWALK_SHA256 + assert receipt["join"] == { + "semantic_key": ["household.SERIALNO", "person.SPORDER"], + "clone_fanout_key": "person_source_id", + "acs_source_people": 3, + "acs_support_rows": 6, + "acs_support_rows_by_clone_index": {"0": 3, "1": 3}, + "selected_raw_person_rows": 3, + "selected_raw_household_rows": 2, + "unmatched_pool_source_people": 0, + "source_identity_collisions": 0, + "semantic_key_sha256": receipt["join"]["semantic_key_sha256"], + } + race_counts = receipt["models"]["scf_wealth"]["predictors"]["PRDTRACE"] + assert race_counts == {"asec_native": 2, "acs_joined": 6, "still_null": 0} + reporter = receipt["models"]["ssi_disability_criteria"]["predictors"][ + "reported_ssi_anchor" + ] + assert reporter["asec_native"] == 2 + assert reporter["acs_joined"] == 4 + assert reporter["still_null"] == 2 + + repeated = _join(result.frame, _archives(tmp_path / "again"), monkeypatch) + assert_frame_equal(repeated.frame.table("person"), person) + + +def test_release_join_refuses_a_missing_raw_person( + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, +) -> None: + raw = _raw_person().iloc[:-1].copy() + with pytest.raises(ValueError, match="not total over pool source people"): + _join( + _stacked_frame(), + _archives(tmp_path, person=raw), + monkeypatch, + ) + + +def test_release_join_refuses_raw_person_key_collision( + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, +) -> None: + raw = pd.concat([_raw_person(), _raw_person().iloc[[0]]], ignore_index=True) + with pytest.raises(ValueError, match="raw person key collision"): + _join( + _stacked_frame(), + _archives(tmp_path, person=raw), + monkeypatch, + ) + + +def test_release_join_refuses_source_identity_collision( + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, +) -> None: + frame = _stacked_frame() + person = frame.table("person") + acs = person["person_support_channel"].eq("acs") + source_id = person.loc[acs, "person_source_id"].iloc[0] + clone = ( + acs + & person["person_source_id"].eq(source_id) + & person["person_support_clone_index"].eq(1) + ) + person.loc[clone, "SPORDER"] = 9 + + with pytest.raises(ValueError, match="source_person_id does not equal"): + _join(frame, _archives(tmp_path), monkeypatch) + + +def test_release_join_verifies_pin_before_opening_archive( + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, +) -> None: + person_path, person_sha, household_path, household_sha = _archives(tmp_path) + monkeypatch.setattr(module, "ACS_2024_PERSON_ZIP_SHA256", "0" * 64) + monkeypatch.setattr(module, "ACS_2024_HOUSEHOLD_ZIP_SHA256", household_sha) + with pytest.raises(ValueError, match="pin must be the reviewed"): + join_acs_release_predictors( + _stacked_frame(), + person_zip=person_path, + person_sha256=person_sha, + household_zip=household_path, + household_sha256=household_sha, + ) + + +def test_no_acs_frame_is_an_identity_without_archive_options() -> None: + frame = _source_frame(acs=False) + result = join_acs_release_predictors( + frame, + person_zip=None, + person_sha256=None, + household_zip=None, + household_sha256=None, + ) + assert result.frame is frame + assert result.receipt == { + "enabled": False, + "reason": "no physical ACS source rows", + } diff --git a/packages/microcosm-build/tests/test_us_spine_blindness.py b/packages/microcosm-build/tests/test_us_spine_blindness.py index 98c59af52..5c1892d55 100644 --- a/packages/microcosm-build/tests/test_us_spine_blindness.py +++ b/packages/microcosm-build/tests/test_us_spine_blindness.py @@ -94,6 +94,8 @@ { # Declares and receipts exact ACS source universes; never mutates rows. "acs_income_universe.py", + # Owner-approved release boundary: exact raw ACS join and receipt. + "acs_release_predictors.py", "base_pool.py", # Legacy late-spine assembly. # Enumerates provenance columns only to reject preassembled source frames. "operator_boundary.py", @@ -216,6 +218,7 @@ "acs_inputs.py", "acs_multispine.py", "acs_pums.py", + "acs_release_predictors.py", # Pinned release join; provenance owner. "acs_sources.py", "acs_transfer_bank.py", # Bounded checkpoint I/O; no population treatment. "asec_checkpoint.py", # Bounded checkpoint I/O; no population treatment. @@ -3110,9 +3113,7 @@ def _function_callers(source: str, callee: str) -> tuple[tuple[str, int], ...]: if not isinstance(node, ast.ImportFrom): continue aliases.update( - alias.asname or alias.name - for alias in node.names - if alias.name == callee + alias.asname or alias.name for alias in node.names if alias.name == callee ) class CallerVisitor(ast.NodeVisitor): @@ -3355,9 +3356,10 @@ def test_physical_source_accessor_is_confined_to_reviewed_gates() -> None: f"or wrapper use is forbidden. Found callers: {call_details}" ) - for (module_name, helper), expected_callers in ( - _GATE_SOURCE_SCOPE_HELPER_CALLERS.items() - ): + for ( + module_name, + helper, + ), expected_callers in _GATE_SOURCE_SCOPE_HELPER_CALLERS.items(): callers = _function_callers((_US_RUNTIME / module_name).read_text(), helper) actual_callers = frozenset(caller for caller, _line in callers) assert actual_callers == expected_callers, ( diff --git a/packages/microcosm-build/tests/test_us_ssi_disability_criteria.py b/packages/microcosm-build/tests/test_us_ssi_disability_criteria.py index adf0e19ca..2a4aeaf8f 100644 --- a/packages/microcosm-build/tests/test_us_ssi_disability_criteria.py +++ b/packages/microcosm-build/tests/test_us_ssi_disability_criteria.py @@ -47,6 +47,35 @@ ) +def test_reported_ssi_anchor_coalesces_native_asec_and_harmonized_acs() -> None: + person = pd.DataFrame( + { + "SSI_VAL": [1_200.0, np.nan, np.nan], + "ssi_reported": [np.nan, 900.0, np.nan], + } + ) + + values = module._reported_ssi_anchor( + person, + age=np.asarray([40.0, 50.0, 10.0]), + ) + + np.testing.assert_array_equal(values, [1_200.0, 900.0, 0.0]) + assert pd.isna(person.loc[2, "ssi_reported"]) + + +def test_reported_ssi_anchor_refuses_an_adult_universe_blank() -> None: + person = pd.DataFrame( + { + "SSI_VAL": [np.nan], + "ssi_reported": [np.nan], + } + ) + + with pytest.raises(ValueError, match="blank only below"): + module._reported_ssi_anchor(person, age=np.asarray([40.0])) + + def _source_row( ssuid: str, pnum: int, From a7108697222f76015bf86033660c77449111e524 Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Thu, 27 Aug 2026 15:12:02 +0200 Subject: [PATCH 37/49] Harden ACS predictor crosswalk contracts --- PROGRESS.md | 35 ++-- .../us_runtime/acs_release_predictors.py | 156 +++++++++++++++--- .../us_runtime/ssi_disability_criteria.py | 24 +-- .../tests/test_us_acs_release_predictors.py | 61 ++++++- .../tests/test_us_ssi_disability_criteria.py | 25 +++ 5 files changed, 255 insertions(+), 46 deletions(-) diff --git a/PROGRESS.md b/PROGRESS.md index 22356a698..3bf72d0ea 100644 --- a/PROGRESS.md +++ b/PROGRESS.md @@ -65,33 +65,48 @@ push, retraining, threshold change, or launcher-contract edit is in scope. fans mapped values to clones only through `person_source_id`. - Added explicit disability, race/Hispanic, 530-code occupation, and tenure tables. A canonical crosswalk payload is pinned at SHA-256 - `cf21e20831dd15479e8f5704743dc5e22e5b8a8b78546107ba5024f22d8f3f1b` + `1d4906242e9c73e31b3283659e5cad8242b8cbc42914ab6fa59547a10c8770e9` and rides the JSON-ready join receipt with per-model/per-predictor ASEC-native, ACS-joined, and still-null counts. - Preserved CPS disability universe semantics (`-1` below the question age) - and the POCCU2 age universe (0 below 15, consumed no-occupation code 53 for - older ACS OCCP blanks). The explicit occupation table covers every one of - the 530 codes in the pinned ACS person archive and every consumed POCCU2 bin. + and the ACS occupation universe. Blank `PEIOOCC` uses the CPS NIU sentinel + `-1`; blank `POCCU2` remains 0 through age 15 and maps to the consumed + no-occupation code 53 only from age 16. This explicitly preserves the + one-year ACS/CPS source-universe gap instead of assigning every ACS + 15-year-old a never-worked status without source evidence. The explicit + occupation table covers every one of the 530 codes in the pinned ACS person + archive and every consumed POCCU2 bin. - Changed the SSI-disability reporter read, without source routing, to row-wise coalesce measured ASEC `SSI_VAL` with harmonized native ACS `ssi_reported`. Adult blanks and conflicting dual reporters fail; genuine below-age-15 ACS blanks remain null in the frame and become false only for the consumer's `> 0` predicate. +- Hardened the join after independent crosswalk review: raw ACS `SSIP` and + `ADJINC` now travel through the pinned join and must agree exactly with every + native clone-0 `ssi_reported` value under the established adjusted-dollar + formula. Raw `ESR`/`OCCP` must obey their exact age-16 universes, and all + ASEC predictor receipt cells must be numeric and finite with complete, + nonnegative `SSI_VAL`. +- Updated the SSI signal diagnostic to use the same row-wise reporter coalesce + as the model consumer, while retaining the archived native-role anchor + scope. A lost positive ACS-native reporter can therefore no longer evade the + release gate merely because `SSI_VAL` is null on physical ACS rows. - Added focused tests for crosswalk identity/all consumed bins, exact join and clone invariance, ASEC byte preservation, receipt contents, missing joins, raw and source-identity collisions, hash refusal, no-ACS identity, and SSI - coalescing/universe refusal. The new join file passes all 7 tests; the SSI - and source-blindness suites passed alongside it before the final join-only - merge-indicator repair, and focused Ruff is green. + coalescing/universe refusal. Coverage now also fixes the age-15 occupation + gap, malformed ESR refusal, malformed ASEC SSI refusal, raw SSI attestation, + and gate-side ACS reporter preservation. The complete join, SSI, and + source-blindness test files pass together, and focused Ruff is green. ## Next - Add the paired release CLI inputs and carry the receipt into both build and release manifests before the six archived model stages, without changing their feature/selection logic or any gate threshold. -- Re-run the complete focused SSI/source-blindness group, exercise the join - read-only against the supplied pool and canonical zips, add the changelog - fragment, then run full shard verification and write `out.md`. +- Exercise the join read-only against the supplied pool and canonical zips, + add the changelog fragment, then run full shard verification and write + `out.md`. # Weeksgate: stacked release gates and integer-week provenance diff --git a/packages/microcosm-build/src/microcosm/build/us_runtime/acs_release_predictors.py b/packages/microcosm-build/src/microcosm/build/us_runtime/acs_release_predictors.py index 7fcd0210c..51501e33d 100644 --- a/packages/microcosm-build/src/microcosm/build/us_runtime/acs_release_predictors.py +++ b/packages/microcosm-build/src/microcosm/build/us_runtime/acs_release_predictors.py @@ -75,7 +75,7 @@ ) ACS_RELEASE_PREDICTOR_CROSSWALK_VERSION = 1 ACS_RELEASE_PREDICTOR_CROSSWALK_SHA256 = ( - "cf21e20831dd15479e8f5704743dc5e22e5b8a8b78546107ba5024f22d8f3f1b" + "1d4906242e9c73e31b3283659e5cad8242b8cbc42914ab6fa59547a10c8770e9" ) _SHA256_PATTERN = re.compile(r"[0-9a-f]{64}") @@ -100,6 +100,8 @@ "HISP", "OCCP", "ESR", + "SSIP", + "ADJINC", ) _HOUSEHOLD_RAW_COLUMNS = ("SERIALNO", "NP", "TYPEHUGQ", "TEN") @@ -149,7 +151,8 @@ # used by the archived ORG/FLSA consumer. The table is intentionally explicit: # the canonical archive pin and crosswalk digest jointly refuse a new Census # code until its consumed category is reviewed. Blank OCCP is handled -# separately: PEIOOCC uses 0, while POCCU2 preserves its age universe. +# separately: PEIOOCC uses the CPS NIU sentinel -1, while POCCU2 preserves +# the ACS age universe rather than inventing an occupation for 15-year-olds. ACS_OCCP_TO_POCCU2: Mapping[int, int] = { 10: 1, 20: 1, @@ -742,11 +745,22 @@ def acs_release_predictor_crosswalk_payload() -> dict[str, Any]: }, }, "occupation": { - "OCCP_to_PEIOOCC": "identity; blank out-of-universe to 0", + "OCCP_to_PEIOOCC": { + "observed": "identity", + "blank": -1, + "blank_semantic": "CPS not-in-universe sentinel", + }, "OCCP_to_POCCU2": { str(key): value for key, value in ACS_OCCP_TO_POCCU2.items() }, - "blank_OCCP_to_POCCU2": {"age_below_15": 0, "age_15_plus": 53}, + "blank_OCCP_to_POCCU2": { + "age_below_16": 0, + "age_16_plus": 53, + "age_15_source_target_universe_gap": ( + "ACS OCCP is not asked; retain the out-of-universe sentinel " + "instead of fabricating CPS no-occupation code 53" + ), + }, }, "tenure": { "TEN_to_SPM_TENMORTSTATUS": { @@ -910,7 +924,7 @@ def join_acs_release_predictors( mapped = _crosswalk_people(joined) mapped["SPM_TENMORTSTATUS"] = joined["SERIALNO"].map(tenure_by_serial).to_numpy() - _canonical_ssi_reporter_values(frame, canonical) + _validate_canonical_ssi_reporter_values(frame, canonical, joined) if mapped.loc[:, list(_OUTPUT_COLUMNS)].isna().any().any(): missing = { column: int(mapped[column].isna().sum()) @@ -1377,20 +1391,52 @@ def _crosswalk_people(joined: pd.DataFrame) -> pd.DataFrame: raise ValueError(f"ACS HISP contains unsupported code(s): {unknown_hisp}.") result["PRDTHSP"] = hisp.map(_ACS_HISP_TO_CONSUMED_PRDTHSP).to_numpy(dtype=np.int16) - occupation = pd.to_numeric(joined["OCCP"], errors="coerce") - employment = pd.to_numeric(joined["ESR"], errors="coerce") - invalid_blank = occupation.isna() & ~(employment.isna() | employment.eq(6)) + occupation_raw = joined["OCCP"] + occupation = pd.to_numeric(occupation_raw, errors="coerce") + employment_raw = joined["ESR"] + employment = pd.to_numeric(employment_raw, errors="coerce") + invalid_employment = ( + (employment_raw.notna() & employment.isna()) + | (age.lt(16) & employment.notna()) + | (age.ge(16) & ~employment.isin(range(1, 7))) + ) + if invalid_employment.any(): + bad = joined.loc[invalid_employment, ["SERIALNO", "SPORDER", "AGEP", "ESR"]] + raise ValueError( + "ACS ESR code/universe mismatch (blank below age 16; codes 1--6 " + "from age 16); examples=" + f"{bad.head().to_dict('records')}." + ) + invalid_occupation = occupation_raw.notna() & occupation.isna() + invalid_observed_universe = occupation.notna() & age.lt(16) + invalid_blank = occupation.isna() & age.ge(16) & ~employment.eq(6) + if invalid_occupation.any() or invalid_observed_universe.any(): + bad_mask = invalid_occupation | invalid_observed_universe + bad = joined.loc[ + bad_mask, + ["SERIALNO", "SPORDER", "AGEP", "OCCP", "ESR"], + ] + raise ValueError( + "ACS OCCP code/universe mismatch (blank below age 16); examples=" + f"{bad.head().to_dict('records')}." + ) if invalid_blank.any(): - bad = joined.loc[invalid_blank, ["SERIALNO", "SPORDER", "OCCP", "ESR"]] + bad = joined.loc[ + invalid_blank, + ["SERIALNO", "SPORDER", "AGEP", "OCCP", "ESR"], + ] raise ValueError( "ACS OCCP is blank inside its observed employment universe; examples=" f"{bad.head().to_dict('records')}." ) observed = occupation.notna() observed_values = occupation.loc[observed].to_numpy(dtype=np.float64) - if not np.equal(observed_values, np.floor(observed_values)).all(): - raise ValueError("ACS OCCP contains non-integer code(s).") - occupation_codes = occupation.fillna(0).astype(np.int64) + if ( + not np.isfinite(observed_values).all() + or not np.equal(observed_values, np.floor(observed_values)).all() + ): + raise ValueError("ACS OCCP contains nonfinite or non-integer code(s).") + occupation_codes = occupation.fillna(-1).astype(np.int64) unknown_occupation = sorted( set(occupation_codes.loc[observed]) - set(ACS_OCCP_TO_POCCU2) ) @@ -1399,19 +1445,21 @@ def _crosswalk_people(joined: pd.DataFrame) -> pd.DataFrame: f"ACS OCCP contains unsupported code(s): {unknown_occupation}." ) result["PEIOOCC"] = occupation_codes.to_numpy(dtype=np.int16) - poccu2 = occupation_codes.map(ACS_OCCP_TO_POCCU2) - # CPS POCCU2 is in universe from age 15 and uses 53 for the no-occupation / - # never-worked consumed bin. ACS OCCP starts at age 16, so age-15 blanks - # also belong to 53; younger children retain the CPS out-of-universe 0. - poccu2.loc[occupation.isna()] = np.where(age.loc[occupation.isna()].ge(15), 53, 0) + poccu2 = occupation.map(ACS_OCCP_TO_POCCU2) + # ACS OCCP starts at age 16, one year later than CPS POCCU2. Preserve the + # ACS NIU state for age 15 instead of fabricating no-occupation evidence. + # From age 16, a blank is admitted only for ESR=6 and maps to the consumed + # no-occupation / never-worked bin 53. + poccu2.loc[occupation.isna()] = np.where(age.loc[occupation.isna()].ge(16), 53, 0) result["POCCU2"] = poccu2.to_numpy(dtype=np.int16) return result -def _canonical_ssi_reporter_values( +def _validate_canonical_ssi_reporter_values( frame: Frame, canonical: pd.DataFrame, -) -> np.ndarray: + joined: pd.DataFrame, +) -> None: person = frame.table("person") source_id_column = support_source_id_column("person") channel_column = support_channel_column("person") @@ -1449,21 +1497,66 @@ def _canonical_ssi_reporter_values( "canonical raw join." ) by_source = pd.Series(reported.to_numpy(), index=native[source_id_column]) - aligned = canonical["person_source_id"].map(by_source) - return aligned.to_numpy(dtype=np.float64) + pool_aligned = canonical["person_source_id"].map(by_source).to_numpy( + dtype=np.float64 + ) + + raw_ssip = pd.to_numeric(joined["SSIP"], errors="coerce") + raw_adjinc = pd.to_numeric(joined["ADJINC"], errors="coerce") + raw_age = _required_integral(joined["AGEP"], label="ACS AGEP", minimum=0) + invalid_ssip = ( + (joined["SSIP"].notna() & raw_ssip.isna()) + | (raw_ssip.notna() & ~np.isfinite(raw_ssip.fillna(0.0))) + | raw_ssip.lt(0) + | (raw_age.lt(15) & raw_ssip.notna()) + | (raw_age.ge(15) & raw_ssip.isna()) + ) + invalid_adjinc = raw_ssip.notna() & ( + raw_adjinc.isna() | ~np.isfinite(raw_adjinc.fillna(0.0)) | raw_adjinc.le(0) + ) + if invalid_ssip.any() or invalid_adjinc.any(): + invalid = invalid_ssip | invalid_adjinc + bad = joined.loc[ + invalid, + ["SERIALNO", "SPORDER", "AGEP", "SSIP", "ADJINC"], + ] + raise ValueError( + "ACS raw SSIP/ADJINC violates the age-15 adjusted-dollar contract; " + f"examples={bad.head().to_dict('records')}." + ) + raw_aligned = raw_ssip.to_numpy(dtype=np.float64) * ( + raw_adjinc.to_numpy(dtype=np.float64) / 1_000_000.0 + ) + equal = (np.isnan(pool_aligned) & np.isnan(raw_aligned)) | np.equal( + pool_aligned, + raw_aligned, + ) + if not equal.all(): + bad = joined.loc[ + ~equal, + ["person_source_id", "SERIALNO", "SPORDER", "SSIP", "ADJINC"], + ].copy() + bad["ssi_reported_pool"] = pool_aligned[~equal] + bad["ssi_reported_raw"] = raw_aligned[~equal] + raise ValueError( + "ACS native ssi_reported disagrees with pinned raw " + "SSIP * ADJINC / 1_000_000; examples=" + f"{bad.head().to_dict('records')}." + ) def _require_asec_native_predictors(person: pd.DataFrame) -> None: channel_column = support_channel_column("person") asec = person[channel_column].eq(_ASEC_CHANNEL) - missing = [column for column in _OUTPUT_COLUMNS if column not in person] + required = (*_OUTPUT_COLUMNS, "SSI_VAL") + missing = [column for column in required if column not in person] if missing: raise ValueError( f"ACS release join requires native ASEC predictor column(s): {missing}." ) null_counts = { column: int(person.loc[asec, column].isna().sum()) - for column in _OUTPUT_COLUMNS + for column in required if person.loc[asec, column].isna().any() } if null_counts: @@ -1471,6 +1564,23 @@ def _require_asec_native_predictors(person: pd.DataFrame) -> None: "ACS release predictor receipt requires complete native ASEC inputs; " f"null_counts={null_counts}." ) + malformed: dict[str, int] = {} + for column in required: + values = person.loc[asec, column] + numeric = pd.to_numeric(values, errors="coerce") + invalid = numeric.isna() | ~np.isfinite(numeric.to_numpy(dtype=np.float64)) + if not pd.api.types.is_numeric_dtype(values.dtype): + invalid = pd.Series(True, index=values.index) + if column == "SSI_VAL": + invalid |= numeric.lt(0) + if invalid.any(): + malformed[column] = int(invalid.sum()) + if malformed: + raise ValueError( + "ACS release predictor receipt requires numeric finite native ASEC " + "inputs and nonnegative SSI_VAL; malformed_counts=" + f"{malformed}." + ) def _receipt( diff --git a/packages/microcosm-build/src/microcosm/build/us_runtime/ssi_disability_criteria.py b/packages/microcosm-build/src/microcosm/build/us_runtime/ssi_disability_criteria.py index a939d2b64..fa7492764 100644 --- a/packages/microcosm-build/src/microcosm/build/us_runtime/ssi_disability_criteria.py +++ b/packages/microcosm-build/src/microcosm/build/us_runtime/ssi_disability_criteria.py @@ -1200,16 +1200,16 @@ def us_ssi_disability_criteria_summary(frame: Frame) -> dict[str, object]: ) clone_divergence_source_people = int((unique > 1).sum()) - reporter_mismatches = 0 - if "SSI_VAL" in person: - reported = pd.to_numeric(person["SSI_VAL"], errors="coerce").fillna(0.0) > 0.0 - age_column = "age" if "age" in person else "A_AGE" - age = pd.to_numeric(person[age_column], errors="coerce") - asec = pd.Series(True, index=person.index) - if channel_values is not None: - asec = channel_values.eq(_BASE_ASEC_SUPPORT_CHANNEL) - anchor = (reported & age.lt(65.0) & asec).to_numpy() - reporter_mismatches = int(np.count_nonzero(anchor & ~positive)) + age_column = "age" if "age" in person else "A_AGE" + age = pd.to_numeric(person[age_column], errors="coerce").to_numpy( + dtype=np.float64 + ) + reported = _reported_ssi_anchor(person, age=age) > 0.0 + native_role = np.ones(len(person), dtype=bool) + if channel_values is not None: + native_role = channel_values.eq(_BASE_ASEC_SUPPORT_CHANNEL).to_numpy() + anchor = reported & (age < 65.0) & native_role + reporter_mismatches = int(np.count_nonzero(anchor & ~positive)) return { "weighted_true_share": float(weights[positive].sum()) / total_weight, @@ -1263,8 +1263,8 @@ def us_ssi_disability_criteria_signal_gate(frame: Frame) -> GateResult: failures.append(f"{_OUTPUT}: weighted false total is not positive.") if summary["reporter_anchor_mismatches"]: failures.append( - f"{_OUTPUT}: {summary['reporter_anchor_mismatches']} under-65 ASEC " - "SSI reporter anchor(s) were lost." + f"{_OUTPUT}: {summary['reporter_anchor_mismatches']} under-65 " + "native-role SSI reporter anchor(s) were lost." ) if summary["support_provenance_missing"]: failures.append( diff --git a/packages/microcosm-build/tests/test_us_acs_release_predictors.py b/packages/microcosm-build/tests/test_us_acs_release_predictors.py index 43d5b9a71..195152220 100644 --- a/packages/microcosm-build/tests/test_us_acs_release_predictors.py +++ b/packages/microcosm-build/tests/test_us_acs_release_predictors.py @@ -126,6 +126,8 @@ def _raw_person() -> pd.DataFrame: "HISP": [1, 1, 2], "OCCP": [np.nan, 1005, 9800], "ESR": [np.nan, 1, 4], + "SSIP": [np.nan, 900.0, 0.0], + "ADJINC": [1_000_000, 1_000_000, 1_000_000], } ) @@ -261,7 +263,7 @@ def test_release_join_is_exact_total_clone_stable_and_receipted( assert sorted(by_source["PEDISDRS"].tolist()) == [-1.0, 2.0, 2.0] assert sorted(by_source["PRDTRACE"].tolist()) == [1.0, 2.0, 4.0] assert sorted(by_source["PRDTHSP"].tolist()) == [0.0, 0.0, 1.0] - assert sorted(by_source["PEIOOCC"].tolist()) == [0.0, 1005.0, 9800.0] + assert sorted(by_source["PEIOOCC"].tolist()) == [-1.0, 1005.0, 9800.0] assert sorted(by_source["POCCU2"].tolist()) == [0.0, 8.0, 52.0] assert sorted(by_source["SPM_TENMORTSTATUS"].tolist()) == [1.0, 1.0, 3.0] @@ -296,6 +298,63 @@ def test_release_join_is_exact_total_clone_stable_and_receipted( assert_frame_equal(repeated.frame.table("person"), person) +def test_occupation_crosswalk_preserves_age_15_source_universe_gap() -> None: + rows = pd.concat([_raw_person().iloc[[1]]] * 2, ignore_index=True) + rows["SERIALNO"] = ["age15", "age16"] + rows["SPORDER"] = [1, 1] + rows["AGEP"] = [15, 16] + rows["OCCP"] = np.nan + rows["ESR"] = [np.nan, 6] + rows["person_source_id"] = [1, 2] + + mapped = module._crosswalk_people(rows) + + assert mapped["PEIOOCC"].tolist() == [-1, -1] + assert mapped["POCCU2"].tolist() == [0, 53] + + +def test_occupation_crosswalk_refuses_malformed_esr_universe() -> None: + row = _raw_person().iloc[[1]].copy() + row["ESR"] = "not-a-code" + row["person_source_id"] = 1 + + with pytest.raises(ValueError, match="ESR code/universe mismatch"): + module._crosswalk_people(row) + + +@pytest.mark.parametrize("bad_ssi", [np.nan, -1.0, np.inf, "not-a-number"]) +def test_release_join_refuses_malformed_native_asec_ssi( + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, + bad_ssi: object, +) -> None: + frame = _stacked_frame() + person = frame.table("person") + asec = person["person_support_channel"].eq("asec") + if isinstance(bad_ssi, str): + person["SSI_VAL"] = person["SSI_VAL"].astype(object) + person.loc[asec, "SSI_VAL"] = bad_ssi + + with pytest.raises(ValueError, match="native ASEC inputs"): + _join(frame, _archives(tmp_path), monkeypatch) + + +def test_release_join_attests_native_ssi_against_raw_archive( + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, +) -> None: + frame = _stacked_frame() + person = frame.table("person") + acs_native = person["person_support_channel"].eq("acs") & person[ + "person_support_clone_index" + ].eq(0) + adult = acs_native & person["age"].eq(40) + person.loc[adult, "ssi_reported"] = 901.0 + + with pytest.raises(ValueError, match="ssi_reported disagrees"): + _join(frame, _archives(tmp_path), monkeypatch) + + def test_release_join_refuses_a_missing_raw_person( tmp_path: Path, monkeypatch: pytest.MonkeyPatch, diff --git a/packages/microcosm-build/tests/test_us_ssi_disability_criteria.py b/packages/microcosm-build/tests/test_us_ssi_disability_criteria.py index 2a4aeaf8f..ca9362ccc 100644 --- a/packages/microcosm-build/tests/test_us_ssi_disability_criteria.py +++ b/packages/microcosm-build/tests/test_us_ssi_disability_criteria.py @@ -630,6 +630,31 @@ def test_stacked_clone_divergence_diagnostic_checks_clone_two() -> None: assert summary["clone_divergence_source_people"] == 1 +def test_summary_checks_harmonized_ssi_on_native_role() -> None: + expanded = clone_us_frame_for_puf_support(_frame()) + person = expanded.table("person") + person["ssi_reported"] = np.nan + native = person["person_support_channel"].astype(str).eq("asec") + source_two = person["person_source_id"].eq(2) + person.loc[native & source_two, "SSI_VAL"] = np.nan + person.loc[native & source_two, "ssi_reported"] = 900.0 + preserved_existing_anchor = ( + native & person["person_source_id"].eq(1) + ).to_numpy() + invalid = _replace_person( + expanded, + **{_OUTPUT: preserved_existing_anchor}, + ) + + summary = us_ssi_disability_criteria_summary(invalid) + gate = us_ssi_disability_criteria_signal_gate(invalid) + + assert summary["reporter_anchor_mismatches"] == 1 + assert any( + "native-role SSI reporter anchor" in failure for failure in gate.failures + ) + + def test_gate_requires_complete_support_provenance( monkeypatch: pytest.MonkeyPatch, ) -> None: From 6b8185e42c78c6b1bbde15341334812ce5bd6831 Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Thu, 27 Aug 2026 15:18:01 +0200 Subject: [PATCH 38/49] Wire ACS predictor join into release builder --- PROGRESS.md | 15 +- .../tests/test_us_fiscal_refresh_builder.py | 138 ++++++++++++++++++ tools/build_us_fiscal_refresh_release.py | 84 +++++++++++ 3 files changed, 234 insertions(+), 3 deletions(-) diff --git a/PROGRESS.md b/PROGRESS.md index 3bf72d0ea..680bf9f0b 100644 --- a/PROGRESS.md +++ b/PROGRESS.md @@ -99,11 +99,20 @@ push, retraining, threshold change, or launcher-contract edit is in scope. and gate-side ACS reporter preservation. The complete join, SSI, and source-blindness test files pass together, and focused Ruff is green. +- The release CLI now accepts the person/household zip and lowercase 64-hex + SHA-256 options as an all-or-none set. It invokes the authenticated join + after the last unrelated native-input gate and before SCF wealth, therefore + before all six archived donor-model stages, then carries the complete join + receipt into both `build_manifest.json` and `release_manifest.json`. +- Added parser refusal tests, a source-order contract over all six model calls, + an end-to-end mocked main corridor that verifies the exact four join + arguments and runtime ordering, an AST contract that binds the saved receipt + to the sole manifest call, and JSON round-trip assertions for both manifests. + Focused Ruff, five parser/order/manifest cases, and all six parametrized main + corridor cases pass. + ## Next -- Add the paired release CLI inputs and carry the receipt into both build and - release manifests before the six archived model stages, without changing - their feature/selection logic or any gate threshold. - Exercise the join read-only against the supplied pool and canonical zips, add the changelog fragment, then run full shard verification and write `out.md`. diff --git a/packages/microcosm-build/tests/test_us_fiscal_refresh_builder.py b/packages/microcosm-build/tests/test_us_fiscal_refresh_builder.py index 3c76b4620..0794037d1 100644 --- a/packages/microcosm-build/tests/test_us_fiscal_refresh_builder.py +++ b/packages/microcosm-build/tests/test_us_fiscal_refresh_builder.py @@ -2006,6 +2006,96 @@ def test_sipp_vehicle_donor_override_parses(monkeypatch) -> None: assert args.sipp_vehicle_donor == Path("pu2023.csv") +def test_acs_release_archive_options_parse_as_one_pinned_pair() -> None: + builder = _load_builder_module() + args = builder._parse_args( + [ + "--ledger-facts", + "facts.jsonl", + "--out", + "release", + "--acs-person-zip", + "csv_pus.zip", + "--acs-person-sha256", + "a" * 64, + "--acs-household-zip", + "csv_hus.zip", + "--acs-household-sha256", + "b" * 64, + ] + ) + + assert args.acs_person_zip == Path("csv_pus.zip") + assert args.acs_person_sha256 == "a" * 64 + assert args.acs_household_zip == Path("csv_hus.zip") + assert args.acs_household_sha256 == "b" * 64 + + +@pytest.mark.parametrize( + "extra", + [ + ["--acs-person-zip", "csv_pus.zip"], + [ + "--acs-person-zip", + "csv_pus.zip", + "--acs-person-sha256", + "A" * 64, + "--acs-household-zip", + "csv_hus.zip", + "--acs-household-sha256", + "b" * 64, + ], + ], +) +def test_acs_release_archive_options_fail_closed(extra: list[str]) -> None: + builder = _load_builder_module() + with pytest.raises(SystemExit): + builder._parse_args( + [ + "--ledger-facts", + "facts.jsonl", + "--out", + "release", + *extra, + ] + ) + + +def test_acs_predictor_join_precedes_all_six_archived_model_stages() -> None: + import ast + import inspect + + builder = _load_builder_module() + source = inspect.getsource(builder._main) + join_position = source.index("join_acs_release_predictors(") + + for stage in ( + "with_us_scf_wealth_inputs(", + "with_us_ssi_disability_criteria(", + "with_us_scf_auto_loan_inputs(", + "with_us_sipp_vehicle_inputs(", + "with_us_sipp_tip_inputs(", + "with_us_org_wages_inputs(", + ): + assert join_position < source.index(stage) + + manifest_calls = [ + node + for node in ast.walk(ast.parse(source)) + if isinstance(node, ast.Call) + and isinstance(node.func, ast.Name) + and node.func.id == "_build_manifests" + ] + assert len(manifest_calls) == 1 + receipt_keyword = next( + keyword + for keyword in manifest_calls[0].keywords + if keyword.arg == "acs_predictor_join" + ) + assert isinstance(receipt_keyword.value, ast.Name) + assert receipt_keyword.value.id == "acs_predictor_join_receipt" + + def test_scf_full_extract_override_parses(monkeypatch) -> None: builder = _load_builder_module() monkeypatch.setattr( @@ -4151,6 +4241,8 @@ def test_main_writes_diagnostics_before_post_calibration_gate_failure( base_h5 = tmp_path / "base.h5" pool_manifest = tmp_path / "pool.manifest.json" weeks_source = tmp_path / "asecpub23csv.zip" + acs_person_zip = tmp_path / "csv_pus.zip" + acs_household_zip = tmp_path / "csv_hus.zip" facts = tmp_path / "facts.jsonl" out = tmp_path / "out" base_h5.write_bytes(b"h5") @@ -4286,6 +4378,16 @@ def table(self, entity): ] if terminal_mode not in {"telemetry", "puf_tail"}: argv.append("--no-staging") + argv += [ + "--acs-person-zip", + str(acs_person_zip), + "--acs-person-sha256", + "d" * 64, + "--acs-household-zip", + str(acs_household_zip), + "--acs-household-sha256", + "f" * 64, + ] if terminal_mode == "crash": # Nonexistent incumbent: the degraded-mode guard must record the # load failure, null the path for the writer (no re-hash replay of @@ -5032,6 +5134,20 @@ def fake_retirement_distributions_signal_gate(frame): details={"checked": True}, ), ) + + def fake_join_acs_release_predictors(frame, **kwargs): + captured["source_stage_events"].append("acs_predictor_join") + captured["acs_predictor_join_kwargs"] = kwargs + return SimpleNamespace( + frame=frame, + receipt={"enabled": True, "join": {"acs_source_people": 3}}, + ) + + monkeypatch.setattr( + builder, + "join_acs_release_predictors", + fake_join_acs_release_predictors, + ) monkeypatch.setattr( builder, "fetch_scf_2022_summary_extract", @@ -5068,6 +5184,7 @@ def fake_load_sipp_financial_asset_donor( def fake_with_scf_wealth_inputs( frame, *, seed, time_period, scf_donor, sipp_donor=None ): + captured["source_stage_events"].append("scf_wealth") captured["sipp_scf_wealth_blend_called"] = sipp_donor is not None return frame @@ -6121,6 +6238,15 @@ def fake_release_gate_failures(*args, **kwargs): assert captured["weeks_unemployed_stage_period"] == builder.PERIOD assert isinstance(captured["weeks_unemployed_stage_source"], pd.DataFrame) assert captured["weeks_unemployed_gate_called"] is True + assert captured["acs_predictor_join_kwargs"] == { + "person_zip": acs_person_zip, + "person_sha256": "d" * 64, + "household_zip": acs_household_zip, + "household_sha256": "f" * 64, + } + assert captured["source_stage_events"].index("acs_predictor_join") < captured[ + "source_stage_events" + ].index("scf_wealth") assert captured["source_stage_events"].index("weeks_stage") < captured[ "source_stage_events" ].index("ssi_reporters") @@ -8836,10 +8962,22 @@ def __len__(self): "n_exported_households": 57_240, "l0_lambda_share": 0.8, }, + acs_predictor_join={ + "enabled": True, + "crosswalk": {"sha256": "c" * 64}, + "join": {"acs_source_people": 856_626}, + }, ) manifest = json.loads((release_dir / "release_manifest.json").read_text()) build_manifest = json.loads((release_dir / "build_manifest.json").read_text()) + expected_acs_join = { + "enabled": True, + "crosswalk": {"sha256": "c" * 64}, + "join": {"acs_source_people": 856_626}, + } + assert build_manifest["acs_predictor_join"] == expected_acs_join + assert manifest["build"]["acs_predictor_join"] == expected_acs_join assert build_manifest["gates"]["target_profile_coverage"]["passed"] assert ( build_manifest["gates"]["target_profile_coverage"]["details"][ diff --git a/tools/build_us_fiscal_refresh_release.py b/tools/build_us_fiscal_refresh_release.py index df0634b20..8c1ed2c75 100644 --- a/tools/build_us_fiscal_refresh_release.py +++ b/tools/build_us_fiscal_refresh_release.py @@ -209,6 +209,9 @@ write_us_ssi_take_up_diagnostics, write_us_take_up_participation_diagnostics, ) +from microcosm.build.us_runtime.acs_release_predictors import ( + join_acs_release_predictors, +) from microcosm.build.us_runtime.demographics import ( CENSUS_NATIONAL_AGE_BENCHMARK, demographics_payload, @@ -1287,6 +1290,32 @@ def _parse_args(argv: Sequence[str] | None = None) -> argparse.Namespace: "official Census archive is fetched and verified." ), ) + parser.add_argument( + "--acs-person-zip", + type=Path, + help=( + "Local canonical 2024 one-year ACS PUMS person CSV zip used to " + "populate release-model predictors on physical ACS rows. Requires " + "--acs-person-sha256 and both household archive options." + ), + ) + parser.add_argument( + "--acs-person-sha256", + help="Expected SHA-256 of --acs-person-zip.", + ) + parser.add_argument( + "--acs-household-zip", + type=Path, + help=( + "Local canonical 2024 one-year ACS PUMS household CSV zip used by " + "the release predictor join. Requires the other three ACS archive " + "options." + ), + ) + parser.add_argument( + "--acs-household-sha256", + help="Expected SHA-256 of --acs-household-zip.", + ) parser.add_argument( "--scf-summary-extract", dest="scf_summary_extract", @@ -1555,6 +1584,30 @@ def _parse_args(argv: Sequence[str] | None = None) -> argparse.Namespace: parser.error("--evidence-failure-owners requires --evidence-release.") if args.allow_gate_failed_base_pool and args.base_h5 is None: parser.error("--allow-gate-failed-base-pool requires --base-h5.") + acs_archive_options = ( + args.acs_person_zip, + args.acs_person_sha256, + args.acs_household_zip, + args.acs_household_sha256, + ) + if any(value is not None for value in acs_archive_options) and any( + value is None for value in acs_archive_options + ): + parser.error( + "--acs-person-zip, --acs-person-sha256, --acs-household-zip, and " + "--acs-household-sha256 must be provided together." + ) + for flag, value in ( + ("--acs-person-sha256", args.acs_person_sha256), + ("--acs-household-sha256", args.acs_household_sha256), + ): + if value is not None and ( + len(value) != 64 + or any(character not in "0123456789abcdef" for character in value) + ): + parser.error( + f"{flag} must be exactly 64 lowercase hexadecimal characters." + ) if args.evidence_release and args.exact_k is not None: parser.error( "--evidence-release is incompatible with --exact-k: ladder " @@ -7514,6 +7567,7 @@ def _build_manifests( calibration_filename: str = CALIBRATION_FILENAME, exact_k_ladder: Mapping[str, object] | None = None, base_pool: Mapping[str, object] | None = None, + acs_predictor_join: Mapping[str, object] | None = None, evidence_known_failures: Sequence[Mapping[str, str]] | None = None, ) -> None: dataset_path = artifact_root / dataset_filename @@ -7581,6 +7635,11 @@ def _build_manifests( else {} ), **({"base_pool": dict(base_pool)} if base_pool is not None else {}), + **( + {"acs_predictor_join": dict(acs_predictor_join)} + if acs_predictor_join is not None + else {} + ), "dataset": { "filename": dataset_filename, "sha256": dataset_sha, @@ -7805,6 +7864,11 @@ def _build_manifests( else {} ), **({"base_pool": dict(base_pool)} if base_pool is not None else {}), + **( + {"acs_predictor_join": dict(acs_predictor_join)} + if acs_predictor_join is not None + else {} + ), "warm_start_calibration": warm_start_payload, "selection_source": selection_source_payload, "default_dataset": default_dataset_payload, @@ -9761,6 +9825,25 @@ def _main(argv: Sequence[str] | None = None) -> None: for failure in snap_discretionary_exemption_gate.failures ) ) + acs_predictor_join_result = join_acs_release_predictors( + base_frame, + person_zip=args.acs_person_zip, + person_sha256=args.acs_person_sha256, + household_zip=args.acs_household_zip, + household_sha256=args.acs_household_sha256, + ) + base_frame = acs_predictor_join_result.frame + acs_predictor_join_receipt = dict(acs_predictor_join_result.receipt) + if telemetry is not None: + telemetry.stage( + "acs_predictor_join", + message=( + "Authenticated and joined ACS release predictors before all " + "six archived donor-model stages." + ), + enabled=bool(acs_predictor_join_receipt.get("enabled")), + join=dict(acs_predictor_join_receipt.get("join", {})), + ) if telemetry is not None: telemetry.stage( "scf_wealth_inputs", @@ -11792,6 +11875,7 @@ def _main(argv: Sequence[str] | None = None) -> None: calibration_filename=calibration_filename, exact_k_ladder=exact_k_ladder_provenance, base_pool=base_pool_receipt, + acs_predictor_join=acs_predictor_join_receipt, evidence_known_failures=evidence_known_failures, ) if telemetry is not None: From d87be06817033ce0e2b82f1092bb7652196a1428 Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Thu, 27 Aug 2026 15:30:29 +0200 Subject: [PATCH 39/49] Accept numeric H5 predictor cells --- PROGRESS.md | 24 +++++++++++++++++-- .../us_runtime/acs_release_predictors.py | 9 ++++--- .../tests/test_us_acs_release_predictors.py | 14 +++++++++++ 3 files changed, 42 insertions(+), 5 deletions(-) diff --git a/PROGRESS.md b/PROGRESS.md index 680bf9f0b..5f3250733 100644 --- a/PROGRESS.md +++ b/PROGRESS.md @@ -110,11 +110,31 @@ push, retraining, threshold change, or launcher-contract edit is in scope. to the sole manifest call, and JSON round-trip assertions for both manifests. Focused Ruff, five parser/order/manifest cases, and all six parametrized main corridor cases pass. +- Exercised the hardened join read-only on the complete supplied candidate. + The 3,239,263,147-byte H5 matches its frozen manifest SHA-256 + `871b7e6467675a1e9475b54fd1baf64c53c0f75a3258b8357303a8df0d53642d`. + The current official loader refuses that older candidate before H5 loading + because its archived primary-QRF worker binding predates this branch's + execution identity; this is an existing candidate/code-version mismatch. + Loading those already-authenticated bytes with their frozen assembly receipt + allowed the join boundary itself to be tested without writing an artifact. +- The real join passed every source, raw-key, universe, SSI-attestation, + totality, collision, and clone-fan-out check: 856,626 unique ACS source + people matched 856,626 raw people in 382,903 households and populated + 1,736,840 support rows (856,626 each at clone indices 0 and 1, plus 23,588 + at clone index 2). Every CPS-named predictor consumed by the six models has + 234,133 valid ASEC-native and 1,736,840 ACS-joined cells with zero nulls. + The logical SSI reporter anchor has 234,133 ASEC cells, 1,475,235 observed + ACS cells, and exactly 261,605 preserved child-universe null support rows. +- The first real-data attempt exposed fixed-format HDF's expected object dtype + for mixed-source columns. Tightened the ASEC validator to inspect each cell, + accepting object-wrapped real numbers while still refusing strings, + nonfinite values, nulls, and negative SSI. A focused H5-shape regression and + all 15 join tests pass before the successful full-pool rerun. ## Next -- Exercise the join read-only against the supplied pool and canonical zips, - add the changelog fragment, then run full shard verification and write +- Add the changelog fragment, then run full shard verification and write `out.md`. # Weeksgate: stacked release gates and integer-week provenance diff --git a/packages/microcosm-build/src/microcosm/build/us_runtime/acs_release_predictors.py b/packages/microcosm-build/src/microcosm/build/us_runtime/acs_release_predictors.py index 51501e33d..98ba4e4d9 100644 --- a/packages/microcosm-build/src/microcosm/build/us_runtime/acs_release_predictors.py +++ b/packages/microcosm-build/src/microcosm/build/us_runtime/acs_release_predictors.py @@ -1568,9 +1568,12 @@ def _require_asec_native_predictors(person: pd.DataFrame) -> None: for column in required: values = person.loc[asec, column] numeric = pd.to_numeric(values, errors="coerce") - invalid = numeric.isna() | ~np.isfinite(numeric.to_numpy(dtype=np.float64)) - if not pd.api.types.is_numeric_dtype(values.dtype): - invalid = pd.Series(True, index=values.index) + numeric_cells = values.map(pd.api.types.is_number) + invalid = ( + ~numeric_cells + | numeric.isna() + | ~np.isfinite(numeric.to_numpy(dtype=np.float64)) + ) if column == "SSI_VAL": invalid |= numeric.lt(0) if invalid.any(): diff --git a/packages/microcosm-build/tests/test_us_acs_release_predictors.py b/packages/microcosm-build/tests/test_us_acs_release_predictors.py index 195152220..0c298aaea 100644 --- a/packages/microcosm-build/tests/test_us_acs_release_predictors.py +++ b/packages/microcosm-build/tests/test_us_acs_release_predictors.py @@ -355,6 +355,20 @@ def test_release_join_attests_native_ssi_against_raw_archive( _join(frame, _archives(tmp_path), monkeypatch) +def test_release_join_accepts_h5_object_wrapped_native_numbers( + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, +) -> None: + frame = _stacked_frame() + person = frame.table("person") + for column in (*module._OUTPUT_COLUMNS, "SSI_VAL"): + person[column] = person[column].astype(object) + + result = _join(frame, _archives(tmp_path), monkeypatch) + + assert result.receipt["enabled"] is True + + def test_release_join_refuses_a_missing_raw_person( tmp_path: Path, monkeypatch: pytest.MonkeyPatch, From 69ec0fae9d5a989fdb8c45f326221ecb00c0bdc7 Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Thu, 27 Aug 2026 15:30:52 +0200 Subject: [PATCH 40/49] Document ACS release predictor join --- PROGRESS.md | 7 +++++-- changelog.d/acs-release-predictor-join.fixed.md | 1 + 2 files changed, 6 insertions(+), 2 deletions(-) create mode 100644 changelog.d/acs-release-predictor-join.fixed.md diff --git a/PROGRESS.md b/PROGRESS.md index 5f3250733..841f921df 100644 --- a/PROGRESS.md +++ b/PROGRESS.md @@ -131,11 +131,14 @@ push, retraining, threshold change, or launcher-contract edit is in scope. accepting object-wrapped real numbers while still refusing strings, nonfinite values, nulls, and negative SSI. A focused H5-shape regression and all 15 join tests pass before the successful full-pool rerun. +- Added `changelog.d/acs-release-predictor-join.fixed.md`, describing the + pinned release join, reviewed mappings, fail-closed lineage, dual-manifest + receipts, and unchanged model/gate behavior. ## Next -- Add the changelog fragment, then run full shard verification and write - `out.md`. +- Run repository Ruff, the CI inventory verifier, and every full pytest shard + in its own process; then write and commit `out.md` and the completed journal. # Weeksgate: stacked release gates and integer-week provenance diff --git a/changelog.d/acs-release-predictor-join.fixed.md b/changelog.d/acs-release-predictor-join.fixed.md new file mode 100644 index 000000000..987305594 --- /dev/null +++ b/changelog.d/acs-release-predictor-join.fixed.md @@ -0,0 +1 @@ +Populate physical ACS rows' CPS-named inputs for the six archived US release donor models through an exact, SHA-pinned 2024 ACS person/household join; enforce reviewed disability, consumed race/Hispanic, occupation, tenure, and SSI-reporter semantics; refuse ambiguous, partial, or conflicting lineage; and receipt ASEC-native, ACS-joined, and remaining-null counts in both release manifests without changing model selection logic or gate thresholds. From 01a80f49fe92d904b2fe61cafab63623b8564b8f Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Thu, 27 Aug 2026 15:51:05 +0200 Subject: [PATCH 41/49] Enforce structural pregnancy transfer eligibility --- PROGRESS.md | 49 +- ...798-pregnancy-structural-transfer.fixed.md | 1 + .../799-prior-year-rung-floor.fixed.md | 1 + docs/evidence/spec-engine/us-f0-coverage.json | 70 +-- .../build/spec_engine/field_usage.py | 12 +- .../build/spec_engine/imputation_semantics.py | 19 + .../build/spec_engine/inventory_coverage.py | 20 +- .../spec_engine/schema/imputation.schema.json | 106 ++++ .../microcosm/build/us/spec/imputation.yaml | 41 ++ .../build/us_runtime/acs_transfer.py | 553 +++++++++++++++++- .../microcosm/build/us_runtime/pregnancy.py | 193 +++++- .../build/us_runtime/stacked_spine.py | 95 ++- .../us_runtime/us_late_producer_registry.py | 13 + .../tests/test_spec_engine_compiler_ir.py | 2 +- .../tests/test_spec_engine_coverage_tool.py | 14 +- .../tests/test_spec_engine_field_usage.py | 12 +- .../test_spec_engine_imputation_semantics.py | 2 +- .../test_spec_engine_inventory_coverage.py | 2 +- .../tests/test_spec_engine_legacy_adapter.py | 4 +- ...spec_engine_stacked_authority_semantics.py | 6 +- .../tests/test_us_acs_transfer.py | 290 +++++++++ .../tests/test_us_multispine_pool_tool.py | 2 +- .../tests/test_us_pregnancy.py | 63 ++ .../tests/test_us_spec_bundle.py | 19 +- .../tests/test_us_stacked_spine.py | 133 ++++- tools/spec_engine_coverage.py | 2 +- tools/us_bundle_generation/imputation.py | 11 +- 27 files changed, 1602 insertions(+), 133 deletions(-) create mode 100644 changelog.d/798-pregnancy-structural-transfer.fixed.md create mode 100644 changelog.d/799-prior-year-rung-floor.fixed.md diff --git a/PROGRESS.md b/PROGRESS.md index 77d835003..d9b76876f 100644 --- a/PROGRESS.md +++ b/PROGRESS.md @@ -6,9 +6,11 @@ Resumed on 2026-08-27 on branch `pregnancy-prioryear-defects`, based on `stacked-release-gate-alignment` at `606cbd69`. The root-cause findings are adopted and implementation is in progress for the nonfemale-pregnancy producer defect and the owner-approved rung-aware prior-year availability release floor. -The authored 0.05 floor, upper bound, all other bands, thresholds, seeds, and -batteries remain unchanged. This lane will not build, publish, or push pool or -release artifacts. +Both fixes are implemented and their focused runtime suites are green; the +source-attested spec/coverage repin and full repository verification are in +progress. The authored 0.05 floor, upper bound, all other bands, thresholds, +seeds, and batteries remain unchanged. This lane will not build, publish, or +push pool or release artifacts. ## Done @@ -53,18 +55,41 @@ release artifacts. original gate. The prior-year and H5 focused suites pass (23 and 60 tests), focused Ruff and `git diff --check` pass, and the real candidate-25 manifest validates at factor 0.25. +- Implemented pregnancy's hard female-age-15--44 policy before any requested + pregnancy QRF. The transfer validates donor and recipient structure up + front, draws only one eligible clone-0 representative per assembled source + person, fans that value to missing clones, assigns structural false to + ineligible missing rows, and refuses preexisting/final domain or clone + disagreement with explicit counts. +- Added a sealed structural receipt with disjoint QRF, clone-fanout, + preexisting-value-fanout, and ineligible-false accounting. Production receipt + validation authenticates the policy digest, zero-violation postconditions, + source-person topology, and exact equality to the transferred row count. +- Bound the policy into the transfer execution contract used by late-stage + checkpoint/target-bank identity, isolated pregnancy from unrelated bounded + QRF families, declared its structural source-person input, and regenerated + the authored US imputation spec/schema projection. +- Closed two adversarial restart cases: partial ineligible clone surfaces no + longer double-count receipt categories, and a complete pregnancy surface is + still preflighted and carries a zero-imputation structural proof even when a + different requested family remains active. +- Added source, transfer, gate, receipt, identity, all-ineligible, clone-fanout, + mixed-active, and stacked-execution regressions. The complete pregnancy and + ACS-transfer files, the complete stacked-spine file before the final + preflight refactor, and focused post-refactor tests pass; focused Ruff and + `git diff --check` pass. +- Extended the real-pool pregnancy decomposition: the 108 pregnant nonfemale + rows are joined by 58 pregnant female rows outside ages 15--44, for 166 hard- + domain violations, all on ACS and all isolated to one clone. ASEC has zero. ## Next -- Implement hard pregnancy eligibility before QRF, one source-person result - fanned across clones, refusing pre-existing/final domain violations and - recording structural counts. -- Bind the pregnancy structural policy into checkpointed transfer execution - identity and add transfer/source/gate regressions. -- Complete and focused-test the pregnancy source/transfer/gate structural - policy, receipt authentication, and checkpoint execution identity. -- Run the charter's repository Ruff and independent full-shard pytest boundary, - then write the complete handoff to `out.md`. +- Complete the source-attested spec/coverage repin and rerun all directly + affected spec, transfer, and stacked suites. +- Commit the coherent pregnancy implementation and journal state. +- Run repository Ruff, generated/coverage/inventory checks, and all five pytest + shards in independent processes. +- Write and commit the complete handoff to `out.md`. # Weeksgate: stacked release gates and integer-week provenance diff --git a/changelog.d/798-pregnancy-structural-transfer.fixed.md b/changelog.d/798-pregnancy-structural-transfer.fixed.md new file mode 100644 index 000000000..e03221672 --- /dev/null +++ b/changelog.d/798-pregnancy-structural-transfer.fixed.md @@ -0,0 +1 @@ +Make US pregnancy structurally female-age eligible before ACS transfer: draw once for each eligible source person, fan the result across support clones, assign false to ineligible missing recipients, and refuse preexisting or final domain/clone violations with authenticated counts. Bind the policy to the checkpointed transfer identity and carry its proof in the stacked transfer receipt. diff --git a/changelog.d/799-prior-year-rung-floor.fixed.md b/changelog.d/799-prior-year-rung-floor.fixed.md new file mode 100644 index 000000000..0fbb0e266 --- /dev/null +++ b/changelog.d/799-prior-year-rung-floor.fixed.md @@ -0,0 +1 @@ +Scale only the prior-year-income availability gate's authored lower floor by the authenticated stacked sampling rung, recording the applied match-survival factor and floor on sampled assemblies while preserving the authored 0.05 constant, upper bound, every other band, and byte-identical full-rung output. diff --git a/docs/evidence/spec-engine/us-f0-coverage.json b/docs/evidence/spec-engine/us-f0-coverage.json index c3f6bd0a9..5a98d725c 100644 --- a/docs/evidence/spec-engine/us-f0-coverage.json +++ b/docs/evidence/spec-engine/us-f0-coverage.json @@ -6,7 +6,7 @@ "country": "us", "documentation_sha256": "4b39450dbdb8dafb83c3b627123b8026c6f82c660b66fe76f341a67c4f37c77b", "field_usage": { - "authored_normative_field_count": 32352, + "authored_normative_field_count": 32384, "claim_count": 49, "claims": [ { @@ -220,8 +220,8 @@ ], "mode": "legacy_behavior", "pointer_class": "all", - "pointer_count": 80, - "pointer_sha256": "10bd811f86dae7941035533cfe0d51d81cd632f5995c1adb80234bc7ecbf6565", + "pointer_count": 97, + "pointer_sha256": "cabaaa3d96c534f5ec37f51811c3f549d0ed20791e2892c868888c0da6d7d80e", "rationale": null, "relative_sink_prefix": null, "source_prefix": "/authored/spec~1imputation.yaml/transfer_execution", @@ -274,8 +274,8 @@ ], "mode": "compiler_semantic", "pointer_class": "all", - "pointer_count": 24473, - "pointer_sha256": "1b9f7ffb88d31d2bd365c092d04a96038b2d5a2b03f4f2a79f42e95472b213d6", + "pointer_count": 24488, + "pointer_sha256": "6382a3ed07016414c31462b8bedb24590d92b792405a89c927d85fa1bd2a8347", "rationale": null, "relative_sink_prefix": null, "source_prefix": "/authored/spec~1imputation.yaml/producer_graph", @@ -772,20 +772,20 @@ "verifier": "vintages" } ], - "configuration_field_count": 42122, - "consumed_field_count": 42122, + "configuration_field_count": 42154, + "consumed_field_count": 42154, "generation0_effect_counts": { - "legacy_behavior": 38444, + "legacy_behavior": 38476, "no_generation0_effect": 3678 }, "mode_counts": { - "compiler_semantic": 27700, + "compiler_semantic": 27715, "front_end_validation": 348, "identity_only": 103, - "legacy_behavior": 13971 + "legacy_behavior": 13988 }, "multiple_primary_use_field_count": 0, - "pointer_inventory_sha256": "ae7881c07c6137037f43a5992b3e8fffbbdbd5ddf2e1b70956986cb9df043f71", + "pointer_inventory_sha256": "3fc6b9480ea81b9635bd0db56e180c2daf32a5cd2006a70d350586c570f96754", "resolved_binding_field_count": 9770, "unused_field_count": 0 }, @@ -810,7 +810,7 @@ "primary_targets": 65, "producer_authored_outputs": 92, "producer_compiled_outputs": 227, - "producer_inputs": 2742, + "producer_inputs": 2744, "producer_nodes": 38, "producer_virtual_resources": 75, "release_rungs": 5, @@ -1074,11 +1074,11 @@ "legacy_adapter.stacked_checkpoint_static_components" ], "expected": { - "schedule_sha256": "dcf3c6d2eade3449836c49a1dc4d3b8cd395aab9142db700c3c60598fa9c1c79" + "schedule_sha256": "e59c019d3d454eac99ac0ac209b6c5b6faaf9bdfcaeee18c36a25be19bf7da2f" }, "failures": [], "observed": { - "schedule_sha256": "dcf3c6d2eade3449836c49a1dc4d3b8cd395aab9142db700c3c60598fa9c1c79" + "schedule_sha256": "e59c019d3d454eac99ac0ac209b6c5b6faaf9bdfcaeee18c36a25be19bf7da2f" }, "status": "covered" }, @@ -1250,14 +1250,14 @@ "expected": { "edges": 71, "nodes": 38, - "schedule_sha256": "dcf3c6d2eade3449836c49a1dc4d3b8cd395aab9142db700c3c60598fa9c1c79", + "schedule_sha256": "e59c019d3d454eac99ac0ac209b6c5b6faaf9bdfcaeee18c36a25be19bf7da2f", "waves": 6 }, "failures": [], "observed": { "edges": 71, "nodes": 38, - "schedule_sha256": "dcf3c6d2eade3449836c49a1dc4d3b8cd395aab9142db700c3c60598fa9c1c79", + "schedule_sha256": "e59c019d3d454eac99ac0ac209b6c5b6faaf9bdfcaeee18c36a25be19bf7da2f", "waves": 6 }, "status": "covered" @@ -1275,11 +1275,11 @@ ], "expected": { "relation": "source rows preserved exactly", - "rows": 2742 + "rows": 2744 }, "failures": [], "observed": { - "rows": 2742 + "rows": 2744 }, "status": "covered" }, @@ -1338,12 +1338,12 @@ ], "expected": { "nodes": 38, - "sha256": "7125ad28ae2c69f22094a574bbf6ed2ddf1682a2c2c3b416f8f49304b7016ce7" + "sha256": "271a7bb8d0b3f97ff344e0b7e68184fa74738a6585c24fc8781793db669f388b" }, "failures": [], "observed": { "nodes": 38, - "sha256": "7125ad28ae2c69f22094a574bbf6ed2ddf1682a2c2c3b416f8f49304b7016ce7" + "sha256": "271a7bb8d0b3f97ff344e0b7e68184fa74738a6585c24fc8781793db669f388b" }, "status": "covered" }, @@ -1361,12 +1361,12 @@ ], "expected": { "producer_count": 38, - "sha256": "ef491637933945e28645479da70646d9c50366213b6f70477de445ab9e4bb1cf" + "sha256": "afebb6725373abf5b8dd4fdb77bf2814cb6fcc569cb606c0c30963a8f65c0bab" }, "failures": [], "observed": { "producer_count": 38, - "sha256": "ef491637933945e28645479da70646d9c50366213b6f70477de445ab9e4bb1cf" + "sha256": "afebb6725373abf5b8dd4fdb77bf2814cb6fcc569cb606c0c30963a8f65c0bab" }, "status": "covered" }, @@ -1656,13 +1656,13 @@ "compiler_ir.node_slices" ], "expected": { - "map_sha256": "ee1ad8d5196ca48d70657675237294c2dd925a520900d4d476d256a1b514b283", - "protocol_sha256": "c7df307cba6b20cc26f4aeeb1ee5eae246362318cef885cd747c64349e08cc41" + "map_sha256": "9ff34c4ff7c6d744a8385c2aee6e4eb2ea30a630e56b29ac555381987058da09", + "protocol_sha256": "01c2bf7e06a589ec2c7e99c24e1413b3dfbf02206241a216d843a95082547a34" }, "failures": [], "observed": { - "map_sha256": "ee1ad8d5196ca48d70657675237294c2dd925a520900d4d476d256a1b514b283", - "protocol_sha256": "c7df307cba6b20cc26f4aeeb1ee5eae246362318cef885cd747c64349e08cc41" + "map_sha256": "9ff34c4ff7c6d744a8385c2aee6e4eb2ea30a630e56b29ac555381987058da09", + "protocol_sha256": "01c2bf7e06a589ec2c7e99c24e1413b3dfbf02206241a216d843a95082547a34" }, "status": "covered" }, @@ -1677,7 +1677,7 @@ "compiler_ir.seed_stream_map" ], "expected": { - "implementation_sha256": "c7df307cba6b20cc26f4aeeb1ee5eae246362318cef885cd747c64349e08cc41", + "implementation_sha256": "01c2bf7e06a589ec2c7e99c24e1413b3dfbf02206241a216d843a95082547a34", "protocol": "legacy-v1", "streams": [ "build_model", @@ -1698,7 +1698,7 @@ }, "failures": [], "observed": { - "implementation_sha256": "c7df307cba6b20cc26f4aeeb1ee5eae246362318cef885cd747c64349e08cc41", + "implementation_sha256": "01c2bf7e06a589ec2c7e99c24e1413b3dfbf02206241a216d843a95082547a34", "protocol": "legacy-v1", "streams": [ "build_model", @@ -1840,11 +1840,11 @@ "legacy_adapter.stacked_checkpoint_static_components" ], "expected": { - "sha256": "3a980927227704d0589f246eef9cd825c2ae84f3a4134ac835e0e5ed39a563ac" + "sha256": "e660a8ce42b69a39d29c5f0ec37264bc69d61b03f27adc386336ec8889531bb2" }, "failures": [], "observed": { - "sha256": "3a980927227704d0589f246eef9cd825c2ae84f3a4134ac835e0e5ed39a563ac" + "sha256": "e660a8ce42b69a39d29c5f0ec37264bc69d61b03f27adc386336ec8889531bb2" }, "status": "covered" }, @@ -1887,7 +1887,7 @@ "alpha", "zeta" ], - "sha256": "1fa30ae9213c19607168df3cf0035478b9c96f815e1541cc97f4e1b8e7246935" + "sha256": "b88f2d9c0f6f92c6cd81eb14d6b126afe59577b8bb392b394b2c6fbbafd195c5" }, "failures": [], "observed": { @@ -1911,7 +1911,7 @@ "alpha", "zeta" ], - "sha256": "1fa30ae9213c19607168df3cf0035478b9c96f815e1541cc97f4e1b8e7246935" + "sha256": "b88f2d9c0f6f92c6cd81eb14d6b126afe59577b8bb392b394b2c6fbbafd195c5" }, "status": "covered" }, @@ -1975,7 +1975,7 @@ "take_up_contract", "us_qbi_reconciliation_contract" ], - "sha256": "001a2bdc700615cd96771fc258abf5f769d67ea41687efd8cc2ad30361619057" + "sha256": "04899daa491e8f089899c9df64cdb2ed44d61d11da2b4c733db6f20f38a1668a" }, "status": "covered" }, @@ -2599,7 +2599,7 @@ "country": "us", "schema_id": "country_spec", "schema_version": 1, - "spec_sha256": "5f44d96d45e9aabcea2d565ef063d68bfc0652df1b38b08aa31ce6896d15f371" + "spec_sha256": "b7b7037eeab3258d38968a8a5428c65df50bb3b0ee892c660488bf8df3cbe90f" } }, "report_schema_version": 3, @@ -2609,7 +2609,7 @@ "country": "us", "schema_id": "country_spec", "schema_version": 1, - "spec_sha256": "5f44d96d45e9aabcea2d565ef063d68bfc0652df1b38b08aa31ce6896d15f371" + "spec_sha256": "b7b7037eeab3258d38968a8a5428c65df50bb3b0ee892c660488bf8df3cbe90f" }, "status": "pass" } diff --git a/packages/microcosm-build/src/microcosm/build/spec_engine/field_usage.py b/packages/microcosm-build/src/microcosm/build/spec_engine/field_usage.py index ad66f291a..aa6039e3a 100644 --- a/packages/microcosm-build/src/microcosm/build/spec_engine/field_usage.py +++ b/packages/microcosm-build/src/microcosm/build/spec_engine/field_usage.py @@ -26,9 +26,9 @@ ) from .schemas import load_schema_registry -EXPECTED_AUTHORED_FIELD_COUNT = 32_352 +EXPECTED_AUTHORED_FIELD_COUNT = 32_384 EXPECTED_RESOLVED_BINDING_FIELD_COUNT = 9_770 -EXPECTED_CONFIGURATION_FIELD_COUNT = 42_122 +EXPECTED_CONFIGURATION_FIELD_COUNT = 42_154 class FieldUsageError(AssertionError): @@ -425,12 +425,12 @@ def _path_inventory(rows: Sequence[tuple[str, object]]) -> tuple[int, str]: "e1dd7dc5123ab0f39d08ea4939d98dd09a6fdb8e7449a7ca3125fb1ddbd5b4e9", ), "imputation_producer_graph": ( - 24_473, - "1b9f7ffb88d31d2bd365c092d04a96038b2d5a2b03f4f2a79f42e95472b213d6", + 24_488, + "6382a3ed07016414c31462b8bedb24590d92b792405a89c927d85fa1bd2a8347", ), "imputation_transfer_execution": ( - 80, - "10bd811f86dae7941035533cfe0d51d81cd632f5995c1adb80234bc7ecbf6565", + 97, + "cabaaa3d96c534f5ec37f51811c3f549d0ed20791e2892c868888c0da6d7d80e", ), "imputation_waiver_records": ( 70, diff --git a/packages/microcosm-build/src/microcosm/build/spec_engine/imputation_semantics.py b/packages/microcosm-build/src/microcosm/build/spec_engine/imputation_semantics.py index d80fba823..ef6bbf106 100644 --- a/packages/microcosm-build/src/microcosm/build/spec_engine/imputation_semantics.py +++ b/packages/microcosm-build/src/microcosm/build/spec_engine/imputation_semantics.py @@ -196,6 +196,25 @@ def block_columns(block_id: object) -> list[object]: profile = _mapping_like(profiles[profile_id], f"transfer profile {profile_id}") derive_schedule_d = bool(profile["derive_schedule_d"]) target_set = set(targets) + structural_policies = _mapping_like( + result["structural_target_policies"], + "transfer structural target policies", + ) + resolved_structural_policies: dict[str, object] = {} + for policy_id, policy_value in structural_policies.items(): + policy = deepcopy( + dict( + _mapping_like( + policy_value, + f"transfer structural target policy {policy_id}", + ) + ) + ) + policy.pop("sha256", None) + policy["enabled"] = policy.get("target") in target_set + policy["sha256"] = _canonical_sha256(policy) + resolved_structural_policies[policy_id] = policy + result["structural_target_policies"] = resolved_structural_policies post_transfer: dict[str, object] = {} features = _mapping_like( transfer_execution["post_transfer_features"], "post-transfer features" diff --git a/packages/microcosm-build/src/microcosm/build/spec_engine/inventory_coverage.py b/packages/microcosm-build/src/microcosm/build/spec_engine/inventory_coverage.py index f0dd2529c..6758aca87 100644 --- a/packages/microcosm-build/src/microcosm/build/spec_engine/inventory_coverage.py +++ b/packages/microcosm-build/src/microcosm/build/spec_engine/inventory_coverage.py @@ -348,19 +348,19 @@ EXPECTED_HASHES = { "acs_group_predictors": "a927bb7ecf3e84f54c93583ab79318654514ac546aefafba67da5285615fbd60", "acs_person_predictors": "878c788a6f037d7aca12b3586ea034eff04f3034ffa11935a736493042551f25", - "authority": "3a980927227704d0589f246eef9cd825c2ae84f3a4134ac835e0e5ed39a563ac", + "authority": "e660a8ce42b69a39d29c5f0ec37264bc69d61b03f27adc386336ec8889531bb2", "early_families": "4aa9f736fd76e83955477ad1667e58f48f264783f05bdc7f0102cd32d61323bd", - "full_checkpoint": "1fa30ae9213c19607168df3cf0035478b9c96f815e1541cc97f4e1b8e7246935", + "full_checkpoint": "b88f2d9c0f6f92c6cd81eb14d6b126afe59577b8bb392b394b2c6fbbafd195c5", "gap_fill_schedule": "1c31f9868f7884347cc19cf1ff65da43f950b9114941a715bab168246db414a7", - "graph_nodes": "7125ad28ae2c69f22094a574bbf6ed2ddf1682a2c2c3b416f8f49304b7016ce7", + "graph_nodes": "271a7bb8d0b3f97ff344e0b7e68184fa74738a6585c24fc8781793db669f388b", "geography_assignment": "f49425ca8734ac559c73cf44f6458d86d3162a48956b98a27e6e758959361585", "late_families": "d91f9ff0eb52f43e7b6eed3d5c58c37abe1620c3a11021da15dae9c10e16d382", - "late_resource_semantics": "ef491637933945e28645479da70646d9c50366213b6f70477de445ab9e4bb1cf", - "late_schedule": "dcf3c6d2eade3449836c49a1dc4d3b8cd395aab9142db700c3c60598fa9c1c79", + "late_resource_semantics": "afebb6725373abf5b8dd4fdb77bf2814cb6fcc569cb606c0c30963a8f65c0bab", + "late_schedule": "e59c019d3d454eac99ac0ac209b6c5b6faaf9bdfcaeee18c36a25be19bf7da2f", "ownership": "5f64f0aac49e2313177564f71876bffc8c81b3ded4df701e70930e60e9c98356", "primary_tuples": "987b501c695e31f45521c4a178528f75ab3df22c09bc407b182213b2de99ee57", - "seed_map": "ee1ad8d5196ca48d70657675237294c2dd925a520900d4d476d256a1b514b283", - "seed_protocol": "c7df307cba6b20cc26f4aeeb1ee5eae246362318cef885cd747c64349e08cc41", + "seed_map": "9ff34c4ff7c6d744a8385c2aee6e4eb2ea30a630e56b29ac555381987058da09", + "seed_protocol": "01c2bf7e06a589ec2c7e99c24e1413b3dfbf02206241a216d843a95082547a34", "source_manifest": "cd5ba8924d64da5425ee14cca82a774e3f4b2bb5aabe06df291cc3cc457287a9", "take_up": "fa186daea0f8dd641cc470e41d1a2953f887d45282ec990201298f47bedf8d4d", "tail": "ac92829c88a1a4fb6460d61190918d5d99c6c377fc8dd8f62f02b332d09bf59c", @@ -429,7 +429,7 @@ "primary_targets": 65, "producer_authored_outputs": 92, "producer_compiled_outputs": 227, - "producer_inputs": 2_742, + "producer_inputs": 2_744, "producer_nodes": 38, "producer_virtual_resources": 75, "release_rungs": 5, @@ -1005,7 +1005,7 @@ def add( "producer_inputs_exact", clauses={ "producer input rows differ": inputs_exact, - "input row count differs": input_count == 2742, + "input row count differs": input_count == 2744, }, homes=("/imputation/producer_graph/nodes/*/inputs",), consumers=( @@ -1013,7 +1013,7 @@ def add( "compiler_ir.node_slices", ), observed={"rows": input_count}, - expected={"rows": 2742, "relation": "source rows preserved exactly"}, + expected={"rows": 2744, "relation": "source rows preserved exactly"}, ) outputs_exact = set(expected_outputs) == set(compiled_by_id) and all( _json_equal( diff --git a/packages/microcosm-build/src/microcosm/build/spec_engine/schema/imputation.schema.json b/packages/microcosm-build/src/microcosm/build/spec_engine/schema/imputation.schema.json index 0c2a2548e..4a4a09a5a 100644 --- a/packages/microcosm-build/src/microcosm/build/spec_engine/schema/imputation.schema.json +++ b/packages/microcosm-build/src/microcosm/build/spec_engine/schema/imputation.schema.json @@ -613,6 +613,111 @@ "schema_version": { "type": "integer" }, + "structural_target_policies": { + "type": "object", + "additionalProperties": false, + "properties": { + "is_pregnant": { + "type": "object", + "additionalProperties": false, + "properties": { + "assembled_representative": { + "type": "object", + "additionalProperties": false, + "properties": { + "clone_index": { + "type": "integer", + "minimum": 0 + }, + "clone_index_column": { + "type": "string" + } + }, + "required": [ + "clone_index", + "clone_index_column" + ] + }, + "eligibility": { + "type": "object", + "additionalProperties": false, + "properties": { + "is_female": { + "const": true + }, + "maximum_age_inclusive": { + "type": "integer" + }, + "minimum_age_inclusive": { + "type": "integer" + } + }, + "required": [ + "is_female", + "maximum_age_inclusive", + "minimum_age_inclusive" + ] + }, + "enabled": { + "type": "boolean" + }, + "fanout": { + "type": "string" + }, + "final_clone_disagreement": { + "const": "refuse" + }, + "final_domain_violations": { + "const": "refuse" + }, + "ineligible_missing_value": { + "const": false + }, + "preexisting_clone_disagreement": { + "const": "refuse" + }, + "preexisting_domain_violations": { + "const": "refuse" + }, + "qrf_scope": { + "type": "string" + }, + "sha256": { + "$ref": "defs.schema.json#/$defs/sha256" + }, + "source_person_key_precedence": { + "type": "array", + "items": { + "type": "string" + }, + "minItems": 1, + "uniqueItems": true + }, + "target": { + "const": "is_pregnant" + } + }, + "required": [ + "assembled_representative", + "eligibility", + "enabled", + "fanout", + "final_clone_disagreement", + "final_domain_violations", + "ineligible_missing_value", + "preexisting_clone_disagreement", + "preexisting_domain_violations", + "qrf_scope", + "sha256", + "source_person_key_precedence", + "target" + ] + } + }, + "required": [ + "is_pregnant" + ] + }, "tenure_codes": { "type": "object", "additionalProperties": false, @@ -678,6 +783,7 @@ "profiles", "recipient_combined_sources", "schema_version", + "structural_target_policies", "tenure_codes" ] }, diff --git a/packages/microcosm-build/src/microcosm/build/us/spec/imputation.yaml b/packages/microcosm-build/src/microcosm/build/us/spec/imputation.yaml index 99d853a9a..16ae791a1 100644 --- a/packages/microcosm-build/src/microcosm/build/us/spec/imputation.yaml +++ b/packages/microcosm-build/src/microcosm/build/us/spec/imputation.yaml @@ -409,6 +409,28 @@ transfer_execution: - first_home_mortgage_origination_year - second_home_mortgage_origination_year - weeks_unemployed + structural_target_policies: + is_pregnant: + enabled: false + target: is_pregnant + eligibility: + is_female: true + minimum_age_inclusive: 15 + maximum_age_inclusive: 44 + source_person_key_precedence: + - person_source_id + - person_id + assembled_representative: + clone_index_column: person_support_clone_index + clone_index: 0 + qrf_scope: one_eligible_representative_per_source_person + ineligible_missing_value: false + fanout: one_source_person_result_to_every_missing_clone + preexisting_domain_violations: refuse + preexisting_clone_disagreement: refuse + final_domain_violations: refuse + final_clone_disagreement: refuse + sha256: 3290b2dde9a077d1c42834b22e8b7c383afbd00748012fbea52ccf803a356f3a predictor_bindings: person_required: acs_person_required person_optional: @@ -10451,6 +10473,16 @@ producer_graph: producing_stage: post_clone_input_surface required_scope: asec_source tolerated_absence_receipts: [] + - alternatives: + - - column: person_source_id + entity: person + value_kind: finite_numeric + column: '@effective:pregnancy_structural_source_person_id' + entity: person + producing_stage: post_clone_input_surface + required_scope: asec_source + tolerated_absence_receipts: + - optional_input:source:with_us_pregnancy_inputs:pregnancy_structural_source_person_id - alternatives: - - column: A_AGE entity: person @@ -10622,6 +10654,15 @@ producer_graph: producing_stage: primary_puf_qrf required_scope: puf_clone tolerated_absence_receipts: [] + - alternatives: + - - column: person_source_id + entity: person + value_kind: non_null + column: person_source_id + entity: person + producing_stage: primary_puf_qrf + required_scope: puf_clone + tolerated_absence_receipts: [] - alternatives: - - column: person_spm_unit_id entity: person diff --git a/packages/microcosm-build/src/microcosm/build/us_runtime/acs_transfer.py b/packages/microcosm-build/src/microcosm/build/us_runtime/acs_transfer.py index 8688d8468..b90151099 100644 --- a/packages/microcosm-build/src/microcosm/build/us_runtime/acs_transfer.py +++ b/packages/microcosm-build/src/microcosm/build/us_runtime/acs_transfer.py @@ -138,6 +138,50 @@ } ) +_PREGNANCY_TARGET = "is_pregnant" +_PREGNANCY_AGE_RANGE = (15, 44) +_PERSON_SOURCE_ID_COLUMN = "person_source_id" +_PERSON_ID_COLUMN = "person_id" +_PERSON_CLONE_INDEX_COLUMN = "person_support_clone_index" + + +def _pregnancy_structural_policy_identity(*, enabled: bool) -> dict[str, object]: + """Return the complete hard-domain and clone-fanout transfer policy.""" + + payload: dict[str, object] = { + "enabled": enabled, + "target": _PREGNANCY_TARGET, + "eligibility": { + "is_female": True, + "minimum_age_inclusive": _PREGNANCY_AGE_RANGE[0], + "maximum_age_inclusive": _PREGNANCY_AGE_RANGE[1], + }, + "source_person_key_precedence": [ + _PERSON_SOURCE_ID_COLUMN, + _PERSON_ID_COLUMN, + ], + "assembled_representative": { + "clone_index_column": _PERSON_CLONE_INDEX_COLUMN, + "clone_index": 0, + }, + "qrf_scope": "one_eligible_representative_per_source_person", + "ineligible_missing_value": False, + "fanout": "one_source_person_result_to_every_missing_clone", + "preexisting_domain_violations": "refuse", + "preexisting_clone_disagreement": "refuse", + "final_domain_violations": "refuse", + "final_clone_disagreement": "refuse", + } + payload["sha256"] = hashlib.sha256( + json.dumps( + payload, + sort_keys=True, + separators=(",", ":"), + allow_nan=False, + ).encode("utf-8") + ).hexdigest() + return payload + _IMMIGRATION_STATUS_TARGETS = ( "ssn_card_type", "immigration_status_str", @@ -269,6 +313,11 @@ def acs_transfer_execution_contract_identity( "immigration_status_targets": list(_IMMIGRATION_STATUS_TARGETS), "immigration_status_model_target": _IMMIGRATION_STATUS_MODEL_TARGET, "discrete_numeric_targets": sorted(_DISCRETE_NUMERIC_TARGETS), + "structural_target_policies": { + _PREGNANCY_TARGET: _pregnancy_structural_policy_identity( + enabled=_PREGNANCY_TARGET in requested_targets, + ) + }, "post_transfer_structure": { "schedule_d_capital_gain_distributions": { "enabled": schedule_d_enabled, @@ -445,6 +494,9 @@ class AcsImputedInput: #: Post-fit structural reconciliation counts, when the column's surface #: was adjusted to a statute contract after prediction. reconciliation: Mapping[str, int] | None = None + #: Hard-domain and source-person fanout proof for structurally constrained + #: transfer targets. This is receipt-only and never enters the frame. + structural_receipt: Mapping[str, object] | None = None @dataclass(frozen=True) @@ -559,6 +611,19 @@ class _TargetEncoding: category_position: int | None = None +@dataclass(frozen=True) +class _PregnancyStructuralPlan: + """Pre-QRF pregnancy scope plus enough state for deterministic fanout.""" + + qrf_missing: np.ndarray + source_codes: np.ndarray + source_values: np.ndarray + qrf_source_groups: np.ndarray + representative_positions: np.ndarray + eligibility: np.ndarray + receipt: Mapping[str, object] + + def required_acs_transfer_inputs() -> frozenset[str]: """Return every model leaf in the declared production transfer plan.""" @@ -905,6 +970,331 @@ def assert_acs_transfer_targets_are_input_leaves( ) +def _pregnancy_eligibility(table: pd.DataFrame, *, role: str) -> np.ndarray: + """Resolve the hard female-age domain without coercing missing predictors.""" + + _require_columns( + table, + ("age", "is_female"), + context=f"ACS transfer pregnancy {role} domain", + ) + try: + age = _as_float_array(table["age"]) + female = _as_float_array(table["is_female"]) + except (TypeError, ValueError) as exc: + raise TypeError( + f"ACS transfer pregnancy {role} domain requires numeric age and " + "boolean is_female." + ) from exc + invalid_age = ~np.isfinite(age) + invalid_female = ~np.isfinite(female) | ~np.isin(female, [0.0, 1.0]) + if invalid_age.any() or invalid_female.any(): + raise ValueError( + f"ACS transfer pregnancy {role} domain requires complete finite age " + "and boolean is_female; found " + f"invalid_age_rows={int(invalid_age.sum())}, " + f"invalid_is_female_rows={int(invalid_female.sum())}." + ) + low, high = _PREGNANCY_AGE_RANGE + return (female == 1.0) & (age >= low) & (age <= high) + + +def _pregnancy_boolean_values( + series: pd.Series, + *, + role: str, +) -> tuple[np.ndarray, np.ndarray]: + """Return observed/true masks while refusing non-boolean physical values.""" + + observed = series.notna().to_numpy(dtype=bool) + try: + values = _as_float_array(series) + except (TypeError, ValueError) as exc: + raise TypeError( + f"ACS transfer pregnancy {role} values must be physical booleans." + ) from exc + valid = ~observed | np.isclose(values, 0.0) | np.isclose(values, 1.0) + if not valid.all(): + raise ValueError( + f"ACS transfer pregnancy {role} values contain " + f"{int((~valid).sum())} non-boolean row(s)." + ) + return observed, observed & np.isclose(values, 1.0) + + +def _pregnancy_source_groups( + table: pd.DataFrame, +) -> tuple[np.ndarray, int, str, np.ndarray]: + """Return stable source-person codes and one canonical representative.""" + + has_clone_index = _PERSON_CLONE_INDEX_COLUMN in table + if has_clone_index and _PERSON_SOURCE_ID_COLUMN not in table: + raise ValueError( + "ACS transfer pregnancy clone fanout requires person_source_id when " + f"{_PERSON_CLONE_INDEX_COLUMN} is present." + ) + key_column = ( + _PERSON_SOURCE_ID_COLUMN + if _PERSON_SOURCE_ID_COLUMN in table + else _PERSON_ID_COLUMN + ) + _require_columns( + table, + (key_column,), + context="ACS transfer pregnancy source-person identity", + ) + keys = table[key_column] + if keys.isna().any(): + raise ValueError( + f"ACS transfer pregnancy source-person identity {key_column!r} has " + f"{int(keys.isna().sum())} missing row(s)." + ) + codes, unique_keys = pd.factorize(keys, sort=False) + if (codes < 0).any(): # pragma: no cover - missing keys refused above + raise AssertionError("Pregnancy source-person factorization lost a key.") + group_count = len(unique_keys) + representatives = np.full(group_count, -1, dtype=np.int64) + positions = np.arange(len(table), dtype=np.int64) + + if not has_clone_index: + _, first_positions = np.unique(codes, return_index=True) + representatives[:] = first_positions + return codes, group_count, key_column, representatives + + clone_numeric = pd.to_numeric( + table[_PERSON_CLONE_INDEX_COLUMN], + errors="coerce", + ).to_numpy(dtype=np.float64) + invalid_clone = ( + ~np.isfinite(clone_numeric) + | (clone_numeric < 0.0) + | (clone_numeric != np.floor(clone_numeric)) + ) + if invalid_clone.any(): + raise ValueError( + "ACS transfer pregnancy clone fanout found " + f"{int(invalid_clone.sum())} invalid clone-index row(s)." + ) + clone_index = clone_numeric.astype(np.int64) + duplicate_pairs = pd.DataFrame( + {"_source": codes, "_clone": clone_index} + ).duplicated() + if duplicate_pairs.any(): + raise ValueError( + "ACS transfer pregnancy clone fanout found " + f"{int(duplicate_pairs.sum())} duplicate source-person/clone row(s)." + ) + clone_zero = clone_index == 0 + clone_zero_counts = np.bincount(codes[clone_zero], minlength=group_count) + bad_groups = clone_zero_counts != 1 + if bad_groups.any(): + raise ValueError( + "ACS transfer pregnancy clone fanout requires exactly one clone-0 " + f"representative for every source person; {int(bad_groups.sum())} " + "source person(s) violate that contract." + ) + representatives[codes[clone_zero]] = positions[clone_zero] + return codes, group_count, key_column, representatives + + +def _prepare_pregnancy_structural_plan( + donor: Frame, + recipient: Frame, + *, + target_missing: np.ndarray, +) -> _PregnancyStructuralPlan: + """Refuse invalid inputs and select one eligible QRF row per source person.""" + + donor_table = donor.table(donor.schema.person_entity) + donor_eligible = _pregnancy_eligibility(donor_table, role="donor") + donor_observed, donor_true = _pregnancy_boolean_values( + donor_table[_PREGNANCY_TARGET], + role="donor", + ) + donor_domain_violations = donor_observed & donor_true & ~donor_eligible + if donor_domain_violations.any(): + raise ValueError( + "ACS transfer pregnancy structural policy refused " + f"{int(donor_domain_violations.sum())} preexisting donor domain " + "violation row(s); pregnancy requires female ages 15 through 44." + ) + + table = recipient.table(recipient.schema.person_entity) + eligibility = _pregnancy_eligibility(table, role="recipient") + if _PREGNANCY_TARGET in table: + observed, true_values = _pregnancy_boolean_values( + table[_PREGNANCY_TARGET], + role="recipient", + ) + else: + observed = np.zeros(len(table), dtype=bool) + true_values = np.zeros(len(table), dtype=bool) + recipient_domain_violations = observed & true_values & ~eligibility + if recipient_domain_violations.any(): + raise ValueError( + "ACS transfer pregnancy structural policy refused " + f"{int(recipient_domain_violations.sum())} preexisting recipient " + "domain violation row(s); pregnancy requires female ages 15 " + "through 44." + ) + + source_codes, group_count, key_column, representatives = ( + _pregnancy_source_groups(table) + ) + eligible_min = np.ones(group_count, dtype=np.int8) + eligible_max = np.zeros(group_count, dtype=np.int8) + np.minimum.at(eligible_min, source_codes, eligibility.astype(np.int8)) + np.maximum.at(eligible_max, source_codes, eligibility.astype(np.int8)) + inconsistent_domain_groups = eligible_min != eligible_max + if inconsistent_domain_groups.any(): + raise ValueError( + "ACS transfer pregnancy structural policy refused " + f"{int(inconsistent_domain_groups.sum())} source person(s) whose " + "clones disagree on female-age eligibility." + ) + + group_has_true = ( + np.bincount(source_codes[observed & true_values], minlength=group_count) > 0 + ) + group_has_false = ( + np.bincount(source_codes[observed & ~true_values], minlength=group_count) > 0 + ) + preexisting_disagreement = group_has_true & group_has_false + if preexisting_disagreement.any(): + raise ValueError( + "ACS transfer pregnancy structural policy refused " + f"{int(preexisting_disagreement.sum())} source person(s) with " + "preexisting clone disagreement." + ) + + missing = np.asarray(target_missing, dtype=bool) + if missing.shape != (len(table),): + raise ValueError("ACS pregnancy target-missing mask has the wrong row count.") + group_observed = group_has_true | group_has_false + group_eligible = eligibility[representatives] + qrf_source_groups = ~group_observed & group_eligible + qrf_missing = np.zeros(len(table), dtype=bool) + qrf_missing[representatives[qrf_source_groups]] = True + + source_values = np.full(group_count, np.nan, dtype=np.float64) + source_values[group_observed] = group_has_true[group_observed].astype(np.float64) + source_values[~group_eligible] = 0.0 + group_sizes = np.bincount(source_codes, minlength=group_count) + missing_by_group = np.bincount( + source_codes, + weights=missing.astype(np.int64), + minlength=group_count, + ).astype(np.int64) + qrf_group_count = int(qrf_source_groups.sum()) + policy = _pregnancy_structural_policy_identity(enabled=True) + receipt: dict[str, object] = { + "policy_sha256": policy["sha256"], + "source_person_key": key_column, + "source_persons_checked": group_count, + "physical_rows_checked": int(len(table)), + "clone_rows_checked": int(len(table) - group_count), + "donor_rows_checked": int(len(donor_table)), + "qrf_draw_source_persons": qrf_group_count, + "qrf_draw_rows": qrf_group_count, + "qrf_fanout_rows": int( + missing_by_group[qrf_source_groups].sum() - qrf_group_count + ), + "preexisting_value_fanout_rows": int( + missing_by_group[group_observed & group_eligible].sum() + ), + "ineligible_rows_assigned_false": int((missing & ~eligibility).sum()), + "donor_preexisting_domain_violation_rows": 0, + "recipient_preexisting_domain_violation_rows": 0, + "preexisting_clone_disagreement_source_persons": 0, + "inconsistent_eligibility_source_persons": 0, + "maximum_clones_per_source_person": int(group_sizes.max(initial=0)), + } + return _PregnancyStructuralPlan( + qrf_missing=qrf_missing, + source_codes=source_codes, + source_values=source_values, + qrf_source_groups=qrf_source_groups, + representative_positions=representatives, + eligibility=eligibility, + receipt=receipt, + ) + + +def _fan_pregnancy_predictions( + plan: _PregnancyStructuralPlan, + prediction: np.ndarray | pd.api.extensions.ExtensionArray | None, +) -> np.ndarray: + """Fan one decoded QRF result across each source person's missing clones.""" + + source_values = plan.source_values.copy() + if plan.qrf_source_groups.any(): + if prediction is None: # pragma: no cover - caller invariant + raise RuntimeError("Pregnancy structural plan is missing its QRF draw.") + predicted = _as_float_array(pd.Series(prediction)) + positions = plan.representative_positions[plan.qrf_source_groups] + values = predicted[positions] + valid = np.isfinite(values) & ( + np.isclose(values, 0.0) | np.isclose(values, 1.0) + ) + if not valid.all(): + raise ValueError( + "ACS transfer pregnancy QRF produced " + f"{int((~valid).sum())} invalid representative result(s)." + ) + source_values[plan.qrf_source_groups] = np.isclose(values, 1.0).astype( + np.float64 + ) + if not np.isfinite(source_values).all(): # pragma: no cover - plan invariant + raise RuntimeError("Pregnancy structural fanout left unresolved source people.") + return np.isclose(source_values[plan.source_codes], 1.0) + + +def _finalize_pregnancy_structural_receipt( + table: pd.DataFrame, + plan: _PregnancyStructuralPlan, +) -> dict[str, object]: + """Refuse post-transfer violations and seal the successful count receipt.""" + + eligibility = _pregnancy_eligibility(table, role="post-transfer") + observed, true_values = _pregnancy_boolean_values( + table[_PREGNANCY_TARGET], + role="post-transfer", + ) + incomplete = ~observed + domain_violations = observed & true_values & ~eligibility + group_count = len(plan.source_values) + group_has_true = ( + np.bincount( + plan.source_codes[observed & true_values], + minlength=group_count, + ) + > 0 + ) + group_has_false = ( + np.bincount( + plan.source_codes[observed & ~true_values], + minlength=group_count, + ) + > 0 + ) + clone_disagreements = group_has_true & group_has_false + if incomplete.any() or domain_violations.any() or clone_disagreements.any(): + raise ValueError( + "ACS transfer pregnancy structural postcondition refused output: " + f"incomplete_rows={int(incomplete.sum())}, " + f"domain_violation_rows={int(domain_violations.sum())}, " + "clone_disagreement_source_persons=" + f"{int(clone_disagreements.sum())}." + ) + return { + **dict(plan.receipt), + "final_incomplete_rows": 0, + "final_domain_violation_rows": 0, + "final_clone_disagreement_source_persons": 0, + "status": "verified", + } + + def transfer_acs_inputs( recipient: Frame, donor: Frame, @@ -1025,21 +1415,88 @@ def transfer_acs_inputs( ) for entity, family, active_targets in active ] + structural_records: tuple[AcsImputedInput, ...] = () + fit_donor: Frame | None = None + resolved_channel: str | None = None + pregnancy_plan: _PregnancyStructuralPlan | None = None + pregnancy_request = next( + ( + item + for item in requested + if item[0] == recipient.schema.person_entity + and _PREGNANCY_TARGET in item[2] + ), + None, + ) + if pregnancy_request is not None: + _validate_donor_source( + donor_spine=donor_spine, + donor_channel=donor_channel, + ) + fit_donor, resolved_channel = resolve_acs_donor_channel( + donor, + donor_channel, + ) + person = recipient.table(recipient.schema.person_entity) + pregnancy_missing = ( + person[_PREGNANCY_TARGET].isna().to_numpy(dtype=bool) + if _PREGNANCY_TARGET in person + else np.ones(len(person), dtype=bool) + ) + pregnancy_plan = _prepare_pregnancy_structural_plan( + fit_donor, + recipient, + target_missing=pregnancy_missing, + ) + if not pregnancy_missing.any(): + structural_receipt = _finalize_pregnancy_structural_receipt( + person, + pregnancy_plan, + ) + pregnancy_entity, pregnancy_family, pregnancy_targets = ( + pregnancy_request + ) + if pregnancy_targets != (_PREGNANCY_TARGET,): # pragma: no cover + raise AssertionError( + "Pregnancy structural target was not isolated before receipt." + ) + structural_records = ( + AcsImputedInput( + column=_PREGNANCY_TARGET, + entity=pregnancy_entity, + family=pregnancy_family, + donor_spine=donor_spine, + donor_channel=resolved_channel, + predictors=ACS_PERSON_TRANSFER_PREDICTORS, + seed=_family_seed( + seed, + entity=pregnancy_entity, + family=pregnancy_family, + ), + weight_kind="structural", + structural_receipt=structural_receipt, + ), + ) + if not active: return AcsTransferResult( frame=canonicalize_frame_string_dtypes( recipient, boundary="ACS transfer result", ), + imputed_inputs=structural_records, deferred_inputs=deferred_inputs, + resolved_donor_channel=resolved_channel, ) - _validate_donor_source(donor_spine=donor_spine, donor_channel=donor_channel) - fit_donor, resolved_channel = resolve_acs_donor_channel(donor, donor_channel) + if fit_donor is None: + _validate_donor_source(donor_spine=donor_spine, donor_channel=donor_channel) + fit_donor, resolved_channel = resolve_acs_donor_channel(donor, donor_channel) + assert fit_donor is not None output_tables = { entity: recipient.table(entity).copy() for entity in recipient.entities } - provenance: list[AcsImputedInput] = [] + provenance = list(structural_records) fit_records: list[FitWeightRecord] = [] imputed_masks: dict[tuple[str, str], np.ndarray] = {} ordered_bank_targets = [ @@ -1062,14 +1519,34 @@ def transfer_acs_inputs( ) for target in targets } - if target_bank is None: + active_pregnancy_plan: _PregnancyStructuralPlan | None = None + fit_target_missing = target_missing + if _PREGNANCY_TARGET in targets: + if targets != (_PREGNANCY_TARGET,): + raise AssertionError( + "Pregnancy structural target was not isolated before QRF." + ) + if pregnancy_plan is None: # pragma: no cover - preflight invariant + raise AssertionError("Pregnancy structural plan was not prepared.") + active_pregnancy_plan = pregnancy_plan + fit_target_missing = { + _PREGNANCY_TARGET: active_pregnancy_plan.qrf_missing, + } + + fitted: _FamilyFit | None + if ( + active_pregnancy_plan is not None + and not active_pregnancy_plan.qrf_missing.any() + ): + fitted = None + elif target_bank is None: fitted = _fit_family_patterns( fit_donor, recipient, entity=entity, family=family, targets=targets, - target_missing=target_missing, + target_missing=fit_target_missing, seed=seed, n_estimators=n_estimators, regime_evidence_targets=family_regime_evidence_targets, @@ -1081,7 +1558,7 @@ def transfer_acs_inputs( entity=entity, family=family, targets=targets, - target_missing=target_missing, + target_missing=fit_target_missing, seed=seed, n_estimators=n_estimators, target_bank=target_bank, @@ -1094,16 +1571,29 @@ def transfer_acs_inputs( ) patterns_without_regimes = ( tuple(replace(pattern, target_regimes=()) for pattern in fitted.patterns) - if family_regime_evidence_targets + if fitted is not None and family_regime_evidence_targets else fitted.patterns + if fitted is not None + else () ) for target in targets: - predicted = _prediction_values( - fitted.predictions[target], - encoding=fitted.target_encodings[target], - entity=entity, - target=target, + decoded = ( + _prediction_values( + fitted.predictions[target], + encoding=fitted.target_encodings[target], + entity=entity, + target=target, + ) + if fitted is not None + else None ) + predicted = ( + _fan_pregnancy_predictions(active_pregnancy_plan, decoded) + if active_pregnancy_plan is not None + else decoded + ) + if predicted is None: # pragma: no cover - structural invariant + raise AssertionError("ACS transfer target has no prediction values.") merged, imputed = _fill_recipient_nulls( output_tables[entity], target=target, @@ -1114,6 +1604,14 @@ def transfer_acs_inputs( missing_rows = target_missing[target] if not missing_rows.any(): continue + structural_receipt = ( + _finalize_pregnancy_structural_receipt( + output_tables[entity], + active_pregnancy_plan, + ) + if active_pregnancy_plan is not None + else None + ) provenance.append( AcsImputedInput( column=target, @@ -1121,19 +1619,32 @@ def transfer_acs_inputs( family=family, donor_spine=donor_spine, donor_channel=resolved_channel, - predictors=fitted.predictors, - seed=fitted.family_seed, - weight_kind=fitted.weight_kind, + predictors=( + fitted.predictors + if fitted is not None + else ACS_PERSON_TRANSFER_PREDICTORS + ), + seed=( + fitted.family_seed + if fitted is not None + else _family_seed(seed, entity=entity, family=family) + ), + weight_kind=( + fitted.weight_kind if fitted is not None else "structural" + ), patterns=( fitted.patterns - if target in family_regime_evidence_targets + if fitted is not None + and target in family_regime_evidence_targets else patterns_without_regimes ), imputed_recipient_rows=int(imputed.sum()), unmodeled_recipient_rows=int((missing_rows & ~imputed).sum()), + structural_receipt=structural_receipt, ) ) - fit_records.extend(fitted.fit_records) + if fitted is not None: + fit_records.extend(fitted.fit_records) _apply_post_transfer_structure( output_tables, @@ -2629,7 +3140,7 @@ def _split_large_target_families( *, max_targets_per_fit: int, ) -> list[tuple[str, str, tuple[str, ...]]]: - """Bound retained QRF forests without separating joint categorical codecs.""" + """Bound retained QRF forests and isolate hard-domain target draws.""" bounded: list[tuple[str, str, tuple[str, ...]]] = [] immigration_pair = set(_IMMIGRATION_STATUS_TARGETS) @@ -2647,6 +3158,12 @@ def _split_large_target_families( batches: list[tuple[str, ...]] = [] current: list[str] = [] for atom in atoms: + if atom == (_PREGNANCY_TARGET,): + if current: + batches.append(tuple(current)) + current = [] + batches.append(atom) + continue if current and len(current) + len(atom) > max_targets_per_fit: batches.append(tuple(current)) current = [] diff --git a/packages/microcosm-build/src/microcosm/build/us_runtime/pregnancy.py b/packages/microcosm-build/src/microcosm/build/us_runtime/pregnancy.py index aa6c94096..06bfac7a0 100644 --- a/packages/microcosm-build/src/microcosm/build/us_runtime/pregnancy.py +++ b/packages/microcosm-build/src/microcosm/build/us_runtime/pregnancy.py @@ -20,15 +20,15 @@ ``microcosm/build/us/source_stages.json`` with its citation and reaches this module as a manifest operation parameter. -Selection draws are seeded blake2b hashes keyed by the person's stable -source identity (``source_year`` / ``source_household_id`` / -``source_person_id`` when present), so support-channel clones of one -source person always receive the same flag and reruns are -bit-reproducible. - -Healing behavior: a frame that already carries ``is_pregnant`` with -signal passes through untouched (idempotent). A constant column — -indistinguishable from the engine's broadcast default — is reseeded. +Selection draws are seeded blake2b hashes keyed by the person's stable source +identity (``source_year`` / ``source_household_id`` / ``source_person_id`` +when present), so support-channel clones of one source person always receive +the same flag and reruns are bit-reproducible. + +Healing behavior: a frame that already carries a complete, structurally valid +``is_pregnant`` signal passes through untouched (idempotent). A constant +column — indistinguishable from the engine's broadcast default — is reseeded. +Preexisting or final domain violations and clone disagreement are refused. """ from __future__ import annotations @@ -83,6 +83,7 @@ _FEMALE_SEX_CODE = 2 _CHILDBEARING_AGE_RANGE = (15, 44) +_PERSON_SOURCE_ID_COLUMN = "person_source_id" #: Weighted share of all persons flagged pregnant must land in this band. #: Expected ≈ rate (4.1%) x the female-15-44 share of the population @@ -163,6 +164,131 @@ def _stable_person_draws(persons: pd.DataFrame, *, seed: int) -> np.ndarray: ) +def _pregnancy_eligibility(person: pd.DataFrame) -> tuple[np.ndarray, np.ndarray]: + """Return female and age-domain masks from mapped or raw source columns.""" + + if {"is_female", "age"} <= set(person.columns): + female_source = person["is_female"] + age_source = person["age"] + try: + female_numeric = female_source.to_numpy(dtype=np.float64, na_value=np.nan) + except (TypeError, ValueError) as exc: + raise SourceRuntimeError( + "Pregnancy structural policy requires boolean is_female." + ) from exc + female = np.isclose(female_numeric, 1.0) + invalid_female = ~np.isfinite(female_numeric) | ~np.isin( + female_numeric, + [0.0, 1.0], + ) + elif set(US_PREGNANCY_REQUIRED_SOURCE_COLUMNS) <= set(person.columns): + female_source = person["A_SEX"] + age_source = person["A_AGE"] + female_numeric = pd.to_numeric(female_source, errors="coerce").to_numpy( + dtype=np.float64 + ) + female = female_numeric == _FEMALE_SEX_CODE + invalid_female = ~np.isfinite(female_numeric) + else: + raise SourceRuntimeError( + "Pregnancy structural policy requires either mapped age/is_female " + "or raw A_AGE/A_SEX columns." + ) + age = pd.to_numeric(age_source, errors="coerce").to_numpy(dtype=np.float64) + invalid_age = ~np.isfinite(age) + if invalid_age.any() or invalid_female.any(): + raise SourceRuntimeError( + "Pregnancy structural policy requires complete sex and age; found " + f"invalid_sex_rows={int(invalid_female.sum())}, " + f"invalid_age_rows={int(invalid_age.sum())}." + ) + low, high = _CHILDBEARING_AGE_RANGE + return female, female & (age >= low) & (age <= high) + + +def _pregnancy_boolean_masks( + person: pd.DataFrame, +) -> tuple[np.ndarray, np.ndarray, int]: + """Return observed/true masks and count non-boolean physical values.""" + + values = person[US_PREGNANCY_OUTPUT_COLUMN] + observed = values.notna().to_numpy(dtype=bool) + try: + numeric = values.to_numpy(dtype=np.float64, na_value=np.nan) + except (TypeError, ValueError): + return observed, np.zeros(len(values), dtype=bool), int(observed.sum()) + valid = ~observed | np.isclose(numeric, 0.0) | np.isclose(numeric, 1.0) + return observed, observed & np.isclose(numeric, 1.0), int((~valid).sum()) + + +def _pregnancy_structural_counts(person: pd.DataFrame) -> dict[str, int]: + """Count every hard-domain and all-clone structural condition.""" + + female, eligible = _pregnancy_eligibility(person) + observed, pregnant, non_boolean = _pregnancy_boolean_masks(person) + missing = int((~observed).sum()) + nonfemale = int((pregnant & ~female).sum()) + outside_age = int((pregnant & female & ~eligible).sum()) + ineligible = int((pregnant & ~eligible).sum()) + source_persons = clone_disagreements = malformed_source_ids = 0 + if _PERSON_SOURCE_ID_COLUMN in person: + source_ids = person[_PERSON_SOURCE_ID_COLUMN] + malformed_source_ids = int(source_ids.isna().sum()) + if not malformed_source_ids: + source_persons = int(source_ids.nunique()) + work = pd.DataFrame( + { + "_source": source_ids.to_numpy(), + "_observed": observed, + "_pregnant": pregnant, + } + ) + observed_work = work.loc[work["_observed"]] + clone_disagreements = int( + observed_work.groupby("_source", sort=False)["_pregnant"] + .nunique() + .gt(1) + .sum() + ) + return { + "missing_rows": missing, + "non_boolean_rows": non_boolean, + "pregnant_nonfemale_rows": nonfemale, + "pregnant_female_outside_age_range_rows": outside_age, + "pregnant_ineligible_rows": ineligible, + "source_persons_checked": source_persons, + "clone_disagreement_source_persons": clone_disagreements, + "malformed_source_id_rows": malformed_source_ids, + } + + +def _require_valid_pregnancy_structure( + person: pd.DataFrame, + *, + boundary: str, + require_complete: bool, +) -> dict[str, int]: + """Refuse a persisted pregnancy surface that violates its hard contract.""" + + counts = _pregnancy_structural_counts(person) + refused = { + "non_boolean_rows": counts["non_boolean_rows"], + "pregnant_ineligible_rows": counts["pregnant_ineligible_rows"], + "clone_disagreement_source_persons": counts[ + "clone_disagreement_source_persons" + ], + "malformed_source_id_rows": counts["malformed_source_id_rows"], + } + if require_complete: + refused["missing_rows"] = counts["missing_rows"] + if any(refused.values()): + rendered = ", ".join(f"{key}={value}" for key, value in refused.items()) + raise SourceRuntimeError( + f"{boundary}: pregnancy structural policy refused {rendered}." + ) + return counts + + def derive_us_pregnancy_from_manifest( frame: pd.DataFrame | None, operation: SourceOperationSpec, @@ -221,10 +347,10 @@ def _pregnancy_carries_signal(person: pd.DataFrame) -> bool: def with_us_pregnancy_inputs(frame: Frame, *, seed: int, time_period: int) -> Frame: """Run the ``pregnancy`` manifest stage over a US frame. - A frame already carrying a non-constant ``is_pregnant`` passes through - untouched (idempotent). Any other surface — column missing, or - constant at the engine default — is reseeded from the raw ASEC - columns. + A frame already carrying a complete, non-constant, structurally valid + ``is_pregnant`` passes through untouched (idempotent). Any other valid + surface — column missing, or constant at the engine default — is reseeded + from the raw ASEC columns. Domain violations are refused. Args: frame: A US-schema frame whose person table still carries the raw @@ -245,10 +371,15 @@ def with_us_pregnancy_inputs(frame: Frame, *, seed: int, time_period: int) -> Fr if frame.schema != US_SCHEMA: raise ValueError("US pregnancy inputs require the US schema.") person = frame.table("person") - if US_PREGNANCY_OUTPUT_COLUMN in person.columns and _pregnancy_carries_signal( - person - ): - return frame + if US_PREGNANCY_OUTPUT_COLUMN in person.columns: + carries_signal = _pregnancy_carries_signal(person) + _require_valid_pregnancy_structure( + person, + boundary="US pregnancy preexisting input", + require_complete=carries_signal, + ) + if carries_signal: + return frame stage_person = person.copy(deep=True) stage_person[_PERSON_WEIGHT_COLUMN] = frame.resolve_weights("person").values @@ -271,6 +402,11 @@ def with_us_pregnancy_inputs(frame: Frame, *, seed: int, time_period: int) -> Fr tables["person"][US_PREGNANCY_OUTPUT_COLUMN] = aligned[ US_PREGNANCY_OUTPUT_COLUMN ].to_numpy(dtype=bool) + _require_valid_pregnancy_structure( + tables["person"], + boundary="US pregnancy stage output", + require_complete=True, + ) return Frame( tables, frame.schema, @@ -287,7 +423,7 @@ def us_pregnancy_summary(frame: Frame) -> dict[str, object]: person = frame.table("person") weights = np.asarray(frame.resolve_weights("person").values, dtype=np.float64) total_weight = float(weights.sum()) - pregnant = person[US_PREGNANCY_OUTPUT_COLUMN].astype(bool).to_numpy() + _observed, pregnant, _non_boolean = _pregnancy_boolean_masks(person) pregnant_share = ( float(weights[pregnant].sum()) / total_weight if total_weight > 0 else 0.0 ) @@ -295,6 +431,7 @@ def us_pregnancy_summary(frame: Frame) -> dict[str, object]: "pregnant_share": pregnant_share, "pregnant_share_band": list(_PREGNANT_SHARE_BAND), "unique_count": int(person[US_PREGNANCY_OUTPUT_COLUMN].dropna().nunique()), + **_pregnancy_structural_counts(person), } @@ -316,7 +453,15 @@ def us_pregnancy_signal_gate(frame: Frame) -> GateResult: ) failures: list[str] = [] - summary = us_pregnancy_summary(frame) + try: + summary = us_pregnancy_summary(frame) + except SourceRuntimeError as exc: + return GateResult( + name="pregnancy_signal", + passed=False, + failures=(str(exc),), + details={"structural_error": str(exc)}, + ) if int(summary["unique_count"]) < 2: failures.append( f"{US_PREGNANCY_OUTPUT_COLUMN}: constant column (one observed " @@ -328,6 +473,16 @@ def us_pregnancy_signal_gate(frame: Frame) -> GateResult: failures.append( f"pregnant share {share:.4f} outside plausibility band [{low}, {high}]." ) + for count_key, label in ( + ("missing_rows", "missing pregnancy row(s)"), + ("non_boolean_rows", "non-boolean pregnancy row(s)"), + ("pregnant_ineligible_rows", "pregnant row(s) outside female ages 15--44"), + ("clone_disagreement_source_persons", "source-person clone disagreement(s)"), + ("malformed_source_id_rows", "malformed person_source_id row(s)"), + ): + count = int(summary[count_key]) + if count: + failures.append(f"{US_PREGNANCY_OUTPUT_COLUMN}: {count} {label}.") return GateResult( name="pregnancy_signal", passed=not failures, diff --git a/packages/microcosm-build/src/microcosm/build/us_runtime/stacked_spine.py b/packages/microcosm-build/src/microcosm/build/us_runtime/stacked_spine.py index 879445e0b..ae762b467 100644 --- a/packages/microcosm-build/src/microcosm/build/us_runtime/stacked_spine.py +++ b/packages/microcosm-build/src/microcosm/build/us_runtime/stacked_spine.py @@ -4179,6 +4179,82 @@ def _validate_acs_transfer_row_counts( return typed_counts +_PREGNANCY_STRUCTURAL_COUNT_FIELDS = ( + "source_persons_checked", + "physical_rows_checked", + "clone_rows_checked", + "donor_rows_checked", + "qrf_draw_source_persons", + "qrf_draw_rows", + "qrf_fanout_rows", + "preexisting_value_fanout_rows", + "ineligible_rows_assigned_false", + "donor_preexisting_domain_violation_rows", + "recipient_preexisting_domain_violation_rows", + "preexisting_clone_disagreement_source_persons", + "inconsistent_eligibility_source_persons", + "maximum_clones_per_source_person", + "final_incomplete_rows", + "final_domain_violation_rows", + "final_clone_disagreement_source_persons", +) + + +def _validate_pregnancy_structural_receipt( + structural: object, + *, + row_counts: Mapping[str, int], + boundary: str, +) -> None: + """Authenticate pregnancy's hard domain and source-person accounting.""" + + expected_policy = acs_transfer_runtime.acs_transfer_execution_contract_identity( + targets=("is_pregnant",), + derive_schedule_d=False, + )["structural_target_policies"]["is_pregnant"] + assert isinstance(expected_policy, Mapping) + if ( + not isinstance(structural, Mapping) + or structural.get("policy_sha256") != expected_policy.get("sha256") + or structural.get("source_person_key") != "person_source_id" + or structural.get("status") != "verified" + ): + raise ValueError(f"{boundary}: pregnancy structural policy is invalid.") + counts = {field: structural.get(field) for field in _PREGNANCY_STRUCTURAL_COUNT_FIELDS} + if any( + not isinstance(value, int) or isinstance(value, bool) or value < 0 + for value in counts.values() + ): + raise ValueError(f"{boundary}: pregnancy structural counts are invalid.") + zero_fields = ( + "donor_preexisting_domain_violation_rows", + "recipient_preexisting_domain_violation_rows", + "preexisting_clone_disagreement_source_persons", + "inconsistent_eligibility_source_persons", + "final_incomplete_rows", + "final_domain_violation_rows", + "final_clone_disagreement_source_persons", + ) + if any(counts[field] != 0 for field in zero_fields): + raise ValueError(f"{boundary}: pregnancy structural guard is not green.") + if ( + counts["physical_rows_checked"] - counts["source_persons_checked"] + != counts["clone_rows_checked"] + or counts["qrf_draw_rows"] != counts["qrf_draw_source_persons"] + or counts["maximum_clones_per_source_person"] < 1 + or ( + counts["qrf_draw_rows"] + + counts["qrf_fanout_rows"] + + counts["preexisting_value_fanout_rows"] + + counts["ineligible_rows_assigned_false"] + != row_counts["imputed_rows"] + ) + ): + raise ValueError( + f"{boundary}: pregnancy structural accounting is invalid." + ) + + def _acs_imputed_pattern_evidence(record: AcsImputedInput) -> dict[str, object]: """Return deterministic JSON-ready evidence for one ACS transfer record.""" @@ -4786,11 +4862,24 @@ def validate_stacked_post_puf_transfer_receipt( f"{boundary}: stacked post-PUF target {target_key!r} has " "no receipt." ) - _validate_acs_transfer_row_counts( + row_counts = _validate_acs_transfer_row_counts( target_receipt, boundary=f"{boundary} target {target_key}", required=True, ) + target_name = target_key.rsplit("/", 1)[1] + structural = target_receipt.get("structural_policy") + if target_name == "is_pregnant": + _validate_pregnancy_structural_receipt( + structural, + row_counts=row_counts, + boundary=f"{boundary} target {target_key}", + ) + elif structural is not None: + raise ValueError( + f"{boundary}: undeclared pregnancy structural evidence is " + f"attached to {target_key!r}." + ) owner_receipt = target_receipt.get("post_transfer_calibration") spec = expected_calibrations.get(target_key) if spec is None: @@ -10499,6 +10588,10 @@ def _verify_post_puf_transfer_outcome( target_receipt["qrf_pattern_evidence"] = ( _acs_imputed_pattern_evidence(record) ) + if record is not None and record.structural_receipt is not None: + target_receipt["structural_policy"] = dict( + record.structural_receipt + ) target_receipts[target_receipt_key] = target_receipt if failures: raise ValueError( diff --git a/packages/microcosm-build/src/microcosm/build/us_runtime/us_late_producer_registry.py b/packages/microcosm-build/src/microcosm/build/us_runtime/us_late_producer_registry.py index 1b0f0cc6c..4b8206fa6 100644 --- a/packages/microcosm-build/src/microcosm/build/us_runtime/us_late_producer_registry.py +++ b/packages/microcosm-build/src/microcosm/build/us_runtime/us_late_producer_registry.py @@ -671,6 +671,13 @@ def _inventory( "with_us_pregnancy_inputs", *_raw_person_requirements(("A_SEX", "A_AGE")), _single("person_id", "person", "person_id"), + _single( + "pregnancy_structural_source_person_id", + "person", + "person_source_id", + optional=True, + value_kind="finite_numeric", + ), _single("resolved_person_weight", "person", "@resolved_weight"), _requirement( "stable_source_identity", @@ -1360,6 +1367,12 @@ def _bounded_transfer_groups( batches: list[tuple[str, ...]] = [] current: list[str] = [] for atom in atoms: + if atom == ("is_pregnant",): + if current: + batches.append(tuple(current)) + current = [] + batches.append(atom) + continue if current and len(current) + len(atom) > max_targets_per_fit: batches.append(tuple(current)) current = [] diff --git a/packages/microcosm-build/tests/test_spec_engine_compiler_ir.py b/packages/microcosm-build/tests/test_spec_engine_compiler_ir.py index 393d98dd1..fb141dec1 100644 --- a/packages/microcosm-build/tests/test_spec_engine_compiler_ir.py +++ b/packages/microcosm-build/tests/test_spec_engine_compiler_ir.py @@ -23,7 +23,7 @@ from microcosm.build.spec_engine.resolver import F0_CONTRACT_ONLY_KERNEL_IDS US_SCHEDULE_SHA256 = ( - "dcf3c6d2eade3449836c49a1dc4d3b8cd395aab9142db700c3c60598fa9c1c79" + "e59c019d3d454eac99ac0ac209b6c5b6faaf9bdfcaeee18c36a25be19bf7da2f" ) diff --git a/packages/microcosm-build/tests/test_spec_engine_coverage_tool.py b/packages/microcosm-build/tests/test_spec_engine_coverage_tool.py index 0bd070285..099e68813 100644 --- a/packages/microcosm-build/tests/test_spec_engine_coverage_tool.py +++ b/packages/microcosm-build/tests/test_spec_engine_coverage_tool.py @@ -52,21 +52,21 @@ def test_us_coverage_is_exact_complete_and_honest( assert_coverage_complete(coverage_report) assert coverage_report["status"] == "pass" fields = coverage_report["field_usage"] - assert fields["configuration_field_count"] == 42_122 - assert fields["authored_normative_field_count"] == 32_352 + assert fields["configuration_field_count"] == 42_154 + assert fields["authored_normative_field_count"] == 32_384 assert fields["resolved_binding_field_count"] == 9_770 - assert fields["consumed_field_count"] == 42_122 + assert fields["consumed_field_count"] == 42_154 assert fields["unused_field_count"] == 0 assert fields["multiple_primary_use_field_count"] == 0 assert fields["claim_count"] == 49 assert fields["mode_counts"] == { - "legacy_behavior": 13_971, - "compiler_semantic": 27_700, + "legacy_behavior": 13_988, + "compiler_semantic": 27_715, "front_end_validation": 348, "identity_only": 103, } assert fields["generation0_effect_counts"] == { - "legacy_behavior": 38_444, + "legacy_behavior": 38_476, "no_generation0_effect": 3_678, } @@ -75,7 +75,7 @@ def test_us_coverage_is_exact_complete_and_honest( assert inventory["covered_item_count"] == 41 assert inventory["missing_item_count"] == 0 assert inventory["missing_items"] == [] - assert inventory["counts"]["producer_inputs"] == 2_742 + assert inventory["counts"]["producer_inputs"] == 2_744 assert inventory["counts"]["ownership_rows"] == 18 assert inventory["counts"]["tail_control_fields"] == 934 assert inventory["counts"]["seed_owner_bindings"] == 112 diff --git a/packages/microcosm-build/tests/test_spec_engine_field_usage.py b/packages/microcosm-build/tests/test_spec_engine_field_usage.py index fb6b525b4..d33908b81 100644 --- a/packages/microcosm-build/tests/test_spec_engine_field_usage.py +++ b/packages/microcosm-build/tests/test_spec_engine_field_usage.py @@ -92,22 +92,22 @@ def _mutated_bundle( def test_exact_complete_ledger_has_one_primary_mode_per_pointer(field_ledger) -> None: - assert len(field_ledger.fields) == EXPECTED_CONFIGURATION_FIELD_COUNT == 42_122 + assert len(field_ledger.fields) == EXPECTED_CONFIGURATION_FIELD_COUNT == 42_154 assert field_ledger.source_counts == { - "authored": 32_352, + "authored": 32_384, "resolved_bindings": 9_770, } assert field_ledger.mode_counts == { - "legacy_behavior": 13_971, - "compiler_semantic": 27_700, + "legacy_behavior": 13_988, + "compiler_semantic": 27_715, "front_end_validation": 348, "identity_only": 103, } assert field_ledger.generation0_effect_counts == { - "legacy_behavior": 38_444, + "legacy_behavior": 38_476, "no_generation0_effect": 3_678, } - assert len({field.pointer for field in field_ledger.fields}) == 42_122 + assert len({field.pointer for field in field_ledger.fields}) == 42_154 def test_eligibility_concepts_are_validation_not_generation0_behavior( diff --git a/packages/microcosm-build/tests/test_spec_engine_imputation_semantics.py b/packages/microcosm-build/tests/test_spec_engine_imputation_semantics.py index 749a793b2..c9ae76814 100644 --- a/packages/microcosm-build/tests/test_spec_engine_imputation_semantics.py +++ b/packages/microcosm-build/tests/test_spec_engine_imputation_semantics.py @@ -98,7 +98,7 @@ def test_imputation_projector_matches_live_plans_and_graph_receipts( for key, expected in live.items(): assert canonical_json_bytes(projected[key]) == canonical_json_bytes(expected) assert projected["late_producer_schedule_receipt"]["schedule_sha256"] == ( - "dcf3c6d2eade3449836c49a1dc4d3b8cd395aab9142db700c3c60598fa9c1c79" + "e59c019d3d454eac99ac0ac209b6c5b6faaf9bdfcaeee18c36a25be19bf7da2f" ) assert projected["overlap_ownership"]["sha256"] == ( "5f64f0aac49e2313177564f71876bffc8c81b3ded4df701e70930e60e9c98356" diff --git a/packages/microcosm-build/tests/test_spec_engine_inventory_coverage.py b/packages/microcosm-build/tests/test_spec_engine_inventory_coverage.py index 8f4e2673c..922d23379 100644 --- a/packages/microcosm-build/tests/test_spec_engine_inventory_coverage.py +++ b/packages/microcosm-build/tests/test_spec_engine_inventory_coverage.py @@ -79,7 +79,7 @@ "primary_targets": 65, "producer_authored_outputs": 92, "producer_compiled_outputs": 227, - "producer_inputs": 2_742, + "producer_inputs": 2_744, "producer_nodes": 38, "producer_virtual_resources": 75, "release_rungs": 5, diff --git a/packages/microcosm-build/tests/test_spec_engine_legacy_adapter.py b/packages/microcosm-build/tests/test_spec_engine_legacy_adapter.py index 53f05be22..9edc4e020 100644 --- a/packages/microcosm-build/tests/test_spec_engine_legacy_adapter.py +++ b/packages/microcosm-build/tests/test_spec_engine_legacy_adapter.py @@ -195,10 +195,10 @@ def test_adapter_preserves_generation_zero_identity_components( imputation = legacy_payload["imputation"] assert isinstance(imputation, dict) assert legacy_payload["stacked_authority_receipt"]["sha256"] == ( - "3a980927227704d0589f246eef9cd825c2ae84f3a4134ac835e0e5ed39a563ac" + "e660a8ce42b69a39d29c5f0ec37264bc69d61b03f27adc386336ec8889531bb2" ) assert imputation["late_producer_schedule_receipt"]["schedule_sha256"] == ( - "dcf3c6d2eade3449836c49a1dc4d3b8cd395aab9142db700c3c60598fa9c1c79" + "e59c019d3d454eac99ac0ac209b6c5b6faaf9bdfcaeee18c36a25be19bf7da2f" ) assert imputation["overlap_ownership"]["sha256"] == ( "5f64f0aac49e2313177564f71876bffc8c81b3ded4df701e70930e60e9c98356" diff --git a/packages/microcosm-build/tests/test_spec_engine_stacked_authority_semantics.py b/packages/microcosm-build/tests/test_spec_engine_stacked_authority_semantics.py index c378f3dbd..1096b4092 100644 --- a/packages/microcosm-build/tests/test_spec_engine_stacked_authority_semantics.py +++ b/packages/microcosm-build/tests/test_spec_engine_stacked_authority_semantics.py @@ -73,7 +73,7 @@ def test_authority_projection_is_field_and_byte_identical_to_live_generation_zer assert projected == live assert stacked_identity_bytes(projected) == _canonical_bytes(live) assert projected["sha256"] == ( - "3a980927227704d0589f246eef9cd825c2ae84f3a4134ac835e0e5ed39a563ac" + "e660a8ce42b69a39d29c5f0ec37264bc69d61b03f27adc386336ec8889531bb2" ) assert { name: component["sha256"] for name, component in projected["components"].items() @@ -88,7 +88,7 @@ def test_authority_projection_is_field_and_byte_identical_to_live_generation_zer "cacc6c11e114dbae3aaa2761cc6b3fcb1191cd9b689b1c2bd096614c51ebff8b" ), "late_producer_schedule": ( - "9b15db577b85c796944e8eb267500d5d662f2a0eee77b25c1e4241c7d9620473" + "1b81157b0e21e4763884620ec27b5c4e6e36cc28237273c24eeccdef05a7fbca" ), "metric_registry": ( "d75cb9b29f8b0a9a085471a11f4c19c32ba04cbe5419053df94ea81cbe6125a9" @@ -191,7 +191,7 @@ def test_checkpoint_projection_is_field_and_byte_identical_to_live_oracle( ] assert ( projected["pool_code"]["late_producer_schedule"]["schedule_sha256"] - == "dcf3c6d2eade3449836c49a1dc4d3b8cd395aab9142db700c3c60598fa9c1c79" + == "e59c019d3d454eac99ac0ac209b6c5b6faaf9bdfcaeee18c36a25be19bf7da2f" ) diff --git a/packages/microcosm-build/tests/test_us_acs_transfer.py b/packages/microcosm-build/tests/test_us_acs_transfer.py index f08c84c30..9ef1acbec 100644 --- a/packages/microcosm-build/tests/test_us_acs_transfer.py +++ b/packages/microcosm-build/tests/test_us_acs_transfer.py @@ -1430,6 +1430,296 @@ def test_integer_supported_weeks_predictions_snap_to_observed_donor_support( assert "weeks_unemployed" in contract["discrete_numeric_targets"] +def _pregnancy_donor() -> Frame: + return _with_columns( + _with_columns( + _donor_frame(), + "person", + { + "age": [30.0] * 8, + "is_female": [True] * 8, + }, + ), + "person", + {"is_pregnant": [True] * 6 + [False] * 2}, + ) + + +def test_pregnancy_draws_once_per_eligible_source_person_and_fans_to_clones( + monkeypatch: pytest.MonkeyPatch, +) -> None: + base = _with_full_us_schema(_recipient_frame()) + assembled = assemble_spines( + {"asec": base, "acs": base}, + household_mass_shares={"asec": 0.5, "acs": 0.5}, + ) + recipient = clone_us_frame_for_puf_support(assembled) + monkeypatch.setattr(acs_transfer_module, "QRF", _MeanQRF) + _MeanQRF.calls = [] + + result = transfer_acs_inputs( + recipient, + _with_full_us_schema(_pregnancy_donor()), + target_families={"person": {"pregnancy": ("is_pregnant",)}}, + donor_channel=None, + seed=9, + n_estimators=1, + ) + + person = result.frame.person + eligible = ( + person["is_female"].astype(bool) + & person["age"].between(15, 44, inclusive="both") + ) + assert person.loc[eligible, "is_pregnant"].all() + assert not person.loc[~eligible, "is_pregnant"].any() + assert ( + person.groupby("person_source_id", sort=False)["is_pregnant"] + .nunique() + .eq(1) + .all() + ) + expected_qrf_sources = int( + pd.DataFrame( + { + "source": person["person_source_id"], + "eligible": eligible, + } + ) + .groupby("source", sort=False)["eligible"] + .first() + .sum() + ) + record = next(item for item in result.imputed_inputs if item.column == "is_pregnant") + receipt = record.structural_receipt + assert receipt is not None + assert sum(pattern.recipient_rows for pattern in record.patterns) == ( + expected_qrf_sources + ) + assert receipt["qrf_draw_source_persons"] == expected_qrf_sources + assert receipt["qrf_draw_rows"] == expected_qrf_sources + assert receipt["qrf_fanout_rows"] == expected_qrf_sources + assert receipt["final_domain_violation_rows"] == 0 + assert receipt["final_clone_disagreement_source_persons"] == 0 + + +def test_pregnancy_all_ineligible_recipients_bypass_qrf( + monkeypatch: pytest.MonkeyPatch, +) -> None: + recipient = _with_columns( + _recipient_frame(), + "person", + { + "age": [30.0] * 6, + "is_female": [False] * 6, + }, + ) + monkeypatch.setattr(acs_transfer_module, "QRF", _MeanQRF) + _MeanQRF.calls = [] + + result = transfer_acs_inputs( + recipient, + _pregnancy_donor(), + target_families={"person": {"pregnancy": ("is_pregnant",)}}, + donor_channel=None, + n_estimators=1, + ) + + assert not result.frame.person["is_pregnant"].any() + assert not _MeanQRF.calls + record = result.imputed_inputs[0] + assert record.weight_kind == "structural" + receipt = record.structural_receipt + assert receipt is not None + assert receipt["qrf_draw_source_persons"] == 0 + assert receipt["qrf_draw_rows"] == 0 + assert receipt["ineligible_rows_assigned_false"] == 6 + + +def test_pregnancy_partial_clone_fanout_receipt_categories_are_disjoint( + monkeypatch: pytest.MonkeyPatch, +) -> None: + base = _with_full_us_schema(_recipient_frame()) + assembled = assemble_spines( + {"asec": base, "acs": base}, + household_mass_shares={"asec": 0.5, "acs": 0.5}, + ) + cloned = clone_us_frame_for_puf_support(assembled) + person = cloned.person + clone_zero = person["person_support_clone_index"].eq(0).to_numpy() + pregnancy = pd.array([pd.NA] * len(person), dtype="boolean") + pregnancy[clone_zero] = False + recipient = _with_columns( + cloned, + "person", + {"is_pregnant": pregnancy}, + ) + monkeypatch.setattr(acs_transfer_module, "QRF", _MeanQRF) + _MeanQRF.calls = [] + + result = transfer_acs_inputs( + recipient, + _with_full_us_schema(_pregnancy_donor()), + target_families={"person": {"pregnancy": ("is_pregnant",)}}, + donor_channel=None, + n_estimators=1, + ) + + assert not _MeanQRF.calls + output = result.frame.person + assert not output["is_pregnant"].any() + missing = ~clone_zero + eligible = ( + output["is_female"].astype(bool).to_numpy() + & output["age"].between(15, 44, inclusive="both").to_numpy() + ) + record = result.imputed_inputs[0] + receipt = record.structural_receipt + assert receipt is not None + assert receipt["preexisting_value_fanout_rows"] == int( + (missing & eligible).sum() + ) + assert receipt["ineligible_rows_assigned_false"] == int( + (missing & ~eligible).sum() + ) + assert ( + receipt["preexisting_value_fanout_rows"] + + receipt["ineligible_rows_assigned_false"] + == record.imputed_recipient_rows + ) + + +def test_pregnancy_invalid_donor_is_refused_before_qrf( + monkeypatch: pytest.MonkeyPatch, +) -> None: + donor = _with_columns( + _donor_frame(), + "person", + {"is_pregnant": [True] + [False] * 7}, + ) + monkeypatch.setattr(acs_transfer_module, "QRF", _MeanQRF) + _MeanQRF.calls = [] + + with pytest.raises( + ValueError, + match=r"1 preexisting donor domain violation", + ): + transfer_acs_inputs( + _recipient_frame(), + donor, + target_families={"person": {"pregnancy": ("is_pregnant",)}}, + donor_channel=None, + n_estimators=1, + ) + assert not _MeanQRF.calls + + +def test_pregnancy_complete_invalid_recipient_is_still_refused( + monkeypatch: pytest.MonkeyPatch, +) -> None: + recipient = _with_columns( + _recipient_frame(), + "person", + {"is_pregnant": [True] + [False] * 5}, + ) + monkeypatch.setattr(acs_transfer_module, "QRF", _MeanQRF) + _MeanQRF.calls = [] + + with pytest.raises( + ValueError, + match=r"1 preexisting recipient domain violation", + ): + transfer_acs_inputs( + recipient, + _pregnancy_donor(), + target_families={"person": {"pregnancy": ("is_pregnant",)}}, + donor_channel=None, + n_estimators=1, + ) + assert not _MeanQRF.calls + + +def test_complete_invalid_pregnancy_is_refused_before_other_active_target( + monkeypatch: pytest.MonkeyPatch, +) -> None: + recipient = _with_columns( + _recipient_frame(), + "person", + {"is_pregnant": [True] + [False] * 5}, + ) + monkeypatch.setattr(acs_transfer_module, "QRF", _MeanQRF) + _MeanQRF.calls = [] + + with pytest.raises( + ValueError, + match=r"1 preexisting recipient domain violation", + ): + transfer_acs_inputs( + recipient, + _pregnancy_donor(), + target_families={ + "person": { + "pregnancy": ("is_pregnant",), + "income": ("qualified_dividend_income",), + } + }, + donor_channel=None, + n_estimators=1, + ) + assert not _MeanQRF.calls + + +def test_pregnancy_complete_valid_recipient_retains_structural_receipt( + monkeypatch: pytest.MonkeyPatch, +) -> None: + recipient = _with_columns( + _recipient_frame(), + "person", + {"is_pregnant": [False] * 6}, + ) + monkeypatch.setattr(acs_transfer_module, "QRF", _MeanQRF) + _MeanQRF.calls = [] + + result = transfer_acs_inputs( + recipient, + _pregnancy_donor(), + target_families={"person": {"pregnancy": ("is_pregnant",)}}, + donor_channel=None, + n_estimators=1, + ) + + assert not _MeanQRF.calls + assert len(result.imputed_inputs) == 1 + record = result.imputed_inputs[0] + assert record.imputed_recipient_rows == 0 + assert record.unmodeled_recipient_rows == 0 + assert record.structural_receipt is not None + assert record.structural_receipt["status"] == "verified" + + +def test_pregnancy_policy_is_bound_into_transfer_execution_identity() -> None: + enabled = acs_transfer_module.acs_transfer_execution_contract_identity( + targets=("is_pregnant",), + derive_schedule_d=False, + ) + disabled = acs_transfer_module.acs_transfer_execution_contract_identity( + targets=(), + derive_schedule_d=False, + ) + + policy = enabled["structural_target_policies"]["is_pregnant"] + assert policy["enabled"] is True + assert policy["eligibility"] == { + "is_female": True, + "minimum_age_inclusive": 15, + "maximum_age_inclusive": 44, + } + assert policy["qrf_scope"] == "one_eligible_representative_per_source_person" + assert policy["preexisting_domain_violations"] == "refuse" + assert disabled["structural_target_policies"]["is_pregnant"]["enabled"] is False + assert enabled["sha256"] != disabled["sha256"] + + def test_engine_boolean_metadata_restores_primary_qrf_float_h5_donor( monkeypatch: pytest.MonkeyPatch, tmp_path: Path, diff --git a/packages/microcosm-build/tests/test_us_multispine_pool_tool.py b/packages/microcosm-build/tests/test_us_multispine_pool_tool.py index 473daf364..51e738dfb 100644 --- a/packages/microcosm-build/tests/test_us_multispine_pool_tool.py +++ b/packages/microcosm-build/tests/test_us_multispine_pool_tool.py @@ -2465,7 +2465,7 @@ def capture_equality(expected: object, actual: object) -> None: "country": "us", "schema_id": "country_spec", "schema_version": 1, - "spec_sha256": "5f44d96d45e9aabcea2d565ef063d68bfc0652df1b38b08aa31ce6896d15f371", + "spec_sha256": "b7b7037eeab3258d38968a8a5428c65df50bb3b0ee892c660488bf8df3cbe90f", }, } diff --git a/packages/microcosm-build/tests/test_us_pregnancy.py b/packages/microcosm-build/tests/test_us_pregnancy.py index 056a869b3..6748966aa 100644 --- a/packages/microcosm-build/tests/test_us_pregnancy.py +++ b/packages/microcosm-build/tests/test_us_pregnancy.py @@ -248,6 +248,57 @@ def test_missing_raw_columns_without_signal_raise(self) -> None: with pytest.raises(SourceRuntimeError, match="A_SEX"): with_us_pregnancy_inputs(stripped, seed=0, time_period=TIME_PERIOD) + @pytest.mark.parametrize( + "invalid_row", + ( + {"A_SEX": _MALE, "A_AGE": 25, "is_pregnant": True}, + {"A_SEX": _FEMALE, "A_AGE": 14, "is_pregnant": True}, + {"A_SEX": _FEMALE, "A_AGE": 45, "is_pregnant": True}, + ), + ) + def test_preexisting_domain_violations_are_refused( + self, + invalid_row: dict[str, object], + ) -> None: + rows = [ + invalid_row, + {"A_SEX": _FEMALE, "A_AGE": 25, "is_pregnant": False}, + ] + with pytest.raises( + SourceRuntimeError, + match=r"pregnant_ineligible_rows=1", + ): + with_us_pregnancy_inputs( + _us_frame(rows), + seed=0, + time_period=TIME_PERIOD, + ) + + def test_preexisting_clone_disagreement_is_refused(self) -> None: + rows = [ + { + "A_SEX": _FEMALE, + "A_AGE": 25, + "person_source_id": 42, + "is_pregnant": True, + }, + { + "A_SEX": _FEMALE, + "A_AGE": 25, + "person_source_id": 42, + "is_pregnant": False, + }, + ] + with pytest.raises( + SourceRuntimeError, + match=r"clone_disagreement_source_persons=1", + ): + with_us_pregnancy_inputs( + _us_frame(rows), + seed=0, + time_period=TIME_PERIOD, + ) + class TestGate: def test_plausible_seeded_share_passes(self) -> None: @@ -277,3 +328,15 @@ def test_everyone_pregnant_fails_the_share_band(self) -> None: gate = us_pregnancy_signal_gate(_us_frame(rows)) assert not gate.passed assert any("pregnant share" in failure for failure in gate.failures) + + def test_domain_violation_fails_with_structural_counts(self) -> None: + rows = [ + {"A_SEX": _MALE, "A_AGE": 25, "is_pregnant": True}, + {"A_SEX": _FEMALE, "A_AGE": 25, "is_pregnant": False}, + ] + gate = us_pregnancy_signal_gate(_us_frame(rows)) + + assert not gate.passed + assert gate.details["pregnant_nonfemale_rows"] == 1 + assert gate.details["pregnant_ineligible_rows"] == 1 + assert any("outside female ages 15--44" in item for item in gate.failures) diff --git a/packages/microcosm-build/tests/test_us_spec_bundle.py b/packages/microcosm-build/tests/test_us_spec_bundle.py index 586907804..892ece523 100644 --- a/packages/microcosm-build/tests/test_us_spec_bundle.py +++ b/packages/microcosm-build/tests/test_us_spec_bundle.py @@ -382,11 +382,11 @@ def test_constant_derived_domain_counts_are_complete( assert len(compiled_schedule["waves"]) == 6 assert ( compiled_schedule["schedule_sha256"] - == "dcf3c6d2eade3449836c49a1dc4d3b8cd395aab9142db700c3c60598fa9c1c79" + == "e59c019d3d454eac99ac0ac209b6c5b6faaf9bdfcaeee18c36a25be19bf7da2f" ) assert ( compiled_schedule["payload_sha256"] - == "5921cda83725b2801f2713242003e99ba54766851808b94a4f483666bce604c5" + == "02e618cc656eb39990ed99dca2b30a52794e01e2b06a3c2df87ca4a7d85ab086" ) assert len(take_up["programs"]) == 17 @@ -743,7 +743,20 @@ def collect_sha256(value: object, path: tuple[str, ...] = ()) -> None: "sha256" ] ), - ) + ), + ( + ( + "transfer_execution", + "structural_target_policies", + "is_pregnant", + "sha256", + ), + str( + imputation["transfer_execution"]["structural_target_policies"][ + "is_pregnant" + ]["sha256"] + ), + ), ] assert len(asset_pins) == 2 assert set(asset_pins) == { diff --git a/packages/microcosm-build/tests/test_us_stacked_spine.py b/packages/microcosm-build/tests/test_us_stacked_spine.py index 6036bdfc1..0d4556814 100644 --- a/packages/microcosm-build/tests/test_us_stacked_spine.py +++ b/packages/microcosm-build/tests/test_us_stacked_spine.py @@ -3555,10 +3555,13 @@ def _post_puf_transfer_fixture() -> Frame: index=person.index, dtype="boolean", ) - person.loc[source_producer_rows, "is_pregnant"] = np.resize( - np.asarray([True, False]), - int(source_producer_rows.sum()), + source_eligible = ( + source_producer_rows + & person["is_female"].astype(bool) + & person["age"].between(15, 44, inclusive="both") ) + person.loc[source_producer_rows, "is_pregnant"] = False + person.loc[source_eligible, "is_pregnant"] = True tables = {entity: attached.table(entity) for entity in attached.entities} tables["person"] = person return Frame( @@ -5442,6 +5445,32 @@ def transfer( "unmodeled_rows": 0, "residual_null_rows": 0, } + if target == "is_pregnant": + pregnancy_policy = execution_contract[ + "structural_target_policies" + ]["is_pregnant"] + target_receipt["structural_policy"] = { + "policy_sha256": pregnancy_policy["sha256"], + "source_person_key": "person_source_id", + "source_persons_checked": 1, + "physical_rows_checked": 1, + "clone_rows_checked": 0, + "donor_rows_checked": 1, + "qrf_draw_source_persons": 1, + "qrf_draw_rows": 1, + "qrf_fanout_rows": 0, + "preexisting_value_fanout_rows": 0, + "ineligible_rows_assigned_false": 0, + "donor_preexisting_domain_violation_rows": 0, + "recipient_preexisting_domain_violation_rows": 0, + "preexisting_clone_disagreement_source_persons": 0, + "inconsistent_eligibility_source_persons": 0, + "maximum_clones_per_source_person": 1, + "final_incomplete_rows": 0, + "final_domain_violation_rows": 0, + "final_clone_disagreement_source_persons": 0, + "status": "verified", + } if key in late_specs: target_receipt["qrf_pattern_evidence"] = ( stacked_spine_module._acs_imputed_pattern_evidence(record) @@ -6316,6 +6345,104 @@ def test_post_puf_transfer_preserves_complete_asec_source_producers() -> None: if item.column == "is_pregnant" ) assert all(not pattern.target_regimes for pattern in record.patterns) + structural = receipt["structural_policy"] + assert structural == record.structural_receipt + assert structural["status"] == "verified" + assert structural["source_person_key"] == "person_source_id" + assert structural["qrf_draw_rows"] < receipt["imputed_rows"] + assert structural["final_domain_violation_rows"] == 0 + assert structural["final_clone_disagreement_source_persons"] == 0 + assert ( + person.groupby("person_source_id", sort=False)["is_pregnant"] + .nunique() + .eq(1) + .all() + ) + + +def test_complete_pregnancy_surface_retains_zero_imputation_structural_receipt() -> None: + frame = _post_puf_transfer_fixture() + person = frame.table("person").copy() + recipient_rows = person[support_channel_column("person")].astype(str).eq("acs") + person.loc[recipient_rows, "is_pregnant"] = False + tables = {entity: frame.table(entity) for entity in frame.entities} + tables["person"] = person + complete = Frame( + tables, + frame.schema, + {entity: frame.weights_for(entity) for entity in frame.weighted_entities}, + frame.strata, + mass_log=frame.mass_log, + metadata=frame.metadata, + ) + surface = {"person": {"model_required_boolean": ("is_pregnant",)}} + authority = stacked_spine_module._make_test_stacked_authority( + declared_surface=surface, + gap_fill_plan=(), + post_puf_transfer_surface=surface, + ) + + result = stacked_spine_module._transfer_stacked_post_puf_inputs_with_test_authority( + complete, + authority=authority, + seed=578, + n_estimators=10, + ) + + receipt = result.receipt["targets"][ + "person/model_required_boolean/is_pregnant" + ] + assert receipt["authorized_null_rows"] == 0 + assert receipt["imputed_rows"] == 0 + assert receipt["structural_policy"]["status"] == "verified" + record = next( + item + for item in result.transfer_result.imputed_inputs + if item.column == "is_pregnant" + ) + assert record.imputed_recipient_rows == 0 + assert record.structural_receipt == receipt["structural_policy"] + + +def test_post_puf_transfer_refuses_invalid_pregnancy_source_producer() -> None: + frame = _post_puf_transfer_fixture() + person = frame.table("person").copy() + source_rows = person[support_channel_column("person")].astype(str).eq("asec") + ineligible = source_rows & ~( + person["is_female"].astype(bool) + & person["age"].between(15, 44, inclusive="both") + ) + donor_ineligible = ineligible & person[ + support_clone_index_column("person") + ].eq(1) + person.loc[person.index[donor_ineligible][0], "is_pregnant"] = True + tables = {entity: frame.table(entity) for entity in frame.entities} + tables["person"] = person + invalid = Frame( + tables, + frame.schema, + {entity: frame.weights_for(entity) for entity in frame.weighted_entities}, + frame.strata, + mass_log=frame.mass_log, + metadata=frame.metadata, + ) + surface = {"person": {"model_required_boolean": ("is_pregnant",)}} + authority = stacked_spine_module._make_test_stacked_authority( + declared_surface=surface, + gap_fill_plan=(), + post_puf_transfer_surface=surface, + ) + + with pytest.raises( + ValueError, + match=r"preexisting donor domain violation", + ): + stacked_spine_module._transfer_stacked_post_puf_inputs_with_test_authority( + invalid, + authority=authority, + seed=578, + n_estimators=10, + ) def test_late_calibration_owner_mutates_only_acs_clone_zero_transfer_cells() -> None: diff --git a/tools/spec_engine_coverage.py b/tools/spec_engine_coverage.py index 5bd0d07d3..3f5ae7e76 100644 --- a/tools/spec_engine_coverage.py +++ b/tools/spec_engine_coverage.py @@ -41,7 +41,7 @@ REPORT_SCHEMA_VERSION = 3 EXPECTED_POINTER_INVENTORY_SHA256 = ( - "ae7881c07c6137037f43a5992b3e8fffbbdbd5ddf2e1b70956986cb9df043f71" + "3fc6b9480ea81b9635bd0db56e180c2daf32a5cd2006a70d350586c570f96754" ) DEFAULT_REPORT_PATH = ( Path(__file__).resolve().parents[1] diff --git a/tools/us_bundle_generation/imputation.py b/tools/us_bundle_generation/imputation.py index d80696bf4..42a11953a 100644 --- a/tools/us_bundle_generation/imputation.py +++ b/tools/us_bundle_generation/imputation.py @@ -2144,15 +2144,20 @@ def _assert_invariants( for requirement in node["inputs"] for receipt in requirement["tolerated_absence_receipts"] ] - if ( + graph_counts = ( input_count, authored_output_count, compiled_output_count, len(primary_node["outputs"]), late_authored_output_count, len(tolerated_receipts), - ) != (2742, 92, 227, 35, 0, 212): - raise RuntimeError("US producer graph input/output/absence counts changed.") + ) + expected_graph_counts = (2744, 92, 227, 35, 0, 213) + if graph_counts != expected_graph_counts: + raise RuntimeError( + "US producer graph input/output/absence counts changed: " + f"observed={graph_counts}, expected={expected_graph_counts}." + ) if len(set(tolerated_receipts)) != len(tolerated_receipts): raise RuntimeError("US producer graph absence receipt IDs are not unique.") waived = { From e59ab046c946c64f593ab41c6c218f3c9a72d7b4 Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Thu, 27 Aug 2026 16:15:00 +0200 Subject: [PATCH 42/49] Harden pregnancy and rung receipts --- PROGRESS.md | 22 +++++- docs/evidence/spec-engine/us-f0-coverage.json | 16 ++-- .../build/spec_engine/inventory_coverage.py | 4 +- .../build/us_runtime/acs_transfer.py | 14 ++-- .../src/microcosm/build/us_runtime/h5_io.py | 17 ++++- .../microcosm/build/us_runtime/pregnancy.py | 6 +- .../tests/test_us_acs_transfer.py | 22 ++++++ .../tests/test_us_multispine_pool_h5_io.py | 76 +++++++++++++++++++ .../tests/test_us_multispine_pool_tool.py | 60 ++++++++++++++- .../tests/test_us_pregnancy.py | 12 +++ 10 files changed, 219 insertions(+), 30 deletions(-) diff --git a/PROGRESS.md b/PROGRESS.md index d9b76876f..5de43f21a 100644 --- a/PROGRESS.md +++ b/PROGRESS.md @@ -81,14 +81,28 @@ push pool or release artifacts. - Extended the real-pool pregnancy decomposition: the 108 pregnant nonfemale rows are joined by 58 pregnant female rows outside ages 15--44, for 166 hard- domain violations, all on ACS and all isolated to one clone. ASEC has zero. +- Committed the coherent pregnancy implementation, generated authority, + coverage proof, changelog, tests, and journal as `01a80f49`. +- An adversarial pre-certification audit found that two canonical late-DAG test + fixtures predated the required pregnancy structural receipt. Stopped the + preliminary build-shard run, attached valid zero-imputation proofs to both + fixtures, and added a forged-policy regression. +- Tightened structural types while that integration repair was open: near-0/1 + pregnancy values no longer pass as booleans, and nested stack/arm + fraction/seed/count fields no longer accept JSON boolean equality aliases. + Added source, transfer, and four manifest-alias regressions. The complete + combined pregnancy/ACS/H5 run reached its final live-spec fixture with every + preceding test green; the repinned fixture and forged-policy controls pass. +- Recomputed the source-attested identity after hardening: final resolved US + spec SHA is `11e310c7619cbac91f6703b9679649cdd15f6fb09274ad29904c65881aa93316`; + no authored YAML changed. Regenerated coverage remains complete at + 42,154/42,154 fields and 41/41 inventory checks. ## Next -- Complete the source-attested spec/coverage repin and rerun all directly - affected spec, transfer, and stacked suites. -- Commit the coherent pregnancy implementation and journal state. +- Commit the fixture/type-hardening integration repair and journal state. - Run repository Ruff, generated/coverage/inventory checks, and all five pytest - shards in independent processes. + shards in independent processes, restarting the build shard from zero. - Write and commit the complete handoff to `out.md`. # Weeksgate: stacked release gates and integer-week provenance diff --git a/docs/evidence/spec-engine/us-f0-coverage.json b/docs/evidence/spec-engine/us-f0-coverage.json index 5a98d725c..5a24b5c77 100644 --- a/docs/evidence/spec-engine/us-f0-coverage.json +++ b/docs/evidence/spec-engine/us-f0-coverage.json @@ -1656,13 +1656,13 @@ "compiler_ir.node_slices" ], "expected": { - "map_sha256": "9ff34c4ff7c6d744a8385c2aee6e4eb2ea30a630e56b29ac555381987058da09", - "protocol_sha256": "01c2bf7e06a589ec2c7e99c24e1413b3dfbf02206241a216d843a95082547a34" + "map_sha256": "545d8007bf4f20638cf43f097bcac11559b6aa92f71b9282e965ed672c50cdb9", + "protocol_sha256": "b07caa958217fe3602ca52cf9e39f5b577fa9a94eb4a5722533b223b6714e0e3" }, "failures": [], "observed": { - "map_sha256": "9ff34c4ff7c6d744a8385c2aee6e4eb2ea30a630e56b29ac555381987058da09", - "protocol_sha256": "01c2bf7e06a589ec2c7e99c24e1413b3dfbf02206241a216d843a95082547a34" + "map_sha256": "545d8007bf4f20638cf43f097bcac11559b6aa92f71b9282e965ed672c50cdb9", + "protocol_sha256": "b07caa958217fe3602ca52cf9e39f5b577fa9a94eb4a5722533b223b6714e0e3" }, "status": "covered" }, @@ -1677,7 +1677,7 @@ "compiler_ir.seed_stream_map" ], "expected": { - "implementation_sha256": "01c2bf7e06a589ec2c7e99c24e1413b3dfbf02206241a216d843a95082547a34", + "implementation_sha256": "b07caa958217fe3602ca52cf9e39f5b577fa9a94eb4a5722533b223b6714e0e3", "protocol": "legacy-v1", "streams": [ "build_model", @@ -1698,7 +1698,7 @@ }, "failures": [], "observed": { - "implementation_sha256": "01c2bf7e06a589ec2c7e99c24e1413b3dfbf02206241a216d843a95082547a34", + "implementation_sha256": "b07caa958217fe3602ca52cf9e39f5b577fa9a94eb4a5722533b223b6714e0e3", "protocol": "legacy-v1", "streams": [ "build_model", @@ -2599,7 +2599,7 @@ "country": "us", "schema_id": "country_spec", "schema_version": 1, - "spec_sha256": "b7b7037eeab3258d38968a8a5428c65df50bb3b0ee892c660488bf8df3cbe90f" + "spec_sha256": "11e310c7619cbac91f6703b9679649cdd15f6fb09274ad29904c65881aa93316" } }, "report_schema_version": 3, @@ -2609,7 +2609,7 @@ "country": "us", "schema_id": "country_spec", "schema_version": 1, - "spec_sha256": "b7b7037eeab3258d38968a8a5428c65df50bb3b0ee892c660488bf8df3cbe90f" + "spec_sha256": "11e310c7619cbac91f6703b9679649cdd15f6fb09274ad29904c65881aa93316" }, "status": "pass" } diff --git a/packages/microcosm-build/src/microcosm/build/spec_engine/inventory_coverage.py b/packages/microcosm-build/src/microcosm/build/spec_engine/inventory_coverage.py index 6758aca87..436b65e5b 100644 --- a/packages/microcosm-build/src/microcosm/build/spec_engine/inventory_coverage.py +++ b/packages/microcosm-build/src/microcosm/build/spec_engine/inventory_coverage.py @@ -359,8 +359,8 @@ "late_schedule": "e59c019d3d454eac99ac0ac209b6c5b6faaf9bdfcaeee18c36a25be19bf7da2f", "ownership": "5f64f0aac49e2313177564f71876bffc8c81b3ded4df701e70930e60e9c98356", "primary_tuples": "987b501c695e31f45521c4a178528f75ab3df22c09bc407b182213b2de99ee57", - "seed_map": "9ff34c4ff7c6d744a8385c2aee6e4eb2ea30a630e56b29ac555381987058da09", - "seed_protocol": "01c2bf7e06a589ec2c7e99c24e1413b3dfbf02206241a216d843a95082547a34", + "seed_map": "545d8007bf4f20638cf43f097bcac11559b6aa92f71b9282e965ed672c50cdb9", + "seed_protocol": "b07caa958217fe3602ca52cf9e39f5b577fa9a94eb4a5722533b223b6714e0e3", "source_manifest": "cd5ba8924d64da5425ee14cca82a774e3f4b2bb5aabe06df291cc3cc457287a9", "take_up": "fa186daea0f8dd641cc470e41d1a2953f887d45282ec990201298f47bedf8d4d", "tail": "ac92829c88a1a4fb6460d61190918d5d99c6c377fc8dd8f62f02b332d09bf59c", diff --git a/packages/microcosm-build/src/microcosm/build/us_runtime/acs_transfer.py b/packages/microcosm-build/src/microcosm/build/us_runtime/acs_transfer.py index b90151099..a3fb00a6b 100644 --- a/packages/microcosm-build/src/microcosm/build/us_runtime/acs_transfer.py +++ b/packages/microcosm-build/src/microcosm/build/us_runtime/acs_transfer.py @@ -1013,13 +1013,13 @@ def _pregnancy_boolean_values( raise TypeError( f"ACS transfer pregnancy {role} values must be physical booleans." ) from exc - valid = ~observed | np.isclose(values, 0.0) | np.isclose(values, 1.0) + valid = ~observed | (values == 0.0) | (values == 1.0) if not valid.all(): raise ValueError( f"ACS transfer pregnancy {role} values contain " f"{int((~valid).sum())} non-boolean row(s)." ) - return observed, observed & np.isclose(values, 1.0) + return observed, observed & (values == 1.0) def _pregnancy_source_groups( @@ -1233,20 +1233,16 @@ def _fan_pregnancy_predictions( predicted = _as_float_array(pd.Series(prediction)) positions = plan.representative_positions[plan.qrf_source_groups] values = predicted[positions] - valid = np.isfinite(values) & ( - np.isclose(values, 0.0) | np.isclose(values, 1.0) - ) + valid = np.isfinite(values) & ((values == 0.0) | (values == 1.0)) if not valid.all(): raise ValueError( "ACS transfer pregnancy QRF produced " f"{int((~valid).sum())} invalid representative result(s)." ) - source_values[plan.qrf_source_groups] = np.isclose(values, 1.0).astype( - np.float64 - ) + source_values[plan.qrf_source_groups] = (values == 1.0).astype(np.float64) if not np.isfinite(source_values).all(): # pragma: no cover - plan invariant raise RuntimeError("Pregnancy structural fanout left unresolved source people.") - return np.isclose(source_values[plan.source_codes], 1.0) + return source_values[plan.source_codes] == 1.0 def _finalize_pregnancy_structural_receipt( diff --git a/packages/microcosm-build/src/microcosm/build/us_runtime/h5_io.py b/packages/microcosm-build/src/microcosm/build/us_runtime/h5_io.py index af546794b..55de34273 100644 --- a/packages/microcosm-build/src/microcosm/build/us_runtime/h5_io.py +++ b/packages/microcosm-build/src/microcosm/build/us_runtime/h5_io.py @@ -328,6 +328,8 @@ def _validated_stacked_sampling_manifest_binding( isinstance(sampling_seed, bool) or not isinstance(sampling_seed, int) or sampling_seed < 0 + or isinstance(stack_seed, bool) + or not isinstance(stack_seed, int) or stack_seed != sampling_seed ): raise ValueError( @@ -358,22 +360,31 @@ def _validated_stacked_sampling_manifest_binding( survey_samples[channel], label=f"{label}.stack_manifest.survey_samples.{channel}", ) - if sample.get("fraction") != sampling_fraction: + sample_fraction = sample.get("fraction") + if type(sample_fraction) is not float or sample_fraction != sampling_fraction: raise ValueError( f"{label} {channel} survey-sample fraction differs from the " "production sampling rung." ) - if sample.get("seed") != sampling_seed: + sample_seed = sample.get("seed") + if ( + isinstance(sample_seed, bool) + or not isinstance(sample_seed, int) + or sample_seed != sampling_seed + ): raise ValueError( f"{label} {channel} survey-sample seed differs from the " "production sample seed." ) realized = sample.get("realized_household_count") + top_realized = realized_households[channel] if ( isinstance(realized, bool) or not isinstance(realized, int) or realized < 1 - or realized_households[channel] != realized + or isinstance(top_realized, bool) + or not isinstance(top_realized, int) + or top_realized != realized ): raise ValueError( f"{label} {channel} realized-household count is malformed or " diff --git a/packages/microcosm-build/src/microcosm/build/us_runtime/pregnancy.py b/packages/microcosm-build/src/microcosm/build/us_runtime/pregnancy.py index 06bfac7a0..99b868c10 100644 --- a/packages/microcosm-build/src/microcosm/build/us_runtime/pregnancy.py +++ b/packages/microcosm-build/src/microcosm/build/us_runtime/pregnancy.py @@ -176,7 +176,7 @@ def _pregnancy_eligibility(person: pd.DataFrame) -> tuple[np.ndarray, np.ndarray raise SourceRuntimeError( "Pregnancy structural policy requires boolean is_female." ) from exc - female = np.isclose(female_numeric, 1.0) + female = female_numeric == 1.0 invalid_female = ~np.isfinite(female_numeric) | ~np.isin( female_numeric, [0.0, 1.0], @@ -217,8 +217,8 @@ def _pregnancy_boolean_masks( numeric = values.to_numpy(dtype=np.float64, na_value=np.nan) except (TypeError, ValueError): return observed, np.zeros(len(values), dtype=bool), int(observed.sum()) - valid = ~observed | np.isclose(numeric, 0.0) | np.isclose(numeric, 1.0) - return observed, observed & np.isclose(numeric, 1.0), int((~valid).sum()) + valid = ~observed | (numeric == 0.0) | (numeric == 1.0) + return observed, observed & (numeric == 1.0), int((~valid).sum()) def _pregnancy_structural_counts(person: pd.DataFrame) -> dict[str, int]: diff --git a/packages/microcosm-build/tests/test_us_acs_transfer.py b/packages/microcosm-build/tests/test_us_acs_transfer.py index 9ef1acbec..e9e615bec 100644 --- a/packages/microcosm-build/tests/test_us_acs_transfer.py +++ b/packages/microcosm-build/tests/test_us_acs_transfer.py @@ -1614,6 +1614,28 @@ def test_pregnancy_invalid_donor_is_refused_before_qrf( assert not _MeanQRF.calls +def test_pregnancy_near_boolean_recipient_is_refused_before_qrf( + monkeypatch: pytest.MonkeyPatch, +) -> None: + recipient = _with_columns( + _recipient_frame(), + "person", + {"is_pregnant": [1.0 + 1e-12] + [False] * 5}, + ) + monkeypatch.setattr(acs_transfer_module, "QRF", _MeanQRF) + _MeanQRF.calls = [] + + with pytest.raises(ValueError, match=r"1 non-boolean row"): + transfer_acs_inputs( + recipient, + _pregnancy_donor(), + target_families={"person": {"pregnancy": ("is_pregnant",)}}, + donor_channel=None, + n_estimators=1, + ) + assert not _MeanQRF.calls + + def test_pregnancy_complete_invalid_recipient_is_still_refused( monkeypatch: pytest.MonkeyPatch, ) -> None: diff --git a/packages/microcosm-build/tests/test_us_multispine_pool_h5_io.py b/packages/microcosm-build/tests/test_us_multispine_pool_h5_io.py index 6eb1f5b68..4bdf237e7 100644 --- a/packages/microcosm-build/tests/test_us_multispine_pool_h5_io.py +++ b/packages/microcosm-build/tests/test_us_multispine_pool_h5_io.py @@ -8,6 +8,7 @@ import pandas as pd import pytest +import microcosm.build.us_runtime.acs_transfer as acs_transfer_module import microcosm.build.us_runtime.h5_io as h5_io import microcosm.build.us_runtime.post_transfer_calibration as post_transfer_calibration_runtime import microcosm.build.us_runtime.stacked_spine as stacked_spine_module @@ -884,6 +885,35 @@ def _canonical_late_calibration_owner_receipt( return owner +def _canonical_pregnancy_structural_receipt() -> dict[str, object]: + policy = acs_transfer_module.acs_transfer_execution_contract_identity( + targets=("is_pregnant",), + derive_schedule_d=False, + )["structural_target_policies"]["is_pregnant"] + return { + "policy_sha256": policy["sha256"], + "source_person_key": "person_source_id", + "source_persons_checked": 1, + "physical_rows_checked": 1, + "clone_rows_checked": 0, + "donor_rows_checked": 1, + "qrf_draw_source_persons": 0, + "qrf_draw_rows": 0, + "qrf_fanout_rows": 0, + "preexisting_value_fanout_rows": 0, + "ineligible_rows_assigned_false": 0, + "donor_preexisting_domain_violation_rows": 0, + "recipient_preexisting_domain_violation_rows": 0, + "preexisting_clone_disagreement_source_persons": 0, + "inconsistent_eligibility_source_persons": 0, + "maximum_clones_per_source_person": 1, + "final_incomplete_rows": 0, + "final_domain_violation_rows": 0, + "final_clone_disagreement_source_persons": 0, + "status": "verified", + } + + def _canonical_stacked_late_dag_receipt() -> dict[str, object]: """Build a signed fixture receipt over the live canonical contracts.""" @@ -945,6 +975,11 @@ def _canonical_stacked_late_dag_receipt() -> dict[str, object]: } for target in group.targets } + pregnancy_key = f"{group.entity}/{group.family}/is_pregnant" + if pregnancy_key in group_targets: + group_targets[pregnancy_key]["structural_policy"] = ( + _canonical_pregnancy_structural_receipt() + ) calibrated_keys = sorted(set(group_targets) & set(late_specs)) for key in calibrated_keys: group_targets[key]["post_transfer_calibration"] = ( @@ -1499,6 +1534,47 @@ def test_ready_stacked_pool_loader_rejects_malformed_sampling_receipt( load_simulation_ready_us_multispine_pool(manifest_path) +@pytest.mark.parametrize( + ("mutation", "error_match"), + ( + ("stack_seed_bool", "same non-negative integer"), + ("arm_fraction_bool", "survey-sample fraction differs"), + ("arm_seed_bool", "survey-sample seed differs"), + ("top_realized_bool", "realized-household count is malformed"), + ), +) +def test_ready_stacked_pool_loader_rejects_boolean_sampling_aliases( + tmp_path: Path, + mutation: str, + error_match: str, +) -> None: + pytest.importorskip("tables") + manifest_path = _write_ready_pool(tmp_path, stacked=True) + manifest = json.loads(manifest_path.read_text(encoding="utf-8")) + stack = manifest["stack_manifest"] + samples = stack["survey_samples"] + if mutation == "stack_seed_bool": + manifest["sampling"]["sample_seed"] = 1 + stack["sample_seed"] = True + for sample in samples.values(): + sample["seed"] = 1 + elif mutation == "arm_fraction_bool": + samples["asec"]["fraction"] = True + elif mutation == "arm_seed_bool": + manifest["sampling"]["sample_seed"] = 1 + stack["sample_seed"] = 1 + samples["asec"]["seed"] = True + samples["acs"]["seed"] = 1 + else: + samples["asec"]["realized_household_count"] = 1 + manifest["sampling"]["realized_households"]["asec"] = True + manifest["sampling"]["stack_manifest_sha256"] = _json_sha256(stack) + manifest_path.write_text(json.dumps(manifest), encoding="utf-8") + + with pytest.raises(ValueError, match=error_match): + load_simulation_ready_us_multispine_pool(manifest_path) + + def test_ready_stacked_pool_loader_binds_h5_cd_vintage_attrs( tmp_path: Path, ) -> None: diff --git a/packages/microcosm-build/tests/test_us_multispine_pool_tool.py b/packages/microcosm-build/tests/test_us_multispine_pool_tool.py index 51e738dfb..ddf7f0712 100644 --- a/packages/microcosm-build/tests/test_us_multispine_pool_tool.py +++ b/packages/microcosm-build/tests/test_us_multispine_pool_tool.py @@ -1222,6 +1222,35 @@ def _canonical_late_calibration_owner_receipt( return owner +def _canonical_pregnancy_structural_receipt() -> dict[str, object]: + policy = acs_transfer_module.acs_transfer_execution_contract_identity( + targets=("is_pregnant",), + derive_schedule_d=False, + )["structural_target_policies"]["is_pregnant"] + return { + "policy_sha256": policy["sha256"], + "source_person_key": "person_source_id", + "source_persons_checked": 1, + "physical_rows_checked": 1, + "clone_rows_checked": 0, + "donor_rows_checked": 1, + "qrf_draw_source_persons": 0, + "qrf_draw_rows": 0, + "qrf_fanout_rows": 0, + "preexisting_value_fanout_rows": 0, + "ineligible_rows_assigned_false": 0, + "donor_preexisting_domain_violation_rows": 0, + "recipient_preexisting_domain_violation_rows": 0, + "preexisting_clone_disagreement_source_persons": 0, + "inconsistent_eligibility_source_persons": 0, + "maximum_clones_per_source_person": 1, + "final_incomplete_rows": 0, + "final_domain_violation_rows": 0, + "final_clone_disagreement_source_persons": 0, + "status": "verified", + } + + def _canonical_late_transfer_receipt( pool_tool: ModuleType, *, @@ -1258,6 +1287,11 @@ def _canonical_late_transfer_receipt( } for target in group.targets } + pregnancy_key = f"{group.entity}/{group.family}/is_pregnant" + if pregnancy_key in group_targets: + group_targets[pregnancy_key]["structural_policy"] = ( + _canonical_pregnancy_structural_receipt() + ) calibrated_keys = sorted(set(group_targets) & set(late_specs)) for key in calibrated_keys: group_targets[key]["post_transfer_calibration"] = ( @@ -2465,7 +2499,7 @@ def capture_equality(expected: object, actual: object) -> None: "country": "us", "schema_id": "country_spec", "schema_version": 1, - "spec_sha256": "b7b7037eeab3258d38968a8a5428c65df50bb3b0ee892c660488bf8df3cbe90f", + "spec_sha256": "11e310c7619cbac91f6703b9679649cdd15f6fb09274ad29904c65881aa93316", }, } @@ -2989,6 +3023,30 @@ def test_late_transfer_validator_rejects_stripped_calibration_evidence( ) +def test_late_transfer_validator_rejects_forged_pregnancy_policy( + pool_tool: ModuleType, +) -> None: + receipt = _canonical_late_transfer_receipt(pool_tool) + stacked_spine_module.validate_stacked_post_puf_transfer_receipt( + receipt, + boundary="canonical pregnancy policy control", + ) + forged = copy.deepcopy(receipt) + pregnancy = next( + target_receipt + for group in forged["groups"].values() + for target_key, target_receipt in group["targets"].items() + if target_key.endswith("/is_pregnant") + ) + pregnancy["structural_policy"]["policy_sha256"] = "0" * 64 + + with pytest.raises(ValueError, match="pregnancy structural policy is invalid"): + stacked_spine_module.validate_stacked_post_puf_transfer_receipt( + forged, + boundary="forged pregnancy policy regression", + ) + + @pytest.mark.parametrize( ("target", "constraint_column", "replacement", "error_match"), ( diff --git a/packages/microcosm-build/tests/test_us_pregnancy.py b/packages/microcosm-build/tests/test_us_pregnancy.py index 6748966aa..13667e301 100644 --- a/packages/microcosm-build/tests/test_us_pregnancy.py +++ b/packages/microcosm-build/tests/test_us_pregnancy.py @@ -299,6 +299,18 @@ def test_preexisting_clone_disagreement_is_refused(self) -> None: time_period=TIME_PERIOD, ) + def test_preexisting_near_boolean_value_is_refused(self) -> None: + rows = [ + {"A_SEX": _FEMALE, "A_AGE": 25, "is_pregnant": 1.0 + 1e-12}, + {"A_SEX": _FEMALE, "A_AGE": 25, "is_pregnant": False}, + ] + with pytest.raises(SourceRuntimeError, match=r"non_boolean_rows=1"): + with_us_pregnancy_inputs( + _us_frame(rows), + seed=0, + time_period=TIME_PERIOD, + ) + class TestGate: def test_plausible_seeded_share_passes(self) -> None: From de5e5d03d7ff7500b304d7cbfe5b113b8f8d7d8f Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Thu, 27 Aug 2026 16:31:46 +0200 Subject: [PATCH 43/49] Repin source-attested spec identities --- PROGRESS.md | 24 +++++++++++++++++-- docs/evidence/spec-engine/us-f0-coverage.json | 16 ++++++------- .../build/spec_engine/inventory_coverage.py | 4 ++-- .../tests/test_spec_engine_country_bundles.py | 4 ++-- .../tests/test_spec_engine_loader.py | 2 +- .../tests/test_us_multispine_pool_tool.py | 2 +- 6 files changed, 36 insertions(+), 16 deletions(-) diff --git a/PROGRESS.md b/PROGRESS.md index 841f921df..265f7f5d0 100644 --- a/PROGRESS.md +++ b/PROGRESS.md @@ -134,11 +134,31 @@ push, retraining, threshold change, or launcher-contract edit is in scope. - Added `changelog.d/acs-release-predictor-join.fixed.md`, describing the pinned release join, reviewed mappings, fail-closed lineage, dual-manifest receipts, and unchanged model/gate behavior. +- Repository Ruff passes, and the CI inventory verifier reports 310 tracked + tests with `verification=ok`. Four complete pytest shards pass in separate + processes: frame 295 passed/36 skipped, fit 93 passed, calibrate 203 passed, + and data 318 passed/2 skipped. +- The first complete build-shard process reached 100% with 6,575 passed and 45 + skipped, plus five failures and six fixture errors. All eleven were the same + expected source-attestation drift: `ssi_disability_criteria.py` belongs to + both the direct and QRF seed-kernel inventories, so this task's runtime edit + moved the seed protocol, compiled US seed map, every country spec identity, + the minimal loader golden, and the generated coverage evidence. No ACS join, + release CLI, manifest, archived-model behavior, or gate test failed. +- Applied the repository's established five-file source-identity repin only: + seed protocol `59a098f9...31d8b`, US seed map `ce3850d8...e42ab`, US spec + `16b7d5e6...dca38`, UK spec `2f921e4c...33a62`, BE spec + `c87a0012...34ba`, and minimal-loader golden `b4946105...f2af`; regenerated + `docs/evidence/spec-engine/us-f0-coverage.json`. All 25 affected cases and + focused Ruff pass. The US bundle generator `--check` passes at the new spec + identity, and coverage `--check` passes at 42,122/42,122 fields and 41/41 + inventory checks. ## Next -- Run repository Ruff, the CI inventory verifier, and every full pytest shard - in its own process; then write and commit `out.md` and the completed journal. +- Re-run the complete build shard after the reviewed source-identity repin, + re-run final repository Ruff, then write and commit `out.md` and the + completed journal. # Weeksgate: stacked release gates and integer-week provenance diff --git a/docs/evidence/spec-engine/us-f0-coverage.json b/docs/evidence/spec-engine/us-f0-coverage.json index c3f6bd0a9..b0304b9f6 100644 --- a/docs/evidence/spec-engine/us-f0-coverage.json +++ b/docs/evidence/spec-engine/us-f0-coverage.json @@ -1656,13 +1656,13 @@ "compiler_ir.node_slices" ], "expected": { - "map_sha256": "ee1ad8d5196ca48d70657675237294c2dd925a520900d4d476d256a1b514b283", - "protocol_sha256": "c7df307cba6b20cc26f4aeeb1ee5eae246362318cef885cd747c64349e08cc41" + "map_sha256": "ce3850d87d9f0a5d37a0b7e1b06ac0ad014b1e487b2340e81ec8b11e750e42ab", + "protocol_sha256": "59a098f9fba730ea046903bd7d4b90be2c0caf7f85494c723661f8067b031d8b" }, "failures": [], "observed": { - "map_sha256": "ee1ad8d5196ca48d70657675237294c2dd925a520900d4d476d256a1b514b283", - "protocol_sha256": "c7df307cba6b20cc26f4aeeb1ee5eae246362318cef885cd747c64349e08cc41" + "map_sha256": "ce3850d87d9f0a5d37a0b7e1b06ac0ad014b1e487b2340e81ec8b11e750e42ab", + "protocol_sha256": "59a098f9fba730ea046903bd7d4b90be2c0caf7f85494c723661f8067b031d8b" }, "status": "covered" }, @@ -1677,7 +1677,7 @@ "compiler_ir.seed_stream_map" ], "expected": { - "implementation_sha256": "c7df307cba6b20cc26f4aeeb1ee5eae246362318cef885cd747c64349e08cc41", + "implementation_sha256": "59a098f9fba730ea046903bd7d4b90be2c0caf7f85494c723661f8067b031d8b", "protocol": "legacy-v1", "streams": [ "build_model", @@ -1698,7 +1698,7 @@ }, "failures": [], "observed": { - "implementation_sha256": "c7df307cba6b20cc26f4aeeb1ee5eae246362318cef885cd747c64349e08cc41", + "implementation_sha256": "59a098f9fba730ea046903bd7d4b90be2c0caf7f85494c723661f8067b031d8b", "protocol": "legacy-v1", "streams": [ "build_model", @@ -2599,7 +2599,7 @@ "country": "us", "schema_id": "country_spec", "schema_version": 1, - "spec_sha256": "5f44d96d45e9aabcea2d565ef063d68bfc0652df1b38b08aa31ce6896d15f371" + "spec_sha256": "16b7d5e622e8a68e008165bb44a5836695d94a2b0dd8d4c51b3c9e8ca89dca38" } }, "report_schema_version": 3, @@ -2609,7 +2609,7 @@ "country": "us", "schema_id": "country_spec", "schema_version": 1, - "spec_sha256": "5f44d96d45e9aabcea2d565ef063d68bfc0652df1b38b08aa31ce6896d15f371" + "spec_sha256": "16b7d5e622e8a68e008165bb44a5836695d94a2b0dd8d4c51b3c9e8ca89dca38" }, "status": "pass" } diff --git a/packages/microcosm-build/src/microcosm/build/spec_engine/inventory_coverage.py b/packages/microcosm-build/src/microcosm/build/spec_engine/inventory_coverage.py index f0dd2529c..ac3e6accc 100644 --- a/packages/microcosm-build/src/microcosm/build/spec_engine/inventory_coverage.py +++ b/packages/microcosm-build/src/microcosm/build/spec_engine/inventory_coverage.py @@ -359,8 +359,8 @@ "late_schedule": "dcf3c6d2eade3449836c49a1dc4d3b8cd395aab9142db700c3c60598fa9c1c79", "ownership": "5f64f0aac49e2313177564f71876bffc8c81b3ded4df701e70930e60e9c98356", "primary_tuples": "987b501c695e31f45521c4a178528f75ab3df22c09bc407b182213b2de99ee57", - "seed_map": "ee1ad8d5196ca48d70657675237294c2dd925a520900d4d476d256a1b514b283", - "seed_protocol": "c7df307cba6b20cc26f4aeeb1ee5eae246362318cef885cd747c64349e08cc41", + "seed_map": "ce3850d87d9f0a5d37a0b7e1b06ac0ad014b1e487b2340e81ec8b11e750e42ab", + "seed_protocol": "59a098f9fba730ea046903bd7d4b90be2c0caf7f85494c723661f8067b031d8b", "source_manifest": "cd5ba8924d64da5425ee14cca82a774e3f4b2bb5aabe06df291cc3cc457287a9", "take_up": "fa186daea0f8dd641cc470e41d1a2953f887d45282ec990201298f47bedf8d4d", "tail": "ac92829c88a1a4fb6460d61190918d5d99c6c377fc8dd8f62f02b332d09bf59c", diff --git a/packages/microcosm-build/tests/test_spec_engine_country_bundles.py b/packages/microcosm-build/tests/test_spec_engine_country_bundles.py index bcac668ec..7b17976a2 100644 --- a/packages/microcosm-build/tests/test_spec_engine_country_bundles.py +++ b/packages/microcosm-build/tests/test_spec_engine_country_bundles.py @@ -32,7 +32,7 @@ [ ( "be", - "f7fd69bf4964faa2eeea1094323ce3a5ace180d76acfff0c30ed0d592ebc129e", + "c87a0012894fc6389873d3e651923ffc59206db7e8c45273af332df3f49434ba", { "household.household_id", "person.person_id", @@ -42,7 +42,7 @@ ), ( "uk", - "e040368ecf8fb89da14ff0b8f2b1f1c8f6845a18423dfe9792c0c2abc094bbbf", + "2f921e4c92e51301f9bbe19d1c168340c6172e9b98f368b93e80a78085833a62", { "benunit.benunit_id", "household.household_id", diff --git a/packages/microcosm-build/tests/test_spec_engine_loader.py b/packages/microcosm-build/tests/test_spec_engine_loader.py index 8a3e0e232..887a03fdb 100644 --- a/packages/microcosm-build/tests/test_spec_engine_loader.py +++ b/packages/microcosm-build/tests/test_spec_engine_loader.py @@ -235,7 +235,7 @@ def test_semantic_hash_has_golden_vector_and_surface_separation(tmp_path) -> Non # Pin the domain separator, normalization rules, schema-set receipt, and # exact normative projection as one reviewable golden vector. assert first.spec_sha256 == ( - "2473f29cdd4aef89cb7998e7effeeb14272f4aaaa91f277f48163d8f5137d1de" + "b4946105f8c371a84950780a01d92e787055823cfd44d35f34d16b5319f9f2af" ) second_root = _rich_minimal(tmp_path / "xy", note="second", store="local:b") diff --git a/packages/microcosm-build/tests/test_us_multispine_pool_tool.py b/packages/microcosm-build/tests/test_us_multispine_pool_tool.py index 473daf364..7e41919e0 100644 --- a/packages/microcosm-build/tests/test_us_multispine_pool_tool.py +++ b/packages/microcosm-build/tests/test_us_multispine_pool_tool.py @@ -2465,7 +2465,7 @@ def capture_equality(expected: object, actual: object) -> None: "country": "us", "schema_id": "country_spec", "schema_version": 1, - "spec_sha256": "5f44d96d45e9aabcea2d565ef063d68bfc0652df1b38b08aa31ce6896d15f371", + "spec_sha256": "16b7d5e622e8a68e008165bb44a5836695d94a2b0dd8d4c51b3c9e8ca89dca38", }, } From d2b75e1e23e35f93face58326d1facb4c683eae3 Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Thu, 27 Aug 2026 16:33:32 +0200 Subject: [PATCH 44/49] Repin shared spec attestations --- PROGRESS.md | 13 ++++++++++--- .../tests/test_spec_engine_country_bundles.py | 4 ++-- .../tests/test_spec_engine_loader.py | 2 +- 3 files changed, 13 insertions(+), 6 deletions(-) diff --git a/PROGRESS.md b/PROGRESS.md index 5de43f21a..8d91f8236 100644 --- a/PROGRESS.md +++ b/PROGRESS.md @@ -97,12 +97,19 @@ push pool or release artifacts. spec SHA is `11e310c7619cbac91f6703b9679649cdd15f6fb09274ad29904c65881aa93316`; no authored YAML changed. Regenerated coverage remains complete at 42,154/42,154 fields and 41/41 inventory checks. +- Committed the fixture, exact-boolean, nested-manifest-type, tamper-test, and + final US identity repair as `e59ab046`. Final-state calibrate, data, fit, and + frame shard runs then passed. +- The restarted build shard reached 11% and exposed only the expected shared + source-attestation cascade in BE, UK, and the rich-minimal golden spec hashes. + Re-pinned those three observed semantic identities; the complete country- + bundle/loader/spec-only/country-spec focused matrix passes. ## Next -- Commit the fixture/type-hardening integration repair and journal state. -- Run repository Ruff, generated/coverage/inventory checks, and all five pytest - shards in independent processes, restarting the build shard from zero. +- Commit the shared country/minimal source-attestation repin. +- Restart the full build shard from zero; the other four final-state shards, + repository Ruff, generated/coverage/inventory checks already pass. - Write and commit the complete handoff to `out.md`. # Weeksgate: stacked release gates and integer-week provenance diff --git a/packages/microcosm-build/tests/test_spec_engine_country_bundles.py b/packages/microcosm-build/tests/test_spec_engine_country_bundles.py index bcac668ec..25d341927 100644 --- a/packages/microcosm-build/tests/test_spec_engine_country_bundles.py +++ b/packages/microcosm-build/tests/test_spec_engine_country_bundles.py @@ -32,7 +32,7 @@ [ ( "be", - "f7fd69bf4964faa2eeea1094323ce3a5ace180d76acfff0c30ed0d592ebc129e", + "8cf81df2cc66a11883281122b0397550af458c9bf3f6bf6a5895e0bbda8110da", { "household.household_id", "person.person_id", @@ -42,7 +42,7 @@ ), ( "uk", - "e040368ecf8fb89da14ff0b8f2b1f1c8f6845a18423dfe9792c0c2abc094bbbf", + "8e4f30d2541c52b946edd19ef277cb0cbac06c7ed0d64fb5375626a2d3939132", { "benunit.benunit_id", "household.household_id", diff --git a/packages/microcosm-build/tests/test_spec_engine_loader.py b/packages/microcosm-build/tests/test_spec_engine_loader.py index 8a3e0e232..f794e60f2 100644 --- a/packages/microcosm-build/tests/test_spec_engine_loader.py +++ b/packages/microcosm-build/tests/test_spec_engine_loader.py @@ -235,7 +235,7 @@ def test_semantic_hash_has_golden_vector_and_surface_separation(tmp_path) -> Non # Pin the domain separator, normalization rules, schema-set receipt, and # exact normative projection as one reviewable golden vector. assert first.spec_sha256 == ( - "2473f29cdd4aef89cb7998e7effeeb14272f4aaaa91f277f48163d8f5137d1de" + "50081b197daa9b9aad3397482a84f6ef00e05b4e4710b95cc9bcb1fed4f739be" ) second_root = _rich_minimal(tmp_path / "xy", note="second", store="local:b") From 7caf69acc185e39a3b935b1e74c39c00107d83d7 Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Thu, 27 Aug 2026 17:00:47 +0200 Subject: [PATCH 45/49] Repin ACS transfer provenance fixture --- PROGRESS.md | 7 ++++++- packages/microcosm-build/tests/test_us_acs_multispine.py | 7 ++++--- 2 files changed, 10 insertions(+), 4 deletions(-) diff --git a/PROGRESS.md b/PROGRESS.md index 8d91f8236..eea8e5b49 100644 --- a/PROGRESS.md +++ b/PROGRESS.md @@ -104,10 +104,15 @@ push pool or release artifacts. source-attestation cascade in BE, UK, and the rich-minimal golden spec hashes. Re-pinned those three observed semantic identities; the complete country- bundle/loader/spec-only/country-spec focused matrix passes. +- Committed the shared semantic-hash repin as `d2b75e1e`. The next clean build + run passed that boundary and reached 39%, where the legacy optional-ACS + provenance golden needed the new additive `structural_receipt: null` field + on an unrelated transfer record. Updated only that JSON-ready shape; the + complete optional-ACS multispine test file and focused Ruff pass. ## Next -- Commit the shared country/minimal source-attestation repin. +- Commit the additive optional-ACS provenance fixture repin. - Restart the full build shard from zero; the other four final-state shards, repository Ruff, generated/coverage/inventory checks already pass. - Write and commit the complete handoff to `out.md`. diff --git a/packages/microcosm-build/tests/test_us_acs_multispine.py b/packages/microcosm-build/tests/test_us_acs_multispine.py index 98b21c6ab..537a27f80 100644 --- a/packages/microcosm-build/tests/test_us_acs_multispine.py +++ b/packages/microcosm-build/tests/test_us_acs_multispine.py @@ -213,9 +213,10 @@ def fake_pool(actual_base, actual_acs, *, acs_share): "patterns": [], "imputed_recipient_rows": 0, "unmodeled_recipient_rows": 0, - "derivation": None, - "reconciliation": None, - } + "derivation": None, + "reconciliation": None, + "structural_receipt": None, + } ], "deferred_inputs": ["congressional_district_geoid"], "adult_care_recipient_gate": None, From d1fa5bbccf2eed9a66242cd40463b4f4ea9a490c Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Thu, 27 Aug 2026 17:30:55 +0200 Subject: [PATCH 46/49] Finalize ACS predictor join report --- PROGRESS.md | 32 +- out.md | 856 +++++++++++++++++++--------------------------------- 2 files changed, 329 insertions(+), 559 deletions(-) diff --git a/PROGRESS.md b/PROGRESS.md index 265f7f5d0..ac4177532 100644 --- a/PROGRESS.md +++ b/PROGRESS.md @@ -2,13 +2,15 @@ ## State -In progress on 2026-08-27. Implementing the owner-approved release-time join -from stacked-pool ACS person source IDs to the sha-pinned 2024 one-year ACS -person/household CSV zips. Scope is limited to populating the six archived -donor models' CPS-named predictors through reviewed native-ACS crosswalks, -strict join/receipt contracts, unchanged model selection logic, and unchanged -gate thresholds. No network access, pool build, release build, publication, -push, retraining, threshold change, or launcher-contract edit is in scope. +Complete on 2026-08-27. The owner-approved release-time join from stacked-pool +ACS source lineage to the SHA-pinned 2024 one-year ACS person/household zips is +implemented, receipted, real-pool exercised, and fully verified. It populates +the six archived donor models' CPS-named predictors through reviewed +native-ACS crosswalks with strict hash, lineage, collision, totality, universe, +and clone-fan-out contracts. Model selection logic and every gate threshold +remain unchanged. The completed evidence and handoff are in `out.md`. No +network access, pool build, release build, publication, push, retraining, +threshold change, or launcher-contract edit occurred. ## Done @@ -116,8 +118,9 @@ push, retraining, threshold change, or launcher-contract edit is in scope. The current official loader refuses that older candidate before H5 loading because its archived primary-QRF worker binding predates this branch's execution identity; this is an existing candidate/code-version mismatch. - Loading those already-authenticated bytes with their frozen assembly receipt - allowed the join boundary itself to be tested without writing an artifact. + Loading those independently manifest-hash-verified bytes with their frozen + assembly receipt allowed the join boundary itself to be tested without + writing an artifact. - The real join passed every source, raw-key, universe, SSI-attestation, totality, collision, and clone-fan-out check: 856,626 unique ACS source people matched 856,626 raw people in 382,903 households and populated @@ -153,12 +156,17 @@ push, retraining, threshold change, or launcher-contract edit is in scope. focused Ruff pass. The US bundle generator `--check` passes at the new spec identity, and coverage `--check` passes at 42,122/42,122 fields and 41/41 inventory checks. +- Re-ran the complete build shard after the reviewed repin: 6,586 passed and + 45 skipped, with exit code 0. Re-ran final repository Ruff, the 310-file CI + inventory verifier, both retained spec `--check` commands, and + `git diff --check`; all pass. Wrote the required final report to `out.md`. ## Next -- Re-run the complete build shard after the reviewed source-identity repin, - re-run final repository Ruff, then write and commit `out.md` and the - completed journal. +- No work remains in this lane. The dispatcher owns rebasing and the launcher + contract update. A future authorized build must produce a pool whose current + source-attested worker identity passes the official release loader; the + supplied older candidate is useful join evidence but cannot be promoted. # Weeksgate: stacked release gates and integer-week provenance diff --git a/out.md b/out.md index 6b9d8437a..eb90666a6 100644 --- a/out.md +++ b/out.md @@ -1,560 +1,322 @@ -# Weeksgate: stacked release gates and fractional-weeks final report +# ACSPRED: release-side ACS predictor join final report Date: 2026-08-27 -Branch: `stacked-release-gate-alignment` +Branch: `acs-predictor-release-join` -Lane base: `4f453746` (`origin/main` at kickoff; includes #786) +Lane base: `606cbd69` (`stacked-release-gate-alignment`) ## Outcome -The 369 fractional `weeks_unemployed` values are genuine pool-content -defects, but the proposed post-transfer-calibration mechanism is refuted. They -were emitted earlier by the ACS-transfer QRF because PolicyEngine-US physically -types this integer-supported variable as `float`. The calibration maps ACS -clone 0 onto actual observed ASEC support and removes all 342 fractions in that -scope; it never touches the 360 ACS clone-1 and 9 ACS clone-2 fractions that -survive into the pool. - -The source fix registers `weeks_unemployed` as a discrete numeric ACS-transfer -target. Predictions now snap deterministically to actual donor support, with -lower-support tie breaking, and are returned on integer support. The transfer -execution contract and generated imputation authority receipt the policy. -The strict post-transfer calibration receipt and validator remain unchanged -because that kernel did not cause the defect; their complete contract suite -still passes. - -The weeks release gate now distinguishes physical source channel from -clone-operator role. It derives an assembled roster (`asec+acs`) or retains a -legacy roster (`asec+puf_tax_detail`), validates raw `LKWEEKS` only where an -ASEC source exists, reconciles direct carries only on native ASEC rows, and -checks the UC rule only on rows owned by that constraint. Every plausibility -band and numeric threshold is unchanged. - -The full release-side sweep fixed all unambiguous physical-source, -clone-layout, and stable-identity archaeology. Six archived-model predictor -contracts remain owner decisions and are reported precisely below. In -particular, ORG wages/FLSA is guaranteed to fail on the stacked pool, while -SIPP tips' global tipped-occupation component passes despite a dead ACS -channel. Those are not safe gate-only edits. - -No network access, pool build, release build, artifact publication, push, -plausibility-band change, or by-origin battery change was performed. Issue -#782's weeks-incidence-band adjudication was not touched. - -## Task 1: provenance of all 369 noninteger weeks - -### Evidence read - -The analysis read the supplied fixed-format pool directly: - -`/Users/maxghenis/PolicyEngine/_buildo-runtime/out/candidate-25/pool/pool.h5` - -It contains 1,970,973 person rows: 234,133 physical ASEC rows and 1,736,840 -physical ACS rows. It also read the exact pre-calibration late-transfer target -bank: - -`/Users/maxghenis/PolicyEngine/_buildo-runtime/out/candidate-25/pool/checkpoints/stacked/3847d30e1488ae18891780a7a5de6a1b75d1329f2491796e35639fc9e802c26e/acs-transfer/0914d1b858dff5171f7743f0e535349be4459fb99b3386bbc2e3be72d40dd5fc/late_producer_dag/person/source_operator_weeks_unemployed/targets/000__weeks_unemployed.h5` - -The raw-draw value SHA-256 is -`6ea20782bcb53fa730bb7e7045c7d79ab096dfe69bc5ceb540158d93f6672db4`. -The final fractional-row-index SHA-256 is -`3431befe89f007c353296b0c59792bc31e45a1031e62a6f9c99e003df286305f`; -the sorted 369-value SHA-256 is -`314ba4459bb57169eaf492dd28c8ea04af460104b962bb29285104e70eee976c`. - -### Counts by physical channel, clone, and UC - -| Physical channel | Clone index | All rows | Fractional, UC=0 | Fractional, UC>0 | -|---|---:|---:|---:|---:| -| ASEC | 0 | 108,073 | 0 | 0 | -| ASEC | 1 | 108,073 | 0 | 0 | -| ASEC | 2 | 17,987 | 0 | 0 | -| ACS | 0 | 856,626 | 0 | 0 | -| ACS | 1 | 856,626 | 355 | 5 | -| ACS | 2 | 23,588 | 9 | 0 | -| **Total** | | **1,970,973** | **364** | **5** | - -All 369 values are positive, have null ASEC `LKWEEKS`, and are distinct at -their exact IEEE-754 representation. Every fractional row maps one-to-one by -`person_source_id` to a unique ACS clone-0 sibling, and -`unemployment_compensation` is identical across that sibling group. Of the -364 UC-zero rows, the clone-0 sibling has zero weeks. The five UC-positive -rows have clone-0 integer weeks in `{2, 4, 40, 48, 50}`. - -The five fractional value / annual-UC pairs are: - -| Fractional weeks | Unemployment compensation | -|---:|---:| -| 3.3297787140375217 | 10,800 | -| 3.948788299342264 | 20,000 | -| 17.23699569220113 | 7,200 | -| 18.64447048256536 | 10,800 | -| 21.97077954067273 | 4,400 | - -### Value distribution - -| Statistic | Value | -|---|---:| -| Distinct values | 369 | -| Minimum | 1.0003521955067698 | -| p10 | 2.306610095577396 | -| p25 | 6.226837695984924 | -| Median | 12.319331262236448 | -| Mean | 14.643462963265243 | -| p75 | 23.618846787450813 | -| p90 | 27.706623369972394 | -| Maximum | 37.796501228614694 | - -| Weeks interval | All | ACS clone 1, UC=0 | ACS clone 1, UC>0 | ACS clone 2, UC=0 | -|---|---:|---:|---:|---:| -| [1, 5) | 74 | 70 | 2 | 2 | -| [5, 10) | 85 | 84 | 0 | 1 | -| [10, 15) | 45 | 44 | 0 | 1 | -| [15, 20) | 40 | 36 | 2 | 2 | -| [20, 25) | 60 | 58 | 1 | 1 | -| [25, 30) | 39 | 39 | 0 | 0 | -| [30, 35) | 16 | 15 | 0 | 1 | -| [35, 40) | 10 | 9 | 0 | 1 | - -The nine clone-2 values are -`1.3112775468533615`, `3.155651627690991`, -`8.163243626548894`, `13.03279351472897`, -`15.40528737169489`, `15.477036717133474`, -`24.60208367015022`, `31.658750512333974`, and -`37.10384792802646`. - -### Mechanism verdict: calibration hypothesis refuted - -The late-transfer target bank contains 711 fractional predictions: - -| ACS clone | Fractional before calibration | Fractional in final pool | Rows changed by calibration | -|---:|---:|---:|---:| -| 0 | 342 | 0 | 13,417 | -| 1 | 360 | 360 | 0 | -| 2 | 9 | 9 | 0 | - -Every surviving fractional bit is identical between the target bank and final -pool. The stacked calibration explicitly defines ASEC clone 0 as reference and -ACS clone 0 as recipient in -`packages/microcosm-build/src/microcosm/build/us_runtime/stacked_spine.py:8975-8985`. -Its amount mapping selects values from the sorted observed donor array rather -than interpolating in -`packages/microcosm-build/src/microcosm/build/us_runtime/post_transfer_calibration.py:577-680`. - -The pool receipt confirms the calibration ran: - -- 8,419 allowed-carrier/addition-candidate rows within 856,626 mutable ACS - clone-0 rows; -- 13,417 changed clone-0 rows, including 4,998 cleared and 8,290 added; -- `capacity_limited=true`; -- 8,419 mapped positive amounts; -- reference quantiles `[2, 6, 12, 26, 36]`; -- recipient quantiles `[2, 8, 22, 24, 32]` before and - `[2, 6, 12, 26, 36]` after; -- amount QED `0.5882352941176471 -> 0.0`; -- zero donor-support violations. - -Weeks' declared post-transfer calibration spec is at -`packages/microcosm-build/src/microcosm/build/us_runtime/post_transfer_calibration.py:237-263`. -The actual cause precedes calibration. Before this branch, -`_target_encoding` treated numeric `weeks_unemployed` as continuous because -the engine metadata says `float`; a quantile-regression-forest prediction may -therefore interpolate between integer observations. The relevant encoding and -decoding seam is -`packages/microcosm-build/src/microcosm/build/us_runtime/acs_transfer.py:3037-3120` -and `:3240-3270`. Calibration then repaired only clone 0 by design. - -### Why the gate said “5,218 PUF rows” - -There is no physical PUF channel in this pool. The compatibility helper -`support_role_series` deliberately labels clone 0 as the ASEC-compatible -operator role and every clone above zero as the PUF-tax-detail operator role, -independent of physical source. See -`packages/microcosm-build/src/microcosm/build/us_runtime/support_provenance.py:390-489`. -The old summary confused that operator role with physical channel. - -The 5,218 rows are all ACS-origin, UC-zero, positive-week non-native clones: - -| Actual rows | Integer weeks | Fractional weeks | Total | -|---|---:|---:|---:| -| ACS clone 1 | 4,733 | 355 | 5,088 | -| ACS clone 2 | 121 | 9 | 130 | -| **Total** | **4,854** | **364** | **5,218** | - -The UC-zero constraint is not defined for those ACS non-native QRF clones. -The updated gate preserves the compatibility role for operator plausibility -but uses physical source and clone provenance to own source/UC checks. - -## Task 2: source fix and receipt contract - -`weeks_unemployed` is now in `_DISCRETE_NUMERIC_TARGETS` at -`packages/microcosm-build/src/microcosm/build/us_runtime/acs_transfer.py:133-141`. -The existing discrete codec: - -1. rejects nonintegral donor support; -2. snaps finite predictions to the nearest value in actual observed support; -3. resolves exact-distance ties to the lower support value; -4. emits integer/nullable-integer output. - -The snap is at -`packages/microcosm-build/src/microcosm/build/us_runtime/acs_transfer.py:3327-3339`. -The sorted discrete-target roster is part of the transfer execution identity at -`:271`; the generated authority records it at -`packages/microcosm-build/src/microcosm/build/us/spec/imputation.yaml:408-411`. -The final regenerated bundle spec SHA-256 is -`5f44d96d45e9aabcea2d565ef063d68bfc0652df1b38b08aa31ce6896d15f371`. -This invalidates affected target-bank identities instead of silently changing -content under an old receipt. - -The release-gate and WIC edits also change exact source bytes attested by the -shared legacy seed protocol. Its reviewed WIC grammar now explicitly records -assembled `person_source_id` before the unchanged legacy fallbacks at -`packages/microcosm-build/src/microcosm/build/spec_engine/seeds.py:720-737`. -The fail-closed spec proof was coherently re-pinned to 42,122/42,122 claimed -fields (32,352 authored and 9,770 resolved) and 41/41 inventory checks. The -committed coverage report, producer-semantics digest, seed protocol/map, -checkpoint identity, pointer inventory, and affected country-envelope goldens -all move together. A detached `origin/main` comparison confirmed that the -BE/UK/minimal-envelope changes come from this branch's attested source and seed -grammar, not environment drift. - -The regression at -`packages/microcosm-build/tests/test_us_acs_transfer.py:1391-1430` sends a -fraction-producing mean QRF through the real target codec, verifies that output -lies on observed integer donor support, and verifies that the execution -contract declares `weeks_unemployed`. - -Because the hypothesis was refuted, no change was made to -`post_transfer_calibration.py` or its strict validator. Its observed-support -amount mapping, before/after margin fidelity, hashes, anchors, capacity status, -and donor-support validation remain mandatory; the complete 47-test receipt -contract passes. - -**Artifact consequence:** pool content changes. The host session must rerun -`late_transfer -> simulated -> terminal-gates` from checkpoints for the -candidate pool. This lane did not run any pool or release build. - -## Task 3: stacked-aware weeks release gate - -The gate now uses the narrow, read-only physical-source accessor -`support_gate_source_channel_series` at -`packages/microcosm-build/src/microcosm/build/us_runtime/support_provenance.py:496-524`. -That accessor is restricted to reviewed gates/reporters by the static -source-blindness contract. - -| Frame layout | Plausibility roster | Raw LKWEEKS validity | Exact LKWEEKS carry | UC-zero consistency | -|---|---|---|---|---| -| No provenance | ASEC | all rows | all rows | none, matching legacy behavior | -| Legacy ASEC+PUF roles | ASEC + PUF | ASEC role | ASEC role | PUF role | -| Assembled ASEC+ACS | actual physical channels | every physical ASEC clone | physical ASEC clone 0 | physical ASEC non-native clones + non-ASEC clone 0 | - -The assembled scope logic is at -`packages/microcosm-build/src/microcosm/build/us_runtime/weeks_unemployed.py:1320-1380`; -summary and failure construction are at `:1204-1317` and `:1383-1435`. -The ASEC plausibility bands and the legacy recipient/PUF bands remain -byte-identical, and every non-ASEC physical channel uses the unchanged -recipient band. - -A read-only replay on the supplied pool reports: - -- roster `asec, acs`; -- 234,133 raw ASEC source rows, zero invalid; -- 108,073 direct native reconciliation rows, zero mismatches; -- 982,686 UC-constrained rows, zero mismatches; -- ASEC positive share `0.01956717285126748`, weighted mean - `0.32416674108944377`, weighted weeks `52,743,981.18128455`; -- ACS positive share `0.00800370929739283`, weighted mean - `0.1376988711272782`, weighted weeks `21,987,101.082573153`; -- the unchanged channel bands pass; -- the gate fails only on the genuine 369 noninteger values. - -Stacked ASEC+ACS and legacy ASEC+PUF fixtures cover roster derivation, raw -source scoping, native reconciliation, and UC ownership. The complete focused -weeks file passed: 26 passed, 1 skipped. - -## Task 4: complete release-side archaeology audit - -The audit started from every release call in -`tools/build_us_fiscal_refresh_release.py:8690-11726`, traced each gate and -its immediately preceding producer/wrapper, and classified physical-source, -operator-role, clone-layout, entity-layout, and raw-column assumptions. The -table groups gates only where they share the same architectural conclusion; -every release gate in the roster is named. - -| Gate/stage | Stacked assumption found | Disposition | +The release builder now populates every CPS-named predictor consumed by the six +archived donor models on every physical ACS support row. It does so through an +exact release-time join to the two canonical, SHA-pinned 2024 one-year ACS PUMS +archives and a versioned, digest-pinned crosswalk. It does not fill unknowns +with silent defaults, weaken a gate, retrain a model, or change model selection +logic. + +The join passed read-only on the complete supplied candidate: 856,626 unique +ACS source people matched exactly 856,626 raw person records in 382,903 raw +households and fanned out to 1,736,840 ACS support rows. All model predictors +have zero remaining nulls. The SSI reporter path separately preserves 261,605 +genuine below-age-15 ACS support-row nulls and treats them as false only at the +consumer's `> 0` predicate. + +The implementation adds four all-or-none release CLI inputs, writes the full +receipt to both manifests, keeps legacy non-ACS releases as an explicit +disabled identity path, and leaves the launcher contract to the dispatcher as +required. + +No network access, pool build, release build, wheel build, publication, push, +or launcher edit was performed. + +## Join-key verification + +### Why `person_source_id` is not the raw key + +The ACS loader rejects duplicate `(SERIALNO, SPORDER)` pairs, stably sorts by +that pair, and only then assigns a zero-based raw `source_row_id` +(`acs_pums.py:204-238`). Source construction stores `SPORDER` separately as +`source_person_id` and links people to the household identified by `SERIALNO` +(`acs_pums.py:264-279`). Assembly then collision-offsets structural IDs and +creates assembly-unique support source IDs (`spine_assembly.py:366-430`; +`support_provenance.py:346-371`). Therefore neither the final structural ID nor +`person_source_id` is a reversible Census record key. + +The release join instead recovers the retained semantic identity: + +| Role | Retained evidence | Enforced relation | +|---|---|---| +| Household key | linked household `SERIALNO` | nonblank, one source household identity per serial | +| Person key | integral `SPORDER` and `source_person_id` | exactly equal and at least 1 | +| Raw ordinal | `source_row_id`, `person_spine_source_id` | exactly equal, but never used as the Census join key | +| Source identity | `person_source_id` | one raw identity across all clones; fan-out only | +| Clone identity | `person_support_clone_index` | unique `(person_source_id, clone_index)` and exactly one clone 0 | +| Vintage | `source_year` | exactly 2024 on physical ACS rows | + +The implementation derives these relations at +`acs_release_predictors.py:1078-1289`, including person-to-household channel +agreement, clone agreement, raw ordinal agreement, and unique canonical +`(SERIALNO, SPORDER)`. It then performs a pandas `one_to_one` left merge on +that semantic key, rejects any unmatched pool source person, and refuses a row +count change (`acs_release_predictors.py:893-923`). Only after that proof does +it map values back through `person_source_id` to every clone +(`acs_release_predictors.py:939-968`). + +## Artifact and refusal contract + +The reviewed archive identities are declared at +`acs_release_predictors.py:70-88`: + +| Archive | SHA-256 | Size observed | +|---|---|---:| +| 2024 ACS person `csv_pus.zip` | `afdc6d90c6e2f0bab365ed32d95ba4c4d8ac651162f46ac7861295b2dc469894` | 602,847,146 bytes | +| 2024 ACS household `csv_hus.zip` | `8281008e53de98f0ef81e7a2ee5a8725991dda1ecfd2713ead73246425e515d0` | 251,500,587 bytes | + +Before either zip is opened, the boundary requires lowercase 64-hex syntax, +requires the caller's expected hash to equal the reviewed canonical pin, and +hashes the actual file (`acs_release_predictors.py:991-1021`). It then requires +exact `psam_pusa/b.csv` or `psam_husa/b.csv` membership, verifies required +headers, and streams only selected serials (`acs_release_predictors.py:1024-1075`). + +The boundary also refuses: + +- partial CLI/archive inputs or archive inputs on a frame with no ACS rows; +- a stale crosswalk digest; +- malformed or incomplete assembly provenance; +- duplicate raw person or household keys; +- source-identity, semantic-key, or clone-index collisions; +- orphaned or cross-channel person/household links; +- pool household `TEN` disagreement with the pinned household archive; +- an incomplete one-to-one person join; +- an unsupported raw code or age/universe violation; +- a conflicting pre-existing ACS predictor value; +- missing, nonnumeric, or nonfinite native ASEC predictors, or negative + `SSI_VAL`; and +- any remaining null in a model-consumed predictor. + +The relevant executable checks are at +`acs_release_predictors.py:830-968,1078-1356,1548-1586`. + +## Reviewed crosswalks + +The canonical crosswalk payload is source-controlled at +`acs_release_predictors.py:726-790` and pinned as: + +`1d4906242e9c73e31b3283659e5cad8242b8cbc42914ab6fa59547a10c8770e9` + +| Source | Target | Exact reviewed mapping | Consumed model bin | +|---|---|---|---| +| `DDRS` | `PEDISDRS` | age 5+: `1 -> 1`, `2 -> 2`; younger blank `-> -1` | SSI tests only `== 1` | +| `DEAR` | `PEDISEAR` | all ages: `1 -> 1`, `2 -> 2` | SSI tests only `== 1` | +| `DEYE` | `PEDISEYE` | all ages: `1 -> 1`, `2 -> 2` | SSI tests only `== 1` | +| `DOUT` | `PEDISOUT` | age 15+: `1 -> 1`, `2 -> 2`; younger blank `-> -1` | SSI tests only `== 1` | +| `DPHY` | `PEDISPHY` | age 5+: `1 -> 1`, `2 -> 2`; younger blank `-> -1` | SSI tests only `== 1` | +| `DREM` | `PEDISREM` | age 5+: `1 -> 1`, `2 -> 2`; younger blank `-> -1` | SSI tests only `== 1` | +| `RAC1P` | `PRDTRACE` | `1 -> 1` White; `2 -> 2` Black; `6 -> 4` Asian; `3/4/5/7/8/9 -> 3` residual Other | SCF White/Black/Asian/Other; ORG White/Black/Other | +| `HISP` | `PRDTHSP` | `1 -> 0` non-Hispanic; `2..24 -> 1` positive Hispanic representative | both consumers test zero versus positive | +| observed `OCCP` | `PEIOOCC` | identity over all 530 pinned detailed codes | SIPP tips exact detailed-code membership | +| blank `OCCP` | `PEIOOCC` | `-1`, the CPS NIU sentinel | unlisted/non-tipped | +| observed `OCCP` | `POCCU2` | explicit 530-key table covering consumed bins 1 through 53 | ORG exact categories and EAP set | +| blank `OCCP` | `POCCU2` | age below 16 `-> 0`; age 16+ permitted only with `ESR=6`, then `-> 53` | preserve NIU; adult no occupation/never worked | +| household `TEN` | `SPM_TENMORTSTATUS` | `1 -> 1`, `2 -> 2`, `3/4 -> 3`; verified group-quarters blank `-> 3` | SIPP vehicle homeowner iff `{1,2}` | + +The disability, race, occupation, and tenure executable mappings are at +`acs_release_predictors.py:108-687,689-694,1364-1454`. The occupation map's +notable reviewed edges include `3250 -> 26` while `3255/3256/3258 -> 25`, ACS +military `9800/9810/9825/9830 -> 52`, and `9920 -> 53`. The table maps the +shared detailed code vocabulary to the modal 2024 ASEC category; it is not +represented as a fictional rowwise CPS identity. + +### Age-15 occupation judgment + +ACS `OCCP` begins at age 16, while CPS `POCCU2` is already in universe at age +15. In source-year-2024 ASEC, 2,174 people are age 15: 1,931 have `POCCU2=53` +but 243 have categories 1 through 52. Mapping every ACS age-15 blank to 53 +would therefore invent never-worked evidence. The crosswalk preserves 0 for +that one-year source/target universe gap and uses 53 only for defensible adult +blanks. Raw `ESR` is required blank below 16 and in `1..6` from 16; an adult +`OCCP` blank is allowed only at `ESR=6`. + +## SSI reporter semantics + +`SSI_VAL` is the measured ASEC amount. The pool's native ACS mapping already +stores adjusted `SSIP` as `ssi_reported`. The release join now reads raw `SSIP` +and `ADJINC`, enforces the exact age-15 universe, and proves clone-zero +`ssi_reported == SSIP * ADJINC / 1_000_000` source person by source person +(`acs_release_predictors.py:1458-1545`). + +The SSI receiver coalesces `SSI_VAL` and `ssi_reported` rowwise, without +origin-routing the model, rejects conflicting dual reporter statuses, permits +a blank only below age 15, and fills that blank only transiently for the +existing `> 0` predicate (`ssi_disability_criteria.py:758-813,1082-1093`). The +gate summary uses the same coalesced anchor and native-role scope, so a lost +positive ACS-native reporter cannot evade the diagnostic. Source nulls remain +null in the frame. + +## Release integration and receipts + +The release builder declares the four inputs at +`build_us_fiscal_refresh_release.py:1293-1318` and validates all-or-none plus +lowercase SHA syntax at `:1587-1610`: + +- `--acs-person-zip` +- `--acs-person-sha256` +- `--acs-household-zip` +- `--acs-household-sha256` + +It invokes the join at `build_us_fiscal_refresh_release.py:9828-9846`, before +SCF wealth and consequently before all six archived-model stages. The receipt +is embedded at top level in `build_manifest.json` (`:7534-7642`) and under +`release_manifest.json.build` (`:7841-7871`); the sole writer call receives the +saved runtime receipt at `:11860-11879`. + +The complete real-data receipt included this core evidence: + +```json +{ + "enabled": true, + "crosswalk": { + "version": 1, + "sha256": "1d4906242e9c73e31b3283659e5cad8242b8cbc42914ab6fa59547a10c8770e9" + }, + "join": { + "semantic_key": ["household.SERIALNO", "person.SPORDER"], + "clone_fanout_key": "person_source_id", + "acs_source_people": 856626, + "acs_support_rows": 1736840, + "acs_support_rows_by_clone_index": {"0": 856626, "1": 856626, "2": 23588}, + "selected_raw_person_rows": 856626, + "selected_raw_household_rows": 382903, + "unmatched_pool_source_people": 0, + "source_identity_collisions": 0, + "semantic_key_sha256": "c7723adc889fc655b46103426d21f6c74937434eff51e8f3e6025bdf60972b74" + } +} +``` + +Per-model counts were: + +| Model | Predictors | ASEC-native each | ACS-joined each | Still null | +|---|---|---:|---:|---:| +| SSI disability | six `PEDIS*` | 234,133 | 1,736,840 | 0 | +| SCF wealth | `PRDTRACE`, `PRDTHSP` | 234,133 | 1,736,840 | 0 | +| SCF auto loans | `PRDTRACE`, `PRDTHSP` | 234,133 | 1,736,840 | 0 | +| SIPP vehicles | `SPM_TENMORTSTATUS` | 234,133 | 1,736,840 | 0 | +| SIPP tips | `PEIOOCC` | 234,133 | 1,736,840 | 0 | +| ORG wages/FLSA | `PRDTRACE`, `PRDTHSP`, `POCCU2` | 234,133 | 1,736,840 | 0 | +| SSI logical reporter anchor | ASEC `SSI_VAL`; ACS `ssi_reported` | 234,133 | 1,475,235 | 261,605 child-universe nulls | + +The receipt producer is at `acs_release_predictors.py:1589-1677`. + +## Archived-model behavior retained + +The five non-SSI consumer modules are byte-unchanged from the lane base. The +SSI model's predictor list, QRF feature/selection logic, role routing, and +thresholds are unchanged; only its source-faithful reporter read and matching +gate diagnostic changed. + +| Consumer | Executable consumed bins | Evidence | |---|---|---| -| Validation input coverage; register consistency; release target parity; target-profile coverage; base-population scale | Manifest, registry, target, or aggregate-weight contracts; no legacy per-row source roster | Already compatible; no change | -| PUF capital-gains-tail presence and post-selection preservation | Authenticated tail-column presence/preservation checks, not a demand for a physical PUF channel (`build_us_fiscal_refresh_release.py:8837-8841,8937-8941`) | Already compatible; no change | -| Exact-k PUF tail support; exact-k frozen-register fit; fiscal-target materialization/skip/zero-support; critical/SOI fit | Calibrated diagnostics and authenticated exact-k inputs, not physical source labels | Already compatible; no change | -| Weeks input and post-selection weeks input | Used clone operator roles as physical ASEC/PUF channels and applied raw/UC checks to the wrong rows | **Fixed** with physical roster and separate validity/reconciliation/UC scopes | -| QBI input | Already has an explicit stacked ACS path | Already compatible; no change | -| Workers' compensation; alimony; Medicare; retirement contributions; retirement distributions | Raw ASEC-only columns were validated/reconciled through legacy operator roles | **Fixed**: all physical ASEC clones own raw validity; native physical ASEC rows own exact direct-carry reconciliation where applicable | -| SSI reporter capture/take-up | Strict `SSI_VAL` validation treated null ACS raw-source cells as invalid | **Fixed**: capture reporters from physical ASEC; assignment remains source-blind and consumes the captured ID set | -| SIPP Head Start; voluntary filing | Assumed exactly an occurrence pair / two clones | **Fixed**: explicit assembly clone index, arbitrary clone count, duplicate source+clone refusal, clone-0-or-lowest canonical decision, fanout to every clone | -| Prior-year income clone diagnostic | Compared only the first two occurrences | **Fixed**: group all assembled clones; existing availability band unchanged | -| SSI disability clone-divergence diagnostic | Compared only a pair and missed clone-2-only divergence | **Fixed** for all assembled clones; whether divergence becomes fatal still requires owner ruling | -| WIC deterministic draw | Raw source-local identity could collide across physical ASEC and ACS | **Fixed**: assembled frames prefer unique `person_source_id`; clones remain draw-stable and legacy precedence is unchanged | -| Farm business; domestic production; child support; disability benefits; educator expenses; Form 4952; SALT refund; capital-gain details; energy subsidy; housing; other health insurance | These use clone-operator roles for reviewed producer/plausibility semantics, not as claims about physical source | Intentionally unchanged; stacked-compatible role diagnostics | -| Childcare; casualty loss; miscellaneous itemized; immigration; generic take-up; hours; SNAP take-up; relationship; eligibility; education; pregnancy; reported-coverage vintage; SNAP discretionary exemption | Output/input signal checks do not assume a physical PUF channel or require raw ASEC cells on all rows | Already compatible; no change | -| Local health input; Medicaid take-up; SNAP state take-up; SSI final/delivery | Output or diagnostics-based checks | Already compatible after SSI reporter fix | -| Input-mass reference; degenerate input; eCPS parity; release input coverage; export input mass; QRF tail concentration; reform-coverage smoke; source coverage | Export-wide schema/mass/tail/simulation contracts, not legacy source rosters | Already compatible; no change | -| Export count-calibrated take-up staleness | Export consistency against recorded count-calibration receipts (`build_us_fiscal_refresh_release.py:11702-11726`), independent of physical source roster | Already compatible; no change | -| SCF wealth | Archived predictors silently map ACS-null CPS race/Hispanic source cells to `Other` | **Owner ruling required**, detailed below | -| SSI disability criteria | Archived receiver strictly requires six ASEC disability predictors and `SSI_VAL` on every row | **Owner ruling required**, detailed below | -| SCF auto loans | Imports the SCF wealth CPS-race mapping, silently mapping ACS recipients to `Other` | **Owner ruling required**, detailed below | -| SIPP vehicles | Missing ACS `SPM_TENMORTSTATUS` defaults every ACS recipient to tenant code 3 | **Owner ruling required**, detailed below | -| SIPP tips | Missing ACS `PEIOOCC` defaults to non-tipped; global band can conceal a dead ACS channel | **Owner ruling required**, detailed below | -| ORG wages/FLSA | Missing ACS race/ethnicity/occupation defaults to zero; unchanged global gates are guaranteed to fail | **Owner ruling required**, detailed below | - -### Unambiguous repairs - -- `support_provenance.py:331-354,496-524` centralizes detection of assembled - metadata and physical channels. General derive/imputation code cannot call - the physical accessor: `test_us_spine_blindness.py` pins the exact reviewed - call graph. -- `alimony.py:282-327` and `workers_compensation.py:560-604` split physical - source validity from direct-carry reconciliation while preserving all role - bands. -- `medicare_take_up.py:134-143,248-270`, - `retirement_contributions.py:172-199,602-628`, and - `retirement_distributions.py:274-301,709-739` scope release diagnostics - without routing their population operators by physical origin. -- `ssi_take_up.py:515-550` captures reporters from physical ASEC and accepts - ACS nulls; the assignment consumes captured lineage rather than reading - origin. -- `sipp_head_start.py:579-621` and - `voluntary_filing.py:749-783` use explicit clone indices on assembled - frames. `prior_year_income.py:744-765` and - `ssi_disability_criteria.py:1119-1152` inspect all clones. -- `wic_claim.py:337-376` prefers the assembly-unique identity only when - assembled metadata exists. - -### Owner rulings required - -1. **SSI disability criteria.** Release call - `tools/build_us_fiscal_refresh_release.py:9840-9847`. The receiver - requires and strictly evaluates all six `PEDIS*` predictors on every row - at - `packages/microcosm-build/src/microcosm/build/us_runtime/ssi_disability_criteria.py:755-859`, - then strictly evaluates `SSI_VAL` at the same file's `:1023-1034`; the - support-role prediction loop is `:965-1005`. Each input is finite on all - 234,133 physical ASEC rows and null on all 1,736,840 ACS rows, so the stage - fails before its gate. - Required decision: transfer/map canonical ACS disability inputs and - reporter-anchor semantics, or revise/retrain the archived recipient model. - -2. **SCF wealth.** Release call - `tools/build_us_fiscal_refresh_release.py:9798-9808`. - `packages/microcosm-build/src/microcosm/build/us_runtime/scf_wealth.py:654-667` - requires `PRDTRACE`, `PRDTHSP`, `A_MARITL`, `PEPAR1/2`, `PH_SEQ`, and - `A_LINENO`; `:670-691` selects the archived reference person, and - `:694-740` builds race/marriage/children predictors. Only - `PRDTRACE/PRDTHSP` are ACS-null; `:611-620` maps those to zero/`Other`. - The other structural fields are finite and mapped on every ACS row. All - three asset leaves are all-null and `net_worth` is absent, so the wrapper's - recompute branch at `:1109-1159` is certain and every ACS recipient is - silently forced to `Other`. Required decision: authoritative ACS - race/Hispanic mapping or transfer, or a reviewed fallback/retrained model. - -3. **SCF auto loans.** Release call - `tools/build_us_fiscal_refresh_release.py:9990-9996`. - `packages/microcosm-build/src/microcosm/build/us_runtime/scf_auto_loans.py:125-137` - requires the archived layout; `:320-362` selects the reference person and - `:365-417` supplies race/marriage/children inputs. `A_LINENO` is required - by the selector's presence check but is not subsequently used; the mapped - structural fields are finite. The actual defect is the imported SCF-wealth - CPS-race mapping of ACS-null `PRDTRACE/PRDTHSP` to `Other`. All three - auto-loan outputs are absent, so `:475-513` certainly takes the QRF path. - Required decision: the same authoritative mapping or reviewed model change - as SCF wealth. - -4. **SIPP vehicles.** Release call - `tools/build_us_fiscal_refresh_release.py:10026-10033`. - `packages/microcosm-build/src/microcosm/build/us_runtime/sipp_vehicles.py:150-157` - declares the archived layout; `:577-622` reads person - `SPM_TENMORTSTATUS` and fills missing values to tenant code 3, while - `:649-763` uses marriage, `A_LINENO` reference selection, and tenure. - The pool's mapped `A_LINENO` is finite, but person - `SPM_TENMORTSTATUS` is ACS-null and the household field is absent. Both - vehicle outputs are absent, so `:872-917` certainly recomputes and - classifies ACS tenure as tenant. Required decision: canonical ACS tenure - mapping/transfer and whether assembled `A_LINENO` preserves the archived - reference-person semantics, or model revision/retraining. - -5. **SIPP tips.** Release call - `tools/build_us_fiscal_refresh_release.py:10103-10109`. - `packages/microcosm-build/src/microcosm/build/us_runtime/sipp_tips.py:249-263` - maps null `PEIOOCC` to tipped code 0; recipient use is `:363-396`, wrapper - branching is `:450-473`, and the gate is `:484-549`. The pool lacks both - tip output columns, so the wrapper takes its imputation branch. - `PEIOOCC` is ASEC-finite and ACS-null. The ASEC-conditional tipped-code - share is `0.0700204513`; ACS is exactly zero; the global share is - nevertheless `0.0353393976`, inside the unchanged `[0.02, 0.15]` band. - This proves that gate component conceals the dead ACS channel, not that the - separate tip-income component necessarily passes. Required decision: - map/transfer ACS occupation or select a reviewed alternate model, then add - channel-aware diagnostics once those semantics are owned. - -6. **ORG wages/FLSA.** Release call - `tools/build_us_fiscal_refresh_release.py:10144-10150`. - `packages/microcosm-build/src/microcosm/build/us_runtime/org_wages.py:535-559` - fills null `PRDTRACE/PRDTHSP/POCCU2` to zero; these enter features at - `:581-625`; wrapper `:901-945` always recomputes; gate `:959-1034`. All - three raw fields are ASEC-finite and ACS-null. On real weights, CPS-race - nonzero share is `0.5047010838` versus unchanged minimum `0.95`, and - detailed-occupation nonzero share is `0.4155259611` versus minimum `0.65`. - The next release is guaranteed to fail here. Required decision: - authoritative ACS race/ethnicity/occupation mapping or a revised model - contract, not threshold weakening. - -### Read-only real-pool replay after the fixes - -The repaired alimony, Medicare, retirement-contribution, -retirement-distribution, workers'-compensation, and SSI-reporter checks pass. -Their physical ASEC source count is 234,133; direct-carry gates use 108,073 -native ASEC rows where applicable. The weeks gate fails only on 369 fractional -values. - -Two other observed failures are genuine data/spec outcomes, not row-label -archaeology: - -- prior-year-income weighted availability is `0.042839`, outside the - unchanged `[0.05, 0.50]` band; -- WIC finds `is_pregnant=true` on nonfemale rows, with example row positions - `129405, 167076, 171133, 192443, 195546`. - -No threshold was changed to hide either outcome. - -## Diff summary by file - -| File | Rationale | -|---|---| -| `PROGRESS.md` | Maintains the required state/done/next journal from kickoff through verified handoff while preserving historical lanes | -| `changelog.d/stacked-release-gate-alignment.fixed.md` | Records the user-visible integer-support and stacked-gate fix under repository convention | -| `out.md` | This provenance, audit, verification, and handoff report | -| `us/spec/imputation.yaml` | Generated authority now receipts `weeks_unemployed` in the discrete numeric transfer roster | -| `us_runtime/acs_transfer.py` | Declares weeks integer-supported through the existing deterministic observed-support codec | -| `us_runtime/support_provenance.py` | Adds assembled-metadata detection and a narrow read-only physical-source accessor | -| `us_runtime/__init__.py`, `us_runtime/puf_support.py` | Re-export assembled-layout detection while keeping the physical accessor confined to its owner | -| `us_runtime/weeks_unemployed.py` | Derives stacked/legacy rosters and separates source validity, native reconciliation, and UC scopes | -| `us_runtime/alimony.py`, `us_runtime/workers_compensation.py` | Scope raw-source and exact-carry checks to their physical/native owners | -| `us_runtime/medicare_take_up.py`, `us_runtime/retirement_contributions.py`, `us_runtime/retirement_distributions.py` | Make release summaries source-aware while retaining source-blind producer kernels | -| `us_runtime/ssi_take_up.py` | Captures SSI reporters from physical ASEC and permits null ACS source cells without origin-routing assignment | -| `us_runtime/sipp_head_start.py`, `us_runtime/voluntary_filing.py` | Replace pair/occurrence layouts with explicit arbitrary-clone assembled layouts | -| `us_runtime/prior_year_income.py`, `us_runtime/ssi_disability_criteria.py` | Detect divergence across all assembled clones | -| `us_runtime/wic_claim.py` | Uses assembly-unique person identity for cross-origin deterministic draws | -| `spec_engine/seeds.py` | Receipts assembled `person_source_id` as WIC's first seed-key source | -| `spec_engine/field_usage.py`, `spec_engine/inventory_coverage.py`, `tools/spec_engine_coverage.py` | Re-pin the exact two-field expansion and downstream producer/seed/checkpoint/pointer identities | -| `docs/evidence/spec-engine/us-f0-coverage.json` | Regenerates the closed 42,122-field, 41-item coverage attestation | -| `tests/test_us_acs_transfer.py` | Proves fractional QRF output is snapped to observed integer weeks support and receipted | -| `tests/test_us_weeks_unemployed.py` | Covers stacked ASEC+ACS and legacy ASEC+PUF roster/scope contracts | -| `tests/test_us_alimony.py`, `test_us_workers_compensation.py`, `test_us_medicare_take_up.py`, `test_us_retirement_contributions.py`, `test_us_retirement_distributions.py`, `test_us_ssi_take_up.py` | Cover physical ASEC source validity, native reconciliation, legacy behavior, and SSI reporter lineage | -| `tests/test_us_sipp_head_start.py`, `test_us_voluntary_filing.py`, `test_us_prior_year_income.py`, `test_us_ssi_disability_criteria.py` | Cover clone 2+, duplicate clone refusal, deterministic canonical rows/fanout, and all-clone divergence | -| `tests/test_us_wic_claim.py` | Covers cross-origin identity collision avoidance and clone-stable draws | -| `tests/test_us_multispine_puf_clone.py`, `test_us_spine_blindness.py` | Cover assembled provenance and pin physical-source access to the reviewed gate/reporter call graph | -| `tests/test_spec_engine_seeds.py` | Pins WIC's assembled-first seed grammar | -| `tests/test_spec_engine_field_usage.py`, `test_spec_engine_coverage_tool.py`, `test_spec_engine_country_bundles.py`, `test_spec_engine_loader.py` | Pin the reviewed field totals, report identities, and source-attested envelope goldens | -| `tests/test_us_multispine_pool_tool.py` | Re-pins the constants adapter's live US spec identity after regeneration while retaining the separate arbitrary checkpoint identity fixture | - -All source paths in the table are under -`packages/microcosm-build/src/microcosm/build/`; all abbreviated test paths -are under `packages/microcosm-build/tests/`. - -## Judgment calls - -- The post-transfer calibration was not modified merely because its receipt is - visible near the symptom. Checkpoint bits prove the surviving values predate - it, and its mapping already uses actual donor support. -- The integer contract is explicit by target rather than inferred from the - current PolicyEngine physical dtype. That dtype is `float` and caused the - bug; reviewed domain semantics are the authority. -- Snapping to observed donor support was preferred over generic rounding. It - cannot invent an unsupported week count, is deterministic, and reuses an - existing receipted codec. -- Physical source identity is exposed only to read-only gates/reporters. - Medicare and retirement producer changes considered during review were - reverted: routing population treatment by origin would violate the - source-blind operator boundary. The authenticated release path consumes - already-produced pool surfaces. -- Raw validity covers all physical ASEC clones because they carry the raw - source. Exact reconciliation covers only native ASEC where transferred - non-native clones are intentionally allowed to differ. -- Operator roles remain appropriate for producer/plausibility bands. They are - not aliases for physical ASEC/ACS source, which was the old weeks bug. -- Clone divergence in SSI disability remains diagnostic rather than newly - fatal; changing that policy needs an owner ruling. -- The six archived-model cases were not “fixed” with zero fills, gate - rescoping, or weaker thresholds. Each requires substantive decisions about - ACS predictors and model semantics. - -## Verification evidence - -All commands ran offline against the prebuilt environment, with -`UV_CACHE_DIR=/private/tmp/microcosm-weeksgate-uv-cache` so the required -`uv run --no-sync` command could operate inside the sandbox. Each pytest -shard ran in one independent process. +| SSI disability | six difficulty inputs consumed only as `== 1`; reporter is under 65 and `> 0` | `ssi_disability_criteria.py:143-150,902-908,1082-1093` | +| SCF wealth | White 1, Black 2, Asian 4, Hispanic positive, residual Other | `scf_wealth.py:611-621,654-724` | +| SCF auto loans | reuses the SCF wealth race helper | `scf_auto_loans.py:38-42,365-380` | +| SIPP vehicles | homeowner iff tenure code is 1 or 2 | `sipp_vehicles.py:577-621,740-763` | +| SIPP tips | exact detailed Census occupation membership; unlisted NIU is zero | `sipp_tips.py:129-184,249-263,363-375` | +| ORG wages/FLSA | POCCU2 53 never worked, 52 military, 8 computer, 41 farmer/fisher, explicit EAP set; Hispanic/White/Black/Other | `org_wages.py:133-183,408-415,535-559,610-625` | + +No gate band or numeric threshold was edited. Whether ORG and tips pass or fail +on a future release is therefore determined by the real joined values, as +required. + +## Real-pool audit and operational caveat + +The supplied candidate pool contains 1,970,973 people and 865,460 households. +Its 3,239,263,147-byte H5 exactly matches the frozen manifest SHA-256 +`871b7e6467675a1e9475b54fd1baf64c53c0f75a3258b8357303a8df0d53642d`. + +The current official release loader refuses that older candidate before H5 +loading because the manifest's archived primary-QRF worker binding predates +this branch's source-attested execution identity: `late primary-QRF worker +binding changed`. To isolate and test this lane without building an artifact, +the audit independently verified the H5 bytes, read its entity tables through +the repository HDF reader, restored the frozen `assembly_receipt`, and ran the +strict join. Every join and crosswalk assertion passed. + +This is not an unresolved crosswalk issue, but it is an operational handoff: +the old candidate cannot be promoted through the current authenticated release +loader. A fresh/currently authenticated candidate is required outside this +headless no-build lane. + +## Verification + +All commands ran offline against the prebuilt environment. Each package shard +ran in its own pytest process. Packaging was not touched, so wheels were not +built. - `uv run --no-sync ruff check .`: PASS, `All checks passed!` - `uv run --no-sync python tools/ci_test_groups.py --verify`: PASS, - `tracked_test_files=309`, `verification=ok` -- Eight directly affected spec/receipt files: PASS, 102 tests -- `tools/generate_us_bundle_from_constants.py --check`: PASS, final US spec - SHA-256 - `5f44d96d45e9aabcea2d565ef063d68bfc0652df1b38b08aa31ce6896d15f371` + `tracked_test_files=310`, `verification=ok` +- `tools/generate_us_bundle_from_constants.py --check`: PASS, US spec + `16b7d5e622e8a68e008165bb44a5836695d94a2b0dd8d4c51b3c9e8ca89dca38` - `tools/spec_engine_coverage.py --check`: PASS, 42,122/42,122 fields and 41/41 inventory checks -- `uv run --no-sync pytest packages/microcosm-calibrate/tests -q`: PASS, - 203 passed -- `uv run --no-sync pytest packages/microcosm-data/tests -q`: PASS, - 318 passed, 2 skipped -- `uv run --no-sync pytest packages/microcosm-fit/tests -q`: PASS, - 93 passed -- `uv run --no-sync pytest packages/microcosm-frame/tests -q`: PASS, - 295 passed, 36 skipped -- `uv run --no-sync pytest packages/microcosm-build/tests -q`: PASS, - 6,608 tests collected, 100% reached, exit code 0, expected skips only -- Complete ACS-transfer file: PASS, 65 tests -- Complete post-transfer calibration receipt-contract file: PASS, 47 tests -- Complete weeks file: PASS, 26 passed, 1 skipped -- Complete alimony/workers'/SSI files: PASS, 28 / 21 / 71 tests -- Medicare + retirement contribution/distribution focused files: PASS, 65 tests -- Repository `git diff --check`: PASS - -The shard warnings observed are existing numerical, sparse-tensor, -joblib core-detection, and PolicyEngine divide warnings; none is a failure and -none originates in the new transfer/source-scope paths. - -## Commit inventory - -Implementation and evidence commits preceding the final report carrier: - -1. `a3331db6` Start stacked release gate alignment journal -2. `9979d101` Record fractional weeks provenance -3. `d7ad753e` Preserve integer support for transferred weeks -4. `0c5b05a8` Receipt integer weeks transfer support -5. `c22a6799` Align weeks gate with stacked source roles -6. `9bbd6dfe` Centralize stacked source channel scopes -7. `49052e6e` Scope alimony source checks to physical ASEC -8. `17ed9a0b` Key WIC draws by stacked person identity -9. `c7dd53ad` Support stacked Head Start clone layouts -10. `3f934688` Scope Medicare source gate to stacked ASEC -11. `5852f4ba` Scope retirement contribution source diagnostics -12. `e40d5db7` Support stacked voluntary filing clones -13. `148881c6` Check all stacked prior-year income clones -14. `52b27f78` Separate alimony source validation and carry scopes -15. `058f27b5` Record stacked clone-layout repairs -16. `97de22e7` Scope workers compensation sources in stacked pools -17. `74e96187` Scope retirement distribution source gates -18. `41007992` Anchor SSI reporters to physical ASEC sources -19. `eb5cd4b6` Keep SSI assignment source blind -20. `3dc5df51` Keep stacked source repairs gate scoped -21. `9b36720b` Record stacked raw-source gate repairs -22. `9520c03f` Report divergence across all SSI disability clones -23. `b8e04e9c` Confine physical source access to release gates -24. `b2de92b9` Document stacked release gate alignment -25. `cd8c55a7` Repin spec engine for integer weeks -26. `12a918ed` Repin multispine live spec fixture - -The final report/journal carrier follows this inventory and is necessarily -self-excluded; the clean-worktree handoff lists it in the final response. +- frame shard: PASS, 295 passed, 36 skipped +- fit shard: PASS, 93 passed +- calibrate shard: PASS, 203 passed +- data shard: PASS, 318 passed, 2 skipped +- build shard after reviewed source-attestation repin: PASS, 6,586 passed, + 45 skipped +- join-focused file: PASS, 15 passed +- combined join, SSI, and source-blindness files: PASS +- parser/order/manifest focused cases and all six parametrized mocked-main + corridor cases: PASS +- repository `git diff --check`: PASS + +The first complete build-shard run reached 100% with 6,575 passed and 45 +skipped plus five failures and six setup errors, all caused by stale expected +source-attestation hashes after changing `ssi_disability_criteria.py`. That +module participates in both seed-kernel source inventories. The narrow +established repin updated the two seed digests, BE/UK/US spec identities, the +minimal-loader golden, and generated coverage evidence. All 25 affected tests +then passed, both retained `--check` commands passed, and the complete build +shard rerun above was green. + +Existing warnings were numerical overflow/divide, pandas chained-assignment, +and fragmented-fixture performance warnings; none was a test failure. + +## Judgment calls and unresolved items + +- The join uses retained semantic lineage, not a guessed arithmetic inversion + of assembly IDs. +- Race and Hispanic mappings stop at the bins the consumers actually read; + they do not invent CPS detail absent from ACS. +- Disability codes preserve `1/2` plus the CPS NIU sentinel rather than + collapsing the source to booleans. +- Age-15 ACS occupation remains NIU because assigning 53 would fabricate + never-worked evidence for a group whose CPS distribution is not degenerate. +- `PEIOOCC` blank uses CPS NIU `-1`, not a fictional detailed occupation 0. +- SSI child nulls remain source nulls; only the predicate interprets them as + non-reporters. +- Pool household `TEN` is preferred for lineage and must agree exactly with the + pinned household zip; group-quarters non-owner status is applied only after + raw universe checks. +- ASEC values are validated but never rewritten, preserving their native model + inputs byte-for-byte. +- No crosswalk semantic remains unresolved. The only open operational item is + obtaining a current authenticated pool and adding the four launcher + arguments; both are explicitly outside this lane and owned by the + dispatcher/build process. + +## Commits before final journal handoff + +- `2aa14e84` Start ACS predictor release join journal +- `c1a41ccf` Record ACS predictor join contracts +- `ecea55a2` Add strict ACS release predictor join +- `a7108697` Harden ACS predictor crosswalk contracts +- `6b8185e4` Wire ACS predictor join into release builder +- `d87be068` Accept numeric H5 predictor cells +- `69ec0fae` Document ACS release predictor join +- `de5e5d03` Repin source-attested spec identities + +The final local commit adds this report and marks `PROGRESS.md` complete. From 011563db8bb0fd766da3ba8c04f9b472789b79cb Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Thu, 27 Aug 2026 21:20:14 +0200 Subject: [PATCH 47/49] Record completed provenance repin --- PROGRESS.md | 10 +++++----- .../microcosm-build/tests/test_us_acs_multispine.py | 8 ++++---- 2 files changed, 9 insertions(+), 9 deletions(-) diff --git a/PROGRESS.md b/PROGRESS.md index eea8e5b49..bfc95a2f1 100644 --- a/PROGRESS.md +++ b/PROGRESS.md @@ -7,10 +7,10 @@ Resumed on 2026-08-27 on branch `pregnancy-prioryear-defects`, based on adopted and implementation is in progress for the nonfemale-pregnancy producer defect and the owner-approved rung-aware prior-year availability release floor. Both fixes are implemented and their focused runtime suites are green; the -source-attested spec/coverage repin and full repository verification are in -progress. The authored 0.05 floor, upper bound, all other bands, thresholds, -seeds, and batteries remain unchanged. This lane will not build, publish, or -push pool or release artifacts. +source-attested spec/coverage repins are committed, and only the clean build- +shard rerun and final report remain. The authored 0.05 floor, upper bound, all +other bands, thresholds, seeds, and batteries remain unchanged. This lane will +not build, publish, or push pool or release artifacts. ## Done @@ -109,10 +109,10 @@ push pool or release artifacts. provenance golden needed the new additive `structural_receipt: null` field on an unrelated transfer record. Updated only that JSON-ready shape; the complete optional-ACS multispine test file and focused Ruff pass. +- Committed the additive optional-ACS provenance fixture repin as `7caf69ac`. ## Next -- Commit the additive optional-ACS provenance fixture repin. - Restart the full build shard from zero; the other four final-state shards, repository Ruff, generated/coverage/inventory checks already pass. - Write and commit the complete handoff to `out.md`. diff --git a/packages/microcosm-build/tests/test_us_acs_multispine.py b/packages/microcosm-build/tests/test_us_acs_multispine.py index 537a27f80..afc38b72d 100644 --- a/packages/microcosm-build/tests/test_us_acs_multispine.py +++ b/packages/microcosm-build/tests/test_us_acs_multispine.py @@ -213,10 +213,10 @@ def fake_pool(actual_base, actual_acs, *, acs_share): "patterns": [], "imputed_recipient_rows": 0, "unmodeled_recipient_rows": 0, - "derivation": None, - "reconciliation": None, - "structural_receipt": None, - } + "derivation": None, + "reconciliation": None, + "structural_receipt": None, + } ], "deferred_inputs": ["congressional_district_geoid"], "adult_care_recipient_gate": None, From c1a4bfb98f5d219c89a8a9f96a1e155b5363f665 Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Thu, 27 Aug 2026 22:22:41 +0200 Subject: [PATCH 48/49] Record full shard verification --- PROGRESS.md | 28 ++++++++++++++++------------ 1 file changed, 16 insertions(+), 12 deletions(-) diff --git a/PROGRESS.md b/PROGRESS.md index bfc95a2f1..747159afd 100644 --- a/PROGRESS.md +++ b/PROGRESS.md @@ -3,14 +3,14 @@ ## State Resumed on 2026-08-27 on branch `pregnancy-prioryear-defects`, based on -`stacked-release-gate-alignment` at `606cbd69`. The root-cause findings are -adopted and implementation is in progress for the nonfemale-pregnancy producer -defect and the owner-approved rung-aware prior-year availability release floor. +`stacked-release-gate-alignment` at `606cbd69`. The root-cause findings and +implementations are complete for the nonfemale-pregnancy producer defect and +the owner-approved rung-aware prior-year availability release floor. Both fixes are implemented and their focused runtime suites are green; the -source-attested spec/coverage repins are committed, and only the clean build- -shard rerun and final report remain. The authored 0.05 floor, upper bound, all -other bands, thresholds, seeds, and batteries remain unchanged. This lane will -not build, publish, or push pool or release artifacts. +source-attested spec/coverage repins and full repository verification are +complete, and only the final report remains. The authored 0.05 floor, upper +bound, all other bands, thresholds, seeds, and batteries remain unchanged. This +lane did not build, publish, or push pool or release artifacts. ## Done @@ -23,8 +23,8 @@ not build, publish, or push pool or release artifacts. transfer defect. - Rebuilt a transient local GitNexus graph offline and queried pregnancy/WIC execution paths. Registration alone failed because the sandbox forbids the - CLI's global `~/.gitnexus` registry; the usable worktree-local index remains - untracked and will be removed before handoff. + CLI's global `~/.gitnexus` registry; the usable worktree-local index was + removed before final verification. - Inspected all 1,970,973 person rows in the supplied 25% pool. There are 108 `is_pregnant=true` nonfemale rows, all on physical ACS records: 45 clone 0, 61 clone 1, and 2 clone 2. ASEC has zero; sex/channel/clone assembly is @@ -95,7 +95,8 @@ not build, publish, or push pool or release artifacts. preceding test green; the repinned fixture and forged-policy controls pass. - Recomputed the source-attested identity after hardening: final resolved US spec SHA is `11e310c7619cbac91f6703b9679649cdd15f6fb09274ad29904c65881aa93316`; - no authored YAML changed. Regenerated coverage remains complete at + hardening required no additional authored YAML edit beyond the already- + generated pregnancy authority. Regenerated coverage remains complete at 42,154/42,154 fields and 41/41 inventory checks. - Committed the fixture, exact-boolean, nested-manifest-type, tamper-test, and final US identity repair as `e59ab046`. Final-state calibrate, data, fit, and @@ -110,11 +111,14 @@ not build, publish, or push pool or release artifacts. on an unrelated transfer record. Updated only that JSON-ready shape; the complete optional-ACS multispine test file and focused Ruff pass. - Committed the additive optional-ACS provenance fixture repin as `7caf69ac`. +- Restarted the complete build shard from zero on final implementation state: + `6601 passed, 45 skipped` in 55m20s, exit 0. Together with the already-green + final-state calibrate, data, fit, and frame shard processes, repository Ruff, + generated authority/coverage checks, and CI test inventory, this completes + the charter's verification boundary. ## Next -- Restart the full build shard from zero; the other four final-state shards, - repository Ruff, generated/coverage/inventory checks already pass. - Write and commit the complete handoff to `out.md`. # Weeksgate: stacked release gates and integer-week provenance From c314963f16f6411316797c6433cedec7f9e3596c Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Thu, 27 Aug 2026 22:34:01 +0200 Subject: [PATCH 49/49] Complete pregnancy and prior-year handoff --- PROGRESS.md | 25 +- out.md | 897 +++++++++++++++++++++------------------------------- 2 files changed, 369 insertions(+), 553 deletions(-) diff --git a/PROGRESS.md b/PROGRESS.md index 747159afd..8da306ad3 100644 --- a/PROGRESS.md +++ b/PROGRESS.md @@ -6,9 +6,8 @@ Resumed on 2026-08-27 on branch `pregnancy-prioryear-defects`, based on `stacked-release-gate-alignment` at `606cbd69`. The root-cause findings and implementations are complete for the nonfemale-pregnancy producer defect and the owner-approved rung-aware prior-year availability release floor. -Both fixes are implemented and their focused runtime suites are green; the -source-attested spec/coverage repins and full repository verification are -complete, and only the final report remains. The authored 0.05 floor, upper +Both fixes, source-attested spec/coverage repins, full repository verification, +and the final `out.md` handoff are complete. The authored 0.05 floor, upper bound, all other bands, thresholds, seeds, and batteries remain unchanged. This lane did not build, publish, or push pool or release artifacts. @@ -116,12 +115,18 @@ lane did not build, publish, or push pool or release artifacts. final-state calibrate, data, fit, and frame shard processes, repository Ruff, generated authority/coverage checks, and CI test inventory, this completes the charter's verification boundary. +- Replaced the stale prior-lane output with the complete pregprior report in + `out.md`: both real-pool decompositions, source and gate mechanisms, receipt + and identity chains, regressions, verification, judgment calls, and the + host-owned next action are recorded there. ## Next -- Write and commit the complete handoff to `out.md`. +- Host session: rebuild the affected pregnancy transfer from the invalidated + checkpoint/target bank, then run terminal pool/release gates. This headless + lane performed none of those artifact operations. -# Weeksgate: stacked release gates and integer-week provenance +# Historical: Weeksgate stacked release gates and integer-week provenance ## State @@ -134,7 +139,8 @@ release-call roster is classified; six archived-model input assumptions require owner rulings and are deliberately reported instead of guessed. Repository-wide Ruff, the CI inventory verifier, and all five full pytest shards pass in their required independent processes. The completed provenance, audit, verification, -judgment calls, and host-owned checkpoint-rerun consequence are in `out.md`. +judgment calls, and host-owned checkpoint-rerun consequence were recorded in +`out.md` at commit `606cbd69`. No network access, artifact build, publication, push, pool build, or release build is in scope. @@ -301,14 +307,15 @@ build is in scope. coverage proof, the CI test inventory, and `git diff --check` are green. - Wrote the final provenance tables, mechanism verdict, per-file rationale, exhaustive release-gate audit, owner-ruling list, verification evidence, and - judgment calls to `out.md`. + judgment calls to `out.md`, preserved at commit `606cbd69`. ## Next - Host session: rerun `late_transfer -> simulated -> terminal-gates` from the candidate checkpoints because the discrete weeks codec changes pool content. -- Review the six archived-model owner rulings in `out.md`; do not reinterpret - their missing ACS predictors through a gate-only threshold/scope change. +- Review the six archived-model owner rulings recorded in `out.md` at commit + `606cbd69`; do not reinterpret their missing ACS predictors through a + gate-only threshold/scope change. # Historical: gate-failed base-pool release lane diff --git a/out.md b/out.md index 6b9d8437a..f150d8d4e 100644 --- a/out.md +++ b/out.md @@ -1,560 +1,369 @@ -# Weeksgate: stacked release gates and fractional-weeks final report +# Pregnancy and prior-year defects: final report Date: 2026-08-27 -Branch: `stacked-release-gate-alignment` +Branch: `pregnancy-prioryear-defects` -Lane base: `4f453746` (`origin/main` at kickoff; includes #786) +Lane base: `606cbd69` (`stacked-release-gate-alignment`) ## Outcome -The 369 fractional `weeks_unemployed` values are genuine pool-content -defects, but the proposed post-transfer-calibration mechanism is refuted. They -were emitted earlier by the ACS-transfer QRF because PolicyEngine-US physically -types this integer-supported variable as `float`. The calibration maps ACS -clone 0 onto actual observed ASEC support and removes all 342 fractions in that -scope; it never touches the 360 ACS clone-1 and 9 ACS clone-2 fractions that -survive into the pool. - -The source fix registers `weeks_unemployed` as a discrete numeric ACS-transfer -target. Predictions now snap deterministically to actual donor support, with -lower-support tie breaking, and are returned on integer support. The transfer -execution contract and generated imputation authority receipt the policy. -The strict post-transfer calibration receipt and validator remain unchanged -because that kernel did not cause the defect; their complete contract suite -still passes. - -The weeks release gate now distinguishes physical source channel from -clone-operator role. It derives an assembled roster (`asec+acs`) or retains a -legacy roster (`asec+puf_tax_detail`), validates raw `LKWEEKS` only where an -ASEC source exists, reconciles direct carries only on native ASEC rows, and -checks the UC rule only on rows owned by that constraint. Every plausibility -band and numeric threshold is unchanged. - -The full release-side sweep fixed all unambiguous physical-source, -clone-layout, and stable-identity archaeology. Six archived-model predictor -contracts remain owner decisions and are reported precisely below. In -particular, ORG wages/FLSA is guaranteed to fail on the stacked pool, while -SIPP tips' global tipped-occupation component passes despite a dead ACS -channel. Those are not safe gate-only edits. - -No network access, pool build, release build, artifact publication, push, -plausibility-band change, or by-origin battery change was performed. Issue -#782's weeks-incidence-band adjudication was not touched. - -## Task 1: provenance of all 369 noninteger weeks - -### Evidence read - -The analysis read the supplied fixed-format pool directly: +Both assigned defects are resolved in code and covered by fail-closed +contracts: pregnancy has a source/transfer fix, while prior-year availability +has the owner-approved release-gate fix. -`/Users/maxghenis/PolicyEngine/_buildo-runtime/out/candidate-25/pool/pool.h5` +- Pregnancy is now structurally limited to female people ages 15 through 44 + before the ACS QRF draw. One source-person result is fanned across every + missing assembled clone, missing ineligible values become deterministic + false, and preexisting or final domain/clone violations are refused with + explicit counts. Production transfer validation authenticates the structural + receipt; the structural policy and execution contract are bound into + checkpoint and target-bank identity. +- The prior-year availability shortfall is a sampling-order effect, not ACS + dilution and not a transfer hole. Under the owner ruling, only the gate's + applied lower availability floor is scaled by the authenticated production + sampling rung. At rung 1.0 the gate and report follow the original path + byte-for-byte. The authored `0.05` constant, the `0.50` upper bound, all + other bands, thresholds, seeds, and batteries are unchanged. + +Full repository verification is green: all five package shards passed in +independent pytest processes, repository Ruff passed, generated bundle and +coverage bytes are current, the CI test inventory passed, and the worktree has +no leftover graph artifact or untracked file. -It contains 1,970,973 person rows: 234,133 physical ASEC rows and 1,736,840 -physical ACS rows. It also read the exact pre-calibration late-transfer target -bank: +This lane used the supplied 25% pool only as read-only diagnostic evidence. It +did not use the network, build or publish a pool/release, push, or publish any +artifact. The supplied pool still contains the old pregnancy outputs and is not +evidence that the source fix has run. -`/Users/maxghenis/PolicyEngine/_buildo-runtime/out/candidate-25/pool/checkpoints/stacked/3847d30e1488ae18891780a7a5de6a1b75d1329f2491796e35639fc9e802c26e/acs-transfer/0914d1b858dff5171f7743f0e535349be4459fb99b3386bbc2e3be72d40dd5fc/late_producer_dag/person/source_operator_weeks_unemployed/targets/000__weeks_unemployed.h5` +## Task 1: pregnancy structural eligibility -The raw-draw value SHA-256 is -`6ea20782bcb53fa730bb7e7045c7d79ab096dfe69bc5ceb540158d93f6672db4`. -The final fractional-row-index SHA-256 is -`3431befe89f007c353296b0c59792bc31e45a1031e62a6f9c99e003df286305f`; -the sorted 369-value SHA-256 is -`314ba4459bb57169eaf492dd28c8ea04af460104b962bb29285104e70eee976c`. +### Real-pool decomposition -### Counts by physical channel, clone, and UC +The supplied pool is: -| Physical channel | Clone index | All rows | Fractional, UC=0 | Fractional, UC>0 | +`/Users/maxghenis/PolicyEngine/_buildo-runtime/out/candidate-25/pool/pool.h5` + +It contains 1,970,973 person rows. The complete hard-domain decomposition is: + +| Physical channel | Clone | Person rows | Pregnant nonfemale | Pregnant female outside 15--44 | Total violations | +|---|---:|---:|---:|---:|---:| +| ASEC | 0 | 108,073 | 0 | 0 | 0 | +| ASEC | 1 | 108,073 | 0 | 0 | 0 | +| ASEC | 2 | 17,987 | 0 | 0 | 0 | +| **ASEC all** | | **234,133** | **0** | **0** | **0** | +| ACS | 0 | 856,626 | 45 | 30 | 75 | +| ACS | 1 | 856,626 | 61 | 28 | 89 | +| ACS | 2 | 23,588 | 2 | 0 | 2 | +| **ACS all** | | **1,736,840** | **108** | **58** | **166** | +| **Pool** | | **1,970,973** | **108** | **58** | **166** | + +The initially reported row positions 129405, 167076, 171133, 192443, and +195546 are members of the 108-row nonfemale subset. All 166 hard-domain +violations occur on ACS physical records and are isolated to one clone within +their source-person group. There are 11,287 ACS source people whose assembled +clones disagree on pregnancy; ASEC has zero clone disagreements. Sex values, +ages, physical channel, and clone attachment are internally consistent, which +rules out assembly or sex-code corruption. + +### Root cause + +The defect was the ACS pregnancy QRF path. The ASEC source producer already +conditions its stable pregnancy draw on female ages 15 through 44 +(`packages/microcosm-build/src/microcosm/build/us_runtime/pregnancy.py:292` and +`:330`). ACS transfer instead used sex and age only as soft QRF predictors, +modeled physical clone rows separately, and had neither a hard eligibility +precondition nor a domain/clone postcondition. That combination explains both +the 166 impossible positives and the 11,287 source-person disagreements. + +### Source fix and construction invariant + +The policy is explicit and identity-bearing at +`packages/microcosm-build/src/microcosm/build/us_runtime/acs_transfer.py:141`: + +1. Female status must be an exact `0/1` Boolean-domain value and age must be + complete, finite, and within the inclusive range 15--44. Donor and recipient + surfaces are checked before any requested pregnancy QRF runs + (`acs_transfer.py:973`). +2. Assembled recipients are grouped by `person_source_id`, must have valid + clone indices and exactly one clone 0, and must agree on eligibility and any + preexisting pregnancy value (`acs_transfer.py:1025` and `:1100`). +3. Only one eligible, unresolved clone-0 representative per source person is + sent to the QRF. Existing valid source-person values are preserved; missing + ineligible rows are resolved as structural false without entering the QRF + (`acs_transfer.py:1181`). +4. The one decoded source-person result is fanned to every missing clone + (`acs_transfer.py:1223`). Completeness, hard-domain validity, and all-clone + equality are then enforced as fail-closed postconditions + (`acs_transfer.py:1248`). +5. Pregnancy preflight still runs when its surface is already complete or when + some other requested family remains active. A complete surface therefore + carries a zero-imputation structural proof instead of bypassing validation + (`acs_transfer.py:1417`). Pregnancy is also isolated from unrelated bounded + QRF families (`acs_transfer.py:3157`). + +The ASEC source-stage producer and the release pregnancy signal gate apply +the same hard domain and exact-Boolean contract. Their details report the +actual domain and clone counts, and they refuse violations rather than +silently zeroing them +(`packages/microcosm-build/src/microcosm/build/us_runtime/pregnancy.py:167`, +`:224`, `:265`, and `:438`). + +### Receipt and execution identity + +The sealed structural receipt distinguishes four disjoint fill categories: +QRF representative rows, QRF clone fanout, preexisting-value fanout, and +ineligible-false assignments. It also records source-person topology and the +preexisting/final domain and clone-violation counts +(`acs_transfer.py:1189` and `:1285`). A successful new transfer must receipt +zero violations; a rejected surface names the nonzero count in its failure. + +Production validation authenticates the policy digest, exact integer count +types, zero-violation postconditions, source-person/clone arithmetic, one QRF +row per QRF source person, and exact equality between the four fill categories +and the transferred-row count +(`packages/microcosm-build/src/microcosm/build/us_runtime/stacked_spine.py:4203` +and `:4873`). Receipt emission is at `stacked_spine.py:10591`. + +The structural policy is inside the target-specific transfer execution +contract and its SHA (`acs_transfer.py:276`). Each late-transfer model binding +includes that contract (`stacked_spine.py:6691`), the outer checkpoint identity +authenticates those resource semantics (`stacked_spine.py:6809` and +`tools/build_us_multispine_pool.py:1410`), and late target-bank identities +extend the checkpoint identity (`tools/build_us_multispine_pool.py:3300` and +`:3726`). A policy change therefore invalidates both checkpoint reuse and +target-bank reuse. + +The generated US imputation authority declares the generic policy template at +`packages/microcosm-build/src/microcosm/build/us/spec/imputation.yaml:412` with +`enabled: false`; the resolver enables and rehashes it exactly for transfer +groups containing `is_pregnant` +(`packages/microcosm-build/src/microcosm/build/spec_engine/imputation_semantics.py:195`). +The late producer also declares `person_source_id` as its structural grouping +input (`packages/microcosm-build/src/microcosm/build/us_runtime/us_late_producer_registry.py:670`). +The final resolved US spec SHA is +`11e310c7619cbac91f6703b9679649cdd15f6fb09274ad29904c65881aa93316`. + +### Pregnancy regressions + +- Source refusal, gate counts/clone disagreement, and near-Boolean refusal: + `packages/microcosm-build/tests/test_us_pregnancy.py:251`, `:302`, and `:344`. +- One draw per source person, clone fanout, structural false, and disjoint + receipt accounting: `packages/microcosm-build/tests/test_us_acs_transfer.py:1448`. +- Donor/recipient pre-QRF and mixed-active refusal: `test_us_acs_transfer.py:1592`. +- Zero-imputation receipt and policy-dependent execution identity: + `test_us_acs_transfer.py:1694` and `:1722`. +- Integrated stacked transfer, receipt propagation, all-clone equality, + complete-surface proof, and invalid source refusal: + `packages/microcosm-build/tests/test_us_stacked_spine.py:6311`. +- Bound execution-contract emission and stale-contract refusal: + `test_us_stacked_spine.py:4651` and `:4693`. +- Forged policy-digest rejection: + `packages/microcosm-build/tests/test_us_multispine_pool_tool.py:3026`. + +## Task 2: prior-year availability + +### Real-pool decomposition and verdict + +| Physical channel | Clone | Person rows | Available rows | Weighted availability | |---|---:|---:|---:|---:| -| ASEC | 0 | 108,073 | 0 | 0 | -| ASEC | 1 | 108,073 | 0 | 0 | -| ASEC | 2 | 17,987 | 0 | 0 | -| ACS | 0 | 856,626 | 0 | 0 | -| ACS | 1 | 856,626 | 355 | 5 | -| ACS | 2 | 23,588 | 9 | 0 | -| **Total** | | **1,970,973** | **364** | **5** | - -All 369 values are positive, have null ASEC `LKWEEKS`, and are distinct at -their exact IEEE-754 representation. Every fractional row maps one-to-one by -`person_source_id` to a unique ACS clone-0 sibling, and -`unemployment_compensation` is identical across that sibling group. Of the -364 UC-zero rows, the clone-0 sibling has zero weeks. The five UC-positive -rows have clone-0 integer weeks in `{2, 4, 40, 48, 50}`. - -The five fractional value / annual-UC pairs are: - -| Fractional weeks | Unemployment compensation | -|---:|---:| -| 3.3297787140375217 | 10,800 | -| 3.948788299342264 | 20,000 | -| 17.23699569220113 | 7,200 | -| 18.64447048256536 | 10,800 | -| 21.97077954067273 | 4,400 | - -### Value distribution - -| Statistic | Value | +| ASEC | 0 | 108,073 | 4,724 | 0.04308325 | +| ASEC | 1 | 108,073 | 4,724 | 0.04308351 | +| ASEC | 2 | 17,987 | 752 | 0.04277737 | +| **ASEC all** | | **234,133** | **10,200** | **0.04308325** | +| ACS | 0 | 856,626 | 38,228 | 0.04258969 | +| ACS | 1 | 856,626 | 38,228 | 0.04258837 | +| ACS | 2 | 23,588 | 928 | 0.04374427 | +| **ACS all** | | **1,736,840** | **77,384** | **0.04258969** | +| **Pool** | | **1,970,973** | **87,584** | **0.04283879** | + +This is neither ACS dilution nor an ACS transfer hole. ASEC and ACS are both +about 4.3% available. ACS is intended to receive +`previous_year_income_available`: the generated authority declares the early +ASEC-to-ACS QRF over all recipient rows at +`packages/microcosm-build/src/microcosm/build/us/spec/imputation.yaml:1077`. +The nearly identical ASEC and ACS incidence is evidence that transfer is +operating. + +The cause is the assembly's sampling order. ASEC households are sampled before, +and without coupling to, their adjacent-year `PERIDNUM` partners +(`packages/microcosm-build/src/microcosm/build/us_runtime/stacked_spine.py:633` +and `prior_year_income.py:346`). Of 18,518 sampled current-year records that +match the intact full predecessor files, only 4,724 retain a predecessor after +the sampled-to-sampled join. Weighted match survival is 25.4117%, the expected +25% rung effect. As controls, full pooled ASEC availability is 16.9147%, while +selected current rows joined to intact prior files are 16.9541%. + +The owner accepted this sampling-order verdict. It falls under the charter's +"something else" category: the transfer is sound, and the low incidence is +the expected consequence of independently sampling both join sides. + +### Rung-aware release floor + +The authored bands remain exactly: + +- `_PREVIOUS_YEAR_AVAILABLE_SHARE_BAND = (0.05, 0.50)` +- `_SELF_EMPLOYMENT_NONZERO_SHARE_BAND = (0.01, 0.25)` + +They are at +`packages/microcosm-build/src/microcosm/build/us_runtime/prior_year_income.py:183`. +For a production sampling rung `r`, only the applied lower availability floor +becomes `0.05 * r`; the upper stays `0.50`. + +| Rung | Applied availability floor | Upper bound | +|---:|---:|---:| +| 0.01 | 0.0005 | 0.50 | +| 0.04 | 0.0020 | 0.50 | +| 0.10 | 0.0050 | 0.50 | +| 0.25 | 0.0125 | 0.50 | +| 1.00 | 0.0500 | 0.50 | + +The supplied candidate is gate-failed and contains stale pregnancy output; it +is not simulation-ready. Its sibling manifest nevertheless records and +digest-binds a valid sampling receipt at rung `0.25` (version 4, seed 578), so +the prior-year gate's applied floor is `0.0125`. The configured rung, not the +empirical 25.4117% diagnostic estimate, is the execution input. + +On sampled production assemblies, the gate adds these report fields so the +reader can see exactly why the floor moved: + +- `previous_year_income_available_sampled_match_survival_factor` +- `previous_year_income_available_applied_floor` +- `previous_year_income_available_applied_share_band` + +The conditional logic is at `prior_year_income.py:889`. At rung 1.0 the branch +is not entered: no new fields are emitted, the original band key is evaluated, +and the result is byte-identical to the pre-change gate. No-manifest frames and +legacy version-1 frames also retain factor 1.0. Malformed version-4 metadata or +unsupported manifest versions fail closed (`prior_year_income.py:189`). No +other prior-year check or band changed. + +### Manifest provenance and regressions + +Production assembly writes the version-4 common fraction/seed and both arm +receipts into frame metadata (`stacked_spine.py:679`). Pool publication carries +the summarized sampling receipt, full stack manifest, and canonical SHA-256 +binding (`tools/build_us_multispine_pool.py:4193` and `:4224`). The H5 loader +checks the schema-9 envelope, approved rung token, strict finite-float fraction, +non-Boolean seeds/counts, exact ASEC+ACS arms, arm/top agreement, positive +realized counts, and canonical manifest digest before restoring a deep copy to +frame metadata. The surrounding loader also authenticates the H5 SHA, size, and +run identity and rechecks the digest after reading +(`packages/microcosm-build/src/microcosm/build/us_runtime/h5_io.py:238`, `:270`, +`:819`, `:1750`, and `:1813`). On the production release path the gate therefore +consumes loader-authenticated, not caller-invented, rung metadata +(`tools/build_us_fiscal_refresh_release.py:1696` and `:9514`). + +Regression coverage includes: + +- scaled-only lower-floor behavior and unchanged bands: + `packages/microcosm-build/tests/test_us_prior_year_income.py:439`; +- lower and upper enforcement: `test_us_prior_year_income.py:468`; +- exact rung-1 gate/report bytes: `test_us_prior_year_income.py:486`; +- legacy behavior and malformed metadata: `test_us_prior_year_income.py:510`; +- receipt restoration and top/arm disagreement: + `packages/microcosm-build/tests/test_us_multispine_pool_h5_io.py:1456`; +- nested Boolean/type-alias refusal: `test_us_multispine_pool_h5_io.py:1537`. + +Pre-sampling adjacent-year joins at assembly remain a possible architectural +end state only. This lane did not move the joins or change source assembly. + +## Diff summary + +Before replacing this report, the lane diff comprised 34 files with 2,352 +insertions and 126 deletions. The substantive changes are: + +- pregnancy source, ACS transfer, stacked receipt/authentication, late-producer + structural inputs, execution identity, and their source/transfer/gate/tool + regressions; +- authenticated H5 sampling-receipt restoration and the rung-aware prior-year + gate/report regressions; +- generated US imputation authority, schema/semantic projection, compiler + field-usage and inventory proofs, and the source-attested BE/UK/minimal hash + repins caused by the shared semantic change; +- coverage evidence and the additive optional-ACS provenance golden; and +- changelog fragments + `changelog.d/798-pregnancy-structural-transfer.fixed.md` and + `changelog.d/799-prior-year-rung-floor.fixed.md`. + +## Verification + +All commands used the prebuilt environment with no sync or network access. + +### Complete independent pytest shards + +| Shard target | Result | |---|---:| -| Distinct values | 369 | -| Minimum | 1.0003521955067698 | -| p10 | 2.306610095577396 | -| p25 | 6.226837695984924 | -| Median | 12.319331262236448 | -| Mean | 14.643462963265243 | -| p75 | 23.618846787450813 | -| p90 | 27.706623369972394 | -| Maximum | 37.796501228614694 | - -| Weeks interval | All | ACS clone 1, UC=0 | ACS clone 1, UC>0 | ACS clone 2, UC=0 | -|---|---:|---:|---:|---:| -| [1, 5) | 74 | 70 | 2 | 2 | -| [5, 10) | 85 | 84 | 0 | 1 | -| [10, 15) | 45 | 44 | 0 | 1 | -| [15, 20) | 40 | 36 | 2 | 2 | -| [20, 25) | 60 | 58 | 1 | 1 | -| [25, 30) | 39 | 39 | 0 | 0 | -| [30, 35) | 16 | 15 | 0 | 1 | -| [35, 40) | 10 | 9 | 0 | 1 | - -The nine clone-2 values are -`1.3112775468533615`, `3.155651627690991`, -`8.163243626548894`, `13.03279351472897`, -`15.40528737169489`, `15.477036717133474`, -`24.60208367015022`, `31.658750512333974`, and -`37.10384792802646`. - -### Mechanism verdict: calibration hypothesis refuted - -The late-transfer target bank contains 711 fractional predictions: - -| ACS clone | Fractional before calibration | Fractional in final pool | Rows changed by calibration | -|---:|---:|---:|---:| -| 0 | 342 | 0 | 13,417 | -| 1 | 360 | 360 | 0 | -| 2 | 9 | 9 | 0 | - -Every surviving fractional bit is identical between the target bank and final -pool. The stacked calibration explicitly defines ASEC clone 0 as reference and -ACS clone 0 as recipient in -`packages/microcosm-build/src/microcosm/build/us_runtime/stacked_spine.py:8975-8985`. -Its amount mapping selects values from the sorted observed donor array rather -than interpolating in -`packages/microcosm-build/src/microcosm/build/us_runtime/post_transfer_calibration.py:577-680`. - -The pool receipt confirms the calibration ran: - -- 8,419 allowed-carrier/addition-candidate rows within 856,626 mutable ACS - clone-0 rows; -- 13,417 changed clone-0 rows, including 4,998 cleared and 8,290 added; -- `capacity_limited=true`; -- 8,419 mapped positive amounts; -- reference quantiles `[2, 6, 12, 26, 36]`; -- recipient quantiles `[2, 8, 22, 24, 32]` before and - `[2, 6, 12, 26, 36]` after; -- amount QED `0.5882352941176471 -> 0.0`; -- zero donor-support violations. - -Weeks' declared post-transfer calibration spec is at -`packages/microcosm-build/src/microcosm/build/us_runtime/post_transfer_calibration.py:237-263`. -The actual cause precedes calibration. Before this branch, -`_target_encoding` treated numeric `weeks_unemployed` as continuous because -the engine metadata says `float`; a quantile-regression-forest prediction may -therefore interpolate between integer observations. The relevant encoding and -decoding seam is -`packages/microcosm-build/src/microcosm/build/us_runtime/acs_transfer.py:3037-3120` -and `:3240-3270`. Calibration then repaired only clone 0 by design. - -### Why the gate said “5,218 PUF rows” - -There is no physical PUF channel in this pool. The compatibility helper -`support_role_series` deliberately labels clone 0 as the ASEC-compatible -operator role and every clone above zero as the PUF-tax-detail operator role, -independent of physical source. See -`packages/microcosm-build/src/microcosm/build/us_runtime/support_provenance.py:390-489`. -The old summary confused that operator role with physical channel. - -The 5,218 rows are all ACS-origin, UC-zero, positive-week non-native clones: - -| Actual rows | Integer weeks | Fractional weeks | Total | -|---|---:|---:|---:| -| ACS clone 1 | 4,733 | 355 | 5,088 | -| ACS clone 2 | 121 | 9 | 130 | -| **Total** | **4,854** | **364** | **5,218** | - -The UC-zero constraint is not defined for those ACS non-native QRF clones. -The updated gate preserves the compatibility role for operator plausibility -but uses physical source and clone provenance to own source/UC checks. - -## Task 2: source fix and receipt contract - -`weeks_unemployed` is now in `_DISCRETE_NUMERIC_TARGETS` at -`packages/microcosm-build/src/microcosm/build/us_runtime/acs_transfer.py:133-141`. -The existing discrete codec: - -1. rejects nonintegral donor support; -2. snaps finite predictions to the nearest value in actual observed support; -3. resolves exact-distance ties to the lower support value; -4. emits integer/nullable-integer output. - -The snap is at -`packages/microcosm-build/src/microcosm/build/us_runtime/acs_transfer.py:3327-3339`. -The sorted discrete-target roster is part of the transfer execution identity at -`:271`; the generated authority records it at -`packages/microcosm-build/src/microcosm/build/us/spec/imputation.yaml:408-411`. -The final regenerated bundle spec SHA-256 is -`5f44d96d45e9aabcea2d565ef063d68bfc0652df1b38b08aa31ce6896d15f371`. -This invalidates affected target-bank identities instead of silently changing -content under an old receipt. - -The release-gate and WIC edits also change exact source bytes attested by the -shared legacy seed protocol. Its reviewed WIC grammar now explicitly records -assembled `person_source_id` before the unchanged legacy fallbacks at -`packages/microcosm-build/src/microcosm/build/spec_engine/seeds.py:720-737`. -The fail-closed spec proof was coherently re-pinned to 42,122/42,122 claimed -fields (32,352 authored and 9,770 resolved) and 41/41 inventory checks. The -committed coverage report, producer-semantics digest, seed protocol/map, -checkpoint identity, pointer inventory, and affected country-envelope goldens -all move together. A detached `origin/main` comparison confirmed that the -BE/UK/minimal-envelope changes come from this branch's attested source and seed -grammar, not environment drift. - -The regression at -`packages/microcosm-build/tests/test_us_acs_transfer.py:1391-1430` sends a -fraction-producing mean QRF through the real target codec, verifies that output -lies on observed integer donor support, and verifies that the execution -contract declares `weeks_unemployed`. - -Because the hypothesis was refuted, no change was made to -`post_transfer_calibration.py` or its strict validator. Its observed-support -amount mapping, before/after margin fidelity, hashes, anchors, capacity status, -and donor-support validation remain mandatory; the complete 47-test receipt -contract passes. - -**Artifact consequence:** pool content changes. The host session must rerun -`late_transfer -> simulated -> terminal-gates` from checkpoints for the -candidate pool. This lane did not run any pool or release build. - -## Task 3: stacked-aware weeks release gate - -The gate now uses the narrow, read-only physical-source accessor -`support_gate_source_channel_series` at -`packages/microcosm-build/src/microcosm/build/us_runtime/support_provenance.py:496-524`. -That accessor is restricted to reviewed gates/reporters by the static -source-blindness contract. - -| Frame layout | Plausibility roster | Raw LKWEEKS validity | Exact LKWEEKS carry | UC-zero consistency | -|---|---|---|---|---| -| No provenance | ASEC | all rows | all rows | none, matching legacy behavior | -| Legacy ASEC+PUF roles | ASEC + PUF | ASEC role | ASEC role | PUF role | -| Assembled ASEC+ACS | actual physical channels | every physical ASEC clone | physical ASEC clone 0 | physical ASEC non-native clones + non-ASEC clone 0 | - -The assembled scope logic is at -`packages/microcosm-build/src/microcosm/build/us_runtime/weeks_unemployed.py:1320-1380`; -summary and failure construction are at `:1204-1317` and `:1383-1435`. -The ASEC plausibility bands and the legacy recipient/PUF bands remain -byte-identical, and every non-ASEC physical channel uses the unchanged -recipient band. - -A read-only replay on the supplied pool reports: - -- roster `asec, acs`; -- 234,133 raw ASEC source rows, zero invalid; -- 108,073 direct native reconciliation rows, zero mismatches; -- 982,686 UC-constrained rows, zero mismatches; -- ASEC positive share `0.01956717285126748`, weighted mean - `0.32416674108944377`, weighted weeks `52,743,981.18128455`; -- ACS positive share `0.00800370929739283`, weighted mean - `0.1376988711272782`, weighted weeks `21,987,101.082573153`; -- the unchanged channel bands pass; -- the gate fails only on the genuine 369 noninteger values. - -Stacked ASEC+ACS and legacy ASEC+PUF fixtures cover roster derivation, raw -source scoping, native reconciliation, and UC ownership. The complete focused -weeks file passed: 26 passed, 1 skipped. - -## Task 4: complete release-side archaeology audit - -The audit started from every release call in -`tools/build_us_fiscal_refresh_release.py:8690-11726`, traced each gate and -its immediately preceding producer/wrapper, and classified physical-source, -operator-role, clone-layout, entity-layout, and raw-column assumptions. The -table groups gates only where they share the same architectural conclusion; -every release gate in the roster is named. - -| Gate/stage | Stacked assumption found | Disposition | -|---|---|---| -| Validation input coverage; register consistency; release target parity; target-profile coverage; base-population scale | Manifest, registry, target, or aggregate-weight contracts; no legacy per-row source roster | Already compatible; no change | -| PUF capital-gains-tail presence and post-selection preservation | Authenticated tail-column presence/preservation checks, not a demand for a physical PUF channel (`build_us_fiscal_refresh_release.py:8837-8841,8937-8941`) | Already compatible; no change | -| Exact-k PUF tail support; exact-k frozen-register fit; fiscal-target materialization/skip/zero-support; critical/SOI fit | Calibrated diagnostics and authenticated exact-k inputs, not physical source labels | Already compatible; no change | -| Weeks input and post-selection weeks input | Used clone operator roles as physical ASEC/PUF channels and applied raw/UC checks to the wrong rows | **Fixed** with physical roster and separate validity/reconciliation/UC scopes | -| QBI input | Already has an explicit stacked ACS path | Already compatible; no change | -| Workers' compensation; alimony; Medicare; retirement contributions; retirement distributions | Raw ASEC-only columns were validated/reconciled through legacy operator roles | **Fixed**: all physical ASEC clones own raw validity; native physical ASEC rows own exact direct-carry reconciliation where applicable | -| SSI reporter capture/take-up | Strict `SSI_VAL` validation treated null ACS raw-source cells as invalid | **Fixed**: capture reporters from physical ASEC; assignment remains source-blind and consumes the captured ID set | -| SIPP Head Start; voluntary filing | Assumed exactly an occurrence pair / two clones | **Fixed**: explicit assembly clone index, arbitrary clone count, duplicate source+clone refusal, clone-0-or-lowest canonical decision, fanout to every clone | -| Prior-year income clone diagnostic | Compared only the first two occurrences | **Fixed**: group all assembled clones; existing availability band unchanged | -| SSI disability clone-divergence diagnostic | Compared only a pair and missed clone-2-only divergence | **Fixed** for all assembled clones; whether divergence becomes fatal still requires owner ruling | -| WIC deterministic draw | Raw source-local identity could collide across physical ASEC and ACS | **Fixed**: assembled frames prefer unique `person_source_id`; clones remain draw-stable and legacy precedence is unchanged | -| Farm business; domestic production; child support; disability benefits; educator expenses; Form 4952; SALT refund; capital-gain details; energy subsidy; housing; other health insurance | These use clone-operator roles for reviewed producer/plausibility semantics, not as claims about physical source | Intentionally unchanged; stacked-compatible role diagnostics | -| Childcare; casualty loss; miscellaneous itemized; immigration; generic take-up; hours; SNAP take-up; relationship; eligibility; education; pregnancy; reported-coverage vintage; SNAP discretionary exemption | Output/input signal checks do not assume a physical PUF channel or require raw ASEC cells on all rows | Already compatible; no change | -| Local health input; Medicaid take-up; SNAP state take-up; SSI final/delivery | Output or diagnostics-based checks | Already compatible after SSI reporter fix | -| Input-mass reference; degenerate input; eCPS parity; release input coverage; export input mass; QRF tail concentration; reform-coverage smoke; source coverage | Export-wide schema/mass/tail/simulation contracts, not legacy source rosters | Already compatible; no change | -| Export count-calibrated take-up staleness | Export consistency against recorded count-calibration receipts (`build_us_fiscal_refresh_release.py:11702-11726`), independent of physical source roster | Already compatible; no change | -| SCF wealth | Archived predictors silently map ACS-null CPS race/Hispanic source cells to `Other` | **Owner ruling required**, detailed below | -| SSI disability criteria | Archived receiver strictly requires six ASEC disability predictors and `SSI_VAL` on every row | **Owner ruling required**, detailed below | -| SCF auto loans | Imports the SCF wealth CPS-race mapping, silently mapping ACS recipients to `Other` | **Owner ruling required**, detailed below | -| SIPP vehicles | Missing ACS `SPM_TENMORTSTATUS` defaults every ACS recipient to tenant code 3 | **Owner ruling required**, detailed below | -| SIPP tips | Missing ACS `PEIOOCC` defaults to non-tipped; global band can conceal a dead ACS channel | **Owner ruling required**, detailed below | -| ORG wages/FLSA | Missing ACS race/ethnicity/occupation defaults to zero; unchanged global gates are guaranteed to fail | **Owner ruling required**, detailed below | - -### Unambiguous repairs - -- `support_provenance.py:331-354,496-524` centralizes detection of assembled - metadata and physical channels. General derive/imputation code cannot call - the physical accessor: `test_us_spine_blindness.py` pins the exact reviewed - call graph. -- `alimony.py:282-327` and `workers_compensation.py:560-604` split physical - source validity from direct-carry reconciliation while preserving all role - bands. -- `medicare_take_up.py:134-143,248-270`, - `retirement_contributions.py:172-199,602-628`, and - `retirement_distributions.py:274-301,709-739` scope release diagnostics - without routing their population operators by physical origin. -- `ssi_take_up.py:515-550` captures reporters from physical ASEC and accepts - ACS nulls; the assignment consumes captured lineage rather than reading - origin. -- `sipp_head_start.py:579-621` and - `voluntary_filing.py:749-783` use explicit clone indices on assembled - frames. `prior_year_income.py:744-765` and - `ssi_disability_criteria.py:1119-1152` inspect all clones. -- `wic_claim.py:337-376` prefers the assembly-unique identity only when - assembled metadata exists. - -### Owner rulings required - -1. **SSI disability criteria.** Release call - `tools/build_us_fiscal_refresh_release.py:9840-9847`. The receiver - requires and strictly evaluates all six `PEDIS*` predictors on every row - at - `packages/microcosm-build/src/microcosm/build/us_runtime/ssi_disability_criteria.py:755-859`, - then strictly evaluates `SSI_VAL` at the same file's `:1023-1034`; the - support-role prediction loop is `:965-1005`. Each input is finite on all - 234,133 physical ASEC rows and null on all 1,736,840 ACS rows, so the stage - fails before its gate. - Required decision: transfer/map canonical ACS disability inputs and - reporter-anchor semantics, or revise/retrain the archived recipient model. - -2. **SCF wealth.** Release call - `tools/build_us_fiscal_refresh_release.py:9798-9808`. - `packages/microcosm-build/src/microcosm/build/us_runtime/scf_wealth.py:654-667` - requires `PRDTRACE`, `PRDTHSP`, `A_MARITL`, `PEPAR1/2`, `PH_SEQ`, and - `A_LINENO`; `:670-691` selects the archived reference person, and - `:694-740` builds race/marriage/children predictors. Only - `PRDTRACE/PRDTHSP` are ACS-null; `:611-620` maps those to zero/`Other`. - The other structural fields are finite and mapped on every ACS row. All - three asset leaves are all-null and `net_worth` is absent, so the wrapper's - recompute branch at `:1109-1159` is certain and every ACS recipient is - silently forced to `Other`. Required decision: authoritative ACS - race/Hispanic mapping or transfer, or a reviewed fallback/retrained model. - -3. **SCF auto loans.** Release call - `tools/build_us_fiscal_refresh_release.py:9990-9996`. - `packages/microcosm-build/src/microcosm/build/us_runtime/scf_auto_loans.py:125-137` - requires the archived layout; `:320-362` selects the reference person and - `:365-417` supplies race/marriage/children inputs. `A_LINENO` is required - by the selector's presence check but is not subsequently used; the mapped - structural fields are finite. The actual defect is the imported SCF-wealth - CPS-race mapping of ACS-null `PRDTRACE/PRDTHSP` to `Other`. All three - auto-loan outputs are absent, so `:475-513` certainly takes the QRF path. - Required decision: the same authoritative mapping or reviewed model change - as SCF wealth. - -4. **SIPP vehicles.** Release call - `tools/build_us_fiscal_refresh_release.py:10026-10033`. - `packages/microcosm-build/src/microcosm/build/us_runtime/sipp_vehicles.py:150-157` - declares the archived layout; `:577-622` reads person - `SPM_TENMORTSTATUS` and fills missing values to tenant code 3, while - `:649-763` uses marriage, `A_LINENO` reference selection, and tenure. - The pool's mapped `A_LINENO` is finite, but person - `SPM_TENMORTSTATUS` is ACS-null and the household field is absent. Both - vehicle outputs are absent, so `:872-917` certainly recomputes and - classifies ACS tenure as tenant. Required decision: canonical ACS tenure - mapping/transfer and whether assembled `A_LINENO` preserves the archived - reference-person semantics, or model revision/retraining. - -5. **SIPP tips.** Release call - `tools/build_us_fiscal_refresh_release.py:10103-10109`. - `packages/microcosm-build/src/microcosm/build/us_runtime/sipp_tips.py:249-263` - maps null `PEIOOCC` to tipped code 0; recipient use is `:363-396`, wrapper - branching is `:450-473`, and the gate is `:484-549`. The pool lacks both - tip output columns, so the wrapper takes its imputation branch. - `PEIOOCC` is ASEC-finite and ACS-null. The ASEC-conditional tipped-code - share is `0.0700204513`; ACS is exactly zero; the global share is - nevertheless `0.0353393976`, inside the unchanged `[0.02, 0.15]` band. - This proves that gate component conceals the dead ACS channel, not that the - separate tip-income component necessarily passes. Required decision: - map/transfer ACS occupation or select a reviewed alternate model, then add - channel-aware diagnostics once those semantics are owned. - -6. **ORG wages/FLSA.** Release call - `tools/build_us_fiscal_refresh_release.py:10144-10150`. - `packages/microcosm-build/src/microcosm/build/us_runtime/org_wages.py:535-559` - fills null `PRDTRACE/PRDTHSP/POCCU2` to zero; these enter features at - `:581-625`; wrapper `:901-945` always recomputes; gate `:959-1034`. All - three raw fields are ASEC-finite and ACS-null. On real weights, CPS-race - nonzero share is `0.5047010838` versus unchanged minimum `0.95`, and - detailed-occupation nonzero share is `0.4155259611` versus minimum `0.65`. - The next release is guaranteed to fail here. Required decision: - authoritative ACS race/ethnicity/occupation mapping or a revised model - contract, not threshold weakening. - -### Read-only real-pool replay after the fixes - -The repaired alimony, Medicare, retirement-contribution, -retirement-distribution, workers'-compensation, and SSI-reporter checks pass. -Their physical ASEC source count is 234,133; direct-carry gates use 108,073 -native ASEC rows where applicable. The weeks gate fails only on 369 fractional -values. - -Two other observed failures are genuine data/spec outcomes, not row-label -archaeology: - -- prior-year-income weighted availability is `0.042839`, outside the - unchanged `[0.05, 0.50]` band; -- WIC finds `is_pregnant=true` on nonfemale rows, with example row positions - `129405, 167076, 171133, 192443, 195546`. - -No threshold was changed to hide either outcome. - -## Diff summary by file - -| File | Rationale | -|---|---| -| `PROGRESS.md` | Maintains the required state/done/next journal from kickoff through verified handoff while preserving historical lanes | -| `changelog.d/stacked-release-gate-alignment.fixed.md` | Records the user-visible integer-support and stacked-gate fix under repository convention | -| `out.md` | This provenance, audit, verification, and handoff report | -| `us/spec/imputation.yaml` | Generated authority now receipts `weeks_unemployed` in the discrete numeric transfer roster | -| `us_runtime/acs_transfer.py` | Declares weeks integer-supported through the existing deterministic observed-support codec | -| `us_runtime/support_provenance.py` | Adds assembled-metadata detection and a narrow read-only physical-source accessor | -| `us_runtime/__init__.py`, `us_runtime/puf_support.py` | Re-export assembled-layout detection while keeping the physical accessor confined to its owner | -| `us_runtime/weeks_unemployed.py` | Derives stacked/legacy rosters and separates source validity, native reconciliation, and UC scopes | -| `us_runtime/alimony.py`, `us_runtime/workers_compensation.py` | Scope raw-source and exact-carry checks to their physical/native owners | -| `us_runtime/medicare_take_up.py`, `us_runtime/retirement_contributions.py`, `us_runtime/retirement_distributions.py` | Make release summaries source-aware while retaining source-blind producer kernels | -| `us_runtime/ssi_take_up.py` | Captures SSI reporters from physical ASEC and permits null ACS source cells without origin-routing assignment | -| `us_runtime/sipp_head_start.py`, `us_runtime/voluntary_filing.py` | Replace pair/occurrence layouts with explicit arbitrary-clone assembled layouts | -| `us_runtime/prior_year_income.py`, `us_runtime/ssi_disability_criteria.py` | Detect divergence across all assembled clones | -| `us_runtime/wic_claim.py` | Uses assembly-unique person identity for cross-origin deterministic draws | -| `spec_engine/seeds.py` | Receipts assembled `person_source_id` as WIC's first seed-key source | -| `spec_engine/field_usage.py`, `spec_engine/inventory_coverage.py`, `tools/spec_engine_coverage.py` | Re-pin the exact two-field expansion and downstream producer/seed/checkpoint/pointer identities | -| `docs/evidence/spec-engine/us-f0-coverage.json` | Regenerates the closed 42,122-field, 41-item coverage attestation | -| `tests/test_us_acs_transfer.py` | Proves fractional QRF output is snapped to observed integer weeks support and receipted | -| `tests/test_us_weeks_unemployed.py` | Covers stacked ASEC+ACS and legacy ASEC+PUF roster/scope contracts | -| `tests/test_us_alimony.py`, `test_us_workers_compensation.py`, `test_us_medicare_take_up.py`, `test_us_retirement_contributions.py`, `test_us_retirement_distributions.py`, `test_us_ssi_take_up.py` | Cover physical ASEC source validity, native reconciliation, legacy behavior, and SSI reporter lineage | -| `tests/test_us_sipp_head_start.py`, `test_us_voluntary_filing.py`, `test_us_prior_year_income.py`, `test_us_ssi_disability_criteria.py` | Cover clone 2+, duplicate clone refusal, deterministic canonical rows/fanout, and all-clone divergence | -| `tests/test_us_wic_claim.py` | Covers cross-origin identity collision avoidance and clone-stable draws | -| `tests/test_us_multispine_puf_clone.py`, `test_us_spine_blindness.py` | Cover assembled provenance and pin physical-source access to the reviewed gate/reporter call graph | -| `tests/test_spec_engine_seeds.py` | Pins WIC's assembled-first seed grammar | -| `tests/test_spec_engine_field_usage.py`, `test_spec_engine_coverage_tool.py`, `test_spec_engine_country_bundles.py`, `test_spec_engine_loader.py` | Pin the reviewed field totals, report identities, and source-attested envelope goldens | -| `tests/test_us_multispine_pool_tool.py` | Re-pins the constants adapter's live US spec identity after regeneration while retaining the separate arbitrary checkpoint identity fixture | - -All source paths in the table are under -`packages/microcosm-build/src/microcosm/build/`; all abbreviated test paths -are under `packages/microcosm-build/tests/`. - -## Judgment calls - -- The post-transfer calibration was not modified merely because its receipt is - visible near the symptom. Checkpoint bits prove the surviving values predate - it, and its mapping already uses actual donor support. -- The integer contract is explicit by target rather than inferred from the - current PolicyEngine physical dtype. That dtype is `float` and caused the - bug; reviewed domain semantics are the authority. -- Snapping to observed donor support was preferred over generic rounding. It - cannot invent an unsupported week count, is deterministic, and reuses an - existing receipted codec. -- Physical source identity is exposed only to read-only gates/reporters. - Medicare and retirement producer changes considered during review were - reverted: routing population treatment by origin would violate the - source-blind operator boundary. The authenticated release path consumes - already-produced pool surfaces. -- Raw validity covers all physical ASEC clones because they carry the raw - source. Exact reconciliation covers only native ASEC where transferred - non-native clones are intentionally allowed to differ. -- Operator roles remain appropriate for producer/plausibility bands. They are - not aliases for physical ASEC/ACS source, which was the old weeks bug. -- Clone divergence in SSI disability remains diagnostic rather than newly - fatal; changing that policy needs an owner ruling. -- The six archived-model cases were not “fixed” with zero fills, gate - rescoping, or weaker thresholds. Each requires substantive decisions about - ACS predictors and model semantics. - -## Verification evidence - -All commands ran offline against the prebuilt environment, with -`UV_CACHE_DIR=/private/tmp/microcosm-weeksgate-uv-cache` so the required -`uv run --no-sync` command could operate inside the sandbox. Each pytest -shard ran in one independent process. - -- `uv run --no-sync ruff check .`: PASS, `All checks passed!` -- `uv run --no-sync python tools/ci_test_groups.py --verify`: PASS, +| `pytest packages/microcosm-build/tests` | 6,601 passed, 45 skipped | +| `pytest packages/microcosm-calibrate/tests` | 203 passed | +| `pytest packages/microcosm-data/tests` | 318 passed, 2 skipped | +| `pytest packages/microcosm-fit/tests` | 93 passed | +| `pytest packages/microcosm-frame/tests` | 295 passed, 36 skipped | +| **Aggregate** | **7,510 passed, 83 skipped** | + +The clean final build-shard process took 55m20s and exited 0. Its 2,351 +warnings are preexisting overflow, pandas chained-assignment/fragmentation, +PolicyEngine division, and donor-snap warnings; there were no test errors. + +Focused evidence also passed: all 23 prior-year tests and all 64 current H5 +loader tests (the initial 60-test run plus four later nested-type adversarial +cases), the complete pregnancy and ACS-transfer files, integrated stacked +execution, forged-policy controls, the complete optional-ACS multispine file, +and the country-bundle/loader/spec matrix. + +### Static, generated, and workspace gates + +- `uv run --no-sync ruff check .`: `All checks passed!` +- `uv run --no-sync python tools/ci_test_groups.py --verify`: `tracked_test_files=309`, `verification=ok` -- Eight directly affected spec/receipt files: PASS, 102 tests -- `tools/generate_us_bundle_from_constants.py --check`: PASS, final US spec - SHA-256 - `5f44d96d45e9aabcea2d565ef063d68bfc0652df1b38b08aa31ce6896d15f371` -- `tools/spec_engine_coverage.py --check`: PASS, 42,122/42,122 fields and - 41/41 inventory checks -- `uv run --no-sync pytest packages/microcosm-calibrate/tests -q`: PASS, - 203 passed -- `uv run --no-sync pytest packages/microcosm-data/tests -q`: PASS, - 318 passed, 2 skipped -- `uv run --no-sync pytest packages/microcosm-fit/tests -q`: PASS, - 93 passed -- `uv run --no-sync pytest packages/microcosm-frame/tests -q`: PASS, - 295 passed, 36 skipped -- `uv run --no-sync pytest packages/microcosm-build/tests -q`: PASS, - 6,608 tests collected, 100% reached, exit code 0, expected skips only -- Complete ACS-transfer file: PASS, 65 tests -- Complete post-transfer calibration receipt-contract file: PASS, 47 tests -- Complete weeks file: PASS, 26 passed, 1 skipped -- Complete alimony/workers'/SSI files: PASS, 28 / 21 / 71 tests -- Medicare + retirement contribution/distribution focused files: PASS, 65 tests -- Repository `git diff --check`: PASS - -The shard warnings observed are existing numerical, sparse-tensor, -joblib core-detection, and PolicyEngine divide warnings; none is a failure and -none originates in the new transfer/source-scope paths. - -## Commit inventory - -Implementation and evidence commits preceding the final report carrier: - -1. `a3331db6` Start stacked release gate alignment journal -2. `9979d101` Record fractional weeks provenance -3. `d7ad753e` Preserve integer support for transferred weeks -4. `0c5b05a8` Receipt integer weeks transfer support -5. `c22a6799` Align weeks gate with stacked source roles -6. `9bbd6dfe` Centralize stacked source channel scopes -7. `49052e6e` Scope alimony source checks to physical ASEC -8. `17ed9a0b` Key WIC draws by stacked person identity -9. `c7dd53ad` Support stacked Head Start clone layouts -10. `3f934688` Scope Medicare source gate to stacked ASEC -11. `5852f4ba` Scope retirement contribution source diagnostics -12. `e40d5db7` Support stacked voluntary filing clones -13. `148881c6` Check all stacked prior-year income clones -14. `52b27f78` Separate alimony source validation and carry scopes -15. `058f27b5` Record stacked clone-layout repairs -16. `97de22e7` Scope workers compensation sources in stacked pools -17. `74e96187` Scope retirement distribution source gates -18. `41007992` Anchor SSI reporters to physical ASEC sources -19. `eb5cd4b6` Keep SSI assignment source blind -20. `3dc5df51` Keep stacked source repairs gate scoped -21. `9b36720b` Record stacked raw-source gate repairs -22. `9520c03f` Report divergence across all SSI disability clones -23. `b8e04e9c` Confine physical source access to release gates -24. `b2de92b9` Document stacked release gate alignment -25. `cd8c55a7` Repin spec engine for integer weeks -26. `12a918ed` Repin multispine live spec fixture - -The final report/journal carrier follows this inventory and is necessarily -self-excluded; the clean-worktree handoff lists it in the final response. +- `uv run --no-sync python tools/generate_us_bundle_from_constants.py --check`: + `US bundle spec_sha256=11e310c7619cbac91f6703b9679649cdd15f6fb09274ad29904c65881aa93316` +- `uv run --no-sync python tools/spec_engine_coverage.py --check`: + `42154/42154` configuration fields and `41/41` inventory checks +- `git diff --check 606cbd69..HEAD`: passed +- Untracked files and local `.gitnexus` directories: zero + +## Judgment calls and boundaries + +- Missing values on ineligible recipients are constructed as false and + receipted; preexisting invalid true values are refused with counts. This + preserves valid existing data without silently repairing corrupted input. +- One eligible clone-0 representative owns the random draw. Every missing clone + of the source person receives the same result, while valid existing values + are preserved, eliminating clone-specific random pregnancy states by + construction. +- Boolean structure is exact `0/1`; near-Boolean floats do not pass through + numerical tolerance. Nested manifest fractions, seeds, and counts likewise + reject JSON Boolean equality aliases. +- A complete pregnancy surface cannot bypass structural preflight during a + restart and carries a zero-imputation proof. +- The prior-year gate uses the authenticated configured rung as the + sampled-to-sampled survival factor. It does not substitute a noisy empirical + estimate, change the authored 0.05 constant, relax the upper bound, or touch + another band. +- Legacy/no-rung and full-rung behavior remains unchanged. At rung 1.0 the + gate report is byte-identical to the old output. +- Joining adjacent raw ASEC years before sampling is documented only as a + future architectural end state, per the ruling; it is not implemented here. + +## Commit inventory before this report carrier + +1. `b7be9c35` — Start pregnancy and prior-year defect journal +2. `364f4525` — Record pregnancy and prior-year root causes +3. `6d3351fd` — Record prior-year release-gate ruling +4. `f5284a07` — Scale prior-year availability gate by sampling rung +5. `4aa6269a` — Record prior-year gate implementation +6. `01a80f49` — Enforce structural pregnancy transfer eligibility +7. `e59ab046` — Harden pregnancy and rung receipts +8. `d2b75e1e` — Repin shared spec attestations +9. `7caf69ac` — Repin ACS transfer provenance fixture +10. `011563db` — Record completed provenance repin +11. `c1a4bfb9` — Record full shard verification + +## Host-owned next action + +Rebuild the affected pregnancy transfer from the invalidated checkpoint and +target bank, then run the terminal pool/release gates on the rebuilt artifact. +The prior-year gate will read the existing authenticated sampling receipt and +report the applied factor/floor. Those build, certification, publication, and +push actions are deliberately outside this headless lane.