diff --git a/PROGRESS.md b/PROGRESS.md index 32ab6f09e..c6c9f696b 100644 --- a/PROGRESS.md +++ b/PROGRESS.md @@ -1,65 +1,371 @@ -# Progress: historical scorer columns +# ACS predictor release join -## State +> **Historical note (2026-08-28).** This journal describes the +> `acs-predictor-release-join` lane as of 2026-08-27. The branch has since +> been merged into the `stacked-release-fix-train` integration branch +> together with the #794 gate-alignment and #798/#799 pregnancy/prior-year +> fixes, with spec envelope digests and coverage evidence regenerated over +> the union tree. Treat the "State"/"Next" sections below as history; +> check git/GitHub for current truth. -Final lane state as of 2026-08-26: implementation and complete workspace -verification are green. The real acceptance invocation cleared the original -incumbent formula-column defect and completed all incumbent scoring, then -stopped at a separate authenticated candidate-manifest failure: -late primary-QRF worker binding changed. No scorecard files were written. +## State -The goal is to make head-to-head scoring deterministically remove and receipt -formula-owned columns from loaded historical incumbent or candidate H5 -artifacts, while keeping the fresh-release leaf-only export gate strict. +Complete on 2026-08-27. The owner-approved release-time join from stacked-pool +ACS source lineage to the SHA-pinned 2024 one-year ACS person/household zips is +implemented, receipted, real-pool exercised, and fully verified. It populates +the six archived donor models' CPS-named predictors through reviewed +native-ACS crosswalks with strict hash, lineage, collision, totality, universe, +and clone-fan-out contracts. Model selection logic and every gate threshold +remain unchanged. The completed evidence and handoff are in `out.md`. No +network access, pool build, release build, publication, push, retraining, +threshold change, or launcher-contract edit occurred. ## Done -- Read the repository operating instructions and confirmed the worktree starts - clean on `scorer-legacy-incumbent-columns`. -- Confirmed the requested defect boundary: historical scorer inputs only; no - pool build, release build, publishing, push, or validation bypass is in scope. -- Started this committed lane journal before implementation work. -- Reused the release gate's cached, period-sensitive - `PolicyEngineUSVariableMetadataIndex` classification at the scorer loading - boundary without changing the builder gate. -- Added fail-closed dependency-closure checks on each dropped output, including - entity-correct leaf presence, before removing any artifact column. -- Added schema-3 JSON and Markdown receipts with deterministic total count and - sorted column names per entity; clean artifacts seal an explicit empty - receipt. -- Preserved tables, weights, strata, mass log, and frame metadata across the - scorer-only normalization. -- Added the changelog fragment and three requested H5 tests: drop-and-score, - missing-leaf refusal, and clean empty receipt. The full targeted scorer test - file passes (16 passed). Its first explicit receipt run exposed and fixed an - empty-entity pandas drop edge before full-shard verification. -- Passed repository-wide Ruff through the required `uv run --no-sync` command - using a writable offline cache, and passed CI test-inventory verification for - all 309 tracked test files. -- Passed all five full pytest shards in separate processes: build 6,516 passed / - 45 skipped; frame 295 / 36; calibrate 203 / 0; data 318 / 2; fit 93 / 0. - Aggregate: 7,425 passed, 83 skipped, 0 failed. -- Ran the exact acceptance scorer from this worktree. It loaded the live - incumbent, dropped and receipted one person column - (has_marketplace_health_coverage), scored all five incumbent chunks and 12 - household slices per chunk, and released that state at 19.20 GiB peak RSS. -- Captured the complete subsequent traceback in out.md. Candidate - authentication refused the pool manifest before its H5 loaded, so the run - correctly stopped without bypassing the late-producer binding or emitting a - partial JSON/Markdown scorecard. +- Read `CLAUDE.md` and the prior weeksgate report's six owner-ruling items with + their release-call and model-consumer evidence. +- Confirmed the requested branch `acs-predictor-release-join` is clean at + `606cbd69`, based on `stacked-release-gate-alignment`. +- Read the GitNexus exploration and impact-analysis workflows. This workspace + exposes neither GitNexus repository resources nor query tools, so the same + call/dependency analysis will be performed directly from source and tests. +- Recorded the required source zips and SHA-256 pins, strict exact/total join + contract, explicit crosswalk and receipt requirements, and verification + boundary. +- Proved that `person_source_id` is not a reversible ACS key: ACS people are + sorted by `(SERIALNO, SPORDER)`, receive a zero-based raw spine ID, and then + receive a collision-dependent assembly offset. The pool retains the raw + spine ID, `source_row_id`, `source_person_id`, household `SERIALNO`, and clone + metadata, so the release join will use the retained semantic + `(SERIALNO, integral SPORDER)` key and treat `person_source_id` only as the + one-to-many clone fan-out identity. +- Audited the supplied candidate pool read-only: 856,626 distinct ACS source + people expand to 1,736,840 rows (856,626 clone 0, 856,626 clone 1, and + 23,588 clone 2), with no duplicate `(person_source_id, clone_index)` pair. + Every ACS row agrees with its raw spine/source lineage, and all selected + people match the pinned raw person archive exactly. +- Verified both local archives against the charter pins. The person archive + has 3,422,888 unique `(SERIALNO, SPORDER)` rows and no household orphans; + the household archive has 1,631,969 unique serials, including 1,531,614 + occupied records. Both contain every requested native predictor. +- Established the disability universes from the pinned archive and the + archived repository mapping: DEAR/DEYE are complete at every age; + DREM/DPHY/DDRS are asked from age 5; DOUT from age 15; native code 1 is the + consumer's difficulty bin and code 2 (plus an age-valid universe blank) is + its non-difficulty bin. +- Established the consumed race/Hispanic bins: both SCF models distinguish + White, Black, Asian, Hispanic, and Other; ORG distinguishes Hispanic, + non-Hispanic White, non-Hispanic Black, and Other. `RAC1P`/`HISP` can map + exactly to those bins without inventing detailed CPS combinations. +- Recovered the complete 2024 Census detailed-occupation-to-`POCCU2` consumed + grouping from the native ASEC relationship and confirmed that ACS `OCCP` + uses the same detailed codes. `PEIOOCC` is therefore a direct carry, while + `POCCU2` will use an explicit reviewed 53-bin table; blank out-of-universe + occupation maps to code 0, military to 52, and code 9920 to 53. +- Confirmed ACS `TEN` maps to the SPM vehicle model's three consumed tenure + bins (mortgaged owner, outright owner, non-owner); no-cash-rent and verified + group-quarters blanks belong to the non-owner bin. Confirmed the SSI model's + `SSI_VAL` use is only the `> 0` reporter anchor and that native ACS `SSIP` is + already carried as harmonized `ssi_reported`, observed exactly from age 15. +- Added the dedicated `acs_release_predictors` release boundary. It verifies + the two canonical archive pins before opening either zip, streams only + selected households, validates exact archive members and headers, rejects + raw/person/clone collisions, binds retained pool lineage to + `(SERIALNO, SPORDER)`, requires total one-to-one source-person matching, and + fans mapped values to clones only through `person_source_id`. +- Added explicit disability, race/Hispanic, 530-code occupation, and tenure + tables. A canonical crosswalk payload is pinned at SHA-256 + `1d4906242e9c73e31b3283659e5cad8242b8cbc42914ab6fa59547a10c8770e9` + and rides the JSON-ready join receipt with per-model/per-predictor + ASEC-native, ACS-joined, and still-null counts. +- Preserved CPS disability universe semantics (`-1` below the question age) + and the ACS occupation universe. Blank `PEIOOCC` uses the CPS NIU sentinel + `-1`; blank `POCCU2` remains 0 through age 15 and maps to the consumed + no-occupation code 53 only from age 16. This explicitly preserves the + one-year ACS/CPS source-universe gap instead of assigning every ACS + 15-year-old a never-worked status without source evidence. The explicit + occupation table covers every one of the 530 codes in the pinned ACS person + archive and every consumed POCCU2 bin. +- Changed the SSI-disability reporter read, without source routing, to + row-wise coalesce measured ASEC `SSI_VAL` with harmonized native ACS + `ssi_reported`. Adult blanks and conflicting dual reporters fail; genuine + below-age-15 ACS blanks remain null in the frame and become false only for + the consumer's `> 0` predicate. +- Hardened the join after independent crosswalk review: raw ACS `SSIP` and + `ADJINC` now travel through the pinned join and must agree exactly with every + native clone-0 `ssi_reported` value under the established adjusted-dollar + formula. Raw `ESR`/`OCCP` must obey their exact age-16 universes, and all + ASEC predictor receipt cells must be numeric and finite with complete, + nonnegative `SSI_VAL`. +- Updated the SSI signal diagnostic to use the same row-wise reporter coalesce + as the model consumer, while retaining the archived native-role anchor + scope. A lost positive ACS-native reporter can therefore no longer evade the + release gate merely because `SSI_VAL` is null on physical ACS rows. +- Added focused tests for crosswalk identity/all consumed bins, exact join and + clone invariance, ASEC byte preservation, receipt contents, missing joins, + raw and source-identity collisions, hash refusal, no-ACS identity, and SSI + coalescing/universe refusal. Coverage now also fixes the age-15 occupation + gap, malformed ESR refusal, malformed ASEC SSI refusal, raw SSI attestation, + and gate-side ACS reporter preservation. The complete join, SSI, and + source-blindness test files pass together, and focused Ruff is green. + +- The release CLI now accepts the person/household zip and lowercase 64-hex + SHA-256 options as an all-or-none set. It invokes the authenticated join + after the last unrelated native-input gate and before SCF wealth, therefore + before all six archived donor-model stages, then carries the complete join + receipt into both `build_manifest.json` and `release_manifest.json`. +- Added parser refusal tests, a source-order contract over all six model calls, + an end-to-end mocked main corridor that verifies the exact four join + arguments and runtime ordering, an AST contract that binds the saved receipt + to the sole manifest call, and JSON round-trip assertions for both manifests. + Focused Ruff, five parser/order/manifest cases, and all six parametrized main + corridor cases pass. +- Exercised the hardened join read-only on the complete supplied candidate. + The 3,239,263,147-byte H5 matches its frozen manifest SHA-256 + `871b7e6467675a1e9475b54fd1baf64c53c0f75a3258b8357303a8df0d53642d`. + The current official loader refuses that older candidate before H5 loading + because its archived primary-QRF worker binding predates this branch's + execution identity; this is an existing candidate/code-version mismatch. + Loading those independently manifest-hash-verified bytes with their frozen + assembly receipt allowed the join boundary itself to be tested without + writing an artifact. +- The real join passed every source, raw-key, universe, SSI-attestation, + totality, collision, and clone-fan-out check: 856,626 unique ACS source + people matched 856,626 raw people in 382,903 households and populated + 1,736,840 support rows (856,626 each at clone indices 0 and 1, plus 23,588 + at clone index 2). Every CPS-named predictor consumed by the six models has + 234,133 valid ASEC-native and 1,736,840 ACS-joined cells with zero nulls. + The logical SSI reporter anchor has 234,133 ASEC cells, 1,475,235 observed + ACS cells, and exactly 261,605 preserved child-universe null support rows. +- The first real-data attempt exposed fixed-format HDF's expected object dtype + for mixed-source columns. Tightened the ASEC validator to inspect each cell, + accepting object-wrapped real numbers while still refusing strings, + nonfinite values, nulls, and negative SSI. A focused H5-shape regression and + all 15 join tests pass before the successful full-pool rerun. +- Added `changelog.d/acs-release-predictor-join.fixed.md`, describing the + pinned release join, reviewed mappings, fail-closed lineage, dual-manifest + receipts, and unchanged model/gate behavior. +- Repository Ruff passes, and the CI inventory verifier reports 310 tracked + tests with `verification=ok`. Four complete pytest shards pass in separate + processes: frame 295 passed/36 skipped, fit 93 passed, calibrate 203 passed, + and data 318 passed/2 skipped. +- The first complete build-shard process reached 100% with 6,575 passed and 45 + skipped, plus five failures and six fixture errors. All eleven were the same + expected source-attestation drift: `ssi_disability_criteria.py` belongs to + both the direct and QRF seed-kernel inventories, so this task's runtime edit + moved the seed protocol, compiled US seed map, every country spec identity, + the minimal loader golden, and the generated coverage evidence. No ACS join, + release CLI, manifest, archived-model behavior, or gate test failed. +- Applied the repository's established five-file source-identity repin only: + seed protocol `59a098f9...31d8b`, US seed map `ce3850d8...e42ab`, US spec + `16b7d5e6...dca38`, UK spec `2f921e4c...33a62`, BE spec + `c87a0012...34ba`, and minimal-loader golden `b4946105...f2af`; regenerated + `docs/evidence/spec-engine/us-f0-coverage.json`. All 25 affected cases and + focused Ruff pass. The US bundle generator `--check` passes at the new spec + identity, and coverage `--check` passes at 42,122/42,122 fields and 41/41 + inventory checks. +- Re-ran the complete build shard after the reviewed repin: 6,586 passed and + 45 skipped, with exit code 0. Re-ran final repository Ruff, the 310-file CI + inventory verifier, both retained spec `--check` commands, and + `git diff --check`; all pass. Wrote the required final report to `out.md`. ## Next -1. Resolve or rebuild candidate-25 so its sealed primary-PUF-QRF worker binding - matches the current authenticated late-producer contract. -2. Rerun the unchanged acceptance scorer command; do not weaken or bypass pool - authentication. +- No work remains in this lane. The dispatcher owns rebasing and the launcher + contract update. A future authorized build must produce a pool whose current + source-attested worker identity passes the official release loader; the + supplied older candidate is useful join evidence but cannot be promoted. -## Historical prior lane +# Weeksgate: stacked release gates and integer-week provenance + +## State + +Complete on 2026-08-27. Real-pool provenance has refuted the proposed +post-transfer amount-mapping mechanism: every fractional week is an ACS-origin +non-native clone prediction outside the calibration's clone-0 recipient scope. +The source codec, weeks-gate architecture, source-scope, clone-layout, and +stable-identity repairs are implemented and focused-tested. The complete +release-call roster is classified; six archived-model input assumptions require +owner rulings and are deliberately reported instead of guessed. Repository-wide +Ruff, the CI inventory verifier, and all five full pytest shards pass in their +required independent processes. The completed provenance, audit, verification, +judgment calls, and host-owned checkpoint-rerun consequence are in `out.md`. +No network access, artifact build, publication, push, pool build, or release +build is in scope. + +## Done + +- Read `CLAUDE.md` and the GitNexus debugging workflow. +- Confirmed branch `stacked-release-gate-alignment` is clean at `4f453746`. +- Confirmed the local GitNexus CLI is installed but the repository is not yet + indexed. Its offline analyzer parsed the repository but could not register + the index because the sandbox forbids writes to `~/.gitnexus`; the generated + local index was moved out of the worktree to `/private/tmp`. +- Recorded the four requested workstreams: fractional-week provenance and PUF + misclassification; integer-support calibration repair; stacked/legacy weeks + gate alignment; and the full release-side gate archaeology sweep. +- Recorded the required verification boundary: repository Ruff plus one pytest + process per shard, with no pool/release builds. +- Read the fixed-format HDF5 blocks directly and classified all 369 noninteger + `weeks_unemployed` rows: 360 are ACS clone 1 (355 UC=0, 5 UC>0) and 9 are ACS + clone 2 (all UC=0); all are positive, all 369 values are distinct, and the + exact range is 1.0003521955067698--37.796501228614694. +- Confirmed zero nonintegers on ASEC rows and ACS clone 0. The receipted + calibration covers exactly the 856,626 ACS clone-0 rows, maps 8,419 carrier + amounts onto observed ASEC support with zero donor-support violations, and + records QED 0.5882352941176471 to 0.0. +- Reproduced the 5,218-row false "PUF" classification: the legacy role helper + calls every clone index above zero `puf_tax_detail`, regardless of raw source + channel. The rows are all ACS-origin clones: 4,733 integer clone-1 rows, 355 + fractional clone-1 rows, 121 integer clone-2 rows, and 9 fractional clone-2 + rows with nonzero weeks while UC is nonpositive. +- Traced the actual fractional mechanism to the ACS transfer target codec: + PolicyEngine-US declares `weeks_unemployed` as physical `float`, so the + generic QRF path treats it as continuous even though its reviewed source + contract is integer-supported. The later calibration repairs clone 0 only. +- Bound every final fractional value bit-for-bit to the late-transfer target + bank. Its raw QRF draw contains 711 nonintegers: 342 ACS clone 0, 360 clone 1, + and 9 clone 2. Post-transfer calibration changes 13,417 clone-0 rows and + eliminates all 342 clone-0 fractions; the 369 out-of-scope clone fractions + pass through unchanged. +- Added `weeks_unemployed` to the ACS transfer's authority-bound discrete + numeric target registry. The existing codec now snaps every prediction to + actual observed ASEC donor support, and the execution-contract identity + receipts the policy alongside the two mortgage-year targets. +- Added a focused ACS-transfer regression proving weeks predictions retain + observed integer support and the execution contract declares the target. +- Regenerated the checked-in US imputation authority so + `transfer_execution.discrete_numeric_targets` includes `weeks_unemployed`. + The generator's compile and byte-staleness checks pass at bundle spec SHA + `821d5838da3ac368170e61e017f1a72648f93e8a011aa40e33b8c2b4b14511f3` + at that source-fix step; the later runtime/seed identity repin below + supersedes this intermediate digest. The spec-bundle and imputation- + semantics suites pass. +- Passed the complete ACS-transfer test file (65 tests), the complete + post-transfer calibration receipt-contract file (47 tests), focused Ruff, + and `git diff --check` using the prebuilt `.venv` directly. A task-local + `UV_CACHE_DIR` later allowed the required `uv run --no-sync` commands to run + against that same prebuilt environment without accessing `~/.cache/uv`. +- Modernized the weeks summary/gate to derive its roster from actual assembled + source channels, while retaining the legacy ASEC/PUF role path. The ASEC + source-validity scope, direct native-clone reconciliation scope, and reviewed + UC-constraint scope are now distinct and explicitly receipted in details. +- Kept all four plausibility-band tuples and every numeric threshold unchanged; + non-ASEC assembled channels use the unchanged legacy recipient band. +- Added stacked ASEC+ACS and legacy ASEC+PUF fixtures covering roster detection, + raw-source scoping, native reconciliation, and UC constraint ownership. The + complete weeks file passes (26 passed, 1 skipped) with focused Ruff. +- Replayed the updated gate over the supplied pool's exact live arrays and + weights. It now reports 234,133 valid ASEC source rows, 108,073 exact native + reconciliation rows, 982,686 UC-constrained rows, zero source/UC failures, + both unchanged channel bands passing, and only the genuine 369 nonintegers. +- Added a centralized provenance-owner API that distinguishes validated + physical source channels from legacy clone-operator roles. The weeks gate + now consumes that API rather than reading provenance columns directly, and + both repository source-blindness tripwires pass alongside the stacked and + legacy provenance/weekly-signal suites. +- Made WIC's deterministic draw key prefer the assembly-unique + `person_source_id` on multispine frames, before the source-local raw identity + triple. Distinct ASEC/ACS records can no longer collide when their vintages + align, while all support clones of one assembled person retain the same draw + and the legacy key order remains unchanged. +- Completed the release-tool call-roster audit. It found unambiguous physical + ASEC scoping repairs in SSI take-up, workers' compensation, alimony, + retirement contributions/distributions, and Medicare; clone-2 layout fixes + in Head Start, voluntary filing, and prior-year income; and a stable-key fix + in WIC. Archived ASEC-only predictor assumptions in SSI disability, SCF + wealth, SCF auto loans, and SIPP vehicles require explicit model-owner + rulings and will be reported rather than guessed. +- Reworked the assembled Head Start and voluntary-filing receiver layouts to + key by assembly-unique source ID plus explicit clone index, accept clone 2 + and later, reject duplicate source/clone rows, choose clone 0 (or the lowest + surviving clone) deterministically, and fan one source-level decision to all + clones. Their legacy role-only duplicate contract remains unchanged. +- Replaced occurrence-pair clone diagnostics in voluntary filing and + prior-year income with all-clone grouping on assembled source IDs, so a + clone-2-only divergence is now detected. Focused tests and Ruff passed for + all three module/test pairs. +- Scoped Medicare and retirement release diagnostics to physical ASEC source + rows, separating raw-source validity from native direct-carry reconciliation + where transferred clones intentionally differ. Kept the producer kernels + origin-blind: the authenticated-pool release path skips those producers, and + indirect physical-source routing would violate the repository's population- + operator boundary. Their 60 focused module/source-blindness tests and Ruff + pass after that review correction. +- Scoped alimony and workers' compensation raw validity to every physical ASEC + clone and exact source-carry checks to physical ASEC native rows, leaving all + clone-operator plausibility bands unchanged. Stacked ASEC+ACS and legacy + ASEC+PUF fixtures pass (28 alimony and 21 workers' compensation tests). +- Changed SSI reporter-lineage capture to validate `SSI_VAL` only on physical + ASEC rows and accept null ACS raw-source cells. Assignment remains source- + blind: it consumes the source-ID set captured before L0, or obtains that set + through the reporter helper when no explicit set is supplied. The complete + SSI take-up file passes (71 tests). +- Fixed SSI-disability's non-fatal clone-divergence diagnostic to group every + assembled clone by source person, so clone-2-only divergence is reported. + The existing decision not to make divergence gate-fatal remains unchanged + for an owner ruling; the complete focused file and source-blindness checks + pass. +- Narrowed physical-channel resolution to a gate/reporter-only provenance API, + removed its general runtime/PUF-support re-exports, and added a static exact- + caller contract (including internal mask-helper callers). A future derive, + impute, or wrapper use now fails the source-blindness suite instead of + passing through indirection. +- Replayed the repaired release gates read-only against the supplied pool. + Alimony, Medicare, retirement contributions/distributions, workers' + compensation, and SSI reporter capture pass; the weeks gate now fails only + on the genuine 369 fractional values. Prior-year income remains outside its + unchanged availability band and WIC finds pregnant nonfemale rows, both + genuine data/spec outcomes rather than stacked-layout archaeology. +- Completed an adversarial review of the repaired code and focused tests with + no additional implementation defect found. It confirmed six owner-ruling + items: SSI disability criteria, SCF wealth, SCF auto loans, SIPP vehicles, + SIPP tips, and ORG wages/FLSA all consume ASEC-only archived predictors on a + frame whose 1,736,840 physical ACS rows carry null source cells. ORG is + guaranteed to fail its unchanged race/occupation bands; SIPP tips' unchanged + tipped-occupation band passes while the ACS channel is dead. +- Passed repository-wide Ruff and the CI test-group inventory verifier. The + calibrate, data, fit, and frame shards pass in four independent pytest + processes. +- Corrected the reviewed WIC seed protocol to match the implemented assembled- + multispine key precedence: assembly-unique `person_source_id` first, then the + unchanged legacy raw/support/person fallbacks. An exact seed-grammar test now + binds that order. +- Re-pinned the fail-closed spec-engine proof after adding one authored + transfer-execution field and one resolved seed-protocol field: 42,122 total + fields (32,352 authored and 9,770 resolved), complete exact-pointer claims, + and all 41 inventory checks. Regenerated the committed coverage report and + validated the final US spec SHA + `5f44d96d45e9aabcea2d565ef063d68bfc0652df1b38b08aa31ce6896d15f371`. +- Verified in a detached `origin/main` worktree, using the same prebuilt venv, + that the old BE, UK, and minimal-spec golden vectors still pass there. Their + current repins therefore reflect this branch's attested runtime and seed- + protocol changes rather than environment drift. All 102 tests in the eight + directly affected spec-engine files pass; generated-bundle and coverage- + report byte checks, focused Ruff, and `git diff --check` also pass. +- Ran the full build shard after that coherent spec repin. It reached 100% with + exactly one failure and no errors: the multispine constants-adapter fixture + still expected the former live US spec SHA. Updated only that live-binding + expectation to the regenerated final SHA; the separate arbitrary checkpoint + identity fixture remains deliberately unchanged. +- Passed the complete multispine-pool-tool file after that correction and + committed the coherent fixture repin as `12a918ed`. +- Reran the entire build shard from zero in one process: 6,608 tests collected, + 100% reached, and pytest exited 0 with expected skips only. All five full + package shards, repository-wide Ruff, generated-artifact checks, the spec + coverage proof, the CI test inventory, and `git diff --check` are green. +- Wrote the final provenance tables, mechanism verdict, per-file rationale, + exhaustive release-gate audit, owner-ruling list, verification evidence, and + judgment calls to `out.md`. + +## Next + +- Host session: rerun `late_transfer -> simulated -> terminal-gates` from the + candidate checkpoints because the discrete weeks codec changes pool content. +- Review the six archived-model owner rulings in `out.md`; do not reinterpret + their missing ACS predictors through a gate-only threshold/scope change. -The stacked-pool CD-vintage provenance journal previously in this file is -historical. Its final state remains available at commit `19854a9f`. -# Gate-failed base-pool release lane +# Historical: gate-failed base-pool release lane ## State diff --git a/changelog.d/798-pregnancy-structural-transfer.fixed.md b/changelog.d/798-pregnancy-structural-transfer.fixed.md new file mode 100644 index 000000000..e03221672 --- /dev/null +++ b/changelog.d/798-pregnancy-structural-transfer.fixed.md @@ -0,0 +1 @@ +Make US pregnancy structurally female-age eligible before ACS transfer: draw once for each eligible source person, fan the result across support clones, assign false to ineligible missing recipients, and refuse preexisting or final domain/clone violations with authenticated counts. Bind the policy to the checkpointed transfer identity and carry its proof in the stacked transfer receipt. diff --git a/changelog.d/799-prior-year-rung-floor.fixed.md b/changelog.d/799-prior-year-rung-floor.fixed.md new file mode 100644 index 000000000..0fbb0e266 --- /dev/null +++ b/changelog.d/799-prior-year-rung-floor.fixed.md @@ -0,0 +1 @@ +Scale only the prior-year-income availability gate's authored lower floor by the authenticated stacked sampling rung, recording the applied match-survival factor and floor on sampled assemblies while preserving the authored 0.05 constant, upper bound, every other band, and byte-identical full-rung output. diff --git a/changelog.d/acs-release-predictor-join.fixed.md b/changelog.d/acs-release-predictor-join.fixed.md new file mode 100644 index 000000000..987305594 --- /dev/null +++ b/changelog.d/acs-release-predictor-join.fixed.md @@ -0,0 +1 @@ +Populate physical ACS rows' CPS-named inputs for the six archived US release donor models through an exact, SHA-pinned 2024 ACS person/household join; enforce reviewed disability, consumed race/Hispanic, occupation, tenure, and SSI-reporter semantics; refuse ambiguous, partial, or conflicting lineage; and receipt ASEC-native, ACS-joined, and remaining-null counts in both release manifests without changing model selection logic or gate thresholds. diff --git a/changelog.d/stacked-release-gate-alignment.fixed.md b/changelog.d/stacked-release-gate-alignment.fixed.md new file mode 100644 index 000000000..4fe48f469 --- /dev/null +++ b/changelog.d/stacked-release-gate-alignment.fixed.md @@ -0,0 +1 @@ +Preserve integer support for transferred ``weeks_unemployed`` values and align US release-side source, clone, and deterministic-identity checks with stacked ASEC+ACS pools while retaining legacy ASEC+PUF behavior and every existing plausibility threshold. diff --git a/docs/evidence/spec-engine/us-f0-coverage.json b/docs/evidence/spec-engine/us-f0-coverage.json index 68f70290f..b202fe632 100644 --- a/docs/evidence/spec-engine/us-f0-coverage.json +++ b/docs/evidence/spec-engine/us-f0-coverage.json @@ -6,7 +6,7 @@ "country": "us", "documentation_sha256": "4b39450dbdb8dafb83c3b627123b8026c6f82c660b66fe76f341a67c4f37c77b", "field_usage": { - "authored_normative_field_count": 32351, + "authored_normative_field_count": 32384, "claim_count": 49, "claims": [ { @@ -220,8 +220,8 @@ ], "mode": "legacy_behavior", "pointer_class": "all", - "pointer_count": 79, - "pointer_sha256": "1575513251c72be60edc75f378221fa878ea76e009512f1352a111586d0a60f8", + "pointer_count": 97, + "pointer_sha256": "cabaaa3d96c534f5ec37f51811c3f549d0ed20791e2892c868888c0da6d7d80e", "rationale": null, "relative_sink_prefix": null, "source_prefix": "/authored/spec~1imputation.yaml/transfer_execution", @@ -274,8 +274,8 @@ ], "mode": "compiler_semantic", "pointer_class": "all", - "pointer_count": 24473, - "pointer_sha256": "1b9f7ffb88d31d2bd365c092d04a96038b2d5a2b03f4f2a79f42e95472b213d6", + "pointer_count": 24488, + "pointer_sha256": "6382a3ed07016414c31462b8bedb24590d92b792405a89c927d85fa1bd2a8347", "rationale": null, "relative_sink_prefix": null, "source_prefix": "/authored/spec~1imputation.yaml/producer_graph", @@ -736,8 +736,8 @@ "legacy_sinks": [], "mode": "compiler_semantic", "pointer_class": "all", - "pointer_count": 823, - "pointer_sha256": "c05ebb88538195798e9365f325381c14d92b74007f846f1fef50a08268e6d27d", + "pointer_count": 824, + "pointer_sha256": "7537385c3fd399a2dbb7dcd8ed7cf1ff2481ed336db621eafcbfd741d5792f40", "rationale": null, "relative_sink_prefix": null, "source_prefix": "/resolved/seed_protocol", @@ -772,21 +772,21 @@ "verifier": "vintages" } ], - "configuration_field_count": 42120, - "consumed_field_count": 42120, + "configuration_field_count": 42154, + "consumed_field_count": 42154, "generation0_effect_counts": { - "legacy_behavior": 38443, - "no_generation0_effect": 3677 + "legacy_behavior": 38476, + "no_generation0_effect": 3678 }, "mode_counts": { - "compiler_semantic": 27699, + "compiler_semantic": 27715, "front_end_validation": 348, "identity_only": 103, - "legacy_behavior": 13970 + "legacy_behavior": 13988 }, "multiple_primary_use_field_count": 0, - "pointer_inventory_sha256": "bc4a948ab632191954600da8474c5b011f977a65e24c399d126f3dc4a79f23e5", - "resolved_binding_field_count": 9769, + "pointer_inventory_sha256": "3fc6b9480ea81b9635bd0db56e180c2daf32a5cd2006a70d350586c570f96754", + "resolved_binding_field_count": 9770, "unused_field_count": 0 }, "inventory_coverage": { @@ -810,7 +810,7 @@ "primary_targets": 65, "producer_authored_outputs": 92, "producer_compiled_outputs": 227, - "producer_inputs": 2742, + "producer_inputs": 2744, "producer_nodes": 38, "producer_virtual_resources": 75, "release_rungs": 5, @@ -1074,11 +1074,11 @@ "legacy_adapter.stacked_checkpoint_static_components" ], "expected": { - "schedule_sha256": "dcf3c6d2eade3449836c49a1dc4d3b8cd395aab9142db700c3c60598fa9c1c79" + "schedule_sha256": "e59c019d3d454eac99ac0ac209b6c5b6faaf9bdfcaeee18c36a25be19bf7da2f" }, "failures": [], "observed": { - "schedule_sha256": "dcf3c6d2eade3449836c49a1dc4d3b8cd395aab9142db700c3c60598fa9c1c79" + "schedule_sha256": "e59c019d3d454eac99ac0ac209b6c5b6faaf9bdfcaeee18c36a25be19bf7da2f" }, "status": "covered" }, @@ -1250,14 +1250,14 @@ "expected": { "edges": 71, "nodes": 38, - "schedule_sha256": "dcf3c6d2eade3449836c49a1dc4d3b8cd395aab9142db700c3c60598fa9c1c79", + "schedule_sha256": "e59c019d3d454eac99ac0ac209b6c5b6faaf9bdfcaeee18c36a25be19bf7da2f", "waves": 6 }, "failures": [], "observed": { "edges": 71, "nodes": 38, - "schedule_sha256": "dcf3c6d2eade3449836c49a1dc4d3b8cd395aab9142db700c3c60598fa9c1c79", + "schedule_sha256": "e59c019d3d454eac99ac0ac209b6c5b6faaf9bdfcaeee18c36a25be19bf7da2f", "waves": 6 }, "status": "covered" @@ -1275,11 +1275,11 @@ ], "expected": { "relation": "source rows preserved exactly", - "rows": 2742 + "rows": 2744 }, "failures": [], "observed": { - "rows": 2742 + "rows": 2744 }, "status": "covered" }, @@ -1338,12 +1338,12 @@ ], "expected": { "nodes": 38, - "sha256": "7125ad28ae2c69f22094a574bbf6ed2ddf1682a2c2c3b416f8f49304b7016ce7" + "sha256": "271a7bb8d0b3f97ff344e0b7e68184fa74738a6585c24fc8781793db669f388b" }, "failures": [], "observed": { "nodes": 38, - "sha256": "7125ad28ae2c69f22094a574bbf6ed2ddf1682a2c2c3b416f8f49304b7016ce7" + "sha256": "271a7bb8d0b3f97ff344e0b7e68184fa74738a6585c24fc8781793db669f388b" }, "status": "covered" }, @@ -1361,12 +1361,12 @@ ], "expected": { "producer_count": 38, - "sha256": "3850554eb804fde5e4f86a34ac1bb8a7a07aafff7e8b48396a3d5fca844798e8" + "sha256": "afebb6725373abf5b8dd4fdb77bf2814cb6fcc569cb606c0c30963a8f65c0bab" }, "failures": [], "observed": { "producer_count": 38, - "sha256": "3850554eb804fde5e4f86a34ac1bb8a7a07aafff7e8b48396a3d5fca844798e8" + "sha256": "afebb6725373abf5b8dd4fdb77bf2814cb6fcc569cb606c0c30963a8f65c0bab" }, "status": "covered" }, @@ -1656,13 +1656,13 @@ "compiler_ir.node_slices" ], "expected": { - "map_sha256": "36a9d819ef196c312888591936d49c025b0407df9928440cceefabc5458f72af", - "protocol_sha256": "15840b380329410a7094f60b0f1dad453457fc785859f0c372f2c8e2d59b0246" + "map_sha256": "271c23eca777b63bbfecc5b0bdbfffee634e13f79dd37273172f5fadda4afefb", + "protocol_sha256": "870f9b43766d4c11f6dc1538bf97e360c6a85dd0d4a4a68c149595dfc0902f0a" }, "failures": [], "observed": { - "map_sha256": "36a9d819ef196c312888591936d49c025b0407df9928440cceefabc5458f72af", - "protocol_sha256": "15840b380329410a7094f60b0f1dad453457fc785859f0c372f2c8e2d59b0246" + "map_sha256": "271c23eca777b63bbfecc5b0bdbfffee634e13f79dd37273172f5fadda4afefb", + "protocol_sha256": "870f9b43766d4c11f6dc1538bf97e360c6a85dd0d4a4a68c149595dfc0902f0a" }, "status": "covered" }, @@ -1677,7 +1677,7 @@ "compiler_ir.seed_stream_map" ], "expected": { - "implementation_sha256": "15840b380329410a7094f60b0f1dad453457fc785859f0c372f2c8e2d59b0246", + "implementation_sha256": "870f9b43766d4c11f6dc1538bf97e360c6a85dd0d4a4a68c149595dfc0902f0a", "protocol": "legacy-v1", "streams": [ "build_model", @@ -1698,7 +1698,7 @@ }, "failures": [], "observed": { - "implementation_sha256": "15840b380329410a7094f60b0f1dad453457fc785859f0c372f2c8e2d59b0246", + "implementation_sha256": "870f9b43766d4c11f6dc1538bf97e360c6a85dd0d4a4a68c149595dfc0902f0a", "protocol": "legacy-v1", "streams": [ "build_model", @@ -1736,7 +1736,7 @@ }, "failures": [], "observed": { - "sha256": "99710bf212c3801cfd0434d56d2d1d2e4902a2a958978cdbd91cd9acce81d2fb", + "sha256": "a0366cd518f42a240b589af3ef6580aac44ad7ed70969bcc669b365d315146ab", "sites": 53 }, "status": "covered" @@ -1840,11 +1840,11 @@ "legacy_adapter.stacked_checkpoint_static_components" ], "expected": { - "sha256": "3a980927227704d0589f246eef9cd825c2ae84f3a4134ac835e0e5ed39a563ac" + "sha256": "e660a8ce42b69a39d29c5f0ec37264bc69d61b03f27adc386336ec8889531bb2" }, "failures": [], "observed": { - "sha256": "3a980927227704d0589f246eef9cd825c2ae84f3a4134ac835e0e5ed39a563ac" + "sha256": "e660a8ce42b69a39d29c5f0ec37264bc69d61b03f27adc386336ec8889531bb2" }, "status": "covered" }, @@ -1887,7 +1887,7 @@ "alpha", "zeta" ], - "sha256": "a128a85f877fb32def9382b841b8b340f974e8a9148ac029c1f04becdc956c18" + "sha256": "b88f2d9c0f6f92c6cd81eb14d6b126afe59577b8bb392b394b2c6fbbafd195c5" }, "failures": [], "observed": { @@ -1911,7 +1911,7 @@ "alpha", "zeta" ], - "sha256": "a128a85f877fb32def9382b841b8b340f974e8a9148ac029c1f04becdc956c18" + "sha256": "b88f2d9c0f6f92c6cd81eb14d6b126afe59577b8bb392b394b2c6fbbafd195c5" }, "status": "covered" }, @@ -1975,7 +1975,7 @@ "take_up_contract", "us_qbi_reconciliation_contract" ], - "sha256": "91c65a9ff36839d575036264c4bf57ffde6457e8fd180a0856f9be712ada371d" + "sha256": "04899daa491e8f089899c9df64cdb2ed44d61d11da2b4c733db6f20f38a1668a" }, "status": "covered" }, @@ -2599,7 +2599,7 @@ "country": "us", "schema_id": "country_spec", "schema_version": 1, - "spec_sha256": "5378bb9189aec96f50da22aac71e5bd2c3d919e9795f6ef2147e0bc9c739dd8e" + "spec_sha256": "835b4d61de3b153b13c536e371dca6a16fad6d41e003d070f9804c7c251f3590" } }, "report_schema_version": 3, @@ -2609,7 +2609,7 @@ "country": "us", "schema_id": "country_spec", "schema_version": 1, - "spec_sha256": "5378bb9189aec96f50da22aac71e5bd2c3d919e9795f6ef2147e0bc9c739dd8e" + "spec_sha256": "835b4d61de3b153b13c536e371dca6a16fad6d41e003d070f9804c7c251f3590" }, "status": "pass" } diff --git a/out.md b/out.md index 1b0bc2883..e460cf492 100644 --- a/out.md +++ b/out.md @@ -1,495 +1,327 @@ -# scorerlegacy final report +# ACSPRED: release-side ACS predictor join final report + +> **Historical note (2026-08-28).** Lane report, accurate as of 2026-08-27. +> The branch was subsequently merged into `stacked-release-fix-train` +> alongside the #794 and #798/#799 fixes; attested digests were regenerated +> over the merged tree. Currency claims below are historical. + +Date: 2026-08-27 + +Branch: `acs-predictor-release-join` + +Lane base: `606cbd69` (`stacked-release-gate-alignment`) ## Outcome -The historical-incumbent defect is fixed and locally certified. The scorer now -normalizes every loaded historical H5 artifact against the same -period-sensitive PolicyEngine-US metadata index used by the strict release -builder gate. It proves the current formula dependency leaves are present on -their declared entities, drops formula-owned artifact columns, preserves all -Frame authority, and emits deterministic JSON and Markdown receipts. - -The exact real acceptance command cleared the reported failure, loaded and -fully scored the live incumbent, and reached 19.20 GiB peak RSS without -crossing the 20 GiB guard. It then stopped before loading the candidate H5 -because the authenticated candidate-25 manifest failed an unrelated existing -late-producer contract: - - ValueError: US stacked pool manifest - /Users/maxghenis/PolicyEngine/_buildo-runtime/out/candidate-25/pool/pool.manifest.json - late producer 'primary_puf_qrf': late primary-QRF worker binding changed. - -Per the lane instruction, I did not bypass, weaken, or repair that separate -boundary. The scorer writes only after both sides finish, so neither -pool-vs-incumbent.json nor pool-vs-incumbent.md exists. - -## Diff summary and rationale - -- tools/score_us_release_head_to_head.py - - Bumped the deterministic scorecard schema from 2 to 3. - - Added one scorer-only historical-artifact normalization seam and applied it - to entity H5, legacy flat H5, and authenticated pool-manifest loading. - - Reused release._formula_owned_gate_adapter() and - _engine_computed_columns(..., period=release.PERIOD), exactly matching the - fresh-builder ownership authority without editing the release gate. - - For every detected output, resolves the metadata index's authenticated - dependency closure and refuses before dropping anything when any input leaf - is absent from its declared entity table. The error deterministically maps - each output to its sorted missing leaves. - - Rebuilds a cleaned Frame only when needed and preserves weights, strata, - mass log, and metadata. - - Records normalization_receipts.historical_formula_owned_columns with count - and sorted columns_by_entity for every artifact; clean artifacts explicitly - seal count 0 and an empty entity map. - - Renders the same receipt in Markdown and uses stable symbol-based mechanism - citations for the scorer code. -- packages/microcosm-build/tests/test_us_release_head_to_head_scorer.py - - Added a real entity-H5 fixture containing the formula-owned Marketplace - alias and its interview leaf; it loads, drops, scores, receipts, and renders. - - Added a fixture missing the leaf; loading refuses and names both the output - and missing leaf. - - Added a clean H5 fixture; scoring emits the exact empty JSON and Markdown - receipt. -- changelog.d/us-historical-scorer-formula-columns.fixed.md - - Added the repository-convention fix fragment. -- PROGRESS.md - - Maintained the committed state/done/next lane journal from the first step. - -Rationale: a historical release artifact records the input surface of the -engine that built it. When today's locked engine formula-owns one of those -columns, retaining the historical value would make one side of the comparison -override a current formula. Dropping it only after proving its current leaves -are present makes both artifacts flow through one engine version, while the -unchanged fresh-release builder continues to reject formula-owned source -columns. - -## Real dropped-column receipt - -The acceptance invocation exercised the same incumbent load seam before -scoring. After the later candidate-authentication failure, a fresh read-only -print of that exact loaded receipt produced: - -~~~json +The release builder now populates every CPS-named predictor consumed by the six +archived donor models on every physical ACS support row. It does so through an +exact release-time join to the two canonical, SHA-pinned 2024 one-year ACS PUMS +archives and a versioned, digest-pinned crosswalk. It does not fill unknowns +with silent defaults, weaken a gate, retrain a model, or change model selection +logic. + +The join passed read-only on the complete supplied candidate: 856,626 unique +ACS source people matched exactly 856,626 raw person records in 382,903 raw +households and fanned out to 1,736,840 ACS support rows. All model predictors +have zero remaining nulls. The SSI reporter path separately preserves 261,605 +genuine below-age-15 ACS support-row nulls and treats them as false only at the +consumer's `> 0` predicate. + +The implementation adds four all-or-none release CLI inputs, writes the full +receipt to both manifests, keeps legacy non-ACS releases as an explicit +disabled identity path, and leaves the launcher contract to the dispatcher as +required. + +No network access, pool build, release build, wheel build, publication, push, +or launcher edit was performed. + +## Join-key verification + +### Why `person_source_id` is not the raw key + +The ACS loader rejects duplicate `(SERIALNO, SPORDER)` pairs, stably sorts by +that pair, and only then assigns a zero-based raw `source_row_id` +(`acs_pums.py:204-238`). Source construction stores `SPORDER` separately as +`source_person_id` and links people to the household identified by `SERIALNO` +(`acs_pums.py:264-279`). Assembly then collision-offsets structural IDs and +creates assembly-unique support source IDs (`spine_assembly.py:366-430`; +`support_provenance.py:346-371`). Therefore neither the final structural ID nor +`person_source_id` is a reversible Census record key. + +The release join instead recovers the retained semantic identity: + +| Role | Retained evidence | Enforced relation | +|---|---|---| +| Household key | linked household `SERIALNO` | nonblank, one source household identity per serial | +| Person key | integral `SPORDER` and `source_person_id` | exactly equal and at least 1 | +| Raw ordinal | `source_row_id`, `person_spine_source_id` | exactly equal, but never used as the Census join key | +| Source identity | `person_source_id` | one raw identity across all clones; fan-out only | +| Clone identity | `person_support_clone_index` | unique `(person_source_id, clone_index)` and exactly one clone 0 | +| Vintage | `source_year` | exactly 2024 on physical ACS rows | + +The implementation derives these relations at +`acs_release_predictors.py:1078-1289`, including person-to-household channel +agreement, clone agreement, raw ordinal agreement, and unique canonical +`(SERIALNO, SPORDER)`. It then performs a pandas `one_to_one` left merge on +that semantic key, rejects any unmatched pool source person, and refuses a row +count change (`acs_release_predictors.py:893-923`). Only after that proof does +it map values back through `person_source_id` to every clone +(`acs_release_predictors.py:939-968`). + +## Artifact and refusal contract + +The reviewed archive identities are declared at +`acs_release_predictors.py:70-88`: + +| Archive | SHA-256 | Size observed | +|---|---|---:| +| 2024 ACS person `csv_pus.zip` | `afdc6d90c6e2f0bab365ed32d95ba4c4d8ac651162f46ac7861295b2dc469894` | 602,847,146 bytes | +| 2024 ACS household `csv_hus.zip` | `8281008e53de98f0ef81e7a2ee5a8725991dda1ecfd2713ead73246425e515d0` | 251,500,587 bytes | + +Before either zip is opened, the boundary requires lowercase 64-hex syntax, +requires the caller's expected hash to equal the reviewed canonical pin, and +hashes the actual file (`acs_release_predictors.py:991-1021`). It then requires +exact `psam_pusa/b.csv` or `psam_husa/b.csv` membership, verifies required +headers, and streams only selected serials (`acs_release_predictors.py:1024-1075`). + +The boundary also refuses: + +- partial CLI/archive inputs or archive inputs on a frame with no ACS rows; +- a stale crosswalk digest; +- malformed or incomplete assembly provenance; +- duplicate raw person or household keys; +- source-identity, semantic-key, or clone-index collisions; +- orphaned or cross-channel person/household links; +- pool household `TEN` disagreement with the pinned household archive; +- an incomplete one-to-one person join; +- an unsupported raw code or age/universe violation; +- a conflicting pre-existing ACS predictor value; +- missing, nonnumeric, or nonfinite native ASEC predictors, or negative + `SSI_VAL`; and +- any remaining null in a model-consumed predictor. + +The relevant executable checks are at +`acs_release_predictors.py:830-968,1078-1356,1548-1586`. + +## Reviewed crosswalks + +The canonical crosswalk payload is source-controlled at +`acs_release_predictors.py:726-790` and pinned as: + +`1d4906242e9c73e31b3283659e5cad8242b8cbc42914ab6fa59547a10c8770e9` + +| Source | Target | Exact reviewed mapping | Consumed model bin | +|---|---|---|---| +| `DDRS` | `PEDISDRS` | age 5+: `1 -> 1`, `2 -> 2`; younger blank `-> -1` | SSI tests only `== 1` | +| `DEAR` | `PEDISEAR` | all ages: `1 -> 1`, `2 -> 2` | SSI tests only `== 1` | +| `DEYE` | `PEDISEYE` | all ages: `1 -> 1`, `2 -> 2` | SSI tests only `== 1` | +| `DOUT` | `PEDISOUT` | age 15+: `1 -> 1`, `2 -> 2`; younger blank `-> -1` | SSI tests only `== 1` | +| `DPHY` | `PEDISPHY` | age 5+: `1 -> 1`, `2 -> 2`; younger blank `-> -1` | SSI tests only `== 1` | +| `DREM` | `PEDISREM` | age 5+: `1 -> 1`, `2 -> 2`; younger blank `-> -1` | SSI tests only `== 1` | +| `RAC1P` | `PRDTRACE` | `1 -> 1` White; `2 -> 2` Black; `6 -> 4` Asian; `3/4/5/7/8/9 -> 3` residual Other | SCF White/Black/Asian/Other; ORG White/Black/Other | +| `HISP` | `PRDTHSP` | `1 -> 0` non-Hispanic; `2..24 -> 1` positive Hispanic representative | both consumers test zero versus positive | +| observed `OCCP` | `PEIOOCC` | identity over all 530 pinned detailed codes | SIPP tips exact detailed-code membership | +| blank `OCCP` | `PEIOOCC` | `-1`, the CPS NIU sentinel | unlisted/non-tipped | +| observed `OCCP` | `POCCU2` | explicit 530-key table covering consumed bins 1 through 53 | ORG exact categories and EAP set | +| blank `OCCP` | `POCCU2` | age below 16 `-> 0`; age 16+ permitted only with `ESR=6`, then `-> 53` | preserve NIU; adult no occupation/never worked | +| household `TEN` | `SPM_TENMORTSTATUS` | `1 -> 1`, `2 -> 2`, `3/4 -> 3`; verified group-quarters blank `-> 3` | SIPP vehicle homeowner iff `{1,2}` | + +The disability, race, occupation, and tenure executable mappings are at +`acs_release_predictors.py:108-687,689-694,1364-1454`. The occupation map's +notable reviewed edges include `3250 -> 26` while `3255/3256/3258 -> 25`, ACS +military `9800/9810/9825/9830 -> 52`, and `9920 -> 53`. The table maps the +shared detailed code vocabulary to the modal 2024 ASEC category; it is not +represented as a fictional rowwise CPS identity. + +### Age-15 occupation judgment + +ACS `OCCP` begins at age 16, while CPS `POCCU2` is already in universe at age +15. In source-year-2024 ASEC, 2,174 people are age 15: 1,931 have `POCCU2=53` +but 243 have categories 1 through 52. Mapping every ACS age-15 blank to 53 +would therefore invent never-worked evidence. The crosswalk preserves 0 for +that one-year source/target universe gap and uses 53 only for defensible adult +blanks. Raw `ESR` is required blank below 16 and in `1..6` from 16; an adult +`OCCP` blank is allowed only at `ESR=6`. + +## SSI reporter semantics + +`SSI_VAL` is the measured ASEC amount. The pool's native ACS mapping already +stores adjusted `SSIP` as `ssi_reported`. The release join now reads raw `SSIP` +and `ADJINC`, enforces the exact age-15 universe, and proves clone-zero +`ssi_reported == SSIP * ADJINC / 1_000_000` source person by source person +(`acs_release_predictors.py:1458-1545`). + +The SSI receiver coalesces `SSI_VAL` and `ssi_reported` rowwise, without +origin-routing the model, rejects conflicting dual reporter statuses, permits +a blank only below age 15, and fills that blank only transiently for the +existing `> 0` predicate (`ssi_disability_criteria.py:758-813,1082-1093`). The +gate summary uses the same coalesced anchor and native-role scope, so a lost +positive ACS-native reporter cannot evade the diagnostic. Source nulls remain +null in the frame. + +## Release integration and receipts + +The release builder declares the four inputs at +`build_us_fiscal_refresh_release.py:1293-1318` and validates all-or-none plus +lowercase SHA syntax at `:1587-1610`: + +- `--acs-person-zip` +- `--acs-person-sha256` +- `--acs-household-zip` +- `--acs-household-sha256` + +It invokes the join at `build_us_fiscal_refresh_release.py:9828-9846`, before +SCF wealth and consequently before all six archived-model stages. The receipt +is embedded at top level in `build_manifest.json` (`:7534-7642`) and under +`release_manifest.json.build` (`:7841-7871`); the sole writer call receives the +saved runtime receipt at `:11860-11879`. + +The complete real-data receipt included this core evidence: + +```json { - "columns_by_entity": { - "person": [ - "has_marketplace_health_coverage" - ] + "enabled": true, + "crosswalk": { + "version": 1, + "sha256": "1d4906242e9c73e31b3283659e5cad8242b8cbc42914ab6fa59547a10c8770e9" }, - "count": 1 + "join": { + "semantic_key": ["household.SERIALNO", "person.SPORDER"], + "clone_fanout_key": "person_source_id", + "acs_source_people": 856626, + "acs_support_rows": 1736840, + "acs_support_rows_by_clone_index": {"0": 856626, "1": 856626, "2": 23588}, + "selected_raw_person_rows": 856626, + "selected_raw_household_rows": 382903, + "unmatched_pool_source_people": 0, + "source_identity_collisions": 0, + "semantic_key_sha256": "c7723adc889fc655b46103426d21f6c74937434eff51e8f3e6025bdf60972b74" + } } -~~~ - -The required leaf -has_marketplace_health_coverage_at_interview was present on person. The -candidate receipt was not produced because authenticated manifest validation -failed before the candidate H5 could load. Focused coverage proves that a clean -loaded artifact emits {"count": 0, "columns_by_entity": {}}. - -## Headline score result - -Unavailable because the head-to-head did not complete and the scorer -deliberately writes no partial output. There is therefore no -pool-vs-incumbent.md from which to quote weighted loss or percent-within-10 -figures. The incumbent alone completed all 32,842 registry targets, but its -in-memory partial payload was correctly not persisted after candidate -authentication failed. - -## Verification evidence - -- Required Ruff command: passed. - - The managed sandbox rejected uv's default cache before execution. - - Re-run offline with UV_CACHE_DIR set to a writable task-specific directory: - uv run --no-sync ruff check . -> All checks passed. -- CI test inventory: tools/ci_test_groups.py --verify passed for 309 tracked - test files; the scorer test remains in engine lane us-qs. -- Targeted scorer suite: 16 passed in 34.34 seconds. -- Full pytest, one separate process per shard: - - build: 6,516 passed, 45 skipped, 0 failed in 3,461.09 seconds; - - frame: 295 passed, 36 skipped, 0 failed in 97.72 seconds; - - calibrate: 203 passed, 0 skipped, 0 failed in 11.61 seconds; - - data: 318 passed, 2 skipped, 0 failed in 7.59 seconds; - - fit: 93 passed, 0 skipped, 0 failed in 27.28 seconds. - - aggregate: 7,425 passed, 83 skipped, 0 failed. -- git diff --check passed throughout. -- No network, push, pool/release build, publication, or validation bypass was - used. - -## Acceptance execution - -Command run from this worktree: - - .venv/bin/python tools/score_us_release_head_to_head.py \ - --incumbent /Users/maxghenis/.cache/huggingface/hub/datasets--policyengine--populace-us/snapshots/26dcad66867687f15735dc4926523e3741920836/populace_us_2024.h5 \ - --candidate /Users/maxghenis/PolicyEngine/_buildo-runtime/out/candidate-25/pool/pool.manifest.json \ - --ledger-facts /Users/maxghenis/PolicyEngine/_buildh-runtime/inputs/consumer_facts_buildn_v9_4.jsonl \ - --out-prefix /Users/maxghenis/PolicyEngine/_buildo-runtime/out/candidate-25/score-pool/pool-vs-incumbent - -Observed milestones: - -- Fiscal yardstick compiled at 2.45 GiB peak RSS. -- Live incumbent loaded at 3.70 GiB, clearing the original formula-owned-column - export failure. -- All five incumbent chunks completed, each over 12 fixed household slices. -- Incumbent scoring state released at 19.20 GiB peak RSS. -- Candidate manifest authentication then failed before candidate H5 loading. -- Both requested scorecard output paths were confirmed absent. - -Complete emitted traceback: - -~~~text -Traceback (most recent call last): - File "/Users/maxghenis/PolicyEngine/_worktrees/microcosm-scorer-legacy/tools/score_us_release_head_to_head.py", line 2300, in - raise SystemExit(main()) - ~~~~^^ - File "/Users/maxghenis/PolicyEngine/_worktrees/microcosm-scorer-legacy/tools/score_us_release_head_to_head.py", line 2268, in main - payload = score_head_to_head( - incumbent=args.incumbent, - ...<8 lines>... - candidate_manifest_sha256=args.candidate_manifest_sha256, - ) - File "/Users/maxghenis/PolicyEngine/_worktrees/microcosm-scorer-legacy/tools/score_us_release_head_to_head.py", line 1797, in score_head_to_head - loaded = load_artifact( - path, - expected_manifest_sha256=expected_manifest_sha256, - ) - File "/Users/maxghenis/PolicyEngine/_worktrees/microcosm-scorer-legacy/tools/score_us_release_head_to_head.py", line 598, in load_artifact - artifact = _load_pool_manifest( - resolved, - expected_manifest_sha256=expected_manifest_sha256, - ) - File "/Users/maxghenis/PolicyEngine/_worktrees/microcosm-scorer-legacy/tools/score_us_release_head_to_head.py", line 505, in _load_pool_manifest - frame, manifest, authenticated = load_authenticated_us_multispine_pool_for_scoring( - ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^ - manifest_path, - ^^^^^^^^^^^^^^ - expected_manifest_sha256=expected_manifest_sha256, - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - ) - ^ - File "/Users/maxghenis/PolicyEngine/_worktrees/microcosm-scorer-legacy/packages/microcosm-build/src/microcosm/build/us_runtime/h5_io.py", line 1281, in load_authenticated_us_multispine_pool_for_scoring - return _load_us_multispine_pool( - path, - expected_manifest_sha256=expected_manifest_sha256, - require_simulation_ready=False, - ) - File "/Users/maxghenis/PolicyEngine/_worktrees/microcosm-scorer-legacy/packages/microcosm-build/src/microcosm/build/us_runtime/h5_io.py", line 1297, in _load_us_multispine_pool - manifest, authenticated_pool_h5 = _load_authenticated_us_multispine_pool_manifest( - ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^ - manifest_path, - ^^^^^^^^^^^^^^ - expected_manifest_sha256=expected_manifest_sha256, - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - allow_terminal_gate_failure=not require_simulation_ready, - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - ) - ^ - File "/Users/maxghenis/PolicyEngine/_worktrees/microcosm-scorer-legacy/packages/microcosm-build/src/microcosm/build/us_runtime/h5_io.py", line 440, in _load_authenticated_us_multispine_pool_manifest - _validate_stacked_late_dag_manifest_binding( - ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^ - manifest, - ^^^^^^^^^ - manifest_path=manifest_path, - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - ) - ^ - File "/Users/maxghenis/PolicyEngine/_worktrees/microcosm-scorer-legacy/packages/microcosm-build/src/microcosm/build/us_runtime/h5_io.py", line 644, in _validate_stacked_late_dag_manifest_binding - validate_stacked_late_producer_receipt( - ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^ - dag, - ^^^^ - boundary=f"US stacked pool manifest {manifest_path}", - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - ) - ^ - File "/Users/maxghenis/PolicyEngine/_worktrees/microcosm-scorer-legacy/packages/microcosm-build/src/microcosm/build/us_runtime/stacked_spine.py", line 8051, in validate_stacked_late_producer_receipt - previous_sha256 = _validate_late_execution_row( - raw_row, - ...<3 lines>... - boundary=boundary, - ) - File "/Users/maxghenis/PolicyEngine/_worktrees/microcosm-scorer-legacy/packages/microcosm-build/src/microcosm/build/us_runtime/stacked_spine.py", line 7744, in _validate_late_execution_row - _validate_late_available_input_receipt( - ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^ - receipt, - ^^^^^^^^ - ...<3 lines>... - boundary=f"{boundary} late producer {contract.name!r}", - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - ) - ^ - File "/Users/maxghenis/PolicyEngine/_worktrees/microcosm-scorer-legacy/packages/microcosm-build/src/microcosm/build/us_runtime/stacked_spine.py", line 5768, in _validate_late_available_input_receipt - _validate_late_resource_binding( - ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^ - binding, - ^^^^^^^^ - ...<3 lines>... - boundary=boundary, - ^^^^^^^^^^^^^^^^^^ - ) - ^ - File "/Users/maxghenis/PolicyEngine/_worktrees/microcosm-scorer-legacy/packages/microcosm-build/src/microcosm/build/us_runtime/stacked_spine.py", line 5506, in _validate_late_resource_binding - raise ValueError(f"{boundary}: late primary-QRF worker binding changed.") -ValueError: US stacked pool manifest /Users/maxghenis/PolicyEngine/_buildo-runtime/out/candidate-25/pool/pool.manifest.json late producer 'primary_puf_qrf': late primary-QRF worker binding changed. -~~~ - -## Judgment calls - -- Applied automatic normalization to every scorer artifact route because all - those Frames originate from persisted historical H5s; no skip flag exists. -- Used the gate's period-sensitive ownership classifier, not the runtime - PolicyEngineUSEngine helper and not period-insensitive - formula_owned_outputs(). -- Required every dependency-closure leaf on its metadata-declared entity. The - source closure is a conservative all-vintage static union, so a future dated - formula can theoretically over-require an irrelevant leaf; fail-closed is the - correct direction for this task, and the locked Marketplace alias is a - direct one-edge closure with no ambiguity. -- Kept the fresh release builder's _assert_no_formula_owned_columns function - and semantics byte-for-byte untouched. -- Bumped the output schema because the always-present receipt changes the - deterministic JSON contract. -- Did not treat the later candidate authentication failure as authorization to - weaken pool provenance. The next action belongs to candidate artifact - production, not this scorer lane. - -## Commits before the final report seal - -- 61311bf5 Start historical scorer column lane -- 169a441a Normalize historical scorer formula columns -- 7a44561c Handle clean entities during scorer normalization -- 2a2df350 Record complete scorer verification -- 34dd3129 Keep scorer mechanism citations stable -# Gate-failed base-pool release lane: final report - -Date: 2026-08-26 - -Branch: `release-from-gate-failed-pool` - -Base: `origin/main` at `2263df36` - -## Outcome - -The legacy `--base-h5` release and preflight paths now fail closed when the H5 -identifies as a US multispine pool. They require and authenticate the canonical -sibling pool manifest, bind it to the exact requested H5, and reject a red -terminal agreement battery by default. The new explicit opt-in is: - -`--allow-gate-failed-base-pool` - -The flag is valid only with `--base-h5` and only for an authenticated current -stacked pool with `status=gate_failed` and `simulation_ready=false`. It is -rejected for generic H5 inputs, green pools, and the exact-k manifest arm. A -release built with the flag carries the full authenticated red verdict so a -reader can see `battery: red, N failures` without fetching the pool. - -Publication preflight authenticates the same pool and displays the red receipt -as prominent human-review evidence. The red battery alone does not change the -existing preflight PASS/AT_RISK/FAIL calculation or exit code. Publication -remains a separate human-gated operation. - -No network access, pool build, release build, publication, push, battery -threshold change, or gate-logic change was performed. - -## Diff summary and rationale - -- `CLAUDE.md`: documents the authenticated `--base-h5` boundary, explicit red - opt-in, verdict carriage, strict exact-k arm, and separate publication step. -- `PROGRESS.md`: maintains the required state/done/next journal from kickoff - through the final verified handoff while preserving the prior lane history. -- `changelog.d/gate-failed-pool-release.fixed.md`: records the closed receipt - bypass and explicit opt-in under the repository changelog convention. -- `packages/microcosm-build/src/microcosm/build/us_runtime/h5_io.py`: adds the - shared pool classifier, exact-H5 binding check, release-specific authenticated - loader, and normalized self-contained release receipt. The existing strict - simulation-ready and scoring loader contracts were not weakened. -- `packages/microcosm-build/src/microcosm/build/us_runtime/release_gate_preflight.py`: - authenticates identified base pools through the shared seam, records their - receipt in `PreflightReport`, and renders authenticated red evidence before - the ordinary check table without including it in exit-code calculation. -- `tools/build_us_fiscal_refresh_release.py`: adds - `--allow-gate-failed-base-pool`, authenticates pool-like legacy base H5s before - the generic loader can run, uses the authenticated frame/H5 identity, and - passes the receipt into both generated manifests. The exact-k arm remains - simulation-ready-only. -- `tools/preflight_us_release_gates.py`: adds the same explicit opt-in and an - optional `--release-manifest`; validates the carried full verdict, requires - its receipt to exactly equal the pool authenticated from `--base-h5`, emits a - large red/human-review banner, and adds machine-readable carried evidence. - Existing required `--base-h5` and `--selection-source-manifest` arguments and - all other preflight behavior remain intact. -- `packages/microcosm-build/tests/test_us_multispine_pool_h5_io.py`: exercises - real stamped H5 classification, strict/default refusal, explicit red - acceptance, redundant-green refusal, receipt contents, identity binding, and - malformed aggregate/nested verdict refusal. PyTables-dependent cases use the - sibling-test `importorskip` idiom. -- `packages/microcosm-build/tests/test_us_fiscal_refresh_builder.py`: covers the - old bare-H5 bypass directly, red-manifest default refusal, flag routing, - generic/non-pool refusal when the flag is present, both-manifest carriage, - exact-k isolation, and AST-level ordinary base-pool wiring guards. -- `packages/microcosm-build/tests/test_us_release_gate_preflight.py`: covers - bare/missing/red receipt authentication, prominent text and JSON carriage, - nonblocking PASS/AT_RISK/FAIL exit semantics, required static inputs, and - exact mismatch refusal across manifest/H5/publication/gates/full-verdict - identities. -- `out.md`: this final implementation, audit, verification, and handoff report. - -## Flag and manifest contract - -The exact flag is `--allow-gate-failed-base-pool`. The release builder permits -it only on its legacy `--base-h5` arm. Publication preflight uses the same flag -to authenticate and inspect the same red pool; neither invocation is an -authorization to publish. - -The build manifest stores the receipt at top-level `base_pool`. The release -manifest stores the same receipt at `build.base_pool`: - -- `artifact_kind` -- `status` -- `simulation_ready` -- `manifest_sha256` -- `publication_run_id` -- `pool_h5_sha256` -- `pool_h5_size_bytes` -- `allow_gate_failed_base_pool` -- `agreement_gate_reference.battery_status` -- `agreement_gate_reference.passed` -- `agreement_gate_reference.gates_json_sha256` -- `agreement_gate_reference.failure_count` -- `agreement_gate_reference.failures`, a flattened list of `{gate, message}` -- `agreement_gate_reference.verdict`, the complete agreement-gate verdict - -The receipt accepts only coherent pairs: simulation-ready/green/passed or -gate-failed/red/failed. Every nested gate must carry a boolean `passed` and a -coherent failure list; the nested aggregate must match the terminal status. -For a red pool the failure list must be nonempty. SHA-256, publication run ID, -H5 size, full verdict, and all other receipt fields are bound through exact -preflight equality. - -The preflight report also exposes the authenticated receipt as `base_pool`. -When `--release-manifest` supplies a red carried receipt, JSON adds -`carried_base_pool_agreement_battery`, including `battery_status=red`, the -failure count/list, gates digest, full agreement reference, -`publication_decision=human_review_required`, and `affects_exit_code=false`. - -## §4 consumer audit - -The requested exhaustive source audit used -`simulation_ready|gate_failed|load_simulation_ready` across `tools/` and -`packages/`, followed by source tracing of every non-test match. - -- `h5_io.py` owns the authentication boundary. The new release wrapper selects - the unchanged strict loader by default and the existing private terminal - loader only when its required explicit boolean is true. The public scoring - and strict simulation-ready contracts remain unchanged. -- `tools/build_us_exact_k_ladder_release.py` remains deliberately strict: it - uses the simulation-ready loader, validates config/release identities, checks - `agreement_gate.passed`, and never forwards the new flag. -- `tools/score_us_release_head_to_head.py` retains its existing authenticated - terminal-evidence exception. It requires a pool manifest and terminal gates; - it cannot authorize a release or accept a naked pool H5. -- `tools/build_us_multispine_pool.py` and - `us_runtime/multispine_pool.py` are producers of the status, readiness, H5 - stamp, manifest, diagnostics, and terminal verdict rather than downstream - release consumers. -- `tools/_legacy/build_us_acs_multispine_base.py` writes its own pre-calibration - `simulation_ready=false` state; it is not a current terminal-pool consumer. -- `tools/build_us_acs_local_release.py` uses its own calibrated-release - readiness summary. A derivative keeps donor release/revision provenance but - does not project the nested `build.base_pool` verdict into its own manifest. -- The microcosm-data release contract, loader, TRACE conversion, and publisher - tolerate but otherwise ignore the additive `build.base_pool` object. The - designated publication preflight is therefore the prominent human-facing - red-verdict surface. - -Report-only interactions left unchanged, as directed: - -1. `tools/build_us_multispine_pool.py::_stacked_manifest_payload` still labels - `calibration.consumer` as `k-ladder` and says - `requires_manifest_simulation_ready=true`. That metadata is now incomplete - for the explicit legacy base-H5 red-pool route, but it is informational and - unenforced. -2. A red pool producer writes its H5, manifest, gates evidence, and failed - Logbook row, then returns status 1. Existing `set -e` candidate chains stop - there, so an operator must deliberately start the separate release command - to use the new opt-in. -3. ACS-local derivative manifests retain fetchable donor provenance but do not - self-contain the nested red receipt. -4. Generic data/release consumers accept the additive receipt but do not - surface it. This is not a publication bypass because the publication tool's - designated preflight now authenticates and displays it. - -No audit-only consumer was modified. - -## Judgment calls - -- Pool detection uses either a canonical sibling manifest whose artifact kind - is the pool kind or the H5's own artifact-metadata stamp. Either positive - identity requires the sidecar; sidecar existence alone is never trusted. -- A dedicated release loader with a required - `allow_terminal_gate_failure` argument keeps the strict and scoring APIs - semantically stable. -- The opt-in is rejected for a green pool rather than silently accepted, so - every recorded flag use has one unambiguous meaning. -- Preflight requires exact equality between the release-carried receipt and the - freshly authenticated base receipt. This prevents displaying release A's red - evidence while checking release B's pool. -- The earlier salvage branch was used as a reference and selected commits were - replayed only after line-by-line review. The final tree corrected its - manifest-only preflight expansion, loader naming/contract ambiguity, - redundant-green behavior, receipt binding, nested-verdict coherence, and an - inaccurate exact-k diagnostic. Full verification ran on the corrected tree. - -## Verification evidence - -All commands ran offline against the pre-built environment with -`uv run --no-sync` (and a task-local UV cache where required). Each pytest -shard ran in one independent process. +``` + +Per-model counts were: + +| Model | Predictors | ASEC-native each | ACS-joined each | Still null | +|---|---|---:|---:|---:| +| SSI disability | six `PEDIS*` | 234,133 | 1,736,840 | 0 | +| SCF wealth | `PRDTRACE`, `PRDTHSP` | 234,133 | 1,736,840 | 0 | +| SCF auto loans | `PRDTRACE`, `PRDTHSP` | 234,133 | 1,736,840 | 0 | +| SIPP vehicles | `SPM_TENMORTSTATUS` | 234,133 | 1,736,840 | 0 | +| SIPP tips | `PEIOOCC` | 234,133 | 1,736,840 | 0 | +| ORG wages/FLSA | `PRDTRACE`, `PRDTHSP`, `POCCU2` | 234,133 | 1,736,840 | 0 | +| SSI logical reporter anchor | ASEC `SSI_VAL`; ACS `ssi_reported` | 234,133 | 1,475,235 | 261,605 child-universe nulls | + +The receipt producer is at `acs_release_predictors.py:1589-1677`. + +## Archived-model behavior retained + +The five non-SSI consumer modules are byte-unchanged from the lane base. The +SSI model's predictor list, QRF feature/selection logic, role routing, and +thresholds are unchanged; only its source-faithful reporter read and matching +gate diagnostic changed. + +| Consumer | Executable consumed bins | Evidence | +|---|---|---| +| SSI disability | six difficulty inputs consumed only as `== 1`; reporter is under 65 and `> 0` | `ssi_disability_criteria.py:143-150,902-908,1082-1093` | +| SCF wealth | White 1, Black 2, Asian 4, Hispanic positive, residual Other | `scf_wealth.py:611-621,654-724` | +| SCF auto loans | reuses the SCF wealth race helper | `scf_auto_loans.py:38-42,365-380` | +| SIPP vehicles | homeowner iff tenure code is 1 or 2 | `sipp_vehicles.py:577-621,740-763` | +| SIPP tips | exact detailed Census occupation membership; unlisted NIU is zero | `sipp_tips.py:129-184,249-263,363-375` | +| ORG wages/FLSA | POCCU2 53 never worked, 52 military, 8 computer, 41 farmer/fisher, explicit EAP set; Hispanic/White/Black/Other | `org_wages.py:133-183,408-415,535-559,610-625` | + +No gate band or numeric threshold was edited. Whether ORG and tips pass or fail +on a future release is therefore determined by the real joined values, as +required. + +## Real-pool audit and operational caveat + +The supplied candidate pool contains 1,970,973 people and 865,460 households. +Its 3,239,263,147-byte H5 exactly matches the frozen manifest SHA-256 +`871b7e6467675a1e9475b54fd1baf64c53c0f75a3258b8357303a8df0d53642d`. + +The current official release loader refuses that older candidate before H5 +loading because the manifest's archived primary-QRF worker binding predates +this branch's source-attested execution identity: `late primary-QRF worker +binding changed`. To isolate and test this lane without building an artifact, +the audit independently verified the H5 bytes, read its entity tables through +the repository HDF reader, restored the frozen `assembly_receipt`, and ran the +strict join. Every join and crosswalk assertion passed. + +This is not an unresolved crosswalk issue, but it is an operational handoff: +the old candidate cannot be promoted through the current authenticated release +loader. A fresh/currently authenticated candidate is required outside this +headless no-build lane. + +## Verification + +All commands ran offline against the prebuilt environment. Each package shard +ran in its own pytest process. Packaging was not touched, so wheels were not +built. - `uv run --no-sync ruff check .`: PASS, `All checks passed!` -- `uv run --no-sync pytest -q packages/microcosm-build/tests`: PASS, 6,545 - passed, 45 skipped, 2,351 warnings in 3,326.74 seconds. -- `uv run --no-sync pytest -q packages/microcosm-calibrate/tests`: PASS, 203 - passed, 2 warnings in 11.39 seconds. -- `uv run --no-sync pytest -q packages/microcosm-data/tests`: PASS, 318 passed, - 2 skipped in 9.11 seconds. -- `uv run --no-sync pytest -q packages/microcosm-fit/tests`: PASS, 93 passed, - 1 warning in 30.27 seconds. -- `uv run --no-sync pytest -q packages/microcosm-frame/tests`: PASS, 295 - passed, 36 skipped, 1 warning in 89.30 seconds. -- Full aggregate: 7,454 passed, 83 skipped. -- `.venv/bin/python tools/ci_test_groups.py --verify`: PASS, - `tracked_test_files=309`, `verification=ok`. -- Focused changed-file suites plus exact-k E2E/launcher and data - contract/release/publish-guard regressions: PASS; the full shard runs above - subsequently covered the same tests on the final code tree. -- `git diff --check origin/main...HEAD`: PASS. - -The build-shard warnings are expected numerical, pandas chained-assignment, -PolicyEngine division, and intentionally fragmented-frame test warnings; no -warning is a test failure and none originates in the new receipt path. - -## Commit inventory - -Implementation and journal commits before this final report carrier: - -1. `34375fe6` Document gate-failed release lane kickoff -2. `5be9e49a` Preserve prior progress journal history -3. `c5a24c0f` Record release containment design review -4. `72274910` feat: allow explicit gate-failed pool release inputs -5. `e3d847d6` feat: surface carried red pool verdict in preflight -6. `46811fdb` fix: preserve default exact-k receipt fixtures -7. `dd1ad19a` Add shared multispine base-pool authentication seam -8. `271ee1fe` Close legacy base-H5 pool receipt bypass -9. `92134322` Authenticate and surface gate-failed pools in preflight -10. `cd39f756` Document gate-failed base-pool release boundary -11. `9db4694a` Tighten the authenticated red-pool release opt-in -12. `72c4c7a1` Bind carried pool verdicts to preflight inputs -13. `59c5759d` Record completed release containment work -14. `af20d3f5` Lock ordinary base-pool release wiring in tests - -Commits after the list carry this report, the final `PROGRESS.md` state, and -their handoff cleanup. A report cannot embed the hash of the commit containing -its own final bytes, so `git log --oneline origin/main..HEAD` is the -authoritative complete inventory. At final handoff the branch is 17 commits -ahead of `origin/main` with no uncommitted paths. +- `uv run --no-sync python tools/ci_test_groups.py --verify`: PASS, + `tracked_test_files=310`, `verification=ok` +- `tools/generate_us_bundle_from_constants.py --check`: PASS, US spec + `16b7d5e622e8a68e008165bb44a5836695d94a2b0dd8d4c51b3c9e8ca89dca38` +- `tools/spec_engine_coverage.py --check`: PASS, 42,122/42,122 fields and + 41/41 inventory checks +- frame shard: PASS, 295 passed, 36 skipped +- fit shard: PASS, 93 passed +- calibrate shard: PASS, 203 passed +- data shard: PASS, 318 passed, 2 skipped +- build shard after reviewed source-attestation repin: PASS, 6,586 passed, + 45 skipped +- join-focused file: PASS, 15 passed +- combined join, SSI, and source-blindness files: PASS +- parser/order/manifest focused cases and all six parametrized mocked-main + corridor cases: PASS +- repository `git diff --check`: PASS + +The first complete build-shard run reached 100% with 6,575 passed and 45 +skipped plus five failures and six setup errors, all caused by stale expected +source-attestation hashes after changing `ssi_disability_criteria.py`. That +module participates in both seed-kernel source inventories. The narrow +established repin updated the two seed digests, BE/UK/US spec identities, the +minimal-loader golden, and generated coverage evidence. All 25 affected tests +then passed, both retained `--check` commands passed, and the complete build +shard rerun above was green. + +Existing warnings were numerical overflow/divide, pandas chained-assignment, +and fragmented-fixture performance warnings; none was a test failure. + +## Judgment calls and unresolved items + +- The join uses retained semantic lineage, not a guessed arithmetic inversion + of assembly IDs. +- Race and Hispanic mappings stop at the bins the consumers actually read; + they do not invent CPS detail absent from ACS. +- Disability codes preserve `1/2` plus the CPS NIU sentinel rather than + collapsing the source to booleans. +- Age-15 ACS occupation remains NIU because assigning 53 would fabricate + never-worked evidence for a group whose CPS distribution is not degenerate. +- `PEIOOCC` blank uses CPS NIU `-1`, not a fictional detailed occupation 0. +- SSI child nulls remain source nulls; only the predicate interprets them as + non-reporters. +- Pool household `TEN` is preferred for lineage and must agree exactly with the + pinned household zip; group-quarters non-owner status is applied only after + raw universe checks. +- ASEC values are validated but never rewritten, preserving their native model + inputs byte-for-byte. +- No crosswalk semantic remains unresolved. The only open operational item is + obtaining a current authenticated pool and adding the four launcher + arguments; both are explicitly outside this lane and owned by the + dispatcher/build process. + +## Commits before final journal handoff + +- `2aa14e84` Start ACS predictor release join journal +- `c1a41ccf` Record ACS predictor join contracts +- `ecea55a2` Add strict ACS release predictor join +- `a7108697` Harden ACS predictor crosswalk contracts +- `6b8185e4` Wire ACS predictor join into release builder +- `d87be068` Accept numeric H5 predictor cells +- `69ec0fae` Document ACS release predictor join +- `de5e5d03` Repin source-attested spec identities + +The final local commit adds this report and marks `PROGRESS.md` complete. diff --git a/packages/microcosm-build/src/microcosm/build/spec_engine/field_usage.py b/packages/microcosm-build/src/microcosm/build/spec_engine/field_usage.py index a5dd282ab..aa6039e3a 100644 --- a/packages/microcosm-build/src/microcosm/build/spec_engine/field_usage.py +++ b/packages/microcosm-build/src/microcosm/build/spec_engine/field_usage.py @@ -26,9 +26,9 @@ ) from .schemas import load_schema_registry -EXPECTED_AUTHORED_FIELD_COUNT = 32_351 -EXPECTED_RESOLVED_BINDING_FIELD_COUNT = 9_769 -EXPECTED_CONFIGURATION_FIELD_COUNT = 42_120 +EXPECTED_AUTHORED_FIELD_COUNT = 32_384 +EXPECTED_RESOLVED_BINDING_FIELD_COUNT = 9_770 +EXPECTED_CONFIGURATION_FIELD_COUNT = 42_154 class FieldUsageError(AssertionError): @@ -425,12 +425,12 @@ def _path_inventory(rows: Sequence[tuple[str, object]]) -> tuple[int, str]: "e1dd7dc5123ab0f39d08ea4939d98dd09a6fdb8e7449a7ca3125fb1ddbd5b4e9", ), "imputation_producer_graph": ( - 24_473, - "1b9f7ffb88d31d2bd365c092d04a96038b2d5a2b03f4f2a79f42e95472b213d6", + 24_488, + "6382a3ed07016414c31462b8bedb24590d92b792405a89c927d85fa1bd2a8347", ), "imputation_transfer_execution": ( - 79, - "1575513251c72be60edc75f378221fa878ea76e009512f1352a111586d0a60f8", + 97, + "cabaaa3d96c534f5ec37f51811c3f549d0ed20791e2892c868888c0da6d7d80e", ), "imputation_waiver_records": ( 70, @@ -457,8 +457,8 @@ def _path_inventory(rows: Sequence[tuple[str, object]]) -> tuple[int, str]: "6a781915fd491d2c4b16d2b7d482f69cf362c904130093c59f9629f7a319269b", ), "resolved_seed_protocol": ( - 823, - "c05ebb88538195798e9365f325381c14d92b74007f846f1fef50a08268e6d27d", + 824, + "7537385c3fd399a2dbb7dcd8ed7cf1ff2481ed336db621eafcbfd741d5792f40", ), "resolved_seed_site_bindings": ( 277, diff --git a/packages/microcosm-build/src/microcosm/build/spec_engine/imputation_semantics.py b/packages/microcosm-build/src/microcosm/build/spec_engine/imputation_semantics.py index d80fba823..ef6bbf106 100644 --- a/packages/microcosm-build/src/microcosm/build/spec_engine/imputation_semantics.py +++ b/packages/microcosm-build/src/microcosm/build/spec_engine/imputation_semantics.py @@ -196,6 +196,25 @@ def block_columns(block_id: object) -> list[object]: profile = _mapping_like(profiles[profile_id], f"transfer profile {profile_id}") derive_schedule_d = bool(profile["derive_schedule_d"]) target_set = set(targets) + structural_policies = _mapping_like( + result["structural_target_policies"], + "transfer structural target policies", + ) + resolved_structural_policies: dict[str, object] = {} + for policy_id, policy_value in structural_policies.items(): + policy = deepcopy( + dict( + _mapping_like( + policy_value, + f"transfer structural target policy {policy_id}", + ) + ) + ) + policy.pop("sha256", None) + policy["enabled"] = policy.get("target") in target_set + policy["sha256"] = _canonical_sha256(policy) + resolved_structural_policies[policy_id] = policy + result["structural_target_policies"] = resolved_structural_policies post_transfer: dict[str, object] = {} features = _mapping_like( transfer_execution["post_transfer_features"], "post-transfer features" diff --git a/packages/microcosm-build/src/microcosm/build/spec_engine/inventory_coverage.py b/packages/microcosm-build/src/microcosm/build/spec_engine/inventory_coverage.py index b48afbf58..6e214e980 100644 --- a/packages/microcosm-build/src/microcosm/build/spec_engine/inventory_coverage.py +++ b/packages/microcosm-build/src/microcosm/build/spec_engine/inventory_coverage.py @@ -348,19 +348,19 @@ EXPECTED_HASHES = { "acs_group_predictors": "a927bb7ecf3e84f54c93583ab79318654514ac546aefafba67da5285615fbd60", "acs_person_predictors": "878c788a6f037d7aca12b3586ea034eff04f3034ffa11935a736493042551f25", - "authority": "3a980927227704d0589f246eef9cd825c2ae84f3a4134ac835e0e5ed39a563ac", + "authority": "e660a8ce42b69a39d29c5f0ec37264bc69d61b03f27adc386336ec8889531bb2", "early_families": "4aa9f736fd76e83955477ad1667e58f48f264783f05bdc7f0102cd32d61323bd", - "full_checkpoint": "a128a85f877fb32def9382b841b8b340f974e8a9148ac029c1f04becdc956c18", + "full_checkpoint": "b88f2d9c0f6f92c6cd81eb14d6b126afe59577b8bb392b394b2c6fbbafd195c5", "gap_fill_schedule": "1c31f9868f7884347cc19cf1ff65da43f950b9114941a715bab168246db414a7", - "graph_nodes": "7125ad28ae2c69f22094a574bbf6ed2ddf1682a2c2c3b416f8f49304b7016ce7", + "graph_nodes": "271a7bb8d0b3f97ff344e0b7e68184fa74738a6585c24fc8781793db669f388b", "geography_assignment": "f49425ca8734ac559c73cf44f6458d86d3162a48956b98a27e6e758959361585", "late_families": "d91f9ff0eb52f43e7b6eed3d5c58c37abe1620c3a11021da15dae9c10e16d382", - "late_resource_semantics": "3850554eb804fde5e4f86a34ac1bb8a7a07aafff7e8b48396a3d5fca844798e8", - "late_schedule": "dcf3c6d2eade3449836c49a1dc4d3b8cd395aab9142db700c3c60598fa9c1c79", + "late_resource_semantics": "afebb6725373abf5b8dd4fdb77bf2814cb6fcc569cb606c0c30963a8f65c0bab", + "late_schedule": "e59c019d3d454eac99ac0ac209b6c5b6faaf9bdfcaeee18c36a25be19bf7da2f", "ownership": "5f64f0aac49e2313177564f71876bffc8c81b3ded4df701e70930e60e9c98356", "primary_tuples": "987b501c695e31f45521c4a178528f75ab3df22c09bc407b182213b2de99ee57", - "seed_map": "36a9d819ef196c312888591936d49c025b0407df9928440cceefabc5458f72af", - "seed_protocol": "15840b380329410a7094f60b0f1dad453457fc785859f0c372f2c8e2d59b0246", + "seed_map": "271c23eca777b63bbfecc5b0bdbfffee634e13f79dd37273172f5fadda4afefb", + "seed_protocol": "870f9b43766d4c11f6dc1538bf97e360c6a85dd0d4a4a68c149595dfc0902f0a", "source_manifest": "cd5ba8924d64da5425ee14cca82a774e3f4b2bb5aabe06df291cc3cc457287a9", "take_up": "fa186daea0f8dd641cc470e41d1a2953f887d45282ec990201298f47bedf8d4d", "tail": "ac92829c88a1a4fb6460d61190918d5d99c6c377fc8dd8f62f02b332d09bf59c", @@ -429,7 +429,7 @@ "primary_targets": 65, "producer_authored_outputs": 92, "producer_compiled_outputs": 227, - "producer_inputs": 2_742, + "producer_inputs": 2_744, "producer_nodes": 38, "producer_virtual_resources": 75, "release_rungs": 5, @@ -1005,7 +1005,7 @@ def add( "producer_inputs_exact", clauses={ "producer input rows differ": inputs_exact, - "input row count differs": input_count == 2742, + "input row count differs": input_count == 2744, }, homes=("/imputation/producer_graph/nodes/*/inputs",), consumers=( @@ -1013,7 +1013,7 @@ def add( "compiler_ir.node_slices", ), observed={"rows": input_count}, - expected={"rows": 2742, "relation": "source rows preserved exactly"}, + expected={"rows": 2744, "relation": "source rows preserved exactly"}, ) outputs_exact = set(expected_outputs) == set(compiled_by_id) and all( _json_equal( diff --git a/packages/microcosm-build/src/microcosm/build/spec_engine/schema/imputation.schema.json b/packages/microcosm-build/src/microcosm/build/spec_engine/schema/imputation.schema.json index 0c2a2548e..4a4a09a5a 100644 --- a/packages/microcosm-build/src/microcosm/build/spec_engine/schema/imputation.schema.json +++ b/packages/microcosm-build/src/microcosm/build/spec_engine/schema/imputation.schema.json @@ -613,6 +613,111 @@ "schema_version": { "type": "integer" }, + "structural_target_policies": { + "type": "object", + "additionalProperties": false, + "properties": { + "is_pregnant": { + "type": "object", + "additionalProperties": false, + "properties": { + "assembled_representative": { + "type": "object", + "additionalProperties": false, + "properties": { + "clone_index": { + "type": "integer", + "minimum": 0 + }, + "clone_index_column": { + "type": "string" + } + }, + "required": [ + "clone_index", + "clone_index_column" + ] + }, + "eligibility": { + "type": "object", + "additionalProperties": false, + "properties": { + "is_female": { + "const": true + }, + "maximum_age_inclusive": { + "type": "integer" + }, + "minimum_age_inclusive": { + "type": "integer" + } + }, + "required": [ + "is_female", + "maximum_age_inclusive", + "minimum_age_inclusive" + ] + }, + "enabled": { + "type": "boolean" + }, + "fanout": { + "type": "string" + }, + "final_clone_disagreement": { + "const": "refuse" + }, + "final_domain_violations": { + "const": "refuse" + }, + "ineligible_missing_value": { + "const": false + }, + "preexisting_clone_disagreement": { + "const": "refuse" + }, + "preexisting_domain_violations": { + "const": "refuse" + }, + "qrf_scope": { + "type": "string" + }, + "sha256": { + "$ref": "defs.schema.json#/$defs/sha256" + }, + "source_person_key_precedence": { + "type": "array", + "items": { + "type": "string" + }, + "minItems": 1, + "uniqueItems": true + }, + "target": { + "const": "is_pregnant" + } + }, + "required": [ + "assembled_representative", + "eligibility", + "enabled", + "fanout", + "final_clone_disagreement", + "final_domain_violations", + "ineligible_missing_value", + "preexisting_clone_disagreement", + "preexisting_domain_violations", + "qrf_scope", + "sha256", + "source_person_key_precedence", + "target" + ] + } + }, + "required": [ + "is_pregnant" + ] + }, "tenure_codes": { "type": "object", "additionalProperties": false, @@ -678,6 +783,7 @@ "profiles", "recipient_combined_sources", "schema_version", + "structural_target_policies", "tenure_codes" ] }, diff --git a/packages/microcosm-build/src/microcosm/build/spec_engine/seeds.py b/packages/microcosm-build/src/microcosm/build/spec_engine/seeds.py index 36e6c27e8..04fb87ad8 100644 --- a/packages/microcosm-build/src/microcosm/build/spec_engine/seeds.py +++ b/packages/microcosm-build/src/microcosm/build/spec_engine/seeds.py @@ -726,6 +726,7 @@ def _stable_site( "wic_claim_assignment", salt="would_claim_wic", key_grammar=( + "source:person_source_id_if_assembled_multispine", "source_year:source_household_id:source_person_id_if_complete", "else_support:person_support_source_id", "else_person:person_id", diff --git a/packages/microcosm-build/src/microcosm/build/us/spec/imputation.yaml b/packages/microcosm-build/src/microcosm/build/us/spec/imputation.yaml index f0a7ed69c..16ae791a1 100644 --- a/packages/microcosm-build/src/microcosm/build/us/spec/imputation.yaml +++ b/packages/microcosm-build/src/microcosm/build/us/spec/imputation.yaml @@ -408,6 +408,29 @@ transfer_execution: discrete_numeric_targets: - first_home_mortgage_origination_year - second_home_mortgage_origination_year + - weeks_unemployed + structural_target_policies: + is_pregnant: + enabled: false + target: is_pregnant + eligibility: + is_female: true + minimum_age_inclusive: 15 + maximum_age_inclusive: 44 + source_person_key_precedence: + - person_source_id + - person_id + assembled_representative: + clone_index_column: person_support_clone_index + clone_index: 0 + qrf_scope: one_eligible_representative_per_source_person + ineligible_missing_value: false + fanout: one_source_person_result_to_every_missing_clone + preexisting_domain_violations: refuse + preexisting_clone_disagreement: refuse + final_domain_violations: refuse + final_clone_disagreement: refuse + sha256: 3290b2dde9a077d1c42834b22e8b7c383afbd00748012fbea52ccf803a356f3a predictor_bindings: person_required: acs_person_required person_optional: @@ -10450,6 +10473,16 @@ producer_graph: producing_stage: post_clone_input_surface required_scope: asec_source tolerated_absence_receipts: [] + - alternatives: + - - column: person_source_id + entity: person + value_kind: finite_numeric + column: '@effective:pregnancy_structural_source_person_id' + entity: person + producing_stage: post_clone_input_surface + required_scope: asec_source + tolerated_absence_receipts: + - optional_input:source:with_us_pregnancy_inputs:pregnancy_structural_source_person_id - alternatives: - - column: A_AGE entity: person @@ -10621,6 +10654,15 @@ producer_graph: producing_stage: primary_puf_qrf required_scope: puf_clone tolerated_absence_receipts: [] + - alternatives: + - - column: person_source_id + entity: person + value_kind: non_null + column: person_source_id + entity: person + producing_stage: primary_puf_qrf + required_scope: puf_clone + tolerated_absence_receipts: [] - alternatives: - - column: person_spm_unit_id entity: person diff --git a/packages/microcosm-build/src/microcosm/build/us_runtime/__init__.py b/packages/microcosm-build/src/microcosm/build/us_runtime/__init__.py index e9e756473..518baacb6 100644 --- a/packages/microcosm-build/src/microcosm/build/us_runtime/__init__.py +++ b/packages/microcosm-build/src/microcosm/build/us_runtime/__init__.py @@ -593,6 +593,7 @@ PufTaxDetailChainInputs, clone_us_frame_for_puf_support, finalize_us_puf_tax_detail_predictions, + has_assembled_support_metadata, has_support_role_metadata, impute_us_puf_tax_detail_support, prepare_us_puf_tax_detail_chain_inputs, @@ -1953,6 +1954,7 @@ "validate_spine_agreement_registry", "assemble_spines", "PUF_TAX_DETAIL_CLONE_INDEX", + "has_assembled_support_metadata", "has_support_role_metadata", "puf_tax_detail_clone_mask", "spine_source_id_column", diff --git a/packages/microcosm-build/src/microcosm/build/us_runtime/acs_release_predictors.py b/packages/microcosm-build/src/microcosm/build/us_runtime/acs_release_predictors.py new file mode 100644 index 000000000..98ba4e4d9 --- /dev/null +++ b/packages/microcosm-build/src/microcosm/build/us_runtime/acs_release_predictors.py @@ -0,0 +1,1697 @@ +"""Release-time ACS crosswalks for CPS-named archived-model predictors. + +The stacked pool deliberately preserves native source columns rather than +pretending that ACS fields are CPS fields. Six release-stage archived donor +models nevertheless consume CPS-named predictors. Until a cold pool rebuild +can carry the reviewed harmonization, this module is the single origin-aware +release boundary that joins selected ACS people back to the exact 2024 one-year +PUMS archives and materializes only the bins those models consume. + +``person_source_id`` is an assembly identity, not a Census key. Assembly can +offset it to avoid cross-spine collisions. The semantic join therefore uses +the raw lineage retained by the pool: parent-household ``SERIALNO`` plus +integral person ``SPORDER``. ``person_source_id`` is used only after the +one-to-one raw join, to fan one source person's values to all support clones. + +The mappings below cite their executable consumers rather than inventing CPS +detail the models never read: + +* ``ssi_disability_criteria._ASEC_DIFFICULTY_SOURCE_COLUMNS`` consumes each + disability field only as ``== 1``; +* ``scf_wealth._recipient_cps_race`` and ``org_wages._derive_wbho`` consume + White, Black, Asian, Hispanic, and residual Other bins; +* ``sipp_tips.CENSUS_OCCUPATION_CODE_TO_TTOC`` consumes detailed Census + occupation codes directly; +* ``org_wages.FLSA_OVERTIME_OCCUPATION_CODES`` and its EAP set consume the + 53-category CPS detailed occupation recode; and +* ``sipp_vehicles._household_tenure_status`` consumes only mortgaged owner, + outright owner, and non-owner tenure codes. + +Every mapping is explicit and included in a canonical digest carried by the +release receipt. Changing one code or universe boundary requires a deliberate +digest repin and focused contract-test change. +""" + +from __future__ import annotations + +import hashlib +import json +import re +from collections.abc import Mapping, Sequence +from dataclasses import dataclass +from pathlib import Path +from typing import Any +from zipfile import ZipFile + +import numpy as np +import pandas as pd + +from microcosm.build.us_runtime.support_provenance import ( + support_channel_column, + support_clone_index_column, + support_source_id_column, + validate_assembly_provenance, +) +from microcosm.frame import US_SCHEMA, Frame + +__all__ = [ + "ACS_2024_HOUSEHOLD_ZIP_SHA256", + "ACS_2024_PERSON_ZIP_SHA256", + "ACS_DIFFICULTY_TO_CPS", + "ACS_OCCP_TO_POCCU2", + "ACS_RAC1P_TO_CONSUMED_PRDTRACE", + "ACS_RELEASE_PREDICTOR_CROSSWALK_SHA256", + "ACS_TEN_TO_SPM_TENMORTSTATUS", + "AcsReleasePredictorJoinResult", + "acs_release_predictor_crosswalk_payload", + "join_acs_release_predictors", +] + +ACS_2024_PERSON_ZIP_SHA256 = ( + "afdc6d90c6e2f0bab365ed32d95ba4c4d8ac651162f46ac7861295b2dc469894" +) +ACS_2024_HOUSEHOLD_ZIP_SHA256 = ( + "8281008e53de98f0ef81e7a2ee5a8725991dda1ecfd2713ead73246425e515d0" +) +ACS_RELEASE_PREDICTOR_CROSSWALK_VERSION = 1 +ACS_RELEASE_PREDICTOR_CROSSWALK_SHA256 = ( + "1d4906242e9c73e31b3283659e5cad8242b8cbc42914ab6fa59547a10c8770e9" +) + +_SHA256_PATTERN = re.compile(r"[0-9a-f]{64}") +_DEFAULT_CHUNKSIZE = 250_000 +_ACS_CHANNEL = "acs" +_ASEC_CHANNEL = "asec" +_ACS_VINTAGE = 2024 + +_PERSON_MEMBERS = ("psam_pusa.csv", "psam_pusb.csv") +_HOUSEHOLD_MEMBERS = ("psam_husa.csv", "psam_husb.csv") +_PERSON_RAW_COLUMNS = ( + "SERIALNO", + "SPORDER", + "AGEP", + "DEAR", + "DEYE", + "DREM", + "DPHY", + "DDRS", + "DOUT", + "RAC1P", + "HISP", + "OCCP", + "ESR", + "SSIP", + "ADJINC", +) +_HOUSEHOLD_RAW_COLUMNS = ("SERIALNO", "NP", "TYPEHUGQ", "TEN") + +ACS_DIFFICULTY_TO_CPS: Mapping[str, str] = { + "DDRS": "PEDISDRS", + "DEAR": "PEDISEAR", + "DEYE": "PEDISEYE", + "DOUT": "PEDISOUT", + "DPHY": "PEDISPHY", + "DREM": "PEDISREM", +} +_ACS_DIFFICULTY_MIN_AGE: Mapping[str, int] = { + "DEAR": 0, + "DEYE": 0, + "DREM": 5, + "DPHY": 5, + "DDRS": 5, + "DOUT": 15, +} + +# The consumers distinguish White (1), Black (2), Asian (4), and residual +# Other. ACS codes 3--5 are American Indian / Alaska Native detail, 7 is +# Native Hawaiian / Pacific Islander, 8 is another race, and 9 is multiracial; +# none has a separate consumed model bin, so code 3 is the canonical CPS Other +# representative rather than a fabricated detailed multiracial code. +ACS_RAC1P_TO_CONSUMED_PRDTRACE: Mapping[int, int] = { + 1: 1, + 2: 2, + 3: 3, + 4: 3, + 5: 3, + 6: 4, + 7: 3, + 8: 3, + 9: 3, +} + +# ACS HISP=1 is non-Hispanic. Codes 2--24 are Hispanic-origin detail; the +# consumers test only zero versus positive, so 1 is the canonical positive +# CPS representative. +_ACS_HISP_TO_CONSUMED_PRDTHSP: Mapping[int, int] = { + 1: 0, + **{code: 1 for code in range(2, 25)}, +} + +# 2024 Census detailed occupation codes mapped to the 53 POCCU2 categories +# used by the archived ORG/FLSA consumer. The table is intentionally explicit: +# the canonical archive pin and crosswalk digest jointly refuse a new Census +# code until its consumed category is reviewed. Blank OCCP is handled +# separately: PEIOOCC uses the CPS NIU sentinel -1, while POCCU2 preserves +# the ACS age universe rather than inventing an occupation for 15-year-olds. +ACS_OCCP_TO_POCCU2: Mapping[int, int] = { + 10: 1, + 20: 1, + 40: 1, + 51: 1, + 52: 1, + 60: 1, + 101: 1, + 102: 1, + 110: 1, + 120: 1, + 135: 2, + 136: 2, + 137: 2, + 140: 2, + 150: 2, + 160: 2, + 205: 2, + 220: 2, + 230: 3, + 300: 3, + 310: 3, + 335: 3, + 340: 3, + 350: 3, + 360: 3, + 410: 3, + 420: 3, + 425: 3, + 440: 3, + 500: 4, + 510: 5, + 520: 5, + 530: 5, + 540: 5, + 565: 5, + 600: 5, + 630: 5, + 640: 5, + 650: 5, + 700: 5, + 705: 5, + 710: 5, + 725: 5, + 726: 5, + 735: 5, + 750: 5, + 800: 6, + 810: 7, + 820: 7, + 830: 7, + 845: 7, + 850: 7, + 860: 7, + 900: 7, + 910: 7, + 930: 7, + 940: 7, + 960: 7, + 1005: 8, + 1006: 8, + 1007: 8, + 1010: 8, + 1021: 8, + 1022: 8, + 1031: 8, + 1032: 8, + 1050: 8, + 1065: 8, + 1105: 8, + 1106: 8, + 1108: 8, + 1200: 9, + 1220: 9, + 1240: 9, + 1305: 10, + 1306: 10, + 1310: 11, + 1320: 12, + 1340: 12, + 1350: 12, + 1360: 12, + 1400: 12, + 1410: 12, + 1420: 12, + 1430: 12, + 1440: 12, + 1450: 12, + 1460: 12, + 1520: 12, + 1530: 12, + 1541: 12, + 1545: 12, + 1551: 12, + 1555: 12, + 1560: 12, + 1600: 13, + 1610: 13, + 1640: 13, + 1650: 13, + 1700: 13, + 1710: 13, + 1720: 13, + 1745: 13, + 1750: 13, + 1760: 13, + 1800: 14, + 1821: 15, + 1822: 15, + 1825: 15, + 1840: 15, + 1860: 15, + 1900: 16, + 1910: 16, + 1920: 16, + 1935: 16, + 1970: 16, + 1980: 16, + 2001: 17, + 2002: 17, + 2003: 17, + 2004: 17, + 2005: 17, + 2006: 17, + 2011: 17, + 2012: 17, + 2013: 17, + 2014: 17, + 2015: 17, + 2016: 17, + 2025: 17, + 2040: 17, + 2050: 17, + 2060: 17, + 2100: 18, + 2105: 18, + 2145: 19, + 2170: 19, + 2180: 19, + 2205: 20, + 2300: 21, + 2310: 21, + 2320: 21, + 2330: 21, + 2350: 21, + 2360: 21, + 2400: 22, + 2435: 22, + 2440: 22, + 2545: 22, + 2555: 22, + 2600: 23, + 2631: 23, + 2632: 23, + 2633: 23, + 2634: 23, + 2635: 23, + 2636: 23, + 2640: 23, + 2700: 23, + 2710: 23, + 2721: 23, + 2722: 23, + 2723: 23, + 2740: 23, + 2751: 23, + 2752: 23, + 2755: 23, + 2770: 23, + 2805: 23, + 2810: 23, + 2825: 23, + 2830: 23, + 2840: 23, + 2850: 23, + 2861: 23, + 2862: 23, + 2865: 23, + 2905: 23, + 2910: 23, + 2920: 23, + 3000: 24, + 3010: 24, + 3030: 24, + 3040: 24, + 3050: 24, + 3090: 24, + 3100: 24, + 3110: 24, + 3120: 24, + 3140: 25, + 3150: 25, + 3160: 25, + 3200: 25, + 3210: 25, + 3220: 25, + 3230: 25, + 3245: 25, + 3250: 26, + 3255: 25, + 3256: 25, + 3258: 25, + 3261: 27, + 3270: 27, + 3300: 27, + 3310: 27, + 3321: 27, + 3322: 27, + 3323: 27, + 3324: 27, + 3330: 27, + 3401: 27, + 3402: 27, + 3421: 27, + 3422: 27, + 3423: 27, + 3424: 27, + 3430: 27, + 3500: 27, + 3515: 27, + 3520: 27, + 3545: 27, + 3550: 27, + 3601: 28, + 3602: 28, + 3603: 28, + 3605: 28, + 3610: 28, + 3620: 28, + 3630: 28, + 3640: 28, + 3645: 28, + 3646: 28, + 3647: 28, + 3648: 28, + 3649: 28, + 3655: 28, + 3700: 29, + 3710: 29, + 3720: 29, + 3725: 29, + 3740: 30, + 3750: 30, + 3801: 30, + 3802: 30, + 3820: 30, + 3840: 30, + 3870: 30, + 3900: 31, + 3910: 31, + 3930: 31, + 3940: 31, + 3945: 31, + 3946: 31, + 3960: 31, + 4000: 32, + 4010: 32, + 4020: 32, + 4030: 33, + 4040: 33, + 4055: 33, + 4110: 33, + 4120: 33, + 4130: 33, + 4140: 33, + 4150: 33, + 4160: 33, + 4200: 34, + 4210: 34, + 4220: 35, + 4230: 35, + 4240: 35, + 4251: 35, + 4252: 35, + 4255: 35, + 4330: 36, + 4340: 37, + 4350: 37, + 4400: 37, + 4420: 37, + 4435: 37, + 4461: 37, + 4465: 37, + 4500: 37, + 4510: 37, + 4521: 37, + 4522: 37, + 4525: 37, + 4530: 37, + 4540: 37, + 4600: 37, + 4621: 37, + 4622: 37, + 4640: 37, + 4655: 37, + 4700: 38, + 4710: 38, + 4720: 39, + 4740: 39, + 4750: 39, + 4760: 39, + 4800: 39, + 4810: 39, + 4820: 39, + 4830: 39, + 4840: 39, + 4850: 39, + 4900: 39, + 4920: 39, + 4930: 39, + 4940: 39, + 4950: 39, + 4965: 39, + 5000: 40, + 5010: 40, + 5020: 40, + 5040: 40, + 5100: 40, + 5110: 40, + 5120: 40, + 5140: 40, + 5150: 40, + 5160: 40, + 5165: 40, + 5220: 40, + 5230: 40, + 5240: 40, + 5250: 40, + 5260: 40, + 5300: 40, + 5310: 40, + 5320: 40, + 5330: 40, + 5340: 40, + 5350: 40, + 5360: 40, + 5400: 40, + 5410: 40, + 5420: 40, + 5500: 40, + 5510: 40, + 5521: 40, + 5522: 40, + 5530: 40, + 5540: 40, + 5550: 40, + 5560: 40, + 5600: 40, + 5610: 40, + 5630: 40, + 5710: 40, + 5720: 40, + 5730: 40, + 5740: 40, + 5810: 40, + 5820: 40, + 5840: 40, + 5850: 40, + 5860: 40, + 5900: 40, + 5910: 40, + 5920: 40, + 5940: 40, + 6005: 41, + 6010: 41, + 6040: 41, + 6050: 41, + 6115: 41, + 6120: 41, + 6130: 41, + 6200: 42, + 6210: 42, + 6220: 42, + 6230: 43, + 6240: 44, + 6250: 44, + 6260: 44, + 6305: 44, + 6330: 44, + 6355: 45, + 6360: 46, + 6400: 46, + 6410: 46, + 6441: 46, + 6442: 46, + 6460: 46, + 6515: 46, + 6520: 46, + 6530: 46, + 6540: 46, + 6600: 46, + 6660: 46, + 6700: 46, + 6710: 46, + 6720: 46, + 6730: 46, + 6740: 46, + 6765: 46, + 6800: 47, + 6825: 47, + 6835: 47, + 6850: 47, + 6950: 47, + 7000: 48, + 7010: 48, + 7020: 48, + 7030: 48, + 7040: 48, + 7100: 48, + 7120: 48, + 7130: 48, + 7140: 48, + 7150: 48, + 7160: 48, + 7200: 48, + 7210: 48, + 7220: 48, + 7240: 48, + 7260: 48, + 7300: 48, + 7315: 48, + 7320: 48, + 7330: 48, + 7340: 48, + 7350: 48, + 7360: 48, + 7410: 48, + 7420: 48, + 7430: 48, + 7510: 48, + 7540: 48, + 7560: 48, + 7610: 48, + 7640: 48, + 7700: 49, + 7720: 49, + 7730: 49, + 7740: 49, + 7750: 49, + 7800: 49, + 7810: 49, + 7830: 49, + 7840: 49, + 7850: 49, + 7855: 49, + 7905: 49, + 7925: 49, + 7950: 49, + 8000: 49, + 8025: 49, + 8030: 49, + 8040: 49, + 8100: 49, + 8130: 49, + 8140: 49, + 8225: 49, + 8250: 49, + 8255: 49, + 8256: 49, + 8300: 49, + 8310: 49, + 8320: 49, + 8335: 49, + 8350: 49, + 8365: 49, + 8450: 49, + 8465: 49, + 8500: 49, + 8510: 49, + 8530: 49, + 8540: 49, + 8555: 49, + 8600: 49, + 8610: 49, + 8620: 49, + 8630: 49, + 8640: 49, + 8650: 49, + 8710: 49, + 8720: 49, + 8730: 49, + 8740: 49, + 8750: 49, + 8760: 49, + 8800: 49, + 8810: 49, + 8830: 49, + 8850: 49, + 8910: 49, + 8920: 49, + 8930: 49, + 8940: 49, + 8950: 49, + 8990: 49, + 9005: 50, + 9030: 50, + 9040: 50, + 9050: 50, + 9110: 51, + 9121: 51, + 9122: 51, + 9130: 51, + 9141: 51, + 9142: 51, + 9150: 51, + 9210: 51, + 9240: 51, + 9265: 51, + 9300: 51, + 9310: 51, + 9350: 51, + 9365: 51, + 9410: 51, + 9415: 51, + 9430: 51, + 9510: 51, + 9570: 51, + 9600: 51, + 9610: 51, + 9620: 51, + 9630: 51, + 9640: 51, + 9645: 51, + 9650: 51, + 9720: 51, + 9760: 51, + 9800: 52, + 9810: 52, + 9825: 52, + 9830: 52, + 9920: 53, +} + +ACS_TEN_TO_SPM_TENMORTSTATUS: Mapping[int, int] = { + 1: 1, # owned with a mortgage or loan + 2: 2, # owned free and clear + 3: 3, # rented for cash + 4: 3, # occupied without cash rent: non-owner consumed bin +} + +_MODEL_PREDICTORS: Mapping[str, tuple[str, ...]] = { + "ssi_disability_criteria": ( + "PEDISDRS", + "PEDISEAR", + "PEDISEYE", + "PEDISOUT", + "PEDISPHY", + "PEDISREM", + ), + "scf_wealth": ("PRDTRACE", "PRDTHSP"), + "scf_auto_loans": ("PRDTRACE", "PRDTHSP"), + "sipp_vehicles": ("SPM_TENMORTSTATUS",), + "sipp_tips": ("PEIOOCC",), + "org_wages": ("PRDTRACE", "PRDTHSP", "POCCU2"), +} +_OUTPUT_COLUMNS = tuple( + dict.fromkeys( + column for columns in _MODEL_PREDICTORS.values() for column in columns + ) +) + + +@dataclass(frozen=True) +class AcsReleasePredictorJoinResult: + """A predictor-enriched frame and JSON-ready release receipt.""" + + frame: Frame + receipt: Mapping[str, Any] + + +def acs_release_predictor_crosswalk_payload() -> dict[str, Any]: + """Return the canonical, JSON-ready crosswalk specification.""" + + return { + "version": ACS_RELEASE_PREDICTOR_CROSSWALK_VERSION, + "disability": { + source: { + "target": ACS_DIFFICULTY_TO_CPS[source], + "minimum_question_age": _ACS_DIFFICULTY_MIN_AGE[source], + "codes": {"1": 1, "2": 2, "below_universe_blank": -1}, + } + for source in ACS_DIFFICULTY_TO_CPS + }, + "race": { + "RAC1P_to_consumed_PRDTRACE": { + str(key): value for key, value in ACS_RAC1P_TO_CONSUMED_PRDTRACE.items() + }, + "HISP_to_consumed_PRDTHSP": { + str(key): value for key, value in _ACS_HISP_TO_CONSUMED_PRDTHSP.items() + }, + }, + "occupation": { + "OCCP_to_PEIOOCC": { + "observed": "identity", + "blank": -1, + "blank_semantic": "CPS not-in-universe sentinel", + }, + "OCCP_to_POCCU2": { + str(key): value for key, value in ACS_OCCP_TO_POCCU2.items() + }, + "blank_OCCP_to_POCCU2": { + "age_below_16": 0, + "age_16_plus": 53, + "age_15_source_target_universe_gap": ( + "ACS OCCP is not asked; retain the out-of-universe sentinel " + "instead of fabricating CPS no-occupation code 53" + ), + }, + }, + "tenure": { + "TEN_to_SPM_TENMORTSTATUS": { + str(key): value for key, value in ACS_TEN_TO_SPM_TENMORTSTATUS.items() + }, + "group_quarters_blank": 3, + }, + "ssi_reporter_anchor": { + "source": "person.ssi_reported (native adjusted ACS SSIP)", + "target": "receiver-coalesced reported SSI anchor", + "consumer_semantic": "> 0", + "below_age_15_blank": "preserved", + }, + "model_predictors": { + model: list(columns) for model, columns in _MODEL_PREDICTORS.items() + }, + } + + +def _computed_crosswalk_sha256() -> str: + payload = json.dumps( + acs_release_predictor_crosswalk_payload(), + sort_keys=True, + separators=(",", ":"), + allow_nan=False, + ).encode() + return hashlib.sha256(payload).hexdigest() + + +def join_acs_release_predictors( + frame: Frame, + *, + person_zip: str | Path | None, + person_sha256: str | None, + household_zip: str | Path | None, + household_sha256: str | None, + chunksize: int = _DEFAULT_CHUNKSIZE, +) -> AcsReleasePredictorJoinResult: + """Populate CPS-named predictors for every physical ACS support row. + + Frames without an assembled ACS channel pass through by identity and do + not require archive options. An assembled frame with any ACS row requires + all four explicit CLI values and the two canonical 2024 archive pins. + """ + + if frame.schema != US_SCHEMA: + raise ValueError("ACS release predictor join requires the US schema.") + if chunksize <= 0: + raise ValueError("ACS release predictor join chunksize must be positive.") + person = frame.table("person") + channel_column = support_channel_column("person") + if ( + channel_column not in person + or not person[channel_column].eq(_ACS_CHANNEL).any() + ): + provided = [person_zip, person_sha256, household_zip, household_sha256] + if any(value is not None for value in provided): + raise ValueError( + "ACS release archive options were provided for a frame with no " + "physical ACS source rows." + ) + return AcsReleasePredictorJoinResult( + frame=frame, + receipt={"enabled": False, "reason": "no physical ACS source rows"}, + ) + + missing_options = [ + name + for name, value in ( + ("person_zip", person_zip), + ("person_sha256", person_sha256), + ("household_zip", household_zip), + ("household_sha256", household_sha256), + ) + if value is None + ] + if missing_options: + raise ValueError( + "Physical ACS rows require all pinned release archive options; " + f"missing {missing_options}." + ) + assert person_zip is not None + assert person_sha256 is not None + assert household_zip is not None + assert household_sha256 is not None + + actual_crosswalk_sha256 = _computed_crosswalk_sha256() + if actual_crosswalk_sha256 != ACS_RELEASE_PREDICTOR_CROSSWALK_SHA256: + raise RuntimeError( + "ACS release predictor crosswalk identity is stale: expected " + f"{ACS_RELEASE_PREDICTOR_CROSSWALK_SHA256}, got " + f"{actual_crosswalk_sha256}." + ) + person_identity = _verify_archive( + person_zip, + expected_sha256=person_sha256, + canonical_sha256=ACS_2024_PERSON_ZIP_SHA256, + label="ACS 2024 person zip", + ) + household_identity = _verify_archive( + household_zip, + expected_sha256=household_sha256, + canonical_sha256=ACS_2024_HOUSEHOLD_ZIP_SHA256, + label="ACS 2024 household zip", + ) + + canonical, acs_rows, clone_counts = _canonical_pool_acs_people(frame) + selected_serials = frozenset(canonical["SERIALNO"].astype(str)) + raw_household = _read_filtered_archive( + Path(household_zip), + expected_members=_HOUSEHOLD_MEMBERS, + columns=_HOUSEHOLD_RAW_COLUMNS, + selected_serials=selected_serials, + chunksize=chunksize, + label="ACS household", + ) + tenure_by_serial = _validated_tenure_by_serial(canonical, raw_household) + + raw_person = _read_filtered_archive( + Path(person_zip), + expected_members=_PERSON_MEMBERS, + columns=_PERSON_RAW_COLUMNS, + selected_serials=selected_serials, + chunksize=chunksize, + label="ACS person", + ) + raw_person["SPORDER"] = _required_integral( + raw_person["SPORDER"], label="raw ACS SPORDER", minimum=1 + ) + duplicate_raw_people = raw_person.duplicated(["SERIALNO", "SPORDER"], keep=False) + if duplicate_raw_people.any(): + examples = ( + raw_person.loc[duplicate_raw_people, ["SERIALNO", "SPORDER"]] + .head() + .to_dict("records") + ) + raise ValueError(f"ACS raw person key collision(s): {examples}.") + + joined = canonical.merge( + raw_person, + on=["SERIALNO", "SPORDER"], + how="left", + validate="one_to_one", + indicator=True, + sort=False, + ) + unmatched = joined["_merge"].ne("both") + if unmatched.any(): + examples = ( + joined.loc[unmatched, ["person_source_id", "SERIALNO", "SPORDER"]] + .head() + .to_dict("records") + ) + raise ValueError( + "ACS release person join is not total over pool source people; " + f"unmatched={int(unmatched.sum())}, examples={examples}." + ) + joined = joined.drop(columns="_merge") + if len(joined) != len(canonical): # pragma: no cover - merge validation guard + raise AssertionError("ACS release person join changed canonical row count.") + + mapped = _crosswalk_people(joined) + mapped["SPM_TENMORTSTATUS"] = joined["SERIALNO"].map(tenure_by_serial).to_numpy() + _validate_canonical_ssi_reporter_values(frame, canonical, joined) + if mapped.loc[:, list(_OUTPUT_COLUMNS)].isna().any().any(): + missing = { + column: int(mapped[column].isna().sum()) + for column in _OUTPUT_COLUMNS + if mapped[column].isna().any() + } + raise ValueError( + "ACS release crosswalk must populate every consumed predictor; " + f"missing={missing}." + ) + + tables = {entity: frame.table(entity).copy() for entity in frame.entities} + output_person = tables["person"] + _require_asec_native_predictors(output_person) + by_source = mapped.set_index("person_source_id") + acs_mask = output_person[channel_column].eq(_ACS_CHANNEL) + source_ids = _required_integral( + output_person.loc[acs_mask, support_source_id_column("person")], + label="ACS person_source_id", + minimum=0, + ) + for column in _OUTPUT_COLUMNS: + values = source_ids.map(by_source[column]) + if values.isna().any(): # pragma: no cover - canonical totality guard + raise AssertionError(f"ACS clone fan-out lost {column!r} value(s).") + if column in output_person: + current = pd.to_numeric( + output_person.loc[acs_mask, column], errors="coerce" + ) + observed = current.notna() + if observed.any() and not np.array_equal( + current.loc[observed].to_numpy(dtype=np.float64), + values.loc[observed].to_numpy(dtype=np.float64), + ): + raise ValueError( + "ACS release predictor join refuses to overwrite conflicting " + f"pre-existing ACS {column!r} values." + ) + else: + output_person[column] = np.nan + output_person.loc[acs_mask, column] = values.to_numpy() + + enriched = Frame( + tables, + frame.schema, + {entity: frame.weights_for(entity) for entity in frame.weighted_entities}, + frame.strata, + mass_log=frame.mass_log, + metadata=frame.metadata, + ) + receipt = _receipt( + enriched, + person_identity=person_identity, + household_identity=household_identity, + canonical=canonical, + raw_person_rows=len(raw_person), + raw_household_rows=len(raw_household), + acs_rows=acs_rows, + clone_counts=clone_counts, + ) + return AcsReleasePredictorJoinResult(frame=enriched, receipt=receipt) + + +def _verify_archive( + path: str | Path, + *, + expected_sha256: str, + canonical_sha256: str, + label: str, +) -> dict[str, Any]: + if not isinstance(expected_sha256, str) or not _SHA256_PATTERN.fullmatch( + expected_sha256 + ): + raise ValueError(f"{label} SHA-256 must be 64 lowercase hex characters.") + if expected_sha256 != canonical_sha256: + raise ValueError( + f"{label} pin must be the reviewed {canonical_sha256}; got " + f"{expected_sha256}." + ) + archive = Path(path) + if not archive.is_file(): + raise FileNotFoundError(f"{label} not found: {archive}") + digest = hashlib.sha256() + size = 0 + with archive.open("rb") as stream: + for chunk in iter(lambda: stream.read(1024 * 1024), b""): + digest.update(chunk) + size += len(chunk) + actual = digest.hexdigest() + if actual != expected_sha256: + raise ValueError( + f"{label} SHA-256 mismatch: expected {expected_sha256}, got {actual}." + ) + return {"path": str(archive), "sha256": actual, "size_bytes": size} + + +def _read_filtered_archive( + path: Path, + *, + expected_members: Sequence[str], + columns: Sequence[str], + selected_serials: frozenset[str], + chunksize: int, + label: str, +) -> pd.DataFrame: + pieces: list[pd.DataFrame] = [] + with ZipFile(path) as archive: + csv_members = { + Path(name).name.lower(): name + for name in archive.namelist() + if name.lower().endswith(".csv") + and Path(name) + .name.lower() + .startswith("psam_pus" if "person" in label else "psam_hus") + } + expected = tuple(name.lower() for name in expected_members) + if tuple(sorted(csv_members)) != tuple(sorted(expected)): + raise ValueError( + f"{label} archive members differ: expected {list(expected)}, " + f"got {sorted(csv_members)}." + ) + for basename in expected: + member_name = csv_members[basename] + with archive.open(member_name) as member: + header = pd.read_csv(member, nrows=0).columns.tolist() + missing = sorted(set(columns) - set(header)) + if missing: + raise ValueError( + f"{label} member {member_name!r} missing column(s): {missing}." + ) + with archive.open(member_name) as member: + for chunk in pd.read_csv( + member, + usecols=list(columns), + dtype={"SERIALNO": "string"}, + chunksize=chunksize, + low_memory=False, + ): + retained = chunk.loc[chunk["SERIALNO"].isin(selected_serials)] + if not retained.empty: + pieces.append(retained) + if not pieces: + raise ValueError(f"{label} archive contains no selected pool records.") + result = pd.concat(pieces, ignore_index=True) + if result["SERIALNO"].isna().any(): + raise ValueError(f"{label} selected SERIALNO values must not be blank.") + result["SERIALNO"] = result["SERIALNO"].astype(str) + return result + + +def _canonical_pool_acs_people( + frame: Frame, +) -> tuple[pd.DataFrame, int, dict[str, int]]: + validate_assembly_provenance( + frame, + boundary="ACS release predictor join", + ) + person = frame.table("person") + household = frame.table("household") + person_channel = support_channel_column("person") + household_channel = support_channel_column("household") + person_clone = support_clone_index_column("person") + household_clone = support_clone_index_column("household") + person_source = support_source_id_column("person") + required_person = { + "person_household_id", + "person_spine_source_id", + person_source, + person_channel, + person_clone, + "source_row_id", + "source_year", + "source_household_id", + "source_person_id", + "SPORDER", + "age", + "ssi_reported", + } + required_household = { + "household_id", + "household_spine_source_id", + "household_source_id", + household_channel, + household_clone, + "SERIALNO", + "TEN", + } + missing_person = sorted(required_person - set(person.columns)) + missing_household = sorted(required_household - set(household.columns)) + if missing_person or missing_household: + raise ValueError( + "ACS release predictor join requires complete pool lineage; " + f"missing person={missing_person}, household={missing_household}." + ) + observed_channels = set(person[person_channel].astype(str).unique()) + if observed_channels != {_ASEC_CHANNEL, _ACS_CHANNEL}: + raise ValueError( + "ACS release predictor receipts require exact ASEC/ACS physical " + f"channels; got {sorted(observed_channels)}." + ) + acs_mask = person[person_channel].eq(_ACS_CHANNEL) + acs_rows = int(acs_mask.sum()) + clone_index = _required_integral( + person.loc[acs_mask, person_clone], + label="ACS person clone index", + minimum=0, + ) + source_id = _required_integral( + person.loc[acs_mask, person_source], + label="ACS person_source_id", + minimum=0, + ) + duplicate_clone = pd.DataFrame( + {"person_source_id": source_id, "clone_index": clone_index} + ).duplicated(keep=False) + if duplicate_clone.any(): + examples = ( + pd.DataFrame({"person_source_id": source_id, "clone_index": clone_index}) + .loc[duplicate_clone] + .head() + .to_dict("records") + ) + raise ValueError( + "ACS pool has duplicate (person_source_id, clone_index) " + f"collision(s): {examples}." + ) + + selected = person.loc[ + acs_mask, + [ + "person_household_id", + "person_spine_source_id", + person_source, + person_clone, + "source_row_id", + "source_year", + "source_household_id", + "source_person_id", + "SPORDER", + ], + ].copy() + selected[person_source] = source_id.to_numpy() + selected[person_clone] = clone_index.to_numpy() + selected["_pool_row"] = selected.index + linked = selected.merge( + household.loc[ + :, + [ + "household_id", + "household_spine_source_id", + "household_source_id", + household_channel, + household_clone, + "SERIALNO", + "TEN", + ], + ], + left_on="person_household_id", + right_on="household_id", + how="left", + validate="many_to_one", + indicator=True, + sort=False, + ) + if linked["_merge"].ne("both").any(): + raise ValueError("ACS pool person rows contain orphan household links.") + linked = linked.drop(columns="_merge") + if not linked[household_channel].eq(_ACS_CHANNEL).all(): + raise ValueError("ACS pool person/household physical channels disagree.") + household_clone_values = _required_integral( + linked[household_clone], label="ACS household clone index", minimum=0 + ) + if not np.array_equal( + linked[person_clone].to_numpy(dtype=np.int64), + household_clone_values.to_numpy(dtype=np.int64), + ): + raise ValueError("ACS pool person/household clone indices disagree.") + if linked["SERIALNO"].isna().any(): + raise ValueError("ACS pool household SERIALNO values must be complete.") + linked["SERIALNO"] = linked["SERIALNO"].astype(str) + source_person = _required_integral( + linked["source_person_id"], label="ACS source_person_id", minimum=1 + ) + sporder = _required_integral(linked["SPORDER"], label="ACS pool SPORDER", minimum=1) + if not np.array_equal(source_person.to_numpy(), sporder.to_numpy()): + raise ValueError("ACS pool source_person_id does not equal integral SPORDER.") + source_year = _required_integral( + linked["source_year"], label="ACS source_year", minimum=_ACS_VINTAGE + ) + if not source_year.eq(_ACS_VINTAGE).all(): + raise ValueError("ACS release predictor join is pinned to source_year=2024.") + spine_id = _required_integral( + linked["person_spine_source_id"], + label="ACS person_spine_source_id", + minimum=0, + ) + row_id = _required_integral( + linked["source_row_id"], label="ACS source_row_id", minimum=0 + ) + if not np.array_equal(spine_id.to_numpy(), row_id.to_numpy()): + raise ValueError( + "ACS pool raw ordinal contract failed: person_spine_source_id must " + "equal source_row_id." + ) + source_household = _required_integral( + linked["source_household_id"], + label="ACS source_household_id", + minimum=1, + ) + household_spine = _required_integral( + linked["household_spine_source_id"], + label="ACS household_spine_source_id", + minimum=1, + ) + if not np.array_equal(source_household.to_numpy(), household_spine.to_numpy()): + raise ValueError( + "ACS pool household lineage failed: source_household_id must equal " + "the linked household_spine_source_id." + ) + + invariant_columns = [ + "person_spine_source_id", + "source_row_id", + "source_household_id", + "source_person_id", + "SPORDER", + "SERIALNO", + ] + conflicting_sources = [] + for column in invariant_columns: + counts = linked.groupby(person_source, sort=False)[column].nunique(dropna=False) + if counts.gt(1).any(): + conflicting_sources.extend(counts.index[counts.gt(1)].tolist()[:5]) + if conflicting_sources: + raise ValueError( + "ACS person_source_id maps to conflicting raw identities: " + f"{sorted(set(map(int, conflicting_sources)))[:5]}." + ) + native_counts = linked.loc[linked[person_clone].eq(0), person_source].value_counts() + all_sources = pd.Index(linked[person_source].unique()) + invalid_native = native_counts.reindex(all_sources, fill_value=0).ne(1) + if invalid_native.any(): + raise ValueError( + "Every ACS person_source_id must have exactly one clone-index-zero row." + ) + + canonical = linked.loc[linked[person_clone].eq(0)].copy() + canonical["person_source_id"] = canonical[person_source].astype("int64") + canonical["SPORDER"] = sporder.loc[canonical.index].to_numpy(dtype=np.int64) + duplicate_semantic = canonical.duplicated(["SERIALNO", "SPORDER"], keep=False) + if duplicate_semantic.any(): + examples = ( + canonical.loc[duplicate_semantic, ["SERIALNO", "SPORDER"]] + .head() + .to_dict("records") + ) + raise ValueError(f"ACS pool semantic person key collision(s): {examples}.") + clone_counts = { + str(int(index)): int(count) + for index, count in clone_index.value_counts().sort_index().items() + } + return canonical, acs_rows, clone_counts + + +def _validated_tenure_by_serial( + canonical: pd.DataFrame, + raw_household: pd.DataFrame, +) -> pd.Series: + duplicate = raw_household["SERIALNO"].duplicated(keep=False) + if duplicate.any(): + examples = raw_household.loc[duplicate, "SERIALNO"].head().tolist() + raise ValueError(f"ACS raw household SERIALNO collision(s): {examples}.") + expected_serials = set(canonical["SERIALNO"].astype(str)) + observed_serials = set(raw_household["SERIALNO"].astype(str)) + if observed_serials != expected_serials: + raise ValueError( + "ACS raw household join is not exact over selected serials; " + f"missing={sorted(expected_serials - observed_serials)[:5]}, " + f"extra={sorted(observed_serials - expected_serials)[:5]}." + ) + people = _required_integral(raw_household["NP"], label="ACS NP", minimum=1) + if people.le(0).any(): # pragma: no cover - minimum guard + raise ValueError("Selected ACS households must be occupied.") + kind = _required_integral( + raw_household["TYPEHUGQ"], label="ACS TYPEHUGQ", minimum=1 + ) + if not kind.isin([1, 2, 3]).all(): + bad = sorted(kind.loc[~kind.isin([1, 2, 3])].unique().tolist()) + raise ValueError(f"ACS TYPEHUGQ contains unsupported code(s): {bad}.") + raw_tenure = pd.to_numeric(raw_household["TEN"], errors="coerce") + housing_unit = kind.eq(1) + invalid_hu = housing_unit & ~raw_tenure.isin(ACS_TEN_TO_SPM_TENMORTSTATUS) + invalid_gq = ~housing_unit & raw_tenure.notna() + if invalid_hu.any() or invalid_gq.any(): + raise ValueError( + "ACS TEN/TYPEHUGQ universe mismatch: housing units require TEN 1--4 " + "and group quarters require blank TEN." + ) + + pool = canonical.loc[:, ["SERIALNO", "TEN"]].drop_duplicates("SERIALNO") + if pool["SERIALNO"].duplicated().any(): # pragma: no cover - drop guard + raise AssertionError("Canonical pool serial deduplication failed.") + comparison = pool.merge( + raw_household.loc[:, ["SERIALNO", "TEN"]], + on="SERIALNO", + how="left", + validate="one_to_one", + suffixes=("_pool", "_raw"), + ) + pool_tenure = pd.to_numeric(comparison["TEN_pool"], errors="coerce") + raw_tenure_aligned = pd.to_numeric(comparison["TEN_raw"], errors="coerce") + equal = (pool_tenure.isna() & raw_tenure_aligned.isna()) | pool_tenure.eq( + raw_tenure_aligned + ) + if not equal.all(): + examples = comparison.loc[~equal, ["SERIALNO", "TEN_pool", "TEN_raw"]] + raise ValueError( + "Pool household TEN disagrees with pinned ACS archive; examples=" + f"{examples.head().to_dict('records')}." + ) + + mapped = raw_tenure.map(ACS_TEN_TO_SPM_TENMORTSTATUS) + mapped.loc[~housing_unit] = 3 + if mapped.isna().any(): # pragma: no cover - universe guards above + raise AssertionError("ACS tenure crosswalk produced missing values.") + return pd.Series( + mapped.to_numpy(dtype=np.int16), + index=raw_household["SERIALNO"].astype(str), + ) + + +def _crosswalk_people(joined: pd.DataFrame) -> pd.DataFrame: + result = pd.DataFrame( + {"person_source_id": joined["person_source_id"].to_numpy()}, + index=joined.index, + ) + age = _required_integral(joined["AGEP"], label="ACS AGEP", minimum=0) + for source, target in ACS_DIFFICULTY_TO_CPS.items(): + values = pd.to_numeric(joined[source], errors="coerce") + in_universe = age.ge(_ACS_DIFFICULTY_MIN_AGE[source]) + invalid = (in_universe & ~values.isin([1, 2])) | (~in_universe & values.notna()) + if invalid.any(): + bad = joined.loc[invalid, ["SERIALNO", "SPORDER", "AGEP", source]] + raise ValueError( + f"ACS {source} code/universe mismatch; examples=" + f"{bad.head().to_dict('records')}." + ) + result[target] = np.select( + [values.eq(1), values.eq(2)], + [1, 2], + default=-1, + ).astype(np.int16) + + race = _required_integral(joined["RAC1P"], label="ACS RAC1P", minimum=1) + unknown_race = sorted(set(race) - set(ACS_RAC1P_TO_CONSUMED_PRDTRACE)) + if unknown_race: + raise ValueError(f"ACS RAC1P contains unsupported code(s): {unknown_race}.") + result["PRDTRACE"] = race.map(ACS_RAC1P_TO_CONSUMED_PRDTRACE).to_numpy( + dtype=np.int16 + ) + hisp = _required_integral(joined["HISP"], label="ACS HISP", minimum=1) + unknown_hisp = sorted(set(hisp) - set(_ACS_HISP_TO_CONSUMED_PRDTHSP)) + if unknown_hisp: + raise ValueError(f"ACS HISP contains unsupported code(s): {unknown_hisp}.") + result["PRDTHSP"] = hisp.map(_ACS_HISP_TO_CONSUMED_PRDTHSP).to_numpy(dtype=np.int16) + + occupation_raw = joined["OCCP"] + occupation = pd.to_numeric(occupation_raw, errors="coerce") + employment_raw = joined["ESR"] + employment = pd.to_numeric(employment_raw, errors="coerce") + invalid_employment = ( + (employment_raw.notna() & employment.isna()) + | (age.lt(16) & employment.notna()) + | (age.ge(16) & ~employment.isin(range(1, 7))) + ) + if invalid_employment.any(): + bad = joined.loc[invalid_employment, ["SERIALNO", "SPORDER", "AGEP", "ESR"]] + raise ValueError( + "ACS ESR code/universe mismatch (blank below age 16; codes 1--6 " + "from age 16); examples=" + f"{bad.head().to_dict('records')}." + ) + invalid_occupation = occupation_raw.notna() & occupation.isna() + invalid_observed_universe = occupation.notna() & age.lt(16) + invalid_blank = occupation.isna() & age.ge(16) & ~employment.eq(6) + if invalid_occupation.any() or invalid_observed_universe.any(): + bad_mask = invalid_occupation | invalid_observed_universe + bad = joined.loc[ + bad_mask, + ["SERIALNO", "SPORDER", "AGEP", "OCCP", "ESR"], + ] + raise ValueError( + "ACS OCCP code/universe mismatch (blank below age 16); examples=" + f"{bad.head().to_dict('records')}." + ) + if invalid_blank.any(): + bad = joined.loc[ + invalid_blank, + ["SERIALNO", "SPORDER", "AGEP", "OCCP", "ESR"], + ] + raise ValueError( + "ACS OCCP is blank inside its observed employment universe; examples=" + f"{bad.head().to_dict('records')}." + ) + observed = occupation.notna() + observed_values = occupation.loc[observed].to_numpy(dtype=np.float64) + if ( + not np.isfinite(observed_values).all() + or not np.equal(observed_values, np.floor(observed_values)).all() + ): + raise ValueError("ACS OCCP contains nonfinite or non-integer code(s).") + occupation_codes = occupation.fillna(-1).astype(np.int64) + unknown_occupation = sorted( + set(occupation_codes.loc[observed]) - set(ACS_OCCP_TO_POCCU2) + ) + if unknown_occupation: + raise ValueError( + f"ACS OCCP contains unsupported code(s): {unknown_occupation}." + ) + result["PEIOOCC"] = occupation_codes.to_numpy(dtype=np.int16) + poccu2 = occupation.map(ACS_OCCP_TO_POCCU2) + # ACS OCCP starts at age 16, one year later than CPS POCCU2. Preserve the + # ACS NIU state for age 15 instead of fabricating no-occupation evidence. + # From age 16, a blank is admitted only for ESR=6 and maps to the consumed + # no-occupation / never-worked bin 53. + poccu2.loc[occupation.isna()] = np.where(age.loc[occupation.isna()].ge(16), 53, 0) + result["POCCU2"] = poccu2.to_numpy(dtype=np.int16) + return result + + +def _validate_canonical_ssi_reporter_values( + frame: Frame, + canonical: pd.DataFrame, + joined: pd.DataFrame, +) -> None: + person = frame.table("person") + source_id_column = support_source_id_column("person") + channel_column = support_channel_column("person") + clone_column = support_clone_index_column("person") + native = person.loc[ + person[channel_column].eq(_ACS_CHANNEL) + & pd.to_numeric(person[clone_column], errors="coerce").eq(0), + [source_id_column, "age", "ssi_reported"], + ].copy() + if native[source_id_column].duplicated().any(): + raise ValueError("ACS native SSI reporter rows collide by person_source_id.") + age = pd.to_numeric(native["age"], errors="coerce") + reported = pd.to_numeric(native["ssi_reported"], errors="coerce") + invalid_blank = reported.isna() & age.ge(15) + invalid_observed = reported.notna() & ( + ~np.isfinite(reported.to_numpy(dtype=np.float64)) | reported.lt(0) + ) + invalid_child = reported.notna() & age.lt(15) + if invalid_blank.any() or invalid_observed.any() or invalid_child.any(): + raise ValueError( + "ACS native ssi_reported violates its age-15 amount universe or " + "finite nonnegative contract." + ) + canonical_ids = set(canonical["person_source_id"].astype(int)) + native_ids = set( + _required_integral( + native[source_id_column], + label="ACS SSI person_source_id", + minimum=0, + ) + ) + if canonical_ids != native_ids: + raise ValueError( + "ACS native ssi_reported source identities do not exactly cover the " + "canonical raw join." + ) + by_source = pd.Series(reported.to_numpy(), index=native[source_id_column]) + pool_aligned = canonical["person_source_id"].map(by_source).to_numpy( + dtype=np.float64 + ) + + raw_ssip = pd.to_numeric(joined["SSIP"], errors="coerce") + raw_adjinc = pd.to_numeric(joined["ADJINC"], errors="coerce") + raw_age = _required_integral(joined["AGEP"], label="ACS AGEP", minimum=0) + invalid_ssip = ( + (joined["SSIP"].notna() & raw_ssip.isna()) + | (raw_ssip.notna() & ~np.isfinite(raw_ssip.fillna(0.0))) + | raw_ssip.lt(0) + | (raw_age.lt(15) & raw_ssip.notna()) + | (raw_age.ge(15) & raw_ssip.isna()) + ) + invalid_adjinc = raw_ssip.notna() & ( + raw_adjinc.isna() | ~np.isfinite(raw_adjinc.fillna(0.0)) | raw_adjinc.le(0) + ) + if invalid_ssip.any() or invalid_adjinc.any(): + invalid = invalid_ssip | invalid_adjinc + bad = joined.loc[ + invalid, + ["SERIALNO", "SPORDER", "AGEP", "SSIP", "ADJINC"], + ] + raise ValueError( + "ACS raw SSIP/ADJINC violates the age-15 adjusted-dollar contract; " + f"examples={bad.head().to_dict('records')}." + ) + raw_aligned = raw_ssip.to_numpy(dtype=np.float64) * ( + raw_adjinc.to_numpy(dtype=np.float64) / 1_000_000.0 + ) + equal = (np.isnan(pool_aligned) & np.isnan(raw_aligned)) | np.equal( + pool_aligned, + raw_aligned, + ) + if not equal.all(): + bad = joined.loc[ + ~equal, + ["person_source_id", "SERIALNO", "SPORDER", "SSIP", "ADJINC"], + ].copy() + bad["ssi_reported_pool"] = pool_aligned[~equal] + bad["ssi_reported_raw"] = raw_aligned[~equal] + raise ValueError( + "ACS native ssi_reported disagrees with pinned raw " + "SSIP * ADJINC / 1_000_000; examples=" + f"{bad.head().to_dict('records')}." + ) + + +def _require_asec_native_predictors(person: pd.DataFrame) -> None: + channel_column = support_channel_column("person") + asec = person[channel_column].eq(_ASEC_CHANNEL) + required = (*_OUTPUT_COLUMNS, "SSI_VAL") + missing = [column for column in required if column not in person] + if missing: + raise ValueError( + f"ACS release join requires native ASEC predictor column(s): {missing}." + ) + null_counts = { + column: int(person.loc[asec, column].isna().sum()) + for column in required + if person.loc[asec, column].isna().any() + } + if null_counts: + raise ValueError( + "ACS release predictor receipt requires complete native ASEC inputs; " + f"null_counts={null_counts}." + ) + malformed: dict[str, int] = {} + for column in required: + values = person.loc[asec, column] + numeric = pd.to_numeric(values, errors="coerce") + numeric_cells = values.map(pd.api.types.is_number) + invalid = ( + ~numeric_cells + | numeric.isna() + | ~np.isfinite(numeric.to_numpy(dtype=np.float64)) + ) + if column == "SSI_VAL": + invalid |= numeric.lt(0) + if invalid.any(): + malformed[column] = int(invalid.sum()) + if malformed: + raise ValueError( + "ACS release predictor receipt requires numeric finite native ASEC " + "inputs and nonnegative SSI_VAL; malformed_counts=" + f"{malformed}." + ) + + +def _receipt( + frame: Frame, + *, + person_identity: Mapping[str, Any], + household_identity: Mapping[str, Any], + canonical: pd.DataFrame, + raw_person_rows: int, + raw_household_rows: int, + acs_rows: int, + clone_counts: Mapping[str, int], +) -> dict[str, Any]: + person = frame.table("person") + channel = person[support_channel_column("person")].astype(str) + models: dict[str, Any] = {} + for model, predictors in _MODEL_PREDICTORS.items(): + models[model] = { + "predictors": { + predictor: { + "asec_native": int( + (channel.eq(_ASEC_CHANNEL) & person[predictor].notna()).sum() + ), + "acs_joined": int( + (channel.eq(_ACS_CHANNEL) & person[predictor].notna()).sum() + ), + "still_null": int(person[predictor].isna().sum()), + } + for predictor in predictors + } + } + if "SSI_VAL" not in person or "ssi_reported" not in person: + raise AssertionError("Reported SSI receipt columns unexpectedly absent.") + asec_ssi = pd.to_numeric(person["SSI_VAL"], errors="coerce") + acs_ssi = pd.to_numeric(person["ssi_reported"], errors="coerce") + reported_anchor = pd.Series( + np.where(channel.eq(_ASEC_CHANNEL), asec_ssi, acs_ssi), + index=person.index, + ) + models["ssi_disability_criteria"]["predictors"]["reported_ssi_anchor"] = { + "source_columns": { + "asec_native": "SSI_VAL", + "acs_joined": "ssi_reported (native adjusted ACS SSIP)", + }, + "asec_native": int((channel.eq(_ASEC_CHANNEL) & reported_anchor.notna()).sum()), + "acs_joined": int((channel.eq(_ACS_CHANNEL) & reported_anchor.notna()).sum()), + "still_null": int(reported_anchor.isna().sum()), + "null_semantic": ( + "below-age-15 ACS SSIP universe; the receiver's > 0 predicate " + "treats it as false without rewriting the source value" + ), + } + semantic_keys = canonical.loc[:, ["SERIALNO", "SPORDER"]].sort_values( + ["SERIALNO", "SPORDER"], kind="stable" + ) + key_digest = hashlib.sha256( + "".join( + f"{serial}:{int(sporder)}\n" + for serial, sporder in semantic_keys.itertuples(index=False, name=None) + ).encode() + ).hexdigest() + return { + "enabled": True, + "version": 1, + "artifacts": { + "person": dict(person_identity), + "household": dict(household_identity), + }, + "crosswalk": { + "version": ACS_RELEASE_PREDICTOR_CROSSWALK_VERSION, + "sha256": ACS_RELEASE_PREDICTOR_CROSSWALK_SHA256, + }, + "join": { + "semantic_key": ["household.SERIALNO", "person.SPORDER"], + "clone_fanout_key": "person_source_id", + "acs_source_people": int(len(canonical)), + "acs_support_rows": acs_rows, + "acs_support_rows_by_clone_index": dict(clone_counts), + "selected_raw_person_rows": int(raw_person_rows), + "selected_raw_household_rows": int(raw_household_rows), + "unmatched_pool_source_people": 0, + "source_identity_collisions": 0, + "semantic_key_sha256": key_digest, + }, + "count_semantics": { + "asec_native": "physical ASEC rows with an observed predictor", + "acs_joined": "physical ACS rows populated by this exact join", + "still_null": "all remaining rows with a null predictor", + }, + "models": models, + } + + +def _required_integral( + values: pd.Series, + *, + label: str, + minimum: int, +) -> pd.Series: + numeric = pd.to_numeric(values, errors="coerce") + array = numeric.to_numpy(dtype=np.float64) + if ( + numeric.isna().any() + or not np.isfinite(array).all() + or not np.equal(array, np.floor(array)).all() + or (array < minimum).any() + ): + raise ValueError( + f"{label} must contain finite integers greater than or equal to {minimum}." + ) + return pd.Series(array.astype(np.int64), index=values.index) diff --git a/packages/microcosm-build/src/microcosm/build/us_runtime/acs_transfer.py b/packages/microcosm-build/src/microcosm/build/us_runtime/acs_transfer.py index f791823a0..a3fb00a6b 100644 --- a/packages/microcosm-build/src/microcosm/build/us_runtime/acs_transfer.py +++ b/packages/microcosm-build/src/microcosm/build/us_runtime/acs_transfer.py @@ -126,17 +126,62 @@ } ) -# These inputs are numeric in current dense artifacts, but their domain is a -# finite set of years. QRF quantile interpolation is valid for continuous -# amounts and invalid for a year code, so predictions are snapped to observed -# donor support exactly as the existing PUF support stage does. +# These inputs are numeric in current dense artifacts, but their domain has +# finite integer support. QRF quantile interpolation is valid for continuous +# amounts and invalid for year codes or counts, so predictions are snapped to +# observed donor support exactly as the existing PUF support stage does. _DISCRETE_NUMERIC_TARGETS = frozenset( { "first_home_mortgage_origination_year", "second_home_mortgage_origination_year", + "weeks_unemployed", } ) +_PREGNANCY_TARGET = "is_pregnant" +_PREGNANCY_AGE_RANGE = (15, 44) +_PERSON_SOURCE_ID_COLUMN = "person_source_id" +_PERSON_ID_COLUMN = "person_id" +_PERSON_CLONE_INDEX_COLUMN = "person_support_clone_index" + + +def _pregnancy_structural_policy_identity(*, enabled: bool) -> dict[str, object]: + """Return the complete hard-domain and clone-fanout transfer policy.""" + + payload: dict[str, object] = { + "enabled": enabled, + "target": _PREGNANCY_TARGET, + "eligibility": { + "is_female": True, + "minimum_age_inclusive": _PREGNANCY_AGE_RANGE[0], + "maximum_age_inclusive": _PREGNANCY_AGE_RANGE[1], + }, + "source_person_key_precedence": [ + _PERSON_SOURCE_ID_COLUMN, + _PERSON_ID_COLUMN, + ], + "assembled_representative": { + "clone_index_column": _PERSON_CLONE_INDEX_COLUMN, + "clone_index": 0, + }, + "qrf_scope": "one_eligible_representative_per_source_person", + "ineligible_missing_value": False, + "fanout": "one_source_person_result_to_every_missing_clone", + "preexisting_domain_violations": "refuse", + "preexisting_clone_disagreement": "refuse", + "final_domain_violations": "refuse", + "final_clone_disagreement": "refuse", + } + payload["sha256"] = hashlib.sha256( + json.dumps( + payload, + sort_keys=True, + separators=(",", ":"), + allow_nan=False, + ).encode("utf-8") + ).hexdigest() + return payload + _IMMIGRATION_STATUS_TARGETS = ( "ssn_card_type", "immigration_status_str", @@ -268,6 +313,11 @@ def acs_transfer_execution_contract_identity( "immigration_status_targets": list(_IMMIGRATION_STATUS_TARGETS), "immigration_status_model_target": _IMMIGRATION_STATUS_MODEL_TARGET, "discrete_numeric_targets": sorted(_DISCRETE_NUMERIC_TARGETS), + "structural_target_policies": { + _PREGNANCY_TARGET: _pregnancy_structural_policy_identity( + enabled=_PREGNANCY_TARGET in requested_targets, + ) + }, "post_transfer_structure": { "schedule_d_capital_gain_distributions": { "enabled": schedule_d_enabled, @@ -444,6 +494,9 @@ class AcsImputedInput: #: Post-fit structural reconciliation counts, when the column's surface #: was adjusted to a statute contract after prediction. reconciliation: Mapping[str, int] | None = None + #: Hard-domain and source-person fanout proof for structurally constrained + #: transfer targets. This is receipt-only and never enters the frame. + structural_receipt: Mapping[str, object] | None = None @dataclass(frozen=True) @@ -558,6 +611,19 @@ class _TargetEncoding: category_position: int | None = None +@dataclass(frozen=True) +class _PregnancyStructuralPlan: + """Pre-QRF pregnancy scope plus enough state for deterministic fanout.""" + + qrf_missing: np.ndarray + source_codes: np.ndarray + source_values: np.ndarray + qrf_source_groups: np.ndarray + representative_positions: np.ndarray + eligibility: np.ndarray + receipt: Mapping[str, object] + + def required_acs_transfer_inputs() -> frozenset[str]: """Return every model leaf in the declared production transfer plan.""" @@ -904,6 +970,327 @@ def assert_acs_transfer_targets_are_input_leaves( ) +def _pregnancy_eligibility(table: pd.DataFrame, *, role: str) -> np.ndarray: + """Resolve the hard female-age domain without coercing missing predictors.""" + + _require_columns( + table, + ("age", "is_female"), + context=f"ACS transfer pregnancy {role} domain", + ) + try: + age = _as_float_array(table["age"]) + female = _as_float_array(table["is_female"]) + except (TypeError, ValueError) as exc: + raise TypeError( + f"ACS transfer pregnancy {role} domain requires numeric age and " + "boolean is_female." + ) from exc + invalid_age = ~np.isfinite(age) + invalid_female = ~np.isfinite(female) | ~np.isin(female, [0.0, 1.0]) + if invalid_age.any() or invalid_female.any(): + raise ValueError( + f"ACS transfer pregnancy {role} domain requires complete finite age " + "and boolean is_female; found " + f"invalid_age_rows={int(invalid_age.sum())}, " + f"invalid_is_female_rows={int(invalid_female.sum())}." + ) + low, high = _PREGNANCY_AGE_RANGE + return (female == 1.0) & (age >= low) & (age <= high) + + +def _pregnancy_boolean_values( + series: pd.Series, + *, + role: str, +) -> tuple[np.ndarray, np.ndarray]: + """Return observed/true masks while refusing non-boolean physical values.""" + + observed = series.notna().to_numpy(dtype=bool) + try: + values = _as_float_array(series) + except (TypeError, ValueError) as exc: + raise TypeError( + f"ACS transfer pregnancy {role} values must be physical booleans." + ) from exc + valid = ~observed | (values == 0.0) | (values == 1.0) + if not valid.all(): + raise ValueError( + f"ACS transfer pregnancy {role} values contain " + f"{int((~valid).sum())} non-boolean row(s)." + ) + return observed, observed & (values == 1.0) + + +def _pregnancy_source_groups( + table: pd.DataFrame, +) -> tuple[np.ndarray, int, str, np.ndarray]: + """Return stable source-person codes and one canonical representative.""" + + has_clone_index = _PERSON_CLONE_INDEX_COLUMN in table + if has_clone_index and _PERSON_SOURCE_ID_COLUMN not in table: + raise ValueError( + "ACS transfer pregnancy clone fanout requires person_source_id when " + f"{_PERSON_CLONE_INDEX_COLUMN} is present." + ) + key_column = ( + _PERSON_SOURCE_ID_COLUMN + if _PERSON_SOURCE_ID_COLUMN in table + else _PERSON_ID_COLUMN + ) + _require_columns( + table, + (key_column,), + context="ACS transfer pregnancy source-person identity", + ) + keys = table[key_column] + if keys.isna().any(): + raise ValueError( + f"ACS transfer pregnancy source-person identity {key_column!r} has " + f"{int(keys.isna().sum())} missing row(s)." + ) + codes, unique_keys = pd.factorize(keys, sort=False) + if (codes < 0).any(): # pragma: no cover - missing keys refused above + raise AssertionError("Pregnancy source-person factorization lost a key.") + group_count = len(unique_keys) + representatives = np.full(group_count, -1, dtype=np.int64) + positions = np.arange(len(table), dtype=np.int64) + + if not has_clone_index: + _, first_positions = np.unique(codes, return_index=True) + representatives[:] = first_positions + return codes, group_count, key_column, representatives + + clone_numeric = pd.to_numeric( + table[_PERSON_CLONE_INDEX_COLUMN], + errors="coerce", + ).to_numpy(dtype=np.float64) + invalid_clone = ( + ~np.isfinite(clone_numeric) + | (clone_numeric < 0.0) + | (clone_numeric != np.floor(clone_numeric)) + ) + if invalid_clone.any(): + raise ValueError( + "ACS transfer pregnancy clone fanout found " + f"{int(invalid_clone.sum())} invalid clone-index row(s)." + ) + clone_index = clone_numeric.astype(np.int64) + duplicate_pairs = pd.DataFrame( + {"_source": codes, "_clone": clone_index} + ).duplicated() + if duplicate_pairs.any(): + raise ValueError( + "ACS transfer pregnancy clone fanout found " + f"{int(duplicate_pairs.sum())} duplicate source-person/clone row(s)." + ) + clone_zero = clone_index == 0 + clone_zero_counts = np.bincount(codes[clone_zero], minlength=group_count) + bad_groups = clone_zero_counts != 1 + if bad_groups.any(): + raise ValueError( + "ACS transfer pregnancy clone fanout requires exactly one clone-0 " + f"representative for every source person; {int(bad_groups.sum())} " + "source person(s) violate that contract." + ) + representatives[codes[clone_zero]] = positions[clone_zero] + return codes, group_count, key_column, representatives + + +def _prepare_pregnancy_structural_plan( + donor: Frame, + recipient: Frame, + *, + target_missing: np.ndarray, +) -> _PregnancyStructuralPlan: + """Refuse invalid inputs and select one eligible QRF row per source person.""" + + donor_table = donor.table(donor.schema.person_entity) + donor_eligible = _pregnancy_eligibility(donor_table, role="donor") + donor_observed, donor_true = _pregnancy_boolean_values( + donor_table[_PREGNANCY_TARGET], + role="donor", + ) + donor_domain_violations = donor_observed & donor_true & ~donor_eligible + if donor_domain_violations.any(): + raise ValueError( + "ACS transfer pregnancy structural policy refused " + f"{int(donor_domain_violations.sum())} preexisting donor domain " + "violation row(s); pregnancy requires female ages 15 through 44." + ) + + table = recipient.table(recipient.schema.person_entity) + eligibility = _pregnancy_eligibility(table, role="recipient") + if _PREGNANCY_TARGET in table: + observed, true_values = _pregnancy_boolean_values( + table[_PREGNANCY_TARGET], + role="recipient", + ) + else: + observed = np.zeros(len(table), dtype=bool) + true_values = np.zeros(len(table), dtype=bool) + recipient_domain_violations = observed & true_values & ~eligibility + if recipient_domain_violations.any(): + raise ValueError( + "ACS transfer pregnancy structural policy refused " + f"{int(recipient_domain_violations.sum())} preexisting recipient " + "domain violation row(s); pregnancy requires female ages 15 " + "through 44." + ) + + source_codes, group_count, key_column, representatives = ( + _pregnancy_source_groups(table) + ) + eligible_min = np.ones(group_count, dtype=np.int8) + eligible_max = np.zeros(group_count, dtype=np.int8) + np.minimum.at(eligible_min, source_codes, eligibility.astype(np.int8)) + np.maximum.at(eligible_max, source_codes, eligibility.astype(np.int8)) + inconsistent_domain_groups = eligible_min != eligible_max + if inconsistent_domain_groups.any(): + raise ValueError( + "ACS transfer pregnancy structural policy refused " + f"{int(inconsistent_domain_groups.sum())} source person(s) whose " + "clones disagree on female-age eligibility." + ) + + group_has_true = ( + np.bincount(source_codes[observed & true_values], minlength=group_count) > 0 + ) + group_has_false = ( + np.bincount(source_codes[observed & ~true_values], minlength=group_count) > 0 + ) + preexisting_disagreement = group_has_true & group_has_false + if preexisting_disagreement.any(): + raise ValueError( + "ACS transfer pregnancy structural policy refused " + f"{int(preexisting_disagreement.sum())} source person(s) with " + "preexisting clone disagreement." + ) + + missing = np.asarray(target_missing, dtype=bool) + if missing.shape != (len(table),): + raise ValueError("ACS pregnancy target-missing mask has the wrong row count.") + group_observed = group_has_true | group_has_false + group_eligible = eligibility[representatives] + qrf_source_groups = ~group_observed & group_eligible + qrf_missing = np.zeros(len(table), dtype=bool) + qrf_missing[representatives[qrf_source_groups]] = True + + source_values = np.full(group_count, np.nan, dtype=np.float64) + source_values[group_observed] = group_has_true[group_observed].astype(np.float64) + source_values[~group_eligible] = 0.0 + group_sizes = np.bincount(source_codes, minlength=group_count) + missing_by_group = np.bincount( + source_codes, + weights=missing.astype(np.int64), + minlength=group_count, + ).astype(np.int64) + qrf_group_count = int(qrf_source_groups.sum()) + policy = _pregnancy_structural_policy_identity(enabled=True) + receipt: dict[str, object] = { + "policy_sha256": policy["sha256"], + "source_person_key": key_column, + "source_persons_checked": group_count, + "physical_rows_checked": int(len(table)), + "clone_rows_checked": int(len(table) - group_count), + "donor_rows_checked": int(len(donor_table)), + "qrf_draw_source_persons": qrf_group_count, + "qrf_draw_rows": qrf_group_count, + "qrf_fanout_rows": int( + missing_by_group[qrf_source_groups].sum() - qrf_group_count + ), + "preexisting_value_fanout_rows": int( + missing_by_group[group_observed & group_eligible].sum() + ), + "ineligible_rows_assigned_false": int((missing & ~eligibility).sum()), + "donor_preexisting_domain_violation_rows": 0, + "recipient_preexisting_domain_violation_rows": 0, + "preexisting_clone_disagreement_source_persons": 0, + "inconsistent_eligibility_source_persons": 0, + "maximum_clones_per_source_person": int(group_sizes.max(initial=0)), + } + return _PregnancyStructuralPlan( + qrf_missing=qrf_missing, + source_codes=source_codes, + source_values=source_values, + qrf_source_groups=qrf_source_groups, + representative_positions=representatives, + eligibility=eligibility, + receipt=receipt, + ) + + +def _fan_pregnancy_predictions( + plan: _PregnancyStructuralPlan, + prediction: np.ndarray | pd.api.extensions.ExtensionArray | None, +) -> np.ndarray: + """Fan one decoded QRF result across each source person's missing clones.""" + + source_values = plan.source_values.copy() + if plan.qrf_source_groups.any(): + if prediction is None: # pragma: no cover - caller invariant + raise RuntimeError("Pregnancy structural plan is missing its QRF draw.") + predicted = _as_float_array(pd.Series(prediction)) + positions = plan.representative_positions[plan.qrf_source_groups] + values = predicted[positions] + valid = np.isfinite(values) & ((values == 0.0) | (values == 1.0)) + if not valid.all(): + raise ValueError( + "ACS transfer pregnancy QRF produced " + f"{int((~valid).sum())} invalid representative result(s)." + ) + source_values[plan.qrf_source_groups] = (values == 1.0).astype(np.float64) + if not np.isfinite(source_values).all(): # pragma: no cover - plan invariant + raise RuntimeError("Pregnancy structural fanout left unresolved source people.") + return source_values[plan.source_codes] == 1.0 + + +def _finalize_pregnancy_structural_receipt( + table: pd.DataFrame, + plan: _PregnancyStructuralPlan, +) -> dict[str, object]: + """Refuse post-transfer violations and seal the successful count receipt.""" + + eligibility = _pregnancy_eligibility(table, role="post-transfer") + observed, true_values = _pregnancy_boolean_values( + table[_PREGNANCY_TARGET], + role="post-transfer", + ) + incomplete = ~observed + domain_violations = observed & true_values & ~eligibility + group_count = len(plan.source_values) + group_has_true = ( + np.bincount( + plan.source_codes[observed & true_values], + minlength=group_count, + ) + > 0 + ) + group_has_false = ( + np.bincount( + plan.source_codes[observed & ~true_values], + minlength=group_count, + ) + > 0 + ) + clone_disagreements = group_has_true & group_has_false + if incomplete.any() or domain_violations.any() or clone_disagreements.any(): + raise ValueError( + "ACS transfer pregnancy structural postcondition refused output: " + f"incomplete_rows={int(incomplete.sum())}, " + f"domain_violation_rows={int(domain_violations.sum())}, " + "clone_disagreement_source_persons=" + f"{int(clone_disagreements.sum())}." + ) + return { + **dict(plan.receipt), + "final_incomplete_rows": 0, + "final_domain_violation_rows": 0, + "final_clone_disagreement_source_persons": 0, + "status": "verified", + } + + def transfer_acs_inputs( recipient: Frame, donor: Frame, @@ -1024,21 +1411,88 @@ def transfer_acs_inputs( ) for entity, family, active_targets in active ] + structural_records: tuple[AcsImputedInput, ...] = () + fit_donor: Frame | None = None + resolved_channel: str | None = None + pregnancy_plan: _PregnancyStructuralPlan | None = None + pregnancy_request = next( + ( + item + for item in requested + if item[0] == recipient.schema.person_entity + and _PREGNANCY_TARGET in item[2] + ), + None, + ) + if pregnancy_request is not None: + _validate_donor_source( + donor_spine=donor_spine, + donor_channel=donor_channel, + ) + fit_donor, resolved_channel = resolve_acs_donor_channel( + donor, + donor_channel, + ) + person = recipient.table(recipient.schema.person_entity) + pregnancy_missing = ( + person[_PREGNANCY_TARGET].isna().to_numpy(dtype=bool) + if _PREGNANCY_TARGET in person + else np.ones(len(person), dtype=bool) + ) + pregnancy_plan = _prepare_pregnancy_structural_plan( + fit_donor, + recipient, + target_missing=pregnancy_missing, + ) + if not pregnancy_missing.any(): + structural_receipt = _finalize_pregnancy_structural_receipt( + person, + pregnancy_plan, + ) + pregnancy_entity, pregnancy_family, pregnancy_targets = ( + pregnancy_request + ) + if pregnancy_targets != (_PREGNANCY_TARGET,): # pragma: no cover + raise AssertionError( + "Pregnancy structural target was not isolated before receipt." + ) + structural_records = ( + AcsImputedInput( + column=_PREGNANCY_TARGET, + entity=pregnancy_entity, + family=pregnancy_family, + donor_spine=donor_spine, + donor_channel=resolved_channel, + predictors=ACS_PERSON_TRANSFER_PREDICTORS, + seed=_family_seed( + seed, + entity=pregnancy_entity, + family=pregnancy_family, + ), + weight_kind="structural", + structural_receipt=structural_receipt, + ), + ) + if not active: return AcsTransferResult( frame=canonicalize_frame_string_dtypes( recipient, boundary="ACS transfer result", ), + imputed_inputs=structural_records, deferred_inputs=deferred_inputs, + resolved_donor_channel=resolved_channel, ) - _validate_donor_source(donor_spine=donor_spine, donor_channel=donor_channel) - fit_donor, resolved_channel = resolve_acs_donor_channel(donor, donor_channel) + if fit_donor is None: + _validate_donor_source(donor_spine=donor_spine, donor_channel=donor_channel) + fit_donor, resolved_channel = resolve_acs_donor_channel(donor, donor_channel) + assert fit_donor is not None output_tables = { entity: recipient.table(entity).copy() for entity in recipient.entities } - provenance: list[AcsImputedInput] = [] + provenance = list(structural_records) fit_records: list[FitWeightRecord] = [] imputed_masks: dict[tuple[str, str], np.ndarray] = {} ordered_bank_targets = [ @@ -1061,14 +1515,34 @@ def transfer_acs_inputs( ) for target in targets } - if target_bank is None: + active_pregnancy_plan: _PregnancyStructuralPlan | None = None + fit_target_missing = target_missing + if _PREGNANCY_TARGET in targets: + if targets != (_PREGNANCY_TARGET,): + raise AssertionError( + "Pregnancy structural target was not isolated before QRF." + ) + if pregnancy_plan is None: # pragma: no cover - preflight invariant + raise AssertionError("Pregnancy structural plan was not prepared.") + active_pregnancy_plan = pregnancy_plan + fit_target_missing = { + _PREGNANCY_TARGET: active_pregnancy_plan.qrf_missing, + } + + fitted: _FamilyFit | None + if ( + active_pregnancy_plan is not None + and not active_pregnancy_plan.qrf_missing.any() + ): + fitted = None + elif target_bank is None: fitted = _fit_family_patterns( fit_donor, recipient, entity=entity, family=family, targets=targets, - target_missing=target_missing, + target_missing=fit_target_missing, seed=seed, n_estimators=n_estimators, regime_evidence_targets=family_regime_evidence_targets, @@ -1080,7 +1554,7 @@ def transfer_acs_inputs( entity=entity, family=family, targets=targets, - target_missing=target_missing, + target_missing=fit_target_missing, seed=seed, n_estimators=n_estimators, target_bank=target_bank, @@ -1093,16 +1567,29 @@ def transfer_acs_inputs( ) patterns_without_regimes = ( tuple(replace(pattern, target_regimes=()) for pattern in fitted.patterns) - if family_regime_evidence_targets + if fitted is not None and family_regime_evidence_targets else fitted.patterns + if fitted is not None + else () ) for target in targets: - predicted = _prediction_values( - fitted.predictions[target], - encoding=fitted.target_encodings[target], - entity=entity, - target=target, + decoded = ( + _prediction_values( + fitted.predictions[target], + encoding=fitted.target_encodings[target], + entity=entity, + target=target, + ) + if fitted is not None + else None ) + predicted = ( + _fan_pregnancy_predictions(active_pregnancy_plan, decoded) + if active_pregnancy_plan is not None + else decoded + ) + if predicted is None: # pragma: no cover - structural invariant + raise AssertionError("ACS transfer target has no prediction values.") merged, imputed = _fill_recipient_nulls( output_tables[entity], target=target, @@ -1113,6 +1600,14 @@ def transfer_acs_inputs( missing_rows = target_missing[target] if not missing_rows.any(): continue + structural_receipt = ( + _finalize_pregnancy_structural_receipt( + output_tables[entity], + active_pregnancy_plan, + ) + if active_pregnancy_plan is not None + else None + ) provenance.append( AcsImputedInput( column=target, @@ -1120,19 +1615,32 @@ def transfer_acs_inputs( family=family, donor_spine=donor_spine, donor_channel=resolved_channel, - predictors=fitted.predictors, - seed=fitted.family_seed, - weight_kind=fitted.weight_kind, + predictors=( + fitted.predictors + if fitted is not None + else ACS_PERSON_TRANSFER_PREDICTORS + ), + seed=( + fitted.family_seed + if fitted is not None + else _family_seed(seed, entity=entity, family=family) + ), + weight_kind=( + fitted.weight_kind if fitted is not None else "structural" + ), patterns=( fitted.patterns - if target in family_regime_evidence_targets + if fitted is not None + and target in family_regime_evidence_targets else patterns_without_regimes ), imputed_recipient_rows=int(imputed.sum()), unmodeled_recipient_rows=int((missing_rows & ~imputed).sum()), + structural_receipt=structural_receipt, ) ) - fit_records.extend(fitted.fit_records) + if fitted is not None: + fit_records.extend(fitted.fit_records) _apply_post_transfer_structure( output_tables, @@ -2628,7 +3136,7 @@ def _split_large_target_families( *, max_targets_per_fit: int, ) -> list[tuple[str, str, tuple[str, ...]]]: - """Bound retained QRF forests without separating joint categorical codecs.""" + """Bound retained QRF forests and isolate hard-domain target draws.""" bounded: list[tuple[str, str, tuple[str, ...]]] = [] immigration_pair = set(_IMMIGRATION_STATUS_TARGETS) @@ -2646,6 +3154,12 @@ def _split_large_target_families( batches: list[tuple[str, ...]] = [] current: list[str] = [] for atom in atoms: + if atom == (_PREGNANCY_TARGET,): + if current: + batches.append(tuple(current)) + current = [] + batches.append(atom) + continue if current and len(current) + len(atom) > max_targets_per_fit: batches.append(tuple(current)) current = [] diff --git a/packages/microcosm-build/src/microcosm/build/us_runtime/alimony.py b/packages/microcosm-build/src/microcosm/build/us_runtime/alimony.py index 5ab6c1995..8ae7fe834 100644 --- a/packages/microcosm-build/src/microcosm/build/us_runtime/alimony.py +++ b/packages/microcosm-build/src/microcosm/build/us_runtime/alimony.py @@ -23,7 +23,9 @@ from microcosm.build.source_manifest import SourceStageSpec, load_source_manifest from microcosm.build.us_runtime.support_provenance import ( BASE_ASEC_SUPPORT_CHANNEL, + has_assembled_support_metadata, has_support_role_metadata, + support_gate_source_channel_series, support_role_series, ) from microcosm.frame import Frame @@ -291,18 +293,41 @@ def us_alimony_signal_gate(frame: Frame) -> GateResult: details=summary, ) source_mask = np.ones(len(person), dtype=bool) + source_reconciliation_mask = source_mask.copy() if has_support_role_metadata(person, entity="person"): source_mask = ( - support_role_series(person, entity="person") + support_gate_source_channel_series(person, entity="person") .eq(BASE_ASEC_SUPPORT_CHANNEL) .to_numpy() ) - amounts = pd.to_numeric(person["OI_VAL"], errors="coerce").to_numpy( + source_reconciliation_mask = source_mask.copy() + if has_assembled_support_metadata(person, entity="person"): + source_reconciliation_mask &= ( + support_role_series(person, entity="person") + .eq(BASE_ASEC_SUPPORT_CHANNEL) + .to_numpy() + ) + raw_amounts = pd.to_numeric(person["OI_VAL"], errors="coerce").to_numpy( dtype=np.float64 - )[source_mask] - codes = pd.to_numeric(person["OI_OFF"], errors="coerce").to_numpy( + ) + raw_codes = pd.to_numeric(person["OI_OFF"], errors="coerce").to_numpy( dtype=np.float64 - )[source_mask] + ) + source_invalid = source_mask & ( + ~np.isfinite(raw_amounts) | ~np.isfinite(raw_codes) + ) + summary["asec_source_rows"] = int(np.count_nonzero(source_mask)) + summary["asec_source_invalid"] = int(np.count_nonzero(source_invalid)) + summary["asec_source_reconciliation_rows"] = int( + np.count_nonzero(source_reconciliation_mask) + ) + if bool(source_invalid.any()): + failures.append( + "ASEC OI_VAL/OI_OFF contain nonfinite source values on " + f"{int(np.count_nonzero(source_invalid))} row(s)." + ) + amounts = raw_amounts[source_reconciliation_mask] + codes = raw_codes[source_reconciliation_mask] if bool(np.isfinite(amounts).all() and np.isfinite(codes).all()): integer_codes = codes.astype(np.int64) expected_alimony = np.where( @@ -321,13 +346,16 @@ def us_alimony_signal_gate(frame: Frame) -> GateResult: amounts, ) actual_alimony = pd.to_numeric( - person.loc[source_mask, "alimony_income"], errors="coerce" + person.loc[source_reconciliation_mask, "alimony_income"], + errors="coerce", ).to_numpy(dtype=np.float64) actual_strike_benefits = pd.to_numeric( - person.loc[source_mask, "strike_benefits"], errors="coerce" + person.loc[source_reconciliation_mask, "strike_benefits"], + errors="coerce", ).to_numpy(dtype=np.float64) actual_miscellaneous = pd.to_numeric( - person.loc[source_mask, "miscellaneous_income"], errors="coerce" + person.loc[source_reconciliation_mask, "miscellaneous_income"], + errors="coerce", ).to_numpy(dtype=np.float64) alimony_mismatch = ~np.isclose(actual_alimony, expected_alimony) strike_benefits_mismatch = ~np.isclose( diff --git a/packages/microcosm-build/src/microcosm/build/us_runtime/h5_io.py b/packages/microcosm-build/src/microcosm/build/us_runtime/h5_io.py index 516dead3c..55de34273 100644 --- a/packages/microcosm-build/src/microcosm/build/us_runtime/h5_io.py +++ b/packages/microcosm-build/src/microcosm/build/us_runtime/h5_io.py @@ -16,6 +16,7 @@ import shutil import uuid from collections.abc import Mapping +from copy import deepcopy from dataclasses import dataclass from pathlib import Path @@ -94,6 +95,15 @@ _TIME_PERIOD_KEY = "_time_period" _LOWERCASE_SHA256 = re.compile(r"[0-9a-f]{64}") _STACKED_PIPELINE = "us-stacked-pool" +_STACKED_SPINE_MANIFEST_VERSION = 4 +_STACKED_SURVEY_CHANNELS = ("asec", "acs") +_STACKED_SAMPLE_RUNG_TOKENS: Mapping[float, str] = { + 0.01: "f001", + 0.04: "f004", + 0.10: "f010", + 0.25: "f025", + 1.00: "f100", +} US_STACKED_POOL_OPERATOR_ORDER = ( "assemble_stacked_spine", "assign_us_puma_ladder", @@ -139,6 +149,8 @@ { "pipeline", "operator_order", + "sampling", + "stack_manifest", "geography_assignment", "stage_receipts", } @@ -255,6 +267,150 @@ def _validated_pool_manifest_envelope( ) +def _validated_stacked_sampling_manifest_binding( + manifest: Mapping[str, object], + *, + manifest_path: Path, +) -> Mapping[str, object] | None: + """Authenticate the production-wide survey rung carried by a stack. + + The adjacent-year ASEC join runs after the two survey arms are sampled. + Release gates may therefore consume the configured production rung only + after the pool manifest proves that its top-level sampling receipt, frozen + stack manifest, and both per-arm sample receipts all name the same value. + Legacy two-spine manifests have no production-wide stack receipt and return + ``None``. + """ + + if manifest.get("pipeline") != _STACKED_PIPELINE: + return None + + label = f"US stacked pool manifest {manifest_path}" + sampling = _mapping(manifest.get("sampling"), label=f"{label}.sampling") + stack_manifest = _mapping( + manifest.get("stack_manifest"), + label=f"{label}.stack_manifest", + ) + if stack_manifest.get("version") != _STACKED_SPINE_MANIFEST_VERSION: + raise ValueError( + f"{label} stack manifest must have production version " + f"{_STACKED_SPINE_MANIFEST_VERSION}." + ) + + sampling_fraction = sampling.get("sample_fraction") + stack_fraction = stack_manifest.get("sample_fraction") + for location, value in ( + ("sampling.sample_fraction", sampling_fraction), + ("stack_manifest.sample_fraction", stack_fraction), + ): + if ( + type(value) is not float + or not np.isfinite(value) + or not 0.0 < value <= 1.0 + ): + raise ValueError( + f"{label} {location} must be a finite float in (0, 1]." + ) + if sampling_fraction != stack_fraction: + raise ValueError( + f"{label} sampling.sample_fraction differs from " + "stack_manifest.sample_fraction." + ) + expected_token = _STACKED_SAMPLE_RUNG_TOKENS.get(sampling_fraction) + if expected_token is None or sampling.get("fraction_token") != expected_token: + raise ValueError( + f"{label} sampling fraction/token pair is not an approved stacked rung." + ) + + sampling_seed = sampling.get("sample_seed") + stack_seed = stack_manifest.get("sample_seed") + if ( + isinstance(sampling_seed, bool) + or not isinstance(sampling_seed, int) + or sampling_seed < 0 + or isinstance(stack_seed, bool) + or not isinstance(stack_seed, int) + or stack_seed != sampling_seed + ): + raise ValueError( + f"{label} sampling.sample_seed and stack_manifest.sample_seed must " + "be the same non-negative integer." + ) + + survey_samples = _mapping( + stack_manifest.get("survey_samples"), + label=f"{label}.stack_manifest.survey_samples", + ) + if set(survey_samples) != set(_STACKED_SURVEY_CHANNELS): + raise ValueError( + f"{label} stack survey samples must exactly cover " + f"{list(_STACKED_SURVEY_CHANNELS)}." + ) + realized_households = _mapping( + sampling.get("realized_households"), + label=f"{label}.sampling.realized_households", + ) + if set(realized_households) != set(_STACKED_SURVEY_CHANNELS): + raise ValueError( + f"{label} realized-household counts must exactly cover " + f"{list(_STACKED_SURVEY_CHANNELS)}." + ) + for channel in _STACKED_SURVEY_CHANNELS: + sample = _mapping( + survey_samples[channel], + label=f"{label}.stack_manifest.survey_samples.{channel}", + ) + sample_fraction = sample.get("fraction") + if type(sample_fraction) is not float or sample_fraction != sampling_fraction: + raise ValueError( + f"{label} {channel} survey-sample fraction differs from the " + "production sampling rung." + ) + sample_seed = sample.get("seed") + if ( + isinstance(sample_seed, bool) + or not isinstance(sample_seed, int) + or sample_seed != sampling_seed + ): + raise ValueError( + f"{label} {channel} survey-sample seed differs from the " + "production sample seed." + ) + realized = sample.get("realized_household_count") + top_realized = realized_households[channel] + if ( + isinstance(realized, bool) + or not isinstance(realized, int) + or realized < 1 + or isinstance(top_realized, bool) + or not isinstance(top_realized, int) + or top_realized != realized + ): + raise ValueError( + f"{label} {channel} realized-household count is malformed or " + "inconsistent." + ) + + expected_stack_sha256 = sampling.get("stack_manifest_sha256") + if ( + not isinstance(expected_stack_sha256, str) + or _LOWERCASE_SHA256.fullmatch(expected_stack_sha256) is None + ): + raise ValueError(f"{label} sampling stack-manifest SHA-256 is malformed.") + canonical_stack = json.dumps( + stack_manifest, + allow_nan=False, + ensure_ascii=False, + separators=(",", ":"), + sort_keys=True, + ).encode("utf-8") + if hashlib.sha256(canonical_stack).hexdigest() != expected_stack_sha256: + raise ValueError( + f"{label} sampling stack-manifest SHA-256 does not match its receipt." + ) + return stack_manifest + + class AuthenticatedPoolH5MismatchError(RuntimeError): """A pool H5 no longer matches the bytes authenticated by its manifest.""" @@ -617,6 +773,10 @@ def _load_authenticated_us_multispine_pool_manifest( manifest, manifest_path=manifest_path, ) + _validated_stacked_sampling_manifest_binding( + manifest, + manifest_path=manifest_path, + ) expected_schema_version = ( US_MULTISPINE_POOL_MANIFEST_SCHEMA_VERSION if envelope == "stacked" @@ -1587,6 +1747,16 @@ def _load_us_multispine_pool( manifest_path=manifest_path, ) frame_metadata: dict[str, object] = {} + stack_manifest = _validated_stacked_sampling_manifest_binding( + manifest, + manifest_path=manifest_path, + ) + if stack_manifest is not None: + from microcosm.build.us_runtime.stacked_spine import ( + STACKED_SPINE_MANIFEST_KEY, + ) + + frame_metadata[STACKED_SPINE_MANIFEST_KEY] = deepcopy(stack_manifest) if late_transition is not None: _dag, transition_authority, _transition_authority_sha256 = late_transition from microcosm.build.us_runtime.stacked_spine import ( diff --git a/packages/microcosm-build/src/microcosm/build/us_runtime/medicare_take_up.py b/packages/microcosm-build/src/microcosm/build/us_runtime/medicare_take_up.py index d93ae5066..ba40e479b 100644 --- a/packages/microcosm-build/src/microcosm/build/us_runtime/medicare_take_up.py +++ b/packages/microcosm-build/src/microcosm/build/us_runtime/medicare_take_up.py @@ -27,7 +27,10 @@ run_source_stage, ) from microcosm.build.us_runtime.support_provenance import ( + BASE_ASEC_SUPPORT_CHANNEL, + has_assembled_support_metadata, has_support_role_metadata, + support_gate_source_channel_series, support_role_series, ) from microcosm.frame import Frame @@ -128,6 +131,20 @@ def _source_codes(person: pd.DataFrame, source: str) -> np.ndarray: return numeric.astype(np.int8) +def _asec_source_mask(person: pd.DataFrame) -> np.ndarray: + """Select physical ASEC rows without changing legacy source semantics.""" + + if not has_assembled_support_metadata(person, entity="person"): + return np.ones(len(person), dtype=bool) + source_channels = support_gate_source_channel_series(person, entity="person") + mask = source_channels.eq(BASE_ASEC_SUPPORT_CHANNEL).to_numpy() + if not mask.any(): + raise SourceRuntimeError( + "US Medicare take-up support has no physical ASEC source rows." + ) + return mask + + def derive_us_medicare_take_up_from_manifest( frame: pd.DataFrame | None, operation: SourceOperationSpec, @@ -228,10 +245,14 @@ def us_medicare_take_up_summary(frame: Frame) -> dict[str, object]: "missing_count": int(person[_OUTPUT].isna().sum()), } if _SOURCE in person: - source_values = _source_codes(person, _SOURCE) == _ENROLLED_CODE + source_mask = _asec_source_mask(person) + source_values = ( + _source_codes(person.loc[source_mask], _SOURCE) == _ENROLLED_CODE + ) summary["source_mismatch_count"] = int( - np.count_nonzero(values != source_values) + np.count_nonzero(values[source_mask] != source_values) ) + summary["source_rows"] = int(np.count_nonzero(source_mask)) if has_support_role_metadata(person, entity="person"): channel_shares: dict[str, float] = {} channels = support_role_series(person, entity="person").to_numpy() diff --git a/packages/microcosm-build/src/microcosm/build/us_runtime/pregnancy.py b/packages/microcosm-build/src/microcosm/build/us_runtime/pregnancy.py index aa6c94096..99b868c10 100644 --- a/packages/microcosm-build/src/microcosm/build/us_runtime/pregnancy.py +++ b/packages/microcosm-build/src/microcosm/build/us_runtime/pregnancy.py @@ -20,15 +20,15 @@ ``microcosm/build/us/source_stages.json`` with its citation and reaches this module as a manifest operation parameter. -Selection draws are seeded blake2b hashes keyed by the person's stable -source identity (``source_year`` / ``source_household_id`` / -``source_person_id`` when present), so support-channel clones of one -source person always receive the same flag and reruns are -bit-reproducible. - -Healing behavior: a frame that already carries ``is_pregnant`` with -signal passes through untouched (idempotent). A constant column — -indistinguishable from the engine's broadcast default — is reseeded. +Selection draws are seeded blake2b hashes keyed by the person's stable source +identity (``source_year`` / ``source_household_id`` / ``source_person_id`` +when present), so support-channel clones of one source person always receive +the same flag and reruns are bit-reproducible. + +Healing behavior: a frame that already carries a complete, structurally valid +``is_pregnant`` signal passes through untouched (idempotent). A constant +column — indistinguishable from the engine's broadcast default — is reseeded. +Preexisting or final domain violations and clone disagreement are refused. """ from __future__ import annotations @@ -83,6 +83,7 @@ _FEMALE_SEX_CODE = 2 _CHILDBEARING_AGE_RANGE = (15, 44) +_PERSON_SOURCE_ID_COLUMN = "person_source_id" #: Weighted share of all persons flagged pregnant must land in this band. #: Expected ≈ rate (4.1%) x the female-15-44 share of the population @@ -163,6 +164,131 @@ def _stable_person_draws(persons: pd.DataFrame, *, seed: int) -> np.ndarray: ) +def _pregnancy_eligibility(person: pd.DataFrame) -> tuple[np.ndarray, np.ndarray]: + """Return female and age-domain masks from mapped or raw source columns.""" + + if {"is_female", "age"} <= set(person.columns): + female_source = person["is_female"] + age_source = person["age"] + try: + female_numeric = female_source.to_numpy(dtype=np.float64, na_value=np.nan) + except (TypeError, ValueError) as exc: + raise SourceRuntimeError( + "Pregnancy structural policy requires boolean is_female." + ) from exc + female = female_numeric == 1.0 + invalid_female = ~np.isfinite(female_numeric) | ~np.isin( + female_numeric, + [0.0, 1.0], + ) + elif set(US_PREGNANCY_REQUIRED_SOURCE_COLUMNS) <= set(person.columns): + female_source = person["A_SEX"] + age_source = person["A_AGE"] + female_numeric = pd.to_numeric(female_source, errors="coerce").to_numpy( + dtype=np.float64 + ) + female = female_numeric == _FEMALE_SEX_CODE + invalid_female = ~np.isfinite(female_numeric) + else: + raise SourceRuntimeError( + "Pregnancy structural policy requires either mapped age/is_female " + "or raw A_AGE/A_SEX columns." + ) + age = pd.to_numeric(age_source, errors="coerce").to_numpy(dtype=np.float64) + invalid_age = ~np.isfinite(age) + if invalid_age.any() or invalid_female.any(): + raise SourceRuntimeError( + "Pregnancy structural policy requires complete sex and age; found " + f"invalid_sex_rows={int(invalid_female.sum())}, " + f"invalid_age_rows={int(invalid_age.sum())}." + ) + low, high = _CHILDBEARING_AGE_RANGE + return female, female & (age >= low) & (age <= high) + + +def _pregnancy_boolean_masks( + person: pd.DataFrame, +) -> tuple[np.ndarray, np.ndarray, int]: + """Return observed/true masks and count non-boolean physical values.""" + + values = person[US_PREGNANCY_OUTPUT_COLUMN] + observed = values.notna().to_numpy(dtype=bool) + try: + numeric = values.to_numpy(dtype=np.float64, na_value=np.nan) + except (TypeError, ValueError): + return observed, np.zeros(len(values), dtype=bool), int(observed.sum()) + valid = ~observed | (numeric == 0.0) | (numeric == 1.0) + return observed, observed & (numeric == 1.0), int((~valid).sum()) + + +def _pregnancy_structural_counts(person: pd.DataFrame) -> dict[str, int]: + """Count every hard-domain and all-clone structural condition.""" + + female, eligible = _pregnancy_eligibility(person) + observed, pregnant, non_boolean = _pregnancy_boolean_masks(person) + missing = int((~observed).sum()) + nonfemale = int((pregnant & ~female).sum()) + outside_age = int((pregnant & female & ~eligible).sum()) + ineligible = int((pregnant & ~eligible).sum()) + source_persons = clone_disagreements = malformed_source_ids = 0 + if _PERSON_SOURCE_ID_COLUMN in person: + source_ids = person[_PERSON_SOURCE_ID_COLUMN] + malformed_source_ids = int(source_ids.isna().sum()) + if not malformed_source_ids: + source_persons = int(source_ids.nunique()) + work = pd.DataFrame( + { + "_source": source_ids.to_numpy(), + "_observed": observed, + "_pregnant": pregnant, + } + ) + observed_work = work.loc[work["_observed"]] + clone_disagreements = int( + observed_work.groupby("_source", sort=False)["_pregnant"] + .nunique() + .gt(1) + .sum() + ) + return { + "missing_rows": missing, + "non_boolean_rows": non_boolean, + "pregnant_nonfemale_rows": nonfemale, + "pregnant_female_outside_age_range_rows": outside_age, + "pregnant_ineligible_rows": ineligible, + "source_persons_checked": source_persons, + "clone_disagreement_source_persons": clone_disagreements, + "malformed_source_id_rows": malformed_source_ids, + } + + +def _require_valid_pregnancy_structure( + person: pd.DataFrame, + *, + boundary: str, + require_complete: bool, +) -> dict[str, int]: + """Refuse a persisted pregnancy surface that violates its hard contract.""" + + counts = _pregnancy_structural_counts(person) + refused = { + "non_boolean_rows": counts["non_boolean_rows"], + "pregnant_ineligible_rows": counts["pregnant_ineligible_rows"], + "clone_disagreement_source_persons": counts[ + "clone_disagreement_source_persons" + ], + "malformed_source_id_rows": counts["malformed_source_id_rows"], + } + if require_complete: + refused["missing_rows"] = counts["missing_rows"] + if any(refused.values()): + rendered = ", ".join(f"{key}={value}" for key, value in refused.items()) + raise SourceRuntimeError( + f"{boundary}: pregnancy structural policy refused {rendered}." + ) + return counts + + def derive_us_pregnancy_from_manifest( frame: pd.DataFrame | None, operation: SourceOperationSpec, @@ -221,10 +347,10 @@ def _pregnancy_carries_signal(person: pd.DataFrame) -> bool: def with_us_pregnancy_inputs(frame: Frame, *, seed: int, time_period: int) -> Frame: """Run the ``pregnancy`` manifest stage over a US frame. - A frame already carrying a non-constant ``is_pregnant`` passes through - untouched (idempotent). Any other surface — column missing, or - constant at the engine default — is reseeded from the raw ASEC - columns. + A frame already carrying a complete, non-constant, structurally valid + ``is_pregnant`` passes through untouched (idempotent). Any other valid + surface — column missing, or constant at the engine default — is reseeded + from the raw ASEC columns. Domain violations are refused. Args: frame: A US-schema frame whose person table still carries the raw @@ -245,10 +371,15 @@ def with_us_pregnancy_inputs(frame: Frame, *, seed: int, time_period: int) -> Fr if frame.schema != US_SCHEMA: raise ValueError("US pregnancy inputs require the US schema.") person = frame.table("person") - if US_PREGNANCY_OUTPUT_COLUMN in person.columns and _pregnancy_carries_signal( - person - ): - return frame + if US_PREGNANCY_OUTPUT_COLUMN in person.columns: + carries_signal = _pregnancy_carries_signal(person) + _require_valid_pregnancy_structure( + person, + boundary="US pregnancy preexisting input", + require_complete=carries_signal, + ) + if carries_signal: + return frame stage_person = person.copy(deep=True) stage_person[_PERSON_WEIGHT_COLUMN] = frame.resolve_weights("person").values @@ -271,6 +402,11 @@ def with_us_pregnancy_inputs(frame: Frame, *, seed: int, time_period: int) -> Fr tables["person"][US_PREGNANCY_OUTPUT_COLUMN] = aligned[ US_PREGNANCY_OUTPUT_COLUMN ].to_numpy(dtype=bool) + _require_valid_pregnancy_structure( + tables["person"], + boundary="US pregnancy stage output", + require_complete=True, + ) return Frame( tables, frame.schema, @@ -287,7 +423,7 @@ def us_pregnancy_summary(frame: Frame) -> dict[str, object]: person = frame.table("person") weights = np.asarray(frame.resolve_weights("person").values, dtype=np.float64) total_weight = float(weights.sum()) - pregnant = person[US_PREGNANCY_OUTPUT_COLUMN].astype(bool).to_numpy() + _observed, pregnant, _non_boolean = _pregnancy_boolean_masks(person) pregnant_share = ( float(weights[pregnant].sum()) / total_weight if total_weight > 0 else 0.0 ) @@ -295,6 +431,7 @@ def us_pregnancy_summary(frame: Frame) -> dict[str, object]: "pregnant_share": pregnant_share, "pregnant_share_band": list(_PREGNANT_SHARE_BAND), "unique_count": int(person[US_PREGNANCY_OUTPUT_COLUMN].dropna().nunique()), + **_pregnancy_structural_counts(person), } @@ -316,7 +453,15 @@ def us_pregnancy_signal_gate(frame: Frame) -> GateResult: ) failures: list[str] = [] - summary = us_pregnancy_summary(frame) + try: + summary = us_pregnancy_summary(frame) + except SourceRuntimeError as exc: + return GateResult( + name="pregnancy_signal", + passed=False, + failures=(str(exc),), + details={"structural_error": str(exc)}, + ) if int(summary["unique_count"]) < 2: failures.append( f"{US_PREGNANCY_OUTPUT_COLUMN}: constant column (one observed " @@ -328,6 +473,16 @@ def us_pregnancy_signal_gate(frame: Frame) -> GateResult: failures.append( f"pregnant share {share:.4f} outside plausibility band [{low}, {high}]." ) + for count_key, label in ( + ("missing_rows", "missing pregnancy row(s)"), + ("non_boolean_rows", "non-boolean pregnancy row(s)"), + ("pregnant_ineligible_rows", "pregnant row(s) outside female ages 15--44"), + ("clone_disagreement_source_persons", "source-person clone disagreement(s)"), + ("malformed_source_id_rows", "malformed person_source_id row(s)"), + ): + count = int(summary[count_key]) + if count: + failures.append(f"{US_PREGNANCY_OUTPUT_COLUMN}: {count} {label}.") return GateResult( name="pregnancy_signal", passed=not failures, diff --git a/packages/microcosm-build/src/microcosm/build/us_runtime/prior_year_income.py b/packages/microcosm-build/src/microcosm/build/us_runtime/prior_year_income.py index 254181ca5..ab3e56d6f 100644 --- a/packages/microcosm-build/src/microcosm/build/us_runtime/prior_year_income.py +++ b/packages/microcosm-build/src/microcosm/build/us_runtime/prior_year_income.py @@ -20,6 +20,7 @@ from __future__ import annotations +from collections.abc import Mapping from importlib.resources import files from typing import Any @@ -39,6 +40,7 @@ run_source_stage, ) from microcosm.build.us_runtime.support_provenance import ( + has_assembled_support_metadata, has_support_role_metadata, support_role_series, without_support_role_metadata, @@ -180,6 +182,47 @@ ) _PREVIOUS_YEAR_AVAILABLE_SHARE_BAND = (0.05, 0.50) _SELF_EMPLOYMENT_NONZERO_SHARE_BAND = (0.01, 0.25) +_STACKED_SPINE_MANIFEST_METADATA_KEY = "us_stacked_spine_manifest" +_PRODUCTION_STACKED_SPINE_MANIFEST_VERSION = 4 + + +def _previous_year_availability_match_survival_factor(frame: Frame) -> float: + """Return the production-wide ASEC sampled-match survival factor. + + Production stacked assembly samples every raw ASEC year independently + before the adjacent-year join, so the configured all-survey rung scales + the join's expected availability floor. The version-1 pilot sampled ACS + only and left ASEC whole; unstacked and legacy frames therefore retain the + authored floor exactly. + """ + + manifest = frame.metadata.get(_STACKED_SPINE_MANIFEST_METADATA_KEY) + if manifest is None: + return 1.0 + if not isinstance(manifest, Mapping): + raise ValueError( + "US prior-year-income availability requires the stacked spine " + "manifest metadata to be an object." + ) + version = manifest.get("version") + if version == 1: + return 1.0 + if version != _PRODUCTION_STACKED_SPINE_MANIFEST_VERSION: + raise ValueError( + "US prior-year-income availability received an unsupported stacked " + f"spine manifest version {version!r}." + ) + factor = manifest.get("sample_fraction") + if ( + type(factor) is not float + or not np.isfinite(factor) + or not 0.0 < factor <= 1.0 + ): + raise ValueError( + "US prior-year-income availability requires a finite production " + "stacked sample_fraction in (0, 1]." + ) + return factor def us_prior_year_income_stage_spec() -> SourceStageSpec: @@ -750,7 +793,9 @@ def _share(mask: np.ndarray) -> float: } ) group_columns = ["_source_id"] - if has_support_role_metadata(person, entity="person"): + if has_support_role_metadata( + person, entity="person" + ) and not has_assembled_support_metadata(person, entity="person"): clone_work["_role"] = support_role_series( person, entity="person" ).to_numpy() @@ -841,12 +886,25 @@ def us_prior_year_income_signal_gate(frame: Frame) -> GateResult: details={"nonfinite_rows": rows}, ) + match_survival_factor = _previous_year_availability_match_survival_factor(frame) summary = us_prior_year_income_summary(frame) + availability_band_key = "previous_year_income_available_share_band" + if match_survival_factor != 1.0: + authored_lower, authored_upper = summary[availability_band_key] + applied_floor = float(authored_lower) * match_survival_factor + availability_band_key = ( + "previous_year_income_available_applied_share_band" + ) + summary[ + "previous_year_income_available_sampled_match_survival_factor" + ] = match_survival_factor + summary["previous_year_income_available_applied_floor"] = applied_floor + summary[availability_band_key] = [applied_floor, authored_upper] failures: list[str] = [] checks = ( ( "previous_year_income_available_share", - "previous_year_income_available_share_band", + availability_band_key, "previous-year availability weighted share", ), ( diff --git a/packages/microcosm-build/src/microcosm/build/us_runtime/puf_support.py b/packages/microcosm-build/src/microcosm/build/us_runtime/puf_support.py index c359eaeed..d78c70a23 100644 --- a/packages/microcosm-build/src/microcosm/build/us_runtime/puf_support.py +++ b/packages/microcosm-build/src/microcosm/build/us_runtime/puf_support.py @@ -41,6 +41,7 @@ BASE_ASEC_SUPPORT_CHANNEL, PUF_TAX_DETAIL_CLONE_INDEX, PUF_TAX_DETAIL_SUPPORT_CHANNEL, + has_assembled_support_metadata, has_support_role_metadata, puf_tax_detail_clone_mask, spine_source_id_column, @@ -73,6 +74,7 @@ "bind_puf_clone_attachment_tail_descendant", "clone_us_frame_for_puf_support", "finalize_us_puf_tax_detail_predictions", + "has_assembled_support_metadata", "has_support_role_metadata", "impute_us_puf_tax_detail_support", "puf_tax_detail_clone_mask", diff --git a/packages/microcosm-build/src/microcosm/build/us_runtime/retirement_contributions.py b/packages/microcosm-build/src/microcosm/build/us_runtime/retirement_contributions.py index 753110d3e..3265139ab 100644 --- a/packages/microcosm-build/src/microcosm/build/us_runtime/retirement_contributions.py +++ b/packages/microcosm-build/src/microcosm/build/us_runtime/retirement_contributions.py @@ -38,7 +38,9 @@ run_source_stage, ) from microcosm.build.us_runtime.support_provenance import ( + has_assembled_support_metadata, has_support_role_metadata, + support_gate_source_channel_series, support_role_series, ) from microcosm.frame import Frame @@ -167,6 +169,30 @@ def _numeric_source(frame: pd.DataFrame, column: str) -> np.ndarray: return values +def _asec_source_mask(frame: pd.DataFrame) -> np.ndarray: + """Select physical ASEC rows while retaining the legacy all-row source.""" + + if not has_assembled_support_metadata(frame, entity="person"): + return np.ones(len(frame), dtype=bool) + source_channels = support_gate_source_channel_series(frame, entity="person") + mask = source_channels.eq(_BASE_ASEC_SUPPORT_CHANNEL).to_numpy() + if not mask.any(): + raise SourceRuntimeError( + "US retirement-contribution support has no physical ASEC source rows." + ) + return mask + + +def _source_reconciliation_mask(frame: pd.DataFrame) -> np.ndarray: + """Select direct ASEC operator rows whose allocation remains source-exact.""" + + source_mask = _asec_source_mask(frame) + if not has_assembled_support_metadata(frame, entity="person"): + return source_mask + roles = support_role_series(frame, entity="person").to_numpy() + return source_mask & (roles == _BASE_ASEC_SUPPORT_CHANNEL) + + def derive_us_retirement_contributions_from_manifest( frame: pd.DataFrame | None, operation: SourceOperationSpec, @@ -569,11 +595,16 @@ def us_retirement_contributions_summary(frame: Frame) -> dict[str, object]: ) for column in US_RETIREMENT_CONTRIBUTION_OUTPUT_COLUMNS } - source = ( - _numeric_source(person, "RETCB_VAL") - if "RETCB_VAL" in person - else np.zeros(len(person), dtype=np.float64) - ) + source = np.full(len(person), np.nan, dtype=np.float64) + source_mask = np.zeros(len(person), dtype=bool) + reconciliation_mask = np.zeros(len(person), dtype=bool) + if "RETCB_VAL" in person: + source_mask = _asec_source_mask(person) + source[source_mask] = _numeric_source( + person.loc[source_mask], + "RETCB_VAL", + ) + reconciliation_mask = _source_reconciliation_mask(person) combined = np.sum(np.column_stack(tuple(contributions.values())), axis=1) def _share(values: np.ndarray) -> float: @@ -590,7 +621,7 @@ def _share(values: np.ndarray) -> float: column: int(np.count_nonzero(values < 0)) for column, values in contributions.items() } - source_positive = source > 0 + source_positive = reconciliation_mask & (source > 0) allocation_mismatch = source_positive & ~np.isclose( combined, source, @@ -608,9 +639,13 @@ def _share(values: np.ndarray) -> float: "nonzero_share_band": list(_NONZERO_SHARE_BAND), "nonfinite": nonfinite, "negative": negative, + "source_rows": int(np.count_nonzero(source_mask)), + "source_reconciliation_rows": int(np.count_nonzero(reconciliation_mask)), "source_positive_rows": int(np.count_nonzero(source_positive)), "allocation_mismatch_rows": int(np.count_nonzero(allocation_mismatch)), - "source_total": float(np.sum(source * weights)), + "source_total": float( + np.sum(np.where(reconciliation_mask, source, 0.0) * weights) + ), "allocated_total": float(np.sum(np.nan_to_num(combined) * weights)), } diff --git a/packages/microcosm-build/src/microcosm/build/us_runtime/retirement_distributions.py b/packages/microcosm-build/src/microcosm/build/us_runtime/retirement_distributions.py index 7f8d1427c..7d9659af0 100644 --- a/packages/microcosm-build/src/microcosm/build/us_runtime/retirement_distributions.py +++ b/packages/microcosm-build/src/microcosm/build/us_runtime/retirement_distributions.py @@ -43,7 +43,9 @@ run_source_stage, ) from microcosm.build.us_runtime.support_provenance import ( + has_assembled_support_metadata, has_support_role_metadata, + support_gate_source_channel_series, support_role_series, ) from microcosm.frame import Frame @@ -269,6 +271,30 @@ def _derived_outputs( return outputs +def _asec_source_mask(frame: pd.DataFrame) -> np.ndarray: + """Select physical ASEC rows while retaining the legacy all-row source.""" + + if not has_assembled_support_metadata(frame, entity="person"): + return np.ones(len(frame), dtype=bool) + source_channels = support_gate_source_channel_series(frame, entity="person") + mask = source_channels.eq(_BASE_ASEC_SUPPORT_CHANNEL).to_numpy() + if not mask.any(): + raise SourceRuntimeError( + "US retirement-distribution support has no physical ASEC source rows." + ) + return mask + + +def _source_reconciliation_mask(frame: pd.DataFrame) -> np.ndarray: + """Select direct operator rows whose outputs remain measured-source exact.""" + + compare = _asec_source_mask(frame).copy() + if has_support_role_metadata(frame, entity="person"): + roles = support_role_series(frame, entity="person").to_numpy() + compare &= roles == _BASE_ASEC_SUPPORT_CHANNEL + return compare + + def derive_us_retirement_distributions_from_manifest( frame: pd.DataFrame | None, operation: SourceOperationSpec, @@ -674,17 +700,26 @@ def us_retirement_distributions_summary(frame: Frame) -> dict[str, object]: } source_mismatches: dict[str, int] = {} + source_rows = 0 + source_reconciliation_rows = 0 if all( column in person for column in US_RETIREMENT_DISTRIBUTION_REQUIRED_SOURCE_COLUMNS ): - expected = _derived_outputs(person, _EXPECTED_OUTPUT_BY_ACCOUNT_CODE) - compare = np.ones(len(person), dtype=bool) - if has_support_role_metadata(person, entity="person"): - compare = ( - support_role_series(person, entity="person").to_numpy() - == _BASE_ASEC_SUPPORT_CHANNEL - ) + source_mask = _asec_source_mask(person) + source_expected = _derived_outputs( + person.loc[source_mask], + _EXPECTED_OUTPUT_BY_ACCOUNT_CODE, + ) + expected = { + column: np.full(len(person), np.nan, dtype=np.float64) + for column in US_RETIREMENT_DISTRIBUTION_OUTPUT_COLUMNS + } + for column, source_values in source_expected.items(): + expected[column][source_mask] = source_values + compare = _source_reconciliation_mask(person) + source_rows = int(np.count_nonzero(source_mask)) + source_reconciliation_rows = int(np.count_nonzero(compare)) source_mismatches = { column: int( np.count_nonzero( @@ -722,6 +757,8 @@ def _share(array: np.ndarray) -> float: "negative": { column: int(np.count_nonzero(array < 0)) for column, array in values.items() }, + "source_rows": source_rows, + "source_reconciliation_rows": source_reconciliation_rows, "source_mismatches": source_mismatches, } diff --git a/packages/microcosm-build/src/microcosm/build/us_runtime/sipp_head_start.py b/packages/microcosm-build/src/microcosm/build/us_runtime/sipp_head_start.py index e7c61f01a..5d45b4764 100644 --- a/packages/microcosm-build/src/microcosm/build/us_runtime/sipp_head_start.py +++ b/packages/microcosm-build/src/microcosm/build/us_runtime/sipp_head_start.py @@ -35,7 +35,9 @@ from microcosm.build.gates import GateResult from microcosm.build.source_manifest import SourceStageSpec, load_source_manifest from microcosm.build.us_runtime.support_provenance import ( + has_assembled_support_metadata, has_support_role_metadata, + support_clone_index_column, support_role_series, ) from microcosm.build.us_runtime.voluntary_filing import ( @@ -481,7 +483,7 @@ def _support_group_keys( person: pd.DataFrame, source_id: pd.Series, ) -> tuple[pd.Series, pd.Series]: - """Return clone-pair keys without consulting source-spine channels.""" + """Return source-person keys and validated legacy-compatible roles.""" if not has_support_role_metadata(person, entity="person"): return ( @@ -571,30 +573,58 @@ def _recipient_predictors(frame: Frame) -> tuple[pd.DataFrame, pd.Series, np.nda "US SIPP Head Start source clones disagree on age for " f"person_source_id(s): {inconsistent[:5]}." ) - role_rows = pd.DataFrame({"source_id": source_id, "role": roles}) - duplicate_roles = role_rows.duplicated( - ["source_id", "role"], - keep=False, - ) - if duplicate_roles.any(): - bad = ( - role_rows.loc[duplicate_roles, ["source_id", "role"]] - .drop_duplicates() - .itertuples(index=False, name=None) - ) - raise ValueError( - "US SIPP Head Start source units carry duplicated same-role rows; " - f"invalid source role(s): {list(bad)[:5]}." - ) - order = pd.DataFrame(index=person.index) order["source_id"] = source_id order["source_key"] = source_key - order["role_priority"] = roles.map({_ASEC_CHANNEL: 0, _PUF_CHANNEL: 1}) + if has_assembled_support_metadata(person, entity="person"): + clone_column = support_clone_index_column("person") + clone_indices = pd.to_numeric(person[clone_column], errors="raise").astype( + "int64" + ) + clone_rows = pd.DataFrame( + {"source_id": source_id, "clone_index": clone_indices} + ) + duplicate_clones = clone_rows.duplicated( + ["source_id", "clone_index"], + keep=False, + ) + if duplicate_clones.any(): + bad = ( + clone_rows.loc[ + duplicate_clones, + ["source_id", "clone_index"], + ] + .drop_duplicates() + .itertuples(index=False, name=None) + ) + raise ValueError( + "US SIPP Head Start assembled source units carry duplicated " + f"clone-index rows; invalid source clone(s): {list(bad)[:5]}." + ) + order["clone_priority"] = clone_indices + priority_column = "clone_priority" + else: + role_rows = pd.DataFrame({"source_id": source_id, "role": roles}) + duplicate_roles = role_rows.duplicated( + ["source_id", "role"], + keep=False, + ) + if duplicate_roles.any(): + bad = ( + role_rows.loc[duplicate_roles, ["source_id", "role"]] + .drop_duplicates() + .itertuples(index=False, name=None) + ) + raise ValueError( + "US SIPP Head Start source units carry duplicated same-role " + f"rows; invalid source role(s): {list(bad)[:5]}." + ) + order["role_priority"] = roles.map({_ASEC_CHANNEL: 0, _PUF_CHANNEL: 1}) + priority_column = "role_priority" order["person_key"] = person["person_id"].astype(str) canonical_index = ( order.sort_values( - ["source_id", "role_priority", "person_key"], + ["source_id", priority_column, "person_key"], kind="mergesort", ) .drop_duplicates("source_key", keep="first") diff --git a/packages/microcosm-build/src/microcosm/build/us_runtime/ssi_disability_criteria.py b/packages/microcosm-build/src/microcosm/build/us_runtime/ssi_disability_criteria.py index c90d8169b..fa7492764 100644 --- a/packages/microcosm-build/src/microcosm/build/us_runtime/ssi_disability_criteria.py +++ b/packages/microcosm-build/src/microcosm/build/us_runtime/ssi_disability_criteria.py @@ -19,9 +19,12 @@ The extended-CPS pipeline predicted its ASEC and PUF-support people separately, because the latter carried separately imputed income and asset predictors. We do the same. Direct under-65 ASEC ``SSI_VAL`` reporters are then preserved as -positive anchors; that anchor is not copied onto the PUF channel. An arbitrary -pre-existing criterion column is never trusted: every run recomputes the full -source-backed surface and uses an equality check only for idempotent return. +positive anchors; that anchor is not copied onto the PUF channel. A stacked +ACS row instead contributes its harmonized native ``ssi_reported`` value to the +same ``> 0`` predicate. The row-wise coalesce is source-blind and preserves +the real below-age-15 ACS amount-universe blank. An arbitrary pre-existing +criterion column is never trusted: every run recomputes the full source-backed +surface and uses an equality check only for idempotent return. The full 2023 SIPP public-use file is the same immutable 3.73 GB artifact already pinned by the vehicle and voluntary-filing stages. It contains 39,513 @@ -42,6 +45,7 @@ from microcosm.build.gates import GateResult from microcosm.build.source_manifest import SourceStageSpec, load_source_manifest from microcosm.build.us_runtime.support_provenance import ( + has_assembled_support_metadata, has_support_role_metadata, support_role_series, ) @@ -751,6 +755,64 @@ def _strict_person_aggregate( ) +def _reported_ssi_anchor(person: pd.DataFrame, *, age: np.ndarray) -> np.ndarray: + """Coalesce CPS and harmonized ACS reporter amounts for the ``> 0`` test. + + ``SSI_VAL`` is the measured CPS ASEC amount. ``ssi_reported`` is the + adjusted native ACS SSIP amount produced by ``map_acs_native_inputs``. + Stacked rows carry exactly one of the two. ACS SSIP is out of universe + below age 15, so that genuine blank is interpreted only at predicate time + and is never rewritten into a fake measured zero. + """ + + available = [column for column in ("SSI_VAL", "ssi_reported") if column in person] + if not available: + raise ValueError( + "US SSI disability receiver requires measured SSI_VAL or harmonized " + "ssi_reported for the under-65 reporter anchor." + ) + + numeric: dict[str, pd.Series] = {} + for column in available: + raw = person[column] + values = pd.to_numeric(raw, errors="coerce") + invalid = raw.notna() & values.isna() + finite = np.isfinite(values.fillna(0.0).to_numpy(dtype=np.float64)) + if invalid.any() or not finite.all(): + raise ValueError( + "US SSI disability receiver reported SSI source " + f"{column!r} contains nonnumeric or nonfinite values." + ) + numeric[column] = values + + combined = pd.Series(np.nan, index=person.index, dtype=np.float64) + if "SSI_VAL" in numeric: + combined = numeric["SSI_VAL"].copy() + if "ssi_reported" in numeric: + if "SSI_VAL" in numeric: + both = numeric["SSI_VAL"].notna() & numeric["ssi_reported"].notna() + positivity_conflict = both & numeric["SSI_VAL"].gt(0).ne( + numeric["ssi_reported"].gt(0) + ) + if positivity_conflict.any(): + raise ValueError( + "US SSI disability receiver SSI_VAL and ssi_reported " + "disagree on reporter status." + ) + combined = combined.combine_first(numeric["ssi_reported"]) + + age_values = np.asarray(age, dtype=np.float64) + if len(age_values) != len(combined) or not np.isfinite(age_values).all(): + raise ValueError("US SSI disability receiver age must be finite.") + invalid_blank = combined.isna().to_numpy() & (age_values >= 15.0) + if invalid_blank.any(): + raise ValueError( + "US SSI disability receiver reported SSI amount may be blank only " + "below the ACS age-15 universe." + ) + return combined.fillna(0.0).to_numpy(dtype=np.float64) + + def _person_ssi_disability_predictors(frame: Frame) -> pd.DataFrame: """Build the exact nineteen predictors on every recipient support row.""" @@ -1019,20 +1081,9 @@ def impute_us_ssi_disability_criteria( # The archived direct-CPS pass preserves measured SSI reporters. Its PUF # clone override does not, even though raw ASEC columns were duplicated. - if "SSI_VAL" not in person: - raise ValueError( - "US SSI disability receiver requires measured ASEC SSI_VAL for the " - "under-65 reporter anchor." - ) - reported_ssi = ( - _strict_person_numeric( - person, - ("SSI_VAL",), - label="reported SSI", - ) - > 0.0 - ) - under_65 = receiver["age"].to_numpy(dtype=np.float64) < 65.0 + receiver_age = receiver["age"].to_numpy(dtype=np.float64) + reported_ssi = _reported_ssi_anchor(person, age=receiver_age) > 0.0 + under_65 = receiver_age < 65.0 if has_support_role_metadata(person, entity="person"): channels = support_role_series(person, entity="person") asec = channels.eq(_BASE_ASEC_SUPPORT_CHANNEL).to_numpy() @@ -1137,24 +1188,28 @@ def us_ssi_disability_criteria_summary(frame: Frame) -> dict[str, object]: "value": values, } ) - clone_table["source_occurrence"] = clone_table.groupby( - ["source_id", "role"], sort=False - ).cumcount() - unique = clone_table.groupby( - ["source_id", "source_occurrence"], sort=False - )["value"].nunique(dropna=False) + if has_assembled_support_metadata(person, entity="person"): + clone_groups = ["source_id"] + else: + clone_table["source_occurrence"] = clone_table.groupby( + ["source_id", "role"], sort=False + ).cumcount() + clone_groups = ["source_id", "source_occurrence"] + unique = clone_table.groupby(clone_groups, sort=False)["value"].nunique( + dropna=False + ) clone_divergence_source_people = int((unique > 1).sum()) - reporter_mismatches = 0 - if "SSI_VAL" in person: - reported = pd.to_numeric(person["SSI_VAL"], errors="coerce").fillna(0.0) > 0.0 - age_column = "age" if "age" in person else "A_AGE" - age = pd.to_numeric(person[age_column], errors="coerce") - asec = pd.Series(True, index=person.index) - if channel_values is not None: - asec = channel_values.eq(_BASE_ASEC_SUPPORT_CHANNEL) - anchor = (reported & age.lt(65.0) & asec).to_numpy() - reporter_mismatches = int(np.count_nonzero(anchor & ~positive)) + age_column = "age" if "age" in person else "A_AGE" + age = pd.to_numeric(person[age_column], errors="coerce").to_numpy( + dtype=np.float64 + ) + reported = _reported_ssi_anchor(person, age=age) > 0.0 + native_role = np.ones(len(person), dtype=bool) + if channel_values is not None: + native_role = channel_values.eq(_BASE_ASEC_SUPPORT_CHANNEL).to_numpy() + anchor = reported & (age < 65.0) & native_role + reporter_mismatches = int(np.count_nonzero(anchor & ~positive)) return { "weighted_true_share": float(weights[positive].sum()) / total_weight, @@ -1208,8 +1263,8 @@ def us_ssi_disability_criteria_signal_gate(frame: Frame) -> GateResult: failures.append(f"{_OUTPUT}: weighted false total is not positive.") if summary["reporter_anchor_mismatches"]: failures.append( - f"{_OUTPUT}: {summary['reporter_anchor_mismatches']} under-65 ASEC " - "SSI reporter anchor(s) were lost." + f"{_OUTPUT}: {summary['reporter_anchor_mismatches']} under-65 " + "native-role SSI reporter anchor(s) were lost." ) if summary["support_provenance_missing"]: failures.append( diff --git a/packages/microcosm-build/src/microcosm/build/us_runtime/ssi_take_up.py b/packages/microcosm-build/src/microcosm/build/us_runtime/ssi_take_up.py index 0b8b035cf..86f208960 100644 --- a/packages/microcosm-build/src/microcosm/build/us_runtime/ssi_take_up.py +++ b/packages/microcosm-build/src/microcosm/build/us_runtime/ssi_take_up.py @@ -77,6 +77,7 @@ PUF_TAX_DETAIL_SUPPORT_CHANNEL, has_support_role_metadata, support_clone_index_column, + support_gate_source_channel_series, support_role_series, ) from microcosm.frame import Frame @@ -530,17 +531,20 @@ def us_ssi_take_up_reporter_source_ids(frame: Frame) -> frozenset[str]: if person[_SOURCE_ID].isna().any(): raise ValueError("US SSI take-up reporter lineage requires provenance.") source_ids = _decoded_strings(person[_SOURCE_ID]) - roles = support_role_series(person, entity="person") + source_channels = support_gate_source_channel_series(person, entity="person") + asec_source = source_channels.eq(_ASEC_CHANNEL).to_numpy() reported = pd.to_numeric(person[US_SSI_TAKE_UP_ANCHOR], errors="coerce").to_numpy( dtype=np.float64 ) - if source_ids.str.strip().eq("").any() or not np.isfinite(reported).all(): + if source_ids.str.strip().eq("").any() or not np.isfinite( + reported[asec_source] + ).all(): raise ValueError( "US SSI take-up reporter lineage requires nonblank identities and " - "finite SSI_VAL values." + "finite SSI_VAL values on physical ASEC source rows." ) reporter_ids = frozenset( - source_ids[roles.eq(_ASEC_CHANNEL).to_numpy() & (reported > 0.0)] + source_ids[asec_source & (reported > 0.0)] ) if not reporter_ids: raise ValueError("US SSI take-up found no direct ASEC SSI reporters.") @@ -576,8 +580,6 @@ def _source_table( weights = np.asarray(frame.resolve_weights("person").values, dtype=np.float64) if not np.isfinite(age).all() or (age < 0).any(): raise ValueError("US SSI take-up ages must be finite and nonnegative.") - if not np.isfinite(reported).all(): - raise ValueError("US SSI take-up SSI_VAL anchors must be finite.") if not np.isfinite(potential).all(): raise ValueError("US SSI take-up uncapped_ssi values must be finite.") if not (np.isfinite(weights) & (weights >= 0)).all() or weights.sum() <= 0: @@ -602,21 +604,19 @@ def _source_table( f"unsupported {sorted(observed_channels - _KNOWN_CHANNELS)}." ) - direct_anchor = (reported > 0.0) & channels.eq(_ASEC_CHANNEL).to_numpy() if reporter_source_ids is None: - anchored_source_ids = frozenset(source_ids[direct_anchor]) + anchored_source_ids = us_ssi_take_up_reporter_source_ids(frame) else: anchored_source_ids = frozenset(str(value) for value in reporter_source_ids) if not anchored_source_ids: raise ValueError("US SSI take-up reporter lineage cannot be empty.") if any(not value.strip() for value in anchored_source_ids): raise ValueError("US SSI take-up reporter source identities are nonblank.") - omitted_direct = sorted(set(source_ids[direct_anchor]) - anchored_source_ids) - if omitted_direct: - raise ValueError( - "US SSI take-up reporter lineage omitted direct ASEC anchors; " - f"examples {omitted_direct[:5]}." - ) + anchored_rows = source_ids.isin(anchored_source_ids).to_numpy() + if not np.isfinite(reported[anchored_rows]).all(): + raise ValueError( + "US SSI take-up SSI_VAL anchors must be finite on anchored source rows." + ) rows = pd.DataFrame( { @@ -627,9 +627,9 @@ def _source_table( "weight": weights, "candidate": potential > 0.0, # Capture lineage on the full support before L0. When pruning keeps - # only a PUF clone, the explicit source-ID set still preserves the - # underlying direct ASEC measurement without promoting PUF-only - # SSI_VAL copies into independent anchors. + # only a donor-role clone, the explicit source-ID set still + # preserves the underlying physical ASEC measurement without + # promoting non-ASEC SSI_VAL values into independent anchors. "anchor": source_ids.isin(anchored_source_ids).to_numpy(), }, index=person.index, diff --git a/packages/microcosm-build/src/microcosm/build/us_runtime/stacked_spine.py b/packages/microcosm-build/src/microcosm/build/us_runtime/stacked_spine.py index 879445e0b..ae762b467 100644 --- a/packages/microcosm-build/src/microcosm/build/us_runtime/stacked_spine.py +++ b/packages/microcosm-build/src/microcosm/build/us_runtime/stacked_spine.py @@ -4179,6 +4179,82 @@ def _validate_acs_transfer_row_counts( return typed_counts +_PREGNANCY_STRUCTURAL_COUNT_FIELDS = ( + "source_persons_checked", + "physical_rows_checked", + "clone_rows_checked", + "donor_rows_checked", + "qrf_draw_source_persons", + "qrf_draw_rows", + "qrf_fanout_rows", + "preexisting_value_fanout_rows", + "ineligible_rows_assigned_false", + "donor_preexisting_domain_violation_rows", + "recipient_preexisting_domain_violation_rows", + "preexisting_clone_disagreement_source_persons", + "inconsistent_eligibility_source_persons", + "maximum_clones_per_source_person", + "final_incomplete_rows", + "final_domain_violation_rows", + "final_clone_disagreement_source_persons", +) + + +def _validate_pregnancy_structural_receipt( + structural: object, + *, + row_counts: Mapping[str, int], + boundary: str, +) -> None: + """Authenticate pregnancy's hard domain and source-person accounting.""" + + expected_policy = acs_transfer_runtime.acs_transfer_execution_contract_identity( + targets=("is_pregnant",), + derive_schedule_d=False, + )["structural_target_policies"]["is_pregnant"] + assert isinstance(expected_policy, Mapping) + if ( + not isinstance(structural, Mapping) + or structural.get("policy_sha256") != expected_policy.get("sha256") + or structural.get("source_person_key") != "person_source_id" + or structural.get("status") != "verified" + ): + raise ValueError(f"{boundary}: pregnancy structural policy is invalid.") + counts = {field: structural.get(field) for field in _PREGNANCY_STRUCTURAL_COUNT_FIELDS} + if any( + not isinstance(value, int) or isinstance(value, bool) or value < 0 + for value in counts.values() + ): + raise ValueError(f"{boundary}: pregnancy structural counts are invalid.") + zero_fields = ( + "donor_preexisting_domain_violation_rows", + "recipient_preexisting_domain_violation_rows", + "preexisting_clone_disagreement_source_persons", + "inconsistent_eligibility_source_persons", + "final_incomplete_rows", + "final_domain_violation_rows", + "final_clone_disagreement_source_persons", + ) + if any(counts[field] != 0 for field in zero_fields): + raise ValueError(f"{boundary}: pregnancy structural guard is not green.") + if ( + counts["physical_rows_checked"] - counts["source_persons_checked"] + != counts["clone_rows_checked"] + or counts["qrf_draw_rows"] != counts["qrf_draw_source_persons"] + or counts["maximum_clones_per_source_person"] < 1 + or ( + counts["qrf_draw_rows"] + + counts["qrf_fanout_rows"] + + counts["preexisting_value_fanout_rows"] + + counts["ineligible_rows_assigned_false"] + != row_counts["imputed_rows"] + ) + ): + raise ValueError( + f"{boundary}: pregnancy structural accounting is invalid." + ) + + def _acs_imputed_pattern_evidence(record: AcsImputedInput) -> dict[str, object]: """Return deterministic JSON-ready evidence for one ACS transfer record.""" @@ -4786,11 +4862,24 @@ def validate_stacked_post_puf_transfer_receipt( f"{boundary}: stacked post-PUF target {target_key!r} has " "no receipt." ) - _validate_acs_transfer_row_counts( + row_counts = _validate_acs_transfer_row_counts( target_receipt, boundary=f"{boundary} target {target_key}", required=True, ) + target_name = target_key.rsplit("/", 1)[1] + structural = target_receipt.get("structural_policy") + if target_name == "is_pregnant": + _validate_pregnancy_structural_receipt( + structural, + row_counts=row_counts, + boundary=f"{boundary} target {target_key}", + ) + elif structural is not None: + raise ValueError( + f"{boundary}: undeclared pregnancy structural evidence is " + f"attached to {target_key!r}." + ) owner_receipt = target_receipt.get("post_transfer_calibration") spec = expected_calibrations.get(target_key) if spec is None: @@ -10499,6 +10588,10 @@ def _verify_post_puf_transfer_outcome( target_receipt["qrf_pattern_evidence"] = ( _acs_imputed_pattern_evidence(record) ) + if record is not None and record.structural_receipt is not None: + target_receipt["structural_policy"] = dict( + record.structural_receipt + ) target_receipts[target_receipt_key] = target_receipt if failures: raise ValueError( diff --git a/packages/microcosm-build/src/microcosm/build/us_runtime/support_provenance.py b/packages/microcosm-build/src/microcosm/build/us_runtime/support_provenance.py index da5bf9e08..ec5a51eaf 100644 --- a/packages/microcosm-build/src/microcosm/build/us_runtime/support_provenance.py +++ b/packages/microcosm-build/src/microcosm/build/us_runtime/support_provenance.py @@ -16,6 +16,7 @@ "PUF_TAX_DETAIL_CLONE_INDEX", "PUF_TAX_DETAIL_SUPPORT_CHANNEL", "SPINE_ASSEMBLY_MANIFEST_KEY", + "has_assembled_support_metadata", "has_support_role_metadata", "puf_tax_detail_clone_mask", "spine_assembly_manifest", @@ -25,6 +26,7 @@ "support_channel_column", "support_clone_index_column", "support_role_series", + "support_gate_source_channel_series", "support_source_id_column", "us_reported_coverage_vintage_signal_gate", "validate_assembly_provenance", @@ -326,6 +328,21 @@ def has_support_role_metadata( ) +def has_assembled_support_metadata( + table: pd.DataFrame, + *, + entity: str, +) -> bool: + """Return whether support metadata belongs to a multispine assembly. + + The raw spine-record ID is the assembly discriminator. Historical frames + used the support-channel column for the two operator roles instead, so the + presence of a channel or clone index alone is not sufficient. + """ + + return spine_source_id_column(entity) in table + + def spine_source_id_column(entity: str) -> str: """Return the entity-prefixed raw spine-record ID metadata column.""" @@ -476,6 +493,37 @@ def support_role_series( ) +def support_gate_source_channel_series( + table: pd.DataFrame, + *, + entity: str, +) -> pd.Series: + """Return physical source channels for read-only gates and reporters. + + Assembled frames retain their receipt-declared physical channels (for + example, ``asec`` and ``acs``). Historical frames have no raw spine ID and + use the exact ``asec``/``puf_tax_detail`` operator roles as their channels. + This deliberately narrow accessor must not route population treatments. + Keeping resolution in the provenance owner lets reporting and release gates + inspect source coverage without teaching operators how to read provenance + columns directly. A static call-site contract pins its reviewed consumers. + """ + + roles = support_role_series(table, entity=entity) + if not has_assembled_support_metadata(table, entity=entity): + return pd.Series( + roles.to_numpy(dtype=object, copy=True), + index=table.index, + name=f"{entity}_source_support_channel", + ) + channels = table[support_channel_column(entity)] + return pd.Series( + channels.to_numpy(dtype=object, copy=True), + index=table.index, + name=f"{entity}_source_support_channel", + ) + + def puf_tax_detail_clone_mask( table: pd.DataFrame, *, diff --git a/packages/microcosm-build/src/microcosm/build/us_runtime/us_late_producer_registry.py b/packages/microcosm-build/src/microcosm/build/us_runtime/us_late_producer_registry.py index 1b0f0cc6c..4b8206fa6 100644 --- a/packages/microcosm-build/src/microcosm/build/us_runtime/us_late_producer_registry.py +++ b/packages/microcosm-build/src/microcosm/build/us_runtime/us_late_producer_registry.py @@ -671,6 +671,13 @@ def _inventory( "with_us_pregnancy_inputs", *_raw_person_requirements(("A_SEX", "A_AGE")), _single("person_id", "person", "person_id"), + _single( + "pregnancy_structural_source_person_id", + "person", + "person_source_id", + optional=True, + value_kind="finite_numeric", + ), _single("resolved_person_weight", "person", "@resolved_weight"), _requirement( "stable_source_identity", @@ -1360,6 +1367,12 @@ def _bounded_transfer_groups( batches: list[tuple[str, ...]] = [] current: list[str] = [] for atom in atoms: + if atom == ("is_pregnant",): + if current: + batches.append(tuple(current)) + current = [] + batches.append(atom) + continue if current and len(current) + len(atom) > max_targets_per_fit: batches.append(tuple(current)) current = [] diff --git a/packages/microcosm-build/src/microcosm/build/us_runtime/voluntary_filing.py b/packages/microcosm-build/src/microcosm/build/us_runtime/voluntary_filing.py index e814cb476..b19d32dda 100644 --- a/packages/microcosm-build/src/microcosm/build/us_runtime/voluntary_filing.py +++ b/packages/microcosm-build/src/microcosm/build/us_runtime/voluntary_filing.py @@ -38,7 +38,9 @@ from microcosm.build.gates import GateResult from microcosm.build.source_manifest import SourceStageSpec, load_source_manifest from microcosm.build.us_runtime.support_provenance import ( + has_assembled_support_metadata, has_support_role_metadata, + support_clone_index_column, support_role_series, ) from microcosm.frame import Frame @@ -742,53 +744,77 @@ def _source_receiver_rows( rows["_support_role"] = support_role_series( tax_unit, entity="tax_unit" ).to_numpy() - role_counts = rows.groupby( - ["_source_id", "_support_role"], - sort=False, - ).size() - duplicated_roles = role_counts[role_counts > 1] - if not duplicated_roles.empty: - bad = duplicated_roles.index.tolist() - raise ValueError( - "US voluntary-filing support source units carry duplicated " - f"same-role rows; invalid source role(s) {bad[:5]}." - ) rows["_source_key"] = rows["_source_id"] - asec_counts = ( - rows["_support_role"] - .eq(_BASE_ASEC_SUPPORT_CHANNEL) - .groupby(rows["_source_key"]) - .sum() - ) - if asec_counts.gt(1).any(): - bad = asec_counts.index[asec_counts.gt(1)].tolist() - raise ValueError( - "US voluntary-filing support source units carry duplicated " - f"ASEC rows; invalid source unit(s) {bad[:5]}." - ) - # Prefer each unit's ASEC row, but a frozen-support selection may - # legitimately keep only a unit's PUF clone (the L0-survivor case the - # SSI reporter lineage also handles — Build M's certified 57,240 - # selection does exactly this). Clones carry the unit's source - # predictors, so the surviving row predicts identically; pick it - # deterministically by channel then tax-unit id. ordered_rows = rows.copy() - ordered_rows["_asec_rank"] = ( - ~ordered_rows["_support_role"].eq(_BASE_ASEC_SUPPORT_CHANNEL) - ).astype(int) - source_rows = ( - ordered_rows.sort_values( - [ - "_source_id", - "_asec_rank", - "_support_role", - "_tax_unit_id", - ], - kind="stable", + if has_assembled_support_metadata(tax_unit, entity="tax_unit"): + clone_column = support_clone_index_column("tax_unit") + ordered_rows["_clone_index"] = pd.to_numeric( + tax_unit[clone_column], errors="raise" + ).to_numpy(dtype=np.int64) + clone_counts = ordered_rows.groupby( + ["_source_id", "_clone_index"], + sort=False, + ).size() + duplicated_clones = clone_counts[clone_counts > 1] + if not duplicated_clones.empty: + bad = duplicated_clones.index.tolist() + raise ValueError( + "US voluntary-filing assembled source units carry " + "duplicated clone-index rows; invalid source clone(s) " + f"{bad[:5]}." + ) + source_rows = ( + ordered_rows.sort_values( + ["_source_id", "_clone_index", "_tax_unit_id"], + kind="stable", + ) + .drop_duplicates("_source_key", keep="first") + .drop(columns="_clone_index") + ) + else: + role_counts = rows.groupby( + ["_source_id", "_support_role"], + sort=False, + ).size() + duplicated_roles = role_counts[role_counts > 1] + if not duplicated_roles.empty: + bad = duplicated_roles.index.tolist() + raise ValueError( + "US voluntary-filing support source units carry duplicated " + f"same-role rows; invalid source role(s) {bad[:5]}." + ) + asec_counts = ( + rows["_support_role"] + .eq(_BASE_ASEC_SUPPORT_CHANNEL) + .groupby(rows["_source_key"]) + .sum() + ) + if asec_counts.gt(1).any(): + bad = asec_counts.index[asec_counts.gt(1)].tolist() + raise ValueError( + "US voluntary-filing support source units carry duplicated " + f"ASEC rows; invalid source unit(s) {bad[:5]}." + ) + # Prefer each unit's ASEC row, but a frozen-support selection may + # legitimately keep only a unit's PUF clone. Clones carry the + # unit's source predictors, so the surviving row predicts + # identically; pick it deterministically by role then tax-unit id. + ordered_rows["_asec_rank"] = ( + ~ordered_rows["_support_role"].eq(_BASE_ASEC_SUPPORT_CHANNEL) + ).astype(int) + source_rows = ( + ordered_rows.sort_values( + [ + "_source_id", + "_asec_rank", + "_support_role", + "_tax_unit_id", + ], + kind="stable", + ) + .drop_duplicates("_source_key", keep="first") + .drop(columns="_asec_rank") ) - .drop_duplicates("_source_key", keep="first") - .drop(columns="_asec_rank") - ) else: if rows["_source_id"].duplicated().any(): duplicates = rows.loc[ @@ -1015,10 +1041,13 @@ def us_voluntary_filing_summary(frame: Frame) -> dict[str, object]: "value": values, } ) - clone_table["source_occurrence"] = clone_table.groupby( - ["source_id", "role"], sort=False - ).cumcount() - clone_groups = ["source_id", "source_occurrence"] + if has_assembled_support_metadata(tax_unit, entity="tax_unit"): + clone_groups = ["source_id"] + else: + clone_table["source_occurrence"] = clone_table.groupby( + ["source_id", "role"], sort=False + ).cumcount() + clone_groups = ["source_id", "source_occurrence"] sizes = clone_table.groupby(clone_groups, sort=False).size() clone_source_units = int((sizes > 1).sum()) unique = clone_table.groupby(clone_groups, sort=False)["value"].nunique( diff --git a/packages/microcosm-build/src/microcosm/build/us_runtime/weeks_unemployed.py b/packages/microcosm-build/src/microcosm/build/us_runtime/weeks_unemployed.py index bd156ecbc..dd7e36641 100644 --- a/packages/microcosm-build/src/microcosm/build/us_runtime/weeks_unemployed.py +++ b/packages/microcosm-build/src/microcosm/build/us_runtime/weeks_unemployed.py @@ -41,7 +41,10 @@ run_source_stage, ) from microcosm.build.us_runtime.support_provenance import ( + has_assembled_support_metadata, has_support_role_metadata, + support_clone_index_column, + support_gate_source_channel_series, support_role_series, ) from microcosm.frame import Frame @@ -1220,14 +1223,18 @@ def us_weeks_unemployed_summary(frame: Frame) -> dict[str, object]: in_range = integer & (values >= 0.0) & (values <= 52.0) positive = in_range & (values > 0.0) - if has_support_role_metadata(person, entity="person"): - channel = support_role_series(person, entity="person").to_numpy() - else: - channel = np.full(len(person), _ASEC_CHANNEL, dtype=object) + ( + channel, + channel_roster, + source_rows, + source_reconciliation_rows, + uc_constraint_rows, + ) = _weeks_unemployed_gate_scopes(person) channels: dict[str, dict[str, float | int]] = {} - for name in (_ASEC_CHANNEL, _PUF_CHANNEL): + for name in channel_roster: mask = channel == name channel_weight = float(weights[mask].sum()) + plausibility_role = _ASEC_CHANNEL if name == _ASEC_CHANNEL else _PUF_CHANNEL channels[name] = { "rows": int(np.count_nonzero(mask)), "positive_rows": int(np.count_nonzero(mask & positive)), @@ -1243,8 +1250,12 @@ def us_weeks_unemployed_summary(frame: Frame) -> dict[str, object]: if channel_weight > 0.0 else 0.0 ), - "positive_share_band": list(_CHANNEL_POSITIVE_SHARE_BANDS[name]), - "weighted_mean_weeks_band": list(_CHANNEL_WEIGHTED_MEAN_WEEKS_BANDS[name]), + "positive_share_band": list( + _CHANNEL_POSITIVE_SHARE_BANDS[plausibility_role] + ), + "weighted_mean_weeks_band": list( + _CHANNEL_WEIGHTED_MEAN_WEEKS_BANDS[plausibility_role] + ), } source_missing = _SOURCE not in person @@ -1253,25 +1264,32 @@ def us_weeks_unemployed_summary(frame: Frame) -> dict[str, object]: source_raw = pd.to_numeric(person[_SOURCE], errors="coerce").to_numpy( dtype=np.float64 ) - asec_mask = channel == _ASEC_CHANNEL source_valid = np.isfinite(source_raw) & (source_raw == np.floor(source_raw)) source_valid &= (source_raw == -1.0) | ( (source_raw >= 0.0) & (source_raw <= 52.0) ) - source_invalid = int(np.count_nonzero(asec_mask & ~source_valid)) + source_invalid = int(np.count_nonzero(source_rows & ~source_valid)) expected = np.where(source_raw == -1.0, 0.0, source_raw) source_mismatch = int( - np.count_nonzero(asec_mask & source_valid & finite & (values != expected)) + np.count_nonzero( + source_reconciliation_rows + & source_valid + & finite + & (values != expected) + ) ) - puf_uc_zero_mismatch = 0 + uc_constraint_mismatch = 0 if _OPTIONAL_UC_PREDICTOR in person: uc = pd.to_numeric(person[_OPTIONAL_UC_PREDICTOR], errors="coerce").to_numpy( dtype=np.float64 ) - puf_mask = channel == _PUF_CHANNEL - puf_uc_zero_mismatch = int( + uc_constraint_mismatch = int( np.count_nonzero( - puf_mask & np.isfinite(uc) & (uc <= 0.0) & finite & (values != 0.0) + uc_constraint_rows + & np.isfinite(uc) + & (uc <= 0.0) + & finite + & (values != 0.0) ) ) @@ -1284,15 +1302,86 @@ def us_weeks_unemployed_summary(frame: Frame) -> dict[str, object]: "positive_share": float(weights[positive].sum() / weights.sum()), "weighted_weeks": float(np.dot(weights, np.nan_to_num(values))), "source_missing": source_missing, + "source_rows": int(np.count_nonzero(source_rows)), + "source_reconciliation_rows": int( + np.count_nonzero(source_reconciliation_rows) + ), "source_invalid": source_invalid, "source_mismatch_count": source_mismatch, - "puf_uc_zero_mismatch_count": puf_uc_zero_mismatch, + "uc_constraint_rows": int(np.count_nonzero(uc_constraint_rows)), + "uc_constraint_mismatch_count": uc_constraint_mismatch, + "channel_roster": list(channel_roster), "channels": channels, } +def _weeks_unemployed_gate_scopes( + person: pd.DataFrame, +) -> tuple[np.ndarray, tuple[str, ...], np.ndarray, np.ndarray, np.ndarray]: + """Resolve source channels and the two reviewed constraint scopes.""" + + rows = len(person) + if not has_support_role_metadata(person, entity="person"): + all_rows = np.ones(rows, dtype=bool) + return ( + np.full(rows, _ASEC_CHANNEL, dtype=object), + (_ASEC_CHANNEL,), + all_rows, + all_rows.copy(), + np.zeros(rows, dtype=bool), + ) + + # Validate complete channel/clone metadata even when an assembled frame's + # raw source channels, rather than its compatibility roles, own the gate. + roles = support_role_series(person, entity="person") + assembled = has_assembled_support_metadata(person, entity="person") + if not assembled: + channel = roles.to_numpy(dtype=object, copy=True) + roster = tuple( + name for name in (_ASEC_CHANNEL, _PUF_CHANNEL) if np.any(channel == name) + ) + source_rows = channel == _ASEC_CHANNEL + if not np.any(source_rows): + raise ValueError("US weeks-unemployed support has no ASEC source rows.") + return ( + channel, + roster, + source_rows, + source_rows.copy(), + channel == _PUF_CHANNEL, + ) + + clone_column = support_clone_index_column("person") + source_channel = support_gate_source_channel_series( + person, + entity="person", + ).to_numpy(dtype=object, copy=True) + observed = {str(value) for value in source_channel} + if _ASEC_CHANNEL not in observed: + raise ValueError("US weeks-unemployed support has no ASEC source rows.") + roster = (_ASEC_CHANNEL, *sorted(observed - {_ASEC_CHANNEL})) + clone_index = pd.to_numeric(person[clone_column], errors="raise").to_numpy( + dtype=np.int64 + ) + native = clone_index == 0 + source_rows = source_channel == _ASEC_CHANNEL + # Native ASEC rows are the direct LKWEEKS carry. Non-native ASEC rows are + # the reviewed PUF-detail imputation; native non-ASEC rows are the reviewed + # post-transfer calibration scope. Other non-native source rows have no UC + # zero rule in either producer contract. + source_reconciliation_rows = source_rows & native + uc_constraint_rows = (source_rows & ~native) | (~source_rows & native) + return ( + source_channel, + roster, + source_rows, + source_reconciliation_rows, + uc_constraint_rows, + ) + + def us_weeks_unemployed_signal_gate(frame: Frame) -> GateResult: - """Require exact ASEC carry and integer, nondefault signal on both halves.""" + """Require exact ASEC carry and integer signal on every support channel.""" person = frame.table("person") if _OUTPUT not in person: @@ -1330,15 +1419,12 @@ def us_weeks_unemployed_signal_gate(frame: Frame) -> GateResult: f"{_OUTPUT} has {summary['source_mismatch_count']} ASEC source " "reconciliation mismatch(es)." ) - if int(summary["puf_uc_zero_mismatch_count"]): + if int(summary["uc_constraint_mismatch_count"]): failures.append( - f"{_OUTPUT} has {summary['puf_uc_zero_mismatch_count']} PUF row(s) " - "positive without unemployment compensation." + f"{_OUTPUT} has {summary['uc_constraint_mismatch_count']} row(s) " + "positive outside the unemployment-compensation constraint." ) - has_support_roles = has_support_role_metadata(person, entity="person") - required_channels = ( - (_ASEC_CHANNEL, _PUF_CHANNEL) if has_support_roles else (_ASEC_CHANNEL,) - ) + required_channels = tuple(summary["channel_roster"]) channels = summary["channels"] for name in required_channels: channel = channels[name] diff --git a/packages/microcosm-build/src/microcosm/build/us_runtime/wic_claim.py b/packages/microcosm-build/src/microcosm/build/us_runtime/wic_claim.py index 48accbcef..f6434a741 100644 --- a/packages/microcosm-build/src/microcosm/build/us_runtime/wic_claim.py +++ b/packages/microcosm-build/src/microcosm/build/us_runtime/wic_claim.py @@ -46,6 +46,7 @@ run_source_stage, ) from microcosm.build.us_runtime.support_provenance import ( + has_assembled_support_metadata, has_support_role_metadata, support_role_series, ) @@ -108,6 +109,7 @@ # protocol attests this historical salt separately from the output column. _DRAW_SALT = "would_claim_wic" _PERSON_WEIGHT_COLUMN = "person_weight" +_PERSON_SOURCE_ID_COLUMN = "person_source_id" _PERSON_SUPPORT_SOURCE_ID_COLUMN = "person_support_source_id" _SOURCE_IDENTITY_COLUMNS = ( "source_year", @@ -344,6 +346,25 @@ def _wic_categories(person: pd.DataFrame) -> np.ndarray: def _stable_person_keys(person: pd.DataFrame) -> pd.Series: + # Multispine assembly makes this ID unique across physical source + # channels while retaining it across every support clone. Prefer it before + # the historical source-local triple, whose namespaces can overlap once + # ASEC and ACS vintages align. + assembled = has_assembled_support_metadata(person, entity="person") + if assembled and _PERSON_SOURCE_ID_COLUMN not in person.columns: + raise SourceRuntimeError( + "US WIC claim multispine identity requires person_source_id." + ) + if assembled: + source_id = person[_PERSON_SOURCE_ID_COLUMN] + if source_id.isna().any(): + rows = np.flatnonzero(source_id.isna().to_numpy())[:5].tolist() + raise SourceRuntimeError( + "US WIC claim assembly source identity contains missing values at " + f"row(s): {rows}." + ) + return "source:" + source_id.astype(str) + present = [column in person.columns for column in _SOURCE_IDENTITY_COLUMNS] if any(present) and not all(present): missing = [ diff --git a/packages/microcosm-build/src/microcosm/build/us_runtime/workers_compensation.py b/packages/microcosm-build/src/microcosm/build/us_runtime/workers_compensation.py index b571a1226..80426c5e2 100644 --- a/packages/microcosm-build/src/microcosm/build/us_runtime/workers_compensation.py +++ b/packages/microcosm-build/src/microcosm/build/us_runtime/workers_compensation.py @@ -28,7 +28,9 @@ run_source_stage, ) from microcosm.build.us_runtime.support_provenance import ( + has_assembled_support_metadata, has_support_role_metadata, + support_gate_source_channel_series, support_role_series, ) from microcosm.frame import Frame @@ -570,16 +572,29 @@ def us_workers_compensation_summary(frame: Frame) -> dict[str, object]: dtype=np.float64 ) source_mask = np.ones(len(person), dtype=bool) + source_reconciliation_mask = source_mask.copy() if has_support_role_metadata(person, entity="person"): source_mask = ( - support_role_series(person, entity="person").to_numpy() + support_gate_source_channel_series( + person, entity="person" + ).to_numpy() == _BASE_ASEC_SUPPORT_CHANNEL ) + source_reconciliation_mask = source_mask.copy() + if has_assembled_support_metadata(person, entity="person"): + source_reconciliation_mask &= ( + support_role_series(person, entity="person").to_numpy() + == _BASE_ASEC_SUPPORT_CHANNEL + ) source_valid = np.isfinite(source) & (source >= 0.0) + summary["source_rows"] = int(np.count_nonzero(source_mask)) + summary["source_reconciliation_rows"] = int( + np.count_nonzero(source_reconciliation_mask) + ) summary["source_invalid"] = int(np.count_nonzero(source_mask & ~source_valid)) summary["source_mismatch_count"] = int( np.count_nonzero( - source_mask + source_reconciliation_mask & source_valid & finite & ~np.isclose(values, source, rtol=0.0, atol=0.0) diff --git a/packages/microcosm-build/tests/test_spec_engine_compiler_ir.py b/packages/microcosm-build/tests/test_spec_engine_compiler_ir.py index 393d98dd1..fb141dec1 100644 --- a/packages/microcosm-build/tests/test_spec_engine_compiler_ir.py +++ b/packages/microcosm-build/tests/test_spec_engine_compiler_ir.py @@ -23,7 +23,7 @@ from microcosm.build.spec_engine.resolver import F0_CONTRACT_ONLY_KERNEL_IDS US_SCHEDULE_SHA256 = ( - "dcf3c6d2eade3449836c49a1dc4d3b8cd395aab9142db700c3c60598fa9c1c79" + "e59c019d3d454eac99ac0ac209b6c5b6faaf9bdfcaeee18c36a25be19bf7da2f" ) diff --git a/packages/microcosm-build/tests/test_spec_engine_country_bundles.py b/packages/microcosm-build/tests/test_spec_engine_country_bundles.py index 1bd391ecc..621adc7a6 100644 --- a/packages/microcosm-build/tests/test_spec_engine_country_bundles.py +++ b/packages/microcosm-build/tests/test_spec_engine_country_bundles.py @@ -32,7 +32,7 @@ [ ( "be", - "86143c1c3f98980e34490c75706fe7dbf72e96e8accb8d7d4bbd2c1ae1a29b65", + "7062e38f4d623553fb0604380a8dac0edacb6261c155b6e31fc38ef7c0f1c57c", { "household.household_id", "person.person_id", @@ -42,7 +42,7 @@ ), ( "uk", - "96efa147c0d80107af1e46ec691030aa001a82334d52459a4522a64853fa267e", + "71054f533464eaeac0ea750e4e88d719f69b30bea8dd6977c677e1ed3592334e", { "benunit.benunit_id", "household.household_id", diff --git a/packages/microcosm-build/tests/test_spec_engine_coverage_tool.py b/packages/microcosm-build/tests/test_spec_engine_coverage_tool.py index 7a4e443bf..099e68813 100644 --- a/packages/microcosm-build/tests/test_spec_engine_coverage_tool.py +++ b/packages/microcosm-build/tests/test_spec_engine_coverage_tool.py @@ -52,22 +52,22 @@ def test_us_coverage_is_exact_complete_and_honest( assert_coverage_complete(coverage_report) assert coverage_report["status"] == "pass" fields = coverage_report["field_usage"] - assert fields["configuration_field_count"] == 42_120 - assert fields["authored_normative_field_count"] == 32_351 - assert fields["resolved_binding_field_count"] == 9_769 - assert fields["consumed_field_count"] == 42_120 + assert fields["configuration_field_count"] == 42_154 + assert fields["authored_normative_field_count"] == 32_384 + assert fields["resolved_binding_field_count"] == 9_770 + assert fields["consumed_field_count"] == 42_154 assert fields["unused_field_count"] == 0 assert fields["multiple_primary_use_field_count"] == 0 assert fields["claim_count"] == 49 assert fields["mode_counts"] == { - "legacy_behavior": 13_970, - "compiler_semantic": 27_699, + "legacy_behavior": 13_988, + "compiler_semantic": 27_715, "front_end_validation": 348, "identity_only": 103, } assert fields["generation0_effect_counts"] == { - "legacy_behavior": 38_443, - "no_generation0_effect": 3_677, + "legacy_behavior": 38_476, + "no_generation0_effect": 3_678, } inventory = coverage_report["inventory_coverage"] @@ -75,7 +75,7 @@ def test_us_coverage_is_exact_complete_and_honest( assert inventory["covered_item_count"] == 41 assert inventory["missing_item_count"] == 0 assert inventory["missing_items"] == [] - assert inventory["counts"]["producer_inputs"] == 2_742 + assert inventory["counts"]["producer_inputs"] == 2_744 assert inventory["counts"]["ownership_rows"] == 18 assert inventory["counts"]["tail_control_fields"] == 934 assert inventory["counts"]["seed_owner_bindings"] == 112 diff --git a/packages/microcosm-build/tests/test_spec_engine_field_usage.py b/packages/microcosm-build/tests/test_spec_engine_field_usage.py index 77fad4942..d33908b81 100644 --- a/packages/microcosm-build/tests/test_spec_engine_field_usage.py +++ b/packages/microcosm-build/tests/test_spec_engine_field_usage.py @@ -92,22 +92,22 @@ def _mutated_bundle( def test_exact_complete_ledger_has_one_primary_mode_per_pointer(field_ledger) -> None: - assert len(field_ledger.fields) == EXPECTED_CONFIGURATION_FIELD_COUNT == 42_120 + assert len(field_ledger.fields) == EXPECTED_CONFIGURATION_FIELD_COUNT == 42_154 assert field_ledger.source_counts == { - "authored": 32_351, - "resolved_bindings": 9_769, + "authored": 32_384, + "resolved_bindings": 9_770, } assert field_ledger.mode_counts == { - "legacy_behavior": 13_970, - "compiler_semantic": 27_699, + "legacy_behavior": 13_988, + "compiler_semantic": 27_715, "front_end_validation": 348, "identity_only": 103, } assert field_ledger.generation0_effect_counts == { - "legacy_behavior": 38_443, - "no_generation0_effect": 3_677, + "legacy_behavior": 38_476, + "no_generation0_effect": 3_678, } - assert len({field.pointer for field in field_ledger.fields}) == 42_120 + assert len({field.pointer for field in field_ledger.fields}) == 42_154 def test_eligibility_concepts_are_validation_not_generation0_behavior( diff --git a/packages/microcosm-build/tests/test_spec_engine_imputation_semantics.py b/packages/microcosm-build/tests/test_spec_engine_imputation_semantics.py index 749a793b2..c9ae76814 100644 --- a/packages/microcosm-build/tests/test_spec_engine_imputation_semantics.py +++ b/packages/microcosm-build/tests/test_spec_engine_imputation_semantics.py @@ -98,7 +98,7 @@ def test_imputation_projector_matches_live_plans_and_graph_receipts( for key, expected in live.items(): assert canonical_json_bytes(projected[key]) == canonical_json_bytes(expected) assert projected["late_producer_schedule_receipt"]["schedule_sha256"] == ( - "dcf3c6d2eade3449836c49a1dc4d3b8cd395aab9142db700c3c60598fa9c1c79" + "e59c019d3d454eac99ac0ac209b6c5b6faaf9bdfcaeee18c36a25be19bf7da2f" ) assert projected["overlap_ownership"]["sha256"] == ( "5f64f0aac49e2313177564f71876bffc8c81b3ded4df701e70930e60e9c98356" diff --git a/packages/microcosm-build/tests/test_spec_engine_inventory_coverage.py b/packages/microcosm-build/tests/test_spec_engine_inventory_coverage.py index 8f4e2673c..922d23379 100644 --- a/packages/microcosm-build/tests/test_spec_engine_inventory_coverage.py +++ b/packages/microcosm-build/tests/test_spec_engine_inventory_coverage.py @@ -79,7 +79,7 @@ "primary_targets": 65, "producer_authored_outputs": 92, "producer_compiled_outputs": 227, - "producer_inputs": 2_742, + "producer_inputs": 2_744, "producer_nodes": 38, "producer_virtual_resources": 75, "release_rungs": 5, diff --git a/packages/microcosm-build/tests/test_spec_engine_legacy_adapter.py b/packages/microcosm-build/tests/test_spec_engine_legacy_adapter.py index 53f05be22..9edc4e020 100644 --- a/packages/microcosm-build/tests/test_spec_engine_legacy_adapter.py +++ b/packages/microcosm-build/tests/test_spec_engine_legacy_adapter.py @@ -195,10 +195,10 @@ def test_adapter_preserves_generation_zero_identity_components( imputation = legacy_payload["imputation"] assert isinstance(imputation, dict) assert legacy_payload["stacked_authority_receipt"]["sha256"] == ( - "3a980927227704d0589f246eef9cd825c2ae84f3a4134ac835e0e5ed39a563ac" + "e660a8ce42b69a39d29c5f0ec37264bc69d61b03f27adc386336ec8889531bb2" ) assert imputation["late_producer_schedule_receipt"]["schedule_sha256"] == ( - "dcf3c6d2eade3449836c49a1dc4d3b8cd395aab9142db700c3c60598fa9c1c79" + "e59c019d3d454eac99ac0ac209b6c5b6faaf9bdfcaeee18c36a25be19bf7da2f" ) assert imputation["overlap_ownership"]["sha256"] == ( "5f64f0aac49e2313177564f71876bffc8c81b3ded4df701e70930e60e9c98356" diff --git a/packages/microcosm-build/tests/test_spec_engine_loader.py b/packages/microcosm-build/tests/test_spec_engine_loader.py index a52518ee6..d4d571fd6 100644 --- a/packages/microcosm-build/tests/test_spec_engine_loader.py +++ b/packages/microcosm-build/tests/test_spec_engine_loader.py @@ -235,7 +235,7 @@ def test_semantic_hash_has_golden_vector_and_surface_separation(tmp_path) -> Non # Pin the domain separator, normalization rules, schema-set receipt, and # exact normative projection as one reviewable golden vector. assert first.spec_sha256 == ( - "1ff676072985f104b9d80e3e5fa6e2078778969246246cf1371e982c772550cc" + "9f5b372796f2638378125d97ef5150be4c1f4cba9147b44973e4cd6a5f52f10a" ) second_root = _rich_minimal(tmp_path / "xy", note="second", store="local:b") diff --git a/packages/microcosm-build/tests/test_spec_engine_seeds.py b/packages/microcosm-build/tests/test_spec_engine_seeds.py index ec1a50ab1..c0e181a93 100644 --- a/packages/microcosm-build/tests/test_spec_engine_seeds.py +++ b/packages/microcosm-build/tests/test_spec_engine_seeds.py @@ -292,6 +292,16 @@ def test_exact_blake2b_salts_keys_candidates_and_absence_conditions() -> None: assert count.draw_condition == "only_when_declared_draw_column_is_absent" assert joint.draw_condition == "only_when_declared_draw_column_is_absent" + wic = LEGACY_V1_PROTOCOL.site("wic_claim_assignment") + assert wic.seed_material == ( + "build_model_seed", + "literal_salt=would_claim_wic", + "source:person_source_id_if_assembled_multispine", + "source_year:source_household_id:source_person_id_if_complete", + "else_support:person_support_source_id", + "else_person:person_id", + ) + vectors = ( (0, "snap_take_up", "2024:10:2", 9_193_979_365_434_741_258), (0, "immigration:ead_workers", "2024:2", 3_846_788_339_087_460_008), diff --git a/packages/microcosm-build/tests/test_spec_engine_stacked_authority_semantics.py b/packages/microcosm-build/tests/test_spec_engine_stacked_authority_semantics.py index c378f3dbd..1096b4092 100644 --- a/packages/microcosm-build/tests/test_spec_engine_stacked_authority_semantics.py +++ b/packages/microcosm-build/tests/test_spec_engine_stacked_authority_semantics.py @@ -73,7 +73,7 @@ def test_authority_projection_is_field_and_byte_identical_to_live_generation_zer assert projected == live assert stacked_identity_bytes(projected) == _canonical_bytes(live) assert projected["sha256"] == ( - "3a980927227704d0589f246eef9cd825c2ae84f3a4134ac835e0e5ed39a563ac" + "e660a8ce42b69a39d29c5f0ec37264bc69d61b03f27adc386336ec8889531bb2" ) assert { name: component["sha256"] for name, component in projected["components"].items() @@ -88,7 +88,7 @@ def test_authority_projection_is_field_and_byte_identical_to_live_generation_zer "cacc6c11e114dbae3aaa2761cc6b3fcb1191cd9b689b1c2bd096614c51ebff8b" ), "late_producer_schedule": ( - "9b15db577b85c796944e8eb267500d5d662f2a0eee77b25c1e4241c7d9620473" + "1b81157b0e21e4763884620ec27b5c4e6e36cc28237273c24eeccdef05a7fbca" ), "metric_registry": ( "d75cb9b29f8b0a9a085471a11f4c19c32ba04cbe5419053df94ea81cbe6125a9" @@ -191,7 +191,7 @@ def test_checkpoint_projection_is_field_and_byte_identical_to_live_oracle( ] assert ( projected["pool_code"]["late_producer_schedule"]["schedule_sha256"] - == "dcf3c6d2eade3449836c49a1dc4d3b8cd395aab9142db700c3c60598fa9c1c79" + == "e59c019d3d454eac99ac0ac209b6c5b6faaf9bdfcaeee18c36a25be19bf7da2f" ) diff --git a/packages/microcosm-build/tests/test_us_acs_multispine.py b/packages/microcosm-build/tests/test_us_acs_multispine.py index 98b21c6ab..afc38b72d 100644 --- a/packages/microcosm-build/tests/test_us_acs_multispine.py +++ b/packages/microcosm-build/tests/test_us_acs_multispine.py @@ -215,6 +215,7 @@ def fake_pool(actual_base, actual_acs, *, acs_share): "unmodeled_recipient_rows": 0, "derivation": None, "reconciliation": None, + "structural_receipt": None, } ], "deferred_inputs": ["congressional_district_geoid"], diff --git a/packages/microcosm-build/tests/test_us_acs_release_predictors.py b/packages/microcosm-build/tests/test_us_acs_release_predictors.py new file mode 100644 index 000000000..0c298aaea --- /dev/null +++ b/packages/microcosm-build/tests/test_us_acs_release_predictors.py @@ -0,0 +1,447 @@ +from __future__ import annotations + +import hashlib +import io +from pathlib import Path +from zipfile import ZIP_DEFLATED, ZipFile + +import numpy as np +import pandas as pd +import pytest +from pandas.testing import assert_frame_equal + +import microcosm.build.us_runtime.acs_release_predictors as module +from microcosm.build.us_runtime.acs_release_predictors import ( + ACS_OCCP_TO_POCCU2, + ACS_RELEASE_PREDICTOR_CROSSWALK_SHA256, + acs_release_predictor_crosswalk_payload, + join_acs_release_predictors, +) +from microcosm.build.us_runtime.puf_support import clone_us_frame_for_puf_support +from microcosm.build.us_runtime.spine_assembly import assemble_spines +from microcosm.frame import US_SCHEMA, Frame, WeightKind, Weights + +_CPS_PREDICTORS = ( + "PEDISDRS", + "PEDISEAR", + "PEDISEYE", + "PEDISOUT", + "PEDISPHY", + "PEDISREM", + "SSI_VAL", + "PRDTRACE", + "PRDTHSP", + "PEIOOCC", + "POCCU2", + "SPM_TENMORTSTATUS", +) + + +def _source_frame(*, acs: bool) -> Frame: + if acs: + household_ids = np.asarray([1, 2], dtype=np.int64) + person_ids = np.asarray([0, 1, 2], dtype=np.int64) + memberships = np.asarray([1, 1, 2], dtype=np.int64) + ages = np.asarray([4.0, 40.0, 30.0]) + else: + household_ids = np.asarray([1], dtype=np.int64) + person_ids = np.asarray([0], dtype=np.int64) + memberships = np.asarray([1], dtype=np.int64) + ages = np.asarray([35.0]) + + offsets = { + "tax_unit": 100, + "spm_unit": 200, + "family": 300, + "marital_unit": 400, + } + person = pd.DataFrame( + { + "person_id": person_ids, + "person_household_id": memberships, + "age": ages, + } + ) + for entity, offset in offsets.items(): + person[f"person_{entity}_id"] = memberships + offset + + household = pd.DataFrame({"household_id": household_ids}) + tables: dict[str, pd.DataFrame] = {"person": person, "household": household} + for entity, offset in offsets.items(): + tables[entity] = pd.DataFrame({f"{entity}_id": household_ids + offset}) + + if acs: + person["source_row_id"] = person_ids + person["source_year"] = 2024 + person["source_household_id"] = memberships + person["source_person_id"] = ["1", "2", "1"] + person["SPORDER"] = [1, 2, 1] + person["ssi_reported"] = [np.nan, 900.0, 0.0] + household["SERIALNO"] = ["2024HU0000001", "2024GQ0000002"] + household["TEN"] = [1.0, np.nan] + else: + for column in _CPS_PREDICTORS: + person[column] = 2.0 + person["SSI_VAL"] = 0.0 + person["PRDTRACE"] = 1.0 + person["PRDTHSP"] = 0.0 + person["PEIOOCC"] = 1005.0 + person["POCCU2"] = 8.0 + person["SPM_TENMORTSTATUS"] = 1.0 + + return Frame( + tables, + US_SCHEMA, + { + "household": Weights( + np.ones(len(household_ids), dtype=np.float64), + WeightKind.DESIGN, + ) + }, + pd.Series(["acs_2024_1yr" if acs else "asec_2024"] * len(person)), + ) + + +def _stacked_frame() -> Frame: + assembled = assemble_spines( + {"asec": _source_frame(acs=False), "acs": _source_frame(acs=True)}, + household_mass_shares={"asec": 0.5, "acs": 0.5}, + ) + return clone_us_frame_for_puf_support(assembled) + + +def _raw_person() -> pd.DataFrame: + return pd.DataFrame( + { + "SERIALNO": ["2024HU0000001", "2024HU0000001", "2024GQ0000002"], + "SPORDER": [1, 2, 1], + "AGEP": [4, 40, 30], + "DEAR": [2, 1, 2], + "DEYE": [2, 2, 2], + "DREM": [np.nan, 2, 2], + "DPHY": [np.nan, 2, 2], + "DDRS": [np.nan, 2, 2], + "DOUT": [np.nan, 2, 2], + "RAC1P": [1, 6, 2], + "HISP": [1, 1, 2], + "OCCP": [np.nan, 1005, 9800], + "ESR": [np.nan, 1, 4], + "SSIP": [np.nan, 900.0, 0.0], + "ADJINC": [1_000_000, 1_000_000, 1_000_000], + } + ) + + +def _raw_household() -> pd.DataFrame: + return pd.DataFrame( + { + "SERIALNO": ["2024HU0000001", "2024GQ0000002"], + "NP": [2, 1], + "TYPEHUGQ": [1, 2], + "TEN": [1.0, np.nan], + } + ) + + +def _write_zip(path: Path, members: dict[str, pd.DataFrame]) -> str: + with ZipFile(path, "w", compression=ZIP_DEFLATED) as archive: + for member, table in members.items(): + buffer = io.StringIO() + table.to_csv(buffer, index=False) + archive.writestr(member, buffer.getvalue()) + return hashlib.sha256(path.read_bytes()).hexdigest() + + +def _archives( + tmp_path: Path, + *, + person: pd.DataFrame | None = None, + household: pd.DataFrame | None = None, +) -> tuple[Path, str, Path, str]: + tmp_path.mkdir(parents=True, exist_ok=True) + raw_person = _raw_person() if person is None else person + raw_household = _raw_household() if household is None else household + person_path = tmp_path / "csv_pus.zip" + household_path = tmp_path / "csv_hus.zip" + person_sha = _write_zip( + person_path, + { + "psam_pusa.csv": raw_person.iloc[:2], + "psam_pusb.csv": raw_person.iloc[2:], + }, + ) + household_sha = _write_zip( + household_path, + { + "psam_husa.csv": raw_household.iloc[:1], + "psam_husb.csv": raw_household.iloc[1:], + }, + ) + return person_path, person_sha, household_path, household_sha + + +def _join( + frame: Frame, + archives: tuple[Path, str, Path, str], + monkeypatch: pytest.MonkeyPatch, +): + person_path, person_sha, household_path, household_sha = archives + monkeypatch.setattr(module, "ACS_2024_PERSON_ZIP_SHA256", person_sha) + monkeypatch.setattr(module, "ACS_2024_HOUSEHOLD_ZIP_SHA256", household_sha) + return join_acs_release_predictors( + frame, + person_zip=person_path, + person_sha256=person_sha, + household_zip=household_path, + household_sha256=household_sha, + chunksize=1, + ) + + +def test_crosswalk_digest_and_every_consumed_occupation_bin_are_pinned() -> None: + assert module._computed_crosswalk_sha256() == ( + ACS_RELEASE_PREDICTOR_CROSSWALK_SHA256 + ) + assert len(ACS_OCCP_TO_POCCU2) == 530 + assert set(ACS_OCCP_TO_POCCU2.values()) == set(range(1, 54)) + assert ACS_OCCP_TO_POCCU2[3250] == 26 + assert ACS_OCCP_TO_POCCU2[3255] == 25 + assert ACS_OCCP_TO_POCCU2[1005] == 8 + assert ACS_OCCP_TO_POCCU2[6005] == 41 + assert ACS_OCCP_TO_POCCU2[9800] == 52 + assert ACS_OCCP_TO_POCCU2[9920] == 53 + + payload = acs_release_predictor_crosswalk_payload() + assert payload["disability"]["DREM"]["minimum_question_age"] == 5 + assert payload["disability"]["DOUT"]["minimum_question_age"] == 15 + assert payload["disability"]["DOUT"]["codes"]["below_universe_blank"] == -1 + assert payload["race"]["RAC1P_to_consumed_PRDTRACE"] == { + "1": 1, + "2": 2, + "3": 3, + "4": 3, + "5": 3, + "6": 4, + "7": 3, + "8": 3, + "9": 3, + } + + +def test_release_join_is_exact_total_clone_stable_and_receipted( + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, +) -> None: + frame = _stacked_frame() + before = frame.table("person").copy(deep=True) + + result = _join(frame, _archives(tmp_path), monkeypatch) + person = result.frame.table("person") + asec = person["person_support_channel"].eq("asec") + acs = person["person_support_channel"].eq("acs") + + assert_frame_equal( + person.loc[asec, list(_CPS_PREDICTORS)], + before.loc[asec, list(_CPS_PREDICTORS)], + ) + assert person.loc[acs, list(module._OUTPUT_COLUMNS)].notna().all().all() + assert person.loc[acs, "SSI_VAL"].isna().all() + assert person.loc[acs, "ssi_reported"].isna().sum() == 2 + + by_source = person.loc[ + acs, + [ + "person_source_id", + "PEDISDRS", + "PRDTRACE", + "PRDTHSP", + "PEIOOCC", + "POCCU2", + "SPM_TENMORTSTATUS", + ], + ].drop_duplicates("person_source_id") + assert sorted(by_source["PEDISDRS"].tolist()) == [-1.0, 2.0, 2.0] + assert sorted(by_source["PRDTRACE"].tolist()) == [1.0, 2.0, 4.0] + assert sorted(by_source["PRDTHSP"].tolist()) == [0.0, 0.0, 1.0] + assert sorted(by_source["PEIOOCC"].tolist()) == [-1.0, 1005.0, 9800.0] + assert sorted(by_source["POCCU2"].tolist()) == [0.0, 8.0, 52.0] + assert sorted(by_source["SPM_TENMORTSTATUS"].tolist()) == [1.0, 1.0, 3.0] + + for _, clones in person.loc[acs].groupby("person_source_id"): + for column in module._OUTPUT_COLUMNS: + assert clones[column].nunique(dropna=False) == 1 + + receipt = result.receipt + assert receipt["crosswalk"]["sha256"] == ACS_RELEASE_PREDICTOR_CROSSWALK_SHA256 + assert receipt["join"] == { + "semantic_key": ["household.SERIALNO", "person.SPORDER"], + "clone_fanout_key": "person_source_id", + "acs_source_people": 3, + "acs_support_rows": 6, + "acs_support_rows_by_clone_index": {"0": 3, "1": 3}, + "selected_raw_person_rows": 3, + "selected_raw_household_rows": 2, + "unmatched_pool_source_people": 0, + "source_identity_collisions": 0, + "semantic_key_sha256": receipt["join"]["semantic_key_sha256"], + } + race_counts = receipt["models"]["scf_wealth"]["predictors"]["PRDTRACE"] + assert race_counts == {"asec_native": 2, "acs_joined": 6, "still_null": 0} + reporter = receipt["models"]["ssi_disability_criteria"]["predictors"][ + "reported_ssi_anchor" + ] + assert reporter["asec_native"] == 2 + assert reporter["acs_joined"] == 4 + assert reporter["still_null"] == 2 + + repeated = _join(result.frame, _archives(tmp_path / "again"), monkeypatch) + assert_frame_equal(repeated.frame.table("person"), person) + + +def test_occupation_crosswalk_preserves_age_15_source_universe_gap() -> None: + rows = pd.concat([_raw_person().iloc[[1]]] * 2, ignore_index=True) + rows["SERIALNO"] = ["age15", "age16"] + rows["SPORDER"] = [1, 1] + rows["AGEP"] = [15, 16] + rows["OCCP"] = np.nan + rows["ESR"] = [np.nan, 6] + rows["person_source_id"] = [1, 2] + + mapped = module._crosswalk_people(rows) + + assert mapped["PEIOOCC"].tolist() == [-1, -1] + assert mapped["POCCU2"].tolist() == [0, 53] + + +def test_occupation_crosswalk_refuses_malformed_esr_universe() -> None: + row = _raw_person().iloc[[1]].copy() + row["ESR"] = "not-a-code" + row["person_source_id"] = 1 + + with pytest.raises(ValueError, match="ESR code/universe mismatch"): + module._crosswalk_people(row) + + +@pytest.mark.parametrize("bad_ssi", [np.nan, -1.0, np.inf, "not-a-number"]) +def test_release_join_refuses_malformed_native_asec_ssi( + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, + bad_ssi: object, +) -> None: + frame = _stacked_frame() + person = frame.table("person") + asec = person["person_support_channel"].eq("asec") + if isinstance(bad_ssi, str): + person["SSI_VAL"] = person["SSI_VAL"].astype(object) + person.loc[asec, "SSI_VAL"] = bad_ssi + + with pytest.raises(ValueError, match="native ASEC inputs"): + _join(frame, _archives(tmp_path), monkeypatch) + + +def test_release_join_attests_native_ssi_against_raw_archive( + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, +) -> None: + frame = _stacked_frame() + person = frame.table("person") + acs_native = person["person_support_channel"].eq("acs") & person[ + "person_support_clone_index" + ].eq(0) + adult = acs_native & person["age"].eq(40) + person.loc[adult, "ssi_reported"] = 901.0 + + with pytest.raises(ValueError, match="ssi_reported disagrees"): + _join(frame, _archives(tmp_path), monkeypatch) + + +def test_release_join_accepts_h5_object_wrapped_native_numbers( + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, +) -> None: + frame = _stacked_frame() + person = frame.table("person") + for column in (*module._OUTPUT_COLUMNS, "SSI_VAL"): + person[column] = person[column].astype(object) + + result = _join(frame, _archives(tmp_path), monkeypatch) + + assert result.receipt["enabled"] is True + + +def test_release_join_refuses_a_missing_raw_person( + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, +) -> None: + raw = _raw_person().iloc[:-1].copy() + with pytest.raises(ValueError, match="not total over pool source people"): + _join( + _stacked_frame(), + _archives(tmp_path, person=raw), + monkeypatch, + ) + + +def test_release_join_refuses_raw_person_key_collision( + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, +) -> None: + raw = pd.concat([_raw_person(), _raw_person().iloc[[0]]], ignore_index=True) + with pytest.raises(ValueError, match="raw person key collision"): + _join( + _stacked_frame(), + _archives(tmp_path, person=raw), + monkeypatch, + ) + + +def test_release_join_refuses_source_identity_collision( + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, +) -> None: + frame = _stacked_frame() + person = frame.table("person") + acs = person["person_support_channel"].eq("acs") + source_id = person.loc[acs, "person_source_id"].iloc[0] + clone = ( + acs + & person["person_source_id"].eq(source_id) + & person["person_support_clone_index"].eq(1) + ) + person.loc[clone, "SPORDER"] = 9 + + with pytest.raises(ValueError, match="source_person_id does not equal"): + _join(frame, _archives(tmp_path), monkeypatch) + + +def test_release_join_verifies_pin_before_opening_archive( + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, +) -> None: + person_path, person_sha, household_path, household_sha = _archives(tmp_path) + monkeypatch.setattr(module, "ACS_2024_PERSON_ZIP_SHA256", "0" * 64) + monkeypatch.setattr(module, "ACS_2024_HOUSEHOLD_ZIP_SHA256", household_sha) + with pytest.raises(ValueError, match="pin must be the reviewed"): + join_acs_release_predictors( + _stacked_frame(), + person_zip=person_path, + person_sha256=person_sha, + household_zip=household_path, + household_sha256=household_sha, + ) + + +def test_no_acs_frame_is_an_identity_without_archive_options() -> None: + frame = _source_frame(acs=False) + result = join_acs_release_predictors( + frame, + person_zip=None, + person_sha256=None, + household_zip=None, + household_sha256=None, + ) + assert result.frame is frame + assert result.receipt == { + "enabled": False, + "reason": "no physical ACS source rows", + } diff --git a/packages/microcosm-build/tests/test_us_acs_transfer.py b/packages/microcosm-build/tests/test_us_acs_transfer.py index a436f50a8..e9e615bec 100644 --- a/packages/microcosm-build/tests/test_us_acs_transfer.py +++ b/packages/microcosm-build/tests/test_us_acs_transfer.py @@ -1388,6 +1388,360 @@ def test_discrete_year_predictions_snap_to_observed_donor_support( assert pd.api.types.is_integer_dtype(values.dtype) +def test_integer_supported_weeks_predictions_snap_to_observed_donor_support( + monkeypatch: pytest.MonkeyPatch, +) -> None: + donor_support = {0.0, 1.0, 3.0, 8.0, 13.0} + donor = _with_columns( + _donor_frame(), + "person", + { + "weeks_unemployed": [ + 0.0, + 0.0, + 0.0, + 1.0, + 3.0, + 3.0, + 8.0, + 13.0, + ] + }, + ) + monkeypatch.setattr(acs_transfer_module, "QRF", _MeanQRF) + _MeanQRF.calls = [] + + result = transfer_acs_inputs( + _recipient_frame(), + donor, + target_families={ + "person": {"weeks": ("weeks_unemployed",)}, + }, + seed=2, + n_estimators=1, + ) + + values = result.frame.table("person")["weeks_unemployed"] + assert set(values).issubset(donor_support) + assert pd.api.types.is_integer_dtype(values.dtype) + contract = acs_transfer_module.acs_transfer_execution_contract_identity( + targets=("weeks_unemployed",), + ) + assert "weeks_unemployed" in contract["discrete_numeric_targets"] + + +def _pregnancy_donor() -> Frame: + return _with_columns( + _with_columns( + _donor_frame(), + "person", + { + "age": [30.0] * 8, + "is_female": [True] * 8, + }, + ), + "person", + {"is_pregnant": [True] * 6 + [False] * 2}, + ) + + +def test_pregnancy_draws_once_per_eligible_source_person_and_fans_to_clones( + monkeypatch: pytest.MonkeyPatch, +) -> None: + base = _with_full_us_schema(_recipient_frame()) + assembled = assemble_spines( + {"asec": base, "acs": base}, + household_mass_shares={"asec": 0.5, "acs": 0.5}, + ) + recipient = clone_us_frame_for_puf_support(assembled) + monkeypatch.setattr(acs_transfer_module, "QRF", _MeanQRF) + _MeanQRF.calls = [] + + result = transfer_acs_inputs( + recipient, + _with_full_us_schema(_pregnancy_donor()), + target_families={"person": {"pregnancy": ("is_pregnant",)}}, + donor_channel=None, + seed=9, + n_estimators=1, + ) + + person = result.frame.person + eligible = ( + person["is_female"].astype(bool) + & person["age"].between(15, 44, inclusive="both") + ) + assert person.loc[eligible, "is_pregnant"].all() + assert not person.loc[~eligible, "is_pregnant"].any() + assert ( + person.groupby("person_source_id", sort=False)["is_pregnant"] + .nunique() + .eq(1) + .all() + ) + expected_qrf_sources = int( + pd.DataFrame( + { + "source": person["person_source_id"], + "eligible": eligible, + } + ) + .groupby("source", sort=False)["eligible"] + .first() + .sum() + ) + record = next(item for item in result.imputed_inputs if item.column == "is_pregnant") + receipt = record.structural_receipt + assert receipt is not None + assert sum(pattern.recipient_rows for pattern in record.patterns) == ( + expected_qrf_sources + ) + assert receipt["qrf_draw_source_persons"] == expected_qrf_sources + assert receipt["qrf_draw_rows"] == expected_qrf_sources + assert receipt["qrf_fanout_rows"] == expected_qrf_sources + assert receipt["final_domain_violation_rows"] == 0 + assert receipt["final_clone_disagreement_source_persons"] == 0 + + +def test_pregnancy_all_ineligible_recipients_bypass_qrf( + monkeypatch: pytest.MonkeyPatch, +) -> None: + recipient = _with_columns( + _recipient_frame(), + "person", + { + "age": [30.0] * 6, + "is_female": [False] * 6, + }, + ) + monkeypatch.setattr(acs_transfer_module, "QRF", _MeanQRF) + _MeanQRF.calls = [] + + result = transfer_acs_inputs( + recipient, + _pregnancy_donor(), + target_families={"person": {"pregnancy": ("is_pregnant",)}}, + donor_channel=None, + n_estimators=1, + ) + + assert not result.frame.person["is_pregnant"].any() + assert not _MeanQRF.calls + record = result.imputed_inputs[0] + assert record.weight_kind == "structural" + receipt = record.structural_receipt + assert receipt is not None + assert receipt["qrf_draw_source_persons"] == 0 + assert receipt["qrf_draw_rows"] == 0 + assert receipt["ineligible_rows_assigned_false"] == 6 + + +def test_pregnancy_partial_clone_fanout_receipt_categories_are_disjoint( + monkeypatch: pytest.MonkeyPatch, +) -> None: + base = _with_full_us_schema(_recipient_frame()) + assembled = assemble_spines( + {"asec": base, "acs": base}, + household_mass_shares={"asec": 0.5, "acs": 0.5}, + ) + cloned = clone_us_frame_for_puf_support(assembled) + person = cloned.person + clone_zero = person["person_support_clone_index"].eq(0).to_numpy() + pregnancy = pd.array([pd.NA] * len(person), dtype="boolean") + pregnancy[clone_zero] = False + recipient = _with_columns( + cloned, + "person", + {"is_pregnant": pregnancy}, + ) + monkeypatch.setattr(acs_transfer_module, "QRF", _MeanQRF) + _MeanQRF.calls = [] + + result = transfer_acs_inputs( + recipient, + _with_full_us_schema(_pregnancy_donor()), + target_families={"person": {"pregnancy": ("is_pregnant",)}}, + donor_channel=None, + n_estimators=1, + ) + + assert not _MeanQRF.calls + output = result.frame.person + assert not output["is_pregnant"].any() + missing = ~clone_zero + eligible = ( + output["is_female"].astype(bool).to_numpy() + & output["age"].between(15, 44, inclusive="both").to_numpy() + ) + record = result.imputed_inputs[0] + receipt = record.structural_receipt + assert receipt is not None + assert receipt["preexisting_value_fanout_rows"] == int( + (missing & eligible).sum() + ) + assert receipt["ineligible_rows_assigned_false"] == int( + (missing & ~eligible).sum() + ) + assert ( + receipt["preexisting_value_fanout_rows"] + + receipt["ineligible_rows_assigned_false"] + == record.imputed_recipient_rows + ) + + +def test_pregnancy_invalid_donor_is_refused_before_qrf( + monkeypatch: pytest.MonkeyPatch, +) -> None: + donor = _with_columns( + _donor_frame(), + "person", + {"is_pregnant": [True] + [False] * 7}, + ) + monkeypatch.setattr(acs_transfer_module, "QRF", _MeanQRF) + _MeanQRF.calls = [] + + with pytest.raises( + ValueError, + match=r"1 preexisting donor domain violation", + ): + transfer_acs_inputs( + _recipient_frame(), + donor, + target_families={"person": {"pregnancy": ("is_pregnant",)}}, + donor_channel=None, + n_estimators=1, + ) + assert not _MeanQRF.calls + + +def test_pregnancy_near_boolean_recipient_is_refused_before_qrf( + monkeypatch: pytest.MonkeyPatch, +) -> None: + recipient = _with_columns( + _recipient_frame(), + "person", + {"is_pregnant": [1.0 + 1e-12] + [False] * 5}, + ) + monkeypatch.setattr(acs_transfer_module, "QRF", _MeanQRF) + _MeanQRF.calls = [] + + with pytest.raises(ValueError, match=r"1 non-boolean row"): + transfer_acs_inputs( + recipient, + _pregnancy_donor(), + target_families={"person": {"pregnancy": ("is_pregnant",)}}, + donor_channel=None, + n_estimators=1, + ) + assert not _MeanQRF.calls + + +def test_pregnancy_complete_invalid_recipient_is_still_refused( + monkeypatch: pytest.MonkeyPatch, +) -> None: + recipient = _with_columns( + _recipient_frame(), + "person", + {"is_pregnant": [True] + [False] * 5}, + ) + monkeypatch.setattr(acs_transfer_module, "QRF", _MeanQRF) + _MeanQRF.calls = [] + + with pytest.raises( + ValueError, + match=r"1 preexisting recipient domain violation", + ): + transfer_acs_inputs( + recipient, + _pregnancy_donor(), + target_families={"person": {"pregnancy": ("is_pregnant",)}}, + donor_channel=None, + n_estimators=1, + ) + assert not _MeanQRF.calls + + +def test_complete_invalid_pregnancy_is_refused_before_other_active_target( + monkeypatch: pytest.MonkeyPatch, +) -> None: + recipient = _with_columns( + _recipient_frame(), + "person", + {"is_pregnant": [True] + [False] * 5}, + ) + monkeypatch.setattr(acs_transfer_module, "QRF", _MeanQRF) + _MeanQRF.calls = [] + + with pytest.raises( + ValueError, + match=r"1 preexisting recipient domain violation", + ): + transfer_acs_inputs( + recipient, + _pregnancy_donor(), + target_families={ + "person": { + "pregnancy": ("is_pregnant",), + "income": ("qualified_dividend_income",), + } + }, + donor_channel=None, + n_estimators=1, + ) + assert not _MeanQRF.calls + + +def test_pregnancy_complete_valid_recipient_retains_structural_receipt( + monkeypatch: pytest.MonkeyPatch, +) -> None: + recipient = _with_columns( + _recipient_frame(), + "person", + {"is_pregnant": [False] * 6}, + ) + monkeypatch.setattr(acs_transfer_module, "QRF", _MeanQRF) + _MeanQRF.calls = [] + + result = transfer_acs_inputs( + recipient, + _pregnancy_donor(), + target_families={"person": {"pregnancy": ("is_pregnant",)}}, + donor_channel=None, + n_estimators=1, + ) + + assert not _MeanQRF.calls + assert len(result.imputed_inputs) == 1 + record = result.imputed_inputs[0] + assert record.imputed_recipient_rows == 0 + assert record.unmodeled_recipient_rows == 0 + assert record.structural_receipt is not None + assert record.structural_receipt["status"] == "verified" + + +def test_pregnancy_policy_is_bound_into_transfer_execution_identity() -> None: + enabled = acs_transfer_module.acs_transfer_execution_contract_identity( + targets=("is_pregnant",), + derive_schedule_d=False, + ) + disabled = acs_transfer_module.acs_transfer_execution_contract_identity( + targets=(), + derive_schedule_d=False, + ) + + policy = enabled["structural_target_policies"]["is_pregnant"] + assert policy["enabled"] is True + assert policy["eligibility"] == { + "is_female": True, + "minimum_age_inclusive": 15, + "maximum_age_inclusive": 44, + } + assert policy["qrf_scope"] == "one_eligible_representative_per_source_person" + assert policy["preexisting_domain_violations"] == "refuse" + assert disabled["structural_target_policies"]["is_pregnant"]["enabled"] is False + assert enabled["sha256"] != disabled["sha256"] + + def test_engine_boolean_metadata_restores_primary_qrf_float_h5_donor( monkeypatch: pytest.MonkeyPatch, tmp_path: Path, diff --git a/packages/microcosm-build/tests/test_us_alimony.py b/packages/microcosm-build/tests/test_us_alimony.py index d52ab7be9..25aefd15c 100644 --- a/packages/microcosm-build/tests/test_us_alimony.py +++ b/packages/microcosm-build/tests/test_us_alimony.py @@ -374,6 +374,96 @@ def test_signal_gate_rejects_discarded_asec_strike_benefits() -> None: assert any("does not conserve OI_VAL" in failure for failure in result.failures) +def _stacked_alimony_person() -> pd.DataFrame: + source_count = 500 + source_numbers = np.repeat(np.arange(source_count), 2) + clone_indices = np.tile([0, 1], source_count) + source_channels = np.where(source_numbers < 250, "asec", "acs") + asec_source = source_channels == "asec" + + codes = np.full(len(source_numbers), np.nan) + amounts = np.full(len(source_numbers), np.nan) + codes[asec_source] = 0.0 + amounts[asec_source] = 0.0 + reported_alimony = source_numbers == 10 + reported_strike = source_numbers == 20 + codes[reported_alimony] = 20.0 + amounts[reported_alimony] = 2_000.0 + codes[reported_strike] = 12.0 + amounts[reported_strike] = 700.0 + + alimony_income = np.where(reported_alimony, amounts, 0.0) + alimony_expense = np.where(source_numbers == 300, 3_000.0, 0.0) + strike_benefits = np.where(reported_strike, amounts, 0.0) + return pd.DataFrame( + { + "person_spine_source_id": source_numbers, + "person_support_channel": source_channels, + "person_support_clone_index": clone_indices, + "OI_OFF": codes, + "OI_VAL": amounts, + "alimony_income": alimony_income, + "alimony_expense": alimony_expense, + "strike_benefits": strike_benefits, + "miscellaneous_income": np.zeros(len(source_numbers)), + } + ) + + +def test_signal_gate_reconciles_physical_asec_rows_in_stacked_pool() -> None: + person = _stacked_alimony_person() + + result = us_alimony_signal_gate(_PersonFrame(person)) # type: ignore[arg-type] + + assert result.passed, result.failures + + asec_clone = ( + person["person_support_channel"].eq("asec") + & person["person_support_clone_index"].eq(1) + ) + person.loc[asec_clone.idxmax(), "miscellaneous_income"] = 1.0 + transferred_clone = us_alimony_signal_gate( # type: ignore[arg-type] + _PersonFrame(person) + ) + assert transferred_clone.passed, transferred_clone.failures + + asec_native = ( + person["person_support_channel"].eq("asec") + & person["person_support_clone_index"].eq(0) + ) + person.loc[asec_native.idxmax(), "miscellaneous_income"] = 1.0 + mismatch = us_alimony_signal_gate(_PersonFrame(person)) # type: ignore[arg-type] + assert not mismatch.passed + assert any("does not conserve OI_VAL" in failure for failure in mismatch.failures) + + person.loc[asec_native.idxmax(), "miscellaneous_income"] = 0.0 + person.loc[asec_clone.idxmax(), "OI_VAL"] = np.nan + invalid_source = us_alimony_signal_gate( # type: ignore[arg-type] + _PersonFrame(person) + ) + assert not invalid_source.passed + assert invalid_source.details["asec_source_invalid"] == 1 + + +def test_signal_gate_preserves_legacy_asec_puf_source_scope() -> None: + person = _stacked_alimony_person().drop(columns=["person_spine_source_id"]) + person["person_support_channel"] = np.where( + person["person_support_clone_index"].eq(0), + "asec", + "puf_tax_detail", + ) + asec = person["person_support_channel"].eq("asec") + person.loc[asec, ["OI_OFF", "OI_VAL"]] = person.loc[ + asec, ["OI_OFF", "OI_VAL"] + ].fillna(0.0) + puf = person["person_support_channel"].eq("puf_tax_detail") + person.loc[puf, ["OI_OFF", "OI_VAL"]] = np.nan + + result = us_alimony_signal_gate(_PersonFrame(person)) # type: ignore[arg-type] + + assert result.passed, result.failures + + @pytest.mark.parametrize( "person", [ diff --git a/packages/microcosm-build/tests/test_us_fiscal_refresh_builder.py b/packages/microcosm-build/tests/test_us_fiscal_refresh_builder.py index 3c76b4620..0794037d1 100644 --- a/packages/microcosm-build/tests/test_us_fiscal_refresh_builder.py +++ b/packages/microcosm-build/tests/test_us_fiscal_refresh_builder.py @@ -2006,6 +2006,96 @@ def test_sipp_vehicle_donor_override_parses(monkeypatch) -> None: assert args.sipp_vehicle_donor == Path("pu2023.csv") +def test_acs_release_archive_options_parse_as_one_pinned_pair() -> None: + builder = _load_builder_module() + args = builder._parse_args( + [ + "--ledger-facts", + "facts.jsonl", + "--out", + "release", + "--acs-person-zip", + "csv_pus.zip", + "--acs-person-sha256", + "a" * 64, + "--acs-household-zip", + "csv_hus.zip", + "--acs-household-sha256", + "b" * 64, + ] + ) + + assert args.acs_person_zip == Path("csv_pus.zip") + assert args.acs_person_sha256 == "a" * 64 + assert args.acs_household_zip == Path("csv_hus.zip") + assert args.acs_household_sha256 == "b" * 64 + + +@pytest.mark.parametrize( + "extra", + [ + ["--acs-person-zip", "csv_pus.zip"], + [ + "--acs-person-zip", + "csv_pus.zip", + "--acs-person-sha256", + "A" * 64, + "--acs-household-zip", + "csv_hus.zip", + "--acs-household-sha256", + "b" * 64, + ], + ], +) +def test_acs_release_archive_options_fail_closed(extra: list[str]) -> None: + builder = _load_builder_module() + with pytest.raises(SystemExit): + builder._parse_args( + [ + "--ledger-facts", + "facts.jsonl", + "--out", + "release", + *extra, + ] + ) + + +def test_acs_predictor_join_precedes_all_six_archived_model_stages() -> None: + import ast + import inspect + + builder = _load_builder_module() + source = inspect.getsource(builder._main) + join_position = source.index("join_acs_release_predictors(") + + for stage in ( + "with_us_scf_wealth_inputs(", + "with_us_ssi_disability_criteria(", + "with_us_scf_auto_loan_inputs(", + "with_us_sipp_vehicle_inputs(", + "with_us_sipp_tip_inputs(", + "with_us_org_wages_inputs(", + ): + assert join_position < source.index(stage) + + manifest_calls = [ + node + for node in ast.walk(ast.parse(source)) + if isinstance(node, ast.Call) + and isinstance(node.func, ast.Name) + and node.func.id == "_build_manifests" + ] + assert len(manifest_calls) == 1 + receipt_keyword = next( + keyword + for keyword in manifest_calls[0].keywords + if keyword.arg == "acs_predictor_join" + ) + assert isinstance(receipt_keyword.value, ast.Name) + assert receipt_keyword.value.id == "acs_predictor_join_receipt" + + def test_scf_full_extract_override_parses(monkeypatch) -> None: builder = _load_builder_module() monkeypatch.setattr( @@ -4151,6 +4241,8 @@ def test_main_writes_diagnostics_before_post_calibration_gate_failure( base_h5 = tmp_path / "base.h5" pool_manifest = tmp_path / "pool.manifest.json" weeks_source = tmp_path / "asecpub23csv.zip" + acs_person_zip = tmp_path / "csv_pus.zip" + acs_household_zip = tmp_path / "csv_hus.zip" facts = tmp_path / "facts.jsonl" out = tmp_path / "out" base_h5.write_bytes(b"h5") @@ -4286,6 +4378,16 @@ def table(self, entity): ] if terminal_mode not in {"telemetry", "puf_tail"}: argv.append("--no-staging") + argv += [ + "--acs-person-zip", + str(acs_person_zip), + "--acs-person-sha256", + "d" * 64, + "--acs-household-zip", + str(acs_household_zip), + "--acs-household-sha256", + "f" * 64, + ] if terminal_mode == "crash": # Nonexistent incumbent: the degraded-mode guard must record the # load failure, null the path for the writer (no re-hash replay of @@ -5032,6 +5134,20 @@ def fake_retirement_distributions_signal_gate(frame): details={"checked": True}, ), ) + + def fake_join_acs_release_predictors(frame, **kwargs): + captured["source_stage_events"].append("acs_predictor_join") + captured["acs_predictor_join_kwargs"] = kwargs + return SimpleNamespace( + frame=frame, + receipt={"enabled": True, "join": {"acs_source_people": 3}}, + ) + + monkeypatch.setattr( + builder, + "join_acs_release_predictors", + fake_join_acs_release_predictors, + ) monkeypatch.setattr( builder, "fetch_scf_2022_summary_extract", @@ -5068,6 +5184,7 @@ def fake_load_sipp_financial_asset_donor( def fake_with_scf_wealth_inputs( frame, *, seed, time_period, scf_donor, sipp_donor=None ): + captured["source_stage_events"].append("scf_wealth") captured["sipp_scf_wealth_blend_called"] = sipp_donor is not None return frame @@ -6121,6 +6238,15 @@ def fake_release_gate_failures(*args, **kwargs): assert captured["weeks_unemployed_stage_period"] == builder.PERIOD assert isinstance(captured["weeks_unemployed_stage_source"], pd.DataFrame) assert captured["weeks_unemployed_gate_called"] is True + assert captured["acs_predictor_join_kwargs"] == { + "person_zip": acs_person_zip, + "person_sha256": "d" * 64, + "household_zip": acs_household_zip, + "household_sha256": "f" * 64, + } + assert captured["source_stage_events"].index("acs_predictor_join") < captured[ + "source_stage_events" + ].index("scf_wealth") assert captured["source_stage_events"].index("weeks_stage") < captured[ "source_stage_events" ].index("ssi_reporters") @@ -8836,10 +8962,22 @@ def __len__(self): "n_exported_households": 57_240, "l0_lambda_share": 0.8, }, + acs_predictor_join={ + "enabled": True, + "crosswalk": {"sha256": "c" * 64}, + "join": {"acs_source_people": 856_626}, + }, ) manifest = json.loads((release_dir / "release_manifest.json").read_text()) build_manifest = json.loads((release_dir / "build_manifest.json").read_text()) + expected_acs_join = { + "enabled": True, + "crosswalk": {"sha256": "c" * 64}, + "join": {"acs_source_people": 856_626}, + } + assert build_manifest["acs_predictor_join"] == expected_acs_join + assert manifest["build"]["acs_predictor_join"] == expected_acs_join assert build_manifest["gates"]["target_profile_coverage"]["passed"] assert ( build_manifest["gates"]["target_profile_coverage"]["details"][ diff --git a/packages/microcosm-build/tests/test_us_medicare_take_up.py b/packages/microcosm-build/tests/test_us_medicare_take_up.py index 81182315b..0ab8d2202 100644 --- a/packages/microcosm-build/tests/test_us_medicare_take_up.py +++ b/packages/microcosm-build/tests/test_us_medicare_take_up.py @@ -109,6 +109,21 @@ def _operation() -> SourceOperationSpec: ) +def _stacked_frame() -> Frame: + derived = with_us_medicare_take_up_input(_frame(), seed=0, time_period=2024) + stacked = clone_us_frame_for_puf_support(derived) + person = stacked.table("person") + source_record = np.tile(np.arange(5, dtype=np.int64), 2) + person["person_spine_source_id"] = source_record + person["person_support_channel"] = np.where( + source_record < 3, + "asec", + "acs", + ) + person.loc[person["person_support_channel"].eq("acs"), _SOURCE] = np.nan + return stacked + + def _derive(person: pd.DataFrame) -> pd.DataFrame: return derive_us_medicare_take_up_from_manifest(person, _operation(), None) @@ -261,6 +276,26 @@ def test_support_cloning_preserves_measured_values_on_both_channels(self) -> Non } assert us_medicare_take_up_signal_gate(cloned).passed + def test_stacked_gate_reconciles_only_physical_asec_source_rows(self) -> None: + stacked = _stacked_frame() + + gate = us_medicare_take_up_signal_gate(stacked) + + assert gate.passed, gate.failures + assert gate.details["source_rows"] == 6 + assert gate.details["source_mismatch_count"] == 0 + assert gate.details["channel_weighted_enrolled_shares"] == { + "asec": pytest.approx(0.2), + PUF_TAX_DETAIL_SUPPORT_CHANNEL: pytest.approx(0.2), + } + + person = stacked.table("person") + asec = person["person_support_channel"].eq("asec") + person.loc[person.index[asec][0], _SOURCE] = np.nan + failed = us_medicare_take_up_signal_gate(stacked) + assert not failed.passed + assert any("MCARE" in failure for failure in failed.failures) + def test_gate_rejects_missing_constant_bad_share_and_mismatch(self) -> None: missing = _frame() assert not us_medicare_take_up_signal_gate(missing).passed diff --git a/packages/microcosm-build/tests/test_us_multispine_pool_h5_io.py b/packages/microcosm-build/tests/test_us_multispine_pool_h5_io.py index 5544e92a5..4bdf237e7 100644 --- a/packages/microcosm-build/tests/test_us_multispine_pool_h5_io.py +++ b/packages/microcosm-build/tests/test_us_multispine_pool_h5_io.py @@ -8,6 +8,7 @@ import pandas as pd import pytest +import microcosm.build.us_runtime.acs_transfer as acs_transfer_module import microcosm.build.us_runtime.h5_io as h5_io import microcosm.build.us_runtime.post_transfer_calibration as post_transfer_calibration_runtime import microcosm.build.us_runtime.stacked_spine as stacked_spine_module @@ -54,6 +55,17 @@ def _sha256(path: Path) -> str: return hashlib.sha256(path.read_bytes()).hexdigest() +def _json_sha256(value: object) -> str: + payload = json.dumps( + value, + allow_nan=False, + ensure_ascii=False, + separators=(",", ":"), + sort_keys=True, + ).encode("utf-8") + return hashlib.sha256(payload).hexdigest() + + def _pool_frame() -> Frame: ids = np.asarray([10, 20, 30], dtype=np.int64) person = pd.DataFrame( @@ -580,7 +592,46 @@ def _fixture_geography_assignment( } -def _write_ready_pool(tmp_path: Path, *, stacked: bool = False) -> Path: +def _fixture_stacked_sampling( + sample_fraction: float, +) -> tuple[dict[str, object], dict[str, object]]: + fraction_token = { + 0.01: "f001", + 0.04: "f004", + 0.10: "f010", + 0.25: "f025", + 1.00: "f100", + }[sample_fraction] + realized = {"asec": 2, "acs": 1} + stack_manifest: dict[str, object] = { + "version": 4, + "sample_fraction": sample_fraction, + "sample_seed": 578, + "survey_samples": { + channel: { + "fraction": sample_fraction, + "seed": 578, + "realized_household_count": count, + } + for channel, count in realized.items() + }, + } + sampling = { + "sample_fraction": sample_fraction, + "fraction_token": fraction_token, + "sample_seed": 578, + "realized_households": realized, + "stack_manifest_sha256": _json_sha256(stack_manifest), + } + return sampling, stack_manifest + + +def _write_ready_pool( + tmp_path: Path, + *, + stacked: bool = False, + sample_fraction: float = 1.0, +) -> Path: run_id = "fixture-publication" pool_path = tmp_path / "pool.h5" diagnostics_path = tmp_path / "pool.agreement.json" @@ -691,6 +742,7 @@ def _write_ready_pool(tmp_path: Path, *, stacked: bool = False) -> Path: if stacked: dag = _canonical_stacked_late_dag_receipt() assert geography_assignment is not None + sampling, stack_manifest = _fixture_stacked_sampling(sample_fraction) transition_authority = ( stacked_spine_module._late_producer_transition_authority_receipt(dag) ) @@ -698,6 +750,8 @@ def _write_ready_pool(tmp_path: Path, *, stacked: bool = False) -> Path: { "pipeline": "us-stacked-pool", "random_seed": 0, + "sampling": sampling, + "stack_manifest": stack_manifest, "geography_assignment": geography_assignment, "provenance_pins": { role: { @@ -831,6 +885,35 @@ def _canonical_late_calibration_owner_receipt( return owner +def _canonical_pregnancy_structural_receipt() -> dict[str, object]: + policy = acs_transfer_module.acs_transfer_execution_contract_identity( + targets=("is_pregnant",), + derive_schedule_d=False, + )["structural_target_policies"]["is_pregnant"] + return { + "policy_sha256": policy["sha256"], + "source_person_key": "person_source_id", + "source_persons_checked": 1, + "physical_rows_checked": 1, + "clone_rows_checked": 0, + "donor_rows_checked": 1, + "qrf_draw_source_persons": 0, + "qrf_draw_rows": 0, + "qrf_fanout_rows": 0, + "preexisting_value_fanout_rows": 0, + "ineligible_rows_assigned_false": 0, + "donor_preexisting_domain_violation_rows": 0, + "recipient_preexisting_domain_violation_rows": 0, + "preexisting_clone_disagreement_source_persons": 0, + "inconsistent_eligibility_source_persons": 0, + "maximum_clones_per_source_person": 1, + "final_incomplete_rows": 0, + "final_domain_violation_rows": 0, + "final_clone_disagreement_source_persons": 0, + "status": "verified", + } + + def _canonical_stacked_late_dag_receipt() -> dict[str, object]: """Build a signed fixture receipt over the live canonical contracts.""" @@ -892,6 +975,11 @@ def _canonical_stacked_late_dag_receipt() -> dict[str, object]: } for target in group.targets } + pregnancy_key = f"{group.entity}/{group.family}/is_pregnant" + if pregnancy_key in group_targets: + group_targets[pregnancy_key]["structural_policy"] = ( + _canonical_pregnancy_structural_receipt() + ) calibrated_keys = sorted(set(group_targets) & set(late_specs)) for key in calibrated_keys: group_targets[key]["post_transfer_calibration"] = ( @@ -1358,6 +1446,133 @@ def test_ready_stacked_pool_loader_binds_terminal_gate_aliases( transition_authority["sha256"] == manifest["late_producer_transition_authority_sha256"] ) + assert stacked_spine_module._json_ready( + frame.metadata[stacked_spine_module.STACKED_SPINE_MANIFEST_KEY] + ) == ( + manifest["stack_manifest"] + ) + + +def test_ready_stacked_pool_loader_restores_sampled_rung_manifest( + tmp_path: Path, +) -> None: + pytest.importorskip("tables") + manifest_path = _write_ready_pool( + tmp_path, + stacked=True, + sample_fraction=0.25, + ) + + frame, manifest, _ = load_simulation_ready_us_multispine_pool(manifest_path) + + stack_manifest = frame.metadata[ + stacked_spine_module.STACKED_SPINE_MANIFEST_KEY + ] + assert stacked_spine_module._json_ready(stack_manifest) == manifest[ + "stack_manifest" + ] + assert stack_manifest["version"] == 4 + assert stack_manifest["sample_fraction"] == 0.25 + + +def test_ready_stacked_pool_loader_rejects_inconsistent_sampling_factor( + tmp_path: Path, +) -> None: + pytest.importorskip("tables") + manifest_path = _write_ready_pool( + tmp_path, + stacked=True, + sample_fraction=0.25, + ) + manifest = json.loads(manifest_path.read_text(encoding="utf-8")) + manifest["sampling"]["sample_fraction"] = 0.10 + manifest["sampling"]["fraction_token"] = "f010" + manifest_path.write_text(json.dumps(manifest), encoding="utf-8") + + with pytest.raises(ValueError, match="sample_fraction differs"): + load_simulation_ready_us_multispine_pool(manifest_path) + + +def test_ready_stacked_pool_loader_rejects_inconsistent_arm_sampling( + tmp_path: Path, +) -> None: + pytest.importorskip("tables") + manifest_path = _write_ready_pool( + tmp_path, + stacked=True, + sample_fraction=0.25, + ) + manifest = json.loads(manifest_path.read_text(encoding="utf-8")) + manifest["stack_manifest"]["survey_samples"]["asec"]["fraction"] = 0.10 + manifest["sampling"]["stack_manifest_sha256"] = _json_sha256( + manifest["stack_manifest"] + ) + manifest_path.write_text(json.dumps(manifest), encoding="utf-8") + + with pytest.raises(ValueError, match="asec survey-sample fraction differs"): + load_simulation_ready_us_multispine_pool(manifest_path) + + +@pytest.mark.parametrize("sample_fraction", [True, 0, 0.25]) +def test_ready_stacked_pool_loader_rejects_malformed_sampling_receipt( + tmp_path: Path, + sample_fraction: object, +) -> None: + pytest.importorskip("tables") + manifest_path = _write_ready_pool(tmp_path, stacked=True) + manifest = json.loads(manifest_path.read_text(encoding="utf-8")) + if sample_fraction == 0.25: + del manifest["stack_manifest"]["sample_fraction"] + manifest["sampling"]["stack_manifest_sha256"] = _json_sha256( + manifest["stack_manifest"] + ) + else: + manifest["sampling"]["sample_fraction"] = sample_fraction + manifest_path.write_text(json.dumps(manifest), encoding="utf-8") + + with pytest.raises(ValueError, match="sample_fraction.*finite float"): + load_simulation_ready_us_multispine_pool(manifest_path) + + +@pytest.mark.parametrize( + ("mutation", "error_match"), + ( + ("stack_seed_bool", "same non-negative integer"), + ("arm_fraction_bool", "survey-sample fraction differs"), + ("arm_seed_bool", "survey-sample seed differs"), + ("top_realized_bool", "realized-household count is malformed"), + ), +) +def test_ready_stacked_pool_loader_rejects_boolean_sampling_aliases( + tmp_path: Path, + mutation: str, + error_match: str, +) -> None: + pytest.importorskip("tables") + manifest_path = _write_ready_pool(tmp_path, stacked=True) + manifest = json.loads(manifest_path.read_text(encoding="utf-8")) + stack = manifest["stack_manifest"] + samples = stack["survey_samples"] + if mutation == "stack_seed_bool": + manifest["sampling"]["sample_seed"] = 1 + stack["sample_seed"] = True + for sample in samples.values(): + sample["seed"] = 1 + elif mutation == "arm_fraction_bool": + samples["asec"]["fraction"] = True + elif mutation == "arm_seed_bool": + manifest["sampling"]["sample_seed"] = 1 + stack["sample_seed"] = 1 + samples["asec"]["seed"] = True + samples["acs"]["seed"] = 1 + else: + samples["asec"]["realized_household_count"] = 1 + manifest["sampling"]["realized_households"]["asec"] = True + manifest["sampling"]["stack_manifest_sha256"] = _json_sha256(stack) + manifest_path.write_text(json.dumps(manifest), encoding="utf-8") + + with pytest.raises(ValueError, match=error_match): + load_simulation_ready_us_multispine_pool(manifest_path) def test_ready_stacked_pool_loader_binds_h5_cd_vintage_attrs( diff --git a/packages/microcosm-build/tests/test_us_multispine_pool_tool.py b/packages/microcosm-build/tests/test_us_multispine_pool_tool.py index d17303b3b..f4ccd1471 100644 --- a/packages/microcosm-build/tests/test_us_multispine_pool_tool.py +++ b/packages/microcosm-build/tests/test_us_multispine_pool_tool.py @@ -1222,6 +1222,35 @@ def _canonical_late_calibration_owner_receipt( return owner +def _canonical_pregnancy_structural_receipt() -> dict[str, object]: + policy = acs_transfer_module.acs_transfer_execution_contract_identity( + targets=("is_pregnant",), + derive_schedule_d=False, + )["structural_target_policies"]["is_pregnant"] + return { + "policy_sha256": policy["sha256"], + "source_person_key": "person_source_id", + "source_persons_checked": 1, + "physical_rows_checked": 1, + "clone_rows_checked": 0, + "donor_rows_checked": 1, + "qrf_draw_source_persons": 0, + "qrf_draw_rows": 0, + "qrf_fanout_rows": 0, + "preexisting_value_fanout_rows": 0, + "ineligible_rows_assigned_false": 0, + "donor_preexisting_domain_violation_rows": 0, + "recipient_preexisting_domain_violation_rows": 0, + "preexisting_clone_disagreement_source_persons": 0, + "inconsistent_eligibility_source_persons": 0, + "maximum_clones_per_source_person": 1, + "final_incomplete_rows": 0, + "final_domain_violation_rows": 0, + "final_clone_disagreement_source_persons": 0, + "status": "verified", + } + + def _canonical_late_transfer_receipt( pool_tool: ModuleType, *, @@ -1258,6 +1287,11 @@ def _canonical_late_transfer_receipt( } for target in group.targets } + pregnancy_key = f"{group.entity}/{group.family}/is_pregnant" + if pregnancy_key in group_targets: + group_targets[pregnancy_key]["structural_policy"] = ( + _canonical_pregnancy_structural_receipt() + ) calibrated_keys = sorted(set(group_targets) & set(late_specs)) for key in calibrated_keys: group_targets[key]["post_transfer_calibration"] = ( @@ -2465,7 +2499,7 @@ def capture_equality(expected: object, actual: object) -> None: "country": "us", "schema_id": "country_spec", "schema_version": 1, - "spec_sha256": "5378bb9189aec96f50da22aac71e5bd2c3d919e9795f6ef2147e0bc9c739dd8e", + "spec_sha256": "835b4d61de3b153b13c536e371dca6a16fad6d41e003d070f9804c7c251f3590", }, } @@ -2989,6 +3023,30 @@ def test_late_transfer_validator_rejects_stripped_calibration_evidence( ) +def test_late_transfer_validator_rejects_forged_pregnancy_policy( + pool_tool: ModuleType, +) -> None: + receipt = _canonical_late_transfer_receipt(pool_tool) + stacked_spine_module.validate_stacked_post_puf_transfer_receipt( + receipt, + boundary="canonical pregnancy policy control", + ) + forged = copy.deepcopy(receipt) + pregnancy = next( + target_receipt + for group in forged["groups"].values() + for target_key, target_receipt in group["targets"].items() + if target_key.endswith("/is_pregnant") + ) + pregnancy["structural_policy"]["policy_sha256"] = "0" * 64 + + with pytest.raises(ValueError, match="pregnancy structural policy is invalid"): + stacked_spine_module.validate_stacked_post_puf_transfer_receipt( + forged, + boundary="forged pregnancy policy regression", + ) + + @pytest.mark.parametrize( ("target", "constraint_column", "replacement", "error_match"), ( diff --git a/packages/microcosm-build/tests/test_us_multispine_puf_clone.py b/packages/microcosm-build/tests/test_us_multispine_puf_clone.py index 2e0663c05..d9b3dc7b1 100644 --- a/packages/microcosm-build/tests/test_us_multispine_puf_clone.py +++ b/packages/microcosm-build/tests/test_us_multispine_puf_clone.py @@ -10,6 +10,7 @@ BASE_ASEC_SUPPORT_CHANNEL, PUF_TAX_DETAIL_SUPPORT_CHANNEL, clone_us_frame_for_puf_support, + has_assembled_support_metadata, has_support_role_metadata, prepare_us_puf_tax_detail_chain_inputs, puf_tax_detail_clone_mask, @@ -19,7 +20,10 @@ support_role_series, support_source_id_column, ) -from microcosm.build.us_runtime.support_provenance import spine_assembly_manifest +from microcosm.build.us_runtime.support_provenance import ( + spine_assembly_manifest, + support_gate_source_channel_series, +) from microcosm.frame import US_SCHEMA, Frame, WeightKind, Weights @@ -111,6 +115,13 @@ def test_puf_clone_preserves_source_spines_and_routes_by_clone_index() -> None: PUF_TAX_DETAIL_SUPPORT_CHANNEL, PUF_TAX_DETAIL_SUPPORT_CHANNEL, ] + assert has_assembled_support_metadata(table, entity=entity) + assert support_gate_source_channel_series(table, entity=entity).tolist() == [ + "asec", + "acs", + "asec", + "acs", + ] assert puf_tax_detail_clone_mask(table, entity=entity).tolist() == [ False, False, @@ -176,6 +187,11 @@ def test_support_role_legacy_fallback_is_closed_to_known_roles() -> None: ) assert has_support_role_metadata(legacy, entity="person") + assert not has_assembled_support_metadata(legacy, entity="person") + assert support_gate_source_channel_series(legacy, entity="person").tolist() == [ + BASE_ASEC_SUPPORT_CHANNEL, + PUF_TAX_DETAIL_SUPPORT_CHANNEL, + ] assert support_role_series(legacy, entity="person").tolist() == [ BASE_ASEC_SUPPORT_CHANNEL, PUF_TAX_DETAIL_SUPPORT_CHANNEL, diff --git a/packages/microcosm-build/tests/test_us_pregnancy.py b/packages/microcosm-build/tests/test_us_pregnancy.py index 056a869b3..13667e301 100644 --- a/packages/microcosm-build/tests/test_us_pregnancy.py +++ b/packages/microcosm-build/tests/test_us_pregnancy.py @@ -248,6 +248,69 @@ def test_missing_raw_columns_without_signal_raise(self) -> None: with pytest.raises(SourceRuntimeError, match="A_SEX"): with_us_pregnancy_inputs(stripped, seed=0, time_period=TIME_PERIOD) + @pytest.mark.parametrize( + "invalid_row", + ( + {"A_SEX": _MALE, "A_AGE": 25, "is_pregnant": True}, + {"A_SEX": _FEMALE, "A_AGE": 14, "is_pregnant": True}, + {"A_SEX": _FEMALE, "A_AGE": 45, "is_pregnant": True}, + ), + ) + def test_preexisting_domain_violations_are_refused( + self, + invalid_row: dict[str, object], + ) -> None: + rows = [ + invalid_row, + {"A_SEX": _FEMALE, "A_AGE": 25, "is_pregnant": False}, + ] + with pytest.raises( + SourceRuntimeError, + match=r"pregnant_ineligible_rows=1", + ): + with_us_pregnancy_inputs( + _us_frame(rows), + seed=0, + time_period=TIME_PERIOD, + ) + + def test_preexisting_clone_disagreement_is_refused(self) -> None: + rows = [ + { + "A_SEX": _FEMALE, + "A_AGE": 25, + "person_source_id": 42, + "is_pregnant": True, + }, + { + "A_SEX": _FEMALE, + "A_AGE": 25, + "person_source_id": 42, + "is_pregnant": False, + }, + ] + with pytest.raises( + SourceRuntimeError, + match=r"clone_disagreement_source_persons=1", + ): + with_us_pregnancy_inputs( + _us_frame(rows), + seed=0, + time_period=TIME_PERIOD, + ) + + def test_preexisting_near_boolean_value_is_refused(self) -> None: + rows = [ + {"A_SEX": _FEMALE, "A_AGE": 25, "is_pregnant": 1.0 + 1e-12}, + {"A_SEX": _FEMALE, "A_AGE": 25, "is_pregnant": False}, + ] + with pytest.raises(SourceRuntimeError, match=r"non_boolean_rows=1"): + with_us_pregnancy_inputs( + _us_frame(rows), + seed=0, + time_period=TIME_PERIOD, + ) + class TestGate: def test_plausible_seeded_share_passes(self) -> None: @@ -277,3 +340,15 @@ def test_everyone_pregnant_fails_the_share_band(self) -> None: gate = us_pregnancy_signal_gate(_us_frame(rows)) assert not gate.passed assert any("pregnant share" in failure for failure in gate.failures) + + def test_domain_violation_fails_with_structural_counts(self) -> None: + rows = [ + {"A_SEX": _MALE, "A_AGE": 25, "is_pregnant": True}, + {"A_SEX": _FEMALE, "A_AGE": 25, "is_pregnant": False}, + ] + gate = us_pregnancy_signal_gate(_us_frame(rows)) + + assert not gate.passed + assert gate.details["pregnant_nonfemale_rows"] == 1 + assert gate.details["pregnant_ineligible_rows"] == 1 + assert any("outside female ages 15--44" in item for item in gate.failures) diff --git a/packages/microcosm-build/tests/test_us_prior_year_income.py b/packages/microcosm-build/tests/test_us_prior_year_income.py index eb20c37eb..8cd06d53d 100644 --- a/packages/microcosm-build/tests/test_us_prior_year_income.py +++ b/packages/microcosm-build/tests/test_us_prior_year_income.py @@ -3,6 +3,7 @@ from __future__ import annotations import importlib.util +import json from pathlib import Path import numpy as np @@ -10,6 +11,7 @@ import pytest import microcosm.build.us_runtime.prior_year_income as module +from microcosm.build.gates import GateReport from microcosm.build.source_runtime import SourceRuntimeError from microcosm.build.us_runtime.l0_refit_export import ( US_RELEASE_REQUIRED_PERSON_SOURCE_COLUMNS, @@ -395,6 +397,30 @@ def _signal_frame() -> Frame: ) +def _signal_frame_with_availability_rows(rows: int) -> Frame: + frame = _signal_frame() + person = frame.table("person").copy() + person["previous_year_income_available"] = np.arange(len(person)) < rows + return module._replace_person_table(frame, person) + + +def _with_stack_manifest( + frame: Frame, + manifest: object, +) -> Frame: + return Frame( + {entity: frame.table(entity).copy() for entity in frame.entities}, + frame.schema, + { + entity: frame.weights_for(entity) + for entity in frame.weighted_entities + }, + frame.strata, + mass_log=frame.mass_log, + metadata={"us_stacked_spine_manifest": manifest}, + ) + + def test_signal_gate_accepts_signed_source_signal_and_rejects_defaults() -> None: passing = us_prior_year_income_signal_gate(_signal_frame()) assert passing.passed, passing.failures @@ -410,6 +436,146 @@ def test_signal_gate_accepts_signed_source_signal_and_rejects_defaults() -> None assert "availability" in " ".join(failing.failures) +def test_sampled_rung_scales_only_prior_year_availability_floor() -> None: + frame = _with_stack_manifest( + _signal_frame_with_availability_rows(6), + {"version": 4, "sample_fraction": 0.25}, + ) + + gate = us_prior_year_income_signal_gate(frame) + + assert gate.passed, gate.failures + assert gate.details["previous_year_income_available_share"] == pytest.approx( + 0.04101010101010102 + ) + assert gate.details["previous_year_income_available_share_band"] == [0.05, 0.50] + assert gate.details[ + "previous_year_income_available_sampled_match_survival_factor" + ] == pytest.approx(0.25) + assert gate.details["previous_year_income_available_applied_floor"] == pytest.approx( + 0.0125 + ) + assert gate.details["previous_year_income_available_applied_share_band"] == [ + 0.0125, + 0.50, + ] + assert gate.details["self_employment_income_last_year_nonzero_share_band"] == [ + 0.01, + 0.25, + ] + + +def test_sampled_rung_preserves_applied_floor_and_authored_upper_bound() -> None: + below_floor = us_prior_year_income_signal_gate( + _with_stack_manifest( + _signal_frame_with_availability_rows(1), + {"version": 4, "sample_fraction": 0.25}, + ) + ) + above_upper = us_prior_year_income_signal_gate( + _with_stack_manifest( + _signal_frame_with_availability_rows(60), + {"version": 4, "sample_fraction": 0.25}, + ) + ) + + assert any("outside [0.012500, 0.500000]" in row for row in below_floor.failures) + assert any("outside [0.012500, 0.500000]" in row for row in above_upper.failures) + + +def test_full_rung_gate_manifest_is_byte_identical_to_legacy_gate() -> None: + frame = _signal_frame_with_availability_rows(6) + legacy = us_prior_year_income_signal_gate(frame) + full_rung = us_prior_year_income_signal_gate( + _with_stack_manifest( + frame, + {"version": 4, "sample_fraction": 1.0}, + ) + ) + + def manifest_bytes(gate) -> bytes: + return json.dumps( + GateReport((gate,)).to_manifest(), + allow_nan=False, + ensure_ascii=False, + separators=(",", ":"), + sort_keys=True, + ).encode("utf-8") + + assert full_rung == legacy + assert manifest_bytes(full_rung) == manifest_bytes(legacy) + assert not any("applied" in key or "survival_factor" in key for key in full_rung.details) + + +def test_legacy_acs_only_sampling_does_not_scale_asec_match_floor() -> None: + frame = _signal_frame_with_availability_rows(6) + legacy = us_prior_year_income_signal_gate(frame) + pilot = us_prior_year_income_signal_gate( + _with_stack_manifest( + frame, + {"version": 1, "acs_sample_fraction": 0.25}, + ) + ) + + assert not legacy.passed + assert pilot == legacy + + +@pytest.mark.parametrize( + "manifest", + [ + {"version": 4}, + {"version": 4, "sample_fraction": True}, + {"version": 4, "sample_fraction": 0.0}, + {"version": 3, "sample_fraction": 0.25}, + "malformed", + ], +) +def test_signal_gate_rejects_malformed_stacked_sampling_metadata( + manifest: object, +) -> None: + with pytest.raises(ValueError, match="prior-year-income availability"): + us_prior_year_income_signal_gate( + _with_stack_manifest(_signal_frame(), manifest) + ) + + +def test_clone_availability_checks_all_assembled_clones_and_legacy_pairs() -> None: + assembled = _frame( + pd.DataFrame( + { + "person_source_id": [10, 10, 10, 20, 20, 20], + "person_spine_source_id": [1, 1, 1, 2, 2, 2], + "person_support_channel": ["acs"] * 6, + "person_support_clone_index": [0, 1, 2, 0, 1, 2], + "self_employment_income_last_year": [10, 10, 10, -5, -5, -5], + "previous_year_income_available": [True, True, False] + [False] * 3, + } + ) + ) + assembled_summary = module.us_prior_year_income_summary(assembled) + assert assembled_summary["clone_availability_mismatches"] == 1 + assembled_gate = us_prior_year_income_signal_gate(assembled) + assert any("1 source person" in failure for failure in assembled_gate.failures) + + legacy = _frame( + pd.DataFrame( + { + "person_source_id": [10, 10, 20, 20], + "person_support_channel": [ + BASE_ASEC_SUPPORT_CHANNEL, + PUF_TAX_DETAIL_SUPPORT_CHANNEL, + ] + * 2, + "self_employment_income_last_year": [10, 10, -5, -5], + "previous_year_income_available": [True, True, False, False], + } + ) + ) + legacy_summary = module.us_prior_year_income_summary(legacy) + assert legacy_summary["clone_availability_mismatches"] == 0 + + def test_source_reconciliation_detects_plausible_but_wrong_asec_carry() -> None: derived = with_us_prior_year_income_inputs( _source_frame(), seed=0, time_period=2024 diff --git a/packages/microcosm-build/tests/test_us_retirement_contributions.py b/packages/microcosm-build/tests/test_us_retirement_contributions.py index db8c97d5a..c40f22d66 100644 --- a/packages/microcosm-build/tests/test_us_retirement_contributions.py +++ b/packages/microcosm-build/tests/test_us_retirement_contributions.py @@ -14,6 +14,7 @@ derive_us_retirement_contributions_from_manifest, us_retirement_contributions_signal_gate, us_retirement_contributions_stage_spec, + us_retirement_contributions_summary, with_us_retirement_contribution_inputs, ) from microcosm.frame import US_SCHEMA, Frame, WeightKind, Weights @@ -87,6 +88,26 @@ def _derive(frame: pd.DataFrame) -> pd.DataFrame: return derive_us_retirement_contributions_from_manifest(frame, operation, None) +def _stacked_frame() -> Frame: + direct = with_us_retirement_contribution_inputs( + _frame(), + seed=0, + time_period=2024, + ) + stacked = clone_us_frame_for_puf_support(direct) + person = stacked.table("person") + source_record = np.tile(np.arange(4, dtype=np.int64), 2) + person["person_spine_source_id"] = source_record + person["person_support_channel"] = np.where( + source_record < 2, + "asec", + "acs", + ) + acs = person["person_support_channel"].eq("acs") + person.loc[acs, ["RETCB_VAL", "WSAL_VAL", "SEMP_VAL"]] = np.nan + return stacked + + def test_stage_manifest_pins_sources_operations_and_five_desired_leaves() -> None: spec = us_retirement_contributions_stage_spec() @@ -160,6 +181,25 @@ def test_with_inputs_materializes_signal_and_preserves_reported_total() -> None: assert allocated.iloc[3] == 0.0 +def test_stacked_gate_validates_physical_source_and_reconciles_direct_role() -> None: + stacked = _stacked_frame() + + gate = us_retirement_contributions_signal_gate(stacked) + + assert gate.passed, gate.failures + assert gate.details["source_rows"] == 4 + assert gate.details["source_reconciliation_rows"] == 2 + assert gate.details["allocation_mismatch_rows"] == 0 + + person = stacked.table("person") + asec_puf_role = person["person_support_channel"].eq("asec") & person[ + "person_support_clone_index" + ].eq(1) + person.loc[person.index[asec_puf_role][0], "RETCB_VAL"] = np.nan + with pytest.raises(SourceRuntimeError, match="RETCB_VAL"): + us_retirement_contributions_summary(stacked) + + def test_puf_half_uses_qrf_predictions_and_applies_income_constraints( monkeypatch: pytest.MonkeyPatch, ) -> None: diff --git a/packages/microcosm-build/tests/test_us_retirement_distributions.py b/packages/microcosm-build/tests/test_us_retirement_distributions.py index 5ac1b0495..8921b521e 100644 --- a/packages/microcosm-build/tests/test_us_retirement_distributions.py +++ b/packages/microcosm-build/tests/test_us_retirement_distributions.py @@ -28,6 +28,7 @@ derive_us_retirement_distributions_from_manifest, us_retirement_distributions_signal_gate, us_retirement_distributions_stage_spec, + us_retirement_distributions_summary, with_us_retirement_distribution_inputs, ) from microcosm.build.us_runtime.source_runtime import us_source_operation_handlers @@ -125,6 +126,29 @@ def _derive(frame: pd.DataFrame) -> pd.DataFrame: ) +def _stacked_frame() -> Frame: + direct = with_us_retirement_distribution_inputs( + _frame(), + seed=0, + time_period=2024, + ) + stacked = clone_us_frame_for_puf_support(direct) + person = stacked.table("person") + source_record = np.tile(np.arange(8, dtype=np.int64), 2) + person["person_spine_source_id"] = source_record + person["person_support_channel"] = np.where( + source_record < 4, + "asec", + "acs", + ) + acs = person["person_support_channel"].eq("acs") + person.loc[ + acs, + list(US_RETIREMENT_DISTRIBUTION_REQUIRED_SOURCE_COLUMNS), + ] = np.nan + return stacked + + def _sha256(path: Path) -> str: digest = sha256() with path.open("rb") as stream: @@ -274,6 +298,25 @@ def test_frame_integration_gate_and_idempotence() -> None: ) +def test_stacked_gate_validates_physical_source_and_reconciles_direct_role() -> None: + stacked = _stacked_frame() + + gate = us_retirement_distributions_signal_gate(stacked) + + assert gate.passed, gate.failures + assert gate.details["source_rows"] == 8 + assert gate.details["source_reconciliation_rows"] == 4 + assert all(value == 0 for value in gate.details["source_mismatches"].values()) + + person = stacked.table("person") + asec_puf_role = person["person_support_channel"].eq("asec") & person[ + "person_support_clone_index" + ].eq(1) + person.loc[person.index[asec_puf_role][0], "DST_SC1"] = np.nan + with pytest.raises(SourceRuntimeError, match="account codes"): + us_retirement_distributions_summary(stacked) + + def test_puf_half_uses_qrf_and_asec_half_remains_exact( monkeypatch: pytest.MonkeyPatch, ) -> None: diff --git a/packages/microcosm-build/tests/test_us_sipp_head_start.py b/packages/microcosm-build/tests/test_us_sipp_head_start.py index 010a1d2f3..52f13aa02 100644 --- a/packages/microcosm-build/tests/test_us_sipp_head_start.py +++ b/packages/microcosm-build/tests/test_us_sipp_head_start.py @@ -429,6 +429,42 @@ def test_imputer_rejects_identical_duplicate_same_role_source_rows( impute_us_sipp_head_start(duplicate, _donor(), seed=0) +def test_assembled_clone_two_uses_lowest_clone_and_fans_to_every_clone( + monkeypatch: pytest.MonkeyPatch, +) -> None: + monkeypatch.setattr(module, "QRF", _FakeQRF) + frame = _frame( + [10, 10, 10, 20, 20], + ages=[4, 4, 4, 5, 5], + female=[True, False, False, False, True], + channels=["acs"] * 5, + ) + person = frame.table("person").copy() + person["person_spine_source_id"] = [100, 100, 100, 200, 200] + person["person_support_clone_index"] = [0, 1, 2, 1, 2] + assembled = _replace_person(frame, person) + + predicted = impute_us_sipp_head_start(assembled, _donor(), seed=7) + by_source = pd.DataFrame( + { + "source": person["person_source_id"], + "value": predicted, + } + ).groupby("source")["value"] + assert (by_source.nunique() == 1).all() + assert by_source.first().to_dict() == {10: True, 20: False} + + materialized = person.copy() + materialized[_OUTPUT] = predicted.to_numpy() + summary = us_sipp_head_start_summary(_replace_person(frame, materialized)) + assert summary["clone_group_count"] == 2 + assert summary["clone_mismatch_count"] == 0 + + materialized.loc[materialized["person_support_clone_index"].eq(2), _OUTPUT] ^= True + mismatch = us_sipp_head_start_summary(_replace_person(frame, materialized)) + assert mismatch["clone_mismatch_count"] == 2 + + def test_wrapper_heals_stale_output_and_is_exactly_idempotent( monkeypatch: pytest.MonkeyPatch, ) -> None: diff --git a/packages/microcosm-build/tests/test_us_spec_bundle.py b/packages/microcosm-build/tests/test_us_spec_bundle.py index 586907804..892ece523 100644 --- a/packages/microcosm-build/tests/test_us_spec_bundle.py +++ b/packages/microcosm-build/tests/test_us_spec_bundle.py @@ -382,11 +382,11 @@ def test_constant_derived_domain_counts_are_complete( assert len(compiled_schedule["waves"]) == 6 assert ( compiled_schedule["schedule_sha256"] - == "dcf3c6d2eade3449836c49a1dc4d3b8cd395aab9142db700c3c60598fa9c1c79" + == "e59c019d3d454eac99ac0ac209b6c5b6faaf9bdfcaeee18c36a25be19bf7da2f" ) assert ( compiled_schedule["payload_sha256"] - == "5921cda83725b2801f2713242003e99ba54766851808b94a4f483666bce604c5" + == "02e618cc656eb39990ed99dca2b30a52794e01e2b06a3c2df87ca4a7d85ab086" ) assert len(take_up["programs"]) == 17 @@ -743,7 +743,20 @@ def collect_sha256(value: object, path: tuple[str, ...] = ()) -> None: "sha256" ] ), - ) + ), + ( + ( + "transfer_execution", + "structural_target_policies", + "is_pregnant", + "sha256", + ), + str( + imputation["transfer_execution"]["structural_target_policies"][ + "is_pregnant" + ]["sha256"] + ), + ), ] assert len(asset_pins) == 2 assert set(asset_pins) == { diff --git a/packages/microcosm-build/tests/test_us_spine_blindness.py b/packages/microcosm-build/tests/test_us_spine_blindness.py index aa6b80162..5c1892d55 100644 --- a/packages/microcosm-build/tests/test_us_spine_blindness.py +++ b/packages/microcosm-build/tests/test_us_spine_blindness.py @@ -94,6 +94,8 @@ { # Declares and receipts exact ACS source universes; never mutates rows. "acs_income_universe.py", + # Owner-approved release boundary: exact raw ACS join and receipt. + "acs_release_predictors.py", "base_pool.py", # Legacy late-spine assembly. # Enumerates provenance columns only to reject preassembled source frames. "operator_boundary.py", @@ -174,6 +176,36 @@ "workers_compensation.py", ) +_GATE_SOURCE_CHANNEL_ACCESSOR = "support_gate_source_channel_series" +_GATE_SOURCE_CHANNEL_CALLERS = { + "alimony.py": frozenset({"us_alimony_signal_gate"}), + "medicare_take_up.py": frozenset({"_asec_source_mask"}), + "retirement_contributions.py": frozenset({"_asec_source_mask"}), + "retirement_distributions.py": frozenset({"_asec_source_mask"}), + "ssi_take_up.py": frozenset({"us_ssi_take_up_reporter_source_ids"}), + "weeks_unemployed.py": frozenset({"_weeks_unemployed_gate_scopes"}), + "workers_compensation.py": frozenset({"us_workers_compensation_summary"}), +} +_GATE_SOURCE_SCOPE_HELPER_CALLERS = { + ("medicare_take_up.py", "_asec_source_mask"): frozenset( + {"us_medicare_take_up_summary"} + ), + ("retirement_contributions.py", "_asec_source_mask"): frozenset( + {"_source_reconciliation_mask", "us_retirement_contributions_summary"} + ), + ( + "retirement_contributions.py", + "_source_reconciliation_mask", + ): frozenset({"us_retirement_contributions_summary"}), + ("retirement_distributions.py", "_asec_source_mask"): frozenset( + {"_source_reconciliation_mask", "us_retirement_distributions_summary"} + ), + ( + "retirement_distributions.py", + "_source_reconciliation_mask", + ): frozenset({"us_retirement_distributions_summary"}), +} + # Every runtime module must be deliberately classified. This allowlist does # not exempt a module from the all-runtime AST scan below; it only records # modules outside the migrated population-treatment registry. Keeping the @@ -186,6 +218,7 @@ "acs_inputs.py", "acs_multispine.py", "acs_pums.py", + "acs_release_predictors.py", # Pinned release join; provenance owner. "acs_sources.py", "acs_transfer_bank.py", # Bounded checkpoint I/O; no population treatment. "asec_checkpoint.py", # Bounded checkpoint I/O; no population treatment. @@ -3071,6 +3104,44 @@ def _called_function_names(source: str) -> set[str]: } +def _function_callers(source: str, callee: str) -> tuple[tuple[str, int], ...]: + """Return function names and lines that call one imported or local name.""" + + tree = ast.parse(source) + aliases = {callee} + for node in ast.walk(tree): + if not isinstance(node, ast.ImportFrom): + continue + aliases.update( + alias.asname or alias.name for alias in node.names if alias.name == callee + ) + + class CallerVisitor(ast.NodeVisitor): + def __init__(self) -> None: + self.function_stack: list[str] = [] + self.callers: list[tuple[str, int]] = [] + + def visit_FunctionDef(self, node: ast.FunctionDef) -> None: + self.function_stack.append(node.name) + self.generic_visit(node) + self.function_stack.pop() + + def visit_AsyncFunctionDef(self, node: ast.AsyncFunctionDef) -> None: + self.function_stack.append(node.name) + self.generic_visit(node) + self.function_stack.pop() + + def visit_Call(self, node: ast.Call) -> None: + if _call_name(node) in aliases: + caller = self.function_stack[-1] if self.function_stack else "" + self.callers.append((caller, node.lineno)) + self.generic_visit(node) + + visitor = CallerVisitor() + visitor.visit(tree) + return tuple(visitor.callers) + + def _imported_us_runtime_modules(source: str) -> tuple[str, ...]: """Return statically imported, flat ``us_runtime`` module filenames.""" @@ -3267,6 +3338,49 @@ def test_registered_population_operators_do_not_read_any_source_channel() -> Non ) +def test_physical_source_accessor_is_confined_to_reviewed_gates() -> None: + """Origin-aware diagnostics must not become population treatments.""" + + actual: dict[str, frozenset[str]] = {} + call_details: dict[str, tuple[tuple[str, int], ...]] = {} + for path in sorted(_US_RUNTIME.glob("*.py")): + if path.name == "support_provenance.py": + continue + callers = _function_callers(path.read_text(), _GATE_SOURCE_CHANNEL_ACCESSOR) + if callers: + actual[path.name] = frozenset(caller for caller, _line in callers) + call_details[path.name] = callers + assert actual == _GATE_SOURCE_CHANNEL_CALLERS, ( + "The physical support-channel accessor is restricted to reviewed " + "release gates and reporter capture; population derivation, imputation, " + f"or wrapper use is forbidden. Found callers: {call_details}" + ) + + for ( + module_name, + helper, + ), expected_callers in _GATE_SOURCE_SCOPE_HELPER_CALLERS.items(): + callers = _function_callers((_US_RUNTIME / module_name).read_text(), helper) + actual_callers = frozenset(caller for caller, _line in callers) + assert actual_callers == expected_callers, ( + f"Physical source-scope helper {module_name}:{helper} may be called " + "only by its reviewed gate-summary chain; " + f"expected={sorted(expected_callers)}, found={callers}." + ) + + +def test_physical_source_accessor_guard_rejects_a_derivation_call() -> None: + source = """ +from owner import support_gate_source_channel_series as channels + +def derive_population(frame): + return channels(frame, entity=\"person\") +""" + assert _function_callers(source, _GATE_SOURCE_CHANNEL_ACCESSOR) == ( + ("derive_population", 5), + ) + + def test_pool_build_tool_import_graph_is_source_spine_blind() -> None: """The wired CLI and every runtime operator it reaches remain blind.""" diff --git a/packages/microcosm-build/tests/test_us_ssi_disability_criteria.py b/packages/microcosm-build/tests/test_us_ssi_disability_criteria.py index 5c52c49af..ca9362ccc 100644 --- a/packages/microcosm-build/tests/test_us_ssi_disability_criteria.py +++ b/packages/microcosm-build/tests/test_us_ssi_disability_criteria.py @@ -47,6 +47,35 @@ ) +def test_reported_ssi_anchor_coalesces_native_asec_and_harmonized_acs() -> None: + person = pd.DataFrame( + { + "SSI_VAL": [1_200.0, np.nan, np.nan], + "ssi_reported": [np.nan, 900.0, np.nan], + } + ) + + values = module._reported_ssi_anchor( + person, + age=np.asarray([40.0, 50.0, 10.0]), + ) + + np.testing.assert_array_equal(values, [1_200.0, 900.0, 0.0]) + assert pd.isna(person.loc[2, "ssi_reported"]) + + +def test_reported_ssi_anchor_refuses_an_adult_universe_blank() -> None: + person = pd.DataFrame( + { + "SSI_VAL": [np.nan], + "ssi_reported": [np.nan], + } + ) + + with pytest.raises(ValueError, match="blank only below"): + module._reported_ssi_anchor(person, age=np.asarray([40.0])) + + def _source_row( ssuid: str, pnum: int, @@ -584,6 +613,48 @@ def test_signal_gate_requires_each_channel_but_allows_clone_divergence( ) +def test_stacked_clone_divergence_diagnostic_checks_clone_two() -> None: + stacked = _replace_person( + _frame(3), + **{ + "person_source_id": np.asarray([10, 10, 10]), + "person_spine_source_id": np.asarray([1, 1, 1]), + "person_support_channel": np.asarray(["acs", "acs", "acs"]), + "person_support_clone_index": np.asarray([0, 1, 2]), + _OUTPUT: np.asarray([False, False, True]), + }, + ) + + summary = us_ssi_disability_criteria_summary(stacked) + + assert summary["clone_divergence_source_people"] == 1 + + +def test_summary_checks_harmonized_ssi_on_native_role() -> None: + expanded = clone_us_frame_for_puf_support(_frame()) + person = expanded.table("person") + person["ssi_reported"] = np.nan + native = person["person_support_channel"].astype(str).eq("asec") + source_two = person["person_source_id"].eq(2) + person.loc[native & source_two, "SSI_VAL"] = np.nan + person.loc[native & source_two, "ssi_reported"] = 900.0 + preserved_existing_anchor = ( + native & person["person_source_id"].eq(1) + ).to_numpy() + invalid = _replace_person( + expanded, + **{_OUTPUT: preserved_existing_anchor}, + ) + + summary = us_ssi_disability_criteria_summary(invalid) + gate = us_ssi_disability_criteria_signal_gate(invalid) + + assert summary["reporter_anchor_mismatches"] == 1 + assert any( + "native-role SSI reporter anchor" in failure for failure in gate.failures + ) + + def test_gate_requires_complete_support_provenance( monkeypatch: pytest.MonkeyPatch, ) -> None: diff --git a/packages/microcosm-build/tests/test_us_ssi_take_up.py b/packages/microcosm-build/tests/test_us_ssi_take_up.py index 8ae286b8c..5819e6302 100644 --- a/packages/microcosm-build/tests/test_us_ssi_take_up.py +++ b/packages/microcosm-build/tests/test_us_ssi_take_up.py @@ -150,6 +150,29 @@ def _frame(*, stale_output: bool = False) -> tuple[Frame, np.ndarray]: return frame, np.asarray(potential, dtype=np.float64) +def _stacked_frame() -> tuple[Frame, np.ndarray]: + """Convert the legacy operator fixture to physical ASEC/ACS channels.""" + + frame, potential = _frame() + person = frame.table("person").copy() + source_number = person["person_source_id"].str.rsplit(":").str[-1].astype(int) + legacy_roles = person["person_support_channel"].copy() + person["person_spine_source_id"] = person["person_source_id"] + person["person_support_clone_index"] = np.where( + legacy_roles.eq("asec"), + 0, + 1, + ) + person["person_support_channel"] = np.where( + source_number.le(6), + "asec", + "acs", + ) + acs_source = person["person_support_channel"].eq("acs") + person.loc[acs_source, US_SSI_TAKE_UP_ANCHOR] = np.nan + return _replace_person(frame, person), potential + + def _replace_person(frame: Frame, person: pd.DataFrame) -> Frame: tables = {entity: frame.table(entity).copy() for entity in frame.entities} tables["person"] = person @@ -309,6 +332,45 @@ def test_puf_only_ssi_value_is_not_promoted_to_reporter_anchor() -> None: assert diagnostics["age_bands"] == baseline_diagnostics["age_bands"] +def test_stacked_asec_sources_own_reporters_across_operator_clones() -> None: + frame, potential = _stacked_frame() + + reporter_source_ids = us_ssi_take_up_reporter_source_ids(frame) + expected = frozenset( + f"{band}:{source_number}" + for band in _AGES + for source_number in (0, 6) + ) + assert reporter_source_ids == expected + + result, diagnostics = with_us_ssi_take_up( + frame, + uncapped_ssi=potential, + seed=17, + targets=_TARGETS, + ) + assert diagnostics["reporter_anchor_lost_count"] == 0 + assert result.table("person").loc[ + result.table("person")["person_source_id"].isin(expected), _OUTPUT + ].all() + + person = frame.table("person").copy() + asec_clone = person["person_support_channel"].eq("asec") & person[ + "person_support_clone_index" + ].eq(1) + person.loc[asec_clone.idxmax(), US_SSI_TAKE_UP_ANCHOR] = np.nan + invalid = _replace_person(frame, person) + with pytest.raises(ValueError, match="physical ASEC source rows"): + us_ssi_take_up_reporter_source_ids(invalid) + with pytest.raises(ValueError, match="physical ASEC source rows"): + with_us_ssi_take_up( + invalid, + uncapped_ssi=potential, + seed=17, + targets=_TARGETS, + ) + + def test_reporter_lineage_survives_when_l0_keeps_only_the_puf_clone() -> None: full, potential = _frame() reporter_source_ids = us_ssi_take_up_reporter_source_ids(full) diff --git a/packages/microcosm-build/tests/test_us_stacked_spine.py b/packages/microcosm-build/tests/test_us_stacked_spine.py index 6036bdfc1..0d4556814 100644 --- a/packages/microcosm-build/tests/test_us_stacked_spine.py +++ b/packages/microcosm-build/tests/test_us_stacked_spine.py @@ -3555,10 +3555,13 @@ def _post_puf_transfer_fixture() -> Frame: index=person.index, dtype="boolean", ) - person.loc[source_producer_rows, "is_pregnant"] = np.resize( - np.asarray([True, False]), - int(source_producer_rows.sum()), + source_eligible = ( + source_producer_rows + & person["is_female"].astype(bool) + & person["age"].between(15, 44, inclusive="both") ) + person.loc[source_producer_rows, "is_pregnant"] = False + person.loc[source_eligible, "is_pregnant"] = True tables = {entity: attached.table(entity) for entity in attached.entities} tables["person"] = person return Frame( @@ -5442,6 +5445,32 @@ def transfer( "unmodeled_rows": 0, "residual_null_rows": 0, } + if target == "is_pregnant": + pregnancy_policy = execution_contract[ + "structural_target_policies" + ]["is_pregnant"] + target_receipt["structural_policy"] = { + "policy_sha256": pregnancy_policy["sha256"], + "source_person_key": "person_source_id", + "source_persons_checked": 1, + "physical_rows_checked": 1, + "clone_rows_checked": 0, + "donor_rows_checked": 1, + "qrf_draw_source_persons": 1, + "qrf_draw_rows": 1, + "qrf_fanout_rows": 0, + "preexisting_value_fanout_rows": 0, + "ineligible_rows_assigned_false": 0, + "donor_preexisting_domain_violation_rows": 0, + "recipient_preexisting_domain_violation_rows": 0, + "preexisting_clone_disagreement_source_persons": 0, + "inconsistent_eligibility_source_persons": 0, + "maximum_clones_per_source_person": 1, + "final_incomplete_rows": 0, + "final_domain_violation_rows": 0, + "final_clone_disagreement_source_persons": 0, + "status": "verified", + } if key in late_specs: target_receipt["qrf_pattern_evidence"] = ( stacked_spine_module._acs_imputed_pattern_evidence(record) @@ -6316,6 +6345,104 @@ def test_post_puf_transfer_preserves_complete_asec_source_producers() -> None: if item.column == "is_pregnant" ) assert all(not pattern.target_regimes for pattern in record.patterns) + structural = receipt["structural_policy"] + assert structural == record.structural_receipt + assert structural["status"] == "verified" + assert structural["source_person_key"] == "person_source_id" + assert structural["qrf_draw_rows"] < receipt["imputed_rows"] + assert structural["final_domain_violation_rows"] == 0 + assert structural["final_clone_disagreement_source_persons"] == 0 + assert ( + person.groupby("person_source_id", sort=False)["is_pregnant"] + .nunique() + .eq(1) + .all() + ) + + +def test_complete_pregnancy_surface_retains_zero_imputation_structural_receipt() -> None: + frame = _post_puf_transfer_fixture() + person = frame.table("person").copy() + recipient_rows = person[support_channel_column("person")].astype(str).eq("acs") + person.loc[recipient_rows, "is_pregnant"] = False + tables = {entity: frame.table(entity) for entity in frame.entities} + tables["person"] = person + complete = Frame( + tables, + frame.schema, + {entity: frame.weights_for(entity) for entity in frame.weighted_entities}, + frame.strata, + mass_log=frame.mass_log, + metadata=frame.metadata, + ) + surface = {"person": {"model_required_boolean": ("is_pregnant",)}} + authority = stacked_spine_module._make_test_stacked_authority( + declared_surface=surface, + gap_fill_plan=(), + post_puf_transfer_surface=surface, + ) + + result = stacked_spine_module._transfer_stacked_post_puf_inputs_with_test_authority( + complete, + authority=authority, + seed=578, + n_estimators=10, + ) + + receipt = result.receipt["targets"][ + "person/model_required_boolean/is_pregnant" + ] + assert receipt["authorized_null_rows"] == 0 + assert receipt["imputed_rows"] == 0 + assert receipt["structural_policy"]["status"] == "verified" + record = next( + item + for item in result.transfer_result.imputed_inputs + if item.column == "is_pregnant" + ) + assert record.imputed_recipient_rows == 0 + assert record.structural_receipt == receipt["structural_policy"] + + +def test_post_puf_transfer_refuses_invalid_pregnancy_source_producer() -> None: + frame = _post_puf_transfer_fixture() + person = frame.table("person").copy() + source_rows = person[support_channel_column("person")].astype(str).eq("asec") + ineligible = source_rows & ~( + person["is_female"].astype(bool) + & person["age"].between(15, 44, inclusive="both") + ) + donor_ineligible = ineligible & person[ + support_clone_index_column("person") + ].eq(1) + person.loc[person.index[donor_ineligible][0], "is_pregnant"] = True + tables = {entity: frame.table(entity) for entity in frame.entities} + tables["person"] = person + invalid = Frame( + tables, + frame.schema, + {entity: frame.weights_for(entity) for entity in frame.weighted_entities}, + frame.strata, + mass_log=frame.mass_log, + metadata=frame.metadata, + ) + surface = {"person": {"model_required_boolean": ("is_pregnant",)}} + authority = stacked_spine_module._make_test_stacked_authority( + declared_surface=surface, + gap_fill_plan=(), + post_puf_transfer_surface=surface, + ) + + with pytest.raises( + ValueError, + match=r"preexisting donor domain violation", + ): + stacked_spine_module._transfer_stacked_post_puf_inputs_with_test_authority( + invalid, + authority=authority, + seed=578, + n_estimators=10, + ) def test_late_calibration_owner_mutates_only_acs_clone_zero_transfer_cells() -> None: diff --git a/packages/microcosm-build/tests/test_us_voluntary_filing.py b/packages/microcosm-build/tests/test_us_voluntary_filing.py index b3460662b..304f4d63d 100644 --- a/packages/microcosm-build/tests/test_us_voluntary_filing.py +++ b/packages/microcosm-build/tests/test_us_voluntary_filing.py @@ -652,6 +652,55 @@ def test_duplicate_same_role_source_rows_fail_closed() -> None: impute_us_voluntary_filing(expanded, _donor(), seed=17) +def test_assembled_clone_two_uses_explicit_index_and_checks_every_clone() -> None: + tax_unit = pd.DataFrame( + { + "tax_unit_id": [100, 101, 102, 200, 201], + "tax_unit_source_id": [10, 10, 10, 20, 20], + "tax_unit_spine_source_id": [1, 1, 1, 2, 2], + "tax_unit_support_channel": ["acs"] * 5, + "tax_unit_support_clone_index": [0, 1, 2, 1, 2], + } + ) + receiver = pd.DataFrame( + { + predictor: np.arange(5, dtype=np.float64) + offset + for offset, predictor in enumerate( + SIPP_VOLUNTARY_FILING_MODEL_PREDICTORS + ) + }, + index=tax_unit["tax_unit_id"], + ) + + class TaxUnitFrame: + def table(self, entity: str) -> pd.DataFrame: + assert entity == "tax_unit" + return tax_unit + + prediction_rows, fan_keys = module._source_receiver_rows( + TaxUnitFrame(), receiver + ) + assert prediction_rows.index.tolist() == ["10", "20"] + assert fan_keys.tolist() == ["10", "10", "10", "20", "20"] + # Source 10 prefers clone 0; source 20 has no native survivor and picks + # the lowest surviving clone index, clone 1. + assert prediction_rows.iloc[:, 0].tolist() == [0.0, 3.0] + + summary_frame = _replace_tax_unit( + _frame(3), + **{ + _OUTPUT: np.asarray([False, False, True]), + "tax_unit_source_id": np.asarray([7, 7, 7]), + "tax_unit_spine_source_id": np.asarray([70, 70, 70]), + "tax_unit_support_channel": np.asarray(["acs", "acs", "acs"]), + "tax_unit_support_clone_index": np.asarray([0, 1, 2]), + }, + ) + summary = us_voluntary_filing_summary(summary_frame) + assert summary["clone_source_units"] == 1 + assert summary["clone_mismatch_source_units"] == 1 + + def test_real_qrf_recomputation_is_deterministic() -> None: frame = _frame(14) donor = _donor(120) diff --git a/packages/microcosm-build/tests/test_us_weeks_unemployed.py b/packages/microcosm-build/tests/test_us_weeks_unemployed.py index bd6e33419..479b11490 100644 --- a/packages/microcosm-build/tests/test_us_weeks_unemployed.py +++ b/packages/microcosm-build/tests/test_us_weeks_unemployed.py @@ -243,6 +243,46 @@ def _gate_frame() -> Frame: ) +def _stacked_gate_frame() -> Frame: + frame = _gate_frame() + person = frame.table("person").copy() + asec_rows = 1_000 + asec_native_rows = 500 + acs_native_rows = 1_000 + person["person_support_channel"] = ["asec"] * asec_rows + ["acs"] * 2_000 + person["person_support_clone_index"] = np.concatenate( + [ + np.zeros(asec_native_rows, dtype=np.int64), + np.ones(asec_rows - asec_native_rows, dtype=np.int64), + np.zeros(acs_native_rows, dtype=np.int64), + np.ones(2_000 - acs_native_rows, dtype=np.int64), + ] + ) + person["person_spine_source_id"] = np.concatenate( + [ + np.arange(asec_native_rows, dtype=np.int64), + np.arange(asec_native_rows, dtype=np.int64), + np.arange(10_000, 10_000 + acs_native_rows, dtype=np.int64), + np.arange(10_000, 10_000 + acs_native_rows, dtype=np.int64), + ] + ) + weeks = np.zeros(len(person), dtype=np.float64) + weeks[:18] = 17.0 + weeks[asec_native_rows : asec_native_rows + 12] = 17.0 + weeks[asec_rows + acs_native_rows : asec_rows + acs_native_rows + 12] = 16.0 + person[_OUTPUT] = weeks + source = np.full(len(person), np.nan, dtype=np.float64) + source[:asec_rows] = 0.0 + source[:18] = 17.0 + person["LKWEEKS"] = source + unemployment_compensation = np.zeros(len(person), dtype=np.float64) + unemployment_compensation[ + asec_native_rows : asec_native_rows + 12 + ] = 100.0 + person["unemployment_compensation"] = unemployment_compensation + return module._replace_person_table(frame, person) + + def test_public_stage_contract_is_exactly_manifest_pinned() -> None: spec = us_weeks_unemployed_stage_spec() @@ -665,6 +705,74 @@ def test_signal_gate_requires_exact_asec_and_nondefault_integer_both_channels() assert any("puf_tax_detail" in failure for failure in gate.failures) +def test_signal_gate_derives_legacy_asec_puf_roster_and_constraint_scope() -> None: + frame = _gate_frame() + summary = module.us_weeks_unemployed_summary(frame) + + assert summary["channel_roster"] == ["asec", "puf_tax_detail"] + assert summary["source_rows"] == 1_000 + assert summary["source_reconciliation_rows"] == 1_000 + assert summary["uc_constraint_rows"] == 2_000 + assert summary["uc_constraint_mismatch_count"] == 0 + + person = frame.table("person").copy() + first_puf_carrier = person.index[ + person["person_support_channel"].eq("puf_tax_detail") + & person[_OUTPUT].gt(0.0) + ][0] + person.loc[first_puf_carrier, "unemployment_compensation"] = 0.0 + gate = us_weeks_unemployed_signal_gate(module._replace_person_table(frame, person)) + + assert not gate.passed + assert any("unemployment-compensation constraint" in item for item in gate.failures) + + +def test_signal_gate_derives_stacked_asec_acs_roster_and_reviewed_scopes() -> None: + frame = _stacked_gate_frame() + gate = us_weeks_unemployed_signal_gate(frame) + summary = gate.details + + assert gate.passed + assert summary["channel_roster"] == ["asec", "acs"] + assert set(summary["channels"]) == {"asec", "acs"} + assert summary["source_rows"] == 1_000 + assert summary["source_reconciliation_rows"] == 500 + assert summary["source_invalid"] == 0 + assert summary["source_mismatch_count"] == 0 + assert summary["uc_constraint_rows"] == 1_500 + assert summary["uc_constraint_mismatch_count"] == 0 + + +def test_signal_gate_stacked_source_and_uc_checks_ignore_unowned_rows() -> None: + frame = _stacked_gate_frame() + person = frame.table("person").copy() + acs_clone = person["person_support_channel"].eq("acs") + person.loc[acs_clone, "LKWEEKS"] = 99.0 + ignored = module._replace_person_table(frame, person) + assert us_weeks_unemployed_signal_gate(ignored).passed + + asec_clone_one = person["person_support_channel"].eq("asec") & person[ + "person_support_clone_index" + ].eq(1) + person.loc[person.index[asec_clone_one][0], "LKWEEKS"] = 99.0 + invalid_source = us_weeks_unemployed_signal_gate( + module._replace_person_table(frame, person) + ) + assert not invalid_source.passed + assert invalid_source.details["source_invalid"] == 1 + + person = frame.table("person").copy() + acs_native = person["person_support_channel"].eq("acs") & person[ + "person_support_clone_index" + ].eq(0) + person.loc[person.index[acs_native][0], _OUTPUT] = 1.0 + uc_mismatch = us_weeks_unemployed_signal_gate( + module._replace_person_table(frame, person) + ) + assert not uc_mismatch.passed + assert uc_mismatch.details["uc_constraint_mismatch_count"] == 1 + + def test_signal_gate_rejects_collapsed_puf_share_and_weighted_weeks() -> None: frame = _gate_frame() person = frame.table("person").copy() diff --git a/packages/microcosm-build/tests/test_us_wic_claim.py b/packages/microcosm-build/tests/test_us_wic_claim.py index 1375153fe..3b307e063 100644 --- a/packages/microcosm-build/tests/test_us_wic_claim.py +++ b/packages/microcosm-build/tests/test_us_wic_claim.py @@ -47,6 +47,7 @@ RESTORED_REFERENCE_ECPS_REQUIRED_INPUTS, ) from microcosm.build.us_runtime.source_runtime import us_source_operation_handlers +from microcosm.build.us_runtime.wic_claim import _stable_person_keys from microcosm.frame import US_SCHEMA, EntitySchema, Frame, WeightKind, Weights policyengine_us_installed = importlib.util.find_spec("policyengine_us") is not None @@ -333,6 +334,43 @@ def test_draws_are_reproducible_and_keyed_by_source_identity(self) -> None: _derive(many, seed=2)[_OUTPUT].to_numpy(), ) + def test_multispine_identity_precedes_source_local_identity(self) -> None: + person = _frame( + [ + { + "person_source_id": 10, + "person_spine_source_id": 1, + "source_year": 2024, + "source_household_id": 50, + "source_person_id": 7, + }, + { + "person_source_id": 20, + "person_spine_source_id": 2, + "source_year": 2024, + "source_household_id": 50, + "source_person_id": 7, + }, + { + "person_source_id": 10, + "person_spine_source_id": 1, + "source_year": 2024, + "source_household_id": 50, + "source_person_id": 7, + }, + ] + ).table("person") + + assert _stable_person_keys(person).tolist() == [ + "source:10", + "source:20", + "source:10", + ] + + person.loc[person.index[0], "person_source_id"] = np.nan + with pytest.raises(SourceRuntimeError, match="assembly source identity"): + _stable_person_keys(person) + @pytest.mark.parametrize("column", US_WIC_CLAIM_REQUIRED_SOURCE_COLUMNS) def test_missing_source_columns_fail_closed(self, column: str) -> None: person = _frame([{}]).table("person").drop(columns=[column]) diff --git a/packages/microcosm-build/tests/test_us_workers_compensation.py b/packages/microcosm-build/tests/test_us_workers_compensation.py index 1e0fdabe9..786e28614 100644 --- a/packages/microcosm-build/tests/test_us_workers_compensation.py +++ b/packages/microcosm-build/tests/test_us_workers_compensation.py @@ -532,6 +532,77 @@ def test_signal_gate_rejects_either_dead_support_channel(dead_channel: str) -> N assert any(dead_channel in failure for failure in gate.failures) +def _stacked_workers_compensation_frame() -> Frame: + direct = with_us_workers_compensation(_frame(), seed=0, time_period=2024) + expanded = clone_us_frame_for_puf_support(direct) + tables = {entity: expanded.table(entity).copy() for entity in expanded.entities} + person = tables["person"] + physical_asec = person["person_source_id"].le(50) + person["person_spine_source_id"] = person["person_source_id"] + person["person_support_channel"] = np.where(physical_asec, "asec", "acs") + person.loc[~physical_asec, "WC_VAL"] = np.nan + return Frame( + tables, + expanded.schema, + { + entity: expanded.weights_for(entity) + for entity in expanded.weighted_entities + }, + expanded.strata, + mass_log=expanded.mass_log, + metadata=expanded.metadata, + ) + + +def test_signal_gate_reconciles_physical_asec_rows_in_stacked_pool() -> None: + frame = _stacked_workers_compensation_frame() + + gate = us_workers_compensation_signal_gate(frame) + + assert gate.passed, gate.failures + assert gate.details["source_invalid"] == 0 + assert gate.details["source_mismatch_count"] == 0 + + person = frame.table("person") + asec_clone = person["person_support_channel"].eq("asec") & person[ + "person_support_clone_index" + ].eq(1) + person.loc[asec_clone.idxmax(), _OUTPUT] = 1.0 + transferred_clone = us_workers_compensation_signal_gate(frame) + assert transferred_clone.passed, transferred_clone.failures + assert transferred_clone.details["source_mismatch_count"] == 0 + + asec_native = person["person_support_channel"].eq("asec") & person[ + "person_support_clone_index" + ].eq(0) + person.loc[asec_native.idxmax(), _OUTPUT] = 1.0 + mismatch = us_workers_compensation_signal_gate(frame) + assert not mismatch.passed + assert mismatch.details["source_mismatch_count"] == 1 + + person.loc[asec_native.idxmax(), _OUTPUT] = person.loc[ + asec_native.idxmax(), "WC_VAL" + ] + person.loc[asec_clone.idxmax(), "WC_VAL"] = np.nan + invalid_source = us_workers_compensation_signal_gate(frame) + assert not invalid_source.passed + assert invalid_source.details["source_invalid"] == 1 + + +def test_signal_gate_preserves_legacy_asec_puf_source_scope() -> None: + direct = with_us_workers_compensation(_frame(), seed=0, time_period=2024) + legacy = clone_us_frame_for_puf_support(direct) + person = legacy.table("person") + puf = person["person_support_channel"].eq("puf_tax_detail") + person.loc[puf, "WC_VAL"] = np.nan + + gate = us_workers_compensation_signal_gate(legacy) + + assert gate.passed, gate.failures + assert gate.details["source_invalid"] == 0 + assert gate.details["source_mismatch_count"] == 0 + + @requires_us def test_policyengine_us_1_819_0_contract_and_positive_annual_behavior() -> None: from policyengine_us import CountryTaxBenefitSystem, Simulation diff --git a/tools/build_us_fiscal_refresh_release.py b/tools/build_us_fiscal_refresh_release.py index df0634b20..8c1ed2c75 100644 --- a/tools/build_us_fiscal_refresh_release.py +++ b/tools/build_us_fiscal_refresh_release.py @@ -209,6 +209,9 @@ write_us_ssi_take_up_diagnostics, write_us_take_up_participation_diagnostics, ) +from microcosm.build.us_runtime.acs_release_predictors import ( + join_acs_release_predictors, +) from microcosm.build.us_runtime.demographics import ( CENSUS_NATIONAL_AGE_BENCHMARK, demographics_payload, @@ -1287,6 +1290,32 @@ def _parse_args(argv: Sequence[str] | None = None) -> argparse.Namespace: "official Census archive is fetched and verified." ), ) + parser.add_argument( + "--acs-person-zip", + type=Path, + help=( + "Local canonical 2024 one-year ACS PUMS person CSV zip used to " + "populate release-model predictors on physical ACS rows. Requires " + "--acs-person-sha256 and both household archive options." + ), + ) + parser.add_argument( + "--acs-person-sha256", + help="Expected SHA-256 of --acs-person-zip.", + ) + parser.add_argument( + "--acs-household-zip", + type=Path, + help=( + "Local canonical 2024 one-year ACS PUMS household CSV zip used by " + "the release predictor join. Requires the other three ACS archive " + "options." + ), + ) + parser.add_argument( + "--acs-household-sha256", + help="Expected SHA-256 of --acs-household-zip.", + ) parser.add_argument( "--scf-summary-extract", dest="scf_summary_extract", @@ -1555,6 +1584,30 @@ def _parse_args(argv: Sequence[str] | None = None) -> argparse.Namespace: parser.error("--evidence-failure-owners requires --evidence-release.") if args.allow_gate_failed_base_pool and args.base_h5 is None: parser.error("--allow-gate-failed-base-pool requires --base-h5.") + acs_archive_options = ( + args.acs_person_zip, + args.acs_person_sha256, + args.acs_household_zip, + args.acs_household_sha256, + ) + if any(value is not None for value in acs_archive_options) and any( + value is None for value in acs_archive_options + ): + parser.error( + "--acs-person-zip, --acs-person-sha256, --acs-household-zip, and " + "--acs-household-sha256 must be provided together." + ) + for flag, value in ( + ("--acs-person-sha256", args.acs_person_sha256), + ("--acs-household-sha256", args.acs_household_sha256), + ): + if value is not None and ( + len(value) != 64 + or any(character not in "0123456789abcdef" for character in value) + ): + parser.error( + f"{flag} must be exactly 64 lowercase hexadecimal characters." + ) if args.evidence_release and args.exact_k is not None: parser.error( "--evidence-release is incompatible with --exact-k: ladder " @@ -7514,6 +7567,7 @@ def _build_manifests( calibration_filename: str = CALIBRATION_FILENAME, exact_k_ladder: Mapping[str, object] | None = None, base_pool: Mapping[str, object] | None = None, + acs_predictor_join: Mapping[str, object] | None = None, evidence_known_failures: Sequence[Mapping[str, str]] | None = None, ) -> None: dataset_path = artifact_root / dataset_filename @@ -7581,6 +7635,11 @@ def _build_manifests( else {} ), **({"base_pool": dict(base_pool)} if base_pool is not None else {}), + **( + {"acs_predictor_join": dict(acs_predictor_join)} + if acs_predictor_join is not None + else {} + ), "dataset": { "filename": dataset_filename, "sha256": dataset_sha, @@ -7805,6 +7864,11 @@ def _build_manifests( else {} ), **({"base_pool": dict(base_pool)} if base_pool is not None else {}), + **( + {"acs_predictor_join": dict(acs_predictor_join)} + if acs_predictor_join is not None + else {} + ), "warm_start_calibration": warm_start_payload, "selection_source": selection_source_payload, "default_dataset": default_dataset_payload, @@ -9761,6 +9825,25 @@ def _main(argv: Sequence[str] | None = None) -> None: for failure in snap_discretionary_exemption_gate.failures ) ) + acs_predictor_join_result = join_acs_release_predictors( + base_frame, + person_zip=args.acs_person_zip, + person_sha256=args.acs_person_sha256, + household_zip=args.acs_household_zip, + household_sha256=args.acs_household_sha256, + ) + base_frame = acs_predictor_join_result.frame + acs_predictor_join_receipt = dict(acs_predictor_join_result.receipt) + if telemetry is not None: + telemetry.stage( + "acs_predictor_join", + message=( + "Authenticated and joined ACS release predictors before all " + "six archived donor-model stages." + ), + enabled=bool(acs_predictor_join_receipt.get("enabled")), + join=dict(acs_predictor_join_receipt.get("join", {})), + ) if telemetry is not None: telemetry.stage( "scf_wealth_inputs", @@ -11792,6 +11875,7 @@ def _main(argv: Sequence[str] | None = None) -> None: calibration_filename=calibration_filename, exact_k_ladder=exact_k_ladder_provenance, base_pool=base_pool_receipt, + acs_predictor_join=acs_predictor_join_receipt, evidence_known_failures=evidence_known_failures, ) if telemetry is not None: diff --git a/tools/spec_engine_coverage.py b/tools/spec_engine_coverage.py index 7a7a1786b..3f5ae7e76 100644 --- a/tools/spec_engine_coverage.py +++ b/tools/spec_engine_coverage.py @@ -41,7 +41,7 @@ REPORT_SCHEMA_VERSION = 3 EXPECTED_POINTER_INVENTORY_SHA256 = ( - "bc4a948ab632191954600da8474c5b011f977a65e24c399d126f3dc4a79f23e5" + "3fc6b9480ea81b9635bd0db56e180c2daf32a5cd2006a70d350586c570f96754" ) DEFAULT_REPORT_PATH = ( Path(__file__).resolve().parents[1] diff --git a/tools/us_bundle_generation/imputation.py b/tools/us_bundle_generation/imputation.py index d80696bf4..42a11953a 100644 --- a/tools/us_bundle_generation/imputation.py +++ b/tools/us_bundle_generation/imputation.py @@ -2144,15 +2144,20 @@ def _assert_invariants( for requirement in node["inputs"] for receipt in requirement["tolerated_absence_receipts"] ] - if ( + graph_counts = ( input_count, authored_output_count, compiled_output_count, len(primary_node["outputs"]), late_authored_output_count, len(tolerated_receipts), - ) != (2742, 92, 227, 35, 0, 212): - raise RuntimeError("US producer graph input/output/absence counts changed.") + ) + expected_graph_counts = (2744, 92, 227, 35, 0, 213) + if graph_counts != expected_graph_counts: + raise RuntimeError( + "US producer graph input/output/absence counts changed: " + f"observed={graph_counts}, expected={expected_graph_counts}." + ) if len(set(tolerated_receipts)) != len(tolerated_receipts): raise RuntimeError("US producer graph absence receipt IDs are not unique.") waived = {