Skip to content

Add 2023-2026 UK facts and restore source vocabulary - #202

Merged
juaristi22 merged 6 commits into
chronicle-196-uk-targetsfrom
chronicle-200-uk-local-gaps
Aug 27, 2026
Merged

Add 2023-2026 UK facts and restore source vocabulary#202
juaristi22 merged 6 commits into
chronicle-196-uk-targetsfrom
chronicle-200-uk-local-gaps

Conversation

@juaristi22

@juaristi22 juaristi22 commented Aug 26, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • Expand the ONS PIPR area package to 14,656 monthly facts from January 2023 through June 2026 across local authorities, regions, countries, and Scottish BRMAs.
  • Add 1,189 DfC Northern Ireland Universal Credit facts from the May 2026 release, covering January 2023 through May 2026 for 18 PCON24 constituencies and 11 local government districts. This current release supersedes May 2025 and revises 583 of its 841 area-month values.
  • Add 152 DfT bus facts: BUS05i revenue/support for FY2023-25, NTS0705 local-bus trips for CY2023-24, and BUS0415 fare indices at all quarter ends from March 2023 through March 2026.
  • Add 214 HMRC CGT facts for tax year 2023-24 from Tables 3, 5, and 6: gain by taxable-income band, UK country and English region, and age.
  • Restore the stable DWP UC source vocabulary used by consumers and give SLGFS council-tax potential-yield measures distinct IDs from Scottish chargeable-dwelling stock measures.
  • Document source_concept and measure_id as consumer compatibility contracts requiring explicit migrations and regression tests.
  • Register all sources in the UK bundle without adding or changing Chronicle target profiles. Target selection, proxy transformations, period alignment, measurement bindings, and calibration remain Microcosm-owned.

Source coverage

Local facts

  • PIPR: 42 months, with 316 England and Wales local authorities in every month. The package emits all publisher-coded numeric rows; eight Northern Ireland BRMAs remain excluded because they share the placeholder code [z].
  • Northern Ireland UC: the May 2026 release publishes 41 months with 29 geographies per month. It republishes every area-month from May 2025, revises 583 of those 841 values, and adds 348 facts for June 2025 through May 2026. Chronicle keeps the current release only, so consumers receive one fact per area-month. DfC publishes claimants, not households or child-presence splits, so Chronicle emits person-level dfc_ni.uc_claimants and leaves household interpretation to Microcosm.

Consumer vocabulary compatibility

  • DWP children and family-type composition facts again expose source_concept: dwp.uc_households.
  • The deductions-derived national caseload series exposes the surviving dwp.uc_benefit_units / total_units selector.
  • SLGFS potential-yield measures now use potential_yield_band_a through potential_yield_band_h and potential_yield_total; the chargeable-dwelling stock package retains band_a through band_h and total.

Bus facts

  • dft-bus05i-revenue-support-2025: 24 facts from BUS05ai and BUS05bi across every numeric published geography and fiscal year 2023-2025.
  • dft-nts0705-local-bus-trips-2024: 24 facts for London-bus and other-local-bus trips per person across five real household-income quintiles plus all incomes in 2023 and 2024.
  • dft-bus0415-fares-index-2026: 104 facts across eight geographic series and 13 quarter ends from March 2023 through March 2026.
  • These sources reproduce uk-data's England anchors but do not publish exact UK fare-receipt or subsidy totals. DfT excludes Northern Ireland, and BUS04 operating revenue is not substituted for passenger receipts or net support.

Capital Gains Tax facts

  • hmrc-cgt-gain-by-income-2025: 142 numeric Table 3 facts across gain and taxable-income bands plus margins. [Fewer than 1] count cells remain guarded source cells and do not emit invented values.
  • hmrc-cgt-country-region-2025: 42 Table 5 taxpayer-count, gain, and liability facts across UK countries and English regions.
  • hmrc-cgt-age-2025: 30 Table 6 taxpayer-count, gain, and liability facts across nine age bands plus all ages.

Validation

  • Source-package validation and source-suite builds pass for all changed packages with full lineage coverage.
  • Focused source-package tests pass for the six new bus and CGT packages.
  • Facts-only tests pass; no Chronicle target profile changed.
  • Alias and UK bundle registry checks pass.
  • The full default bundle is valid with 187,718 facts from 158 packages, zero aggregate duplicate keys, and 143 expected semantic duplicate keys. These include independently published CGT totals and gain-band margins across HMRC Tables 1, 2.1a, 3, 5, and 6.
  • The full repository suite passes with 736 tests passed and one skipped.
  • Ruff and git diff --check pass.

Closes #200
Closes #203

@juaristi22 juaristi22 changed the title Add NI UC facts and 2025 PIPR local rents Add 2023-2026 PIPR rents and NI UC claimant facts Aug 26, 2026
@juaristi22 juaristi22 changed the title Add 2023-2026 PIPR rents and NI UC claimant facts Add 2023-2026 UK local, bus, and CGT facts Aug 26, 2026
@juaristi22
juaristi22 marked this pull request as ready for review August 26, 2026 19:51
@juaristi22
juaristi22 marked this pull request as draft August 27, 2026 09:07
@juaristi22 juaristi22 changed the title Add 2023-2026 UK local, bus, and CGT facts Add 2023-2026 UK facts and restore source vocabulary Aug 27, 2026
@juaristi22
juaristi22 marked this pull request as ready for review August 27, 2026 10:00
@vahid-ahmadi

Copy link
Copy Markdown

Automated review pass (Claude Code, high effort, diff only — no execution). 37 files, mostly generated package YAML.

Given policyengine-uk-data#468 is open and this PR adds local-area facts, I pointed the review specifically at that failure mode. It is not reproduced here, and the design actively prevents it — worth recording what was checked, since this is the pattern uk-data should be moving toward:

  • Every local-area fact carries a code. PIPR (14,656 rows), DfC NI (2,030 rows), CGT Table 5, BUS05i and BUS0415 all set geography_id on every row. The only code-less areas — NI BRMAs marked [z] — are explicitly excluded and documented.
  • No positional joins. Each row carries expected_row_header (the area code for PIPR, the area name for BUS/CGT) plus guard_cells pinning the reference month, and expected_column_header for column-keyed cuts. A reordered or partial re-extract fails the guard rather than silently permuting — which is precisely the property missing from the uk-data UC path.
  • Codes verified against canonical values. All 18 NI PCON24 codes map to the right names (N05000001 Belfast East … N05000018 West Tyrone); E12000001–09 correct; K02000001 / K03000001 / E92000001 / W92000004 / S92000003 / N92000002 correct. No code→name conflicts in any package.
  • Boundary vintage is explicit. Constituency facts declare geography_vintage: pcon_2024 on every row, so a consumer cannot mistake PCON24-native data for 2010 data and boundary-map it a second time — the compounding step in #468.
  • No duplicate record_set_id / value_id within any package; PIPR's selected_rows selector set matches the emitted (period, code) set exactly in both directions, 0 diff.
  • Manifests: source_url basename == filename == R2 key leaf, sha256 matches the key path, uri mirrors key, across all 7 new manifests. No repeat of the doubled-path defect from Add monthly Universal Credit element caseload sources #188.
  • Bands: CGT age bands 0-15…85+ contiguous with an open top band; Table 3 gain bands contiguous with an open gain_5000000_plus; NTS quintiles 5 + total. Units and aggregations sane (mean for per-person trips, ratio for the index, sum for £).

One finding

chronicle/bundle.py:40 — two NI UC packages overlap by 841 facts, and the collision is only a warning.

dfc-ni-uc-statistics-may-2025 and dfc-ni-uc-statistics-may-2026 are both added to UK_BUNDLE_SOURCES, and the 2026 package is a strict superset: Jan-2023 to May-2026 (1,189 facts) against Jan-2023 to May-2025 (841 facts), same concept dfc_ni.uc_claimants, same geographies, same periods, and even identical record_set_id strings.

Because build_source_release_key includes source_table / vintage, the aggregate keys differ, so the hard duplicate_aggregate_fact_key error never fires. All 841 overlapping facts land as duplicate_semantic_fact_key warnings only, and valid ignores warnings (bundle.py:336-344). A downstream consumer selecting by concept + geography + period therefore gets two claimant values per NI area-month for 29 of the 41 months — and DfC revises earlier months between editions, so the two are not guaranteed to agree.

The precedent for a deliberate overlap is the scotgov council-tax collection case, which docs/pe-uk-source-checklist.md documents as "by design, consumers pin source_table". Here the checklist row mentions only the may-2025 package and says nothing about the overlap or the pinning rule. Either drop dfc-ni-uc-statistics-may-2025 from the bundle — nothing in it is absent from the 2026 package — or document the pinning rule the way the scotgov row does, so a consumer knows the ambiguity exists and how to resolve it.

@juaristi22

Copy link
Copy Markdown
Collaborator Author

Addressed the DfC release-overlap review finding by removing the superseded May 2025 source package, alias, manifest, and workbook. The May 2026 package is now the sole registered UK-bundle source for this series; it republishes all 841 earlier area-months, revises 583 values, and adds 348 facts through May 2026. This preserves the existing exact UK alias/bundle drift guard and gives Microcosm one DfC fact per area-month without a vintage selector.

Validation: 736 passed, 1 skipped; focused bundle/alias/DfC tests passed; Ruff and git diff --check passed. No Chronicle target profiles changed.

@juaristi22
juaristi22 merged commit accfbde into chronicle-196-uk-targets Aug 27, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants