Skip to content

Add the Street Manager Open Data archive (streetworks.opendata.archive) - #17

Merged
KFergusonUK merged 1 commit into
mainfrom
opendata-archive
Oct 28, 2026
Merged

KFergusonUK merged 1 commit into
mainfrom
opendata-archive

Conversation

@KFergusonUK

Copy link
Copy Markdown
Owner

DfT publishes every Open Data event as public monthly zips per topic (permit from 2020/07, activity from 2020/09, section 58 from 2022/03). Probed live before building; the full record is in docs/findings/streetmanager-opendata-archive.md.

  • ArchiveClient: url(), available() (HEAD; 404 = not yet published), files() (the host's public S3 bucket listing), and download() - streamed to a .part file, resumed with Range/If-Range, and verified against the ETag, which is the file's MD5.
  • iter_events(): reads a zip or a folder of zips one member at a time, yielding EventNotification, with filters named after the live feed's SNS attributes (ha_org/promoter_org are SWA codes). Bad zips and members are skipped and counted on ReadStats. No de-duplication: none was found within or across months.
  • Registered as opendata_archive; documented in uk.md, including the sizes (a permit month is ~1 GB), the ~0.7 GB zipfile memory cost for a permit month, and a warning not to pass archive events to from_streetmanager(), which accepts them but drops dates, geometry, status and PAA classification.
  • Tests use real fixtures: two pages of the live listing and small zips of real members from each topic's 2026/09 file.

DfT publishes every Open Data event as public monthly zips per topic
(permit from 2020/07, activity from 2020/09, section 58 from 2022/03).
Probed live before building; the full record is in
docs/findings/streetmanager-opendata-archive.md.

- ArchiveClient: url(), available() (HEAD; 404 = not yet published),
  files() (the host's public S3 bucket listing), and download() -
  streamed to a .part file, resumed with Range/If-Range, and verified
  against the ETag, which is the file's MD5.
- iter_events(): reads a zip or a folder of zips one member at a time,
  yielding EventNotification, with filters named after the live feed's
  SNS attributes (ha_org/promoter_org are SWA codes). Bad zips and
  members are skipped and counted on ReadStats. No de-duplication:
  none was found within or across months.
- Registered as opendata_archive; documented in uk.md, including the
  sizes (a permit month is ~1 GB), the ~0.7 GB zipfile memory cost for
  a permit month, and a warning not to pass archive events to
  from_streetmanager(), which accepts them but drops dates, geometry,
  status and PAA classification.
- Tests use real fixtures: two pages of the live listing and small zips
  of real members from each topic's 2026/09 file.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@KFergusonUK
KFergusonUK merged commit 65a3861 into main Oct 28, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant