Repository navigation
Add the Street Manager Open Data archive (streetworks.opendata.archive) - #17
Merged
Merged
Conversation
DfT publishes every Open Data event as public monthly zips per topic (permit from 2020/07, activity from 2020/09, section 58 from 2022/03). Probed live before building; the full record is in docs/findings/streetmanager-opendata-archive.md. - ArchiveClient: url(), available() (HEAD; 404 = not yet published), files() (the host's public S3 bucket listing), and download() - streamed to a .part file, resumed with Range/If-Range, and verified against the ETag, which is the file's MD5. - iter_events(): reads a zip or a folder of zips one member at a time, yielding EventNotification, with filters named after the live feed's SNS attributes (ha_org/promoter_org are SWA codes). Bad zips and members are skipped and counted on ReadStats. No de-duplication: none was found within or across months. - Registered as opendata_archive; documented in uk.md, including the sizes (a permit month is ~1 GB), the ~0.7 GB zipfile memory cost for a permit month, and a warning not to pass archive events to from_streetmanager(), which accepts them but drops dates, geometry, status and PAA classification. - Tests use real fixtures: two pages of the live listing and small zips of real members from each topic's 2026/09 file. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
DfT publishes every Open Data event as public monthly zips per topic (permit from 2020/07, activity from 2020/09, section 58 from 2022/03). Probed live before building; the full record is in docs/findings/streetmanager-opendata-archive.md.