diff --git a/CHANGELOG.md b/CHANGELOG.md index 8b93c03..e6dc454 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,35 @@ ## [Unreleased] +### Added — Street Manager Open Data archive, every Open Data event as public monthly zips (2026-10-28) + +`streetworks.opendata.archive` +- DfT's archive of the Open Data feed: one zip per topic per month at + `opendata.manage-roadworks.service.gov.uk/{topic}/{yyyy}/{mm}.zip`. Permits from 2020/07, + activities from 2020/09, section 58s from 2022/03, no gaps, no credentials, OGL v3. A permit + month is about 1 GB (about 1.1 million events; about 71 GB in all), so nothing downloads + unless asked. Confirmed live before building - full record in + `docs/findings/streetmanager-opendata-archive.md`. +- `ArchiveClient`: `url()` (no I/O), `available()` (one `HEAD`; a month not yet published is a + 404), `files()` (the host's public S3 bucket listing, so what's published is listed rather + than guessed), and `download()` - streamed to a `.part` file and renamed once complete, + resumed with an HTTP `Range` (+ `If-Range`) request, and verified against the `ETag`, which + is the file's MD5. A resumed `.part` left from an older copy of the file is caught by that + check and the download restarted once. +- `iter_events()` reads a zip, or every zip under a folder, one member at a time, yielding the + existing `EventNotification`. Filters are named after the live feed's SNS message attributes + (`usrn`, `area`, `ha_org`, `promoter_org`, `activity_type`, plus `event_types`); `ha_org` and + `promoter_org` match SWA codes, as the live attributes do. Bad zips and members are skipped + and counted on a `ReadStats`, never raised. No de-duplication: none was found within or + across months. +- Registered as `opendata_archive` (`source_grade="register"`). Documented in + `docs/providers/uk.md` with the sizes, the publication timing (normally the 1st of the next + month; months cut at UK local midnight) and the memory cost: Python's `zipfile` holds about + 0.7 GB for a permit month's member list while reading it. +- Not built yet: folding events into the shared `Works` model, which is a derived view. +- Tests use real fixtures: two pages of the live bucket listing, and small zips of real + members (original names and bytes) from the 2026/09 files of each topic. + ## [0.11.0] - 2026-10-28 ### Removed — `WorkAPI.create_alteration` and `WorkAPI.assess_alteration` (breaking) (2026-10-28) diff --git a/docs/findings/streetmanager-opendata-archive.md b/docs/findings/streetmanager-opendata-archive.md new file mode 100644 index 0000000..6ff0092 --- /dev/null +++ b/docs/findings/streetmanager-opendata-archive.md @@ -0,0 +1,106 @@ +# Street Manager Open Data archive — live probe record (2026-10-28) + +What was checked against `https://opendata.manage-roadworks.service.gov.uk` +before `streetworks.opendata.archive` was built, and what each result decided. +Everything here was observed live on 2026-10-28; nothing is inferred from +documentation alone. + +## The host + +S3 (bucket `prod-sm-archived-notifications-48575862-eu-west-2`) behind +CloudFront. The DfT [Archived notifications](https://department-for-transport-streetmanager.github.io/street-manager-docs/archived-notifications) +page is a small Vue app that reads the bucket listing directly. + +| Probe | Result | Decided | +|---|---|---| +| `HEAD permit/2026/09.zip` | `200`, `Content-Type: application/zip`, `Content-Length: 1044935949`, `Accept-Ranges: bytes`, `ETag`, `Last-Modified` | Downloads stream and can resume | +| `HEAD` on a missing month (`permit/2026/12.zip`, `permit/2019/12.zip`, unknown topic) | `404` (not `403`) | `available()` treats 404 as "not published"; any other error raises | +| `GET /` and `GET /?list-type=2&prefix=...` | Public `ListBucketResult` (V1 and V2, with continuation tokens) | `files()` lists what's published rather than probing URLs | +| `Range: bytes=N-` | `206` | Resume works | +| `Range` + matching `If-Range: ` | `206`; with a stale `If-Range`, `200` and the whole file | Resume is guarded against the file changing mid-download | +| MD5 of the downloaded `activity/2026/09.zip` and `section_58/2026/09.zip` | Equal to their `ETag`s; no `ETag` in the listing has a multipart `-N` suffix | `download()` verifies against the `ETag` | + +## Coverage and publication + +The full listing is 203 keys, all `{topic}/{yyyy}/{mm}.zip`: + +| Topic | Months | Range | Gaps | Total | +|---|---|---|---|---| +| `permit` | 75 | 2020/07–2026/09 | none | 70.98 GB | +| `activity` | 73 | 2020/09–2026/09 | none | 0.48 GB | +| `section_58` | 55 | 2022/03–2026/09 | none | 0.11 GB | + +Each month normally appears on the 1st of the following month, 08:00–09:45 +UTC (2026/07, /08 and /09 all did); 2026/06 appeared on 21 July. All 34 +files for 2020–2021 months carry 2022 `LastModified` dates, so the archive +only began in 2022 and its first years were uploaded then. + +Months are cut at **UK local midnight**: the 2026/08 `section_58` file holds +one event at `2026-07-31T23:19:59Z` (00:19 BST on 1 August). + +## Contents + +- **One event per member, the live feed's envelope.** Every member in the + 2026/09 `activity` (16,531) and `section_58` (3,764) files, and in 183 + members sampled across the 2026/09 `permit` file, is a JSON object with + exactly `event_reference`, `event_type`, `event_time`, `object_type`, + `object_reference`, `version` and `object_data`. No unparseable members, no + folders. +- **Matches DfT's schema.** Every `object_data` key seen is in DfT's + [Event Notifier Message JSON Schema](https://department-for-transport-streetmanager.github.io/street-manager-docs/api-documentation/json/event-notifier-message.json), + and the event types seen are exactly the schema's 18 (11 permit/work, 3 + activity, 4 section 58). +- **Member names** follow + `{object_reference}_{event_type lowercased}_event_{yyyy-mm-dd}_{HH-MM-SS}-{n}.json`. + Two reasons the SDK doesn't parse them: permit references can contain + underscores (e.g. `..._1-01_permit_submitted_event_...`), so splitting is + ambiguous; and the trailing `{n}` isn't milliseconds but the minute and + second run together unpadded (`12-40-33-4033`, `11-15-09-159`). +- **Filter fields are strings.** `usrn`, `highway_authority_swa_code`, + `promoter_swa_code`, `area_name` and `activity_type` are strings in every + topic. Activity and section 58 events have no promoter fields; section 58 + events have no `activity_type`. +- **No personal data fields.** None of the `object_data` keys hold contact + details, so real members can be committed as fixtures. + +## Duplicates + +None found: no repeated `(object_reference, event_type, event_time)` and no +repeated `event_reference` within any of the 2026/08 and 2026/09 `activity` +and `section_58` files, and no overlap between the two months on either key. +The guard in `KFergusonUK/RemedialWorksVisits`'s `remedial_works.py` reads +loose `.json` files alongside zips in the same folder, so it protects against +an extracted member being counted twice - not against anything in DfT's +files. So the SDK doesn't de-duplicate; `event_reference` is unique if a +caller mixes sources. Not checked for `permit` across months (it would need +two 1 GB downloads). + +## The permit file, read over HTTP ranges + +`permit/2026/09.zip` was opened with `zipfile` through a range-backed file +object, fetching about 137 MB rather than 1 GB: + +- **ZIP64**, 1,081,996 members, a 125 MB central directory. +- **Memory:** `zipfile.ZipFile` held 723 MB after opening (865 MB peak), + measured with `tracemalloc`. This is the cost `iter_events()` documents for + a permit month. +- **Every member uses a data descriptor** (general-purpose flag bit 3), all + deflate. A constant-memory reader that walks local headers instead of the + central directory is possible, but it has to find each member's end by + decompressing it, since the local headers carry no sizes. Not built. + +## SNS filter attributes + +The live feed's `MessageAttributes` are `area`, `ha_org`, `activity_type`, +`promoter_org` and `usrn`. DfT's sample notification shows `ha_org` and +`promoter_org` carry SWA codes (`5990`, `STPR`), so `iter_events()`'s filters +of the same names match `highway_authority_swa_code` and `promoter_swa_code`, +not the organisation names. + +## Licence + +The archive host states none. The DfT docs page that hosts the archive browser +carries the site footer "All content is available under the Open Government +Licence v3.0, except where otherwise stated", and nothing on the page states +otherwise. The archive needs no registration, unlike the live SNS feed, whose +sign-up includes accepting terms of use. diff --git a/docs/providers/index.md b/docs/providers/index.md index a6d0719..f5695ba 100644 --- a/docs/providers/index.md +++ b/docs/providers/index.md @@ -70,7 +70,7 @@ Street Manager Credentials: Street Manager API account (email + password) from streetworks.streetmanager import StreetManagerClient -... (5 more — OS Open USRN, DataVIA, D-TRO, Street Manager Open Data, UK Police) +... (8 more — Street Manager Open Data, Street Manager Open Data archive, DataVIA, D-TRO, OS Open USRN, National Highways, TfL, UK Police) >>> DGTClient = get_provider("dgt") # the class, not an instance - constructors vary >>> with DGTClient() as dgt: @@ -115,6 +115,7 @@ client, documented in its own section in [`docs/providers/`](.). |---|---|---| | `streetworks.streetmanager` | [DfT Street Manager](https://department-for-transport-streetmanager.github.io/street-manager-docs/api-documentation/) — all nine APIs (Work, Reporting, Street Lookup, GeoJSON, Party, Data Export, Event, Sampling, Worklist), V6 & V7, sandbox & production | read + write | | `streetworks.opendata` | [Street Manager Open Data](https://department-for-transport-streetmanager.github.io/street-manager-docs/open-data/) — AWS SNS push notifications | receive | +| `streetworks.opendata.archive` | [Street Manager Open Data archive](https://department-for-transport-streetmanager.github.io/street-manager-docs/archived-notifications) — every Open Data event as public monthly zips: permits from 2020/07, activities from 2020/09, section 58s from 2022/03 (no credentials, confirmed live 2026-10-28). A permit month is about 1 GB, about 71 GB in all; nothing downloads unless asked. Licence OGL v3. See [`docs/providers/uk.md`](uk.md#street-manager-open-data-archive) | read | | `streetworks.datavia` | [Geoplace DataVIA](https://datavia.geoplace.co.uk/documentation) — full NSG layer catalogue over OGC WFS and WMS (rendered maps + feature info), Basic + OAuth2 | read | | `streetworks.dtro` | [DfT Digital Traffic Regulation Orders](https://d-tro.dft.gov.uk/api-documentation/) — the legal orders behind speed limits, closures and restrictions; integration & production | read + write | | `streetworks.srwr` | [Scottish Road Works Register](https://roadworks.scot/) — national register via Open Data CSV extracts (no credentials) | read | diff --git a/docs/providers/uk.md b/docs/providers/uk.md index c6dd27d..9206bc6 100644 --- a/docs/providers/uk.md +++ b/docs/providers/uk.md @@ -150,6 +150,78 @@ complete FastAPI receiver. > credentials, though the messages, once flowing, are received the same > credential-free way. This module handles the receiving side of both. +## Street Manager Open Data archive + +DfT also publishes every Open Data event as a public archive, linked from its +[Archived notifications](https://department-for-transport-streetmanager.github.io/street-manager-docs/archived-notifications) +page: one zip per topic per month. No credentials or registration; licence +OGL v3 (the docs site footer on that page). `streetworks.opendata.archive` +lists, downloads and reads it: + +```python +from streetworks.opendata.archive import ArchiveClient, ReadStats, Topic, iter_events + +with ArchiveClient() as arc: + arc.files(Topic.ACTIVITY) # every published month, from the bucket listing + arc.available(Topic.PERMIT, 2026, 10) # one HEAD; False until it's published + arc.download(Topic.PERMIT, 2026, 9, "sm/permit/2026-09.zip") # ~1 GB, streamed to disk + +stats = ReadStats() +for event in iter_events("sm/", event_types={"PERMIT_SUBMITTED"}, ha_org="1355", stats=stats): + print(event.object_reference, event.object_data["activity_type"]) +print(stats) # zips read, members, events yielded, and anything skipped +``` + +Know the sizes before you download. Confirmed live 2026-10-28: + +| Topic | From | Per month | Events per month | +|---|---|---|---| +| `permit` | 2020/07 | 0.85–1.2 GB | about 1.1 million | +| `activity` | 2020/09 | 10–25 MB | about 15,000 | +| `section_58` | 2022/03 | 2–4 MB | about 3,500 | + +The whole permit archive is about 71 GB. Nothing downloads unless you call +`download()`, one month at a time. Each month normally appears on the 1st of +the following month, and months are cut at UK local midnight, not UTC. + +- **Downloads resume and are verified.** Bytes go to a `.part` file that is + renamed only once complete; running `download()` again continues it with an + HTTP range request. The finished file is checked against the server's + `ETag`, which is the file's MD5. +- **Each event is the live feed's message**, as a + `streetworks.opendata.EventNotification` - `event_type`, `event_time`, + `object_reference` and `object_data`, the same fields DfT's Event Notifier + Message schema documents. +- **Filters are named after the live feed's SNS message attributes**: `usrn`, + `area`, `ha_org` and `promoter_org` (SWA codes, not organisation names) and + `activity_type`, plus `event_types`. Each takes one value or a collection. +- **Bad files are skipped, not fatal.** A zip that won't open or a member + that isn't an event is counted on `ReadStats` and skipped. +- **Nothing is de-duplicated.** No duplicate events were found within a month + or across neighbouring months; `event_reference` is unique if you combine + overlapping sources yourself. +- **Memory.** Reading a permit month holds about 0.7 GB (peaking near 0.9 GB) + for Python's `zipfile` member list until that zip is done. Activity and + section 58 months are small. + +The archive is events, not works: one permit appears as several events +(submitted, granted, work start, work stop...). Turning those into the shared +`Works` model means folding events per reference, which this SDK doesn't do +for you yet. + +> **Don't pass archive events to `from_streetmanager()`.** A permit event's +> `object_data` looks like a Reporting permit row, and the converter accepts +> it without error, but the field names differ and the results are wrong: +> actual start/end dates are dropped (the archive calls them +> `actual_start_date_time`/`actual_end_date_time`) and the dates graded +> "estimated"; the geometry is dropped (the archive's +> `works_location_coordinates` is WKT text in British National Grid, not +> GeoJSON); status and works type come out empty; and PAAs become sites +> instead of plannings (the archive's `work_category` reads `"Major (PAA)"`; +> the code `paa` is in `work_category_ref`). Each event would also become its +> own record, so one permit appears several times. Read `object_data` +> directly until the archive has a converter of its own. + ## Transport for London (TfL Road Disruption) London roadworks are already in this SDK via Street Manager (the diff --git a/src/streetworks/opendata/archive.py b/src/streetworks/opendata/archive.py new file mode 100644 index 0000000..86bc2fd --- /dev/null +++ b/src/streetworks/opendata/archive.py @@ -0,0 +1,368 @@ +"""Street Manager Open Data archive - every Open Data event, as monthly zips. + +DfT publishes the events behind the Street Manager Open Data feed (the same +messages the SNS feed pushes - see :mod:`streetworks.opendata`) as a public +archive: one zip per topic per month, at +``https://opendata.manage-roadworks.service.gov.uk/{topic}/{yyyy}/{mm}.zip``, +linked from DfT's "Archived notifications" page. No credentials or +registration. Licence: Open Government Licence v3.0, per the footer of the +DfT docs page that hosts the archive browser (the archive host itself states +none). + +Confirmed live 2026-10-28, from the bucket listing, HEAD/GET requests and real +files: + +* **Three topics**, no gaps: ``permit`` from 2020/07, ``activity`` from + 2020/09, ``section_58`` from 2022/03. +* **Published monthly**, normally on the 1st of the following month (June + 2026 came on 21 July). Months are cut at UK local midnight, not UTC: an + event at ``2026-07-31T23:19Z`` (00:19 BST) is in the August file. +* **Sizes.** ``permit`` is about 0.85-1.2 GB a month (about 1.1 million + events) and about 71 GB for the whole archive; ``activity`` is 10-25 MB a + month; ``section_58`` 2-4 MB. +* **One event per zip member**, named + ``{object_reference}_{event_type}_event_{date}_{time}-{n}.json``; each is + the envelope :class:`~streetworks.opendata.EventNotification` models, and + every ``object_data`` field seen matches DfT's Event Notifier Message + schema. Names aren't parsed here: permit references can contain + underscores, so the JSON is the only reliable source. +* **No duplicates seen** - none within a month, and none across Aug/Sep 2026 + for ``activity`` and ``section_58`` (by ``event_reference``, or by + reference, event type and time). So nothing is de-duplicated. +* **The host is S3 behind CloudFront**: a public bucket listing (so + :meth:`ArchiveClient.files` lists what's published rather than guessing), + 404 for a month not yet published, ``Accept-Ranges: bytes`` (so + :meth:`ArchiveClient.download` resumes), and an ``ETag`` that is the file's + MD5 (so downloads are verified). + +Nothing here downloads anything unless asked: :meth:`ArchiveClient.download` +streams one file to disk, and :func:`iter_events` reads zips already on disk. +""" + +from __future__ import annotations + +import hashlib +import json +import re +import zipfile +import zlib +from collections.abc import Iterable, Iterator +from dataclasses import dataclass +from enum import Enum +from pathlib import Path +from xml.etree import ElementTree + +import httpx +from pydantic import ValidationError + +from .._transport import RetryConfig, SyncTransport, _raise_for_response +from ..exceptions import NotFoundError, TransportError +from .models import EventNotification + +__all__ = [ + "BASE_URL", + "ArchiveClient", + "ArchiveFile", + "ReadStats", + "Topic", + "iter_events", +] + +BASE_URL = "https://opendata.manage-roadworks.service.gov.uk" + +_S3 = {"s3": "http://s3.amazonaws.com/doc/2006-03-01/"} +_KEY = re.compile(r"^(?P[a-z0-9_]+)/(?P\d{4})/(?P\d{2})\.zip$") +_MD5_ETAG = re.compile(r'^"?(?P[0-9a-f]{32})"?$') + + +class Topic(str, Enum): + PERMIT = "permit" + ACTIVITY = "activity" + SECTION_58 = "section_58" + + +_TOPICS = frozenset(t.value for t in Topic) + + +@dataclass(frozen=True) +class ArchiveFile: + """One published month, as the bucket listing describes it.""" + + topic: Topic + year: int + month: int + url: str + size: int + last_modified: str + etag: str + + +class ArchiveClient: + """List and download the Street Manager Open Data archive. No + credentials required. + + >>> from streetworks.opendata.archive import ArchiveClient, Topic + >>> with ArchiveClient() as arc: + ... arc.available(Topic.PERMIT, 2026, 9) + ... arc.download(Topic.PERMIT, 2026, 9, "sm/permit-2026-09.zip") # ~1 GB + """ + + def __init__( + self, + *, + base_url: str = BASE_URL, + retry: RetryConfig | None = None, + timeout: float = 600.0, + client: httpx.Client | None = None, + ): + self.base_url = base_url.rstrip("/") + self._client = client or httpx.Client(timeout=timeout, follow_redirects=True) + self._transport = SyncTransport( + retry=retry or RetryConfig(), timeout=timeout, client=self._client + ) + + def url(self, topic: Topic | str, year: int, month: int) -> str: + """The URL a month's zip is (or will be) published at. No I/O.""" + if not 1 <= month <= 12: + raise ValueError(f"month must be 1-12, got {month}") + return f"{self.base_url}/{Topic(topic).value}/{year:04d}/{month:02d}.zip" + + def available(self, topic: Topic | str, year: int, month: int) -> bool: + """Whether a month has been published yet (one ``HEAD``; a month not + yet published is a 404).""" + try: + self._transport.request("HEAD", self.url(topic, year, month)) + except NotFoundError: + return False + return True + + def files(self, topic: Topic | str | None = None) -> list[ArchiveFile]: + """Every published month, from the public bucket listing, by topic + and oldest first - optionally for one topic. Keys that aren't a + ``{topic}/{yyyy}/{mm}.zip`` for a known topic are left out.""" + params = {"list-type": "2"} + if topic is not None: + params["prefix"] = f"{Topic(topic).value}/" + found: list[ArchiveFile] = [] + while True: + root = ElementTree.fromstring( + self._transport.request("GET", f"{self.base_url}/", params=params).content + ) + for entry in root.findall("s3:Contents", _S3): + key = entry.findtext("s3:Key", "", _S3) + match = _KEY.match(key) + if not match or match["topic"] not in _TOPICS: + continue + found.append( + ArchiveFile( + topic=Topic(match["topic"]), + year=int(match["year"]), + month=int(match["month"]), + url=f"{self.base_url}/{key}", + size=int(entry.findtext("s3:Size", "0", _S3)), + last_modified=entry.findtext("s3:LastModified", "", _S3), + etag=entry.findtext("s3:ETag", "", _S3), + ) + ) + token = root.findtext("s3:NextContinuationToken", None, _S3) + if root.findtext("s3:IsTruncated", "false", _S3) != "true" or not token: + break + params["continuation-token"] = token + return sorted(found, key=lambda f: (f.topic.value, f.year, f.month)) + + def download( + self, + topic: Topic | str, + year: int, + month: int, + dest: str | Path, + *, + resume: bool = True, + verify: bool = True, + ) -> Path: + """Stream one month's zip to ``dest`` and return its path. + + A permit month is about 1 GB; it is streamed in 1 MB chunks, never + held in memory. Bytes go to ``dest`` + ``.part`` and are renamed into + place only once complete, so an interrupted run never leaves a + truncated file that looks like a good zip. With ``resume``, a + leftover ``.part`` is continued with an HTTP ``Range`` request. With + ``verify``, the finished file's MD5 is checked against the ``ETag``; + if a resumed ``.part`` turns out to belong to an older copy of the + file, it is discarded and the download restarted once from scratch. + Raises :class:`~streetworks.exceptions.NotFoundError` for a month not + yet published, and :class:`~streetworks.exceptions.TransportError` if + the file doesn't match its ``ETag``. + """ + url = self.url(topic, year, month) + dest = Path(dest) + dest.parent.mkdir(parents=True, exist_ok=True) + part = dest.with_name(dest.name + ".part") + head = self._transport.request("HEAD", url) + size = int(head.headers["content-length"]) + etag = head.headers.get("etag", "") + md5 = _MD5_ETAG.match(etag) + + for attempt in (1, 2): + offset = part.stat().st_size if resume and part.exists() else 0 + if offset > size or (offset and not etag): + offset = 0 + if offset < size: + headers = {"Range": f"bytes={offset}-", "If-Range": etag} if offset else {} + with self._client.stream("GET", url, headers=headers) as response: + if response.status_code >= 400: + response.read() + _raise_for_response(response) + if response.status_code != 206: + offset = 0 # whole file sent: it changed, or the range was ignored + with open(part, "ab" if offset else "wb") as f: + for chunk in response.iter_bytes(chunk_size=1024 * 1024): + f.write(chunk) + if part.stat().st_size != size: + raise TransportError( + f"{url} downloaded {part.stat().st_size} of {size} bytes; " + f"run again to resume from {part}" + ) + if not verify or md5 is None or _md5(part) == md5["md5"]: + part.replace(dest) + return dest + part.unlink() + if not offset or attempt == 2: + raise TransportError(f"{url} does not match its ETag {etag} after download") + raise AssertionError("unreachable") # pragma: no cover + + def close(self) -> None: + self._transport.close() + + def __enter__(self) -> ArchiveClient: + return self + + def __exit__(self, *exc_info: object) -> None: + self.close() + + +def _md5(path: Path) -> str: + digest = hashlib.md5(usedforsecurity=False) + with open(path, "rb") as f: + for chunk in iter(lambda: f.read(1024 * 1024), b""): + digest.update(chunk) + return digest.hexdigest() + + +@dataclass +class ReadStats: + """Counts from one :func:`iter_events` pass - pass one in to see what was + read and what was skipped.""" + + zips: int = 0 + bad_zips: int = 0 + members: int = 0 + bad_members: int = 0 + events: int = 0 + + +# iter_events filter -> the object_data field it matches. The names are the +# SNS MessageAttributes the live feed can be filtered on; ha_org and +# promoter_org carry SWA codes, not organisation names. +_FILTER_FIELDS = { + "usrn": "usrn", + "area": "area_name", + "ha_org": "highway_authority_swa_code", + "promoter_org": "promoter_swa_code", + "activity_type": "activity_type", +} + +FilterValue = str | int | Iterable[str | int] | None + + +def _as_set(value: FilterValue) -> frozenset[str] | None: + if value is None: + return None + if isinstance(value, (str, int)): + return frozenset({str(value)}) + return frozenset(str(v) for v in value) + + +def iter_events( + source: str | Path, + *, + event_types: str | Iterable[str] | None = None, + usrn: FilterValue = None, + area: FilterValue = None, + ha_org: FilterValue = None, + promoter_org: FilterValue = None, + activity_type: FilterValue = None, + stats: ReadStats | None = None, +) -> Iterator[EventNotification]: + """Yield the events in an archive zip, or in every zip under a folder + (searched recursively, in name order), one member at a time. + + Every filter takes one value or a collection, and keeps an event whose + value matches any of them; filters combine with AND. ``event_types`` + matches ``event_type`` (e.g. ``"PERMIT_SUBMITTED"``); the others match + ``object_data`` and are named after the live feed's SNS + MessageAttributes: ``usrn``, ``area`` (``area_name``), ``ha_org`` + (``highway_authority_swa_code``), ``promoter_org`` + (``promoter_swa_code``) and ``activity_type``. An event without the + field doesn't match - activity and section 58 events carry no promoter, + and section 58 events no ``activity_type``. + + A zip that won't open, or a member that won't decompress or isn't an + event, is skipped and counted on ``stats`` - never raised. + + Memory: Python's ``zipfile`` reads a zip's whole member list on open. + For a permit month (about 1.1 million members) that list holds about + 0.7 GB, peaking near 0.9 GB while it is read, until that zip is + finished. Activity and section 58 months are small. + """ + stats = stats if stats is not None else ReadStats() + wanted_types = _as_set(event_types) + filters = { + "usrn": usrn, + "area": area, + "ha_org": ha_org, + "promoter_org": promoter_org, + "activity_type": activity_type, + } + wanted = { + _FILTER_FIELDS[name]: values + for name, value in filters.items() + if (values := _as_set(value)) is not None + } + source = Path(source) + zips = sorted(source.rglob("*.zip")) if source.is_dir() else [source] + for path in zips: + try: + archive = zipfile.ZipFile(path) + except (zipfile.BadZipFile, OSError): + stats.bad_zips += 1 + continue + stats.zips += 1 + with archive: + for info in archive.infolist(): + if info.is_dir(): + continue + stats.members += 1 + try: + raw = json.loads(archive.read(info)) + except (zipfile.BadZipFile, zlib.error, NotImplementedError, ValueError, OSError): + stats.bad_members += 1 + continue + if not isinstance(raw, dict): + stats.bad_members += 1 + continue + if wanted_types is not None and raw.get("event_type") not in wanted_types: + continue + data = raw.get("object_data") or {} + if any( + field not in data or str(data[field]) not in values + for field, values in wanted.items() + ): + continue + try: + event = EventNotification.model_validate(raw) + except ValidationError: + stats.bad_members += 1 + continue + stats.events += 1 + yield event diff --git a/src/streetworks/registry.py b/src/streetworks/registry.py index 414e0aa..6793a4b 100644 --- a/src/streetworks/registry.py +++ b/src/streetworks/registry.py @@ -509,6 +509,28 @@ def get_provider(key: str) -> Any: _client_name="handle", import_line="from streetworks.opendata import handle, parse_message", ), + ProviderEntry( + key="opendata_archive", + name="Street Manager Open Data archive", + description=( + "Every Street Manager Open Data event (permits since 2020, " + "activities, section 58s) as public monthly zips." + ), + kind=Kind.ROADWORKS, + network_scope=NetworkScope.COMPREHENSIVE, + territories=frozenset({"England"}), + scope_note=( + "Historical, not live: one zip per topic per month, normally " + "published on the 1st of the next month. A permit month is about " + "1 GB (about 71 GB since 2020/07); activity and section 58 months " + "are a few MB. Nothing downloads unless asked." + ), + licence="Open Government Licence v3.0 (OGL v3)", + source_grade="register", + _module="streetworks.opendata.archive", + _client_name="ArchiveClient", + import_line="from streetworks.opendata.archive import ArchiveClient, iter_events", + ), ProviderEntry( key="datavia", name="Geoplace DataVIA", diff --git a/tests/fixtures/streetmanager_archive_activity_2026-09_sample.zip b/tests/fixtures/streetmanager_archive_activity_2026-09_sample.zip new file mode 100644 index 0000000..3b6fcde Binary files /dev/null and b/tests/fixtures/streetmanager_archive_activity_2026-09_sample.zip differ diff --git a/tests/fixtures/streetmanager_archive_listing_page1.xml b/tests/fixtures/streetmanager_archive_listing_page1.xml new file mode 100644 index 0000000..5507759 --- /dev/null +++ b/tests/fixtures/streetmanager_archive_listing_page1.xml @@ -0,0 +1,2 @@ + +prod-sm-archived-notifications-48575862-eu-west-2section_58/1i9EeqEmy3IOJzovtZWYl+dG0hhEraV6Z9yX0Kv3L8rUmEb0taDkKHvLPgkpuv456+UYoaPYkkpe1V9GGPzeoSDpmuuJZt+CUxWKOupa3lK5W3NA0vkJc1A==22truesection_58/2022/03.zip2022-08-18T14:36:44.000Z"5f611da6a6da3741aa36ebbf6d682303"64360STANDARDsection_58/2022/04.zip2022-08-19T11:14:05.000Z"9954fc01153549a74741508e0c764d1f"395976STANDARD \ No newline at end of file diff --git a/tests/fixtures/streetmanager_archive_listing_page2.xml b/tests/fixtures/streetmanager_archive_listing_page2.xml new file mode 100644 index 0000000..894b0ae --- /dev/null +++ b/tests/fixtures/streetmanager_archive_listing_page2.xml @@ -0,0 +1,2 @@ + +prod-sm-archived-notifications-48575862-eu-west-2section_58/1i9EeqEmy3IOJzovtZWYl+dG0hhEraV6Z9yX0Kv3L8rUmEb0taDkKHvLPgkpuv456+UYoaPYkkpe1V9GGPzeoSDpmuuJZt+CUxWKOupa3lK5W3NA0vkJc1A==531000falsesection_58/2022/05.zip2022-08-19T13:40:31.000Z"635d2e164704fa0d71112ba96361fe53"795942STANDARDsection_58/2022/06.zip2022-08-22T08:26:50.000Z"d69105a1e67e704908973f0f25856e4a"955432STANDARDsection_58/2022/07.zip2022-08-22T12:38:22.000Z"11a02f63194a28b8e62f0b8fffa0fa72"855110STANDARDsection_58/2022/08.zip2022-09-01T14:37:24.000Z"38a4b8bde234c1d584a6313c1696446e"638085STANDARDsection_58/2022/09.zip2022-10-01T08:00:26.000Z"a02a0ac9e1ac5c9315985d4ad1c88ef7"905908STANDARDsection_58/2022/10.zip2022-11-01T08:00:14.000Z"04f39cdbdef8e3e91c08f89a3db39866"979529STANDARDsection_58/2022/11.zip2022-12-13T13:52:47.000Z"3c16e52f02952d48e35cbf29867b202f"1276461STANDARDsection_58/2022/12.zip2023-01-01T08:00:28.000Z"0733220a659f2c19f56e9ce8619ed34d"1003342STANDARDsection_58/2023/01.zip2023-02-01T08:00:19.000Z"1b7199bf98ab028c5cd3aef56f1ff2ad"1732880STANDARDsection_58/2023/02.zip2023-03-01T08:00:18.000Z"fdee666c322fb7ee0e289c83d4f3e2bc"1668259STANDARDsection_58/2023/03.zip2023-04-01T08:00:17.000Z"6b2ba7f6f95fb90b67dd0fa46980c9d3"1516461STANDARDsection_58/2023/04.zip2023-05-01T08:00:22.000Z"85cdebf5bd98313f6d3a81b262f7ff18"1722568STANDARDsection_58/2023/05.zip2023-06-01T08:02:51.000Z"7237d749b61ba3ebfffcd5f56f717ed8"1674571STANDARDsection_58/2023/06.zip2023-07-01T08:00:26.000Z"a5762ac05a9eff7e94f95771b6301439"1409692STANDARDsection_58/2023/07.zip2023-08-01T08:18:52.000Z"6226dabb1db103169ccb9dde8d5e19f7"1624383STANDARDsection_58/2023/08.zip2023-09-01T08:02:41.000Z"64b86a573824e48b17b64bf7263056bb"1835008STANDARDsection_58/2023/09.zip2023-10-01T08:26:41.000Z"14deeaf0a5d1bd35574eb69ddbd4fb9d"1540715STANDARDsection_58/2023/10.zip2023-11-01T08:00:18.000Z"34d500d77185e37b42effbc12bdd8976"1433044STANDARDsection_58/2023/11.zip2023-12-01T08:24:21.000Z"d3f8bda1ef9ee4f8799dbcd382600973"1712128STANDARDsection_58/2023/12.zip2024-01-01T08:00:17.000Z"a83642bba01d954adf694d3f16a1a9a2"1213710STANDARDsection_58/2024/01.zip2024-02-08T17:13:50.000Z"7a7d1bfd3d85496b05ea7b6177fd5146"1736704STANDARDsection_58/2024/02.zip2024-03-01T08:00:25.000Z"d272d00d585d0165af93e8955d52e1e4"1952791STANDARDsection_58/2024/03.zip2024-04-01T08:27:58.000Z"4f973ab6e678295b6122b4c3ef258d9b"2514303STANDARDsection_58/2024/04.zip2024-05-01T08:00:36.000Z"20133dd04befc8257eb36d7693a261b6"2009279STANDARDsection_58/2024/05.zip2024-06-01T08:15:24.000Z"be1759422a49d25b5e2f415d342e27f0"1748769STANDARDsection_58/2024/06.zip2024-07-01T08:00:29.000Z"ef4506d26bb12f42454f8b67ba454a9d"1382523STANDARDsection_58/2024/07.zip2024-08-01T08:04:12.000Z"280cdb781465bcf62231d81408017f11"2441216STANDARDsection_58/2024/08.zip2024-09-01T08:02:59.000Z"320a7c1b434c3fa85b38e8975e81b684"1876273STANDARDsection_58/2024/09.zip2024-10-01T08:00:21.000Z"2358b54561754d4123a24dae7092f1c0"1897912STANDARDsection_58/2024/10.zip2024-11-01T08:02:24.000Z"ab6306c6eac4f5d7fc586cae4aa4af1e"1923747STANDARDsection_58/2024/11.zip2024-12-01T08:02:22.000Z"6b57e8b312b123dfc4daab70da00a7f8"1925120STANDARDsection_58/2024/12.zip2025-01-01T08:00:18.000Z"667cc9b895aa58bb007918bac2f71106"1382907STANDARDsection_58/2025/01.zip2025-02-01T08:17:16.000Z"8e8250621b6f48ee46a6949a535a1708"2289664STANDARDsection_58/2025/02.zip2025-03-01T08:00:21.000Z"12d626fc5248de72960e45181efd21a4"CRC64NVMEFULL_OBJECT1907905STANDARDsection_58/2025/03.zip2025-04-01T08:00:32.000Z"b9eb4eea4d67ab79c82e35ba2e5676fc"CRC64NVMEFULL_OBJECT2392286STANDARDsection_58/2025/04.zip2025-05-01T08:02:28.000Z"ca221bea3491b855e4bde6bebbbcdb94"CRC64NVMEFULL_OBJECT2542347STANDARDsection_58/2025/05.zip2025-06-01T08:00:22.000Z"4ad01d99d8a390ff500c132587e0de58"CRC64NVMEFULL_OBJECT2630544STANDARDsection_58/2025/06.zip2025-07-01T09:33:03.000Z"f413c21707da5124709377a48d95559b"CRC64NVMEFULL_OBJECT2150400STANDARDsection_58/2025/07.zip2025-08-01T08:02:24.000Z"7f9540f8e23b2eb6bec4c8ab7f10d10f"CRC64NVMEFULL_OBJECT2527232STANDARDsection_58/2025/08.zip2025-09-01T08:00:31.000Z"3db52d6816712c60f50e7c663ecf16be"CRC64NVMEFULL_OBJECT2306048STANDARDsection_58/2025/09.zip2025-10-01T08:01:06.000Z"578e867db0f32578076395c8a0370127"CRC64NVMEFULL_OBJECT2264258STANDARDsection_58/2025/10.zip2025-11-01T08:00:25.000Z"5ee1d66cc179b147c36a6697f4a9defe"CRC64NVMEFULL_OBJECT2784146STANDARDsection_58/2025/11.zip2025-12-01T08:18:01.000Z"7c8d25dc0e925d98fe3934103d8f245d"CRC64NVMEFULL_OBJECT2574935STANDARDsection_58/2025/12.zip2026-01-01T08:10:31.000Z"618a8cf1a1be799456b4ffcf73c8c84e"CRC64NVMEFULL_OBJECT2815065STANDARDsection_58/2026/01.zip2026-02-01T10:08:02.000Z"3750a2624eb3142503b40da3f6288319"CRC64NVMEFULL_OBJECT2650695STANDARDsection_58/2026/02.zip2026-03-01T08:01:35.000Z"8b1c7aa834a2fe3e5053671defeeef8f"CRC64NVMEFULL_OBJECT2777875STANDARDsection_58/2026/03.zip2026-04-01T08:03:21.000Z"68c2a81a039fc3c89de83b26dc0d4ff8"CRC64NVMEFULL_OBJECT3491155STANDARDsection_58/2026/04.zip2026-05-01T08:00:35.000Z"ec9fb6bcf0c1c6500dd9743bfe90a36c"CRC64NVMEFULL_OBJECT3222340STANDARDsection_58/2026/05.zip2026-06-01T08:03:12.000Z"3cb420da3493b12a0fa922835165789e"CRC32FULL_OBJECT3417353STANDARDsection_58/2026/06.zip2026-07-21T07:19:37.000Z"3222b413f876160dacda5cda3334d211"CRC64NVMEFULL_OBJECT2268076STANDARDsection_58/2026/07.zip2026-08-01T08:00:35.000Z"c3fb28087836899d1efe039f0459c0b5"CRC32FULL_OBJECT4180984STANDARDsection_58/2026/08.zip2026-09-01T08:02:59.000Z"e93cc6cec8ddd2b440d2d266cd2b2165"CRC32FULL_OBJECT2695675STANDARDsection_58/2026/09.zip2026-10-01T08:00:36.000Z"5427bb520282cd020dbd5891d984e07b"CRC32FULL_OBJECT3089473STANDARD \ No newline at end of file diff --git a/tests/fixtures/streetmanager_archive_permit_2026-09_sample.zip b/tests/fixtures/streetmanager_archive_permit_2026-09_sample.zip new file mode 100644 index 0000000..cd8395d Binary files /dev/null and b/tests/fixtures/streetmanager_archive_permit_2026-09_sample.zip differ diff --git a/tests/fixtures/streetmanager_archive_section_58_2026-09_sample.zip b/tests/fixtures/streetmanager_archive_section_58_2026-09_sample.zip new file mode 100644 index 0000000..42fb298 Binary files /dev/null and b/tests/fixtures/streetmanager_archive_section_58_2026-09_sample.zip differ diff --git a/tests/test_opendata_archive.py b/tests/test_opendata_archive.py new file mode 100644 index 0000000..c532d63 --- /dev/null +++ b/tests/test_opendata_archive.py @@ -0,0 +1,226 @@ +"""Tests for streetworks.opendata.archive. + +Fixtures are real, cut 2026-10-28 from the live archive: two pages of the +public bucket listing (``prefix=section_58/``, the first page capped at two +keys), and small zips of real events - two per event type from the +2026/09 ``activity`` and ``section_58`` files, one per event type from the +2026/09 ``permit`` file (read over HTTP ranges, not downloaded whole). Member +names and bytes are DfT's; only the zip around them is rebuilt. +""" + +import hashlib +import shutil +import zipfile +from pathlib import Path + +import httpx +import pytest +import respx + +from streetworks.exceptions import NotFoundError, TransportError +from streetworks.opendata import EventNotification +from streetworks.opendata.archive import ( + BASE_URL, + ArchiveClient, + ReadStats, + Topic, + iter_events, +) + +FIXTURES = Path(__file__).parent / "fixtures" +PERMIT = FIXTURES / "streetmanager_archive_permit_2026-09_sample.zip" +ACTIVITY = FIXTURES / "streetmanager_archive_activity_2026-09_sample.zip" +SECTION_58 = FIXTURES / "streetmanager_archive_section_58_2026-09_sample.zip" +S58_URL = f"{BASE_URL}/section_58/2026/09.zip" + + +def _etag(content: bytes) -> str: + return f'"{hashlib.md5(content).hexdigest()}"' + + +def _head(content: bytes, etag: str | None = None) -> httpx.Response: + return httpx.Response( + 200, + headers={"content-length": str(len(content)), "etag": etag or _etag(content)}, + ) + + +# --- ArchiveClient ---------------------------------------------------------- + + +def test_url_is_built_without_io(): + with ArchiveClient() as arc: + assert arc.url(Topic.PERMIT, 2026, 9) == f"{BASE_URL}/permit/2026/09.zip" + assert arc.url("section_58", 2022, 3) == f"{BASE_URL}/section_58/2022/03.zip" + with pytest.raises(ValueError): + arc.url(Topic.PERMIT, 2026, 13) + + +@respx.mock +def test_available_is_false_for_an_unpublished_month(): + respx.head(f"{BASE_URL}/permit/2026/09.zip").mock(return_value=httpx.Response(200)) + respx.head(f"{BASE_URL}/permit/2026/12.zip").mock(return_value=httpx.Response(404)) + with ArchiveClient() as arc: + assert arc.available(Topic.PERMIT, 2026, 9) is True + assert arc.available(Topic.PERMIT, 2026, 12) is False + + +@respx.mock +def test_files_follows_the_listing_continuation_token(): + page1 = (FIXTURES / "streetmanager_archive_listing_page1.xml").read_bytes() + page2 = (FIXTURES / "streetmanager_archive_listing_page2.xml").read_bytes() + route = respx.get(f"{BASE_URL}/").mock( + side_effect=[httpx.Response(200, content=page1), httpx.Response(200, content=page2)] + ) + with ArchiveClient() as arc: + files = arc.files(Topic.SECTION_58) + + assert route.calls[0].request.url.params["prefix"] == "section_58/" + assert "continuation-token" not in route.calls[0].request.url.params + assert route.calls[1].request.url.params["continuation-token"].startswith("1i9EeqEmy3") + assert len(files) == 55 + first, last = files[0], files[-1] + assert (first.topic, first.year, first.month) == (Topic.SECTION_58, 2022, 3) + assert (last.year, last.month, last.size) == (2026, 9, 3089473) + assert last.etag == '"5427bb520282cd020dbd5891d984e07b"' + assert last.url == S58_URL + + +@respx.mock +def test_download_streams_to_dest_and_verifies_the_etag(tmp_path): + content = SECTION_58.read_bytes() + respx.head(S58_URL).mock(return_value=_head(content)) + respx.get(S58_URL).mock(return_value=httpx.Response(200, content=content)) + with ArchiveClient() as arc: + path = arc.download(Topic.SECTION_58, 2026, 9, tmp_path / "sm" / "s58.zip") + assert path.read_bytes() == content + assert not list(tmp_path.rglob("*.part")) + + +@respx.mock +def test_download_resumes_a_part_file_with_a_range_request(tmp_path): + content = SECTION_58.read_bytes() + dest = tmp_path / "s58.zip" + (tmp_path / "s58.zip.part").write_bytes(content[:1000]) + respx.head(S58_URL).mock(return_value=_head(content)) + route = respx.get(S58_URL).mock(return_value=httpx.Response(206, content=content[1000:])) + with ArchiveClient() as arc: + arc.download(Topic.SECTION_58, 2026, 9, dest) + assert route.calls[0].request.headers["range"] == "bytes=1000-" + assert route.calls[0].request.headers["if-range"] == _etag(content) + assert dest.read_bytes() == content + + +@respx.mock +def test_download_starts_over_when_the_server_sends_the_whole_file(tmp_path): + content = SECTION_58.read_bytes() + dest = tmp_path / "s58.zip" + (tmp_path / "s58.zip.part").write_bytes(b"x" * 1000) + respx.head(S58_URL).mock(return_value=_head(content)) + respx.get(S58_URL).mock(return_value=httpx.Response(200, content=content)) + with ArchiveClient() as arc: + arc.download(Topic.SECTION_58, 2026, 9, dest) + assert dest.read_bytes() == content + + +@respx.mock +def test_download_restarts_once_when_a_resumed_part_was_from_another_copy(tmp_path): + content = SECTION_58.read_bytes() + dest = tmp_path / "s58.zip" + (tmp_path / "s58.zip.part").write_bytes(b"x" * 1000) # same server, older file + respx.head(S58_URL).mock(return_value=_head(content)) + route = respx.get(S58_URL).mock( + side_effect=[ + httpx.Response(206, content=content[1000:]), + httpx.Response(200, content=content), + ] + ) + with ArchiveClient() as arc: + arc.download(Topic.SECTION_58, 2026, 9, dest) + assert "range" not in route.calls[1].request.headers + assert dest.read_bytes() == content + + +@respx.mock +def test_download_raises_and_keeps_nothing_when_a_fresh_file_fails_its_etag(tmp_path): + content = SECTION_58.read_bytes() + respx.head(S58_URL).mock(return_value=_head(content, etag='"' + "0" * 32 + '"')) + respx.get(S58_URL).mock(return_value=httpx.Response(200, content=content)) + with ArchiveClient() as arc, pytest.raises(TransportError): + arc.download(Topic.SECTION_58, 2026, 9, tmp_path / "s58.zip") + assert list(tmp_path.iterdir()) == [] + + +@respx.mock +def test_download_of_an_unpublished_month_raises_not_found(tmp_path): + respx.head(f"{BASE_URL}/permit/2026/12.zip").mock(return_value=httpx.Response(404)) + with ArchiveClient() as arc, pytest.raises(NotFoundError): + arc.download(Topic.PERMIT, 2026, 12, tmp_path / "p.zip") + + +# --- iter_events ------------------------------------------------------------- + + +def test_iter_events_yields_every_real_event_in_a_zip(): + stats = ReadStats() + events = list(iter_events(PERMIT, stats=stats)) + assert all(isinstance(e, EventNotification) for e in events) + assert {e.event_type for e in events} == { + "PERMIT_SUBMITTED", + "PERMIT_GRANTED", + "PERMIT_REFUSED", + "PERMIT_CANCELLED", + "PERMIT_REVOKED", + "PERMIT_ALTERATION_GRANTED", + "WORK_START", + "WORK_STOP", + "WORK_START_REVERTED", + "WORK_STOP_REVERTED", + "CURRENT_TRAFFIC_MANAGEMENT_UPDATED", + } + assert stats == ReadStats(zips=1, members=11, events=11) + assert all(e.object_type == "PERMIT" and e.object_data["usrn"] for e in events) + + +def test_iter_events_reads_every_zip_under_a_folder(tmp_path): + for path in (PERMIT, ACTIVITY, SECTION_58): + (tmp_path / path.stem).mkdir() + shutil.copy(path, tmp_path / path.stem / path.name) + stats = ReadStats() + events = list(iter_events(tmp_path, stats=stats)) + assert stats.zips == 3 + assert len(events) == 11 + 6 + 8 + assert {e.object_type for e in events} == {"PERMIT", "ACTIVITY", "SECTION_58"} + + +def test_iter_events_filters_by_event_type_and_object_data(): + events = list(iter_events(SECTION_58)) + target = events[0] + ha = target.object_data["highway_authority_swa_code"] + + assert {e.event_type for e in iter_events(SECTION_58, event_types="SECTION_58_CLOSED")} == { + "SECTION_58_CLOSED" + } + by_usrn = list(iter_events(SECTION_58, usrn=int(target.object_data["usrn"]))) + assert target.event_reference in {e.event_reference for e in by_usrn} + assert all(e.object_data["usrn"] == target.object_data["usrn"] for e in by_usrn) + assert all( + e.object_data["highway_authority_swa_code"] == ha + for e in iter_events(SECTION_58, ha_org=[ha, "no-such-code"]) + ) + # Section 58 events carry no promoter, so a promoter filter matches none. + assert list(iter_events(SECTION_58, promoter_org="anything")) == [] + + +def test_iter_events_skips_and_counts_bad_zips_and_members(tmp_path): + (tmp_path / "not-a-zip.zip").write_bytes(b"this is not a zip") + with zipfile.ZipFile(tmp_path / "mixed.zip", "w") as z: + with zipfile.ZipFile(ACTIVITY) as real: + first = real.namelist()[0] + z.writestr(first, real.read(first)) + z.writestr("truncated.json", '{"event_type": ') + z.writestr("list.json", "[]") + stats = ReadStats() + events = list(iter_events(tmp_path, stats=stats)) + assert len(events) == 1 + assert stats == ReadStats(zips=1, bad_zips=1, members=3, bad_members=2, events=1)