Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
29 changes: 29 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,35 @@

## [Unreleased]

### Added — Street Manager Open Data archive, every Open Data event as public monthly zips (2026-10-28)

`streetworks.opendata.archive`
- DfT's archive of the Open Data feed: one zip per topic per month at
`opendata.manage-roadworks.service.gov.uk/{topic}/{yyyy}/{mm}.zip`. Permits from 2020/07,
activities from 2020/09, section 58s from 2022/03, no gaps, no credentials, OGL v3. A permit
month is about 1 GB (about 1.1 million events; about 71 GB in all), so nothing downloads
unless asked. Confirmed live before building - full record in
`docs/findings/streetmanager-opendata-archive.md`.
- `ArchiveClient`: `url()` (no I/O), `available()` (one `HEAD`; a month not yet published is a
404), `files()` (the host's public S3 bucket listing, so what's published is listed rather
than guessed), and `download()` - streamed to a `.part` file and renamed once complete,
resumed with an HTTP `Range` (+ `If-Range`) request, and verified against the `ETag`, which
is the file's MD5. A resumed `.part` left from an older copy of the file is caught by that
check and the download restarted once.
- `iter_events()` reads a zip, or every zip under a folder, one member at a time, yielding the
existing `EventNotification`. Filters are named after the live feed's SNS message attributes
(`usrn`, `area`, `ha_org`, `promoter_org`, `activity_type`, plus `event_types`); `ha_org` and
`promoter_org` match SWA codes, as the live attributes do. Bad zips and members are skipped
and counted on a `ReadStats`, never raised. No de-duplication: none was found within or
across months.
- Registered as `opendata_archive` (`source_grade="register"`). Documented in
`docs/providers/uk.md` with the sizes, the publication timing (normally the 1st of the next
month; months cut at UK local midnight) and the memory cost: Python's `zipfile` holds about
0.7 GB for a permit month's member list while reading it.
- Not built yet: folding events into the shared `Works` model, which is a derived view.
- Tests use real fixtures: two pages of the live bucket listing, and small zips of real
members (original names and bytes) from the 2026/09 files of each topic.

## [0.11.0] - 2026-10-28

### Removed — `WorkAPI.create_alteration` and `WorkAPI.assess_alteration` (breaking) (2026-10-28)
Expand Down
106 changes: 106 additions & 0 deletions docs/findings/streetmanager-opendata-archive.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,106 @@
# Street Manager Open Data archive — live probe record (2026-10-28)

What was checked against `https://opendata.manage-roadworks.service.gov.uk`
before `streetworks.opendata.archive` was built, and what each result decided.
Everything here was observed live on 2026-10-28; nothing is inferred from
documentation alone.

## The host

S3 (bucket `prod-sm-archived-notifications-48575862-eu-west-2`) behind
CloudFront. The DfT [Archived notifications](https://department-for-transport-streetmanager.github.io/street-manager-docs/archived-notifications)
page is a small Vue app that reads the bucket listing directly.

| Probe | Result | Decided |
|---|---|---|
| `HEAD permit/2026/09.zip` | `200`, `Content-Type: application/zip`, `Content-Length: 1044935949`, `Accept-Ranges: bytes`, `ETag`, `Last-Modified` | Downloads stream and can resume |
| `HEAD` on a missing month (`permit/2026/12.zip`, `permit/2019/12.zip`, unknown topic) | `404` (not `403`) | `available()` treats 404 as "not published"; any other error raises |
| `GET /` and `GET /?list-type=2&prefix=...` | Public `ListBucketResult` (V1 and V2, with continuation tokens) | `files()` lists what's published rather than probing URLs |
| `Range: bytes=N-` | `206` | Resume works |
| `Range` + matching `If-Range: <etag>` | `206`; with a stale `If-Range`, `200` and the whole file | Resume is guarded against the file changing mid-download |
| MD5 of the downloaded `activity/2026/09.zip` and `section_58/2026/09.zip` | Equal to their `ETag`s; no `ETag` in the listing has a multipart `-N` suffix | `download()` verifies against the `ETag` |

## Coverage and publication

The full listing is 203 keys, all `{topic}/{yyyy}/{mm}.zip`:

| Topic | Months | Range | Gaps | Total |
|---|---|---|---|---|
| `permit` | 75 | 2020/07–2026/09 | none | 70.98 GB |
| `activity` | 73 | 2020/09–2026/09 | none | 0.48 GB |
| `section_58` | 55 | 2022/03–2026/09 | none | 0.11 GB |

Each month normally appears on the 1st of the following month, 08:00–09:45
UTC (2026/07, /08 and /09 all did); 2026/06 appeared on 21 July. All 34
files for 2020–2021 months carry 2022 `LastModified` dates, so the archive
only began in 2022 and its first years were uploaded then.

Months are cut at **UK local midnight**: the 2026/08 `section_58` file holds
one event at `2026-07-31T23:19:59Z` (00:19 BST on 1 August).

## Contents

- **One event per member, the live feed's envelope.** Every member in the
2026/09 `activity` (16,531) and `section_58` (3,764) files, and in 183
members sampled across the 2026/09 `permit` file, is a JSON object with
exactly `event_reference`, `event_type`, `event_time`, `object_type`,
`object_reference`, `version` and `object_data`. No unparseable members, no
folders.
- **Matches DfT's schema.** Every `object_data` key seen is in DfT's
[Event Notifier Message JSON Schema](https://department-for-transport-streetmanager.github.io/street-manager-docs/api-documentation/json/event-notifier-message.json),
and the event types seen are exactly the schema's 18 (11 permit/work, 3
activity, 4 section 58).
- **Member names** follow
`{object_reference}_{event_type lowercased}_event_{yyyy-mm-dd}_{HH-MM-SS}-{n}.json`.
Two reasons the SDK doesn't parse them: permit references can contain
underscores (e.g. `..._1-01_permit_submitted_event_...`), so splitting is
ambiguous; and the trailing `{n}` isn't milliseconds but the minute and
second run together unpadded (`12-40-33-4033`, `11-15-09-159`).
- **Filter fields are strings.** `usrn`, `highway_authority_swa_code`,
`promoter_swa_code`, `area_name` and `activity_type` are strings in every
topic. Activity and section 58 events have no promoter fields; section 58
events have no `activity_type`.
- **No personal data fields.** None of the `object_data` keys hold contact
details, so real members can be committed as fixtures.

## Duplicates

None found: no repeated `(object_reference, event_type, event_time)` and no
repeated `event_reference` within any of the 2026/08 and 2026/09 `activity`
and `section_58` files, and no overlap between the two months on either key.
The guard in `KFergusonUK/RemedialWorksVisits`'s `remedial_works.py` reads
loose `.json` files alongside zips in the same folder, so it protects against
an extracted member being counted twice - not against anything in DfT's
files. So the SDK doesn't de-duplicate; `event_reference` is unique if a
caller mixes sources. Not checked for `permit` across months (it would need
two 1 GB downloads).

## The permit file, read over HTTP ranges

`permit/2026/09.zip` was opened with `zipfile` through a range-backed file
object, fetching about 137 MB rather than 1 GB:

- **ZIP64**, 1,081,996 members, a 125 MB central directory.
- **Memory:** `zipfile.ZipFile` held 723 MB after opening (865 MB peak),
measured with `tracemalloc`. This is the cost `iter_events()` documents for
a permit month.
- **Every member uses a data descriptor** (general-purpose flag bit 3), all
deflate. A constant-memory reader that walks local headers instead of the
central directory is possible, but it has to find each member's end by
decompressing it, since the local headers carry no sizes. Not built.

## SNS filter attributes

The live feed's `MessageAttributes` are `area`, `ha_org`, `activity_type`,
`promoter_org` and `usrn`. DfT's sample notification shows `ha_org` and
`promoter_org` carry SWA codes (`5990`, `STPR`), so `iter_events()`'s filters
of the same names match `highway_authority_swa_code` and `promoter_swa_code`,
not the organisation names.

## Licence

The archive host states none. The DfT docs page that hosts the archive browser
carries the site footer "All content is available under the Open Government
Licence v3.0, except where otherwise stated", and nothing on the page states
otherwise. The archive needs no registration, unlike the live SNS feed, whose
sign-up includes accepting terms of use.
3 changes: 2 additions & 1 deletion docs/providers/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -70,7 +70,7 @@ Street Manager
Credentials: Street Manager API account (email + password)
from streetworks.streetmanager import StreetManagerClient

... (5 more — OS Open USRN, DataVIA, D-TRO, Street Manager Open Data, UK Police)
... (8 more — Street Manager Open Data, Street Manager Open Data archive, DataVIA, D-TRO, OS Open USRN, National Highways, TfL, UK Police)

>>> DGTClient = get_provider("dgt") # the class, not an instance - constructors vary
>>> with DGTClient() as dgt:
Expand Down Expand Up @@ -115,6 +115,7 @@ client, documented in its own section in [`docs/providers/`](.).
|---|---|---|
| `streetworks.streetmanager` | [DfT Street Manager](https://department-for-transport-streetmanager.github.io/street-manager-docs/api-documentation/) — all nine APIs (Work, Reporting, Street Lookup, GeoJSON, Party, Data Export, Event, Sampling, Worklist), V6 & V7, sandbox & production | read + write |
| `streetworks.opendata` | [Street Manager Open Data](https://department-for-transport-streetmanager.github.io/street-manager-docs/open-data/) — AWS SNS push notifications | receive |
| `streetworks.opendata.archive` | [Street Manager Open Data archive](https://department-for-transport-streetmanager.github.io/street-manager-docs/archived-notifications) — every Open Data event as public monthly zips: permits from 2020/07, activities from 2020/09, section 58s from 2022/03 (no credentials, confirmed live 2026-10-28). A permit month is about 1 GB, about 71 GB in all; nothing downloads unless asked. Licence OGL v3. See [`docs/providers/uk.md`](uk.md#street-manager-open-data-archive) | read |
| `streetworks.datavia` | [Geoplace DataVIA](https://datavia.geoplace.co.uk/documentation) — full NSG layer catalogue over OGC WFS and WMS (rendered maps + feature info), Basic + OAuth2 | read |
| `streetworks.dtro` | [DfT Digital Traffic Regulation Orders](https://d-tro.dft.gov.uk/api-documentation/) — the legal orders behind speed limits, closures and restrictions; integration & production | read + write |
| `streetworks.srwr` | [Scottish Road Works Register](https://roadworks.scot/) — national register via Open Data CSV extracts (no credentials) | read |
Expand Down
72 changes: 72 additions & 0 deletions docs/providers/uk.md
Original file line number Diff line number Diff line change
Expand Up @@ -150,6 +150,78 @@ complete FastAPI receiver.
> credentials, though the messages, once flowing, are received the same
> credential-free way. This module handles the receiving side of both.

## Street Manager Open Data archive

DfT also publishes every Open Data event as a public archive, linked from its
[Archived notifications](https://department-for-transport-streetmanager.github.io/street-manager-docs/archived-notifications)
page: one zip per topic per month. No credentials or registration; licence
OGL v3 (the docs site footer on that page). `streetworks.opendata.archive`
lists, downloads and reads it:

```python
from streetworks.opendata.archive import ArchiveClient, ReadStats, Topic, iter_events

with ArchiveClient() as arc:
arc.files(Topic.ACTIVITY) # every published month, from the bucket listing
arc.available(Topic.PERMIT, 2026, 10) # one HEAD; False until it's published
arc.download(Topic.PERMIT, 2026, 9, "sm/permit/2026-09.zip") # ~1 GB, streamed to disk

stats = ReadStats()
for event in iter_events("sm/", event_types={"PERMIT_SUBMITTED"}, ha_org="1355", stats=stats):
print(event.object_reference, event.object_data["activity_type"])
print(stats) # zips read, members, events yielded, and anything skipped
```

Know the sizes before you download. Confirmed live 2026-10-28:

| Topic | From | Per month | Events per month |
|---|---|---|---|
| `permit` | 2020/07 | 0.85–1.2 GB | about 1.1 million |
| `activity` | 2020/09 | 10–25 MB | about 15,000 |
| `section_58` | 2022/03 | 2–4 MB | about 3,500 |

The whole permit archive is about 71 GB. Nothing downloads unless you call
`download()`, one month at a time. Each month normally appears on the 1st of
the following month, and months are cut at UK local midnight, not UTC.

- **Downloads resume and are verified.** Bytes go to a `.part` file that is
renamed only once complete; running `download()` again continues it with an
HTTP range request. The finished file is checked against the server's
`ETag`, which is the file's MD5.
- **Each event is the live feed's message**, as a
`streetworks.opendata.EventNotification` - `event_type`, `event_time`,
`object_reference` and `object_data`, the same fields DfT's Event Notifier
Message schema documents.
- **Filters are named after the live feed's SNS message attributes**: `usrn`,
`area`, `ha_org` and `promoter_org` (SWA codes, not organisation names) and
`activity_type`, plus `event_types`. Each takes one value or a collection.
- **Bad files are skipped, not fatal.** A zip that won't open or a member
that isn't an event is counted on `ReadStats` and skipped.
- **Nothing is de-duplicated.** No duplicate events were found within a month
or across neighbouring months; `event_reference` is unique if you combine
overlapping sources yourself.
- **Memory.** Reading a permit month holds about 0.7 GB (peaking near 0.9 GB)
for Python's `zipfile` member list until that zip is done. Activity and
section 58 months are small.

The archive is events, not works: one permit appears as several events
(submitted, granted, work start, work stop...). Turning those into the shared
`Works` model means folding events per reference, which this SDK doesn't do
for you yet.

> **Don't pass archive events to `from_streetmanager()`.** A permit event's
> `object_data` looks like a Reporting permit row, and the converter accepts
> it without error, but the field names differ and the results are wrong:
> actual start/end dates are dropped (the archive calls them
> `actual_start_date_time`/`actual_end_date_time`) and the dates graded
> "estimated"; the geometry is dropped (the archive's
> `works_location_coordinates` is WKT text in British National Grid, not
> GeoJSON); status and works type come out empty; and PAAs become sites
> instead of plannings (the archive's `work_category` reads `"Major (PAA)"`;
> the code `paa` is in `work_category_ref`). Each event would also become its
> own record, so one permit appears several times. Read `object_data`
> directly until the archive has a converter of its own.

## Transport for London (TfL Road Disruption)

London roadworks are already in this SDK via Street Manager (the
Expand Down
Loading
Loading