Generate the compliance doc reproducibly, and drift-check it

[amc-corey-cox]

Closes #318.

make specification became reachable in #317, and this makes it worth running.

The doc was scraped from pytest's stdout

That's the root of everything #318 complained about. It's why the rule needed perl -pne 's@^\.#@#@' — progress dots collide with headings, 54 times in a current run, and the fixup only strips one dot so it would fail silently on ..#. It's also why the published document's last line is pytest's own summary:

. 55 passed, 2 skipped, 83 warnings in 4.60s

I ran the suite twice and diffed to find everything that varies. It was more than #318 recorded: the date, the absolute path, and the elapsed time and warning count in that summary line. Two of those move on every single run, so no amount of fixing the header would have made this drift-checkable.

So the suite now writes the report itself. The 25 emit sites buffer their fragments, and a module-scoped fixture writes the document when --compliance-out is given. Nothing is scraped, so the dots, the perl pass, and the summary line all go away together.

pandoc

Confirmed redundant rather than assumed. The windmill theme builds its own pageToc from the headings, and after this change it is byte-identical to before — same titles, same anchors:

{title: "Feature Set: test_map_types", url: "#feature-set-test_map_types" }

pandoc was only adding a second, in-body copy of that, plus setext headings and escaped underscores. It was also a system binary declared nowhere in the project, which made the one documented way to build this page depend on a tool we never mention.

Keeping the timestamp

Time_executed stays, but a timestamp that moves daily would make the drift check fail on every PR. So the writer only rewrites when the substance changes — it compares ignoring the stamp. The date now means "when this content last actually changed", which is more useful than "when someone last ran make". tests/test_compliance/test_report.py pins that rule down, and Package is repo-relative now instead of /Users/cjm/....

Drift check

Third one in main.yaml, mirroring the datamodel and gendoc checks. This is the payoff — the doc went two years without an update (#320) because nothing was watching.

Verification

Regenerating twice in a row leaves the tree clean, so the check passes. Stamping the file with 2001-01-01 and regenerating leaves it at 2001; changing the content advances it to today. Site builds, full suite passes (1106 passed, 4 skipped, 1 xfailed), lint and format clean.

Reviewing by commit is easier than by diff — the middle commit is the 1,600-line mechanical regeneration, the other two are the actual change.

Coverage of the suite itself is untouched here; that's #320.

[Copilot]

Pull request overview

This PR makes the compliance specification document reproducible by generating it directly from the compliance suite (instead of scraping pytest stdout), removes the pandoc dependency from the build, and adds a CI drift check to ensure the published compliance doc stays up to date.

Changes:

• Introduces a buffered report writer (tests/test_compliance/report.py) and switches the compliance suite to emit report fragments into it, writing docs/specification/compliance.md only when --compliance-out is provided.
• Updates the specification make target to generate the compliance doc via pytest + --compliance-out (no stdout scraping, no perl/pandoc).
• Adds a GitHub Actions drift-check step that regenerates the compliance doc on PRs and fails if it’s out of date.

Reviewed changes

Copilot reviewed 6 out of 7 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
tests/test_compliance/test_report.py	Adds tests pinning the “timestamp only changes when substance changes” rule.
tests/test_compliance/test_compliance_suite.py	Replaces stdout printing with buffered report.emit(...) and writes the report at module teardown when requested.
tests/test_compliance/report.py	New report buffering + render/write logic with timestamp-ignoring drift semantics.
tests/conftest.py	Registers --compliance-out pytest option used by the report writer.
project.Makefile	Simplifies specification target to run pytest with --compliance-out (removes perl/pandoc pipeline).
docs/specification/compliance.md	Regenerated compliance doc output in the new stable format (no pandoc TOC wrapper, no pytest progress/summary artifacts).
.github/workflows/main.yaml	Adds CI job step to regenerate the compliance doc and fail on drift.

[Copilot]

Pull request overview

Copilot reviewed 6 out of 7 changed files in this pull request and generated no new comments.

Suppressed comments (3)

tests/test_compliance/report.py:77

• For cross-platform reproducibility, avoid Path.write_text() here: it writes with OS-native newline translation (e.g. CRLF on Windows), which breaks the PR goal of byte-identical output across environments. Use an explicit newline="\n" (and ideally encoding="utf-8") when writing.

    if path.exists() and TIMESTAMP.sub("", path.read_text()) == TIMESTAMP.sub("", document):
        return False
    path.parent.mkdir(parents=True, exist_ok=True)
    path.write_text(document)
    return True

tests/test_compliance/report.py:82

• _fragments is never cleared after writing. In long-running processes (or repeated report generation in a single interpreter), this can cause duplicated content and retains memory unnecessarily. Clear the buffer after rendering/writing.

def write(path: Path, package: str) -> bool:
    """Write the accumulated report to ``path``, stamped with today's date."""
    return write_if_changed(path, render(package, date.today().strftime("%Y-%m-%d")))

.github/workflows/main.yaml:92

• This drift check only looks at git diff, so it will miss untracked generated files under docs/specification/. The adjacent schema-doc drift check handles this by also checking git ls-files --others --exclude-standard; mirroring that here makes the check more robust and consistent.

          if ! git diff --quiet docs/specification/; then

[Copilot]

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

[amc-corey-cox]

One more piece of verification while Actions is down: the drift check runs only on 3.12, so I checked the report is byte-stable across the support matrix. Generated it under 3.10 and 3.14 and diffed against the committed copy (ignoring the timestamp line):

=== 3.10 vs committed === IDENTICAL
=== 3.14 vs committed === IDENTICAL

So regenerating on any supported interpreter produces the same bytes, and the single-version drift check isn't hiding a version-dependent diff for anyone who runs make -B specification locally.

[Copilot]

Pull request overview

Copilot reviewed 6 out of 7 changed files in this pull request and generated no new comments.

Suppressed comments (2)

tests/test_compliance/test_compliance_suite.py:62

• The module-scoped compliance_report fixture writes the report in teardown whenever --compliance-out is set, even if the compliance suite had test failures (or stopped early). That can overwrite the published compliance doc with a partial/incorrect report when the run did not successfully complete.

@pytest.fixture(scope="module", autouse=True)
def compliance_report(request: pytest.FixtureRequest):
    """Write the accumulated report once the whole suite has run, if asked to."""
    yield
    out = request.config.getoption("--compliance-out")
    if out:
        report.write(Path(out), PACKAGE)

tests/test_compliance/report.py:78

• For reproducible byte-for-byte output across platforms/locales, it’s safer to read/write the generated markdown using an explicit encoding (e.g. UTF-8). Relying on the process default encoding can produce different bytes on systems where the locale encoding isn’t UTF-8.

    if path.exists() and TIMESTAMP.sub("", path.read_text()) == TIMESTAMP.sub("", document):
        return False
    path.parent.mkdir(parents=True, exist_ok=True)
    path.write_text(document)
    return True

[Copilot]

Pull request overview

Copilot reviewed 6 out of 7 changed files in this pull request and generated 2 comments.

[Copilot]

Pull request overview

Copilot reviewed 6 out of 7 changed files in this pull request and generated 2 comments.

Suppressed comments (1)

Previously missed (1) — in code that hasn't changed since the last review.

project.Makefile:4

• The specification rule only depends on test_compliance_suite.py, but the generated output also depends on the report writer (tests/test_compliance/report.py) and the pytest option registration / filtering guard (tests/conftest.py). Without -B, make specification can incorrectly skip regeneration after changes to those files.

specification: docs/specification/compliance.md

docs/specification/compliance.md: tests/test_compliance/test_compliance_suite.py
	$(RUN) pytest $< -q --compliance-out $@

[Copilot]

Pull request overview

Copilot reviewed 6 out of 7 changed files in this pull request and generated 1 comment.

Suppressed comments (2)

Previously missed (1) — in code that hasn't changed since the last review.

project.Makefile:4

• The compliance doc output depends on tests/test_compliance/report.py and the --compliance-out option registration in tests/conftest.py, but the make rule only lists test_compliance_suite.py as a prerequisite. After changes to the writer or option handling, make specification may incorrectly consider the doc up-to-date and skip regeneration unless the user remembers -B. Add the relevant files as prerequisites so incremental builds stay correct.

specification: docs/specification/compliance.md

docs/specification/compliance.md: tests/test_compliance/test_compliance_suite.py
	$(RUN) pytest $< -q --compliance-out $@

tests/test_compliance/test_report.py:75

• test_changed_content_rewrites_and_carries_the_new_stamp seeds the existing file with Path.write_text() defaults. On Windows this can write CRLF, making the setup inconsistent with the LF-only contract and potentially causing the write to happen partly due to newline normalization. Seed the file with explicit UTF-8 + LF newlines.

    target = tmp_path / "compliance.md"
    target.write_text(DOC.replace("Time_executed: 2026-08-06", "Time_executed: 2001-01-01"))

[Copilot]

Pull request overview

Copilot reviewed 6 out of 7 changed files in this pull request and generated no new comments.

Suppressed comments (2)

Previously missed (2) — in code that hasn't changed since the last review.

tests/conftest.py:45

• pytest_configure only blocks filtered runs via -k/-m, but --compliance-out can still be combined with node-id selection (e.g. pytest tests/test_compliance/test_compliance_suite.py::test_join --compliance-out ...), which would write a partial report and truncate the published doc. Consider treating :: node-id selection as a filtered run too (and update the error message accordingly).

    if config.getoption("--compliance-out") and (config.option.keyword or config.option.markexpr):
        msg = "--compliance-out writes the whole report; -k/-m would silently truncate it"
        raise pytest.UsageError(msg)

.github/workflows/main.yaml:92

• The drift check only looks at git diff, so it won’t fail if make -B specification creates untracked files under docs/specification/ (e.g. if compliance.md is deleted in a PR and then regenerated). Mirror the docs/schema/ drift check by also failing on untracked files.

          if ! git diff --quiet docs/specification/; then