Keep three evidence lanes separate: strict-native FPE support qualification, whole-forward FPM prediction accuracy, and end-to-end serving accuracy. A successful query is not an accuracy result; an accuracy result applies only to the measured model, hardware, backend, topology, and workload.
User-facing coverage is described in performance-model support and Replay features. This page owns the jobs, provenance, publication, and notifications behind those views.
The workflow compares the base and candidate predictions using the same controller and inputs. Keep numerical regression checks separate from measurement-based accuracy: agreeing with the previous version does not establish agreement with silicon. The active tooling is under scripts/prediction_regression. Use the workflow's current admission, sampling and qualification policy rather than the retired implementation rollout document.
The independent E2E Accuracy Matrix
evaluates the scheduled main SHA and every discovered release/* head against
the private ai-dynamo/aisim-infx-cache HF dataset. The resolver freezes one full
HF commit for all branches and retains the manifest as a 90-day artifact.
HF_READ_AISIM_INFX_CACHE is a read-only Actions secret passed explicitly to the
reusable workflow. The cache refreshes weekly; prediction campaigns remain daily.
A branch matrix runs at most two campaigns
at once, with separate wheels, artifacts, and provenance. Main reuses a qualified
amd64 wheel for its exact SHA when available, even while nightly staging waits for
approval; otherwise it builds one. Each release builds its own wheel. Both the
AISim replay and bundled legacy AIC baseline use their branch's wheel. Manual
runs evaluate one explicit main or release/* SHA from the trusted main workflow.
Campaign jobs use the configured CI_JOB_CONTAINER_IMAGE, Python 3.12, and
sudo to install libgomp1. The HF reader downloads only the source index,
snapshot manifest and observation Parquet files at the pinned commit. It verifies
artifact sizes, hashes, schemas and identities before producing local tables.
The evaluated wheel's baseline CLI and config adapter are selected from its own
package layout: aiconfigurator.* on 0.12 releases or aisim.* after the
namespace migration. Ambiguous wheels with both baseline APIs are rejected.
Baseline provenance records the selected API, adapter, and actual console entry point;
the public summary retains that entry point. Wheel byte checks and imports run before downloading measurements.
No PostgreSQL client or database dump is needed. Artifact upload
uses the output directory so the runner container hook remaps the full path.
Each branch prepares measurements once and runs four independent prediction
shards, with two CPU workers per shard and fail-fast: false. Results and
incremental checkpoints are encrypted before upload and retained for seven days.
Actions artifacts in a public repository are not private. AES-256-GCM encryption
uses a key derived from the HF secret and workflow identity; it authenticates each
relative filename. Source rows and resolved evidence are never uploaded in plaintext.
gh run rerun RUN_ID --failed reuses successful shard artifacts from that run;
only failed partitions repeat. A separate qualification job verifies full coverage
and matching provenance before combining per-point results. Preview runs also
retain resolved source evidence per shard. Actual concurrency depends on runner
capacity (up to eight shard jobs across the two admitted branches).
To inspect an encrypted artifact locally, set HF_TOKEN to the token used by
that run, plus GITHUB_REPOSITORY=ai-dynamo/aisim and its GITHUB_RUN_ID.
Run python scripts/e2e_accuracy/private_artifacts.py open DOWNLOADED_DIR OUTPUT_DIR
in the campaign environment. Token rotation requires a new complete workflow run;
old ciphertext cannot be opened with the replacement token.
Complete campaigns upload sanitized e2e-accuracy-web-<branch-key> artifacts.
Pages validates the branch's successful qualification job in the artifact's exact
run attempt, producer, revision, coverage, and checksums before combining it with
qualified FPE data. One branch's failure does not cancel other campaigns or block
their publication. Retried jobs preserve successful branches' original run-attempt
provenance. Failed campaigns retain previous evidence. Accuracy numbers are advisory;
incomplete campaigns cannot publish. A release selector may still show a historical
snapshot until that release has a qualified campaign. See the
accuracy campaign contract
for pinned scheduler settings, measurement selection, and provenance.
Accuracy Slack Weekly combines the day's scheduled E2E and FPM results in one message in #swdl-dynamo-aisim-daily. Failures and comparable regressions appear in that message; separate per-model and per-GPU E2E tables appear in the main message; FPM coverage and comparison details remain in its thread. Delivery runs on Mondays at 09:00 America/Los_Angeles using that day's results. E2E and FPM evaluations stay daily. It is opt-in and has a default dry-run mode plus an explicit test-send mode. See setup, comparisons, and testing.
FPM Accuracy Matrix runs at 10:47 UTC daily and supports manual evaluation of
an exact SHA on main or a release >= 0.12.0. It pins HF data once per campaign,
uses verified exact wheels, and evaluates FPM with KV warmup on/off and regression
on CPU. Branch results remain in Actions artifacts for 90 days. Pages validates
and publishes successful branch results independently; failed refreshes retain
the prior qualified result. Accuracy is advisory, outside PR prediction campaigns
and release staging gates. See FPM details.
Manual dispatch accepts optional hf_revision, a full 40-character dataset commit.
Use it to evaluate a migrated dataset branch with compatible preview consumers
before promoting either change to main. Scheduled runs continue to resolve HF main.
The workflow verifies the requested revision through HF and records the resolved
commit in every artifact. Publish compatible consumers before moving HF main to a
new manifest contract; historical evaluation artifacts are never rewritten.
Native op-level probes qualify the measured query surface; they do not certify Replay, Sweeper, or prediction accuracy.
Install a built repository wheel, then generate one op-level FPE shard per system. The command imports that installed native runtime; it must not prepend the source-only application package to Python's import path:
python python/aisim/tools/support_matrix/generate_fpe_support_matrix.py \
--output-dir fpe-support-matrix/b200_sxm \
--system b200_sxm \
--forward-model op_level \
--max-workers 8After all system shards finish, build the split CSV files consumed by the interactive page:
python python/aisim/tools/support_matrix/build_fpe_support_matrix.py \
fpe-support-matrix \
--output-dir python/aisim/src/aisim_core/systems/fpe_support_matrixUse filters and a deterministic topology cap for a focused smoke run:
python python/aisim/tools/support_matrix/generate_fpe_support_matrix.py \
--output-dir fpe-support-matrix-smoke \
--model Qwen/Qwen3-8B \
--system b200_sxm \
--backend vllm \
--forward-model op_level \
--max-topologies-per-role 1 \
--max-workers 2Each raw shard contains deterministic JSON, CSV, and Markdown coverage
artifacts plus a separate run_metrics.json. The rollup produces mode-neutral
rows for the separate FPE Support Matrix and adds real probe counts, topology
counts, native status counts, phase latencies, and source SHA to the detail view.
Wall time, CPU time, peak RSS, and worker count remain separate from the
deterministic coverage artifacts.
The runner opens a bounded thread pool for one system/backend/version group at
a time. Native database loading and hot-path queries release the Python GIL,
but Python-backed model compilation and database memory still limit scaling.
Increase --max-workers only with measured memory headroom.
Main branch nightly CI calls the reusable FPE workflow after confirming that main has
changed and building its release artifacts. All FPE shards install the exact
amd64 nightly wheel, verified against the artifact checksums, source commit,
and one recorded wheel hash. A manual run requires the full expected_sha and
builds one shared wheel. Neither path rebuilds the native runtime in every
shard.
The workflow discovers one shard per curated system/backend pair and runs at
most 20 shards concurrently on the repository-specific AMD64 CPU runner set.
Other CI runs share this pool, so runner availability can reduce concurrency.
Each shard runs only forward_model=op_level with an eight-thread local pool. A
final job validates reports before combining them into the split web CSV
artifact. Qualification requires every discovered shard, exact source and wheel
identity, consistent package version and workload, complete role-appropriate
phases, and no unexpected build or query failure. The small required-probe
manifest additionally requires all four phases of at least one native topology
for each known-good model/system/backend identity. An empty or entirely
unsupported report cannot satisfy that requirement. Classified exploratory
coverage gaps remain visible; they do not certify support.
fpe-qualification.json records the accepted source, wheel digest, shard count,
and status counts. FPE qualification gates the later GitLab security handoff,
not initial Artifactory staging. The FPE
workflow remains manually dispatchable for an out-of-band refresh. This avoids
leaving runners idle when a small system finishes before the largest systems.
Full runs also suppress repeated SDK warnings at the console while preserving
every classified failure and representative error in the matrix artifacts.
Refresh-time claims must name both runner concurrency and per-runner thread
count, plus the source SHA from the measured run.
GitHub Pages rebuilds after a successful FPE Support Matrix, Main branch nightly CI, or Release branch nightly CI run and on public-documentation changes. For main, every deployment selects the retained qualified FPE artifact with the newest tested source commit in the current main history. Re-running an older commit cannot displace a newer qualified snapshot. The page shows the snapshot's source SHA, artifact creation time, and producing CI run.
Pages checks the producing workflow, successful main-branch run, qualification manifest, row source identities, and complete shard count. It copies only the indexed CSV data; HTML and scripts always come from main. Pull request previews use repository data and cannot deploy. If no eligible artifact remains, Pages fails and preserves the existing website instead of republishing stale committed data. Run FPE Support Matrix on main with the current full main SHA, then rerun GitHub Pages if needed. Web artifacts are retained for 90 days, subject to the repository's retention policy. Artifacts generated before the qualification manifest was introduced cannot be published by this path.
Staging waits for the complete matrix. Each shard has a 480-minute timeout; the 20-shard concurrency limit and runner queues can make the total wait longer than that per-shard limit. A successful wheel build alone does not make the nightly available. No release-latency percentile is promised until complete runs have been measured with this gate enabled.
The final platform-wheel check invokes the package verifier with
--exercise-engine --exercise-fpe on each runner. The FPE flag requires the
repository generator and Git checkout; its subprocess runs from an unrelated
temporary directory against the installed wheel. The reduced Docker build
context runs the package/runtime verifier without the repository-only FPE flag.
The published matrix's Branch selector defaults to main and lists the
repository's release/* branches. Share a selection using
?branch=release%2F0.12.0; the model search (q) is preserved when switching.
Each selection loads a separate packaged dataset with its own tested source
SHA, timestamp, and evidence link. Release results never fall back to main's data.
Pages discovers release branches from the fetched origin refs. For each
branch, it selects the newest retained qualified artifact with a source SHA in
that branch's history. Eligible producers are successful runs of
fpe-support-matrix.yml or nightly-ci.yml on the selected branch, or the
main-hosted release workflow described below with explicit release and tooling
provenance. An old-commit rerun cannot displace a newer tested commit.
A release without retained qualification is labeled unavailable; an
expired release artifact also removes its data from the next deployment.
The page shows a Results not available yet notice with a Check again
button and a link to the release nightly runs. Coverage appears after a
successful qualified nightly run and Pages deployment.
Malformed qualification fails the deployment. Main still requires a retained
qualified snapshot before the site can deploy.
Release branch nightly CI runs daily at 09:23 UTC from trusted main, and can
also be dispatched on main. It discovers every fetched release/<version>
branch and records its current commit SHA before building. New branches such as
release/0.13.0 join the next run automatically, without a workflow edit or
backport. Versions may contain letters, digits, dots, underscores, and hyphens,
starting with a letter or digit. An empty inventory skips qualification.
The scheduler calls the same reusable qualification workflow for each release, one release at a time. It builds one wheel from each unmodified checkout and uses the release's locked dependencies, curated model inventory, SDK, estimator, model definitions, and performance tables. Up to 20 system/backend shards run concurrently with eight probe threads each; the runner pool is shared with other CI. The serial release matrix keeps this limit at 20 across the release nightly run. A failed release does not cancel the remaining releases, but publication requires the entire nightly run to succeed. Every scheduled run refreshes the evidence, even if the release SHA is unchanged, so retained artifacts do not silently expire.
The probe harness and required-probe manifest come from the workflow's exact
main commit. CI records that tooling SHA separately from the tested release
SHA and wheel digest. Before discovery or probing, it verifies the installed
package bytes and import locations against the shared wheel. The release branch
does not need a workflow backport. This job produces qualification evidence;
it does not stage or publish release packages.
After a release's shards and required probes pass, CI uploads its own
fpe-support-matrix-web-release-<version> artifact for 90 days. Wheels and raw
reports also have version-specific names, so releases in one run cannot mix
data. A successful nightly run triggers Pages. The publisher verifies the
trusted producing workflow, its main-history tooling commit, the artifact's release identity,
and the tested source's membership in release history. It ranks snapshots by
tested source history, then artifact ID, while preserving the existing failed
rerun and expired-artifact protections. An older-source rerun cannot replace a
newer tested source.
Release results come only from GitHub Actions artifacts. No manual ZIP, committed-result fallback, or main-data fallback is used. Until the first qualified release run succeeds, the release selector shows unavailable. The page links to the CI run and shows both tested source and probe tooling.
To reproduce a release result, check out the recorded tooling commit at the
workspace root and the recorded release source under release-source/.
For a local restore, set FPE_RELEASE to the recorded release version,
FPE_SOURCE_SHA to its full release source SHA, FPE_TOOLING_SHA to the full
tooling SHA, and FPE_RUN_ID to the producing Release branch nightly CI run ID
(not a local retry or a later consumer run). Obtain WHEEL_STORAGE_URL and
WHEEL_STORAGE_REPO from that run's automated-release environment
configuration, and supply an authorized read credential as ARTIFACTORY_TOKEN
through your local secret manager or environment. Authenticate gh with access
to the producing run; the GitHub token and Artifactory token are separate.
Run the following from the tooling checkout root. It downloads the run's
fpe-release-wheel-<version>-manifest Actions artifact, which contains only
wheel-manifest.json, then invokes the transfer script directly:
set -euo pipefail
gh run download "${FPE_RUN_ID:?}" --repo ai-dynamo/aisim \
--name "fpe-release-wheel-${FPE_RELEASE:?}-manifest" --dir release-wheel-manifest
GITHUB_REPOSITORY=ai-dynamo/aisim \
GITHUB_REPOSITORY_ID="$(gh api repos/ai-dynamo/aisim --jq .id)" \
GITHUB_SHA="${FPE_TOOLING_SHA:?}" \
GITHUB_RUN_ID="${FPE_RUN_ID:?}" \
STORAGE_MODE=download \
STORAGE_SCOPE=official \
STORAGE_PR_NUMBER='' \
STORAGE_NAME="fpe-release-wheel-${FPE_RELEASE:?}" \
STORAGE_PATH=. \
STORAGE_URL="${WHEEL_STORAGE_URL:?}" \
STORAGE_REPO="${WHEEL_STORAGE_REPO:?}" \
STORAGE_TOKEN="${ARTIFACTORY_TOKEN:?}" \
STORAGE_RUN_ID="${FPE_RUN_ID:?}" \
STORAGE_SOURCE_SHA="${FPE_SOURCE_SHA:?}" \
STORAGE_ARCHITECTURE=amd64 \
STORAGE_MANIFEST=release-wheel-manifest/wheel-manifest.json \
python3 .github/actions/wheel-storage/transfer.pyThe script from the recorded tooling commit restores fpe-release-wheel/
(including the wheel and its provenance), fpe-release-shards.json, and
fpe-release-probes.json into the workspace. It binds the release source,
tooling commit, producer run, repository, and amd64 architecture, reads the
producer attempt from the manifest, and verifies each object's size and SHA-256
before replacing local files. No GITHUB_RUN_ATTEMPT is needed for a download.
Install the release's locked environment and exact restored wheel as in the
workflow. The matrix's per-cell command uses release-source/python/aisim/.venv/bin/python
directly to run the verified release wrapper with the recorded source, tooling,
and branch. Each invocation requires a fresh release-probe-harness/ directory;
remove only that generated directory between reproductions. Raw shard reports
remain Actions artifacts for seven days, as does the completion manifest.
Configure Artifactory retention for the handoff payload to at least seven days;
the wheel payload is not retained in Actions. The smaller qualified web dataset
is retained for 90 days.
The deployed data/fpe-support-matrix/branches.json catalog lists available
and unavailable branches. Main retains the existing data path, and release
data lives under data/fpe-support-matrix/branches/release/<version>/.
Repository previews package main's committed snapshot only and require no
GitHub credentials or artifact downloads.
The Legacy AIC Support Matrix displays a Historical snapshot and Qualification not recorded. Its Latest data change timestamp is the commit time of the most recent change to its index or an indexed CSV, with a link to that data commit. It is not the website build time or evidence of a complete matrix rerun. No full-matrix generation time or qualification report was recorded for the retained legacy data.
The Pages builder adds this provenance to the packaged
data/support-matrix/index.json. Website-only changes do not refresh the data
date. A build with modified/untracked data, a shallow Git history, or no Git
history leaves the date unavailable. Direct source-tree previews also show the
missing-date state because the committed legacy index contains no provenance.
To reproduce the browser checks, install Chromium with
uv run --python 3.12 --with playwright playwright install chromium, then run
uv run --python 3.12 --with playwright python scripts/pages/check_legacy_support_matrix_browser.py.
The script builds a temporary site and verifies real data, commit links, missing
and malformed metadata, calendar-date boundaries, and unchanged matrix rows.
Use --browser-executable /path/to/chrome to reuse an installed browser, or
--screenshot /path/to/preview.png to capture the real page before test fixtures.
Accuracy Slack Weekly posts one weekly top-level message to
#swdl-dynamo-aisim-daily (C0BULBSTXJ6). Alerts and results share that message;
E2E per-model and per-GPU tables are included in the main message. No mentions are
sent. The bot does not accept commands or start evaluations.
The message title is :rainbow: *Accuracy Weekly · YYYY-MM-DD*, with a rainbow
emoji and bold text, without a timezone suffix. Immediately below it, a quote
block shows run links with duration and attempt, overview links, and short
alert/recovery summaries. The alert count and comparison-note count appear at
the end of the main message.
- Select today's scheduled E2E and FPM runs by their creation date in
America/Los_Angeles. A rerun uses its latest attempt; successful branch artifacts retained from an earlier attempt are validated against that attempt. - Send on Mondays at 09:00 America/Los_Angeles, including failures/cancellations and missing/unfinished runs. Producer completion no longer triggers the notifier on other days. GitHub schedules can be delayed; 09:00 is the intended time, not a wall-clock SLA.
- Each main message includes both pipeline links. An unstarted pipeline links to its workflow. Failed jobs include their job links and failing step names.
- Include every branch evaluated in those runs. Do not substitute yesterday's data for a missing branch. Manual campaigns never enter production weekly reports.
- After delivery, late completions do not update the frozen report or create another message that day. Subsequent weekly reports can announce recovery.
- Serialize notifier runs. Identify production messages using Slack metadata keyed by repository and local date. Persist the frozen report before posting; on retry, restore it and send only missing thread parts. Rechecks at 30-minute intervals through Monday 11:30 America/Los_Angeles can resume an interrupted delivery. Do not delete the bot's messages or its retained report artifacts; they are delivery records.
- History/thread reads use GET query parameters and omit empty pagination cursors.
- Slack timeouts and
ok:falsefail the notification job. An uncertain POST is not blindly retried: the next run checks Slack history/thread metadata first. An extended Slack/GitHub outage may prevent delivery; no system can guarantee a notification while its delivery service is unavailable.
The E2E table has six columns: Branch, Overall, vLLM, SGLang, TRT-LLM, Coverage.
Each accuracy cell is TPOT%/TTFT% MAPE (for example 24.32%/43.67%)
for AISim, not legacy AIC.
Main-message detail tables independently group by model and by GPU, within each branch,
combining frameworks. They average successful point errors directly; they do
not average already-aggregated MAPEs. Coverage is predicted / eligible points,
not all original source measurements. FPM shows KV-warmup on/off and online
regression MAPE, weighted by successful prediction counts. FPM coverage is in a
thread reply to keep the main table narrow. The parent uses single-line spacing,
shows at most three short alert summaries, and puts comparison notes and full
alert details in the thread. Every numerical MAPE value includes %.
For each branch:
- Start with the most recent earlier qualified scheduled campaign in the retained 90-day history. Download only the requested branch's artifact and comparison evidence during this search. Without one, explicitly report an initial baseline.
- Require identical measurement revision/content and evaluation-rule fingerprints.
The rule fingerprint covers evaluator source and its branch workflow at the
producer revision, not the evaluated AISim package revision. The FPM
fingerprint includes
scripts/notifications/accuracy_digest.py, which supplies its point codec. A change establishes a new baseline and reports comparison unavailable, not recovery. - Compare each E2E topology/configuration and FPM configuration/predictor on the intersection of successfully predicted points. Alert when MAPE increases by at least 2 percentage points AND at least 10% relative. From zero, an increase of at least 2 points qualifies.
- A previously successful point becoming failed, unavailable, or absent alerts separately, even if total coverage is unchanged because other points improved.
- Keep the pre-alert baseline while the regression persists. Do not silently accept yesterday's regression as today's normal. Successful comparable results against that baseline can be marked recovered in a later weekly message.
- Initial prediction failures are shown as attention notes; without prior evidence they are not labeled new regressions. A green pipeline does not imply every prediction succeeded.
Aggregates alone cannot detect exchanged successful and failed points. The FPM
producer writes a separate
fpm-accuracy-comparison-<branch-key> artifact containing a hash of observation order
and compressed little-endian float64 percentage-error sequences (-1 for unsuccessful predictions). It binds
to the public summary checksum and exact producer snapshot. Reruns overwrite
both public and comparison artifacts so their attempt identities stay aligned. The notifier verifies
counts and means against that qualified summary. No raw measured/predicted
latencies are exported. Existing Pages artifacts remain unchanged. Old FPM runs
without point evidence can be displayed, but point regression checks are explicitly
unavailable until a comparable pair with evidence exists.
GitHub Actions runs the notifier; the Slack app supplies its bot identity. No server, Event Subscriptions, Socket Mode, or slash commands are needed.
At https://api.slack.com/apps, choose Create New App > From scratch, name it
AISim Daily, and select the workspace containing the daily channel.
Under OAuth & Permissions > Bot Token Scopes, add:
chat:writeto post the main message and replies.channels:historyto check prior messages and resume threads in this public channel. If using a private channel later, usegroups:historyas well.metadata.message:readto read daily-report and thread-part metadata.
Click Install to Workspace (or request workspace approval), then copy the
Bot User OAuth Token beginning with xoxb-. After changing scopes on an
installed app, reinstall it to apply the new permissions.
In #swdl-dynamo-aisim-daily, open Channel details > Integrations > Add apps
and add AISim Daily. Store its bot token only as repository
secret SLACK_ACCURACY_BOT_TOKEN. Configure repository variables:
| Variable | Value |
|---|---|
SLACK_ACCURACY_CHANNEL_ID |
C0BULBSTXJ6 |
SLACK_ACCURACY_ENABLED |
true only after the test below passes |
The previous SLACK_ACCURACY_WEBHOOK_URL and Slack Workflow Builder trigger
are not used by this version. Messages use Slack mrkdwn: bold headings,
monospaced tables, and named links. This is not GitHub-flavored Markdown; tables
are rendered inside code blocks rather than as Markdown table syntax.
Both test and production use this same channel. Test messages are labeled
[TEST] and use separate metadata, so they do not consume the weekly production
slot or update its baseline. Automatic delivery is disabled unless explicitly
enabled. GitHub permissions are read-only (contents:read, actions:read).
The notifier checks out its own trusted workflow revision; it never executes
code from downloaded artifacts. Pages validation remains strict; explicit manual
preview/test alone permits same-repository producer runs from a non-main branch.
Open Actions → Accuracy Slack Weekly → Run workflow. Choose dry-run
(the default), and provide completed e2e_run_id and fpm_run_id values. Manual
accuracy runs are allowed here. For example:
gh workflow run accuracy-digest.yml --ref main \
-f mode=dry-run -f e2e_run_id=<e2e-run-id> -f fpm_run_id=<fpm-run-id>For a local preview, run the same Python entrypoint from the checkout with a read-only GitHub credential:
GH_TOKEN="$(gh auth token)" python3 scripts/notifications/notify_accuracy.py \
--e2e-run-id <e2e-run-id> --fpm-run-id <fpm-run-id> \
--output /tmp/accuracy-previewInspect the Actions Summary and accuracy-preview / accuracy-report artifacts,
or local preview.md / report.json. The JSON includes alert reasons, comparison
limitations, and the exact baseline snapshots. Dry run does not read Slack
credentials, post messages, or write production baseline state.
After configuring the bot and channel, run once with mode=test:
gh workflow run accuracy-digest.yml --ref main \
-f mode=test -f e2e_run_id=<e2e-run-id> -f fpm_run_id=<fpm-run-id>This sends a real [TEST] parent message plus its detail thread to the daily
channel. Check desktop/mobile table readability, links, and per-model/per-GPU
sections, then set SLACK_ACCURACY_ENABLED=true. Test sending is restricted to
main; a branch dry run receives no Slack token. No manual mode sends a
production weekly report.
A workflow dry run does not rerun predictors. Use the deterministic tests in
tests/fpm_accuracy/test_digest.py to exercise regressions, coverage losses,
changed datasets, missing runs, daylight-saving cutoffs, deduplication, and
interrupted thread delivery without sending fake alarms to Slack.