This page is for whoever makes a release. To install or upgrade one at a site, follow 02 — Install guide §4 (fetch and verify the signed release) and 06 — Operations runbook §2 (upgrade).
A release is a git tag vX.Y.Z (or vX.Y.Z-suffix, published as a pre-release) on a
commit of main that CI passed. Pushing the tag runs .github/workflows/release.yml. It
publishes images and evidence. It deploys nothing. Installation is done by the
installer (install/), from the release manifest. The workflow refuses a tag whose commit
is not on main (git merge-base --is-ancestor), before anything is built or signed.
The old deploy-dev.yml and deploy-prod.yml were deleted on 2026-10-08. They could not
have worked: dev triggered on a develop branch that does not exist, prod retagged a
:main image that nothing pushed, wrote $GITHUB_OUTPUT on the remote host, ran
migrations after up -d, and bound blue/green to hard-coded ports (web 80/8081, api
8080/8180). Prod also kept the
production SSH key in a third-party action.
Per-release notes (known limitations, staging record) live in docs/release-notes/,
e.g. v1.0.0.md; the change list is CHANGELOG.md.
For each of the five services, api, worker, web, scenario-engine and
ai-orchestrator, and for moodle (TrueNorth's Moodle, infra/platform/moodle, which the
installer runs only with tn_moodle_enabled), release.yml:
- builds from the tagged commit with a fresh base image (
pull: true) and no layer cache, withTN_VERSION=<tag>as a build arg. In the five service images the tag becomes theTN_VERSIONenv var (the Moodle Dockerfile declares noARG TN_VERSION); the workflow sets theorg.opencontainers.image.versionlabel on all six. Therevisionlabel holds the commit. BuildKit attaches SLSA provenance (mode=max) and an SBOM as attestations. - pushes the image to
ghcr.io/mblanke/truenorth-<service>by digest only. Nothing is tagged yet. - scans that digest with Trivy. Blocking: any HIGH or CRITICAL vulnerability with a fix available fails the release.
- writes an SPDX SBOM of that digest with syft (
anchore/sbom-action). - signs the scanned digest with cosign keyless and attaches the SPDX SBOM as a signed
attestation (
cosign attest --type spdxjson), then verifies both against the signer identity ("Signatures", below). - only then tags the digest
:<tag>, and checks that the tag resolves to the digest it scanned.
The publish job then signs release-manifest.json and SHA256SUMS and creates the GitHub
release with these assets:
| Asset | What |
|---|---|
release-manifest.json |
the interface for installers: service → image digest (below) |
release-manifest.json.sigstore.json |
its Sigstore bundle: certificate, signature, transparency-log proof |
sbom-<service>.spdx.json |
SBOM per image |
trivy-<service>.json |
the scan that gated the image (fixable HIGH/CRITICAL; empty when clean) |
00-index.pdf ... 06-operations-runbook.pdf |
the guides (docs/guides/) rendered by scripts/build_guides_pdf.sh; best-effort, so a release can lack them if that job failed |
SHA256SUMS |
SHA-256 of every asset above (the manifest's bundle included) |
SHA256SUMS.sigstore.json |
the Sigstore bundle of SHA256SUMS |
Every signature is cosign keyless: GitHub's OIDC token proves which workflow ran, Fulcio issues a short-lived certificate naming it, and Rekor logs the signature publicly. There is no signing key to steal or rotate. A valid signature must name exactly:
| certificate identity | https://github.com/mblanke/TrueNorth/.github/workflows/release.yml@refs/tags/<tag> |
| OIDC issuer | https://token.actions.githubusercontent.com |
The installer (install/roles/tn_release) verifies release-manifest.json against that
identity, for the tag being installed, before it trusts any digest in it, with a cosign
binary it pins and checks by SHA-256 (tn_cosign_version, tn_cosign_sha256; the same
release as the workflow's COSIGN_VERSION/COSIGN_SHA256). A manifest that does not
verify stops 20-fetch-app. -e tn_release_verify_signature=false exists only for a
release made before signing, and says so loudly.
Air-gapped. The bundle carries the certificate, the signature and the Rekor inclusion
proof, so verification needs no Rekor or Fulcio access, only Sigstore's trust root. Copy a
trusted_root.json to the control node with the release files (on a connected machine
cosign caches it under ~/.sigstore/root/; or fetch it from Sigstore's TUF repository) and
pass -e tn_release_trusted_root=<file>: the installer then verifies with --offline.
Not yet exercised against a real release: the first signed tag is the test.
If any image fails its build or scan, no release is created. Digests of images that passed may already be in GHCR, untagged. They are harmless and are not in any manifest.
git fetch github && git checkout github/main
# CI must be green for this commit (Actions tab, or gh run list --commit "$(git rev-parse HEAD)")
git tag -a v1.2.3 -m "TrueNorth v1.2.3"
git push github v1.2.3
gh run watch "$(gh run list --workflow release.yml --limit 1 --json databaseId --jq '.[0].databaseId')"To redo a failed release, delete the tag (git push github :refs/tags/v1.2.3), fix the
cause on main, and tag again. Never move a published release's tag: installs pin digests,
and the manifest must keep describing what was scanned.
The application reports the tag: images carry TN_VERSION (the installer also renders it
into the env file), and the API reads it at start (APP_VERSION in
control-plane/api/app/main.py, "dev" when unset), logs it at start and returns it as
version from GET /health. The OpenAPI document carries the API contract version
instead (API_CONTRACT_VERSION), so a release does not drift the published contract.
The role guides (docs/guides/) are not built by this workflow. To give a site printable
copies, run scripts/build_guides_pdf.sh (pandoc) when the documentation set is cut
(docs/guides/README.md).
V=v1.2.3
mkdir -p "rel-$V" && cd "rel-$V"
gh release download "$V" -R mblanke/TrueNorth
ID="https://github.com/mblanke/TrueNorth/.github/workflows/release.yml@refs/tags/$V"
ISS=https://token.actions.githubusercontent.com
for f in release-manifest.json SHA256SUMS; do # signed by release.yml at this tag
cosign verify-blob --bundle "$f.sigstore.json" --certificate-identity "$ID" --certificate-oidc-issuer "$ISS" "$f"
done
sha256sum -c SHA256SUMS # assets intact
# Each image: signature and SBOM attestation
jq -r '.images[].ref' release-manifest.json | while read -r ref; do
cosign verify "$ref" --certificate-identity "$ID" --certificate-oidc-issuer "$ISS" >/dev/null && echo "signed $ref"
cosign verify-attestation --type spdxjson "$ref" --certificate-identity "$ID" --certificate-oidc-issuer "$ISS" >/dev/null
done
# Every tag still points at the digest the manifest records (and that was scanned).
jq -r '.images | to_entries[] | "\(.value.tag) \(.value.digest)"' release-manifest.json |
while read -r tag digest; do
got=$(docker buildx imagetools inspect "$tag" --format '{{json .Manifest.Digest}}' | tr -d '"')
[ "$got" = "$digest" ] && echo "ok $tag" || echo "DIFF $tag: $got != $digest"
done
# The gating scan found nothing fixable at HIGH/CRITICAL.
for f in trivy-*.json; do
echo "$f: $(jq '[.Results[]?.Vulnerabilities[]?] | length' "$f") findings"
done
# Rescan today (new advisories appear after release). Use the pinned Trivy, below.
trivy image --scanners vuln --severity HIGH,CRITICAL --ignore-unfixed \
"$(jq -r .images.api.ref release-manifest.json)"
# What is inside an image
jq -r '.packages[] | "\(.name) \(.versionInfo)"' sbom-api.spdx.json | sort | headThe installer consumes it (install/roles/tn_release; install/README.md "Images"):
ansible-playbook site.yml -e tn_release_version=v1.2.3. This is the interface:
{
"schema_version": 1,
"version": "v1.2.3",
"git_sha": "<40-hex commit the images were built from>",
"created": "2026-10-08T12:00:00Z",
"images": {
"api": {"image": "ghcr.io/mblanke/truenorth-api", "digest": "sha256:…",
"ref": "ghcr.io/mblanke/truenorth-api@sha256:…",
"tag": "ghcr.io/mblanke/truenorth-api:v1.2.3"},
"worker": {…}, "web": {…}, "scenario-engine": {…}, "ai-orchestrator": {…},
"moodle": {…}
}
}Rules for consumers:
- Pull and run
ref(image@digest). Nevertag: a tag can be moved, a digest cannot. - Check
schema_version. Fields may be added, and a breaking change bumps it. - Verify its Sigstore bundle against the release workflow's identity ("Signatures"), then
check it against
SHA256SUMSfrom the same release, before using it. git_shais the commit whoseinfra/platform/docker/compose.prod.ymland Alembic migrations match these images. Fetch the app source at that commit, not at a branch.- Order is unchanged: datastores, then
alembic upgrade headin the api image, then the rest (the installer's50-stack-upalready does this). compose.prod.ymlhas nobuild:sections: it runs${TN_IMAGE_API},${TN_IMAGE_WORKER},${TN_IMAGE_WEB}and${TN_IMAGE_AI_ORCHESTRATOR}, which the installer renders fromref, and pins every third-party image by digest.compose.build.ymladds the builds back for a lab. Thescenario-engineimage is published but not run by compose (the api mounts the engine from the source atgit_sha). Themoodleimage runs fromcompose.moodle-prod.ymlas${TN_IMAGE_MOODLE}(install/roles/tn_moodle); releases before it have nomoodleentry, and the installer then refuses to enable Moodle rather than run an unscanned image.
| What | Pinned to | How it was verified |
|---|---|---|
| cosign binary | v3.1.3, linux-amd64 SHA-256 4629c757…f7f71 (release.yml; the installer pins the same release for linux/darwin amd64/arm64) |
cosign_checksums.txt of v3.1.3 verified 2026-10-08 with cosign verify-blob --bundle cosign_checksums.txt.sigstore.json --certificate-identity keyless@projectsigstore.iam.gserviceaccount.com --certificate-oidc-issuer https://accounts.google.com; each binary's hash matches its line |
| Trivy binary | v0.74.0, Linux-64bit tarball SHA-256 2ae6fe3e…8be4371a (in ci.yml and release.yml) |
trivy_0.74.0_checksums.txt verified with cosign against aquasecurity/trivy/.github/workflows/reusable-release.yaml@refs/tags/v0.74.0 (OIDC issuer token.actions.githubusercontent.com); the tarball hash matches its line |
aquasecurity/trivy-action |
ed142fd0673e97e23eac54620cfb913e5ce36c25 (v0.36.0, 2026-04-22) |
gh api repos/aquasecurity/trivy-action/git/ref/tags/v0.36.0 → annotated tag a9c7b0f…, signature verified, target ed142fd…; the commit's signature is verified. Run with skip-setup-trivy: true, so the action's installer is not used |
anchore/sbom-action |
3ad7283483fc7af8ff2b4ea19663c2d5ca935e26 (v0.24.2; it pins syft v1.51.1) |
tag resolved via the GitHub API; commit signature verified. v0.24.3 (2026-10-02) was too new |
| other actions | full commit SHA, with a # vX.Y.Z comment — four stragglers are still on bare tags: actions/checkout@v4 (ci.yml lint-install, backup-drill), actions/setup-python@v5 (ci.yml) and actions/upload-artifact@v4 (backup-drill) |
tag resolved via gh api repos/<owner>/<repo>/git/ref/tags/<tag> (annotated tags dereferenced) |
| k6 (CI smoke) | grafana/k6@sha256:23f22790… (v1.8.1) |
docker buildx imagetools inspect grafana/k6:1.8.1 |
Trivy's release channel was compromised on 2026-03-19 (CVE-2026-33634: a malicious
v0.69.4, and hijacked trivy-action/setup-trivy tags). Rules when bumping any pin:
- never
@master,@mainor a bare major tag; - avoid releases younger than a few weeks;
- for Trivy, verify the cosign bundle of the checksums file, then update
TRIVY_VERSIONandTRIVY_SHA256in both workflows.
GitHub-hosted runners share egress addresses, so anonymous Docker Hub pulls fail on its
rate limit (toomanyrequests) and on intermittent 500s. On PR #124 that failed
build-docker, integration, e2e, load-smoke, moodle, greyspace, helm-kind and the backup
drill. CI pulls Docker Hub images through Google's public cache, mirror.gcr.io, instead.
It needs no credentials and serves the same content by the same digests. Digest-pinned
references stay pinned and resolve to identical bytes.
| Pull path | How it uses the mirror |
|---|---|
Docker daemon (docker run/pull/build, compose, kind's node image, the buildx builder image) |
.github/actions/dockerhub-mirror, the first step after checkout: merges registry-mirrors into the runner's /etc/docker/daemon.json, restarts dockerd and fails unless docker info lists the mirror. It also runs docker logout. The runner image logs the client in to Docker Hub (account githubactions), dockerd presents that credential to the mirror, the mirror answers unauthorized, and dockerd silently falls back to Docker Hub. Pulls are therefore anonymous. dockerd still falls back to Docker Hub if the mirror fails |
services: (ci.yml test-python-shard) |
They start before any step, and a dockerd restart would stop them, so the images are named mirror.gcr.io/library/postgres:16 and so on |
BuildKit (docker/setup-buildx-action, docker-container driver), which does not read daemon.json |
buildkitd-config-inline with [registry."docker.io"] mirrors = ["mirror.gcr.io"] in ci.yml build-docker, helm-kind and release.yml image. Unlike dockerd, BuildKit did not fall back to Docker Hub when the mirror was unreachable (tested locally with an unresolvable mirror host), so these builds depend on mirror.gcr.io being up |
kind's node containerd (helm-kind: deps.yaml, the helm test image) |
infra/k8s/kind/ci-cluster.yaml sets containerd's config_path. A step then writes certs.d/docker.io/hosts.toml in each node, with the mirror first and Docker Hub as the fallback |
| Self-hosted runner (lab.yml) | The host's dockerd is shared and is not reconfigured. The image is named on mirror.gcr.io |
tests/contracts/test_ci_dockerhub_mirror.py fails when a hosted job that runs
docker/compose/buildx/kind lacks the action, when a service image or a buildx builder
uses Docker Hub directly, or when the kind configuration drifts. A new job that pulls
images only needs - uses: ./.github/actions/dockerhub-mirror after its checkout.
A release builds with pull: true from base-image tags. Through the mirror, a tag can
briefly resolve to an older digest than Docker Hub's while the cache refreshes. The
release Trivy gate scans what was actually built, and the provenance attestation records
the base digest. To force a fresh base, rerun after the cache catches up. You can also
check the tag with docker buildx imagetools inspect against docker.io and
mirror.gcr.io.
ci.yml build-docker builds each image (no push) and scans it with the same pinned
Trivy. It prints a report of fixable HIGH/CRITICAL (never fails) and fails the PR on a
fixable CRITICAL. A release also blocks on HIGH, so read the report before tagging.
The five service Dockerfiles apply distro security updates at build time (apt-get upgrade / apk upgrade), because base-image tags lag them; the Moodle image instead pins
its base image by digest and applies no distro upgrade. On 2026-10-08 an un-upgraded build had
fixable HIGH findings in pcre2 (all five images) and libexpat (web). After the upgrade
step, all five images scanned clean locally (arm64 builds; CI builds amd64).
When a fixable HIGH appears in a Python or npm dependency, upgrade the pin. An ignore
file (.trivyignore) needs its own ADR, like the pip-audit ignores (ADR 0003).