[new HQR] queue builder uploaded to a wrong NAR path on S3

[vcunat]

Describe the bug

We have a correct .narinfo for this fixed-output derivation:

$ curl -s https://cache.nixos.org/gypp7g48p2fcpjx952i84dhw4ak95npl.narinfo | grep NarHash
NarHash: sha256:033w5cd8z7r02h4r2p8lha6z81qqvky69a7aiahjpd15smss4wjp

git grep -B1 033w5cd8z7r02h4r2p8lha6z81qqvky69a7aiahjpd15smss4wjp | cat
pkgs/tools/typesetting/tex/texlive/fixed-hashes.nix-  lpform-36918 = {
pkgs/tools/typesetting/tex/texlive/fixed-hashes.nix:    run = "033w5cd8z7r02h4r2p8lha6z81qqvky69a7aiahjpd15smss4wjp";

But the contents is very wrong.

$ nix build --store "$(mktemp -d)" /nix/store/gypp7g48p2fcpjx952i84dhw4ak95npl-lpform-36918-tex
error: hash mismatch importing path '/nix/store/gypp7g48p2fcpjx952i84dhw4ak95npl-lpform-36918-tex';
         specified: sha256:033w5cd8z7r02h4r2p8lha6z81qqvky69a7aiahjpd15smss4wjp
         got:       sha256:1f0v56g28nl5v8pmxg9cf7d8rm3jnb2950dknd613z1nlvqw78fc
error: path '/nix/store/gypp7g48p2fcpjx952i84dhw4ak95npl-lpform-36918-tex' is required, but there is no substituter that can build it

An LLM analysis repost:

The object served at cache.nixos.org/nar/033w5cd8….nar.zst is not the lpform TeX file at all. It’s the Haskell half-0.3.3 library built for GHC 9.10.3 (~1 MB, 13 files: libHShalf-*.so/.a, .hi, .hie, a package .conf). Decompressed and restored it hashes to 1f0v56g28… — your exact failure hash.

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/error-when-rebuilding-with-texlivefull-package/78356/12

[vcunat]

Note that part of these commands will probably soon fail to reproduce, as we surely want to fix the bad path on cache.nixos.org

[NyCodeGHG]

nixpkgs master's merge queue is broken because of (at least) these store paths also having the problem:

• /nix/store/bl7rmhhsy7vjb9qm3jfwgqpv3cn7wfb1-openssl-3.6.2
• /nix/store/i8ykl6jr2x9x3bqv2m8l6gn6i2246lyr-nghttp3-1.15.0

[vmfunc]

this hit more than the texlive path, it broke nixpkgs ci repo-wide through the ci/pinned.json tooling bootstrap. two affected tooling paths i caught, state as of 2026-06-18.

nghttp3-1.15.0 (i8ykl6jr2x9x3bqv2m8l6gn6i2246lyr).. same shape as yours, wrong content under a correct narinfo.

narinfo     url nar/157s6q….nar.zst   narhash sha256:157s6q…
ci import   specified sha256:157s6q…  got sha256:07svi88…

now repaired.. the served nar re-hashes to 157s6q… again, matches the narinfo, imports clean.

openssl-3.6.2 (bl7rmhhsy7vjb9qm3jfwgqpv3cn7wfb1).. different shape, worth a look.

ci import     specified sha256:0nagwij5…  got sha256:1fypgxjw…
narinfo now   narhash sha256:1fypgxjw…, served nar also hashes to 1fypgxjw…

so this one is self-consistent now but at 1fypgxjw…, the content ci flagged as wrong, not the 0nagwij5… it expected. for an input-addressed path that looks like the narinfo got aligned to the bad upload rather than the path restored. might be worth confirming 0nagwij5… is the real openssl-3.6.2 and re-uploading it.

impacts.. while it was live, lint / parse, lint / nixpkgs-vet and eval / x86_64-darwin failed on prs at random depending on how warm the runner cache was. examples NixOS/nixpkgs#532910, #532911, #532912. recovering now, newest prs mostly green with a few still catching unrepaired paths.

[vcunat]

These two still succeed for me, but I suppose that Fastly isn't necessarily globally consistent?

(ideal way to test is nix build --store "$(mktemp -d)" /nix/store/$path)

[vmfunc]

on the edge i hit, the path is fully self-consistent right now, nar.zst and narinfo both last-modified Wed, 17 Jun 06:18:34, the narinfo FileHash sha256:1652ryb9… / FileSize 98432 match the served nar.zst, it decompresses to 157s6q…, sig verifies. but ur seeing the nar.zst at Thu, 18 Jun 06:34:05

for an immutable content-addressed nar there should never be two last-modified stamps, so the bad blob got overwritten and fastly pops haven't converged

so it's not local cache warmth like i first guessed, it's edge inconsistency, your globally-consistent doubt confirmed. ci runners pass or fail by which pop they land on, it can't see a poisoned one

i think it needs a hard fastly purge of nar/157s6q….nar.zst plus a check that s3 origin holds the 157s6q blob and not the 07svi88 one

otherwise stale-but-bad copies keep getting served until they age out

[vcunat]

Note that the FOD hash for that texlive package has been in nixpkgs for about 3 years. Apparently even its .narinfo is recently reuploaded because

$ curl -s https://cache.nixos.org/gypp7g48p2fcpjx952i84dhw4ak95npl.narinfo | grep Compression
Compression: zstd

(but the hash in that .narinfo is the correct one, unchanging over those years)

[vcunat]

Reuploads of already existing NARs might be related to the observed "restarting" of already-succeeded jobs observed in the Hydra's web UI.

Details: for example this one got somehow automatically restarted this way (it's not possible to force the UI into such actions, I believe) and then aborted manually. Also note that the whole-jobset view is showing very confusing counts, as this build (and thousands of others) are present in all evaluations but only in the latest ones they get counted red.

[vcunat]

People running into this might get persistent issues because of local caching. That can be nuked by sudo rm -rf /root/.cache/nix/ (in typical cases at least, I think those using nix through daemon running as root)

[Mic92]

Those are the store paths that were affected as far as I can tell:

/nix/store/i8ykl6jr2x9x3bqv2m8l6gn6i2246lyr-nghttp3-1.15.0
/nix/store/53zvl5z0iwjfcgl89dgcnyv9nbiw97yd-tzdata-2026b
/nix/store/klwr4n2snj7ldlpkhwarwlif80myx0vj-git-2.54.0-doc
/nix/store/bl7rmhhsy7vjb9qm3jfwgqpv3cn7wfb1-openssl-3.6.2
/nix/store/lq8laawx5ii4jrs3rcvyjya3gvk5d6lq-ruby3.4-serverengine-2.4.0
/nix/store/gypp7g48p2fcpjx952i84dhw4ak95npl-lpform-36918-tex

I still wait for the inventory of the current day so I can check all NAR files that have been produced.