composefs backend: zstd:chunked images fail to pull with "unexpected EOF reading tar entry"

[andrewdunndev]

Environment

• bootc 1.16.9-1.fc44, composefs backend, sealed UKI + systemd-boot
• Image: OCI, layers application/vnd.oci.image.layer.v1.tar+zstd with a zstd:chunked TOC annotation, pushed by buildah push --compression-format zstd:chunked

Reproduction

1. Publish a bootc image with buildah push --compression-format zstd:chunked.
2. Run bootc upgrade on a composefs-backend host whose origin references it.

Fails deterministically in seconds:

error: Upgrading composefs: ... Unable to pull container image docker://...:
Failed to pull config Descriptor { media_type: ImageConfig, digest: sha256:073a8cb2...,
size: 23125, ... }: unexpected EOF reading tar entry

The registry side was verified intact before filing: skopeo inspect passes, the named config blob fetches byte-exact (size and sha256 match the descriptor), and podman and skopeo consume the same image without complaint.

Cause (from source, at the composefs-rs rev v1.16.9 pins - 0fbc8533, unchanged on current main f22ecda)

• crates/composefs-oci/src/layer.rs decompress_async() routes MediaType::ImageLayerZstd through a plain async_compression ZstdDecoder. A zstd:chunked blob is a multi-frame stream (including skippable frames); the decoder stops at the first frame boundary, truncating the decompressed tar, and crates/composefs-oci/src/tar.rs bails with unexpected EOF reading tar entry. Chunked support for this path is wip: zstd:chunked support via oci-client composefs/composefs-rs#137, a draft dormant since June 2025.
• The error is misattributed: crates/composefs-oci/src/skopeo.rs pull() wraps ensure_config_with_layers() - the config fetch and every layer import - in a single with_context("Failed to pull config ..."), so a layer-side failure is reported against the config descriptor. That sent me down an hour of registry forensics on a blob that was byte-perfect. composefs fails with Docker v2s2 images #1703 looks like another failure class surfacing under the same wrapper.

This is not #509: that issue tracks zstd:chunked in the containers-storage/podman subsystem; this is the composefs backend's own pull path (composefs-rs), which podman never exercises.

Why this is worse than a missing feature

A sealed composefs host whose publisher switches to zstd:chunked is stranded: the component that would carry any fix (bootc) only arrives through the pull that fails. Recovery is publisher-side re-push (plain zstd pulls fine) or reinstall - nothing host-side works.

Asks

1. Split the error context so layer failures name the failing layer, not the config.
2. Until wip: zstd:chunked support via oci-client composefs/composefs-rs#137 lands: detect the chunked TOC annotation in the manifest before pulling and fail with an explicit "zstd:chunked is not supported by the composefs backend" message.
3. A line in docs/src/experimental-composefs.md documenting the limitation.

[andrewdunndev]

Recovery confirmed from the same host: re-publishing with buildah push --compression-format zstd (no chunked TOC) restored the update channel. The same bootc 1.16.9 composefs host staged the rebuilt image on the first bootc upgrade, 31 s wall for a ~1.4 GB image, and booted it clean, with no host-side action beyond the normal upgrade. That matches the stranding assessment above: the recovery lever is entirely publisher-side.

On ask 2: the release-1.16.10 branch pins the same composefs-rs rev (0fbc8533), so the explicit unsupported-format error would be the only signal a 1.16.10 host gets before hitting the same misattributed failure.

[andrewdunndev]

Root cause confirmed at the composefs-rs rev v1.16.9 pins, and unchanged on current main: the plain ZstdDecoder stops at the first frame boundary of a chunked blob, and the pull error context misattributes the failure to the config.

Three follow-ups, smallest first. composefs/composefs-rs#381 enables multi-frame decode so chunked images pull correctly: full pull, no TOC exploitation, the interim described in #509 ("at least doesn't barf on them (but it won't have optimized fetches)"), with decode parity to ostree-ext's Decompressor as suggested in #1703. composefs/composefs-rs#382 splits the error context so layer failures name the failing layer. #2409 adds the Known Issues line until a fixed composefs-rs reaches bootc.

Verified against a skopeo-produced zstd:chunked image: fails at the v1.16.9 pin with exactly the error above, pulls clean with the decode fix. The release-1.16.10 branch carries the same composefs-rs pin, so the limitation ships in it as well.

[cgwalters]

Thanks for the report, this is conceptually a bug in composefs-rs, so we should have a tracker there, but it makes sense to have one here until we've bumped bootc with a fix.

At least on the bootc side though, unified storage would avoid this issue.

But yeah, we are definitely missing integration tests w/ zstd:chunked here.

[andrewdunndev]

Filed the composefs-rs tracker: composefs/composefs-rs#383.