zstd-seekable-format-go

Package seekable adds random-access reads to compressed Zstandard data.

It writes compressed streams with a seek table and reads them by decompressed byte offset through Reader, ReadAt, and Seek. The resulting stream remains valid Zstandard data, so standard Zstandard readers can still decode it sequentially.

The wire format is the Zstandard seekable format: compressed frames followed by a seek table in a skippable frame.

The package uses small encoder/decoder interfaces and is tested with github.com/klauspost/compress/zstd.

Install

go get github.com/SaveTheRbtz/zstd-seekable-format-go/pkg

import seekable "github.com/SaveTheRbtz/zstd-seekable-format-go/pkg"

Write

func writeSeekable(dst io.Writer, chunks [][]byte) error {
	enc, err := zstd.NewWriter(nil, zstd.WithEncoderLevel(zstd.SpeedFastest))
	if err != nil {
		return err
	}
	defer enc.Close()

	w, err := seekable.NewWriter(dst, enc)
	if err != nil {
		return err
	}
	for _, chunk := range chunks {
		if _, err := w.Write(chunk); err != nil {
			return err
		}
	}
	return w.Close()
}

Writer.Close writes the final seek table. Without it, Reader and NewSeekTable cannot find the random-access metadata.

Read

func readAt(src io.ReadSeeker, off int64, p []byte) error {
	dec, err := zstd.NewReader(nil)
	if err != nil {
		return err
	}
	defer dec.Close()

	r, err := seekable.NewReader(src, dec)
	if err != nil {
		return err
	}
	defer r.Close()

	_, err = r.ReadAt(p, off)
	return err
}

Offsets are decompressed byte offsets. Reader implements io.Reader, io.ReaderAt, io.Seeker, and io.Closer.

For an io.ReaderAt, use io.NewSectionReader with the compressed stream's size in bytes. This works with sources that fetch byte ranges from remote storage:

r, err := seekable.NewReader(
	io.NewSectionReader(src, 0, compressedSize), dec,
)

Metadata

func frameForOffset(r *seekable.Reader, off uint64) (seekable.FrameOffsetEntry, error) {
	table, err := r.SeekTable()
	if err != nil {
		return seekable.FrameOffsetEntry{}, err
	}
	entry, ok := table.EntryByDecompressedOffset(off)
	if !ok {
		return seekable.FrameOffsetEntry{}, io.EOF
	}
	return entry, nil
}

SeekTable exposes the decompressed size, frame count, checksum flag, and frame lookup by id or decompressed offset.

Compatibility

Seekable streams are valid Zstandard streams:

func readSequential(src io.Reader) ([]byte, error) {
	zr, err := zstd.NewReader(src)
	if err != nil {
		return nil, err
	}
	defer zr.Close()

	return io.ReadAll(zr)
}

Command Line

Install the zstdseek command with:

go install github.com/SaveTheRbtz/zstd-seekable-format-go/cmd/zstdseek@latest

Compress a file into a seekable Zstandard stream:

zstdseek -f input.dat -o input.dat.zst

Add -t to verify the compressed output after writing:

zstdseek -f input.dat -o input.dat.zst -t

Use - for stdin or stdout. Verification requires a named output file. Input may also be supplied as one positional argument. With neither -f nor a positional input, zstdseek reads stdin. Without -o, it writes stdout. To avoid dumping compressed bytes to a terminal, stdout output is rejected when stdout is a terminal.

zstdseek -f - -o input.dat.zst < input.dat
zstdseek -f input.dat -o - > input.dat.zst
zstdseek input.dat > input.dat.zst
zstdseek < input.dat > input.dat.zst

The main tuning flags are -q for Zstandard compression quality and -c for the average or min:avg:max content-defined chunk size in KiB.