Add optional progress output for long-running subcommands (mpileup, view, norm, index)

[carstenerickson]

Summary

bcftools mpileup, view, norm, and index run silent on multi-hour inputs, making "still working" indistinguishable from "hung" without external tooling. An opt-in periodic stderr progress line would close the gap at near-zero cost.

Current workaround

We read /proc/<pid>/io (Linux only) for bytes-read and divide by input file size. Limitations:

• Linux only; no /proc/<pid>/io on macOS or BSD.
• Bytes only, no record counts.
• Fragile child-PID discovery when bcftools sits behind a shell pipe.
• mpileup needs a special case because the I/O-doing process is the child, not the parent shell.

Our wrapper is ~140 LOC handling these edges.

Suggested approach

Either of:

• Opt-in flag — --progress[=INTERVAL] emits a stderr line every INTERVAL records (default 1M). One extra branch per record; off by default, so scripts are unaffected.
• TTY auto-enable — isatty(STDERR_FILENO)-gated, like tar --checkpoint=.N. Interactive only, no flag.

Format suggestion:

[mpileup] 12.3 / 23.0 GB processed (53.5%), 145M records, 1234s elapsed, 11min remaining

Bytes from htsFile.fp.bgzf->block_offset; records from each subcommand's main loop; ETA from linear extrapolation, omitted when input size is unknown.

Low priority — the workaround works. Filing because the proliferation of /proc/<pid>/io wrappers across pipelines is a UX signal that upstream should own this.

---

bcftools 1.23.1.

[pd3]

I like the idea. However, even though it seems simple, this would not be straightforward to implement. If this is ever supported, it would likely be done together with samtools.

[carstenerickson]

Thanks @pd3, that makes sense, and it matches what we found when we went digging through the sources to scope it ourselves. Sharing in case it's useful, and because I'd be happy to do the work if you're open to a PR.

Two things became clear once I looked, and they line up with both of your points.

1. Three of the four commands can be done in bcftools alone while index cannot.

view, norm, and mpileup each own their driver loop in bcftools (while (bcf_sr_next_line(...)) in vcfview.c and vcfnorm.c, and while (bam_mplp_auto(...)) in mpileup_reg()), so an optional periodic emit can live entirely there with no htslib change. For mpileup the input decode and pileup all run on the main thread (hts_set_threads only touches the output), so the emit is trivially thread safe.

index is the odd one out: vcfindex.c makes a single call to bcf_index_build3() and only ever sees the final return code. The whole read and push loop is inside htslib (bcf_index() and tbx_index()), sitting right next to sam_index(), all over the shared hts_idx_* machinery. So there's no seam on the bcftools side at all; progress for index needs a new htslib entry point (a progress options arg, or a *_build4 variant), and samtools index would get and want the same change. That's the concrete version of your remark. (One wrinkle: samtools index of CRAM routes through cram_index_build(), a separate path, so a hook on the BGZF loop covers BAM, SAM, VCF, and BCF but not CRAM indexing. CRAM parity would be its own follow up.)

2. Our own suggestion for where to read bytes was wrong, and a real percentage only works for local seekable files.

I suggested bgzf->block_offset. That's actually the cursor within the current uncompressed block (it resets every ~64 KB, masked to 16 bits), so it's a sawtooth, not a file position. The right primitive is the public bgzf_tell(fp) >> 16 (that is, block_address), which is the compressed offset of the current block.

The genuine gap is the denominator, htslib has no public accessor that I can see for total size, and size is unknowable for pipes, stdin, and most remote URLs (hseek(SEEK_END) disrupts the stream and fails on backends that cannot seek; stat() only helps for local regular files). So a true percentage plus ETA only works for local seekable inputs; streaming has to fall back to bytes or records processed plus a rate, with no ETA. Worth noting that index is the easy case here: bcf_index_build3 already rejects input that is not BGZF, and vcfindex.c forbids stdin, so the input is always a file on disk with a knowable size, and the loops already hold bgzf_tell().

What we'd propose (happy to adjust):

Phase 1, bcftools only, no ABI impact: an optional --progress[=SEC] on view, norm, and mpileup. Throttled by time (a cheap amortized clock check so we're not calling time() per record or column), a plain stderr line terminated by a newline in the existing [I::...] style. Percentage plus ETA when the input is a local seekable file, otherwise records per second. This lands independently and delivers most of the value.

Phase 2, htslib, shared with samtools: the index case, as a small public progress hook threaded into the index builders. That is the natural place for a shared primitive both tools use, for example an hts_tell() style accessor for the compressed offset that normalizes BGZF, CRAM, and hFILE, dispatching on is_bgzf and is_cram the way hts_hfile already does internally. This is the piece that would be done together with samtools, and we're glad to open it as a separate htslib PR if there's appetite.
Would you prefer the timing and format helper to live in htslib as public API (it benefits samtools symmetrically, but is more work), or a small bcftools helper first? There's already a copy of this pattern in +vrfs, which prints elapsed, estimated total, and ETA via a local ksprint_time().

If a Phase 1 draft PR against view, norm, and mpileup sounds like a reasonable starting point, we'll put one up for review and keep index and htslib as a separate discussion. Totally understand if the priority is low. We just wanted to offer concrete help.

[pd3]

I can imagine adding something relatively simple, such as periodically printing the CHR:POS position to indicate how far the program has progressed. Anything more sophisticated is likely to run into complications, for example when a list of regions is provided. Therefore, I would be willing to consider a simple progress meter along those lines. One advantage is that it would not require any changes in htslib.

Before you start working on this, please check our contributing policy, especially the section on AI-contributed content: https://github.com/samtools/bcftools/blob/develop/CONTRIBUTING.md

Please also note that we are less likely to accept pull requests that make large or invasive code changes.