Skip to content

Read and format a whole file on threads (ramdump -@ N) - #124

Merged
mvassilev merged 1 commit into
developfrom
ramdump-threads
Nov 5, 2026
Merged

mvassilev merged 1 commit into
developfrom
ramdump-threads

Conversation

@mvassilev

Copy link
Copy Markdown
Collaborator

A full dump of chr20 took 16.5 s on one thread, against 13.6 s for samtools view from BAM
on one thread and 3.6 s on eight. Decoding the columns and building the text both ran on
the thread that writes; only qualities and names were decoded ahead on other threads.

ramdump -@ N, with no region, starts N threads. An RNTuple reader is not shared between
threads, so each opens the file for itself and formats every N-th block of records (those
of a block of names and qualities, 10,000 records) into a text of its own; the main thread
writes the texts in order, with at most 2N waiting.

  • A thread never takes two consecutive blocks, so the name and quality readers decode
    nothing ahead. A first version handed the blocks out as they came, and a thread that got
    two in a row started reading ahead blocks that other threads then took: -@ 2 peaked at
    729 MB.
  • The threads open the file without RNTuple's cluster cache, which would have every thread
    decompress the whole cluster of each block it takes.
  • With a region, or with -c, -@ is ignored.

NA12878 chr20 at 30x (17.7 million records) on a Xeon E5-2690 v4, pinned to one socket (14
cores), file in the page cache, medians of 5 runs in alternating order:

ramdump -h wall CPU memory
develop, one thread 16.45 s 26.9 s 107 MB
-@ 2 13.74 s 26.9 s 297 MB
-@ 4 7.85 s 30.2 s 379 MB
-@ 8 4.59 s 34.3 s 546 MB

On the same machine samtools view -h -@ 8 takes 3.6 s from BAM (21.6 s CPU, 94 MB) and
7.7 s from CRAM 3.1 (39.4 s CPU, 964 MB). One thread of -@ does all its decoding itself,
so -@ 2 gains less than half; past that the time is about the CPU time over N.

The output equals the one-thread output, byte for byte, on 1, 2, 3, 4, 8 and 500 threads,
with and without FLAG filters. A new test dumps 25,000 records on three threads and compares.
All tests pass. No format change.

With no region, ramdump -@ N starts N threads. Each opens the file for
itself, without RNTuple's cluster cache, and formats every N-th block of
10,000 records into a text of its own; the texts are written in order,
at most 2N waiting. A thread never takes two consecutive blocks, so the
name and quality readers decode nothing ahead that another thread needs.

On chr20 of NA12878 a full read takes 4.6 s on 8 threads instead of
16.5 s on one; samtools view -@ 8 takes 3.6 s from BAM and 7.7 s from
CRAM 3.1.
@github-actions

github-actions Bot commented Nov 5, 2026

Copy link
Copy Markdown

clang-tidy review says "All clean, LGTM! 👍"

@codecov-commenter

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 82.65306% with 17 lines in your changes missing coverage. Please review.
✅ Project coverage is 88.90%. Comparing base (29428e1) to head (2b10721).

Files with missing lines Patch % Lines
tools/ramdump.cxx 81.39% 9 Missing and 7 partials ⚠️
test/ramdump_test.cxx 91.66% 0 Missing and 1 partial ⚠️

❌ Your patch status has failed because the patch coverage (82.65%) is below the target coverage (85.00%). You can increase the patch coverage or adjust the target coverage.

Additional details and impacted files

Impacted file tree graph

@@             Coverage Diff             @@
##           develop     #124      +/-   ##
===========================================
- Coverage    89.02%   88.90%   -0.13%     
===========================================
  Files           26       26              
  Lines         3408     3506      +98     
  Branches      1155     1198      +43     
===========================================
+ Hits          3034     3117      +83     
- Misses         196      204       +8     
- Partials       178      185       +7     
Flag Coverage Δ
unittests 88.90% <82.65%> (-0.13%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines Coverage Δ
test/ramdump_test.cxx 98.42% <91.66%> (-0.71%) ⬇️
tools/ramdump.cxx 85.49% <81.39%> (-0.65%) ⬇️
Files with missing lines Coverage Δ
test/ramdump_test.cxx 98.42% <91.66%> (-0.71%) ⬇️
tools/ramdump.cxx 85.49% <81.39%> (-0.65%) ⬇️
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@mvassilev
mvassilev merged commit e62acde into develop Nov 5, 2026
4 checks passed
@mvassilev
mvassilev deleted the ramdump-threads branch November 5, 2026 09:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants