Multi-rail parallel read: aggregate multiple RDMA paths for single-worker stripe reads

[fresshman]

Refs #33

Summary

Generalizes the existing per-node concurrent stripe read
(get_descriptor_stripes_sge, wire tag 15) to per-rail concurrency: one
client worker transfers disjoint stripe subsets of the same object over N
independent RDMA paths concurrently, aggregating NIC bandwidth.

Hard invariants: on-disk stripe layout untouched; upper read interface
(LookupObject / ReadByDescriptor) untouched; single-NIC deployments are a
pass-through (byte-for-byte identical behavior); v1 fails the whole request
safely if any required rail fails — no partial data, no in-request retry.

Changes

• multi_rail.rs — RailReader trait, RailManager, MultiRailReader:
  stripe→rail planning (locality + round-robin), scoped-thread concurrent
  reads, completion aggregation, per-stripe checksum verification, and
  RailReadStats with a bottleneck classifier (network / storage / software).
• mock_rail.rs — mock transport with configurable per-rail bandwidth and
  fault injection (disconnect, under-delivery, late_write_after_cancel +
  epoch guard), making the full scheduling / completion / buffer-lifecycle
  path testable without RDMA hardware.
• rdma.rs — RailReader implemented for RdmaClient (real verbs path;
  the default read path registers the destination buffer afresh per read and
  holds the MR only across register → read_stripes, reads go through
  get_descriptor_stripes_sge). Works for Soft-RoCE and physical NICs.
• tests/multi_rail_mock.rs — 11 hardware-free tests (see below).
• tests/multi_rail_mr_lifetime.rs — 3 regression guards for MR lifetime
  (consecutive reads with address reuse, object switching on one reader, and a
  compile-time check of the pooled registration API).
• cs-mock-bench — rail-count sweep capacity model with bottleneck
  attribution.
• examples/softroce_dual_rail.rs — full dual-rail e2e on Soft-RoCE:
  gRPC PUT → descriptor lookup → single-rail baseline vs dual-rail read →
  per-stripe xxh3-64 verification + byte equality → speedup report
  (rxe0/rxe1 against a server listening via CS_RDMA_DEVICES).
• examples/softroce_concurrency.rs — object-size x concurrency sweep on the
  real verbs path, reporting aggregate goodput, p50/p95 latency and process
  resources (RSS, registered memory, inflight bytes) — contest deliverable e.
• setup-wsl2-rxe.sh + kv-service/configs/server-wsl2-softroce.toml +
  docs/wsl2-softroce-setup.md + build-and-test.cmd — reproducible two-rail
  RXE bring-up, demo server config (4 MiB striping threshold, tier_b io_uring
  executor), setup guide and a one-shot build/test/bench driver for WSL2.

Memory safety (late RDMA WRITE hazard)

1. pin-until-completion: the read holds the destination buffer and every
   rail's registration until join; cancel flips a logical flag only, MRs are
   not deregistered early.
2. epoch buffer guard: late writes land only while live == epoch;
   bumping live on cancel makes the guard skip them.
3. two-phase commit + per-stripe checksum: stripes are verified after
   aggregation; a cancelled request never reaches commit.

Test plan

cargo test -p contextstore-client-rs --test multi_rail_mock — 11/11 pass
(default features, no RDMA hardware needed):

• byte-exact multi-rail aggregation; single-rail pass-through;
• late-write hazard A/B (without cancel: corruption reproduced; with cancel:
  epoch guard blocks it);
• stripe→rail round-robin planning;
• per-stripe checksum failure detected; single-rail disconnect safe-fail;
• partial completion / under-delivery detected;
• read timeout: error surfaces, caller's buffer untouched;
• inflight-budget backpressure rejection;
• object version-change (generation) rejection.

cargo test -p contextstore-client-rs --test multi_rail_mr_lifetime — 3/3
(the pooled-API check is rdma-feature gated):

• consecutive reads into per-iteration buffers (address reuse) stay correct and
  register afresh every read;
• switching objects on one reader never serves a stale registration;
• invalidate_mr_cache exists with the documented signature.

cargo test -p contextstore-client-rs --features rdma --lib — 4/4.

Capacity model (Mock, WSL2; labelled accordingly)

64 MiB object, 4 MiB chunks, 16 stripes, 125 MB/s per rail:

rails  agg_MB/s  bal_MB/s  theo_MB/s  speedup  bottleneck
1      123.5     125.0     125.0      0.99x    network: slowest rail caps the object
2      244.9     250.0     250.0      1.96x    software: post-read stripe verification (client CPU)
3      327.4     333.3     375.0      2.62x    software: post-read stripe verification
4      484.1     500.0     500.0      3.87x    software: post-read stripe verification

Soft-RoCE end-to-end (WSL2, rxe0/rxe1 over two veth pairs)

Setup: custom WSL2 kernel 6.6.87.2 with CONFIG_RDMA_RXE=y, two RXE devices
bound to two veth pairs (192.168.96.110/111), server started with
CS_RDMA_DEVICES=rxe0:0.0.0.0:50053:1,rxe1:0.0.0.0:50054:1, then
examples/softroce_dual_rail.rs PUTs a 64 MiB object (16 stripes x 4 MiB)
via gRPC and reads it back single-rail vs dual-rail with per-stripe xxh3-64
verification plus full byte equality:

object   single-rail      dual-rail        speedup  verify
64 MiB   73.8 MB/s        217.6 MB/s       2.95x    xxh3-64 per stripe + byte equality
128 MiB  93.3 MB/s        384.1 MB/s       4.11x    xxh3-64 per stripe + byte equality

(Single runs; run-to-run variance on WSL2 is noticeable, e.g. single-rail
64 MiB measured 73.8–110.1 MB/s across runs. The dual-rail advantage was
consistent in every run.)

The e2e pass surfaced and fixed two real bugs in the existing tag-15 path:

• server: serve_get_stripes compared a stripe's object-space end offset
  against the client window (req.max_size), silently missing every stripe
  beyond index 0 whenever the object exceeded the advertised window. Now
  compared against the descriptor size.
• client: the server maps stripe offsets onto the segment table as
  contiguous object-space coverage from 0 (map_range_to_segments), so the
  sparse per-stripe table shifted writes to wrong offsets. RdmaClient
  now advertises one segment spanning the registered object window (the
  RailReader trait keeps the per-stripe table, which the mock asserts on;
  translation happens in the RDMA implementation with guards).

Also fixes two pre-existing --features rdma build breaks (build_put_request
test missing ttl_seconds; rdma_bench test Args initializer missing
fields), and completes the mandatory [gc] fields in the demo config.

The concurrency sweep surfaced and fixed a third, more subtle bug:

• client, memory safety: register_raw_buffer_cached keyed MRs by
  (base_ptr, length). A client doing consecutive reads into a per-iteration
  Vec silently read back zeros — after the first Vec is freed the allocator
  often returns the same virtual address, the cache hits, and the reused MR
  still pins the old physical pages while the CPU reads the new mapping, so
  the NIC's RDMA WRITE lands where the caller never looks. The trait already
  documented "pins the region for the read's duration", so the cache was the
  deviation. The default path now registers afresh per read (MR held in
  in_flight_mr across register → read_stripes); the pooled path survives
  as explicitly-named register_raw_buffer_pooled with invalidate_mr_cache.
  No public signature changed; wire protocol, disk layout and upper read API
  untouched. Guarded by multi_rail_mr_lifetime.rs.

Soft-RoCE concurrency / resource sweep (deliverable e)

examples/softroce_concurrency.rs, real verbs, same object / layout / server /
client across rows. Every one of the 20 points verified (verify_ok=true):

size   mode    conc  agg_MB/s  p50_ms    RSS_MiB  reg_MiB  infl_MiB
32MiB  single  1      128.4     296.3        70       32       32
32MiB  dual    1      197.1     216.9        70       64       32
32MiB  single  2      489.8     118.1        70       64       64
32MiB  dual    2      490.0     118.2        70      128       64
64MiB  single  1      121.8     657.2       134       64       64
64MiB  dual    1      339.9     213.4       134      128       64
64MiB  single  2      498.6     235.1       134      128      128
64MiB  dual    2      492.9     236.4       134      256      128
128MiB single  1      111.1    1365.9       262      128      128
128MiB dual    1      411.3     305.6       262      256      128
128MiB single  2      495.9     474.0       262      256      256
128MiB dual    2      495.2     471.9       262      512      256
256MiB single  1      116.0    2485.6       518      256      256
256MiB dual    1      171.3    1617.4       518      512      256
256MiB single  2      133.3    3723.7       518      512      512
256MiB dual    2      138.1    3889.5       518     1024      512
512MiB single  1      101.3    5550.9      1030      512      512
512MiB dual    1      112.1    4558.4      1030     1024      512
1024MiB single 1       92.6   12551.7      2054     1024     1024
1024MiB dual   1      112.5    9004.3      2054     2048     1024

Dual-rail speedup vs object size (conc=1): 32 MiB 1.54x, 64 MiB 2.79x,
128 MiB 3.70x (peak), 256 MiB 1.48x, 512 MiB 1.11x, 1024 MiB 1.22x.

Reading it honestly: multi-rail pays off in the 64–128 MiB band; below it
the per-read connect/first-WR cost dominates; above 256 MiB the speedup
collapses and absolute goodput falls to ~112 MB/s because on Soft-RoCE every
rail is emulated in the same host CPU and long chains of 4 MiB WRITEs serialize
through one QP per rail — the CPU, not the NIC, is the ceiling, and a second
rail competes for it. Concurrency helps single-rail more than dual-rail in the
sweet spot (single up to 4.46x vs dual 1.20–2.49x), and at 256 MiB dual
concurrency 1→2 regresses to 0.81x — exactly the ceiling the inflight-budget
guard exists to bound. Resources stay predictable: RSS tracks live buffers
(70 → 2054 MiB), registered memory is size × workers × rails, inflight is
size × workers.

Environment honesty

Soft-RoCE (RXE) is a software RoCE implementation: both rails share the host
CPU. The >2x speedup on 2 rails is therefore not NIC bandwidth aggregation —
the single-rail baseline is capped by CPU serialization on one QP/softirq
path, and a second rail engages additional cores. The claim is the relative
single-vs-dual-rail speedup on identical code paths; no claim of physical
multi-NIC aggregate bandwidth; no GPU involved. Absolute numbers on real
hardware will differ (mock model predicts rail-bound scaling up to the
software verification ceiling).