[milesial]

Why and what changed

Weka replay spends host time serializing rows, normalizing repeated prefix hashes, and inspecting the same trace twice before execution.

This change buffers row writes, parses rows from a reusable byte buffer, and reuses collision-checked source hashes within each play. Python admission consumes hash arrays through a streaming loop and caches completed corpus estimates. Both admission guards still apply current memory headroom.

The measured command takes 85.53 s before this PR, 42.86 s with buffering, and 28.02 s with all changes. This saves 57.51 s, or 67.24%, for a 3.05× overall speedup. The additional changes save 14.84 s beyond buffering, or another 1.53× speedup.

Command and end-to-end ladder

This is the public command measured from process launch through exit, including Python startup and output conversion:

python3 -m aisimulate predict --stack engine \
  --config "$CONFIG" --output-dir "$OUTPUT" --capture-per-request

CONFIG and OUTPUT replace private absolute paths. All three configuration files contain identical JSON. The installed interpreter comes from the experiment virtual environment.

The Linux x86_64 run uses a 64,882,247-byte Weka trace with 16 plays and 4,908 source requests. It uses one replica, 16 agentic lanes, seed 42, warmup, and a 3,600-second profile. The resource limits are 6 GB and four CPUs. The trace block size is 64.

Cumulative stage	Command time	Saving from prior stage	Observed peak RAM	Temporary file data
Before this PR	85.53 s	Baseline	456.94 MiB	Same serialization format; disk probe not repeated
Buffer canonical row writes	42.86 s	42.67 s, 2.00×	449.38 MiB	308.43 MiB logical payload in separate import probe
Add hash reuse, row reader, streaming parser, and metadata cache	28.02 s	14.84 s, 1.53×	444.45 MiB	Same 308.43 MiB logical payload

RAM means resident set size, or memory resident in RAM. The RAM column is the sampled sum for the supervised process tree. It excludes the outer coordinator. One sample per stage does not establish a RAM reduction or timing variance.

Native replay takes 17.15 → 16.84 → 17.18 s. Time outside native replay falls from 68.38 → 26.02 → 10.83 s. That interval includes admission, import, model preparation, startup, and output conversion. The 0.35-second native variation does not establish a replay improvement or regression.

Completed requests, output tokens, simulated duration, predicted throughput, and the request lifecycle digest remain identical across all stages.

Only this public command has an end-to-end comparison. No separate end-to-end ablation exists for each change in the final stage.

Per-change time, disk, and RAM impact

The following component measurements use macOS arm64. Their times do not add to the Linux command ladder.

Change	Time evidence	Disk impact	RAM impact
Buffered row writes	End-to-end: 85.53 → 42.86 s	Fewer small writes. Same serialized bytes, so no space saving.	Adds a bounded writer buffer. Its individual allocation was not measured.
Reuse normalized hashes within each play	Import open, preflight, and lowering: median 6.447 → 2.372 s	Same canonical rows. No persistent cache.	Adds one map per play. Largest sample play has 65,904 unique source hashes.
Reuse a byte buffer for row parsing	Row parsing and builder insertion: median 0.862 → 0.391 s	Same row spool and complete read.	Retains capacity for the largest row. The individual peak contribution was not measured.
Stream hash arrays through an inner loop	One metadata scan: sample 5.449 → 2.349 s; full source 131.687 → 63.583 s	One source scan remains. No new files.	Recorded process high-water mark does not increase: 50,692,096 bytes for sample calls and 50,724,864 bytes for full-source calls.
Cache completed metadata estimates	Full source: 63.556 s cold, then 0.000051 s in the same process	Avoids another metadata scan of 1,847,151,435 source bytes. This saves logical reads, not disk capacity.	At most 16 estimates plus per-file identity keys. Incremental cache allocation was not measured.

The native probe runs these diagnostic commands with the sample trace:

./weka-import-baseline "$TRACE"
./weka-import-optimized "$TRACE"

The probe timer covers import through graph finalization. It excludes process startup and a later digest pass. Three alternating pairs give a total median of 7.934 → 3.386 s, or 2.34×. Both builds already contain buffered writes.

The two native changes raise median process peak RAM from 207,142,912 to 219,971,584 bytes, an increase of 12.23 MiB. That peak includes the digest pass. The measurements do not isolate each native change's RAM cost. Phase medians do not sum to the total median.

The Python measurements time estimator calls, not complete CLI commands. Before/after parser calls share a process, so their RAM values are cumulative high-water marks. The cache series uses a separate process. Its full-source cold and warm calls report the same 47,906,816-byte high-water mark. Estimates remain byte-for-byte equal, and a cache hit still rejects insufficient live headroom.

Disk space boundary

A separate import probe finds the same 258,529,550-byte canonical row spool in both native builds. The source snapshot adds 64,882,247 bytes. They overlap during import, for 323,411,797 bytes of logical temporary file data, or 308.43 MiB. The measured spool-size difference is zero. Both probes remove their temporary files on exit.

This total excludes original input, output reports, filesystem overhead, and unrelated files. The directory sampler does not capture the exact filesystem allocation peak. No disk peak was measured during the Linux commands. Sufficient TMPDIR capacity remains necessary because this PR does not shrink the spool.

Review map and evidence

• Risk level: medium, because admission reuses metadata estimates.
• Start with _estimate_weka_trace in resources.py, then WekaImporter::for_each_row and normalized_hashes in weka.rs.
• Cache keys include file identity and allocation-relevant workload fields. The implementation compares source identities before and after inspection.
• Invalid, changed, empty, and budget-limited scans remain uncached. Native source validation remains active.
• Regression coverage includes parser boundaries, malformed hashes, integer limits, collisions, private tails, repeated spool reads, source changes, eviction, and live headroom.
• Ruff hooks select the same project configuration as CI, including for root-level tests.
• Public and serialized contracts remain unchanged.
• Fast CI, Full CI, and Simulation Performance passed for code commit cd84034e7bab39854203c3b2f2bacbfb8319b204.
• Independent review remains pending.

Modeling and measurement limits

These changes affect host runtime. They do not change modeled GPU performance. Model, workload, scheduler configuration, and other experimental patches remain fixed across the end-to-end stages.

Those other patches, the source trace, diagnostic binaries, and complete model configuration are outside this PR. These observations are not a standalone reproducible benchmark or a general speedup guarantee. The 28.02-second result uses the sample trace. Full-source Python inspection is not a full-corpus simulation. Earlier full-corpus baselines did not complete, so this PR claims no full-corpus end-to-end speedup ratio.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

• Configuration used: Repository: ai-dynamo/aisimulate/.coderabbit.yaml
• Review profile: ASSERTIVE
• Plan: Enterprise
• Run ID: 9b99b12a-b61d-45e0-8d73-8c51fd2eb39b

📥 Commits

Reviewing files that changed from the base of the PR and between a0a35e7 and 7fb214a.

📒 Files selected for processing: 5

• crates/core/src/replay/loadgen/weka.rs
• docs/reference/local-resources.md
• python/aisimulate/.pre-commit-config.yaml
• python/aisimulate/src/aisimulate/resources.py
• tests/test_weka_host_resources.py

🔗 Linked repositories identified

CodeRabbit considers these linked repositories for cross-repo context during reviews:

• ai-dynamo/dynamo (manual)
• ai-dynamo/aiconfigurator (manual)

Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 9 remain after this review.

📜 Recent review details

⏰ Context from checks skipped due to timeout: 24

• GitHub Check: Prediction Regression / Collect prediction snapshot (new)
• GitHub Check: Prediction Regression / Collect prediction snapshot (old)
• GitHub Check: Platform Wheels / Build wheels (manylinux_2_28_aarch64)
• GitHub Check: Collector Data / Check collector data
• GitHub Check: Collector Data / Perf data sanity (informational)
• GitHub Check: Platform Wheels / Build wheels (manylinux_2_28_x86_64)
• GitHub Check: Platform Wheels / Build wheels (macosx_arm64)
• GitHub Check: Python Dependency Licenses
• GitHub Check: readme-commands / README / dynamo
• GitHub Check: Application Test Wheel (arm64)
• GitHub Check: Application Test Wheel (amd64)
• GitHub Check: Rust (arm64)
• GitHub Check: readme-commands / README / source
• GitHub Check: Python 3.13 compatibility
• GitHub Check: Release Artifact Contract (amd64)
• GitHub Check: Python 3.11 compatibility
• GitHub Check: Public API Rust (amd64)
• GitHub Check: Public API Rust (arm64)
• GitHub Check: Engine Golden Regression
• GitHub Check: Release Artifact Contract (arm64)
• GitHub Check: Rust feature modes
• GitHub Check: Rust (amd64)
• GitHub Check: Build simulation wheel (head)
• GitHub Check: Build simulation wheel (base)

🧰 Additional context used

📚 Code guidelines: 3

AGENTS.md — configured
REVIEW.md — configured
DEVELOPMENT.md — auto-discovered

📓 Path-based instructions: 8

Check unified CLI, Replay, Sweeper, and orchestration behavior together.

⚙️ CodeRabbit configuration file

Files:

• python/aisimulate/src/aisimulate/resources.py

Preserve deterministic Replay semantics and explicit validation.

⚙️ CodeRabbit configuration file

Files:

• crates/core/src/replay/loadgen/weka.rs

Require coverage of the changed behavior and its negative or boundary cases.

⚙️ CodeRabbit configuration file

Files:

• tests/test_weka_host_resources.py

Check commands, defaults, supported runtimes, public names, and claims against executable behavior.

⚙️ CodeRabbit configuration file

Files:

• docs/reference/local-resources.md

Read REVIEW.md before commenting.

⚙️ CodeRabbit configuration file

Files:

• docs/reference/local-resources.md
• python/aisimulate/src/aisimulate/resources.py
• crates/core/src/replay/loadgen/weka.rs
• tests/test_weka_host_resources.py

Source excerpt: Keep run logs, investigation notes, and benchmark results out of `docs/`; measured evidence belongs under `benchmarks/evidence/`.

📄 CodeRabbit inference engine (AGENTS.md)

Files:

• docs/reference/local-resources.md
• python/aisimulate/src/aisimulate/resources.py
• crates/core/src/replay/loadgen/weka.rs
• tests/test_weka_host_resources.py

Source excerpt: Binary parquet changes require a machine-readable summary of keys, shapes, units, row counts, coverage, and anomalies plus proof that a real consumer reaches the new data.

📄 CodeRabbit inference engine (REVIEW.md)

Files:

• docs/reference/local-resources.md
• python/aisimulate/src/aisimulate/resources.py
• crates/core/src/replay/loadgen/weka.rs
• tests/test_weka_host_resources.py

Applicability reference: AGENTS.md: # AGENTS **Lint:** `python/aisimulate/.venv/bin/ruff check --config python/aisimulate/pyproject.toml python/aisimulate tests` and `python/aisimulate/.venv/bin/ruff format --check --config python/aisimulat...

📄 CodeRabbit inference engine (DEVELOPMENT.md)

Files:

• python/aisimulate/src/aisimulate/resources.py

🪛 ast-grep (0.45.3)

tests/test_weka_host_resources.py

[info] 425-425: use jsonify instead of json.dumps for JSON output
Context: json.dumps(_play(play_id="added"))
Note: [CWE-116] Improper Encoding or Escaping of Output.

(use-jsonify)

🔀 Multi-repo context ai-dynamo/dynamo, ai-dynamo/aiconfigurator

Linked repositories findings

ai-dynamo/dynamo

• Dynamo pins aisimulate-core and re-exports its replay/loadgen APIs, making this change a direct downstream integration point. [::ai-dynamo/dynamo::Cargo.toml:61-62] [::ai-dynamo/dynamo::lib/mocker/src/loadgen/mod.rs:6-10]
• Weka replay intentionally passes no configured block size so embedded metadata is authoritative. Native loading still supports explicit-size validation, with an existing test requiring the mismatch error to remain unchanged. [::ai-dynamo/dynamo::components/src/dynamo/replay/simulation.py:392-405] [::ai-dynamo/dynamo::lib/mocker/src/replay/entrypoints.rs:72-89] [::ai-dynamo/dynamo::lib/mocker/src/replay/entrypoints.rs:3601-3643]
• Replay hashes cross into Dynamo KV-router placement as ReplayRequestHashes; exact hash parity with router-generated hashes is tested. Weka hash materialization should preserve these local and sequence-hash semantics. [::ai-dynamo/dynamo::lib/mocker/src/replay/offline/extensions/kv_router/mod.rs:156-183] [::ai-dynamo/dynamo::lib/mocker/src/loadgen/tests.rs:50-67]
• Dynamo’s Weka fixtures rely on embedded block_size, hash_id_scope: "local", and repeated source hash IDs. [::ai-dynamo/dynamo::components/src/dynamo/replay/tests/test_agentx_g2.py:65-84]

ai-dynamo/aiconfigurator

• This repository is only a migration/compatibility reference: its documentation identifies AISimulate as the new application/source owner and directs Sweeper integrations to AISimulate. No Weka or replay consumer was found. [::ai-dynamo/aiconfigurator::docs/aisimulate_migration.md:8-19] [::ai-dynamo/aiconfigurator::docs/aisimulate_migration.md:48-67]

🔇 Additional comments: 4

python/aisimulate/src/aisimulate/resources.py: 1

295-295: LGTM!

Also applies to: 338-339, 399-399, 403-418, 456-505, 525-525, 808-812, 817-817, 824-824, 842-842

tests/test_weka_host_resources.py: 1

11-11: LGTM!

Also applies to: 17-17, 315-461

docs/reference/local-resources.md: 1

289-294: LGTM!

python/aisimulate/.pre-commit-config.yaml: 1

10-10: LGTM!

Also applies to: 14-14

📝 Summary

Medium risk. Human attention should focus on:

1. Cache invalidation and source-identity checks.
2. Buffered spool parsing and malformed-tail recovery.
3. Memory-headroom enforcement after cached estimates.

Changed behavior

• Buffers normalized Weka row writes and reads.
• Reuses collision-resolved source hashes within each play.
• Streams Python admission hash arrays.
• Caches completed Weka corpus estimates with bounded eviction.
• Revalidates source identity and applies current admission limits.
• Adds regression coverage for parser, hash, cache, mutation, and headroom cases.
• Removes the Weka benchmark evidence file.
• No public entity declarations or serialized contracts changed.

Evidence supplied

The commit history contains the implementation commits, documentation commit, and final benchmark-file deletion. The reported sample preparation time decreased from 85.53 seconds to 28.02 seconds. Reported outputs, lifecycle digests, serialized contracts, and spool size remained unchanged. Fast CI, Full CI, and Simulation Performance reportedly passed.

Evidence still missing

No independent review findings or severity counts were supplied. Independent review remains pending.

No separate end-to-end ablation measures each final-stage optimization. The benchmark measures host preparation time, not modeled GPU performance or full-corpus end-to-end speedup.

Merge readiness

The change set has targeted regression coverage and documented performance evidence. Merge readiness is not established by CI results alone. Review the cache invalidation, source mutation handling, malformed spool recovery, and live-headroom checks before merging.

Weka replay now buffers row spooling and reuses normalized full-block source hashes within a play. Python Weka resource estimation streams and validates trace metadata, caches eligible estimates, and rechecks source identity and current admission limits.

Changes: Weka replay

Layer / File(s)	Summary
Buffered row spool crates/core/src/replay/loadgen/weka.rs	Row output uses a buffered writer that is flushed before rewind. Row reading processes newline-delimited JSON records and reports malformed trailing data after previously emitted rows. Tests cover large and partial rows, malformed tails, and repeated collection.
Source-hash reuse crates/core/src/replay/loadgen/weka.rs	Full-block source hashes reuse the first collision-resolved value within a play. Missing and partial blocks remain request-private. Tests cover reuse and request-private identities.

Weka resource admission

Layer / File(s)	Summary
Stream and qualify Weka trace estimates python/aisimulate/src/aisimulate/resources.py, tests/test_weka_host_resources.py	Trace inspection streams hash_ids values and validates them as unsigned 64-bit integers. Tests cover large arrays and invalid or truncated metadata.
Cache estimates and recheck admission python/aisimulate/src/aisimulate/resources.py, tests/test_weka_host_resources.py, docs/reference/local-resources.md, python/aisimulate/.pre-commit-config.yaml	A bounded cache reuses eligible estimates using trace identity and workload settings. Admission checks source stability and applies current headroom and profile limits. Tests cover budget handling, cache reuse and eviction, source changes, and inspection-time mutation. Ruff hooks now use the project configuration for linting and formatting.

Priority: ➖ Normal

Estimated code review effort: 3 (Moderate) | ~25 minutes

Merge Risk: ⚪ Minimal · up to 7fb21

No material issue was identified that needs resolution before merging. Weka admission continues to apply current memory limits when reusing a cached estimate.

Pre-merge checks: 5 · 2 · 1

Failed checks: 2 warnings · 1 inconclusive

Check name	Status	Explanation	Resolution
Description check		The description provides detailed motivation, behavior changes, review guidance, commands, benchmark results, limitations, and regression evidence. However, the medium-risk template requires a Codex r…	Add the Codex reviewed commit after review, complete the Tracking section, and document the expected-value derivation for exact numerical assertions before merging.
Modeling And Data Evidence		The PR changes Weka admission formulas and claims large before/after performance gains, but the final range does not retain the required reproducible evidence. docs/reference/local-resources.md prov…	Restore a versioned machine-readable evidence summary in the final PR. Include the exact base and candidate commits, workload identity or checksum, reproducible command and configuration, before/after measurements, output and lifecycle inva…
Review Evidence		The visible description names the measured command and results and reports boundary tests. Repository evidence also shows that only root workflows are hosted, while workflows under `python/aisimulate/…	Provide the complete PR description, including the full Evidence section and all CI links. Verify that each hosted-CI citation refers to a workflow under the repository-root .github/workflows/, not an imported workflow under `python/aisim…

Passed checks: 5 passed

Check name	Status	Explanation
Title check		The title precisely identifies a Weka replay preparation performance improvement. It is directly related to the main changes: buffering, reuse, streaming, and caching.
Linked Issues check		Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check		Check skipped because no linked issues were found for this pull request.
Cross-Layer Contract		No stale cross-layer contract is introduced. The diff changes private Weka spooling and Python admission internals; it does not change public Rust signatures, the Agentic Mooncake row schema, CLI inpu…
Compatibility Boundaries		No compatibility boundary is changed. The PR changes private Weka lowering/reading logic and private Python resource-estimation helpers. Public Weka exports and method signatures remain unchanged in `…

Full details: Cross-Layer Contract

Explanation

No stale cross-layer contract is introduced. The diff changes private Weka spooling and Python admission internals; it does not change public Rust signatures, the Agentic Mooncake row schema, CLI inputs, defaults, or serialized output fields. Rust tests cover normalized-hash reuse, large rows, repeated collection, and malformed spool tails (crates/core/src/replay/loadgen/weka.rs:2020-2071,2472-2496). Python tests cover cache keys, invalidation, budgets, live headroom, eviction, and profile behavior (tests/test_weka_host_resources.py:318-460). The resource-cache behavior is documented (docs/reference/local-resources.md:289-294). Existing Weka integration tests also compare direct native replay with materialized v2 graphs and lifecycle results (crates/core/tests/agentx_qualification.rs:118-153). Downstream Dynamo still consumes the unchanged Weka inputs and canonical schema; its compatibility adapter delegates Weka graph construction to the core loader (dynamo commit 1e5cb3eda45d100963a3f8cfab910841550d2546, components/src/dynamo/replay/simulation.py:396-425, docs/fern/pages/reference/components/dynosim-replay-cli-reference.mdx:230-279).

Full details: Compatibility Boundaries

Explanation

No compatibility boundary is changed. The PR changes private Weka lowering/reading logic and private Python resource-estimation helpers. Public Weka exports and method signatures remain unchanged in crates/core/src/replay/loadgen/mod.rs:53-57 and weka.rs:347-379. The Agentic Mooncake schema/version definitions are unchanged, and the new hash reuse remains inside the private normalized_hashes implementation (weka.rs:1425-1461). Cargo manifests, the Python pyproject.toml, bindings, and versions remain unchanged at 0.13.0. The single-wheel checks still require one unified wheel with no split aisimulate-core, aiconfigurator-core, or Dynamo dependencies (python/aisimulate/tests/cross_package/test_single_distribution.py:83-92). Dynamo's linked consumer continues to document Agentic Mooncake v2 and states that Weka lowering belongs to AISimulate (ai-dynamo/dynamo, commit 1e5cb3eda45d100963a3f8cfab910841550d2546, docs/fern/pages/reference/components/dynosim-replay-cli-reference.mdx:230-278). No new Dynamo dependency or standalone-boundary violation is introduced.

Full details: Description check

Explanation

The description provides detailed motivation, behavior changes, review guidance, commands, benchmark results, limitations, and regression evidence. However, the medium-risk template requires a Codex reviewed commit, while the description states that independent review is pending. The Tracking section is also missing, and expected-value derivation is not provided.

Resolution

Add the Codex reviewed commit after review, complete the Tracking section, and document the expected-value derivation for exact numerical assertions before merging.

Full details: Modeling And Data Evidence

Explanation

The PR changes Weka admission formulas and claims large before/after performance gains, but the final range does not retain the required reproducible evidence. docs/reference/local-resources.md provides formula provenance and qualifications (lines 255-275, 277-328), and the added tests check post-change arithmetic and cache behavior. However, the machine-readable benchmarks/evidence/weka-replay-runtime-20261021/summary.json was added in cd84034e, expanded with exact timings, configuration, invariants, and anomaly limits in 7442163f, then deleted by 7fb214aa; it is absent from the final tree. The remaining PR description is a prose table with private path placeholders, one timing sample per stage, and no standalone source/config or golden diff. The tests are not an independent oracle or held-out before/after comparison.

Resolution

Restore a versioned machine-readable evidence summary in the final PR. Include the exact base and candidate commits, workload identity or checksum, reproducible command and configuration, before/after measurements, output and lifecycle invariants, anomaly checks, and explicit sampling and attribution limits. Alternatively, provide an equivalent explained golden diff, independent oracle, or held-out comparison for the changed admission model and performance claims.

Full details: Review Evidence

Explanation

The visible description names the measured command and results and reports boundary tests. Repository evidence also shows that only root workflows are hosted, while workflows under python/aisimulate/.github/ are migration provenance (REVIEW.md:138-140; docs/ci/README.md:23-25). However, the PR description is truncated before its complete evidence section. The omitted text may contain the required local/fake-runner/parity/production distinctions and the exact CI citations, so the check cannot determine whether inactive nested workflows were cited as hosted evidence.

Resolution

Provide the complete PR description, including the full Evidence section and all CI links. Verify that each hosted-CI citation refers to a workflow under the repository-root .github/workflows/, not an imported workflow under python/aisimulate/.github/.

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

_{Comment @coderabbitai help to get the list of available commands.}

[tedzhouhk]

Reviewed 7fb214aa; no new correctness issue found in the exercised paths. The buffered spool, per-play collision-checked hash memoization, streaming scan, and bounded estimate cache preserve the tested semantics. Cache identity/invalidation, malformed input, eviction, and current-headroom admission were checked.

Validation on the exact head: 164 Python resource tests and 207 Rust load-generator tests passed; 600 cold/warm/budget comparisons over 120 generated corpora also passed. The Python suites used a freshly built head-native extension.

The reported ~3x improvement is end-to-end command wall time dominated by preparation savings, not a 3x core-replay or modeled GPU-throughput improvement. I did not independently reproduce the private-trace performance measurement.