Skip to content

fix: reject unsupported pipeline parallelism in engine replay - #545

Draft
milesial wants to merge 1 commit into
ai-dynamo:mainfrom
milesial:alexandrem/fix-replay-pipeline-validation
Draft

milesial wants to merge 1 commit into
ai-dynamo:mainfrom
milesial:alexandrem/fix-replay-pipeline-validation

Conversation

@milesial

Copy link
Copy Markdown
Contributor

Why and what changed

Engine replay accepts pipeline parallelism even though it has no pipeline-stage scheduler or transfer model. A requested PP value can affect performance-model inputs and capacity while replay still runs a single-stage worker.

Reject pipeline sizes other than one at the shared runner materialization boundary, before KV capacity estimation. Cover deployment parallelism, flat and nested legacy rank arguments, and canonical timing configuration aliases for aggregated, prefill, and decode roles. Update the existing capacity test to use supported PP1.

Review map

  • Risk level: medium.
  • Start with _materialize_engine_role in python/aisimulate/src/aisimulate/runner.py.
  • Public or serialized contract changed: no schema change. Unsupported engine-replay PP values now raise a clear error.
  • Compatibility or rollback concern: callers that previously supplied PP greater than one must select PP1. Standalone performance-model APIs retain their existing behavior.

Evidence

  • Regression coverage exercises five configuration sources across all three worker roles.
  • The negative tests require rejection before the capacity estimator is called.
  • Existing runner tests cover supported PP1 behavior.
  • CI results are pending.

Modeling or data provenance

N/A. This enforces the documented engine-replay support boundary and adds no pipeline model.

Signed-off-by: Alexandre Milesi <milesial@users.noreply.github.com>
@coderabbitai

coderabbitai Bot commented Oct 21, 2026

Copy link
Copy Markdown

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

  • Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Comment @coderabbitai help to get the list of available commands.

@liangqianxing liangqianxing left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Small, non-blocking follow-up at 6035c1ecfaa1c9e82f2793098fdcc7dd494fab9e: the runner guard correctly rejects PP2, but public prediction can still estimate capacity before reaching it, contrary to the documented ordering.

With workers.aggregated.parallelism.pipeline: 2, fixed or polynomial timing, default KV capacity, and no state cache, run_prediction first calls prediction_to_replay_spec. During compilation, _worker_engine_args calls materialize_aic_num_gpu_blocks (compiler.py:694–696), reaching estimate_num_gpu_blocks(pp_size=2) before this guard.

I verified this ordering on the exact Python head with a recording estimator. An injected estimator failure escapes before the PP diagnostic; returning capacity reaches the expected PP rejection. Two unmocked offline checks using the local tiny-model fixture, with default and explicit capacity, also produced the intended rejection; the real estimator succeeded in the default-capacity case. PP2 never reached the native runtime in these checks. This demonstrates ordering and injected failure propagation; no real metadata failure or incorrect replay result was observed.

A shared engine-stack preflight before compiler capacity estimation, plus a public run_prediction regression for these configurations, would close the gap. The current direct-materializer tests cover the later boundary.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants