Skip to content

Latest commit

 

History

History
73 lines (60 loc) · 4.79 KB

README.md

File metadata and controls

73 lines (60 loc) · 4.79 KB

AISim documentation

AISim predicts LLM serving behavior and searches deployment configurations offline. Start with the quickstart to predict a deployment, search under a GPU budget, and evaluate the selected configuration.

Three layers

flowchart LR
    P["Performance model<br/>forward latency and memory"] --> R["Replay<br/>request and worker simulation"]
    R --> S["Sweeper<br/>candidate evaluation and search"]
    A["Framework adapters<br/>configuration, execution, outputs"] -.-> R
    A -.-> S
Loading

The arrows show how capabilities compose. In a typical prediction-driven search, Sweeper invokes a runner, Replay schedules work in virtual time, and the engine queries the performance model for forward timing. Some analytical evaluation paths do not execute the native Replay loop; their scope is stated in the component documentation.

Layer Input Output and responsibility
Performance model Model, hardware, backend identity, parallelism, batch features, performance data Forward-pass latency and memory estimates, with explicit estimator mode and coverage
Replay Workload, deployment topology, scheduling and cache policies, performance model Request timelines, throughput, latency, cache behavior and completion evidence
Sweeper Search domains, workload, optimization goal and constraints Evaluated candidates, ranking/Pareto results and concrete recommended configurations

Adapters are integration boundaries, not a fourth simulation layer. A serving framework can supply configuration, execution and output contracts; native composition can customize placement and scaling without owning the event loop. The Rust crate's engine, replay, and perfmodel namespaces describe its implementation, while the product layers above describe user capabilities.

Choose a task

Task Read
Install and run a first prediction/search Installation, quickstart, understand results
Choose or query an estimator Model configuration, memory, Python API, Rust API
Collect FPM data for a model and target FPM self-service, implementation/reference, examples
Replay traffic or AgentX sessions Workloads, AgentX quickstart, engine, Dynamo integration
Search and materialize a deployment Search space, goals, results, deployment generation
Implement an integration boundary Adapter ABI reference
Extend a component Performance modeling, Collector, Replay, Sweeper SDK
Look up shared syntax CLI, top-level configuration, local resources
Maintain checks and releases CI, release, accuracy publication, upstream sync
Use or migrate AIC workflows AIC backward compatibility

Support and accuracy

Performance-model coverage and Replay features answer different questions. The Feature Support Matrix summarizes Replay features per engine. The public FPE matrix probes a specific strict-native op-level query surface. It does not certify all FPM methods, orchestration paths, or end-to-end accuracy.

Use the FPM accuracy and E2E accuracy dashboards for measurement-backed results within their recorded scope. A configuration that runs successfully is not necessarily an accurate prediction. Replay and Sweeper remain experimental; validate shortlisted configurations on the target serving system before capacity or SLA decisions.

Examples on main can require capabilities newer than published wheels. Follow the installation guide's version and optional Dynamo pairing rules.