EngineConfig.max_model_len is an optional positive prompt-plus-output token
limit for vLLM, TRT-LLM, and SGLang. Recipe adapters can map TRT-LLM
max_seq_len and SGLang context_length to this field.
- A prompt at or above the limit is rejected before prefill computation.
- A decode destination rejects such a prompt before queuing the handoff or reserving KV blocks, even when destination admission is deferred.
- Terminal rejection removes the request's outstanding Belady input demand before subsequent cache admission and eviction.
- Generation stops when prompt plus output reaches the limit, including speculative bursts and requests with explicit output token IDs. Reports retain the requested output length and count only tokens actually generated.
- TRT-LLM completion reservations and SGLang output reservations use the capped output budget. Physical KV capacity remains a separate constraint.
- The limit applies to aggregated workers and both roles in disaggregated replay. An unset limit preserves the existing backend behavior.
The prediction compiler and recommendation deployment builder pass explicit
engine.context_length values to every backend. Prediction with
context_length: max retains its existing behavior: vLLM resolves model
metadata, while TRT-LLM and SGLang leave the scheduler limit unset.
This is the simulator's normalized context-limit contract. It does not model backend-version-specific frontend validation margins or automatic prompt truncation. Successful replay alone does not establish silicon timing accuracy. See the SGLang GPU parity observations for stricter boundary behavior; they do not establish a fixed token offset across versions or configurations.
For native SDK estimates, estimate_kv_cache, estimate_num_gpu_blocks, and
KVCacheEstimator.from_request resolve an omitted backend_version (or None)
through the backend's current data alias for the requested system and data
roots. Explicit version selectors are preserved. FPM profile estimates still
require the profile's literal backend version.
For Mistral3's gated Pixtral encoder, the analytical activation estimate includes the replicated residual alongside the gate and up intermediates. Encoder tensor parallelism shards those intermediates; encoder data parallelism keeps a full tower on each rank. This remains an analytical lower bound, not a measured peak.
SGLang's native estimator treats mem_fraction_static as a static weights/KV
pool. Peak activation/workspace estimates remain visible in
memory_breakdown.activations_bytes, but are not deducted from that pool;
transient execution headroom is already outside the static fraction. Increasing
the prefill token budget alone therefore does not reduce SGLang KV capacity.
Resident runtime/communication estimates reduce the pre-load free-memory pool
before applying the fraction; weights are deducted afterward. The budget is
(capacity - resident_overhead) * mem_fraction_static - weights, less any
explicit additional graph reservation. For ordinary SGLang DeepSeek-V3/R1
(non-CP, non-PP, non-speculative, non-large-EP), the estimator also respects
checkpoint dense/MoE layer counts and TP-sharded embeddings independently of
the unchanged timing graph. Other model layouts retain their prior weight
accounting.
vLLM and TRT-LLM continue to deduct activation memory under their own budget
semantics. No measured server capacity is required by this calculation.
estimate_kv_cache and estimate_num_gpu_blocks accept
cuda_graph_reserved_bytes=<rank-local bytes>. The value must be a
non-negative integer no greater than 2**53 and defaults to zero. It is treated
as fixed non-KV memory before the backend-specific KV fraction is applied. For
SGLang, it is an additional reservation beyond graph/runtime headroom already
encoded by mem_fraction_static. Native estimates return it as
memory_breakdown.cuda_graph_reserved_bytes; naive fallback estimates apply it
but keep memory_breakdown=None because the other components are unavailable.
The Rust KvCacheEstimateRequest exposes the same field. Engine replay accepts
the value in its engine arguments and carries it through native AIC capacity
rematerialization, so the Python estimator and native scheduler use the same
rank-local KV capacity. The aisim predict YAML exposes it at
engine.workers.<role>.kv_cache.capacity.cuda_graph_reserved_bytes. This API
does not estimate the reservation; callers must supply a value from a source
they trust.
Serialized Rust requests and estimates that omit the field remain compatible because deserialization defaults it to zero. See Rust literal migration for required source changes to request and memory-breakdown structs.
FPM resources support cache_layout: linear with a positive
kv_bytes_per_token, or cache_layout: grouped with a nonempty cache_groups
list and no scalar token rate. Existing linear profiles keep their serialized
shape. Python exports FpmCacheGroup from aisim.fpm_profile and
aisim_core.fpm_profile; Rust exports it from aisim_core::perfmodel.
An explicit profile can use cache_layout: unmodeled when the runtime cache
allocator has no AIS model, such as Mamba checkpoint retention.
This layout requires observed runtime_memory and omits both
kv_bytes_per_token and cache_groups. Store the native cache descriptors in
the referenced provenance artifact. Canonical standalone timing queries can
consume this profile. Cache sizing and replay reject it. An observed allocation
does not establish how request lengths, cache reuse, or speculation consume it.
Memory can be pending, declared, or observed at runtime. The four legacy fields
weights_bytes, activations_bytes, runtime_overhead_bytes, and
comm_overhead_bytes are optional. A complete set retains the existing non-KV
budget calculation. A partial set is planning evidence only: missing values are
not zero, and cache sizing or replay fails with an instruction to finalize
runtime memory. Schema validation, collection planning, and direct timing queries
can use a pending profile. Omitted fields remain absent from serialized output.
Alternatively, resources.runtime_memory records the selected deployment's
initialized cache allocation:
{
"kv_cache_bytes": 1073741824,
"gpu_memory_utilization": 0.9,
"max_model_len": 32768,
"provenance": "Verified collection worker initialization; see saved evidence."
}The byte count is a positive integer no greater than 2**53; utilization is
finite and in (0, 1]; context and provenance are required. Runtime memory
cannot coexist with any of the four legacy non-KV fields. Python exposes
FpmRuntimeMemoryProfile and resource properties memory_source
(pending, declared, or runtime) and memory_ready; require_memory()
rejects pending resources. Rust exposes FpmRuntimeMemoryConfig and
FpmResourceConfig::require_memory(). See Rust literal migration
for resource construction changes.
A runtime memory record can also keep the prefix-caching settings of the
runtime probe
that observed it: enable_prefix_caching (a boolean), mamba_cache_mode
(none or align) and hash_block_size_tokens (the positive hash block size,
in tokens, that the runtime resolved). All three are optional, and an absent
field is left out of the serialized record. align requires
enable_prefix_caching to be true. vLLM's all mode is not modeled and is
rejected. Rust exports the mode type as FpmMambaCacheMode. These fields are
evidence of the probe's settings. Selection, sizing and replay do not read them
yet. The simulated deployment decides prefix caching. The vLLM 0.33.0 runtime
probe fills these fields from the prefill phase of an aggregated or prefill
role. A decode role takes them from its decode phase.
The import of a vLLM 0.33.0 runtime probe record also writes
runtime_framework_version: the version of the runtime that observed the cache
groups and the prefix-caching settings, such as 0.33.0. It is optional,
nonempty text, and an absent field is left out of the serialized record. It
can differ from the deployment's backend_version, because
onboard probe-runtime keeps an explicit dataset label and probes the runtime
of the image. The import of a record without hybrid evidence, a vLLM 0.27.0,
0.28.0 or 0.32.0 record, does not write it,
and the linear layout of a measured cache leaves it out with the groups.
Sizing does not read it. A profile imported before this field existed has
none; see the migration guide.
Each group has a unique name, kind (attention, convolution or state),
positive num_layers, block_size_tokens, and page_size_bytes, and an optional
positive sliding_window. An omitted or null window retains full history;
convolution groups require a window. In an attention or convolution group,
page_size_bytes is the rank-local aggregate for all layers in the group,
including runtime padding. Do not
multiply it by num_layers again. Runtime block sizes and padding are deployment
inputs; model geometry alone does not establish them. See the
grouped-profile review workflow.
A state group holds fixed per-request state, such as the recurrent state of
linear-attention layers. It has no sliding_window, and no other kind accepts
blocks_per_request. Three fields have their own meaning in a state group:
page_size_bytesis one pool block: every byte that one block ID reserves in the shared pool. This can be more than the bytes of the group's own layers. vLLM pads each Mamba page to the attention page, and the group with the most bytes per block sets the size of a pool block.block_size_tokensis the block size of vLLM's state spec, as the runtime probe records it. For a Mamba spec, that is the Mamba block: the scheduler block inalignmode andmax_model_lenin modenone. For a ring spec, such as an indexer tail, it is the ring block. Sizing and replay do not read this field yet. Planned grouped prefix caching will read it as the Mamba block ofalignmode. So it stays part of the group shape that onboarding compares with the runtime probe.blocks_per_requestis the positive number of pool blocks that a running request holds between block boundaries. For a Mamba spec, that is1 + k, wherekis the spec'snum_speculative_blocks. Grouped execution forbids speculative decoding, sokis 0 there. A ring spec, such asCircularBufferSpecin vLLM 0.33.0, holds 1 block for the whole life of the request in every mode. The count does not change with the request length. When a request crosses a block boundary inalignmode, a Mamba group also holds the previous block for one or two steps. vLLM's startup sizing in that mode counts2 + k + cblocks per request for a Mamba group, wherecis the spec'snum_prefill_checkpoint_blocks. It counts 1 block for a ring group. See vLLM v0.33.0kv_cache_interface.py:1135-1141andsingle_type_kv_cache_manager.py:1640-1657.
A state group can also hold two optional fields from the runtime probe
import. A profile imported before these fields existed has neither of them, and
an absent field is left out of the serialized group. Sizing and replay do not
read them yet.
prefill_checkpoint_blocksis the number of prefill checkpoint blocks of a Mamba spec: 0 or 1. Only astategroup can have it. A vLLM 0.33.0 import writes it for every Mamba group. It refuses a record with another count. Inalignmode, vLLM's startup sizing counts these blocks ascin2 + k + c.prefill_checkpoint_alignmentis the token alignment of that checkpoint. It must be positive, and it needsprefill_checkpoint_blocksof 1. A vLLM 0.33.0 import writes it for a Mamba group with one checkpoint block. It refuses a record whose alignment does not match the count. In vLLM 0.33.0, a GLM-5.3-Flash KDA layer that prefills with FlashKDA has one checkpoint block with an alignment of 16 tokens. With the Triton kernel it has no checkpoint block and no alignment. See vLLM v0.33.0kda.py:168-176andkda_checkpoint.py:9-10.
Every group can also hold two more optional fields. A profile imported before these fields existed has neither of them, and an absent field is left out of the serialized group. Sizing and replay do not read them yet.
-
prefix_cacheabletells whether vLLM's manager for the group takes part in prefix caching. A ring spec (CircularBufferSpec) does not, and a uniform group takes part only when every inner spec does. Import writes it for every group except a group that holds aSlidingWindowMLASpeclayer. Such a group takes no part if it replays a bounded window, and the probe does not record that setting. See vLLM v0.33.0kv_cache_interface.py:998-1000,:1027-1029and:1305-1307. -
transfer_bytes_per_blockis the bytes that vLLM's P/D connector (NIXL) reads for one block of the group in a transfer on one TP rank, for the vLLM version of the deployment. It is positive and at most 2^53, and it can be larger thanpage_size_bytes, because the connector reads a block in regions that layers of other groups can share. A hand-made profile may state it only for a group whose transfer size has been measured.To measure the full-attention value and the combined transfer size of the other groups, send two prompts of different length through a P/D pair, one at a time, and read the change of
vllm:nixl_bytes_transferred_sumon the decode server for each. Divide each change by the change ofvllm:nixl_bytes_transferred_count, because each TP rank makes its own transfer and the field holds the bytes of one rank. The difference between the two results, divided by the difference in full-attention blocks that move, is the value of the full-attention group. The rest of one transfer is the sum of the values of the other groups when each of them moves one block, as a state group does. With multiple state groups, this determines only their combined transfer size.Per-group values for models with several state groups are not supported by this procedure or the current runtime probe. Leave
transfer_bytes_per_blockunset for those groups until per-group transfer measurements or additional runtime-observer evidence identifies the bytes read for each group on one TP rank. Do not split the combined size evenly or use the page sizes as substitutes.The probe import of a vLLM 0.33.0 record writes it for no group, because what the P/D connector of vLLM 0.33.0 reads for a block depends on facts that the record does not hold. The connector registers one region per layer address and lets layers of different groups share a region. In a model with Mamba layers, it reads a block of an attention group, other than the ring, in every region. The length of a region of a layer that is not a Mamba layer depends on the layout: the padded page divided by a kernel-block ratio that the connector chooses at start, or the whole storage row of a block. A Mamba layer moves its state without padding, and a
tp_replicatedMamba layer moves its whole page. The record holds neither the ratio nortp_replicated. See vLLM v0.33.0base_worker.py:1744-1756,:194-249,:1598-1603,:1693-1719,:1063-1082,:2013-2033and:1774-1779. A P/D model of a grouped profile for vLLM 0.33.0 needs these facts from the runtime observer first. A claim inresource_overrides.cache_groupsthat states this field is refused, because the probe evidence holds no value to compare it with.
Use RustForwardPassPerfModel.estimate_cache_budget(config, budget) with the
same canonical ForwardPassPerfModelConfig used for timing. Rust exposes
ForwardPassPerfModelConfig::estimate_cache_budget(&request) and
ForwardPassPerfModel::estimate_cache_budget(&request), taking
FpmCacheBudgetRequest and returning FpmCacheBudget. The config method checks
the exact profile deployment and scheduler envelope without constructing a
timing model, loading timings, or building an operation graph. Grouped planning
requires the native extension; it does not require GPU access or measured data.
| Budget field | Meaning |
|---|---|
total_gpu_capacity_bytes |
Positive rank-local GPU capacity. |
memory_fraction_kind, memory_fraction_value |
of_total and the fraction of GPU memory available to the worker. |
max_num_tokens, max_batch_size |
Positive rank-local scheduler limits within a declared envelope; exact recorded values for runtime memory. |
context_length |
Optional positive request bound, at most the profile context and runtime max_model_len when present; defaults to the profile context. |
request_occupancy_tokens |
Optional positive logical request length within the selected context, for a separate steady decode footprint with one new token per forward pass. |
cuda_graph_reserved_bytes |
Separate nonnegative reservation for declared memory; must be zero for runtime memory. |
tolerance_fraction |
Optional fraction in [0, 1) deducted from the resulting cache byte budget. |
The result includes total_kv_size_bytes, memory_breakdown,
resource_provenance, cache_layout, cache_groups, and
request_peak_cache_bytes. The latter is a conservative single-request bound
including block alignment and the configured prefill chunk, not a reservation
for every scheduler slot. A smaller context changes this peak bound, not the
declared non-cache resource bounds. tolerance_adjusted, when present, provides
the reduced byte budget. For grouped caches, kv_size_per_token_bytes,
total_kv_size_tokens, and the adjusted token capacity are null (None in Python).
There is no equivalent scalar token capacity.
When request_occupancy_tokens is supplied, the result also includes
request_occupancy_cache_bytes. KV-relative synthetic loads use this native
footprint while retaining the actual scheduler settings and context for resource
admission. Observed linear pools retain their recorded capacity even when timing
coverage is smaller; uncovered queries report timing errors.
Runtime memory uses kv_cache_bytes directly and returns
memory_breakdown: null (Option::None in Rust). It does not invent activation
or other non-KV components. The requested memory fraction must exactly match the
recorded utilization; the device budget must contain the observed cache pool.
A different device capacity argument never rescales the pool or qualifies a new
deployment. Changed scheduler settings require new runtime evidence. Explicit
scalar cache-capacity overrides are rejected for runtime profiles in prediction,
recommendation and replay; use the recorded allocation. A tolerance margin may
reduce it through the canonical budget API.
Python estimate_kv_cache(..., fpm_profile=..., context_length=...) delegates
grouped and runtime-memory profiles to this native budget method.
estimate_num_gpu_blocks also accepts context_length for runtime validation;
it and the
legacy Rust scalar KvCacheEstimate transport reject grouped profiles; use the
groups and shared byte budget instead. Existing complete linear declarations
keep their budget calculation and serialized shape.
Grouped native execution supports cold aggregated vLLM, PP1/CP1, HBM-only cache,
non-speculative decoding and prefix_caching: false. Prefix reuse, host/G3
offload, disaggregation and fixed scalar cache capacity are rejected. Allocation
is atomic across all groups against one shared rank-local byte budget. A window
of W keeps the blocks covering the last W - 1 computed tokens and all tokens
scheduled for the next forward. Expired completed pages are evicted before
subsequent forwards; temporary prefill pages remain charged through completion.
Full-attention groups retain full history. A state group holds its
blocks_per_request blocks from the request's first scheduled forward until the
request finishes, is cancelled or is preempted. A resumed request reserves them
again once. This physical retention never truncates logical request progress or
the context coordinates sent to FPM timing queries.
The Rust Replay observer's ReplaySchedulerMetricsSnapshot reports optional
kv_cache_used_bytes and kv_cache_capacity_bytes for grouped caches; its
active_cache_usage and physical_cache_usage use byte occupancy. active_blocks
counts resident group pages, while total_blocks is zero because heterogeneous
pages have no scalar block capacity. Use the byte fields for capacity comparisons.
These fields are currently exposed through the Rust observer API, not the Python
JSON replay runtime. Linear snapshots retain their existing block metrics.
Cache reuse, eviction, and offload lifecycle are documented in Replay KV cache.
The mamba_align layout describes one full-attention group and padded
Mamba groups that share one physical pool. Supply observed runtime memory,
physical page bytes, block length, state-group count, prefix unit, and
speculative state count. Set mamba_cache.state_storage: full_state explicitly.
The count must match the measured draft depth. Internal prefill checkpoint
slots are unsupported.
Field under resources.mamba_cache |
Meaning |
|---|---|
state_storage |
Required full_state. Other policies are rejected. |
block_size_tokens |
Physical attention block length, at least two tokens. |
page_size_bytes |
Padded page size shared by attention and state groups. |
state_groups |
Number of state pages in one complete working state. |
prefix_match_unit |
Positive divisor of the block length. |
num_speculative_states |
Full-state scratch copies, equal to the measured draft depth, from zero through five. |
The native budget returns replay_cache. This descriptor excludes one reserved
null page and includes working-state and speculative scratch reservations.
Do not interpret its allocation byte rate as logical KV bytes per token.
The profile does not expose an aggregate token capacity.
Use aggregated vLLM replay with the measured block length and scheduler limits. Keep prefix caching enabled. Fixed capacity overrides, offload, and disaggregated FPM replay are unsupported for this layout. Original timing imports remain valid as timing-only evidence.
ReplaySSM and RecoverSSM lay out and size their states differently, so
profile them as unmodeled for timing only; do not relabel them full_state.
Lookups follow vLLM revision db9527a46873454610df6dbedf79a36d6bf1a7f6, which
applies the logits cap before the MTP hash-unit drop, so some exactly aligned
prompts miss the cached tail.