Skip to content

Latest commit

 

History

History
392 lines (345 loc) · 25 KB

memory.md

File metadata and controls

392 lines (345 loc) · 25 KB

GPU memory and cache capacity

Engine context limits

EngineConfig.max_model_len is an optional positive prompt-plus-output token limit for vLLM, TRT-LLM, and SGLang. Recipe adapters can map TRT-LLM max_seq_len and SGLang context_length to this field.

  • A prompt at or above the limit is rejected before prefill computation.
  • A decode destination rejects such a prompt before queuing the handoff or reserving KV blocks, even when destination admission is deferred.
  • Terminal rejection removes the request's outstanding Belady input demand before subsequent cache admission and eviction.
  • Generation stops when prompt plus output reaches the limit, including speculative bursts and requests with explicit output token IDs. Reports retain the requested output length and count only tokens actually generated.
  • TRT-LLM completion reservations and SGLang output reservations use the capped output budget. Physical KV capacity remains a separate constraint.
  • The limit applies to aggregated workers and both roles in disaggregated replay. An unset limit preserves the existing backend behavior.

The prediction compiler and recommendation deployment builder pass explicit engine.context_length values to every backend. Prediction with context_length: max retains its existing behavior: vLLM resolves model metadata, while TRT-LLM and SGLang leave the scheduler limit unset.

This is the simulator's normalized context-limit contract. It does not model backend-version-specific frontend validation margins or automatic prompt truncation. Successful replay alone does not establish silicon timing accuracy. See the SGLang GPU parity observations for stricter boundary behavior; they do not establish a fixed token offset across versions or configurations.

KV-cache capacity reservation

For native SDK estimates, estimate_kv_cache, estimate_num_gpu_blocks, and KVCacheEstimator.from_request resolve an omitted backend_version (or None) through the backend's current data alias for the requested system and data roots. Explicit version selectors are preserved. FPM profile estimates still require the profile's literal backend version.

For Mistral3's gated Pixtral encoder, the analytical activation estimate includes the replicated residual alongside the gate and up intermediates. Encoder tensor parallelism shards those intermediates; encoder data parallelism keeps a full tower on each rank. This remains an analytical lower bound, not a measured peak.

SGLang's native estimator treats mem_fraction_static as a static weights/KV pool. Peak activation/workspace estimates remain visible in memory_breakdown.activations_bytes, but are not deducted from that pool; transient execution headroom is already outside the static fraction. Increasing the prefill token budget alone therefore does not reduce SGLang KV capacity. Resident runtime/communication estimates reduce the pre-load free-memory pool before applying the fraction; weights are deducted afterward. The budget is (capacity - resident_overhead) * mem_fraction_static - weights, less any explicit additional graph reservation. For ordinary SGLang DeepSeek-V3/R1 (non-CP, non-PP, non-speculative, non-large-EP), the estimator also respects checkpoint dense/MoE layer counts and TP-sharded embeddings independently of the unchanged timing graph. Other model layouts retain their prior weight accounting. vLLM and TRT-LLM continue to deduct activation memory under their own budget semantics. No measured server capacity is required by this calculation.

estimate_kv_cache and estimate_num_gpu_blocks accept cuda_graph_reserved_bytes=<rank-local bytes>. The value must be a non-negative integer no greater than 2**53 and defaults to zero. It is treated as fixed non-KV memory before the backend-specific KV fraction is applied. For SGLang, it is an additional reservation beyond graph/runtime headroom already encoded by mem_fraction_static. Native estimates return it as memory_breakdown.cuda_graph_reserved_bytes; naive fallback estimates apply it but keep memory_breakdown=None because the other components are unavailable.

The Rust KvCacheEstimateRequest exposes the same field. Engine replay accepts the value in its engine arguments and carries it through native AIC capacity rematerialization, so the Python estimator and native scheduler use the same rank-local KV capacity. The aisim predict YAML exposes it at engine.workers.<role>.kv_cache.capacity.cuda_graph_reserved_bytes. This API does not estimate the reservation; callers must supply a value from a source they trust.

Serialized Rust requests and estimates that omit the field remain compatible because deserialization defaults it to zero. See Rust literal migration for required source changes to request and memory-breakdown structs.

FPM profile cache groups and byte budgets

FPM resources support cache_layout: linear with a positive kv_bytes_per_token, or cache_layout: grouped with a nonempty cache_groups list and no scalar token rate. Existing linear profiles keep their serialized shape. Python exports FpmCacheGroup from aisim.fpm_profile and aisim_core.fpm_profile; Rust exports it from aisim_core::perfmodel.

An explicit profile can use cache_layout: unmodeled when the runtime cache allocator has no AIS model, such as Mamba checkpoint retention. This layout requires observed runtime_memory and omits both kv_bytes_per_token and cache_groups. Store the native cache descriptors in the referenced provenance artifact. Canonical standalone timing queries can consume this profile. Cache sizing and replay reject it. An observed allocation does not establish how request lengths, cache reuse, or speculation consume it.

Memory can be pending, declared, or observed at runtime. The four legacy fields weights_bytes, activations_bytes, runtime_overhead_bytes, and comm_overhead_bytes are optional. A complete set retains the existing non-KV budget calculation. A partial set is planning evidence only: missing values are not zero, and cache sizing or replay fails with an instruction to finalize runtime memory. Schema validation, collection planning, and direct timing queries can use a pending profile. Omitted fields remain absent from serialized output.

Alternatively, resources.runtime_memory records the selected deployment's initialized cache allocation:

{
  "kv_cache_bytes": 1073741824,
  "gpu_memory_utilization": 0.9,
  "max_model_len": 32768,
  "provenance": "Verified collection worker initialization; see saved evidence."
}

The byte count is a positive integer no greater than 2**53; utilization is finite and in (0, 1]; context and provenance are required. Runtime memory cannot coexist with any of the four legacy non-KV fields. Python exposes FpmRuntimeMemoryProfile and resource properties memory_source (pending, declared, or runtime) and memory_ready; require_memory() rejects pending resources. Rust exposes FpmRuntimeMemoryConfig and FpmResourceConfig::require_memory(). See Rust literal migration for resource construction changes.

A runtime memory record can also keep the prefix-caching settings of the runtime probe that observed it: enable_prefix_caching (a boolean), mamba_cache_mode (none or align) and hash_block_size_tokens (the positive hash block size, in tokens, that the runtime resolved). All three are optional, and an absent field is left out of the serialized record. align requires enable_prefix_caching to be true. vLLM's all mode is not modeled and is rejected. Rust exports the mode type as FpmMambaCacheMode. These fields are evidence of the probe's settings. Selection, sizing and replay do not read them yet. The simulated deployment decides prefix caching. The vLLM 0.33.0 runtime probe fills these fields from the prefill phase of an aggregated or prefill role. A decode role takes them from its decode phase.

The import of a vLLM 0.33.0 runtime probe record also writes runtime_framework_version: the version of the runtime that observed the cache groups and the prefix-caching settings, such as 0.33.0. It is optional, nonempty text, and an absent field is left out of the serialized record. It can differ from the deployment's backend_version, because onboard probe-runtime keeps an explicit dataset label and probes the runtime of the image. The import of a record without hybrid evidence, a vLLM 0.27.0, 0.28.0 or 0.32.0 record, does not write it, and the linear layout of a measured cache leaves it out with the groups. Sizing does not read it. A profile imported before this field existed has none; see the migration guide.

Each group has a unique name, kind (attention, convolution or state), positive num_layers, block_size_tokens, and page_size_bytes, and an optional positive sliding_window. An omitted or null window retains full history; convolution groups require a window. In an attention or convolution group, page_size_bytes is the rank-local aggregate for all layers in the group, including runtime padding. Do not multiply it by num_layers again. Runtime block sizes and padding are deployment inputs; model geometry alone does not establish them. See the grouped-profile review workflow.

A state group holds fixed per-request state, such as the recurrent state of linear-attention layers. It has no sliding_window, and no other kind accepts blocks_per_request. Three fields have their own meaning in a state group:

  • page_size_bytes is one pool block: every byte that one block ID reserves in the shared pool. This can be more than the bytes of the group's own layers. vLLM pads each Mamba page to the attention page, and the group with the most bytes per block sets the size of a pool block.
  • block_size_tokens is the block size of vLLM's state spec, as the runtime probe records it. For a Mamba spec, that is the Mamba block: the scheduler block in align mode and max_model_len in mode none. For a ring spec, such as an indexer tail, it is the ring block. Sizing and replay do not read this field yet. Planned grouped prefix caching will read it as the Mamba block of align mode. So it stays part of the group shape that onboarding compares with the runtime probe.
  • blocks_per_request is the positive number of pool blocks that a running request holds between block boundaries. For a Mamba spec, that is 1 + k, where k is the spec's num_speculative_blocks. Grouped execution forbids speculative decoding, so k is 0 there. A ring spec, such as CircularBufferSpec in vLLM 0.33.0, holds 1 block for the whole life of the request in every mode. The count does not change with the request length. When a request crosses a block boundary in align mode, a Mamba group also holds the previous block for one or two steps. vLLM's startup sizing in that mode counts 2 + k + c blocks per request for a Mamba group, where c is the spec's num_prefill_checkpoint_blocks. It counts 1 block for a ring group. See vLLM v0.33.0 kv_cache_interface.py:1135-1141 and single_type_kv_cache_manager.py:1640-1657.

A state group can also hold two optional fields from the runtime probe import. A profile imported before these fields existed has neither of them, and an absent field is left out of the serialized group. Sizing and replay do not read them yet.

  • prefill_checkpoint_blocks is the number of prefill checkpoint blocks of a Mamba spec: 0 or 1. Only a state group can have it. A vLLM 0.33.0 import writes it for every Mamba group. It refuses a record with another count. In align mode, vLLM's startup sizing counts these blocks as c in 2 + k + c.
  • prefill_checkpoint_alignment is the token alignment of that checkpoint. It must be positive, and it needs prefill_checkpoint_blocks of 1. A vLLM 0.33.0 import writes it for a Mamba group with one checkpoint block. It refuses a record whose alignment does not match the count. In vLLM 0.33.0, a GLM-5.3-Flash KDA layer that prefills with FlashKDA has one checkpoint block with an alignment of 16 tokens. With the Triton kernel it has no checkpoint block and no alignment. See vLLM v0.33.0 kda.py:168-176 and kda_checkpoint.py:9-10.

Every group can also hold two more optional fields. A profile imported before these fields existed has neither of them, and an absent field is left out of the serialized group. Sizing and replay do not read them yet.

  • prefix_cacheable tells whether vLLM's manager for the group takes part in prefix caching. A ring spec (CircularBufferSpec) does not, and a uniform group takes part only when every inner spec does. Import writes it for every group except a group that holds a SlidingWindowMLASpec layer. Such a group takes no part if it replays a bounded window, and the probe does not record that setting. See vLLM v0.33.0 kv_cache_interface.py:998-1000, :1027-1029 and :1305-1307.

  • transfer_bytes_per_block is the bytes that vLLM's P/D connector (NIXL) reads for one block of the group in a transfer on one TP rank, for the vLLM version of the deployment. It is positive and at most 2^53, and it can be larger than page_size_bytes, because the connector reads a block in regions that layers of other groups can share. A hand-made profile may state it only for a group whose transfer size has been measured.

    To measure the full-attention value and the combined transfer size of the other groups, send two prompts of different length through a P/D pair, one at a time, and read the change of vllm:nixl_bytes_transferred_sum on the decode server for each. Divide each change by the change of vllm:nixl_bytes_transferred_count, because each TP rank makes its own transfer and the field holds the bytes of one rank. The difference between the two results, divided by the difference in full-attention blocks that move, is the value of the full-attention group. The rest of one transfer is the sum of the values of the other groups when each of them moves one block, as a state group does. With multiple state groups, this determines only their combined transfer size.

    Per-group values for models with several state groups are not supported by this procedure or the current runtime probe. Leave transfer_bytes_per_block unset for those groups until per-group transfer measurements or additional runtime-observer evidence identifies the bytes read for each group on one TP rank. Do not split the combined size evenly or use the page sizes as substitutes.

    The probe import of a vLLM 0.33.0 record writes it for no group, because what the P/D connector of vLLM 0.33.0 reads for a block depends on facts that the record does not hold. The connector registers one region per layer address and lets layers of different groups share a region. In a model with Mamba layers, it reads a block of an attention group, other than the ring, in every region. The length of a region of a layer that is not a Mamba layer depends on the layout: the padded page divided by a kernel-block ratio that the connector chooses at start, or the whole storage row of a block. A Mamba layer moves its state without padding, and a tp_replicated Mamba layer moves its whole page. The record holds neither the ratio nor tp_replicated. See vLLM v0.33.0 base_worker.py:1744-1756, :194-249, :1598-1603, :1693-1719, :1063-1082, :2013-2033 and :1774-1779. A P/D model of a grouped profile for vLLM 0.33.0 needs these facts from the runtime observer first. A claim in resource_overrides.cache_groups that states this field is refused, because the probe evidence holds no value to compare it with.

Use RustForwardPassPerfModel.estimate_cache_budget(config, budget) with the same canonical ForwardPassPerfModelConfig used for timing. Rust exposes ForwardPassPerfModelConfig::estimate_cache_budget(&request) and ForwardPassPerfModel::estimate_cache_budget(&request), taking FpmCacheBudgetRequest and returning FpmCacheBudget. The config method checks the exact profile deployment and scheduler envelope without constructing a timing model, loading timings, or building an operation graph. Grouped planning requires the native extension; it does not require GPU access or measured data.

Budget field Meaning
total_gpu_capacity_bytes Positive rank-local GPU capacity.
memory_fraction_kind, memory_fraction_value of_total and the fraction of GPU memory available to the worker.
max_num_tokens, max_batch_size Positive rank-local scheduler limits within a declared envelope; exact recorded values for runtime memory.
context_length Optional positive request bound, at most the profile context and runtime max_model_len when present; defaults to the profile context.
request_occupancy_tokens Optional positive logical request length within the selected context, for a separate steady decode footprint with one new token per forward pass.
cuda_graph_reserved_bytes Separate nonnegative reservation for declared memory; must be zero for runtime memory.
tolerance_fraction Optional fraction in [0, 1) deducted from the resulting cache byte budget.

The result includes total_kv_size_bytes, memory_breakdown, resource_provenance, cache_layout, cache_groups, and request_peak_cache_bytes. The latter is a conservative single-request bound including block alignment and the configured prefill chunk, not a reservation for every scheduler slot. A smaller context changes this peak bound, not the declared non-cache resource bounds. tolerance_adjusted, when present, provides the reduced byte budget. For grouped caches, kv_size_per_token_bytes, total_kv_size_tokens, and the adjusted token capacity are null (None in Python). There is no equivalent scalar token capacity.

When request_occupancy_tokens is supplied, the result also includes request_occupancy_cache_bytes. KV-relative synthetic loads use this native footprint while retaining the actual scheduler settings and context for resource admission. Observed linear pools retain their recorded capacity even when timing coverage is smaller; uncovered queries report timing errors.

Runtime memory uses kv_cache_bytes directly and returns memory_breakdown: null (Option::None in Rust). It does not invent activation or other non-KV components. The requested memory fraction must exactly match the recorded utilization; the device budget must contain the observed cache pool. A different device capacity argument never rescales the pool or qualifies a new deployment. Changed scheduler settings require new runtime evidence. Explicit scalar cache-capacity overrides are rejected for runtime profiles in prediction, recommendation and replay; use the recorded allocation. A tolerance margin may reduce it through the canonical budget API.

Python estimate_kv_cache(..., fpm_profile=..., context_length=...) delegates grouped and runtime-memory profiles to this native budget method. estimate_num_gpu_blocks also accepts context_length for runtime validation; it and the legacy Rust scalar KvCacheEstimate transport reject grouped profiles; use the groups and shared byte budget instead. Existing complete linear declarations keep their budget calculation and serialized shape.

Grouped native execution supports cold aggregated vLLM, PP1/CP1, HBM-only cache, non-speculative decoding and prefix_caching: false. Prefix reuse, host/G3 offload, disaggregation and fixed scalar cache capacity are rejected. Allocation is atomic across all groups against one shared rank-local byte budget. A window of W keeps the blocks covering the last W - 1 computed tokens and all tokens scheduled for the next forward. Expired completed pages are evicted before subsequent forwards; temporary prefill pages remain charged through completion. Full-attention groups retain full history. A state group holds its blocks_per_request blocks from the request's first scheduled forward until the request finishes, is cancelled or is preempted. A resumed request reserves them again once. This physical retention never truncates logical request progress or the context coordinates sent to FPM timing queries.

The Rust Replay observer's ReplaySchedulerMetricsSnapshot reports optional kv_cache_used_bytes and kv_cache_capacity_bytes for grouped caches; its active_cache_usage and physical_cache_usage use byte occupancy. active_blocks counts resident group pages, while total_blocks is zero because heterogeneous pages have no scalar block capacity. Use the byte fields for capacity comparisons. These fields are currently exposed through the Rust observer API, not the Python JSON replay runtime. Linear snapshots retain their existing block metrics.

Cache reuse, eviction, and offload lifecycle are documented in Replay KV cache.

Observed Mamba align pools

The mamba_align layout describes one full-attention group and padded Mamba groups that share one physical pool. Supply observed runtime memory, physical page bytes, block length, state-group count, prefix unit, and speculative state count. Set mamba_cache.state_storage: full_state explicitly. The count must match the measured draft depth. Internal prefill checkpoint slots are unsupported.

Field under resources.mamba_cache Meaning
state_storage Required full_state. Other policies are rejected.
block_size_tokens Physical attention block length, at least two tokens.
page_size_bytes Padded page size shared by attention and state groups.
state_groups Number of state pages in one complete working state.
prefix_match_unit Positive divisor of the block length.
num_speculative_states Full-state scratch copies, equal to the measured draft depth, from zero through five.

The native budget returns replay_cache. This descriptor excludes one reserved null page and includes working-state and speculative scratch reservations. Do not interpret its allocation byte rate as logical KV bytes per token. The profile does not expose an aggregate token capacity.

Use aggregated vLLM replay with the measured block length and scheduler limits. Keep prefix caching enabled. Fixed capacity overrides, offload, and disaggregated FPM replay are unsupported for this layout. Original timing imports remain valid as timing-only evidence.

ReplaySSM and RecoverSSM lay out and size their states differently, so profile them as unmodeled for timing only; do not relabel them full_state. Lookups follow vLLM revision db9527a46873454610df6dbedf79a36d6bf1a7f6, which applies the logits cap before the MTP hash-unit drop, so some exactly aligned prompts miss the cached tail.