You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
【WIP】Support model express bootstrap acceleration - #1862
RoleRunning is sticky when a Running Pod merely becomes NotReady: updatePod's default branch does not demote the role, so this can count a currently unready replica as a bootstrap source and release another batch. Either derive source readiness from the current Pods here or update the Pod handler to transition the role out of Running on Ready→NotReady changes.
Missing E2E coverage for batch wake-ups and rolling updates
The promised controller-manager E2E coverage is absent: no test under test/e2e exercises batch wake-ups, real PodGroup gang cores, or rolling-update pacing. The unit tests use fake clients/managers and cannot validate these cluster-level interactions, so add the E2E described here before merging.
The reason will be displayed to describe this comment to others. Learn more.
Heres my example.Plugin way i think is more convient to introduce more P2P backends in future
// ModelServingSpec
type ModelServingSpec struct {
// ...existing fields (replicas, schedulerName, plugins, template, ...)
// BootstrapStrategy controls how role replicas are created during startup,
// scale-up, recovery, and rolling-update re-creation.
// If unset, all missing replicas are created as today.
// This field is revision-neutral: edits do not trigger a rollout.
// +optional
BootstrapStrategy *BootstrapStrategy `json:"bootstrapStrategy,omitempty"`
}
// BootstrapStrategy configures source-aware creation pacing of role replicas.
// It does not select a weight-transfer implementation; engines/plugins do that.
type BootstrapStrategy struct {
// Roles limits pacing to the listed roles. Empty means all roles.
// +listType=set
// +optional
Roles []string `json:"roles,omitempty"`
// SeedReplicas is the max role replicas per role that may be created while
// no ready source exists. Int or percent of (spec.replicas * role.replicas),
// rounded up. Effective budget is at least one gang core when gang scheduling
// is enabled.
// +kubebuilder:validation:XIntOrString
// +kubebuilder:validation:XValidation:rule="type(self) == int ? self >= 1 : self.matches('^([1-9]|[1-9][0-9]|100)%$')"
// +kubebuilder:default=1
// +optional
SeedReplicas *intstr.IntOrString `json:"seedReplicas,omitempty"`
// SourceFanOut is the max starting (created, not yet Running) role replicas
// allowed per ready replica of the same role. Effective budget is at least
// one gang core when gang scheduling is enabled.
// +kubebuilder:validation:Minimum=1
// +kubebuilder:default=1
// +optional
SourceFanOut *int32 `json:"sourceFanOut,omitempty"`
}
// ModelExpress stays out of Spec strategy. It is a BuiltIn plugin config:
//
// plugins:
// - name: modelexpress
// type: BuiltIn
// scope: { roles: ["prefill"], target: All }
// config: { engineContainers, serverAddress, readyURL }
//
// Suggested JSON shape for plugin Config (not a top-level CRD struct field):
type ModelExpressPluginConfig struct {
EngineContainers []string `json:"engineContainers"`
ServerAddress string `json:"serverAddress,omitempty"`
ReadyURL string `json:"readyURL,omitempty"`
}
spec:
replicas: 16
schedulerName: volcano
# Controller: When to Create
bootstrapStrategy:
roles: ["prefill"]
seedReplicas: 1
sourceFanOut: 2
# Plugins: How to modify the Pod (to match existing plugins)
plugins:
- name: modelexpress
type: BuiltIn
scope:
roles: ["prefill"] # It's clearer when aligned with bootstrapStrategy.roles
target: All
config:
engineContainers: ["engine"]
serverAddress: modelexpress-server.modelexpress:8001
readyURL: http://127.0.0.1:8000/health
template:
roles:
- name: prefill
replicas: 1
entryTemplate:
spec:
containers:
- name: engine
image: <vllm-0.23+-with-modelexpress>
args:
- serve
- /models/deepseek-v4
- --load-format
- modelexpress
- --tensor-parallel-size
- "8"
readinessProbe:
httpGet: {path: /health, port: 8000}
hzxuzhonghu
changed the title
Support model express bootstrap acceleration
【WIP】Support model express bootstrap acceleration
Oct 20, 2026
The reason will be displayed to describe this comment to others. Learn more.
🟡 Changes recommended
Bootstrap admission budgets legacy gang recovery using a reduced current replica count, causing perpetual denial when no Ready source exists and preventing partition-protected replicas from recovering.
klog.V(4).Infof("scaleUpServingGroups: creating new ServingGroup at ordinal=%d with newRevision=%s for modelServing=%s", ordinal, newRevision, utils.GetNamespaceName(ms))
if err := createServingGroup(ordinal, newRevision, ms.Spec.Template.Roles); err != nil {
This pull-request has been approved by: Once this PR has been reviewed and has the lgtm label, please ask for approval from hzxuzhonghu. For more information see the Kubernetes Code Review Process.
The full list of commands accepted by this bot can be found here.
Details
Needs approval from an approver in each of these files:
The reason will be displayed to describe this comment to others. Learn more.
🟡 Changes recommended
Gang-recovery admission can deadlock when seed budgets are exhausted by incomplete gangs or conflict with historical replica layouts, leaving workloads unable to recover.
- An MX-enabled inference engine (for example vLLM with `--load-format modelexpress`) that can reach a ModelExpress server.
- The controller manager is started with `--modelexpress-server-address=<host>:<port>` (Helm: `controllerManager.modelExpressServerAddress`). The address is injected into engine containers as `MX_SERVER_ADDRESS`.
This branch has not been deployed
No deployments
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What type of PR is this?
/kind enhancement
What this PR does / why we need it:
Depend on #1849
Which issue(s) this PR fixes:
Fixes #
Bug evidence (required for bug-related PRs):
Special notes for your reviewer:
Does this PR introduce a user-facing change?: