Skip to content

【WIP】Support model express bootstrap acceleration - #1862

Open
hzxuzhonghu wants to merge 9 commits into
volcano-sh:mainfrom
hzxuzhonghu:mx-implement
Open

hzxuzhonghu wants to merge 9 commits into
volcano-sh:mainfrom
hzxuzhonghu:mx-implement

Conversation

@hzxuzhonghu

Copy link
Copy Markdown
Member

What type of PR is this?

/kind enhancement

What this PR does / why we need it:

Depend on #1849

Which issue(s) this PR fixes:
Fixes #

Bug evidence (required for bug-related PRs):

Special notes for your reviewer:

Does this PR introduce a user-facing change?:


Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Bootstrap hash tracking, observation ordering, schema consistency, and required coverage must be corrected before approval.

5 open findings
What changed in this PR

Adds ModelExpress-backed, source-aware bootstrap batching to ModelServing.

Changes:

  • Adds bootstrap strategy APIs, validation, generated clients, CRDs, and documentation.
  • Implements configuration hashing, environment injection, and batch admission.
  • Integrates bootstrap state into reconciliation and datastore tracking.
File Description
pkg/​model-serving-controller/​webhook/​validator.go Validates bootstrap configuration.
pkg/​model-serving-controller/​webhook/​validator_test.go Tests validation behavior.
pkg/​model-serving-controller/​utils/​revision_data.go Exposes plugin normalization.
pkg/​model-serving-controller/​utils/​revision_data_test.go Verifies revision neutrality.
pkg/​model-serving-controller/​plugins/​modelexpress_plugin.go Adds managed ModelExpress plugin.
pkg/​model-serving-controller/​plugins/​modelexpress_plugin_test.go Tests plugin injection and scope.
pkg/​model-serving-controller/​datastore/​store.go Tracks role bootstrap hashes.
pkg/​model-serving-controller/​datastore/​store_test.go Tests hash tracking.
pkg/​model-serving-controller/​datastore/​dump.go Exports bootstrap hashes.
pkg/​model-serving-controller/​controller/​model_serving_controller.go Integrates batch admission.
pkg/​model-serving-controller/​controller/​model_serving_controller_test.go Updates controller tests.
pkg/​model-serving-controller/​controller/​model_serving_controller_bootstrap_test.go Tests bootstrap reconciliation.
pkg/​model-serving-controller/​bootstrap/​config.go Calculates pool configuration hashes.
pkg/​model-serving-controller/​bootstrap/​admitter.go Implements admission budgets.
pkg/​model-serving-controller/​bootstrap/​admitter_test.go Tests admission rules.
pkg/​apis/​workload/​v1alpha1/​zz_generated.deepcopy.go Adds generated deep copies.
pkg/​apis/​workload/​v1alpha1/​model_serving_types.go Defines bootstrap API fields.
pkg/​apis/​workload/​v1alpha1/​labels.go Defines the hash annotation.
examples/​model-serving/​bootstrap-acceleration.yaml Adds an example workload.
docs/​proposal/​modelserving-modelexpress-startup-acceleration.md Documents the design.
docs/​kthena/​sidebars.ts Links the user guide.
docs/​kthena/​docs/​user-guide/​bootstrap-acceleration.md Adds usage documentation.
docs/​kthena/​docs/​reference/​crd/​workload.serving.volcano.sh.md Updates CRD reference.
client-go/​applyconfiguration/​workload/​v1alpha1/​modelservingspec.go Adds apply configuration field.
client-go/​applyconfiguration/​workload/​v1alpha1/​modelexpressconfig.go Adds generated ModelExpress builder.
client-go/​applyconfiguration/​workload/​v1alpha1/​bootstrapacceleratestrategy.go Adds generated strategy builder.
client-go/​applyconfiguration/​utils.go Registers apply configurations.
charts/​kthena/​charts/​workload/​crds/​workload.serving.volcano.sh_modelservings.yaml Updates the packaged CRD.
Files not reviewed (5)
  • client-go/applyconfiguration/utils.go: Generated file
  • client-go/applyconfiguration/workload/v1alpha1/bootstrapacceleratestrategy.go: Generated file
  • client-go/applyconfiguration/workload/v1alpha1/modelexpressconfig.go: Generated file
  • client-go/applyconfiguration/workload/v1alpha1/modelservingspec.go: Generated file
  • pkg/apis/workload/v1alpha1/zz_generated.deepcopy.go: Generated file

🧠 Review effort: Balanced


💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment on lines +54 to +57
// BootstrapAccelerateStrategy creates role replicas in source-aware batches
// for peer-to-peer model weight loading. If unset, creation is unchanged.
// +optional
BootstrapAccelerateStrategy *BootstrapAccelerateStrategy `json:"bootstrapAccelerateStrategy,omitempty"`
Comment on lines +567 to +573
admitter, err := c.newBootstrapAdmitter(ms)
if err != nil {
return fmt.Errorf("failed to build bootstrap admitter: %v", err)
}

// 1. Sync the number of ServingGroups to match the expected replicas defined in spec.
if err := c.syncServingGroupReplicas(ctx, ms, revision); err != nil {
if err := c.syncServingGroupReplicas(ctx, ms, revision, admitter); err != nil {
Copilot AI balanced review requested due to automatic review settings October 9, 2026 11:14

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Source readiness and template identity can be misclassified, and the planned cluster-level E2E coverage is missing.

3 open findings
3 resolved since last review
Files not reviewed (5)
  • client-go/applyconfiguration/utils.go: Generated file
  • client-go/applyconfiguration/workload/v1alpha1/bootstrapacceleratestrategy.go: Generated file
  • client-go/applyconfiguration/workload/v1alpha1/modelexpressconfig.go: Generated file
  • client-go/applyconfiguration/workload/v1alpha1/modelservingspec.go: Generated file
  • pkg/apis/workload/v1alpha1/zz_generated.deepcopy.go: Generated file
Previously missed (2)

In code that hasn't changed since last review

Medium severity Running role remains active when Pod becomes NotReady

pkg/​model-serving-controller/​controller/​model_serving_controller.go:640

RoleRunning is sticky when a Running Pod merely becomes NotReady: updatePod's default branch does not demote the role, so this can count a currently unready replica as a bootstrap source and release another batch. Either derive source readiness from the current Pods here or update the Pod handler to transition the role out of Running on Ready→NotReady changes.

Low severity Missing E2E coverage for batch wake-ups and rolling updates

docs/​proposal/​modelserving-modelexpress-startup-acceleration.md:406

The promised controller-manager E2E coverage is absent: no test under test/e2e exercises batch wake-ups, real PodGroup gang cores, or rolling-update pacing. The unit tests use fake clients/managers and cannot validate these cluster-level interactions, so add the E2E described here before merging.

🧠 Review effort: Balanced

Comment on lines +634 to +639
replicas = append(replicas, bootstrap.Replica{
Pool: bootstrap.PoolKey{
Role: roleName,
RoleTemplateHash: role.RoleTemplateHash,
ConfigHash: role.BootstrapConfigHash,
},
}

// ModelExpressConfig holds values injected into the Pods of in-scope roles.
type ModelExpressConfig struct {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why not put these model express specific config into plugin

roles: ["prefill"]
seedReplicas: 1
sourceFanOut: 2
modelExpress:

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

a little wierd for me

Comment on lines +127 to +227
// ModelServingSpec defines the specification of the ModelServing resource.
type ModelServingSpec struct {
// ...existing fields...

// BootstrapAccelerateStrategy creates role replicas in batches so that most of them
// load model weights peer-to-peer from ready replicas instead of from storage.
// +optional
BootstrapAccelerateStrategy *BootstrapAccelerateStrategy `json:"bootstrapAccelerateStrategy,omitempty"`
}

// BootstrapAccelerateStrategyProvider identifies the peer-to-peer weight transfer system.
// +kubebuilder:validation:Enum={ModelExpress}
type BootstrapAccelerateStrategyProvider string

const (
BootstrapAccelerateStrategyProviderModelExpress BootstrapAccelerateStrategyProvider = "ModelExpress"
)

// BootstrapAccelerateStrategy configures source-aware creation of role replicas.
type BootstrapAccelerateStrategy struct {
// Provider is the weight transfer system used by the inference engines.
// +kubebuilder:default=ModelExpress
// +optional
Provider BootstrapAccelerateStrategyProvider `json:"provider,omitempty"`

// Roles limits bootstrap acceleration to the listed roles. Empty means all roles.
// +listType=set
// +optional
Roles []string `json:"roles,omitempty"`

// SeedReplicas is the number of role replicas per role that may be created while no
// ready source exists. A percentage is taken of the desired role replicas
// (spec.replicas * role.replicas) and rounded up, e.g. 10% of 3 is 1.
// With gang scheduling, the effective budget is at least the role's minRoleReplicas
// (default: all role replicas in one ServingGroup). Must resolve to at least 1
// when the desired count is nonzero; a desired count of zero creates nothing.
// +kubebuilder:validation:XIntOrString
// +kubebuilder:default=1
// +optional
SeedReplicas *intstr.IntOrString `json:"seedReplicas,omitempty"`

// SourceFanOut is the number of role replicas per role that may be starting
// (created but not yet Running) for each ready replica of the same role.
// +kubebuilder:validation:Minimum=1
// +kubebuilder:default=1
// +optional
SourceFanOut *int32 `json:"sourceFanOut,omitempty"`

// ModelExpress configures the ModelExpress provider. If unspecified, no MX
// environment variables are injected and users configure the engine themselves.
// +optional
ModelExpress *ModelExpressConfig `json:"modelExpress,omitempty"`
}

// ModelExpressConfig holds values injected into the Pods of in-scope roles.
type ModelExpressConfig struct {
// EngineContainers names the regular containers running the inference engine
// in entry and worker Pods of in-scope roles. Only these containers receive MX envs.
// +kubebuilder:validation:MinItems=1
// +listType=set
EngineContainers []string `json:"engineContainers"`

// ServerAddress is the gRPC address (host:port) of the ModelExpress server. It is
// injected as MX_SERVER_ADDRESS only if that variable is not already
// set in the container.
// +optional
ServerAddress string `json:"serverAddress,omitempty"`

// ReadyURL is the engine endpoint that MX polls before a source publishes its
// metadata; it may differ from the Pod readiness probe. It is injected as
// MX_ARTIFACT_READY_URL only if that variable is not already set in the container.
// If empty, MX uses the engine's default (http://127.0.0.1:8000/health for vLLM).
// +optional
ReadyURL string `json:"readyURL,omitempty"`
}
```

```yaml
spec:
replicas: 16
bootstrapAccelerateStrategy:
roles: ["prefill"]
seedReplicas: 1
sourceFanOut: 2
modelExpress:
engineContainers: ["engine"]
serverAddress: modelexpress-server.modelexpress:8001
readyURL: http://127.0.0.1:8000/health
template:
roles:
- name: prefill
replicas: 1
entryTemplate:
spec:
containers:
- name: engine
image: <vllm-0.23+-with-modelexpress>
args: ["serve", "/models/deepseek-v4", "--load-format", "modelexpress", "--tensor-parallel-size", "8"]
readinessProbe:
httpGet: {path: /health, port: 8000}
```

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Heres my example.Plugin way i think is more convient to introduce more P2P backends in future

// ModelServingSpec
type ModelServingSpec struct {
	// ...existing fields (replicas, schedulerName, plugins, template, ...)

	// BootstrapStrategy controls how role replicas are created during startup,
	// scale-up, recovery, and rolling-update re-creation.
	// If unset, all missing replicas are created as today.
	// This field is revision-neutral: edits do not trigger a rollout.
	// +optional
	BootstrapStrategy *BootstrapStrategy `json:"bootstrapStrategy,omitempty"`
}

// BootstrapStrategy configures source-aware creation pacing of role replicas.
// It does not select a weight-transfer implementation; engines/plugins do that.
type BootstrapStrategy struct {
	// Roles limits pacing to the listed roles. Empty means all roles.
	// +listType=set
	// +optional
	Roles []string `json:"roles,omitempty"`

	// SeedReplicas is the max role replicas per role that may be created while
	// no ready source exists. Int or percent of (spec.replicas * role.replicas),
	// rounded up. Effective budget is at least one gang core when gang scheduling
	// is enabled.
	// +kubebuilder:validation:XIntOrString
	// +kubebuilder:validation:XValidation:rule="type(self) == int ? self >= 1 : self.matches('^([1-9]|[1-9][0-9]|100)%$')"
	// +kubebuilder:default=1
	// +optional
	SeedReplicas *intstr.IntOrString `json:"seedReplicas,omitempty"`

	// SourceFanOut is the max starting (created, not yet Running) role replicas
	// allowed per ready replica of the same role. Effective budget is at least
	// one gang core when gang scheduling is enabled.
	// +kubebuilder:validation:Minimum=1
	// +kubebuilder:default=1
	// +optional
	SourceFanOut *int32 `json:"sourceFanOut,omitempty"`
}

// ModelExpress stays out of Spec strategy. It is a BuiltIn plugin config:
//
// plugins:
// - name: modelexpress
//   type: BuiltIn
//   scope: { roles: ["prefill"], target: All }
//   config: { engineContainers, serverAddress, readyURL }
//
// Suggested JSON shape for plugin Config (not a top-level CRD struct field):
type ModelExpressPluginConfig struct {
	EngineContainers []string `json:"engineContainers"`
	ServerAddress    string   `json:"serverAddress,omitempty"`
	ReadyURL         string   `json:"readyURL,omitempty"`
}
spec:
  replicas: 16
  schedulerName: volcano
  # Controller: When to Create
  bootstrapStrategy:
    roles: ["prefill"]
    seedReplicas: 1
    sourceFanOut: 2
  # Plugins: How to modify the Pod (to match existing plugins)
  plugins:
  - name: modelexpress
    type: BuiltIn
    scope:
      roles: ["prefill"]   # It's clearer when aligned with bootstrapStrategy.roles
      target: All
    config:
      engineContainers: ["engine"]
      serverAddress: modelexpress-server.modelexpress:8001
      readyURL: http://127.0.0.1:8000/health
  template:
    roles:
    - name: prefill
      replicas: 1
      entryTemplate:
        spec:
          containers:
          - name: engine
            image: <vllm-0.23+-with-modelexpress>
            args:
            - serve
            - /models/deepseek-v4
            - --load-format
            - modelexpress
            - --tensor-parallel-size
            - "8"
            readinessProbe:
              httpGet: {path: /health, port: 8000}

@hzxuzhonghu hzxuzhonghu changed the title Support model express bootstrap acceleration 【WIP】Support model express bootstrap acceleration Oct 20, 2026
hzxuzhonghu and others added 4 commits November 5, 2026 16:40
Signed-off-by: Zhonghu Xu <xuzhonghu.xzh@alibaba-inc.com>
Signed-off-by: Zhonghu Xu <xuzhonghu.xzh@alibaba-inc.com>
Signed-off-by: Zhonghu Xu <zhhxu2011@163.com>
Signed-off-by: Zhonghu Xu <zhhxu2011@163.com>
Copilot AI balanced review requested due to automatic review settings November 5, 2026 11:23

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Bootstrap admission budgets legacy gang recovery using a reduced current replica count, causing perpetual denial when no Ready source exists and preventing partition-protected replicas from recovering.

7 open issues
6 files not reviewed
  • client-go/applyconfiguration/utils.go: Generated file
  • client-go/applyconfiguration/workload/v1alpha1/bootstrapacceleratestrategy.go: Generated file
  • client-go/applyconfiguration/workload/v1alpha1/bootstraprole.go: Generated file
  • client-go/applyconfiguration/workload/v1alpha1/modelexpressconfig.go: Generated file
  • client-go/applyconfiguration/workload/v1alpha1/modelservingspec.go: Generated file
  • pkg/apis/workload/v1alpha1/zz_generated.deepcopy.go: Generated file

Review effort: Balanced

seed = value
}
}
return min(desired, max(a.GangMinimum(spec), max(1, seed)))
TemplateHash: role.RoleTemplateHash,
ConfigHash: hash,
HashKnown: known,
Running: role.Status == datastore.RoleRunning,
}
klog.V(4).Infof("scaleUpServingGroups: creating new ServingGroup at ordinal=%d with newRevision=%s for modelServing=%s", ordinal, newRevision, utils.GetNamespaceName(ms))
if err := createServingGroup(ordinal, newRevision, ms.Spec.Template.Roles); err != nil {
if _, err := createServingGroup(ordinal, newRevision, ms.Spec.Template.Roles); err != nil {
Signed-off-by: Zhonghu Xu <zhhxu2011@163.com>
Signed-off-by: Zhonghu Xu <zhhxu2011@163.com>
Copilot AI balanced review requested due to automatic review settings November 5, 2026 12:00

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Missing ModelExpress plugin symbols prevent compilation, while revision-unaware bootstrap admission can exceed seed budgets or indefinitely block partition-protected replica recovery.

8 open issues
6 files not reviewed
  • client-go/applyconfiguration/utils.go: Generated file
  • client-go/applyconfiguration/workload/v1alpha1/bootstrapacceleratestrategy.go: Generated file
  • client-go/applyconfiguration/workload/v1alpha1/bootstraprole.go: Generated file
  • client-go/applyconfiguration/workload/v1alpha1/modelexpressconfig.go: Generated file
  • client-go/applyconfiguration/workload/v1alpha1/modelservingspec.go: Generated file
  • pkg/apis/workload/v1alpha1/zz_generated.deepcopy.go: Generated file

Review effort: Balanced

Comment on lines +3029 to +3030
if mxPlugin := plugins.NewModelExpressPlugin(ms, c.modelExpressServerAddress); mxPlugin != nil {
chain = chain.WithManagedPlugin(mxPlugin, plugins.ModelExpressPluginScope(ms))
Signed-off-by: Zhonghu Xu <zhhxu2011@163.com>
Signed-off-by: Zhonghu Xu <zhhxu2011@163.com>
Signed-off-by: Zhonghu Xu <zhhxu2011@163.com>
Copilot AI balanced review requested due to automatic review settings November 6, 2026 01:57
@volcano-sh-bot

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please ask for approval from hzxuzhonghu. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Gang-recovery admission can deadlock when seed budgets are exhausted by incomplete gangs or conflict with historical replica layouts, leaving workloads unable to recover.

9 open issues
1 resolved since last review
6 files not reviewed
  • client-go/applyconfiguration/utils.go: Generated file
  • client-go/applyconfiguration/workload/v1alpha1/bootstrapacceleratestrategy.go: Generated file
  • client-go/applyconfiguration/workload/v1alpha1/bootstraprole.go: Generated file
  • client-go/applyconfiguration/workload/v1alpha1/modelexpressconfig.go: Generated file
  • client-go/applyconfiguration/workload/v1alpha1/modelservingspec.go: Generated file
  • pkg/apis/workload/v1alpha1/zz_generated.deepcopy.go: Generated file

Review effort: Balanced

Comment on lines +1212 to +1213
bootstrapAdmits := func(ordinal int, roleToApply workloadv1alpha1.Role) bool {
if bootstrap.AdmitterFrom(ctx).AdmitRole(roleToApply) {
## Prerequisites

- An MX-enabled inference engine (for example vLLM with `--load-format modelexpress`) that can reach a ModelExpress server.
- The controller manager is started with `--modelexpress-server-address=<host>:<port>` (Helm: `controllerManager.modelExpressServerAddress`). The address is injected into engine containers as `MX_SERVER_ADDRESS`.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

Status: Accepted

Development

Successfully merging this pull request may close these issues.

4 participants