[RobuRishabh]

Part of #2962.

Design-only PR for Kubernetes-native Workload-Aware Scheduling (WAS) in Spark Operator. It adds KEP-2962 proposing a top-level SparkApplication.spec.scheduling API aligned with KEP-6089, kubernetes/enhancements#6342, and the Kubeflow Trainer KEP.

Summary

• SparkApplication is the root controller and sole compiler of native Workload objects via upstream workloadbuilder.
• Opt-in API: omitted .spec.scheduling preserves current behavior; SparkApplicationWorkloadAwareScheduling defaults to disabled.
• Driver schedules independently; executor cohort is one leaf PodGroup, avoiding the cluster-mode bootstrap deadlock.
• Gang minCount is controller-computed from Spark's resolved initial executor count; users express scale through executor.instances / sparkConf.
• One stable Workload per SparkApplication; each status.submissionID gets an isolated attempt-scoped PodGroup.
• Native WAS is mutually exclusive with Volcano, YuniKorn, and scheduler-plugins per application.
• Alpha scope is static executors only: dynamic allocation with native WAS is rejected in this KEP and deferred to a follow-up KEP (topology, DRA, client modes, and Kueue are also out of scope).
• Phase 0 prerequisite: pin Kubernetes v1.37+ and the newest KEP-6089 served API stack at implementation time (prefer scheduling.k8s.io/v1 building blocks from #6342 over v1.36 prototype wire types).

Alpha vs upstream beta: "Alpha" here is the Spark Operator feature stage (kep.yaml stage: alpha, gate off by default, narrow MVP). It does not mean KEP-6089 stays alpha — reusable building blocks are graduating to beta via #6342 for Kubernetes v1.38.

The KEP is restructured to match the Trainer layout and includes API examples, validation tables, lifecycle semantics, RBAC, graduation criteria, alternatives, and seven Mermaid diagrams (architecture, compile path, sequence, ownership, bootstrap deadlock, and lifecycle).

Thanks to @Rajadi16 for prototype work in #3093, #3113, and #3116. Those PRs are credited as implementation evidence; this PR asks the community to agree on the durable API and lifecycle before implementation continues.

Review focus

Highest-value feedback:

1. Open decisions in the KEP — especially empty scheduling: {} defaulting (Basic vs Gang), CRD surface area, and behavior when a deployment-level default batch scheduler is also configured.
2. Alpha scope boundary — static gang only in MVP; dynamic allocation explicitly deferred.
3. Kubernetes version and Phase 0 API pinning (v1.37+ provisional minimum).
4. Workload / PodGroup lifecycle — retry isolation, suspend/resume, and immutable .spec.scheduling in alpha.

Test plan

• Design-only change; no runtime code.
• Reviewers: read proposals/2962-workload-aware-scheduling/README.md and kep.yaml.
• Confirm alignment with KEP-6089 root-controller model and upstream workloadbuilder.
• Confirm alpha scope and deferred topics match expectations for #2962.

References

• Kubernetes KEP-6089: https://github.com/kubernetes/enhancements/tree/master/keps/sig-scheduling/6089-was-controller-apis
• KEP-6089 beta promotion (#6342): kep-6089: promote Workload Aware Scheduling Controller APIs to Beta kubernetes/enhancements#6342
• Kubeflow Trainer KEP: feat(api): KEP-3015: Workload Aware Scheduling for TrainJob trainer#3219

[google-oss-prow]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign chenyi015 for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[Copilot]

🟡 Changes recommended

KEP metadata uses a non-standard sentinel value (replaces: "NA") that should be an empty list (or omitted) to avoid breaking or confusing KEP tooling/automation.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

This PR adds a design KEP proposing Kubernetes-native Workload-Aware Scheduling (WAS) support for SparkApplication via a new optional .spec.scheduling API, aligning Spark Operator’s scheduling model with Kubernetes KEP-6089 and related ecosystem direction.

Changes:

• Add a full KEP document describing the proposed .spec.scheduling API surface, defaulting/validation, lifecycle/cleanup, and phased implementation plan.
• Add KEP metadata (kep.yaml) for proposal tracking and review routing.

File summaries

File	Description
proposals/2962-workload-aware-scheduling/README.md	New KEP document detailing API shape, controller/lifecycle design, validation, and rollout plan for native WAS.
proposals/2962-workload-aware-scheduling/kep.yaml	New KEP metadata file (title, number, authors, reviewers, links, feature gate).

Review details

• Files reviewed: 2/2 changed files
• Comments generated: 1
• Review effort level: Lite

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[andreyvelich]

This is awesome, thank you for driving this @RobuRishabh!
/cc @kannon92 @mm4tt @helayoty

[google-oss-prow]

@andreyvelich: GitHub didn't allow me to request PR reviews from the following users: mm4tt, helayoty.

Note that only kubeflow members and repo collaborators can review this PR, and authors cannot review their own PRs.

Details

In response to this:

This is awesome, thank you for driving this @RobuRishabh!
/cc @kannon92 @mm4tt @helayoty

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

[sxivansx]

One thing worth checking before Phase 0. I looked at the runtime shape in k8s.io/api v0.35.4, which this repo already pins in go.mod. In scheduling.k8s.io/v1alpha1, PodGroup isn't a standalone object — it's an inline struct in WorkloadSpec.PodGroups, and that field is immutable. Only Workload/WorkloadList are registered, and Workload has no Status. Pods join via pod.spec.workloadRef{name, podGroup, podGroupReplicaKey}, not spec.schedulingGroup.

If that shape carries into the version you pin, the per-submissionID PodGroup object has nothing to create or GC, and updating minCount in place isn't possible either.

WorkloadReference.PodGroupReplicaKey looks like it covers the first case — its doc says policy is applied separately per replica. Setting it to status.submissionID would give retry isolation with one long-lived Workload, which is the invariant you already want.

Smaller: the RBAC block grants podgroups and podgroups/status, neither of which exists in that shape.

[RobuRishabh]

Hi @sxivansx thanks for the review. Agreed on Phase 0 must pin the exact served API before implementation.
The shape you inspected (scheduling.k8s.io/v1alpha1 in k8s.io/api v0.35.x) is the superseded prototype: inline Workload.spec.podGroups, Pods via spec.workloadRef + podGroupReplicaKey, no standalone PodGroup CRD.

This KEP targets Kubernetes v1.37+ and scheduling.k8s.io/v1beta1, where:

• Workload / PodGroup are standalone resources (PodGroup has status)
• Pods join via spec.schedulingGroup, not workloadRef
• Gang minCount is mutable on the runtime PodGroup
• RBAC for podgroups / podgroups/status applies

Phase 0 will bump k8s.io/api / component-helpers to the v1.37+ stack and compile workloadbuilder against those types, IMO we should not implement against the v1alpha1 wire shape in today’s go.mod.
The per-submissionID PodGroup + schedulingGroup model follows KEP-6089’s centralized management pattern (also used by Job and Trainer).

[tariq-hasan]

Hi @RobuRishabh! I have added comments inline on the KEP.

[RobuRishabh]

Hi @andreyvelich @vjanelle @vikas-saxena02 @alimaredia @shruthis4 @nabuskey @vara-bonthu @venkomirisetti @dineshkumar181094 @jacobsalway @tarekabouzeid please review this proposal. Thank You

[RobuRishabh]

Hello @kannon92 @mm4tt @helayoty could you please review this PR, and provide some feedback. Thank You

[kannon92]

I would go ahead and add support for this now in alpha.

Its what we did for JobSet/Job and its not really that much more of a lift.

[RobuRishabh]

Thanks we aligned with Andrey on the TrainJob/JobSet API approach (extensible .spec.scheduling wrapper so CPG/multi-leaf can land without a break).

For behavior, alpha still keeps exactly one executor leaf: Spark has one executor group today, driver stays outside the gang (deadlock), and nabuskey preferred not putting CPG into this phase-1 surface while there’s only one group. Happy to revisit pulling more into alpha if reviewers want the extra lift now.

[vikas-saxena02]

Thanks @RobuRishabh, the second-round revision is a big step forward. The alpha WorkloadSchedulingReady condition, persist-before-create submissionID with stale PodGroup cleanup, REST injection coverage, typed schedulerName conflicts, PartialRestart exclusion and the static post-admission replacement section all close real gaps.

I've left 5 inline comments. Summary, in priority order:

1. API shape mismatch with Add workload aware scheduling api #3230 (L349): the KEP specifies scheduling.schedulingPolicy.{basic|gang} on upstream KEP-6089 types and rejects gang.minCount in alpha; Add workload aware scheduling api #3230 adds scheduling.policy + scheduling.gang.minCount as Spark-owned types. Spark-owned types are fine by me, but the KEP and the API PR should agree before Add workload aware scheduling api #3230 merges. Add workload aware scheduling api #3230 also doesn't regenerate the CRD YAML/OpenAPI, so the field would be pruned as-is.
2. Feature-gate checks at admission (L433–L435): feature gates are only wired into the controller binary/chart, not the webhook, so "gate disabled → reject" and "PartialRestart enabled → reject" can't be enforced by admission today. Either wire gates into the webhook or enforce at reconcile. Same question for webhook.enable=false.
3. .spec.scheduling mutability (L141, also L447/L802): "rejected" and "invalidates on next submission" both appear; please pick one.
4. Admission deadline (L610): where it's configured, which failure state/retry budget it uses, how it behaves with restartPolicy: Always, and when the clock starts.
5. "Fails closed" on missing membership (L328): what action the controller takes beyond setting the condition, plus whether schedulingGroup is mutable post-create and how this depends on webhook.failurePolicy.

Happy to go through any of these at the next Spark Operator community call if that's easier than async.

[nabuskey]

Hmm so spec.scheduling is new. But we already have similar sounding fields like spec.batchScheduler, spec.batchSchedulerOptions, and schedulerName. So we'd have 4 similar sounding fields. Very confusing.
I think unifying them into the scheduling field, then deprecating the moved field is a better idea. In addition, avoid silently translating existing backends into native WAS. Existing ones do not translate well into WAS.

Each SparkApplication maps to one root Workload containing one or more PodGroupTemplates. The alpha implementation should enforce exactly one executor PodGroupTemplate because currently Spark supports exactly one executor group. Spark may decide to be more flexible in the future. We should be prepared for that.

In our repo, the order (driver pod first ALWAYS, then executors). The executors should be placed in a pod group(s) created by the operator. So this is where CompositePodGroup could be useful. However, I would not put this in this spec. As of right now, it's not useful because there's exactly one group.

As for ownership, in an ideal world, both Spark and the operator must not independently create PodGroups. I honestly don't know if this going to be true in the future. In our ideal world, the operator is the sole owner of PodGroups.

[RobuRishabh]

Hmm so spec.scheduling is new. But we already have similar sounding fields like spec.batchScheduler, spec.batchSchedulerOptions, and schedulerName. So we'd have 4 similar sounding fields. Very confusing. I think unifying them into the scheduling field, then deprecating the moved field is a better idea. In addition, avoid silently translating existing backends into native WAS. Existing ones do not translate well into WAS.

Each SparkApplication maps to one root Workload containing one or more PodGroupTemplates. The alpha implementation should enforce exactly one executor PodGroupTemplate because currently Spark supports exactly one executor group. Spark may decide to be more flexible in the future. We should be prepared for that.

In our repo, the order (driver pod first ALWAYS, then executors). The executors should be placed in a pod group(s) created by the operator. So this is where CompositePodGroup could be useful. However, I would not put this in this spec. As of right now, it's not useful because there's exactly one group.

As for ownership, in an ideal world, both Spark and the operator must not independently create PodGroups. I honestly don't know if this going to be true in the future. In our ideal world, the operator is the sole owner of PodGroups.

Thanks @nabuskey this is helpful. Taking the points in order:

Four similarly named fields
Agreed this KEP is so big that surface is confusing (batchScheduler, batchSchedulerOptions, typed/sparkConf schedulerName, and new .spec.scheduling).

For this alpha KEP I’m keeping:

• No silent translation of Volcano / YuniKorn / scheduler-plugins into native WAS (same as you, those backends don’t map cleanly).
• Mutual exclusion: explicit .spec.scheduling and explicit batchScheduler are rejected together; we don’t run both or invent a bridge.

Full unification under .spec.scheduling + deprecating the moved fields is the right long-term direction, but it’s a compatibility project beyond pinning native WAS. I’ll add it as an explicit Future Goal / Open Question (e.g. nest or replace batchScheduler* under scheduling, deprecate the old fields over a release window) rather than do that migration in alpha. Happy to adjust if you’d rather block alpha until that unify plan is sketched in this KEP.

One Workload, one executor PodGroupTemplate in alpha
Aligned. Alpha enforces exactly one executor leaf (Spark has one executor group today). The API/Workload model should stay open to more templates later without a break, I’ll make that “prepared for N templates / multi-group later” explicit, without exposing multi-group in alpha.

Driver first, then executors / CompositePodGroup
Also aligned with the KEP: driver always schedules first (outside the executor gang); operator creates the executor PodGroup(s). CompositePodGroup is useful if/when we need driver+executor composition or multiple executor groups out of this alpha spec, as you said, while there’s only one group.

Ownership
Alpha position: Spark Operator is the sole creator/owner of Workload + attempt PodGroups for opted-in apps; users don’t set schedulingGroup, and we don’t want Spark to create a second competing PodGroup. I’ll note the future risk if Spark grows its own WAS path, and that we’d need an explicit contract then so they don’t both create groups.

I’ll fold the unify-as-future + “one template now / N later” + operator-owned PodGroups wording into the next KEP revision.

[nabuskey]

Full unification under .spec.scheduling + deprecating the moved fields is the right long-term direction, but it’s a compatibility project beyond pinning native WAS.

I disagree. We are introducing a new field that sounds very very similar to the fields mentioned. What I care the most is the surface at which users operate. The scheduler options are confusing as-is imo. We need a concrete plan for deprecating and moving them under the new field instead of Future Goal / Open Question.

[RobuRishabh]

Full unification under .spec.scheduling + deprecating the moved fields is the right long-term direction, but it’s a compatibility project beyond pinning native WAS.

I disagree. We are introducing a new field that sounds very very similar to the fields mentioned. What I care the most is the surface at which users operate. The scheduler options are confusing as-is imo. We need a concrete plan for deprecating and moving them under the new field instead of Future Goal / Open Question.

@nabuskey ok so help me out here, I’d like your guidance on the target shape so I don’t invent the wrong nesting:

1. Where should legacy backends live under .spec.scheduling?
2. What about typed / sparkConf schedulerName?
3. What Alpha sequencing you’d prefer: ?
   Happy to implement whatever you prefer once that’s clear.

[nabuskey]

1. Yes. I wouldn't call them legacy though.

spec:
  scheduling:
     schedulingPolicy:       
        gang: {}
     batchScheduler:          
      name: volcano          # volcano | yunikorn
      queue: default
      priorityClassName: high
      resources: {}

• CEL on scheduling. Something like: !(has(self.schedulingPolicy) && has(self.batchScheduler)).

2. Typed and sparkConf schedulerName fields stay where they are and are not deprecated. They are Pod-level settings.
3. The flow would be something like:

• API changes. Add scheduling spec.
• Ensure current behaviours are preserved with either fields.
• Native WAS alpha implementations behind feature gate.

Also note that the doc is quite a bit convoluted with duplication.

[RobuRishabh]

1. Yes. I wouldn't call them legacy though.

spec:
  scheduling:
     schedulingPolicy:       
        gang: {}
     batchScheduler:          
      name: volcano          # volcano | yunikorn
      queue: default
      priorityClassName: high
      resources: {}

• CEL on scheduling. Something like: !(has(self.schedulingPolicy) && has(self.batchScheduler)).

2. Typed and sparkConf schedulerName fields stay where they are and are not deprecated. They are Pod-level settings.
3. The flow would be something like:

• API changes. Add scheduling spec.
• Ensure current behaviours are preserved with either fields.
• Native WAS alpha implementations behind feature gate.

Also note that the doc is quite a bit convoluted with duplication.

Thanks @nabuskey this is very clear to lock in. I’ll treat it as the plan. Few quick confirmations:

• the intended shape is scheduling.batchScheduler.{name,queue,priorityClassName,resources} correct?
• Suppose if someone set both spec.batchScheduler and spec.scheduling.batchScheduler means top-level and nested batch fields are both present, should we reject, or prefer nested?
• the concrete plan would be to basically warn on top-level starting when nested ships for deprecation and, keep fields readable for ≥2 minors (or indefinitely with +deprecated), and migrate docs/examples to nested only?
• Controller also treats kube-scheduler / scheduler-plugins style names right ?? so thisscheduling.batchScheduler.name still allows kube-scheduler / scheduler-plugins values, not only volcano | yunikorn? so --default-batch-scheduler still applies when .spec.scheduling is nil / has neither child?

Once you confirm these, I’ll update the KEP with this as the written migration plan (and slim the doc).