Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
44 changes: 44 additions & 0 deletions docs/deploy/example/hyperjob/cluster-affinity-training.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,44 @@
# Case 3: Training with Cluster Affinity
#
# A team runs a fault-tolerant training job across three clusters with a minimum
# availability requirement. The job uses clusterNames to pin each replica to a
# specific cluster, while minAvailable allows the HyperJob to continue even if
# one replica fails.
#
# The controller creates three VCJobs (resilient-training-trainer-0, resilient-training-trainer-1,
# resilient-training-trainer-2), each scheduled to their respective preferred cluster.
# The HyperJob is considered healthy as long as at least 2 out of 3 jobs are running.
Comment on lines +3 to +10

Copilot AI Mar 10, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The header comment says clusterNames pins each replica to a specific cluster and that each VCJob will be scheduled to its “respective” preferred cluster. In practice the controller sets clusterAffinity.clusterNames for every generated PropagationPolicy, which only restricts eligible clusters; it does not guarantee a 1:1 mapping between replicas and clusters or that replicas land on distinct clusters. Please adjust the wording to avoid over-promising placement behavior.

Copilot uses AI. Check for mistakes.
Comment on lines +9 to +10

Copilot AI Mar 10, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This example text implies spec.minAvailable: 2 makes the HyperJob “considered healthy” while 2/3 jobs are running, but HyperJob.spec.minAvailable is currently not enforced by the controller. Consider removing/softening this claim or adding a note that it’s reserved for future fault-tolerance behavior.

Copilot uses AI. Check for mistakes.
apiVersion: training.volcano.sh/v1alpha1
kind: HyperJob
metadata:
name: resilient-training
namespace: default
spec:
minAvailable: 2
replicatedJobs:
- name: trainer
replicas: 3
clusterNames:
- cluster-gpu-west-1
- cluster-gpu-east-1
- cluster-gpu-central-1
templateSpec:
minAvailable: 1
schedulerName: volcano
policies:
- event: PodEvicted
action: RestartJob
tasks:
- name: worker
replicas: 32
template:
spec:
containers:
- name: trainer
image: training-image:v1
resources:
requests:
nvidia.com/gpu: "1"
limits:
nvidia.com/gpu: "1"
restartPolicy: OnFailure
78 changes: 78 additions & 0 deletions docs/deploy/example/hyperjob/heterogeneous-training.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,78 @@
# Case 2: Heterogeneous Clusters Training
#
# An organization has multiple clusters with different generations of Ascend NPU
# accelerators (910B and 910C). This HyperJob splits the workload across these
# heterogeneous clusters, with each ReplicatedJob targeting specific hardware.
#
# The controller creates:
# - llm-heterogeneous-trainer-910b-0 (VCJob + PropagationPolicy) → cluster-ascend-910b-1 or cluster-ascend-910b-2
# - llm-heterogeneous-trainer-910c-0 (VCJob + PropagationPolicy) → cluster-ascend-910c-1
apiVersion: training.volcano.sh/v1alpha1
kind: HyperJob
metadata:
name: llm-heterogeneous
namespace: default
spec:
minAvailable: 2
replicatedJobs:
- name: trainer-910b
replicas: 1
clusterNames:
- cluster-ascend-910b-1
- cluster-ascend-910b-2
templateSpec:
minAvailable: 1
schedulerName: volcano
tasks:
- name: worker
replicas: 64
template:
spec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: hardware-type
operator: In
values:
- Ascend910B
containers:
- name: trainer
image: training-image:v1
resources:
requests:
huawei.com/ascend910b: "1"
limits:
huawei.com/ascend910b: "1"
restartPolicy: OnFailure
- name: trainer-910c
replicas: 1
clusterNames:
- cluster-ascend-910c-1
templateSpec:
minAvailable: 1
schedulerName: volcano
tasks:
- name: worker
replicas: 64
template:
spec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: hardware-type
operator: In
values:
- Ascend910C
containers:
- name: trainer
image: training-image:v1
resources:
requests:
huawei.com/ascend910c: "1"
limits:
huawei.com/ascend910c: "1"
restartPolicy: OnFailure
35 changes: 35 additions & 0 deletions docs/deploy/example/hyperjob/llm-training.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,35 @@
# Case 1: Large-scale Training Job Splitting
#
# A research team wants to train a large language model that requires 256 GPUs,
# but their largest cluster only has 128 GPUs. Using HyperJob, the training job
# is split into two sub-jobs, each with 128 GPUs, running across two clusters.
#
# The controller creates two VCJobs (llm-training-trainer-0, llm-training-trainer-1)
# with a PropagationPolicy for each, distributing them across available clusters.
apiVersion: training.volcano.sh/v1alpha1
kind: HyperJob
metadata:
name: llm-training
namespace: default
spec:
minAvailable: 2
replicatedJobs:
- name: trainer
replicas: 2
templateSpec:
minAvailable: 1
schedulerName: volcano
tasks:
- name: worker
replicas: 128
template:
spec:
containers:
- name: trainer
image: training-image:v1
resources:
requests:
nvidia.com/gpu: "1"
limits:
nvidia.com/gpu: "1"
restartPolicy: OnFailure
Loading
Loading