Repository navigation
Add HyperJob user guide and example manifests #63
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,44 @@ | ||
| # Case 3: Training with Cluster Affinity | ||
| # | ||
| # A team runs a fault-tolerant training job across three clusters with a minimum | ||
| # availability requirement. The job uses clusterNames to pin each replica to a | ||
| # specific cluster, while minAvailable allows the HyperJob to continue even if | ||
| # one replica fails. | ||
| # | ||
| # The controller creates three VCJobs (resilient-training-trainer-0, resilient-training-trainer-1, | ||
| # resilient-training-trainer-2), each scheduled to their respective preferred cluster. | ||
| # The HyperJob is considered healthy as long as at least 2 out of 3 jobs are running. | ||
|
Comment on lines
+9
to
+10
|
||
| apiVersion: training.volcano.sh/v1alpha1 | ||
| kind: HyperJob | ||
| metadata: | ||
| name: resilient-training | ||
| namespace: default | ||
| spec: | ||
| minAvailable: 2 | ||
| replicatedJobs: | ||
| - name: trainer | ||
| replicas: 3 | ||
| clusterNames: | ||
| - cluster-gpu-west-1 | ||
| - cluster-gpu-east-1 | ||
| - cluster-gpu-central-1 | ||
| templateSpec: | ||
| minAvailable: 1 | ||
| schedulerName: volcano | ||
| policies: | ||
| - event: PodEvicted | ||
| action: RestartJob | ||
| tasks: | ||
| - name: worker | ||
| replicas: 32 | ||
| template: | ||
| spec: | ||
| containers: | ||
| - name: trainer | ||
| image: training-image:v1 | ||
| resources: | ||
| requests: | ||
| nvidia.com/gpu: "1" | ||
| limits: | ||
| nvidia.com/gpu: "1" | ||
| restartPolicy: OnFailure | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,78 @@ | ||
| # Case 2: Heterogeneous Clusters Training | ||
| # | ||
| # An organization has multiple clusters with different generations of Ascend NPU | ||
| # accelerators (910B and 910C). This HyperJob splits the workload across these | ||
| # heterogeneous clusters, with each ReplicatedJob targeting specific hardware. | ||
| # | ||
| # The controller creates: | ||
| # - llm-heterogeneous-trainer-910b-0 (VCJob + PropagationPolicy) → cluster-ascend-910b-1 or cluster-ascend-910b-2 | ||
| # - llm-heterogeneous-trainer-910c-0 (VCJob + PropagationPolicy) → cluster-ascend-910c-1 | ||
| apiVersion: training.volcano.sh/v1alpha1 | ||
| kind: HyperJob | ||
| metadata: | ||
| name: llm-heterogeneous | ||
| namespace: default | ||
| spec: | ||
| minAvailable: 2 | ||
| replicatedJobs: | ||
| - name: trainer-910b | ||
| replicas: 1 | ||
| clusterNames: | ||
| - cluster-ascend-910b-1 | ||
| - cluster-ascend-910b-2 | ||
| templateSpec: | ||
| minAvailable: 1 | ||
| schedulerName: volcano | ||
| tasks: | ||
| - name: worker | ||
| replicas: 64 | ||
| template: | ||
| spec: | ||
| affinity: | ||
| nodeAffinity: | ||
| requiredDuringSchedulingIgnoredDuringExecution: | ||
| nodeSelectorTerms: | ||
| - matchExpressions: | ||
| - key: hardware-type | ||
| operator: In | ||
| values: | ||
| - Ascend910B | ||
| containers: | ||
| - name: trainer | ||
| image: training-image:v1 | ||
| resources: | ||
| requests: | ||
| huawei.com/ascend910b: "1" | ||
| limits: | ||
| huawei.com/ascend910b: "1" | ||
| restartPolicy: OnFailure | ||
| - name: trainer-910c | ||
| replicas: 1 | ||
| clusterNames: | ||
| - cluster-ascend-910c-1 | ||
| templateSpec: | ||
| minAvailable: 1 | ||
| schedulerName: volcano | ||
| tasks: | ||
| - name: worker | ||
| replicas: 64 | ||
| template: | ||
| spec: | ||
| affinity: | ||
| nodeAffinity: | ||
| requiredDuringSchedulingIgnoredDuringExecution: | ||
| nodeSelectorTerms: | ||
| - matchExpressions: | ||
| - key: hardware-type | ||
| operator: In | ||
| values: | ||
| - Ascend910C | ||
| containers: | ||
| - name: trainer | ||
| image: training-image:v1 | ||
| resources: | ||
| requests: | ||
| huawei.com/ascend910c: "1" | ||
| limits: | ||
| huawei.com/ascend910c: "1" | ||
| restartPolicy: OnFailure |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,35 @@ | ||
| # Case 1: Large-scale Training Job Splitting | ||
| # | ||
| # A research team wants to train a large language model that requires 256 GPUs, | ||
| # but their largest cluster only has 128 GPUs. Using HyperJob, the training job | ||
| # is split into two sub-jobs, each with 128 GPUs, running across two clusters. | ||
| # | ||
| # The controller creates two VCJobs (llm-training-trainer-0, llm-training-trainer-1) | ||
| # with a PropagationPolicy for each, distributing them across available clusters. | ||
| apiVersion: training.volcano.sh/v1alpha1 | ||
| kind: HyperJob | ||
| metadata: | ||
| name: llm-training | ||
| namespace: default | ||
| spec: | ||
| minAvailable: 2 | ||
| replicatedJobs: | ||
| - name: trainer | ||
| replicas: 2 | ||
| templateSpec: | ||
| minAvailable: 1 | ||
| schedulerName: volcano | ||
| tasks: | ||
| - name: worker | ||
| replicas: 128 | ||
| template: | ||
| spec: | ||
| containers: | ||
| - name: trainer | ||
| image: training-image:v1 | ||
| resources: | ||
| requests: | ||
| nvidia.com/gpu: "1" | ||
| limits: | ||
| nvidia.com/gpu: "1" | ||
| restartPolicy: OnFailure |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The header comment says
clusterNamespins each replica to a specific cluster and that each VCJob will be scheduled to its “respective” preferred cluster. In practice the controller setsclusterAffinity.clusterNamesfor every generated PropagationPolicy, which only restricts eligible clusters; it does not guarantee a 1:1 mapping between replicas and clusters or that replicas land on distinct clusters. Please adjust the wording to avoid over-promising placement behavior.