Skip to content

Migrate remaining Ironwood training recipes from XPK to Cluster Toolkit - #392

Merged
ankitkinra merged 2 commits into
AI-Hypercomputer:mainfrom
cj-klein:migrate-remaining-ironwood-recipes-to-ctk
Nov 15, 2026
Merged

ankitkinra merged 2 commits into
AI-Hypercomputer:mainfrom
cj-klein:migrate-remaining-ironwood-recipes-to-ctk

Conversation

@cj-klein

Copy link
Copy Markdown
Contributor

As part of the XPK deprecation, this PR migrates the remaining three Ironwood (tpu7x) training recipes from XPK to Cluster Toolkit (cluster_toolkit/ and k8s/) and updates training/ironwood/README.md with hardware-validated benchmark metrics:

  • training/ironwood/llama3.1-70b/8k-bf16-tpu7x-4x8x8/ (12.48s step time, 1,179.62 TFLOPs/s/chip, 2,625.91 tokens/s/chip)
  • training/ironwood/llama3.1-70b/8k-fp8-tpu7x-4x8x8/ (8.15s step time, 1,805.78 TFLOPs/s/chip, 4,019.79 tokens/s/chip)
  • training/ironwood/wan2.2-t2v-14b/bf16-tpu7x-2x4x4/ (12.08s step time, 863.44 TFLOPs/s/chip)

For each recipe directory:

  • Removed the legacy XPK recipe files
  • Added cluster_toolkit/ (README.md, run_recipe.sh, run_workload.sh)
  • Added or replaced k8s/k8s_manifest.yaml from the validated Cluster Toolkit (gcluster v1.108.0) execution

Migrates the 3 remaining Ironwood (tpu7x) pretraining recipes (MaxText and MaxDiffusion) from XPK to Cluster Toolkit (gcluster v1.108.0) with rendered Kubernetes JobSet manifests, removes their legacy XPK recipes, and updates training/ironwood/README.md benchmark metrics:

- training/ironwood/llama3.1-70b/8k-bf16-tpu7x-4x8x8/
- training/ironwood/llama3.1-70b/8k-fp8-tpu7x-4x8x8/
- training/ironwood/wan2.2-t2v-14b/bf16-tpu7x-2x4x4/

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request migrates several training recipes on Ironwood TPU (including llama3.1-70b and wan2.2-t2v-14b) from XPK-based deployments to Cluster Toolkit (gcluster) deployments. This involves adding new Cluster Toolkit configuration files, run scripts, and updated Kubernetes manifests, while removing the old XPK-related files. Feedback on the changes highlights several instances of dead code in the newly added run_recipe.sh and run_workload.sh scripts for both the bf16 and fp8 llama3.1-70b recipes, specifically pointing out that the reinforcement learning (RL) conditional blocks always evaluate to false and should be removed.

@cj-klein
cj-klein marked this pull request as ready for review November 15, 2026 07:30
@ankitkinra
ankitkinra merged commit 00250bf into AI-Hypercomputer:main Nov 15, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants