Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -89,13 +89,13 @@ Examples:

The configurations below were verified on a 64-chip TPU Ironwood (`v7x`) slice (`tpu7x-4x4x4`, 128 TensorCore devices, `1,153.5 TFLOP/s/core` / `2,307 TFLOP/s/chip` BF16 peak):

| Context Length | Parallelism & Offload Config | UBench Run Name & BigQuery `run_id` | Steady-State Step Time | Throughput (`TFLOP/s` & `Tokens/s/core`) | Measured MFU (%) |
| :--- | :--- | :--- | :--- | :--- | :--- |
| **1M** (`1,048,576`) | `CP=32, FSDP=4`, `pdbs=0.03125`, `sa_block_*=2048`, `context=device`, `dvfs_p_state=7` | `chengnuojin-ubench-iwwssuhp` (`maxtext_training_with_xpk-llama3_1_8b-2026-10-05_172350-76c5205d-c288-4aed-89e2-375398c92c9a`) | **49.58 s** | **574.8 TFLOP/s** (`1,149.6` /chip, `660.9 tok/s/core`) | **49.98%** (`50.90%` in `B3F_1m_cp32_ideal`) |
| **2M** (`2,097,152`) | `CP=64, FSDP=2`, `pdbs=0.015625`, `sa_block_*=2048`, `context=device`, `dvfs_p_state=7` | `chengnuojin-ubench-c79shbyc` (`maxtext_training_with_xpk-llama3_1_8b-2026-10-05_174711-a6caf10a-50e6-423e-a0ad-ef2090917698`) | **94.60 s** | **586.9 TFLOP/s** (`1,173.8` /chip, `346.4 tok/s/core`) | **51.03%** (`50.80%` in `B3F_2m_cp64_IDEALBRACKET`) |
| **4M** (`4,194,304`) | `CP=128, FSDP=1`, `pdbs=0.0078125`, `sa_block_*=2048`, `context=device` | `chengnuojin-ubench-8vkgzrtc` (`maxtext_training_with_xpk-llama3_1_8b-2026-10-05_182345-03a74883-8a36-41ae-aace-a7e443e87e43`) | **213.68 s** (`186.48 s` w/ `p7`) | **512.7 TFLOP/s** (`1,025.5` /chip, `153.3 tok/s/core`) | **44.60%** (`50.93%` w/ `p7` in `B3F_4m_cp128_IDEALBRACKET`) |
| **10M** (`10,485,760`) | `CP=128, FSDP=1`, `pdbs=0.0078125`, `sa_block_*=2048`, `context=device`, `opt_off=true`, `dvfs_p_state=7` | `chengnuojin-ubench-kabnsj6h` (`maxtext_training_with_xpk-llama3_1_8b-2026-10-05_223247-053f00f3-e418-43ce-b1bf-0e567cec4625`) | **1,124.12 s** | **604.2 TFLOP/s** (`1,208.5` /chip, `72.9 tok/s/core`) | **52.54%** (`52.11%` in `B3F_10m_cp128_IDEALBRACKET`) |
| **14M** (`14,680,064`) | `CP=128, FSDP=1`, `pdbs=0.0078125`, `sa_block_*=2048`, `context=offload`, `decoder_layer_input=offload`, `opt_off=true`, `par_off=true`, `dvfs_p_state=7` | `chengnuojin-ubench-gz6ugsgn` (`maxtext_training_with_xpk-llama3_1_8b-2026-10-06_003849-40f0a6b0-15e9-49c5-a336-e52c036019c4`) | **2,360.00 s** | **563.2 TFLOP/s** (`1,126.5` /chip, `48.6 tok/s/core`) | **48.83%** (`48.71%` in `B3F_14m_cp128_inoff`) |
| Context Length | Parallelism & Offload Config | Steady-State Step Time | Throughput (`TFLOP/s` & `Tokens/s/core`) | Measured MFU (%) |
| :--- | :--- | :--- | :--- | :--- |
| **1M** (`1,048,576`) | `CP=32, FSDP=4`, `pdbs=0.03125`, `sa_block_*=2048`, `context=device`, `dvfs_p_state=7` | **49.58 s** | **574.8 TFLOP/s** (`1,149.6` /chip, `660.9 tok/s/core`) | **49.98%** (`50.90%` in `B3F_1m_cp32_ideal`) |
| **2M** (`2,097,152`) | `CP=64, FSDP=2`, `pdbs=0.015625`, `sa_block_*=2048`, `context=device`, `dvfs_p_state=7` | **94.60 s** | **586.9 TFLOP/s** (`1,173.8` /chip, `346.4 tok/s/core`) | **51.03%** (`50.80%` in `B3F_2m_cp64_IDEALBRACKET`) |
| **4M** (`4,194,304`) | `CP=128, FSDP=1`, `pdbs=0.0078125`, `sa_block_*=2048`, `context=device` | **213.68 s** (`186.48 s` w/ `p7`) | **512.7 TFLOP/s** (`1,025.5` /chip, `153.3 tok/s/core`) | **44.60%** (`50.93%` w/ `p7` in `B3F_4m_cp128_IDEALBRACKET`) |
| **10M** (`10,485,760`) | `CP=128, FSDP=1`, `pdbs=0.0078125`, `sa_block_*=2048`, `context=device`, `opt_off=true`, `dvfs_p_state=7` | **1,124.12 s** | **604.2 TFLOP/s** (`1,208.5` /chip, `72.9 tok/s/core`) | **52.54%** (`52.11%` in `B3F_10m_cp128_IDEALBRACKET`) |
| **14M** (`14,680,064`) | `CP=128, FSDP=1`, `pdbs=0.0078125`, `sa_block_*=2048`, `context=offload`, `decoder_layer_input=offload`, `opt_off=true`, `par_off=true`, `dvfs_p_state=7` | **2,360.00 s** | **563.2 TFLOP/s** (`1,126.5` /chip, `48.6 tok/s/core`) | **48.83%** (`48.71%` in `B3F_14m_cp128_inoff`) |

## Cleanup

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -54,7 +54,7 @@ configuration to your cluster.
Open a terminal and set the following environment variables to match your setup.
**Note:**
- `k8s_manifest.yaml` is in the same directory as this README.
- For WORKLOAD_IMAGE, see [Docker container image](../xpk/README.md#docker-container-image) section.
- For WORKLOAD_IMAGE, see [Docker container image](../cluster_toolkit/README.md#docker-container-image) section.

```bash
# Set variables for your environment
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -120,8 +120,7 @@ export RESERVATION_NAME="<your-reservation-name>"
empty on a cluster created from the blueprint below: it installs a single
LocalQueue (`user-queue`) in `${NAMESPACE}`, which `gcluster job submit`
discovers on its own. Set it only when the namespace holds several
LocalQueues, for example `multislice-queue` on a cluster provisioned with
XPK.
LocalQueues, for example on a custom or shared GKE cluster.
- `BUCKET_NAME` is the GCS bucket the workload writes to. Storage mounts and
paths in `run_recipe.sh` that pointed at the original run's
bucket read it (or `BASE_OUTPUT_DIR`) instead.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -120,8 +120,7 @@ export RESERVATION_NAME="<your-reservation-name>"
empty on a cluster created from the blueprint below: it installs a single
LocalQueue (`user-queue`) in `${NAMESPACE}`, which `gcluster job submit`
discovers on its own. Set it only when the namespace holds several
LocalQueues, for example `multislice-queue` on a cluster provisioned with
XPK.
LocalQueues, for example on a custom or shared GKE cluster.
- `BUCKET_NAME` is the GCS bucket the workload writes to. Storage mounts and
paths in `run_recipe.sh` that pointed at the original run's
bucket read it (or `BASE_OUTPUT_DIR`) instead.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -120,8 +120,7 @@ export RESERVATION_NAME="<your-reservation-name>"
empty on a cluster created from the blueprint below: it installs a single
LocalQueue (`user-queue`) in `${NAMESPACE}`, which `gcluster job submit`
discovers on its own. Set it only when the namespace holds several
LocalQueues, for example `multislice-queue` on a cluster provisioned with
XPK.
LocalQueues, for example on a custom or shared GKE cluster.
- `BUCKET_NAME` is the GCS bucket the workload writes to. Storage mounts and
paths in `run_recipe.sh` that pointed at the original run's
bucket read it (or `BASE_OUTPUT_DIR`) instead.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -120,8 +120,7 @@ export RESERVATION_NAME="<your-reservation-name>"
empty on a cluster created from the blueprint below: it installs a single
LocalQueue (`user-queue`) in `${NAMESPACE}`, which `gcluster job submit`
discovers on its own. Set it only when the namespace holds several
LocalQueues, for example `multislice-queue` on a cluster provisioned with
XPK.
LocalQueues, for example on a custom or shared GKE cluster.
- `BUCKET_NAME` is the GCS bucket the workload writes to. Storage mounts and
paths in `run_recipe.sh` that pointed at the original run's
bucket read it (or `BASE_OUTPUT_DIR`) instead.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -120,8 +120,7 @@ export RESERVATION_NAME="<your-reservation-name>"
empty on a cluster created from the blueprint below: it installs a single
LocalQueue (`user-queue`) in `${NAMESPACE}`, which `gcluster job submit`
discovers on its own. Set it only when the namespace holds several
LocalQueues, for example `multislice-queue` on a cluster provisioned with
XPK.
LocalQueues, for example on a custom or shared GKE cluster.
- `BUCKET_NAME` is the GCS bucket the workload writes to. Storage mounts and
paths in `run_recipe.sh` that pointed at the original run's
bucket read it (or `BASE_OUTPUT_DIR`) instead.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -120,8 +120,7 @@ export RESERVATION_NAME="<your-reservation-name>"
empty on a cluster created from the blueprint below: it installs a single
LocalQueue (`user-queue`) in `${NAMESPACE}`, which `gcluster job submit`
discovers on its own. Set it only when the namespace holds several
LocalQueues, for example `multislice-queue` on a cluster provisioned with
XPK.
LocalQueues, for example on a custom or shared GKE cluster.
- `BUCKET_NAME` is the GCS bucket the workload writes to. Storage mounts and
paths in `run_recipe.sh` that pointed at the original run's
bucket read it (or `BASE_OUTPUT_DIR`) instead.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -48,7 +48,6 @@ spec:
- bash
- -c
- |
echo XPK Start: $(date);
_sigterm() (kill -SIGTERM $! 2>/dev/null;);
trap _sigterm SIGTERM;
(export TPU_STDERR_LOG_LEVEL=0 && export TPU_MIN_LOG_LEVEL=0 && export TF_CPP_MIN_LOG_LEVEL=0 && export TPU_VMODULE=real_program_continuator=1 && set -e && export ENABLE_PATHWAYS_PERSISTENCE='1' && export JAX_PLATFORMS='tpu,cpu' && export ENABLE_PJRT_COMPATIBILITY='true' && pip install . && export LIBTPU_INIT_ARGS='--xla_enable_async_all_gather=true --xla_tpu_enable_async_collective_fusion=false --xla_tpu_enable_async_collective_fusion_fuse_all_gather=false --xla_enable_async_all_reduce=true --xla_tpu_enable_sparse_core_collective_offload_all_reduce=true --xla_max_concurrent_async_all_gathers=4 --xla_tpu_enable_async_all_to_all=true --xla_latency_hiding_scheduler_rerun=5 --xla_tpu_rwb_fusion=false --xla_tpu_enable_sublane_major_scaling_bitcast_fusion=false --xla_tpu_impure_enable_packed_bf16_math_ops=false --xla_tpu_enable_sparse_core_reduce_scatter_v2=true --xla_tpu_enable_sparse_core_collective_offload_all_gather=true --xla_tpu_enable_sparse_core_collective_offload_2d_all_gather=true --xla_tpu_enable_all_gather_offload_tracing=true --xla_tpu_use_tc_device_shape_on_sc=true --xla_tpu_prefer_async_allgather_to_allreduce=true --xla_tpu_enable_sparse_core_collective_offload_reduce_scatter=true --xla_tpu_scoped_vmem_limit_kib=65472 --xla_tpu_enable_tpu_custom_call_scoped_vmem_adjustments=true --xla_enable_transpose_trace=false ' && echo 'Starting WAN training ...' && HF_HUB_CACHE=/dev/shm python3 -m src.maxdiffusion.train_wan src/maxdiffusion/configs/base_wan_14b.yml model_name=wan2.1 attention=flash weights_dtype=bfloat16 activations_dtype=bfloat16 guidance_scale=5.0 flow_shift=5.0 fps=16 skip_jax_distributed_system=False output_dir=${BASE_OUTPUT_DIR}/ train_data_dir=${DATASET_DIR} load_tfrecord_cached=True height=1280 width=720 num_frames=81 num_inference_steps=50 prompt='a japanese pop star young woman with black hair is singing with a smile. She is inside a studio with dim lighting and musical instruments.' jax_cache_dir=${BASE_OUTPUT_DIR}/jax_cache/ max_train_steps=150 enable_profiler=True dataset_save_location=${DATASET_DIR} remat_policy=FULL flash_min_seq_length=0 seed=123456789 skip_first_n_steps_for_profiler=5 profiler_steps=10 per_device_batch_size=0.25 ici_data_parallelism=32 ici_context_parallelism=4 ici_tensor_parallelism=1 vae_spatial=1 replicate_vae=True allow_split_physical_axes=True base_output_directory=${BASE_OUTPUT_DIR} run_name=${WORKLOAD_NAME}) & PID=$!;
Expand All @@ -57,7 +56,6 @@ spec:
done;
wait $PID;
EXIT_CODE=$?;
echo XPK End: $(date);
echo EXIT_CODE=$EXIT_CODE;
exit $EXIT_CODE
resources:
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -122,8 +122,7 @@ export RESERVATION_NAME="<your-reservation-name>"
empty on a cluster created from the blueprint below: it installs a single
LocalQueue (`user-queue`) in `${NAMESPACE}`, which `gcluster job submit`
discovers on its own. Set it only when the namespace holds several
LocalQueues, for example `multislice-queue` on a cluster provisioned with
XPK.
LocalQueues, for example on a custom or shared GKE cluster.
- `BUCKET_NAME` is the GCS bucket the workload writes to. Storage mounts and
paths in `run_recipe.sh` that pointed at the original run's
bucket read it (or `BASE_OUTPUT_DIR`) instead.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -85,13 +85,13 @@ Examples:

The configurations below were verified on a 64-chip TPU v5p slice (`v5p-128`, topology `4x4x4`, 64 Megacore devices, `459 TFLOP/s/device` BF16 peak):

| Context Length | Parallelism & Offload Config | UBench Run Name & BigQuery `run_id` | Steady-State Step Time | Throughput (`TFLOP/s` & `Tokens/s/chip`) | Measured MFU (%) |
| :--- | :--- | :--- | :--- | :--- | :--- |
| **1M** (`1,048,576`) | `CP=64, FSDP=1`, `pdbs=0.03125`, `sa_block_*=1024`, `context=offload` | `chengnuojin-ubench-wm87mo94` (`maxtext_training_with_xpk-llama3_1_8b-2026-10-05_173709-20730566-a238-463e-918e-1f353c039538`) | **95.73 s** | **297.7 TFLOP/s** (`342.4 tok/s/chip`) | **64.86%** (`68.75%` in `O_h2_1024k_ring_cp64_b2_r2`) |
| **2M** (`2,097,152`) | `CP=64, FSDP=1`, `pdbs=0.03125`, `sa_block_*=1024`, `context=offload`, `decoder_layer_input=offload` | `chengnuojin-ubench-w3jvwbkr` (`maxtext_training_with_xpk-llama3_1_8b-2026-10-05_183624-083a4c98-0783-46af-a0a8-cd91be9efeb5`) | **357.68 s** | **310.4 TFLOP/s** (`183.2 tok/s/chip`) | **67.63%** (`69.88%` in `O_r5_2m_ring_b2_inoff`) |
| **4M** (`4,194,304`) | `CP=64, FSDP=1`, `pdbs=0.03125`, `sa_block_*=1024`, `context=offload`, `decoder_layer_input=offload` | `chengnuojin-ubench-asb6kxmv` (`maxtext_training_with_xpk-llama3_1_8b-2026-10-05_215125-766e1e52-4ecf-4a87-a439-8a0d23bacc1c`) | **1,373.73 s** | **319.0 TFLOP/s** (`95.4 tok/s/chip`) | **69.50%** (`70.29%` in `O_r5_4m_ring_b2_inoff`) |
| **10M** (`10,485,760`) | `CP=64, FSDP=1`, `pdbs=0.015625`, `sa_block_*=1024`, `remat_policy=full` | `chengnuojin-ubench-uujda4kd` (`maxtext_training_with_xpk-llama3_1_8b-2026-10-06_140929-befcf112-30da-4290-a424-9d9f39be4deb`) | **8,060.20 s** (`8,056.40 s` ref) | **168.5 TFLOP/s** (`20.3 tok/s/chip`) | **36.72% – 36.74%** |
| **14M** (`14,680,064`) | `CP=64, FSDP=1`, `pdbs=0.015625`, `sa_block_*=1024`, `context=remat`, `decoder_layer_input=offload` | `chengnuojin-ubench-bjzjtsvd` (`maxtext_training_with_xpk-llama3_1_8b-2026-10-06_141408-f2a71578-1159-4612-b1a5-c97197dd1493`) | **15,766.82 s** (`15,623.09 s` ref) | **168.6 TFLOP/s** (`170.2` ref, `14.5 tok/s/chip`) | **36.73% – 37.07%** |
| Context Length | Parallelism & Offload Config | Steady-State Step Time | Throughput (`TFLOP/s` & `Tokens/s/chip`) | Measured MFU (%) |
| :--- | :--- | :--- | :--- | :--- |
| **1M** (`1,048,576`) | `CP=64, FSDP=1`, `pdbs=0.03125`, `sa_block_*=1024`, `context=offload` | **95.73 s** | **297.7 TFLOP/s** (`342.4 tok/s/chip`) | **64.86%** (`68.75%` in `O_h2_1024k_ring_cp64_b2_r2`) |
| **2M** (`2,097,152`) | `CP=64, FSDP=1`, `pdbs=0.03125`, `sa_block_*=1024`, `context=offload`, `decoder_layer_input=offload` | **357.68 s** | **310.4 TFLOP/s** (`183.2 tok/s/chip`) | **67.63%** (`69.88%` in `O_r5_2m_ring_b2_inoff`) |
| **4M** (`4,194,304`) | `CP=64, FSDP=1`, `pdbs=0.03125`, `sa_block_*=1024`, `context=offload`, `decoder_layer_input=offload` | **1,373.73 s** | **319.0 TFLOP/s** (`95.4 tok/s/chip`) | **69.50%** (`70.29%` in `O_r5_4m_ring_b2_inoff`) |
| **10M** (`10,485,760`) | `CP=64, FSDP=1`, `pdbs=0.015625`, `sa_block_*=1024`, `remat_policy=full` | **8,060.20 s** (`8,056.40 s` ref) | **168.5 TFLOP/s** (`20.3 tok/s/chip`) | **36.72% – 36.74%** |
| **14M** (`14,680,064`) | `CP=64, FSDP=1`, `pdbs=0.015625`, `sa_block_*=1024`, `context=remat`, `decoder_layer_input=offload` | **15,766.82 s** (`15,623.09 s` ref) | **168.6 TFLOP/s** (`170.2` ref, `14.5 tok/s/chip`) | **36.73% – 37.07%** |

## Cleanup

Expand Down