Skip to content

[Ironwood][MaxText] Add Qwen3-0.6B GRPO RL recipe (Tunix + vLLM/tpu-inference + MaxText under Pathways) on tpu7x 4x4x4 - #387

Merged
joeywan-google merged 1 commit into
AI-Hypercomputer:mainfrom
gutianyu-google:rl-grpo-qwen3-0.6b-ironwood
Nov 13, 2026
Merged

joeywan-google merged 1 commit into
AI-Hypercomputer:mainfrom
gutianyu-google:rl-grpo-qwen3-0.6b-ironwood

Conversation

@gutianyu-google

@gutianyu-google gutianyu-google commented Nov 6, 2026 •

Copy link
Copy Markdown
Contributor

Adds a top-level rl/ directory for reinforcement-learning post-training recipes and
its first recipe: GRPO of Qwen3-0.6B on OpenMathInstruct-2 with MaxText + Tunix +
vLLM TPU (tpu-inference) under Pathways, on one 64-chip tpu7x 4x4x4 slice
(16 hosts, 128 TensorCores), trainer and sampler colocated.

Path: rl/ironwood/qwen3-0.6b-grpo/math-bf16-tpu7x-4x4x4/k8s/

File Contents
k8s_manifest.yaml the JobSet of the measured run, with five envsubst placeholders (run name, image, output bucket, checkpoint path, CPU node pool)
README.md cluster creation with the Cluster Toolkit gke-tpu-7x blueprint, image build with the dependency-pin change and a version check, checkpoint conversion, run, monitor, step-time rule, silent-failure table
read_step_time.py standard-library reader of perf/train/global_step_time applying the measurement rule
rl/README.md index of RL recipes, same table style as the training and inference indexes
README.md ./rl entry in the repository organization list

Reference result

Metric Value
perf/train/global_step_time, median over TensorBoard steps 3..20 (n = 18) 56.94 s
p10 / p90 55.2 s / 59.3 s
Measured 2026-09-21, 20 steps, next to a concurrent control run of the previous best recipe on the same cluster (56.96 s)

Stack

MaxText c28166ca7 with the three post-training pins in
src/dependencies/extra_deps/post_train_github_deps.txt moved to tunix a3e92fba,
tpu-inference c2289ee4b, vLLM 51da0ca6; Pathways 20260918-jax_0.11.0 by digest.
No source patches. All performance work behind this number is merged upstream in the
three repositories; the pin change is needed because MaxText's own pin file predates
those merges, and an image built without it runs about 2x slower with no error. The
launcher checks the installed commits at start-up and fails loudly if the pins were
not moved. The dataset and the model checkpoint are pinned to the Hugging Face
revisions the run used.

Workload

  • 256 prompts x 8 generations per step, prompt cap 4,096, completion cap 8,192
  • temperature 0.8, top-k 50, top-p 0.95; rollout TP 1 x DP 128; sequence packing on the trainer
  • lr 1e-6, no KL, clip 0.2 / 0.28; 20 steps

Validation

Measured on a tpu7x 4x4x4 GKE cluster with Pathways, run tianyu-rl-head-03,
2026-09-21, with the image built exactly as the README describes (image digest
verified against the four pinned commits and jax 0.11.0 before the run).

Check Result
Steps completed 0..19, exit code 0, 0 pod restarts
Launcher IMAGE-CHECK all pins present, 0 runtime patches
Step time, median steps 3..20 (n = 18) 56.94 s (p10 55.2 / p90 59.3)
Concurrent control, previous best configuration on the same cluster 56.96 s
Earlier build of the same recipe, 2026-09-18 57.92 s (its control 58.46 s; cluster about 1 s slower that day)
Generation contract across both builds mean completion length 4,148 to 4,747 tokens and reward sum 0.418 to 0.497 over steps 3..20, identical
read_step_time.py on the saved event files prints 56.94 s; also checked on a synthetic file with a step-0 record

Not covered: checkpoint restore and behaviour beyond 20 steps; the Cluster Toolkit
cluster-creation section was written from the gke-tpu-7x blueprint and its module
defaults, the measured cluster was created separately with an equivalent TPU pool.

@gutianyu-google
gutianyu-google marked this pull request as ready for review November 6, 2026 06:56

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces a new reinforcement learning recipe for Qwen3-0.6B using GRPO on Ironwood TPU GKE clusters, adding a comprehensive README, a Kubernetes JobSet manifest, and a helper Python script to parse step times from TensorBoard event files. The review feedback identifies a potential TypeError in the step-time parsing script when handling default protobuf values and suggests using a safer file-reading pattern. Additionally, it recommends using the captured PID variable instead of the generic shell variable in the manifest's signal handler to ensure robust process termination.

Comment on lines +72 to +80
def read(path):
data = open(path, 'rb').read()
i, out = 0, []
while i + 12 <= len(data):
(n,) = struct.unpack_from('<Q', data, i)
i += 12 # length + length-crc
if i + n + 4 > len(data): break
rec = data[i:i+n]; i += n + 4 # payload + payload-crc
wall = step = None; vals = []

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

In Python 3, comparing None with an integer (e.g., step >= a.min_step) raises a TypeError. In proto3, default values (such as 0 for the int64 step field) are omitted from the serialized representation. Consequently, for any event at step 0, the step field will not be present in the binary payload, leaving step as None. When the script processes these events, it will crash with a TypeError during the filtering step.\n\nAdditionally, using a with statement to open the file ensures that the file descriptor is properly closed.\n\nWe can fix both issues by defaulting step to 0 (the correct protobuf default value) and using a with statement to read the file.

def read(path):\n    with open(path, 'rb') as f:\n        data = f.read()\n    i, out = 0, []\n    while i + 12 <= len(data):\n        (n,) = struct.unpack_from('<Q', data, i)\n        i += 12                       # length + length-crc\n        if i + n + 4 > len(data): break\n        rec = data[i:i+n]; i += n + 4  # payload + payload-crc\n        wall = None; step = 0; vals = []

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

- -c
- |
echo Pathways Helm Start: $(date);
_sigterm() (kill -SIGTERM $! 2>/dev/null;);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

Using $! inside the signal handler can be unreliable if other background processes or subshells are spawned later, as $! always points to the most recently spawned background process. Since the PID is explicitly captured in the PID variable, it is safer and more robust to use "$PID" in the signal handler.

                _sigterm() { kill -SIGTERM \"$PID\" 2>/dev/null; };

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

@gutianyu-google
gutianyu-google force-pushed the rl-grpo-qwen3-0.6b-ironwood branch 3 times, most recently from c7bcbbb to 6a006a0 Compare November 13, 2026 08:48
@gutianyu-google gutianyu-google changed the title [Ironwood][MaxText] Add Qwen3-0.6B GRPO reinforcement-learning recipe on tpu7x 4x4x4 [Ironwood][MaxText] Add Qwen3-0.6B GRPO RL recipe (Tunix + vLLM/tpu-inference + MaxText under Pathways) on tpu7x 4x4x4 Nov 13, 2026
@gutianyu-google
gutianyu-google force-pushed the rl-grpo-qwen3-0.6b-ironwood branch from 6a006a0 to 209a599 Compare November 13, 2026 09:05
… on tpu7x 4x4x4

Adds a top-level rl/ directory for reinforcement-learning post-training recipes
and its first recipe: GRPO of Qwen3-0.6B on OpenMathInstruct-2 with MaxText +
Tunix + vLLM TPU under Pathways, one 64-chip tpu7x 4x4x4 slice, trainer and
sampler colocated. Reference step time 56.94 s (median of TensorBoard steps
3..20) measured on 2026-09-21 with an image built from MaxText c28166ca7 and its
three post-training pins moved to tunix a3e92fba / tpu-inference c2289ee4b /
vLLM 51da0ca6, no source patches.

Files:
- rl/README.md: index of RL recipes (layout mirrors training/ and inference/)
- rl/ironwood/qwen3-0.6b-grpo/math-bf16-tpu7x-4x4x4/k8s/k8s_manifest.yaml: the
  exact JobSet of the measured run with five envsubst placeholders (run name,
  image, output bucket, checkpoint path, CPU node pool); the dataset is pinned to
  the Hugging Face revision the run used
- .../k8s/README.md: cluster creation with the Cluster Toolkit gke-tpu-7x
  blueprint (Pathways CPU pool cpu-np), image build with the pin change plus a
  version check, checkpoint conversion at a pinned model revision, run, monitor,
  step-time measurement rule, silent-failure table
- .../k8s/read_step_time.py: stdlib-only reader of perf/train/global_step_time
- README.md: ./rl entry in the repository organization list

Review follow-ups folded in: Apache license header, JobSet TTL and TPU
toleration, explanatory comments; read_step_time.py defaults a missing proto3
step field to 0 instead of None and reads the event file with a context manager;
the manifest's SIGTERM handler kills the captured $PID rather than $!.
@gutianyu-google
gutianyu-google force-pushed the rl-grpo-qwen3-0.6b-ironwood branch from 209a599 to 2241dc8 Compare November 13, 2026 09:45
@joeywan-google
joeywan-google merged commit 73cdb04 into AI-Hypercomputer:main Nov 13, 2026
6 checks passed
lokic233 added a commit to lokic233/vllm-project.github.io that referenced this pull request Nov 13, 2026
Link AI-Hypercomputer/tpu-recipes#387 (Qwen3-0.6B GRPO on tpu7x
4x4x4: JobSet manifest, pinned image build, step-time reader; no source
patches) at the top of Appendix B and from the Takeaways.

Signed-off-by: Loki Chen <dengcchi@meta.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants