Repository navigation
[Ironwood][MaxText] Add Qwen3-0.6B GRPO RL recipe (Tunix + vLLM/tpu-inference + MaxText under Pathways) on tpu7x 4x4x4 - #387
Conversation
There was a problem hiding this comment.
Code Review
This pull request introduces a new reinforcement learning recipe for Qwen3-0.6B using GRPO on Ironwood TPU GKE clusters, adding a comprehensive README, a Kubernetes JobSet manifest, and a helper Python script to parse step times from TensorBoard event files. The review feedback identifies a potential TypeError in the step-time parsing script when handling default protobuf values and suggests using a safer file-reading pattern. Additionally, it recommends using the captured PID variable instead of the generic shell variable in the manifest's signal handler to ensure robust process termination.
| def read(path): | ||
| data = open(path, 'rb').read() | ||
| i, out = 0, [] | ||
| while i + 12 <= len(data): | ||
| (n,) = struct.unpack_from('<Q', data, i) | ||
| i += 12 # length + length-crc | ||
| if i + n + 4 > len(data): break | ||
| rec = data[i:i+n]; i += n + 4 # payload + payload-crc | ||
| wall = step = None; vals = [] |
There was a problem hiding this comment.
In Python 3, comparing None with an integer (e.g., step >= a.min_step) raises a TypeError. In proto3, default values (such as 0 for the int64 step field) are omitted from the serialized representation. Consequently, for any event at step 0, the step field will not be present in the binary payload, leaving step as None. When the script processes these events, it will crash with a TypeError during the filtering step.\n\nAdditionally, using a with statement to open the file ensures that the file descriptor is properly closed.\n\nWe can fix both issues by defaulting step to 0 (the correct protobuf default value) and using a with statement to read the file.
def read(path):\n with open(path, 'rb') as f:\n data = f.read()\n i, out = 0, []\n while i + 12 <= len(data):\n (n,) = struct.unpack_from('<Q', data, i)\n i += 12 # length + length-crc\n if i + n + 4 > len(data): break\n rec = data[i:i+n]; i += n + 4 # payload + payload-crc\n wall = None; step = 0; vals = []| - -c | ||
| - | | ||
| echo Pathways Helm Start: $(date); | ||
| _sigterm() (kill -SIGTERM $! 2>/dev/null;); |
There was a problem hiding this comment.
Using $! inside the signal handler can be unreliable if other background processes or subshells are spawned later, as $! always points to the most recently spawned background process. Since the PID is explicitly captured in the PID variable, it is safer and more robust to use "$PID" in the signal handler.
_sigterm() { kill -SIGTERM \"$PID\" 2>/dev/null; };c7bcbbb to
6a006a0
Compare
6a006a0 to
209a599
Compare
… on tpu7x 4x4x4 Adds a top-level rl/ directory for reinforcement-learning post-training recipes and its first recipe: GRPO of Qwen3-0.6B on OpenMathInstruct-2 with MaxText + Tunix + vLLM TPU under Pathways, one 64-chip tpu7x 4x4x4 slice, trainer and sampler colocated. Reference step time 56.94 s (median of TensorBoard steps 3..20) measured on 2026-09-21 with an image built from MaxText c28166ca7 and its three post-training pins moved to tunix a3e92fba / tpu-inference c2289ee4b / vLLM 51da0ca6, no source patches. Files: - rl/README.md: index of RL recipes (layout mirrors training/ and inference/) - rl/ironwood/qwen3-0.6b-grpo/math-bf16-tpu7x-4x4x4/k8s/k8s_manifest.yaml: the exact JobSet of the measured run with five envsubst placeholders (run name, image, output bucket, checkpoint path, CPU node pool); the dataset is pinned to the Hugging Face revision the run used - .../k8s/README.md: cluster creation with the Cluster Toolkit gke-tpu-7x blueprint (Pathways CPU pool cpu-np), image build with the pin change plus a version check, checkpoint conversion at a pinned model revision, run, monitor, step-time measurement rule, silent-failure table - .../k8s/read_step_time.py: stdlib-only reader of perf/train/global_step_time - README.md: ./rl entry in the repository organization list Review follow-ups folded in: Apache license header, JobSet TTL and TPU toleration, explanatory comments; read_step_time.py defaults a missing proto3 step field to 0 instead of None and reads the event file with a context manager; the manifest's SIGTERM handler kills the captured $PID rather than $!.
209a599 to
2241dc8
Compare
Link AI-Hypercomputer/tpu-recipes#387 (Qwen3-0.6B GRPO on tpu7x 4x4x4: JobSet manifest, pinned image build, step-time reader; no source patches) at the top of Appendix B and from the Takeaways. Signed-off-by: Loki Chen <dengcchi@meta.com>
Adds a top-level
rl/directory for reinforcement-learning post-training recipes andits first recipe: GRPO of Qwen3-0.6B on OpenMathInstruct-2 with MaxText + Tunix +
vLLM TPU (tpu-inference) under Pathways, on one 64-chip tpu7x
4x4x4slice(16 hosts, 128 TensorCores), trainer and sampler colocated.
Path:
rl/ironwood/qwen3-0.6b-grpo/math-bf16-tpu7x-4x4x4/k8s/k8s_manifest.yamlenvsubstplaceholders (run name, image, output bucket, checkpoint path, CPU node pool)README.mdgke-tpu-7xblueprint, image build with the dependency-pin change and a version check, checkpoint conversion, run, monitor, step-time rule, silent-failure tableread_step_time.pyperf/train/global_step_timeapplying the measurement rulerl/README.mdREADME.md./rlentry in the repository organization listReference result
perf/train/global_step_time, median over TensorBoard steps 3..20 (n = 18)Stack
MaxText
c28166ca7with the three post-training pins insrc/dependencies/extra_deps/post_train_github_deps.txtmoved to tunixa3e92fba,tpu-inference
c2289ee4b, vLLM51da0ca6; Pathways20260918-jax_0.11.0by digest.No source patches. All performance work behind this number is merged upstream in the
three repositories; the pin change is needed because MaxText's own pin file predates
those merges, and an image built without it runs about 2x slower with no error. The
launcher checks the installed commits at start-up and fails loudly if the pins were
not moved. The dataset and the model checkpoint are pinned to the Hugging Face
revisions the run used.
Workload
Validation
Measured on a tpu7x
4x4x4GKE cluster with Pathways, runtianyu-rl-head-03,2026-09-21, with the image built exactly as the README describes (image digest
verified against the four pinned commits and jax 0.11.0 before the run).
IMAGE-CHECKread_step_time.pyon the saved event filesNot covered: checkpoint restore and behaviour beyond 20 steps; the Cluster Toolkit
cluster-creation section was written from the
gke-tpu-7xblueprint and its moduledefaults, the measured cluster was created separately with an equivalent TPU pool.