Repository navigation
feat: add opt-in single_host_tpu_init for node-local diagnostics on multi-host slices - #144
simrankaurb wants to merge 11 commits into
Conversation
There was a problem hiding this comment.
Code Review
This pull request introduces a helper method _replicated_normal to generate replicated arrays of normal samples locally on each device, preventing device-to-device transfers that can hang when ICI routing is not configured. The generate_inputs method is updated to utilize this helper for input and scaling factor generation. The review feedback suggests several improvements: simplifying the creation of replicated_sharding with an empty PartitionSpec(), using the idiomatic self.mesh.local_devices property instead of manual filtering, and catching both RuntimeError and IndexError when accessing the CPU device to prevent potential crashes.
d6baf11 to
b78b61c
Compare
…-host slices
Adds a benchmark-level `single_host_tpu_init: true` config key and a
`--single_host_tpu_init` common execution flag (default off, accepted by
both `benchmark run` and `benchmark run-config`) that initialize the
current host standalone inside a multi-host slice, so single-host
benchmarks such as `gemm` and `hbm` can be used for node-local diagnostics
(e.g. Cluster Health Scanner) without the other hosts of the slice
participating.
When enabled, `core/single_host.py` (applied after op_flags and before the
first JAX device call) forces a 1-host topology (`TPU_HOST_BOUNDS`,
`TPU_WORKER_ID`, `TPU_WORKER_HOSTNAMES`, `TPU_SKIP_MDS_QUERY`), skips
SliceBuilder/tpunetd (`--deepsea_hal_test_skip_slicebuilder=true`,
`--deepsea_hal_test_allow_multichip_skip_slicebuilder=true`) and disables
the cross-host launch barrier. LIBTPU_INIT_ARGS is merged by flag name so
pre-existing values are overridden rather than duplicated, and every
overridden env var / flag is logged with its previous value.
Because the slice's ICI routing is left untouched, device-to-device
transfers hang in this mode. `GeneralizedGemmBenchmark.generate_inputs`
previously sampled the operands on the default device and broadcast them
with `jax.device_put(x, replicated_sharding)`, which hung for any matrix
size. It now generates each operand under `jax.jit(...,
out_shardings=replicated_sharding)` (the pattern already used by `hbm.py`),
so every device produces the same replicated values and no broadcast is
needed. This is a single code path for both modes: with the flag off the
only behavioural change is that GEMM operand generation no longer
broadcasts over ICI (values are bit-identical). Only `gemm`,
`gemm_throttling` (same input path) and `hbm` are supported in this mode;
the runner prints a warning when the flag is used with any other benchmark.
Verification:
- TPU, earlier revision of this change (replicated operands built via
`jax.make_array_from_single_device_arrays` instead of
`jax.jit(out_shardings=...)`), on a tpu7x-standard-4t host of a 2x2x4
slice (jax 0.12.0, libtpu 0.0.50), no manual environment set up:
`single_host_tpu_init: true`, gemm 16384^3 bf16 x20, xprof
local_slowest_device: 2127 TFLOP/s per chip, 20/20 xprof runs, 78 s;
`--single_host_tpu_init` gemm 1024^3: 20 s; hbm copy 1G elems:
2790 GB/s; control without the flag on the same node fails at init
("Expected 4 worker addresses, got 1").
- CPU: the current `jax.jit(out_shardings=...)` path yields arrays
bit-identical to the previous default path with fully replicated,
identical shards on a 4-device CPU mesh; it has not yet been re-run on
TPU. New unit tests cover config/CLI plumbing, LIBTPU_INIT_ARGS merging,
runner stage ordering and the unsupported-benchmark warning.
b78b61c to
a80aaba
Compare
|
अगर तुम ऐसा AI बनाना चाहते हो जिसकी दुनिया में करोड़ों लोगों को जरूरत हो, तो मेरी सलाह है कि केवल एक और ChatGPT जैसा AI मत बनाओ। ऐसा AI बनाओ जो लोगों की कोई बड़ी, रोज़ की समस्या हल करे और उनका समय, पैसा या मेहनत बचाए।
|
Both benchmarks only move data between host buffers and each local device (device_id -1 selects every local device individually), so they perform no device-to-device transfers and work with the standalone single-host init. Add them to SUPPORTED_BENCHMARKS so the runner no longer warns for them.
With single_host_tpu_init the process is the only participant, but jax.distributed.initialize() still derives the process list from the slice environment. On GKE, TPU_PROCESS_ADDRESSES (preferred over TPU_WORKER_HOSTNAMES) lists every host of the job, so each host tried to register with the others' coordinator and aborted: F client.h:86] Terminating process because the JAX distributed service detected fatal errors. ... CoordinationService/RegisterTask Only initialize the backend in that mode; the distributed coordinator is unchanged for normal runs.
_replicated_normal generates the operands under jit with a replicated out_shardings, but the key it was given lives on the default device, so JAX first broadcast it to the other mesh devices with a cross-device copy (libtpu: "Slow PjRt TPU operation ... CopyToMemorySpace CrossDeviceSrc"). Under single_host_tpu_init that copy never completes and gemm hangs on its first run. Convert the key to a host array first; the transfer is then host->device on every shard. Output is unchanged.
Restore the default GEMM input path to the pre-existing behaviour (sample on the default device, jax.device_put onto the replicated sharding) and only use the per-device jit generation when single_host_tpu_init is on, selected via single_host.is_single_host_tpu_init(). Normal single- and multi-host runs are unchanged; the two paths produce identical arrays. Tests: default path broadcasts with device_put; single-host path uses no device_put; is_single_host_tpu_init() follows configure_single_host_tpu().
…is off generate_inputs now only gains an early return to the single-host path; its body is unchanged from main (no lines removed or modified), so runs without the flag execute exactly the pre-existing code.
Docstrings, the CLI help, the runner warning and the README now state the mode's contract (host<->device data movement only, no inter-device communication) and the supported benchmarks, instead of failure modes.
68cdd95 to
08bf044
Compare
…_normal Matches the fully replicated NamedSharding idiom used elsewhere in the file. No change to the upstream generate_inputs path.
Launching the replicated program back-to-back on the same buffers slows one local device on v6e (10.4 ms -> 12-17 ms, high variance). Regenerating and synchronizing the inputs between runs keeps every device at its standalone kernel time; gemm_throttling keeps reusing inputs during soaking.
bc762dd to
ef51c08
Compare
On v6e the timeline ts/dur of TPU events are corrupted (multiple steps share identical ts with dur=0 or huge spans), so the enclosure bounds and step ends were wrong on healthy devices. Prefer the hardware-reported device_offset_ps and device_duration_ps, falling back to ts/dur.
ef51c08 to
cef341a
Compare
Summary
Adds an opt‑in
single_host_tpu_init(YAML keysingle_host_tpu_init: true/ CLI--single_host_tpu_init, default off) that initializes the current host as a standalone 1‑host topology inside a multi‑host slice, so single‑host benchmarks (gemm,hbm,host_to_device,device_to_host) can be run for node‑local diagnostics — e.g. by Cluster Health Scanner — without the other hosts of the slice participating. With the flag off every code path is the pre‑existing one. This brings thechsbranch's single‑host init (#142) tomain.What the flag does
core/single_host.py::configure_single_host_tpu(), called by the runner afterop_flags.yamlis applied and before the first JAX device call:TPU_HOST_BOUNDS=1,1,1,TPU_WORKER_ID=0,TPU_WORKER_HOSTNAMES=localhost,TPU_SKIP_MDS_QUERY=true(each change is logged)LIBTPU_INIT_ARGS, merged by flag name:--deepsea_hal_test_skip_slicebuilder=true --deepsea_hal_test_allow_multichip_skip_slicebuilder=true --xla_tpu_use_enhanced_launch_barrier=false— the same recipe the TPU preflight health check uses for a 1x1 sliceSUPPORTED_BENCHMARKSlists the benchmarks whose data movement stays on the host↔device path (gemm,gemm_throttling,hbm,host_to_device,device_to_hostand aliases) and a warning is printed for othersGEMM inputs under the flag
GeneralizedGemmBenchmark.generate_inputsreturns early into_generate_inputs_single_host()whensingle_host.is_single_host_tpu_init()is true. There the operands (and optional scaling factors / accumulator) are produced underjitwith a replicatedout_shardings— the patternhbm.pyuses — from a host‑array key, so each shard is generated on its own device and the only transfers are host→device. The values equal eager sampling with the same key, and the sharding is the same replicatedNamedShardingas before. The upstream body ofgenerate_inputsis unchanged (matmul.pyis +60/−0).Plumbing mirrors
xprof_timing:BenchmarkKey.SINGLE_HOST_TPU_INIT+ bool validation inconfig.py,--single_host_tpu_initonbenchmark run/run-config,runner.run_benchmarks(single_host_tpu_init=…). The only pre‑existing lines modified in the PR are the ones givinginit_jax_distributedadistributed=Trueparameter; its default behaviour is unchanged. README and CLI help updated; unit tests cover config, CLI, runner ordering,single_host, and both GEMM paths.Verification
tpu7x‑standard‑4t hosts of 4x4x4 and 2x2x4 slices on GKE (jax 0.12.0 / libtpu 0.0.50), both as bare pods and inside 2‑pod Cluster Health Scanner jobs:
single_host_tpu_init: true, gemm 16384³ bf16 ×100,xprof_device_timing_aggregation: local_slowest_devicehbmcopy sweep 1 M → 4 G elements, device 0host_to_device+device_to_host,tpu_sync, 4096 MiB, all 8 local devicesExpected 4 worker addresses, got 1), as expectedCPU (4‑device host mesh): with the flag off
generate_inputsperforms the pre‑existingdevice_putbroadcasts; with the flag on it performs none; arrays and GEMM outputs are identical between the two.Note: upstream
tests/importaccelerator_microbenchmarks.tests, which is not resolvable in the OSS checkout, sotest_matmul.pycould not be run here; its new cases were exercised standalone with the same assertions.