You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
This repository was archived by the owner on Jun 5, 2026. It is now read-only.
Repository navigation
This repository was archived by the owner on Jun 5, 2026. It is now read-only.
Data seems not sharded across processes in multi-host single-slice setting #267
This seems to imply we are getting the global batch (8 samples) on each host even if the Minibatch is enabled. Any idea why this happened? Also the printed loss is the same on the two hosts in the VM. Many thanks!!!
Hi @weirayao , thanks for checking out our code base. I think what you're seeing is consistent with the distributed execution model of torchprime, although the behavior is not intuitive. Here's what's happening:
torchprime uses SPMD instead of multi-processing based distributed execution.
The torchprime trainer instantiates a stack of dataloaders each wrapping the previous one.
The base DataLoader outputs local tensors stored on the cpu.
The MpDeviceLoader outputs sharded global tensors stored on the TPU. If we print its output, that would give the appearance of duplication.
We can get a better understanding of this process by instrumenting the data loaders. I've uploaded an example at
This shows that the base data loader is outputting local tensors, but the pl.MpDeviceLoader outputs global tensors. In short, MpDeviceLoader combines the local tensors from each host into a larger sharded global tensor without communication overhead. The console table printed by visualize_tensor_sharding further proves that the MpDeviceLoader output is sharded.
When you print(batch), that will pull all the sharded data into the host that is doing the print. So it would appear that each host stores all the data, when in fact the data is sharded over all TPUs, unless a print forces the all-gathering of shards.
A lot of this is hidden by the GSPMD execution model (https://docs.pytorch.org/xla/master/perf/spmd_basic.html). In this model, a tensor's data may be backed by several TPU chips including those from other hosts, but a layer of abstraction presents you the global shape as if the tensor lives locally.
This is the same reason why the loss values printed by different hosts are the same. When we print the loss, all the hosts will reduce-scatter or all-reduce the loss and end up computing the same loss.
Activity
tengyifei commentedon Jun 2, 2025
Hi @weirayao , thanks for checking out our code base. I think what you're seeing is consistent with the distributed execution model of torchprime, although the behavior is not intuitive. Here's what's happening:
DataLoaderoutputs local tensors stored on thecpu.MpDeviceLoaderoutputs sharded global tensors stored on the TPU. If we print its output, that would give the appearance of duplication.We can get a better understanding of this process by instrumenting the data loaders. I've uploaded an example at
torchprime/torchprime/torch_xla_models/train.py
Lines 178 to 219 in 21633ae
With that branch checked out when I run
to train on two 4-chip TPU VMs, I see the following logs:
This shows that the
basedata loader is outputting local tensors, but thepl.MpDeviceLoaderoutputs global tensors. In short,MpDeviceLoadercombines the local tensors from each host into a larger sharded global tensor without communication overhead. The console table printed byvisualize_tensor_shardingfurther proves that theMpDeviceLoaderoutput is sharded.When you
print(batch), that will pull all the sharded data into the host that is doing theprint. So it would appear that each host stores all the data, when in fact the data is sharded over all TPUs, unless aprintforces the all-gathering of shards.A lot of this is hidden by the GSPMD execution model (https://docs.pytorch.org/xla/master/perf/spmd_basic.html). In this model, a tensor's data may be backed by several TPU chips including those from other hosts, but a layer of abstraction presents you the global shape as if the tensor lives locally.
This is the same reason why the loss values printed by different hosts are the same. When we print the loss, all the hosts will
reduce-scatterorall-reducethe loss and end up computing the same loss.LMK if this answer makes sense.