Repository navigation
Conversation
Codex Review Summary: ✅ CompletedThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: a6feceda4c
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| hold a multi-axis box as a dense run, which is how ZeRO stores a tensor-parallel shard, or at the | ||
| parameter's own row-major strides, which is how a ZeRO partition holds a box cut out of the rows it | ||
| stores. ``_shards_tile`` proves the second kind by its shift, not by its address interval. |
There was a problem hiding this comment.
Document the newly accepted flat-shard layout
When callers build these parameter-stride boxes, the checked-in contract still tells them the opposite: deepspeed/checkpoint/affine_ir_spec.md:671-688 says every box in a 1-D shard is dense, is treated as [dest_offset, dest_offset + numel), and that a narrower block must remain one piece per row. Following that guidance recreates the max_pieces_per_map failure this change is intended to remove, so update the storage-map contract and derivation guidance alongside this new behavior.
AGENTS.md reference: AGENTS.md:L71-L71
Useful? React with 👍 / 👎.
| monkeypatch.setattr(transfer_module, '_validate_holders', transfer_module._validate_holders_pairwise) | ||
| assert fast == _outcome(target, source) |
There was a problem hiding this comment.
Test holder validation only through public behavior
This test replaces the production _validate_holders binding with another private helper and treats the exact plan or error text from that implementation as its oracle. A correct refactor that removes or renames either helper, or changes diagnostics without changing plan_transfer behavior, will fail this test, while a defect shared by both paths can still pass; exercise the public planner against independently derived valid/invalid maps or observable transfer results instead.
AGENTS.md reference: AGENTS.md:L76-L80
Useful? React with 👍 / 👎.
|
@0z5a, I checked #9052 on top of this. Your commit and #9052's two commits apply onto master (892ed83) in that order without conflicts, and all eight affine test files pass on CPU (gloo, 4 ranks): 725 passed, against 715 with #9052 alone. On the random two-layout sets from #9052 (ZeRO-3 storage with 1–6 partitions plus surviving TP shards of uneven sizes, 300 cases cut along rows and 300 along columns), every union now plans exactly into both a row-split and a column-split target. That includes the 82 column-cut cases per target that #9052 alone refuses with @LiRunGuo, the combined branch is Achyuthan-S/DeepSpeed:union-on-9070, for your rerun of the weights from #9074 that are planned from both sources. |
A ZeRO-3 partition cut at a tensor-parallel column edge holds boxes narrower than the parameter, stored at the parameter's own row-major strides. Their address intervals interleave, so plan_transfer refused them, and the only workaround was one piece per row, which needs a larger budget. Accept them when a flat shard keeps one shift for all its pieces: the address of logical row-major index i is then i + shift, which is injective, so disjoint logical boxes, the shard bounds and the volume sum prove the shard is tiled. Other mixes stay refused. Signed-off-by: 0z5a <dezhen.lu@student.uni-tuebingen.de>
The spec said every box in a 1-D shard is dense and is planned as the interval [dest_offset, dest_offset + numel), and that a block narrower than a shard's rows keeps one piece per row. plan_transfer now also accepts boxes at the parameter's own row-major strides with one shift per shard, so describe that form, its tiling argument and the shards it fits, and note that nest_affine_maps still emits dense boxes only. Signed-off-by: 0z5a <dezhen.lu@student.uni-tuebingen.de>
a6feced to
8cc4b52
Compare
|
#9011 is in, so I rebased this onto master as promised. The column-case commit is unchanged, so @Achyuthan-S's The first Codex note is fair: the spec still described every box in a flat shard as dense. A second commit adds the parameter-order form and its tiling argument to §10.2 and §10.3 of On CPU (gloo, up to 4 processes), |
Follow-up to the column case in #9026. #9011, which this was stacked on, has merged; the branch is now on master.
A ZeRO-3 partition cut at a tensor-parallel column edge holds boxes narrower than the parameter, and it stores them at the parameter's own row-major strides (
(cols, 1)), not as dense runs.plan_transferrefuses those today (dest axis 0 strides …). The only workaround is one piece per row, which runs intomax_pieces_per_mapfor real weights.Change. In
_frame_of, a flat shard may now hold a piece at the parameter's own strides, alongside the dense run it already accepts. Each such piece records its shift: its shard offset minus the row-major index of its origin. The tiling check for these pieces is a separate branch,_in_order_shard_tiles:isits at addressi + shift, which is injective.max_pair_checks, the same as in [AffineIR] Check affine holders and source regions without pairwise comparisons #9011.Shards whose pieces are all dense runs take the existing path unchanged.
Docs. A second commit updates
affine_ir_spec.md§10.2–10.3 to match: the two forms a flat shard may take, the tiling argument for the second, and thatnest_affine_mapsstill emits dense boxes only, since the parameter's own strides fit only a shard that is a run of the parameter's flattening.Tests (
tests/unit/checkpoint/test_affine_transfer_plan.py, all single-process):Without the planner change, the 5 acceptance cases are refused. The 5 refusal cases refuse either way; the tests pin the new reasons. On CPU (gloo, up to 4 processes),
test_affine_*.py,test_autoep_affine*.py,test_zero_affine.pyandv1/moe/test_autoep_checkpoint.pypass: 765 passed and 4 skipped, against 755 and 4 on master.Outside the unit tests (CPU, against this branch). These use the two-layout map from #9026: ZeRO-3 partitions plus the surviving TP shards, cut at each other's edges.
execute_transferson 4 gloo processes, in both directions, at staging budgets of 8 MiB and 256 B. Nothing is wrong: 0 of 200 target shards reading from the cut map, and 0 of 483 writing into it.