Repository navigation
[Cherry-Pick][Distributed] Key Muon sharding 2D buckets by (color, group ranks) (#79913) - #79914
Closed
DanielSun11 wants to merge 1 commit into
Closed
DanielSun11 wants to merge 1 commit into
DanielSun11 wants to merge 1 commit into
Conversation
…oup ranks) (PaddlePaddle#79913) The keep-whole (2D) buckets of MuonShardingOptimizer were keyed by color name only, and the color -> group table kept the first group seen for each name. When one color name is used by several comm groups on the same rank (e.g. 'moe_expert' for two sub-models with different expert-parallel sizes), all those params were partitioned, reduced and broadcast in the first group, which silently corrupts the params of the other groups. Key the buckets by (color name or None, group ranks) instead, the same key the machine-balanced plan already used. clear/reset_param_storage still take a color name and now handle every bucket of that name. Behavior is unchanged when each color name maps to a single group. MixedShardingOptimizer does not exist on this branch, so only the Muon part of PaddlePaddle#79913 is picked.
Contributor
Paddle-Bot Review Board (正在review中...)
Powered by Nyanpasu claude with Opus 5.5 默认推理级别, please check the suggestions carefully. |
Paddle-Review-Bot
approved these changes
Oct 30, 2026
| 'rank': group.rank if group else 0, | ||
| } | ||
| name, group = cls._param_color_and_group(p, default_group) | ||
| key = (name, cls._group_ranks(group)) |
Contributor
There was a problem hiding this comment.
💡 可维护性建议(底层优雅度,不阻塞合入):这里用 (name, cls._group_ranks(group)) 手工重拼了一遍 bucket key,与上方新增的 _param_bucket_key 是两处独立构造;后续若给 key 增加维度,容易只改一处导致 _color_to_group_info 与 _params_2d_by_color 的键不对齐。建议改为 key = cls._param_bucket_key(p, default_group),仅在 key not in color_to_info 时再取 group(也可在 develop 的 #79913 一并调整,保持两分支一致)。
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
PR Category
Distributed Strategy
PR Types
Bug fixes
Description
Cherry-pick of #79913 to release/3.4.
MuonShardingOptimizerkeeps keep-whole (2D) params, e.g. Muon or Hyperball params, whole on one owner rank, bucketed byparam.color. The buckets and the color -> group table were keyed by color name only, and the table kept the first group it saw for each name. When one color name is used by several comm groups on the same rank (e.g.moe_expertfor an encoder and a decoder with different expert-parallel sizes), all params of that color were partitioned, reduced and broadcast in the first group. Training does not fail; the params of the other groups are silently corrupted.The fix keys a 2D bucket by
(color name or None, group ranks), the key the machine-balanced plan already used.clear_param_storage/reset_param_storagestill take a color name and handle every bucket with that name. When each color name maps to a single group, the partition, comm groups and owners are the same as before.Differences from #79913:
MixedShardingOptimizerand its tests do not exist on release/3.4, so only the Muon change is picked. The change tomuon_sharding_optimizer.pyis identical to #79913. The new single-process testtest/legacy_test/test_sharding_2d_bucket_key.pycovers only Muon here.是否引起精度变化
否