Skip to content

[Cherry-Pick][Distributed] Key Muon sharding 2D buckets by (color, group ranks) (#79913) - #79914

Closed
DanielSun11 wants to merge 1 commit into
PaddlePaddle:release/3.4from
DanielSun11:cherry-pick/3.4-sharding-2d-bucket-key
Closed

DanielSun11 wants to merge 1 commit into
PaddlePaddle:release/3.4from
DanielSun11:cherry-pick/3.4-sharding-2d-bucket-key

Conversation

@DanielSun11

Copy link
Copy Markdown
Contributor

PR Category

Distributed Strategy

PR Types

Bug fixes

Description

Cherry-pick of #79913 to release/3.4.

MuonShardingOptimizer keeps keep-whole (2D) params, e.g. Muon or Hyperball params, whole on one owner rank, bucketed by param.color. The buckets and the color -> group table were keyed by color name only, and the table kept the first group it saw for each name. When one color name is used by several comm groups on the same rank (e.g. moe_expert for an encoder and a decoder with different expert-parallel sizes), all params of that color were partitioned, reduced and broadcast in the first group. Training does not fail; the params of the other groups are silently corrupted.

The fix keys a 2D bucket by (color name or None, group ranks), the key the machine-balanced plan already used. clear_param_storage / reset_param_storage still take a color name and handle every bucket with that name. When each color name maps to a single group, the partition, comm groups and owners are the same as before.

Differences from #79913: MixedShardingOptimizer and its tests do not exist on release/3.4, so only the Muon change is picked. The change to muon_sharding_optimizer.py is identical to #79913. The new single-process test test/legacy_test/test_sharding_2d_bucket_key.py covers only Muon here.

是否引起精度变化

否

…oup ranks) (PaddlePaddle#79913)

The keep-whole (2D) buckets of MuonShardingOptimizer were keyed by color
name only, and the color -> group table kept the first group seen for each
name. When one color name is used by several comm groups on the same rank
(e.g. 'moe_expert' for two sub-models with different expert-parallel sizes),
all those params were partitioned, reduced and broadcast in the first group,
which silently corrupts the params of the other groups.

Key the buckets by (color name or None, group ranks) instead, the same key
the machine-balanced plan already used. clear/reset_param_storage still take
a color name and now handle every bucket of that name. Behavior is unchanged
when each color name maps to a single group.

MixedShardingOptimizer does not exist on this branch, so only the Muon part
of PaddlePaddle#79913 is picked.
@Paddle-Review-Bot

Copy link
Copy Markdown
Contributor

Paddle-Bot Review Board (正在review中...)

序号 位置 优先级 规则来源 状态

Powered by Nyanpasu claude with Opus 5.5 默认推理级别, please check the suggestions carefully.

'rank': group.rank if group else 0,
}
name, group = cls._param_color_and_group(p, default_group)
key = (name, cls._group_ranks(group))

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 可维护性建议(底层优雅度,不阻塞合入):这里用 (name, cls._group_ranks(group)) 手工重拼了一遍 bucket key,与上方新增的 _param_bucket_key 是两处独立构造;后续若给 key 增加维度,容易只改一处导致 _color_to_group_info 与 _params_2d_by_color 的键不对齐。建议改为 key = cls._param_bucket_key(p, default_group),仅在 key not in color_to_info 时再取 group(也可在 develop 的 #79913 一并调整,保持两分支一致)。

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants