Repository navigation
[FSDP] Sum cp-partial grads over cp in the sharding reduce - #79948
Conversation
The sharding group spans cp, so averaging over it divided cp-partial grads by cp too. Divide by sharding_degree / cp instead; grads replicated across cp (context_parallel_disable_scale_grad, or expert_cp_replicated for experts) keep the full sharding_degree and get their own buffer.
Paddle-Bot Review Board (review完成)
可维护性建议(总分 89.2 / 100,只给建议,不影响评审结论)
各维度得分(仅评本 PR 改动,点击展开)
Powered by Nyanpasu claude with Opus 5.5 默认推理级别, please check the suggestions carefully. |
| grad_div=self._fsdp_group.nranks | ||
| // (1 if group.cp_replicated else cp_degree), |
There was a problem hiding this comment.
这一行改变了 cp>1 时所有 grads buffer 的梯度除数,属于精度关键路径,但 PR 没有补测试;
test/legacy_test/test_fsdp_fully_shard_fusion.py 和 test/collective/fleet/dygraph_group_fsdp.py 都没有覆盖 cp_degree > 1。建议补一个单进程用例:用 fake hcg(get_context_parallel_world_size() 返回 2、sharding 组 nranks=4)构造 FSDPBufferManager,断言未标记参数的 grads_buffer.grad_div == 2、带 context_parallel_disable_scale_grad / expert_cp_replicated 的参数落在独立 buffer 且 grad_div == 4,防止后续改动把系数改回去。
There was a problem hiding this comment.
已在 90896eb 补充 TestCpGradDiv:cp=2 时校验部分梯度 grad_div == 2、两类复制梯度 grad_div == 4 且落在独立 buffer,cp=1 时校验分组不变、grad_div == 4。
At cp = 1 both kinds of grads use the same grad_div, so splitting buffers only changes buffer count and fuse_params_<gid> numbering. Keep the grouping identical to before when cp is disabled.
Check that cp-partial grads divide by sharding / cp, cp-replicated grads divide by sharding in their own buffers, and cp = 1 keeps the grouping.
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## develop #79948 +/- ##
===========================================
Coverage ? 100.00%
===========================================
Files ? 1
Lines ? 2
Branches ? 0
===========================================
Hits ? 2
Misses ? 0
Partials ? 0 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
12c3908
into
PaddlePaddle:develop
PR Category
Distributed Strategy
PR Types
Bug fixes
Description
开启 context parallel(cp > 1)时,manual-parallel FSDP 算出的梯度偏小 cp 倍。
sharding 组包含 cp 维度(sharding_degree = cp_sharding × cp),grads buffer 的 reduce-scatter 按 sharding_degree 求平均。但大多数参数在每个 cp rank 上只算了 1/cp 序列的部分梯度,正确做法是在 cp 上求和、只在 cp_sharding 上求平均,所以除数应为 sharding_degree / cp。
本 PR 只修改
FSDPBufferManager的 grads buffer 除数:grad_div = sharding_degree // cpgrad_div = sharding_degree。非 expert 参数用context_parallel_disable_scale_grad标记,例如 embedding;expert 参数用expert_cp_replicated标记,例如 HyperBody encoder 的 expert。这两类参数通过分组 key 放进独立的 buffer,不与部分梯度混在同一个 buffer 里。验证
ERNIE-5.5-Flash,单机 8 卡,cp=8,ep=8,200 步,与 sharding(非 FSDP)对比:
是否引起精度变化
否。修复了 cp > 1 时 FSDP 梯度偏小 cp 倍的问题,修复后与 sharding 对齐;cp = 1 时无变化。