Repository navigation
Conversation
The forward post hook wrapped every hookable value of a dict output in its
own FusionBackwardHook. The first value whose gradient arrives (e.g. an entry
a later layer passes through untouched) then all-gathers the unit before its
recompute backward runs, and _release_params_after_unit frees later units
whose recompute backward has not run yet ("Tensor holds no memory").
Route all hookable dict values through one FusionBackwardHook, as the tuple
branch already does, so the backward all_gather fires once, after every
output gradient has arrived. Non-tensor, stop_gradient and exposed values are
left untouched.
Paddle-Bot Review Board (review完成)
可维护性建议(总分 91.2 / 100,只给建议,不影响评审结论)
各维度得分(仅评本 PR 改动,点击展开)
Powered by Nyanpasu claude with Opus 5.5 默认推理级别, please check the suggestions carefully. |
| def test_unhookable_values_kept_untouched(self): | ||
| hidden = paddle.to_tensor([1.0, 2.0], stop_gradient=False) * 1.0 | ||
| frozen = paddle.to_tensor([5.0]) # stop_gradient=True | ||
| weight = paddle.to_tensor([3.0, 4.0]) # exposed LM-head weight |
There was a problem hiding this comment.
这里的
weight 由 paddle.to_tensor 创建,默认 stop_gradient=True,dict 分支第一个条件 not value.stop_gradient(fully_shard_fusion.py:2509)就已把它排除。所以即使删掉 value.name not in exposed 这个按名字过滤 exposed 参数的条件,assertIs(result["w"], weight) 依然成立,用例没有覆盖 PR 描述中“exposed 参数保持不变”的行为。实际的 exposed 参数(如 LM head weight)是 stop_gradient=False 的叶子变量,送进 PyLayer 会报错。
建议改为 weight = paddle.to_tensor([3.0, 4.0], stop_gradient=False),让按名字过滤成为该断言成立的唯一原因。
There was a problem hiding this comment.
收到。这是 P3 的测试覆盖问题,确实不影响修复本身的正确性,接受先合入,建议后续顺手补上 stop_gradient=False 的用例。
| comm_manager=self.comm_manager, | ||
| recursive=recursive, | ||
| ) | ||
| if not isinstance(hooked, (tuple, list)): |
There was a problem hiding this comment.
💡 可维护性建议(逻辑清晰度,不阻塞合入):FusionBackwardHook.forward(1484 行)只会返回 tuple 或单个 Tensor,list 分支走不到,而且与紧邻 tuple 分支 2546 行的 isinstance(hooked, tuple) 写法不一致,读者容易以为两处的返回约定不同。建议统一为 if not isinstance(hooked, tuple):。
There was a problem hiding this comment.
收到,可维护性建议不阻塞合入,已标记为接受。
| """Covers the dict branch of the forward post hook built by | ||
| ``_register_fusion_layer_hooks``.""" | ||
|
|
||
| def _capture_post_hook(self, param_names): |
There was a problem hiding this comment.
💡 可维护性建议(AI 生成债务,不阻塞合入):这个 _capture_post_hook 与 TestForwardPostHookTupleBranch._capture_post_hook(192 行)逐行相同,只有 comm_manager 类型和返回值不同。_register_fusion_layer_hooks 的签名变化时需要同步改两处。建议抽成一个模块级 helper,例如 _make_hooked_layer(param_names, comm_manager) 返回 (fsdp, layer),两个测试类共用。
There was a problem hiding this comment.
收到,可维护性建议不阻塞合入,已标记为接受。
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## develop #79959 +/- ##
===========================================
Coverage ? 100.00%
===========================================
Files ? 1
Lines ? 8
Branches ? 0
===========================================
Hits ? 8
Misses ? 0
Partials ? 0 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
PR Category
Distributed Strategy
PR Types
Bug fixes
Description
Problem
FullyShardFusion's forward post hook wrapped every hookable value of a dict output in its ownFusionBackwardHook. The first value whose gradient arrives (e.g. an entry that a later layer passes through untouched) then all-gathers the unit before its recompute backward runs, and_release_params_after_unitfrees later units whose recompute backward has not run yet. Backward then fails withTensor holds no memory.PaddleFleet's
TransformerLayertakes and returns dicts, so this reproduces deterministically withreshard_policy="after_forward"+ full recompute (ERNIE-5.5-Omni-Flash, HyperBody + PaddleFleet).after_stepavoids it at the cost of memory.Fix
Route all hookable dict values through ONE
FusionBackwardHook.apply(*values), as the tuple branch already does. The PyLayer's backward runs once, after every output gradient has arrived, so the unit is all-gathered once at the right time. Non-tensor,stop_gradientand exposed-parameter values are left untouched; key order is preserved. The tuple / single-tensor branches are unchanged.Tests
Added
TestForwardPostHookDictBranchtotest/legacy_test/test_fsdp_fully_shard_fusion.py:2 != 1), gradients unchanged;All 58 tests in the file pass.
E2E
ERNIE-5.5-Omni-Flash with FSDP (
after_forward, full recompute): before the fix, single-node smoke runs crashed in backward. After the fix, single-node CP1/CP2 losses match the sharding stage1 baseline (first 2 steps bit-identical), and the full 43-layer model runs 50 steps on 64 x B30Z (CP4, EP64).是否引起精度变化
否