Skip to content

[FSDP] Route dict outputs through a single backward hook - #79959

Merged
swgu98 merged 1 commit into
PaddlePaddle:developfrom
liym27:liym27/fsdp-dict-outputs-single-backward-hook
Nov 20, 2026
Merged

swgu98 merged 1 commit into
PaddlePaddle:developfrom
liym27:liym27/fsdp-dict-outputs-single-backward-hook

Conversation

@liym27

@liym27 liym27 commented Nov 18, 2026

Copy link
Copy Markdown
Contributor

PR Category

Distributed Strategy

PR Types

Bug fixes

Description

Problem

FullyShardFusion's forward post hook wrapped every hookable value of a dict output in its own FusionBackwardHook. The first value whose gradient arrives (e.g. an entry that a later layer passes through untouched) then all-gathers the unit before its recompute backward runs, and _release_params_after_unit frees later units whose recompute backward has not run yet. Backward then fails with Tensor holds no memory.

PaddleFleet's TransformerLayer takes and returns dicts, so this reproduces deterministically with reshard_policy="after_forward" + full recompute (ERNIE-5.5-Omni-Flash, HyperBody + PaddleFleet). after_step avoids it at the cost of memory.

Fix

Route all hookable dict values through ONE FusionBackwardHook.apply(*values), as the tuple branch already does. The PyLayer's backward runs once, after every output gradient has arrived, so the unit is all-gathered once at the right time. Non-tensor, stop_gradient and exposed-parameter values are left untouched; key order is preserved. The tuple / single-tensor branches are unchanged.

Tests

Added TestForwardPostHookDictBranch to test/legacy_test/test_fsdp_fully_shard_fusion.py:

  • two requires-grad dict values -> exactly 1 backward all_gather (the old code issues 2; the test fails on current develop with 2 != 1), gradients unchanged;
  • non-tensor / stop_gradient / exposed values are kept as the same objects;
  • nothing hookable -> same dict returned, no hook applied;
  • single hookable value (bare-tensor return of the PyLayer) is written back correctly.

All 58 tests in the file pass.

E2E

ERNIE-5.5-Omni-Flash with FSDP (after_forward, full recompute): before the fix, single-node smoke runs crashed in backward. After the fix, single-node CP1/CP2 losses match the sharding stage1 baseline (first 2 steps bit-identical), and the full 43-layer model runs 50 steps on 64 x B30Z (CP4, EP64).

是否引起精度变化

否

The forward post hook wrapped every hookable value of a dict output in its
own FusionBackwardHook. The first value whose gradient arrives (e.g. an entry
a later layer passes through untouched) then all-gathers the unit before its
recompute backward runs, and _release_params_after_unit frees later units
whose recompute backward has not run yet ("Tensor holds no memory").

Route all hookable dict values through one FusionBackwardHook, as the tuple
branch already does, so the backward all_gather fires once, after every
output gradient has arrived. Non-tensor, stop_gradient and exposed values are
left untouched.
@Paddle-Review-Bot

Paddle-Review-Bot commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

Paddle-Bot Review Board (review完成)

序号 位置 优先级 规则来源 状态
1 dict 分支 exposed 参数测试 P3 仓库规则:测试质量与验证 🟡

可维护性建议(总分 91.2 / 100,只给建议,不影响评审结论)

序号 位置 维度 状态
1 dict 分支返回值归一化 逻辑清晰度 🟡
2 dict 分支测试 hook 捕获辅助函数 AI 生成债务 🟡
各维度得分(仅评本 PR 改动,点击展开)
87 AI 生成债务
2
87 中层优雅度
87 设计一致性
88 抽象适配度
89 契约一致性
90 命名质量
90 逻辑清晰度
1
90 类型安全
90 错误处理一致性
90 高层优雅度
90 底层优雅度
92 测试策略
93 API 表面一致性
94 约定偏离
95 跨模块架构
95 初始化耦合
95 包组织结构
100 依赖健康度
— 鉴权一致性
— 未完成迁移

Powered by Nyanpasu claude with Opus 5.5 默认推理级别, please check the suggestions carefully.

def test_unhookable_values_kept_untouched(self):
hidden = paddle.to_tensor([1.0, 2.0], stop_gradient=False) * 1.0
frozen = paddle.to_tensor([5.0]) # stop_gradient=True
weight = paddle.to_tensor([3.0, 4.0]) # exposed LM-head weight

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P3 这里的 weight 由 paddle.to_tensor 创建,默认 stop_gradient=True,dict 分支第一个条件 not value.stop_gradient(fully_shard_fusion.py:2509)就已把它排除。所以即使删掉 value.name not in exposed 这个按名字过滤 exposed 参数的条件,assertIs(result["w"], weight) 依然成立,用例没有覆盖 PR 描述中“exposed 参数保持不变”的行为。实际的 exposed 参数(如 LM head weight)是 stop_gradient=False 的叶子变量,送进 PyLayer 会报错。

建议改为 weight = paddle.to_tensor([3.0, 4.0], stop_gradient=False),让按名字过滤成为该断言成立的唯一原因。

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

这个不影响正确性,暂时可以先推进合入

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

收到。这是 P3 的测试覆盖问题,确实不影响修复本身的正确性,接受先合入,建议后续顺手补上 stop_gradient=False 的用例。

comm_manager=self.comm_manager,
recursive=recursive,
)
if not isinstance(hooked, (tuple, list)):

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 可维护性建议(逻辑清晰度,不阻塞合入):FusionBackwardHook.forward(1484 行)只会返回 tuple 或单个 Tensor,list 分支走不到,而且与紧邻 tuple 分支 2546 行的 isinstance(hooked, tuple) 写法不一致,读者容易以为两处的返回约定不同。建议统一为 if not isinstance(hooked, tuple):。

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

这个不影响正确性,暂时可以先推进合入

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

收到,可维护性建议不阻塞合入,已标记为接受。

"""Covers the dict branch of the forward post hook built by
``_register_fusion_layer_hooks``."""

def _capture_post_hook(self, param_names):

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 可维护性建议(AI 生成债务,不阻塞合入):这个 _capture_post_hook 与 TestForwardPostHookTupleBranch._capture_post_hook(192 行)逐行相同,只有 comm_manager 类型和返回值不同。_register_fusion_layer_hooks 的签名变化时需要同步改两处。建议抽成一个模块级 helper,例如 _make_hooked_layer(param_names, comm_manager) 返回 (fsdp, layer),两个测试类共用。

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

这个不影响正确性,暂时可以先推进合入

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

收到,可维护性建议不阻塞合入,已标记为接受。

@Xing-lil Xing-lil left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@codecov-commenter

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
⚠️ Please upload report for BASE (develop@7076654). Learn more about missing BASE report.

Additional details and impacted files
@@             Coverage Diff             @@
##             develop    #79959   +/-   ##
===========================================
  Coverage           ?   100.00%           
===========================================
  Files              ?         1           
  Lines              ?         8           
  Branches           ?         0           
===========================================
  Hits               ?         8           
  Misses             ?         0           
  Partials           ?         0           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@swgu98
swgu98 merged commit 7aecda1 into PaddlePaddle:develop Nov 20, 2026
121 of 129 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants