Skip to content

[Fsdp] support fp8 and fused linear - #79813

Merged
zhengshengning merged 10 commits into
PaddlePaddle:developfrom
zhengshengning:fsdp_fp8_fused_linear
Oct 15, 2026
Merged

zhengshengning merged 10 commits into
PaddlePaddle:developfrom
zhengshengning:fsdp_fp8_fused_linear

Conversation

@zhengshengning

Copy link
Copy Markdown
Contributor

PR Category

Distributed Strategy

PR Types

New features

Description

[Fsdp] support fp8 and fused linear

是否引起精度变化

否

@risemeup1111

This comment was marked as outdated.

if group.grads_buffer is not None and (
group.is_expert_param or not group.grads_buffer.is_sharded
):
return

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P3

此处移除按计数触发 reduce 的逻辑(原 if group.grads_use_cnt == group.grads_use_sum: self._reduce_group_grads(group))后,reduce 已全部由 _flush_expert_grads_after_unit 与 finish_grads_sync 结构化触发。grads_use_sum(构造时按 len(params) 赋值)以及 grads_use_cnt(reduce_scatter_grads 中自增、多处清零)在全仓库范围内已无任何读取点。

建议一并清理这些残留字段与自增,避免误导后续维护者以为计数触发仍在生效。

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

收到,此项按作者确认接受,不再要求修改。

已在当前 head a7bd5e0 复核:grads_use_sum / grads_use_cnt(第 67-68 行声明,第 493、769、779、862 行写入)在本文件内没有任何读取点,不影响功能正确性,仅为计数触发移除后的残留字段,作为可选清理项保留。

@risemeup1111 risemeup1111 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

本次同步为 formatting-only 变更(Codestyle-Check),未引入新增问题。此前两项未解决项仍需处理:

  • P1:新增 FSDP fp8 / 融合线性 / Muon 的数值与多卡通信行为仍缺少单卡 + 多卡测试覆盖;
  • P3:按计数触发 reduce 的逻辑移除后,grads_use_cnt / grads_use_sum 已成死代码待清理。

详见对应行内评论。

Paddle-Review-Bot

This comment was marked as outdated.

Paddle-Review-Bot

This comment was marked as outdated.

Paddle-Review-Bot

This comment was marked as outdated.


ctx.comm_manager.all_gather_params(trainable_params, is_backward=True)

return args

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1

最新两个 commit 删除了本 PR 自己在 fsdp_support_fused_linear 中加入的 duplicable slot 返回修正(ctx._n_forward_slots 与 if n_slots == 1 and len(args) > 1: return list(args)),但消费该修正的调用路径仍然存在,会让「main + MTP heads」场景在反向直接报错。

触发路径:_forward_post_hook(第 1824-1830 行 else 分支)对返回 list 的 layer 输出走 FusionBackwardHook.apply(outputs, ...),把整个 list 作为一个位置实参传入:

  • paddle/fluid/pybind/eager_py_layer.cc:382-391 把 list 实参登记为 1 个 duplicable 输入 slot,即 forward_input_tensor_is_duplicable.size() == 1;
  • forward 返回 inputs[0](仍是该 list),eager_py_layer.cc:439-440 用 PyList_AsTuple 把它展开成 N 个输出 slot;
  • 反向因此收到 N 个独立梯度实参,此处 return args 返回 N 元 tuple,命中 paddle/fluid/eager/pylayer/py_layer_node.cc:201-208 的检查并抛出 The number of outputs of PyLayer.backward should be 1, but received N。

同时第 1679-1686 行仍保留「several fused calls (main + MTP heads)」的 fp32 梯度 list 累加逻辑,说明该场景仍在本 PR 支持范围内,接口两端已不一致。

建议恢复按 slot 数聚合的返回(1 个 duplicable slot 时以单个 list 返回该位置;注意不能把多个梯度求和,求和能跑通但梯度错误,属静默数值错误),或说明该路径已由其它改动覆盖。

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

已确认本PR 此处不做修改,请重新review

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

已按当前 head 21367f0 复查,接受此处不改,并修正此前的优先级判断:相对 base 无回归,该项由 P1 下调为 P2 并标记为已接受。

依据:merge-base 9460e89 上 FusionBackwardHook.backward 同样是 return args,_forward_post_hook 的 else 分支也同样把返回 list 的 layer 输出整体交给该 PyLayer。因此「返回 list 的 layer 在反向报错」是 develop 既有行为,本 PR 删除该补丁后与 base 一致,不构成本 PR 引入的回归。此前将其判为 P1 回归并不准确。

仅保留一点供后续判断(不影响本 PR):第 1679-1686 行仍保留「several fused calls (main + MTP heads)」的 fp32 梯度 list 累加逻辑,若该场景确实在支持范围内,返回 list 的 layer 仍无法走完反向;若该场景不在本 PR 范围内可忽略。

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

已确认没有问题

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

收到,此项维持 P2 / 已接受,不再重复提出。

已按当前 head 6f81ae5 再次确认:该 commit 为 Merge branch 'PaddlePaddle:develop',第 967 行 return args 与上一轮一致;相对新 merge-base 785b2a1,FusionBackwardHook.backward 与 _forward_post_hook 的 else 分支行为与基准相同,本 PR 不引入回归。此前提到的「several fused calls (main + MTP heads)」fp32 梯度 list 累加(第 1679-1686 行)仅作为后续场景的备注,不作为本 PR 的问题项。

"fully_shard_fusion.py",
)
)
spec = importlib.util.spec_from_file_location("fsf_under_test", src)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2

新增用例通过 spec_from_file_location 按相对路径加载源码文件,而不是 import paddle.distributed.fsdp.fully_shard_fusion,存在两个问题:

  1. 进程内会同时存在两份同一模块(fsf_under_test 与 paddle.distributed.fsdp.fully_shard_fusion),fsf.FullyShardFusion / fsf.FusionBackwardHook 与 paddle 实际导出的类不是同一对象。当测试环境的 paddle 来自已安装包、而相对路径指向另一份源码时,用例校验的并不是真正被发布的那份实现——与第 17-19 行 docstring 想规避的情况恰好相反。
  2. 路径硬编码为 __file__/../../python/paddle/distributed/fsdp/,测试文件或被测模块任一方移动位置即失效,且无法覆盖 paddle.distributed.fsdp 包导出装配是否正常。

建议改为 from paddle.distributed.fsdp import fully_shard_fusion as fsf,让用例与实际导入路径保持一致。

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

已修复,标记为已解决。head a7bd5e0 中用例已改为 from paddle.distributed.fsdp import fully_shard_fusion as fsf,走 paddle 公开导入路径,importlib/sys.path 相关的自定义加载函数已删除。

self.pre_hook = hook


# PLACEHOLDER_TESTS

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P3

第 127、174、216 行残留 # PLACEHOLDER_TESTS / # PLACEHOLDER_TESTS2 / # PLACEHOLDER_TESTS3 三处占位注释,属于生成文件时的脚手架标记,无实际含义,建议一并删除。

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

已修复,标记为已解决。head a7bd5e0 中三处 # PLACEHOLDER_TESTS / PLACEHOLDER_TESTS2 / PLACEHOLDER_TESTS3 占位注释均已删除。

@codecov-commenter

codecov-commenter commented Oct 14, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 96.07843% with 2 lines in your changes missing coverage. Please review.
⚠️ Please upload report for BASE (develop@785b2a1). Learn more about missing BASE report.

Files with missing lines Patch % Lines
...thon/paddle/distributed/fsdp/fully_shard_fusion.py 96.07% 2 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             develop   #79813   +/-   ##
==========================================
  Coverage           ?   96.07%           
==========================================
  Files              ?        1           
  Lines              ?       51           
  Branches           ?        0           
==========================================
  Hits               ?       49           
  Misses             ?        2           
  Partials           ?        0           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Comment on lines +86 to +89
class _CompletedTask:
# Placeholder for the broadcast (sync) path where callers still .wait().
def wait(self):
pass

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P3

本 PR 新增的 TensorFusionBuffer._CompletedTask(第 86-89 行)在仓库内没有任何引用,grep -rn "_CompletedTask" 仅命中此处定义。注释提到的 broadcast(同步)路径并未使用它返回占位 task,属于未接入的死代码,建议删除;若后续确实要让同步路径返回可 .wait() 的对象,请在同一 PR 内补上调用点。

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

收到,此项按作者确认接受,不再要求修改。

已在当前 head a7bd5e0 复核确认不存在隐藏风险:do_all_gather / do_reduce_scatter 均以 sync_op=False 返回真实 task,三处 sync_op=True 的 all_reduce / reduce / broadcast(第 865、872、1633 行)都直接丢弃返回值、未赋给 comm_task,因此第 654、850 行的 .wait() 不会遇到 None。该占位类仅为未接入的冗余代码,作为可选清理项保留。

"fully_shard_fusion.py",
)
)
spec = importlib.util.spec_from_file_location("fsf_under_test", src)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

已修复,标记为已解决。head a7bd5e0 中用例已改为 from paddle.distributed.fsdp import fully_shard_fusion as fsf,走 paddle 公开导入路径,importlib/sys.path 相关的自定义加载函数已删除。

self.pre_hook = hook


# PLACEHOLDER_TESTS

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

已修复,标记为已解决。head a7bd5e0 中三处 # PLACEHOLDER_TESTS / PLACEHOLDER_TESTS2 / PLACEHOLDER_TESTS3 占位注释均已删除。

@liym27 liym27 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@zhengshengning
zhengshengning merged commit 2be7d96 into PaddlePaddle:develop Oct 15, 2026
80 of 81 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants