Skip to content

[Cherry-Pick][Operator Mechanism] Fix O(n^2) python stack accumulation in SetPythonStack - #79862

Merged
sneaxiy merged 3 commits into
PaddlePaddle:release/3.4from
feixi139:cp79831_release34
Oct 15, 2026
Merged

sneaxiy merged 3 commits into
PaddlePaddle:release/3.4from
feixi139:cp79831_release34

Conversation

@feixi139

@feixi139 feixi139 commented Oct 13, 2026 •

Copy link
Copy Markdown
Contributor

PR Category

Operator Mechanism

PR Types

Bug fixes

Description

修复动态图下开启 FLAGS_check_nan_inf(默认 FLAGS_check_nan_inf_level=0)时,每个 op 调用都把当前 Python 栈拼接到历史栈上且从不清理,导致进程内 O(n²) 退化的性能 bug:

  1. 性能 bug(根因修复):SetPythonStack()(paddle/fluid/pybind/eager_utils.cc)每次 op 都执行 str + 历史整串 再回存。GetPythonStack() 返回完整 Python 调用栈文本(GIL 获取、traceback 模块调用、逐行字符串拼接),该 thread_local 串每个 op 增长一段栈,每次调用还要整串拷贝数遍——单个 op 的开销随进程累计 op 数线性增长,整体 O(n²)。实测 2600 个小 tensor 的造数从 1.88 ms/tensor 单调恶化到 35.37 ms/tensor(总计 44.17s);每个 grad node 的 SetForwardTrace 还会把整串再复制一份;
  2. 调试语义 bug:反向报 nan 时打印的「该 op 的前向栈」实际是从进程启动以来所有 op 栈的拼接串(PaddleAPITest 长 run 中可达数 MB),并非该 op 自己的前向栈——这正好违背了引入该逻辑的目的。

两处均源自 2023-04 PR #52639:其目的是反向出 nan 时打印该 op 的前向栈,但拼接写法并不服务于该目的(同函数 FLAGS_call_stack_level == 3 分支本来就是直接覆盖)。该代码在上游 develop 至今未改。

修复

将拼接改为覆盖,只保留当前栈(-2/+1 行):

std::string str = GetPythonStack();
egr::Controller::Instance().SetPythonStack(str);

实测数据

四调试 FLAG 全开(FLAGS_use_system_allocator=1 FLAGS_check_nan_inf=true FLAGS_alloc_fill_value=255 FLAGS_check_cuda_error=true),逐 tensor 执行 rand → scale → scale → cast → assign:

2600 tensor 总耗时 分段 ms/tensor RSS 增长
修复前 44.17 s 1.88 → 4.91 → 25.79 → 35.37(单调恶化) 37 MB
修复后 1.75 s 0.64 – 0.72(平稳) 15 MB

PaddleAPITest 真实路径(engineV4,单个 2606-tensor 的 paddle.concat case):修复前 1009s 最终 timeout;修复后 18s pass。原生栈采样显示修复前 88% 时间耗在 SetPythonStack 的字符串构造/拷贝(_M_construct/_M_mutate)上。

影响面

  • 默认路径(不开 FLAGS_check_nan_inf):不进入该分支,行为不变;
  • FLAGS_check_nan_inf_level=1:该分支本来就被跳过,行为不变;
  • FLAGS_call_stack_level == 3 分支:本来就是覆盖,行为不变;
  • 开 FLAGS_check_nan_inf 且 level=0(默认):唯一行为变化是反向 nan 时打印的前向栈从「历史拼接串」变为「该 op 自己的前向栈」,即恢复其本来的语义。

回归测试

新增 test/legacy_test/check_nan_inf_forward_trace.py 及 test_nan_inf.py::TestNanInfStack::test_forward_trace_not_accumulated:先在 _stale_frame_marker()(一个已返回的调用帧)中调一次 paddle.rand,再做 sqrt(0)(反向为 inf),断言打印的前向栈包含 z = paddle.sqrt(x) 且不包含已返回的 _stale_frame_marker 帧。该测试在未修复版本上 FAIL(前向栈混入 marker 帧,正是 bug 原样),修复后 PASS;test_nan_inf.py 全部 13 个测试修复后均通过。

另注:现有 check_nan_inf_backward_stack.py 的 pow 用例在当前版本反向不产生 nan,断言实际未被触发(空过),本次未改动。

devPR:#79831

是否引起精度变化

否

test_forward_trace_not_accumulated asserted on the raw source line
'z = paddle.sqrt(x)', whose presence depends on linecache being able to
read the script source; this is not guaranteed on all CI runners (e.g.
DCU), causing a false failure. Match the traceback frame header
(co_filename + co_name), which traceback.format_stack always emits, and
invoke the helper via an absolute path like the sibling tests.
check_nan_inf_forward_trace.py only exercises the device-independent
Python call-stack mechanism, but _stale_frame_marker() ran paddle.rand on
the process default device. On XPU runners that produced an XPU tensor
whose forward check_nan_inf dispatched a CPUContext check-numerics kernel
and segfaulted, so the subprocess aborted before printing the forward
trace. Pin the script to CPU via paddle.set_device('cpu') so it no longer
depends on the CI runner's default device.
@sneaxiy
sneaxiy merged commit ba4147b into PaddlePaddle:release/3.4 Oct 15, 2026
253 of 276 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants