Repository navigation
[Cherry-Pick][Operator Mechanism] Fix O(n^2) python stack accumulation in SetPythonStack - #79862
Merged
Merged
Conversation
test_forward_trace_not_accumulated asserted on the raw source line 'z = paddle.sqrt(x)', whose presence depends on linecache being able to read the script source; this is not guaranteed on all CI runners (e.g. DCU), causing a false failure. Match the traceback frame header (co_filename + co_name), which traceback.format_stack always emits, and invoke the helper via an absolute path like the sibling tests.
check_nan_inf_forward_trace.py only exercises the device-independent
Python call-stack mechanism, but _stale_frame_marker() ran paddle.rand on
the process default device. On XPU runners that produced an XPU tensor
whose forward check_nan_inf dispatched a CPUContext check-numerics kernel
and segfaulted, so the subprocess aborted before printing the forward
trace. Pin the script to CPU via paddle.set_device('cpu') so it no longer
depends on the CI runner's default device.
Paddle-Review-Bot
approved these changes
Oct 13, 2026
wanghuancoder
approved these changes
Oct 14, 2026
sneaxiy
approved these changes
Oct 15, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
PR Category
Operator Mechanism
PR Types
Bug fixes
Description
修复动态图下开启
FLAGS_check_nan_inf(默认FLAGS_check_nan_inf_level=0)时,每个 op 调用都把当前 Python 栈拼接到历史栈上且从不清理,导致进程内 O(n²) 退化的性能 bug:SetPythonStack()(paddle/fluid/pybind/eager_utils.cc)每次 op 都执行str + 历史整串再回存。GetPythonStack()返回完整 Python 调用栈文本(GIL 获取、traceback 模块调用、逐行字符串拼接),该 thread_local 串每个 op 增长一段栈,每次调用还要整串拷贝数遍——单个 op 的开销随进程累计 op 数线性增长,整体 O(n²)。实测 2600 个小 tensor 的造数从 1.88 ms/tensor 单调恶化到 35.37 ms/tensor(总计 44.17s);每个 grad node 的SetForwardTrace还会把整串再复制一份;两处均源自 2023-04 PR #52639:其目的是反向出 nan 时打印该 op 的前向栈,但拼接写法并不服务于该目的(同函数
FLAGS_call_stack_level == 3分支本来就是直接覆盖)。该代码在上游 develop 至今未改。修复
将拼接改为覆盖,只保留当前栈(-2/+1 行):
std::string str = GetPythonStack(); egr::Controller::Instance().SetPythonStack(str);实测数据
四调试 FLAG 全开(
FLAGS_use_system_allocator=1 FLAGS_check_nan_inf=true FLAGS_alloc_fill_value=255 FLAGS_check_cuda_error=true),逐 tensor 执行rand → scale → scale → cast → assign:PaddleAPITest 真实路径(engineV4,单个 2606-tensor 的
paddle.concatcase):修复前 1009s 最终 timeout;修复后 18s pass。原生栈采样显示修复前 88% 时间耗在SetPythonStack的字符串构造/拷贝(_M_construct/_M_mutate)上。影响面
FLAGS_check_nan_inf):不进入该分支,行为不变;FLAGS_check_nan_inf_level=1:该分支本来就被跳过,行为不变;FLAGS_call_stack_level == 3分支:本来就是覆盖,行为不变;FLAGS_check_nan_inf且 level=0(默认):唯一行为变化是反向 nan 时打印的前向栈从「历史拼接串」变为「该 op 自己的前向栈」,即恢复其本来的语义。回归测试
新增
test/legacy_test/check_nan_inf_forward_trace.py及test_nan_inf.py::TestNanInfStack::test_forward_trace_not_accumulated:先在_stale_frame_marker()(一个已返回的调用帧)中调一次paddle.rand,再做sqrt(0)(反向为 inf),断言打印的前向栈包含z = paddle.sqrt(x)且不包含已返回的_stale_frame_marker帧。该测试在未修复版本上 FAIL(前向栈混入 marker 帧,正是 bug 原样),修复后 PASS;test_nan_inf.py全部 13 个测试修复后均通过。另注:现有
check_nan_inf_backward_stack.py的 pow 用例在当前版本反向不产生 nan,断言实际未被触发(空过),本次未改动。devPR:#79831
是否引起精度变化
否