Repository navigation
Conversation
Paddle-Bot Review Board (review完成)
可维护性建议(总分 84.7 / 100,只给建议,不影响评审结论)
各维度得分(仅评本 PR 改动,点击展开)
Powered by Nyanpasu claude with Opus 5.5 默认推理级别, please check the suggestions carefully. |
| dense_buckets.resize(num_dense_buckets); | ||
| for (int i = 0; i < num_dense_buckets; ++i) { | ||
| const long long* row = dense_table.data() + 7 * (size_t)i; | ||
| dense_buckets[i] = | ||
| DenseBucket{row[0], row[1], row[2], row[3], row[4], row[5], row[6]}; | ||
| } | ||
| moe_buckets.resize(num_moe_buckets); | ||
| for (int i = 0; i < num_moe_buckets; ++i) { | ||
| const long long* row = moe_table.data() + 5 * (size_t)i; | ||
| moe_buckets[i] = MoeBucket{row[0], row[1], row[2], row[3], row[4]}; | ||
| } |
There was a problem hiding this comment.
setup 不校验每桶参数表里的偏移,但放行判断会在 CPU 上直接按这些偏移读 host window。
blocked_dense_inter_node_reduce/blocked_moe_reduce读host + signal_offset/8*8,blocked_intra_node_reduce读intra_arrive_offset/8 + bucket*16 + l,全程不和host_bytes比较。Python 侧的布局一旦算错,结果是训练进程里 CPU 越界读(段错误),拿不到错误码。- 偏移不是 8 字节对齐时,
/8会截断到相邻的 word,而 GPU 的 signal 写在原偏移上。这个桶的分片求和会永远等不到目标值,症状是卡住,原因难查。 setup_gather_table校验了send_offset >= 0和shard_numel > 0,没校验row[8]/row[9]非负;负数转成size_t后就是 GPU 写 owner host window 时用的巨大偏移。
建议在 setup 里对 signal_offset、intra_arrive_offset(含 bucket*16+ranks_per_node 那一段)、各 recv_offset_* / send_offset 校验非负、对齐(signal/arrive 8 字节,数据 4 字节),并检查是否落在对应 window 内(host_bytes / send_bytes);不满足时返回 cudaErrorInvalidValue。setup_gather_table 也补上 row[8]、row[9] 的同类检查。
There was a problem hiding this comment.
已在当前提交修复:tables_in_bounds 对每桶的 send/recv/signal 偏移、intra_arrive 区和三个 credit 区做了非负、对齐(计数 8 B,数据 4 B)和窗口范围检查,setup_gather_table 也补上了 row[8] / row[9] 的检查。
| if (mark_ < 0 || mark_ >= tail_mark_index) return 2; | ||
| if (control->state == DONE) | ||
| return kind < 0 ? 0 : 2; // 已全部放完:只换 mark 的消息直接丢弃 | ||
| if (control->state != RUN) return 2; | ||
| control->queue.push_back(ReadyMessage{kind, bucket, mark_, scale}); | ||
| control->cv_proxy.notify_all(); | ||
| return 0; |
There was a problem hiding this comment.
There was a problem hiding this comment.
已在当前提交修复:新增只由主线程维护的 reported_* 位图,IDLE 和 RUN 两个分支都拒绝重复上报,step_reset 负责清零。
| int bucket_ready(int kind, int bucket, float scale, int mark); | ||
| int release_status(std::vector<int>& positions, | ||
| std::vector<int>& counts, | ||
| std::vector<int>& posted, | ||
| int* done); | ||
| int release_stuck_report(std::vector<long long>& flat); | ||
| int release_error(int* code, int* stage, int* bucket); | ||
| int release_activity(); | ||
| int tail_mark(); | ||
| int tail_start(int intra_node_reduce_blocks, | ||
| int shard_reduce_blocks, | ||
| int mark, | ||
| int all_posted); | ||
| int tail_all_posted(); | ||
| int tail_wait(double timeout_s); | ||
| int tail_abort(double wait_s); |
There was a problem hiding this comment.
💡 可维护性建议(契约一致性,不阻塞合入):类注释的「约定」写着返回 int 返回码(0 成功,<10000 = cudaError_t,>=10000 = 10000 + ncclResult_t),但本 PR 新增的几个方法不走这套编码:bucket_ready / tail_all_posted 返回 0/1/2;tail_wait / tail_abort 返回 TAIL_* 状态,其中 TAIL_RUN = 1 与 cudaErrorInvalidValue 数值相同;tail_mark 返回槽号或 -1/-2。调用方套用通用的返回码检查就会误判。建议在「约定」里逐条列出这些例外及其取值,或改成「返回码 + 出参」的形式(同 release_error)。
There was a problem hiding this comment.
已处理:bucket_ready、tail_all_posted、tail_mark、tail_wait、tail_abort、release_activity 的返回值都在声明旁写明了,类注释也说明了例外方法看各自的声明。
| kernels/put.cu | ||
| kernels/signal.cu | ||
| kernels/scale.cu | ||
| kernels/release.cu) |
There was a problem hiding this comment.
💡 可维护性建议(包组织结构,不阻塞合入):kernels/release.cu 里没有 __global__,也没有 <<<>>>,内容是 host 侧状态机和 std::thread。这和 kernels/kernels.h 文件头「kernels/ 只放 kernel 的 launch 包装」的约定不符,还会让这段 host 代码多经过一次 nvcc。建议移到 gin_overlap/release.cc,与 gin_engine.cc 并列,并同步更新这里和第 12 行的注释。
There was a problem hiding this comment.
已处理:放行层已移到 gin_overlap/release.cc,CMake 和各处引用同步更新了。
| // - pos_moe_put:专家 put。 | ||
| // - stage_positions[3]:专家分片求和。 | ||
| // 一个阶段的待放行队列 = 它的位置到上一阶段位置之间的那些桶。 | ||
| int pos_dense_copy = 0, stage_positions[4] = {0, 0, 0, 0}, pos_moe_put = 0; |
There was a problem hiding this comment.
💡 可维护性建议(底层优雅度,不阻塞合入):六个阶段的进度里,两个是具名变量,另外四个放在 stage_positions[0..3] 里,下标含义只写在上面的注释中。所以 release_next_put、pending_counts、stuck_stage 里出现 stage_positions[1] >= stage_positions[0] 这类表达式,要对着注释才能看懂。建议拆成与 STAGE_* 对应的具名变量(如 pos_dense_intra_node_reduce / pos_dense_inter_node_put / pos_dense_inter_node_reduce / pos_moe_reduce)。step_reset 里的清零逻辑也随之变直白。
There was a problem hiding this comment.
已处理:六个阶段的进度都改成了具名变量,stage_positions 数组已删除。
| control = new TailControl(); | ||
| try { | ||
| std::thread(thread_main).detach(); | ||
| } catch (...) { | ||
| return (int)cudaErrorUnknown; | ||
| } |
There was a problem hiding this comment.
💡 可维护性建议(初始化耦合,不阻塞合入):control 在尾部线程启动之前就已经发布。std::thread 构造失败时只返回 cudaErrorUnknown,control 留在非空状态,于是 setup 再也不能重试(入口检查 control 非空)。之后 advance 照常可用;tail_start 会把状态改成 RUN,但没有线程处理,tail_wait 只能等到超时。建议在 catch 分支里 delete control; control = nullptr; 后再返回,或者等线程成功启动后再给 control 赋值。
There was a problem hiding this comment.
已处理:线程启动失败时会 delete control 并置空,setup 可以重试。
| inline unsigned long long credit_target(long long num_buckets, | ||
| long long position, | ||
| long long depth) { | ||
| if (depth < 0) return 0; | ||
| const long long target = step * num_buckets + position - depth + 1; | ||
| return target > 0 ? (unsigned long long)target : 0ull; | ||
| } |
There was a problem hiding this comment.
💡 可维护性建议(测试策略,不阻塞合入):放行层的判断大多是纯 host 逻辑:credit_target 的跨步递推、blocked_* 对 host window 计数的比较、六个阶段位置的先后约束,以及 IDLE/RUN/DONE/ERR/ABORTED 的状态转换。这些逻辑目前只能靠多机实跑覆盖。建议后续的单测 PR 把这部分和 submit_* 解耦(例如用一块普通内存模拟 host window,用桩函数替代 submit),至少覆盖 credit 目标、阶段推进顺序和 bucket_ready / tail_* 的状态检查。
badc00c to
f6bc21c
Compare
| if (!in_window(row[0], m * dense.size, 4, 4, dense.send_bytes) || | ||
| !in_window(row[1], 1, 8, 8, dense.host_bytes)) | ||
| return false; | ||
| if (row[5] >= 0 && | ||
| !in_window(row[3], (L - 1) * Nn * m, 4, 4, dense.recv_bytes)) |
There was a problem hiding this comment.
dense 的
send_offset(row[0])和 recv_offset_intra(row[3])这里只要求 4 字节对齐,但 submit_dense_intra_node_reduce 把 send + send_offset、recv + recv_offset 直接当 float4* 交给 k_reduce_intra_node。这个 kernel 没有标量回退(kernels/reduce.cu 71-111 行),要求 16 字节对齐。
结果是一个 4 字节对齐、但不是 16 字节对齐的偏移能通过 setup,要到 kernel 才报 misaligned address。这是 sticky 错误,进程只能重启,拿不到 cudaErrorInvalidValue。上一轮建议里写的“数据 4 字节”对这两列不够。
建议把这两列的对齐要求改成 16(Python 侧按 256 对齐,不受影响),或者在 submit_dense_intra_node_reduce 里检查两个指针的对齐,不满足时返回 cudaErrorInvalidValue。
There was a problem hiding this comment.
已在当前提交修复:dense 的 send_offset(row[0])和 recv_offset_intra(row[3])现在按 16 字节对齐校验,与 k_reduce_intra_node 的 float4 访问一致。
f6bc21c to
ae8eee4
Compare
ae8eee4 to
f1cb51e
Compare
f1cb51e to
c10b645
Compare
…treams Second piece of the move from PaddleFleet's extension: the host side that builds everything the per-step code runs on. Nothing calls it yet (the Python binding comes last), and nothing is submitted to the GPU per step yet (next commit). * gin_engine.h / gin_engine.cc, initialization only: the CommGroup state of the two communication groups (sharding / moe_sharding), the error-check macros, and the GradCommEngine methods that the Python layer calls once at startup, in this order: set_comms (reuse Paddle's ncclComm_t for each ring id and cross-check size / rank), alloc_windows (send / recv / signal as NCCL_WIN_COLL_SYMMETRIC windows), alloc_host_counters (one HOST_NUMA host window per group, registered with NCCL_WIN_STRICT_ORDERING), create_dev_comm (ncclDevCommCreate + LSA pointers), set_group_topology, and configure (side streams, per-bucket events, the window-mark ring, the credit offsets, the host-window LSA pointers, and preloading every kernel so that the first launch of a module does not land on the backward critical path). Also error_string and the NCCL runtime / build version queries. * paddle_comm.cc: takes the ncclComm_t out of Paddle's ProcessGroup by ring id, the way DeepEP does (ProcessGroupMapFromGid -> ProcessGroupNCCL -> NCCLCommContext::GetNcclComm). * paddle_tensor.cc / .h: Paddle's current compute stream, the zero-copy paddle::Tensor over a send-window slice (paddle::from_blob with deleter = nullptr, so the memory stays owned by the NCCL window, plus a post-wrap self-check), and the pitched copy_2d through cuMemcpy2DAsync. The file header of gin_engine.h is the one full description of the C++ side (terms, the two groups, the dense / expert / 2D stages, the receive pools, the host-window layout and the constraints). It is added here in full: it is organised by concept, not by layer, so the parts about the per-stage submits and the release layer (next two commits) cannot be moved out without rewriting it. paddle_comm.cu and paddle_tensor.cu become .cc: they were .cu only because an out-of-tree build has to hand any translation unit that includes Paddle headers to nvcc (thrust); in-tree they are ordinary sources. The pybind registration that was stranded in paddle_tensor.cu moves to the pybind layer, so paddle_tensor.h declares the real paddle::Tensor signatures. The library now also depends on process_group_nccl, for paddle_comm.cc. Like the kernels, the code is moved, not rewritten: same checks, same order of collective calls. Co-Authored-By: Claude Code <noreply@anthropic.com>
Translate every comment, docstring, error message and log line of the gin_overlap sources (C++, CUDA, CMake, pybind and the Python layer) into English, and tighten the wording on the way. The terms are defined once, in the gin_engine.h file header (C++) and in the engine.py module docstring (Python); every other file uses them as defined there. Strings only. comment_clean_check.py --ignore-strings passes on every changed file: C++ / CUDA / CMake token streams are identical except for string literals (printf conversions inside merged literals keep their order), and the Python ASTs are identical except for str constants and the literal parts of f-strings. Log lines all carry the [gin_overlap] prefix; copy_2d errors now start with "copy_2d:". Co-Authored-By: Claude Code <noreply@anthropic.com>
…ne API
Argument checks (values come from Python):
- scale_range, gather_put_to_owner, send_window_tensor and copy_2d check
their bounds without unsigned wrap-around, and reject numel < 0.
copy_2d also rejects non-contiguous tensors (a stride-0 or transposed
view passes a numel-based bound but the copy reads contiguous bytes).
- release::setup checks every offset of the per-bucket tables and the
host window areas: non-negative, aligned (counters 8 B, data 4 B) and
inside its window. setup_gather_table checks both arrival counter
offsets of each entry the same way. CommGroup records recv_bytes for
this, and alloc_symmetric_window returns the size it actually
registered instead of the caller repeating the 4096 B minimum.
- A wrong state returns cudaErrorNotReady and a bad argument
cudaErrorInvalidValue (gather_setup, gather_put_to_owner, scale_range).
- bucket_ready rejects a bucket reported twice in a step in the RUN state
too, from a bitmap only the main thread touches.
Init failures:
- configure destroys the streams and events it created, frees its
temporary buffer and resets the globals it set when a later step fails,
so it can be retried. set_comms clears the groups it set on failure.
release::setup unpublishes its control block if the tail thread cannot
be started. The steps that cannot be rolled back (window registration,
devcomm creation, the host window) say so: restart the process.
- The gin_engine.h constraints state that the engine lives as long as
the process; shutdown_process_group / restart_process_group are not
supported.
API:
- GIN_CHECK replaces the dozen { rc = launch_x(...); if (rc) return rc; }
blocks. launch_* / preload_* return cudaError_t.
- launch_put_to_expert_peers takes (send_offset, recv_offset) like
launch_put_to_node_peers.
- join_bucket_streams / gather_wait_bucket take kind (0 dense, 1 expert),
as bucket_ready and the gather table do, instead of is_dense.
- The six release positions are named variables, one per stage.
- The methods that do not return a code (record_window_mark, tail_mark,
bucket_ready, tail_all_posted, tail_wait, tail_abort, release_activity,
the version getters) document what they return.
- tail_wait / tail_abort release the GIL while they block.
No kernel, reduction order, release condition or credit target changes.
Co-Authored-By: Claude Code <noreply@anthropic.com>
kernels/release.cu has no kernel and no <<<>>>: it is the host-side release state machine and its std::thread. kernels/ holds only the kernel launch wrappers, so the file moves next to gin_engine.cc as release.cc and is compiled by the host compiler instead of nvcc. The content is unchanged apart from the file name in comments. Co-Authored-By: Claude Code <noreply@anthropic.com>
Remove gin_overlap/ from the cpplint exclude list and fix every finding: C-style casts become static_cast / reinterpret_cast, long long and unsigned long long become int64_t / uint64_t, non-const reference out-parameters become pointers (call sites updated), and C system headers come before C++ ones. atomicAdd_system keeps its unsigned long long prototype through one aliased type. No change to values, types seen by Python, or arithmetic. Co-Authored-By: Claude Code <noreply@anthropic.com>
…e slot as an out-parameter C++: both return 0 or a cudaError_t code and write the slot to *slot: cudaErrorNotReady without configure / release_setup, the error of paddle_compute_stream or cudaEventRecord, and cudaErrorLaunchOutOfResources when this step's tail mark slots are used up (was -1 / -2). The Python binding still returns the slot and raises RuntimeError with the error text on failure. CPU test of the binding before setup. Co-Authored-By: Claude Code <noreply@anthropic.com>
…ther Third piece of the move from PaddleFleet's extension: the host functions that hand one stage of one bucket to its side stream, and the stream / event plumbing around them. Still no caller: the release layer that decides when each stage may go (next commit) is the only code that calls submit_*. * The six stage submits: submit_dense_intra_node_copy (cudaMemcpyBatchAsync of the column to the other ranks on the node + the arrival write), submit_dense_intra_node_reduce (+ intra-node credit), submit_dense_inter_node_put, submit_moe_put, submit_dense_inter_node_reduce and submit_moe_reduce (+ GIN credit). Each one only waits on work this rank has already handed out (a compute-stream mark or a local side-stream event), never on a remote counter: whether the remote side is ready is decided on the host from the host window before the call. * record_window_mark: records a mark on Paddle's current compute stream into the ring configure() created. * join_bucket_streams (compute stream waits for a bucket's side-stream events), scale_range, end_step (GIN flush of both groups so the send window can be reused), stats. * Muon 2D-bucket gather to the owner: gather_setup, gather_gate, gather_wait_bucket, gather_mark_ready, gather_put_to_owner (GIN put across nodes, CE + release store within a node) and gather_join. The table-driven entry point, gather_put_entry, forwards to the release layer and comes with it. Moved as is from the extension, like the previous commits. Co-Authored-By: Claude Code <noreply@anthropic.com>
Translate every comment, docstring, error message and log line of the gin_overlap sources (C++, CUDA, CMake, pybind and the Python layer) into English, and tighten the wording on the way. The terms are defined once, in the gin_engine.h file header (C++) and in the engine.py module docstring (Python); every other file uses them as defined there. Strings only. comment_clean_check.py --ignore-strings passes on every changed file: C++ / CUDA / CMake token streams are identical except for string literals (printf conversions inside merged literals keep their order), and the Python ASTs are identical except for str constants and the literal parts of f-strings. Log lines all carry the [gin_overlap] prefix; copy_2d errors now start with "copy_2d:". Co-Authored-By: Claude Code <noreply@anthropic.com>
…the gate Both cudaMemcpyBatchAsync calls use the CUDA 13 signature (8 arguments, const void** sources). CUDA 12.8 / 12.9 declare a 9-argument version, so a build with the gate open on those toolkits would not compile. Raise the gate to CUDA >= 13.0, the only toolkit the code has been built and run with, instead of adding an untested 12.x branch; gin_dynload.h gets the matching #error backstop. The batch attribute index is named attrs_idx, which is what that argument is. The arch condition reuses ARCH_BIN_CONTAINS_GE_90, computed at the top of the same file for DeepEP, instead of a second loop over NVCC_ARCH_BIN. _BUILD_HINT in the Python layer now lists exactly the gate's conditions, and the CMake comment says to change both together. Co-Authored-By: Claude Code <noreply@anthropic.com>
…ne API
Argument checks (values come from Python):
- scale_range, gather_put_to_owner, send_window_tensor and copy_2d check
their bounds without unsigned wrap-around, and reject numel < 0.
copy_2d also rejects non-contiguous tensors (a stride-0 or transposed
view passes a numel-based bound but the copy reads contiguous bytes).
- release::setup checks every offset of the per-bucket tables and the
host window areas: non-negative, aligned (counters 8 B, data 4 B) and
inside its window. setup_gather_table checks both arrival counter
offsets of each entry the same way. CommGroup records recv_bytes for
this, and alloc_symmetric_window returns the size it actually
registered instead of the caller repeating the 4096 B minimum.
- A wrong state returns cudaErrorNotReady and a bad argument
cudaErrorInvalidValue (gather_setup, gather_put_to_owner, scale_range).
- bucket_ready rejects a bucket reported twice in a step in the RUN state
too, from a bitmap only the main thread touches.
Init failures:
- configure destroys the streams and events it created, frees its
temporary buffer and resets the globals it set when a later step fails,
so it can be retried. set_comms clears the groups it set on failure.
release::setup unpublishes its control block if the tail thread cannot
be started. The steps that cannot be rolled back (window registration,
devcomm creation, the host window) say so: restart the process.
- The gin_engine.h constraints state that the engine lives as long as
the process; shutdown_process_group / restart_process_group are not
supported.
API:
- GIN_CHECK replaces the dozen { rc = launch_x(...); if (rc) return rc; }
blocks. launch_* / preload_* return cudaError_t.
- launch_put_to_expert_peers takes (send_offset, recv_offset) like
launch_put_to_node_peers.
- join_bucket_streams / gather_wait_bucket take kind (0 dense, 1 expert),
as bucket_ready and the gather table do, instead of is_dense.
- The six release positions are named variables, one per stage.
- The methods that do not return a code (record_window_mark, tail_mark,
bucket_ready, tail_all_posted, tail_wait, tail_abort, release_activity,
the version getters) document what they return.
- tail_wait / tail_abort release the GIL while they block.
No kernel, reduction order, release condition or credit target changes.
Co-Authored-By: Claude Code <noreply@anthropic.com>
kernels/release.cu has no kernel and no <<<>>>: it is the host-side release state machine and its std::thread. kernels/ holds only the kernel launch wrappers, so the file moves next to gin_engine.cc as release.cc and is compiled by the host compiler instead of nvcc. The content is unchanged apart from the file name in comments. Co-Authored-By: Claude Code <noreply@anthropic.com>
Co-Authored-By: Claude Code <noreply@anthropic.com>
Remove gin_overlap/ from the cpplint exclude list and fix every finding: C-style casts become static_cast / reinterpret_cast, long long and unsigned long long become int64_t / uint64_t, non-const reference out-parameters become pointers (call sites updated), and C system headers come before C++ ones. atomicAdd_system keeps its unsigned long long prototype through one aliased type. No change to values, types seen by Python, or arithmetic. Co-Authored-By: Claude Code <noreply@anthropic.com>
…e slot as an out-parameter C++: both return 0 or a cudaError_t code and write the slot to *slot: cudaErrorNotReady without configure / release_setup, the error of paddle_compute_stream or cudaEventRecord, and cudaErrorLaunchOutOfResources when this step's tail mark slots are used up (was -1 / -2). The Python binding still returns the slot and raises RuntimeError with the error text on failure. CPU test of the binding before setup. Co-Authored-By: Claude Code <noreply@anthropic.com>
…rd to it Fourth piece of the move from PaddleFleet's extension: the code that decides, for each bucket and stage, whether it may be handed to the GPU now, and then calls the submit_* functions of the previous commit. With this the C++ side is complete; it is still not reachable from Python (next commit). kernels/release.cu: the release decision lives here exactly once and is shared by two releasers that never run at the same time (both take the control mutex; advance() only releases in IDLE, the tail thread only in RUN), so there is one writer of the per-step state at any time: * the main thread, which calls advance() in every window where DeepEP uses neither the network nor the SMs, with the set of stages the caller picked for that window, and returns right away; * the tail thread, which takes over after the last combine of the step (tail_start) and loops over the same release_next_* until every bucket is out; the main thread then only reports buckets that became ready, says when all are posted, waits on a condition variable and finishes. Readiness is read from the host window (credits, intra-node arrivals, GIN signals), never waited on from the GPU. Also here: the tail mark ring, status / stuck_report / last error for diagnostics, the Muon 2D gather table and its entry point, and the constants table (ADVANCE_* / STAGE_* / COUNTER_* / TAIL_*), whose values are defined only here so the Python side reads them by name. The file is .cu because it sits with the kernels, but it is host code (no kernel is defined in it). gin_engine.h / gin_engine.cc: the release:: declarations and the thin GradCommEngine wrappers the Python layer calls (release_setup, release_step_reset, advance, bucket_ready, release_status, release_stuck_report, release_error, release_activity, tail_*, tail_gather_*, gather_put_entry). They only check arguments and forward. kernels/release.cu is appended to GIN_OVERLAP_SRCS. Moved as is from the extension, like the previous commits. Co-Authored-By: Claude Code <noreply@anthropic.com>
Translate every comment, docstring, error message and log line of the gin_overlap sources (C++, CUDA, CMake, pybind and the Python layer) into English, and tighten the wording on the way. The terms are defined once, in the gin_engine.h file header (C++) and in the engine.py module docstring (Python); every other file uses them as defined there. Strings only. comment_clean_check.py --ignore-strings passes on every changed file: C++ / CUDA / CMake token streams are identical except for string literals (printf conversions inside merged literals keep their order), and the Python ASTs are identical except for str constants and the literal parts of f-strings. Log lines all carry the [gin_overlap] prefix; copy_2d errors now start with "copy_2d:". Co-Authored-By: Claude Code <noreply@anthropic.com>
…ne API
Argument checks (values come from Python):
- scale_range, gather_put_to_owner, send_window_tensor and copy_2d check
their bounds without unsigned wrap-around, and reject numel < 0.
copy_2d also rejects non-contiguous tensors (a stride-0 or transposed
view passes a numel-based bound but the copy reads contiguous bytes).
- release::setup checks every offset of the per-bucket tables and the
host window areas: non-negative, aligned (counters 8 B, data 4 B) and
inside its window. setup_gather_table checks both arrival counter
offsets of each entry the same way. CommGroup records recv_bytes for
this, and alloc_symmetric_window returns the size it actually
registered instead of the caller repeating the 4096 B minimum.
- A wrong state returns cudaErrorNotReady and a bad argument
cudaErrorInvalidValue (gather_setup, gather_put_to_owner, scale_range).
- bucket_ready rejects a bucket reported twice in a step in the RUN state
too, from a bitmap only the main thread touches.
Init failures:
- configure destroys the streams and events it created, frees its
temporary buffer and resets the globals it set when a later step fails,
so it can be retried. set_comms clears the groups it set on failure.
release::setup unpublishes its control block if the tail thread cannot
be started. The steps that cannot be rolled back (window registration,
devcomm creation, the host window) say so: restart the process.
- The gin_engine.h constraints state that the engine lives as long as
the process; shutdown_process_group / restart_process_group are not
supported.
API:
- GIN_CHECK replaces the dozen { rc = launch_x(...); if (rc) return rc; }
blocks. launch_* / preload_* return cudaError_t.
- launch_put_to_expert_peers takes (send_offset, recv_offset) like
launch_put_to_node_peers.
- join_bucket_streams / gather_wait_bucket take kind (0 dense, 1 expert),
as bucket_ready and the gather table do, instead of is_dense.
- The six release positions are named variables, one per stage.
- The methods that do not return a code (record_window_mark, tail_mark,
bucket_ready, tail_all_posted, tail_wait, tail_abort, release_activity,
the version getters) document what they return.
- tail_wait / tail_abort release the GIL while they block.
No kernel, reduction order, release condition or credit target changes.
Co-Authored-By: Claude Code <noreply@anthropic.com>
kernels/release.cu has no kernel and no <<<>>>: it is the host-side release state machine and its std::thread. kernels/ holds only the kernel launch wrappers, so the file moves next to gin_engine.cc as release.cc and is compiled by the host compiler instead of nvcc. The content is unchanged apart from the file name in comments. Co-Authored-By: Claude Code <noreply@anthropic.com>
…fsets in release setup Co-Authored-By: Claude Code <noreply@anthropic.com>
Remove gin_overlap/ from the cpplint exclude list and fix every finding: C-style casts become static_cast / reinterpret_cast, long long and unsigned long long become int64_t / uint64_t, non-const reference out-parameters become pointers (call sites updated), and C system headers come before C++ ones. atomicAdd_system keeps its unsigned long long prototype through one aliased type. No change to values, types seen by Python, or arithmetic. Co-Authored-By: Claude Code <noreply@anthropic.com>
…e slot as an out-parameter C++: both return 0 or a cudaError_t code and write the slot to *slot: cudaErrorNotReady without configure / release_setup, the error of paddle_compute_stream or cudaEventRecord, and cudaErrorLaunchOutOfResources when this step's tail mark slots are used up (was -1 / -2). The Python binding still returns the slot and raises RuntimeError with the error text on failure. CPU test of the binding before setup. Co-Authored-By: Claude Code <noreply@anthropic.com>
c10b645 to
00624af
Compare
PR Category
Distributed Strategy
PR Types
New features
Description
[WIP] gin_overlap 2/3: the engine (setup, per-stage submits, 2D-bucket gather) and the release layer of the NCCL-GIN primitive (#79964). Depends on #79967. The branch contains only this PR's changes on top of develop (
63c4ef64e6), so Files changed shows only them, with one exception:gin_overlap/CMakeLists.txtis created by #79967 and extended here, so this PR carries the whole final file (about 23 of its lines are #79967's). It does not build until #79967 is merged; it will then be rebased onto develop. CI fails until then, which is expected. #79971 depends on this PR. It replaces the earlier #79968 and #79969, which are closed; their descriptions are merged below.Part 3 (formerly #79968): engine setup (comms, windows, devcomm, side streams)
What this part adds
The host side that builds everything the per-step code runs on. Nothing is submitted to the GPU
per step yet (Part 4).
gin_engine.h: the file header is the one full description of the C++ side: terms (bucket,shard, GIN, LSA, devcomm, CE, window, mark, release, credit, owner, tail thread), the two groups,
the four dense stages / two expert stages / Muon 2D gather, the receive pools, the host-window
layout, and the constraints every later file relies on. It is added here in full. It is
organised by concept, not by PR, and the parts about the release layer (Part 5) are single
bullets or single sentences inside it (e.g. "the release layer reads remote counters on the
CPU", "only one writer of the release state: advance or the tail thread"); cutting those out
would mean rewriting the description, so it is not split.
gin_engine.h/gin_engine.cc, initialization only:CommGroupstate of the two groups(sharding / moe_sharding), the
CUDA_CHECK/NCCL_CHECK/CUDA_DRIVER_CHECKmacros, and theGradCommEnginemethods called once at startup, in this order:set_comms(reuse Paddle'sncclComm_tper ring id, cross-check size / rank),alloc_windows(send / recv / signal asNCCL_WIN_COLL_SYMMETRIC),alloc_host_counters(one HOST_NUMA host window per group,registered with
NCCL_WIN_STRICT_ORDERING),create_dev_comm(ncclDevCommCreate+ LSApointers),
set_group_topology,configure(side streams, per-bucket events, the window-markring, credit offsets, host-window LSA pointers, preloading every kernel module). Also
error_stringand the NCCL runtime / build version queries.paddle_comm.cc: thencclComm_tof Paddle'sProcessGroupby ring id, the way DeepEP gets it(
ProcessGroupMapFromGid→ProcessGroupNCCL→NCCLCommContext::GetNcclComm).paddle_tensor.cc/.h: Paddle's current compute stream, the zero-copypaddle::Tensorover asend-window slice (
paddle::from_blob,deleter = nullptr, so the NCCL window keeps owning thememory), and the pitched
copy_2dthroughcuMemcpy2DAsync.gin_overlap/CMakeLists.txt: the three.ccfiles joinGIN_OVERLAP_SRCS;DEPSgainsprocess_group_nccl(forpaddle_comm.cc).Comments in this part mention methods and files that arrive later (
release_setup, the pybind file,the Python layer); they are kept as they are in #79964.
Callers after merge: none. The methods are called from Python, which is bound in #79971.
What CI can show: on a build with the gate on (see #79967 (Part 2)), that these files compile against
Paddle's headers and that the declarations in
gin_engine.hmatch the definitions (a staticcheck of declarations, includes and
GIN_OVERLAP_SRCSpasses for this branch). No test callsthe engine.
How correctness is established: moved from PaddleFleet's extension, same checks and same order
of collective calls;
paddle_comm.cu/paddle_tensor.cubecame.cc(in-tree they need nonvcc). The bitwise evidence so far is for earlier, out-of-tree states of the engine (4 nodes /
32 cards, 10 steps, identical to Paddle's
reduce_scatterbaseline; see #79967 (Part 2)), not for this tree.The acceptance gate is the same 10-step AdamW and Muon run on a build of the whole series.
The whole series has been compiled (see #79967 (Part 2)); nothing has been run on a GPU yet.
Changes after the first review
copy_2drejects non-contiguous tensors and checks its bounds without unsigned wrap-around; its errors start withcopy_2d:.send_window_tensorchecks its bounds the same way.configuredestroys the streams and events it created, frees its temporary buffer and resets the globals it set, so it can be retried. A failedset_commsclears the groups it set. The steps that cannot be rolled back (alloc_windows,alloc_host_counters,create_dev_comm: registration and devcomm creation are collective) say in their comments that a failure needs a process restart.gin_engine.hheader and the Python docstring state that the engine lives as long as the process and thatshutdown_process_group/restart_process_groupare not supported (using the engine after them is undefined). No Python-side interception was added:restart_process_grouprebuilds the comm inside the same group object (same.id), so a cheap Python check cannot see it, and a reliable one needs a hook inProcessGroupNCCL.alloc_symmetric_windowreturns the size it registered;CommGrouprecordsrecv_bytestoo.GIN_CHECKreplaces the{ rc = launch_x(...); if (rc) return rc; }blocks. The methods that do not return a code in the usual convention say what they return.CUresultinCUDA_DRIVER_CHECK, and one name formoe/expert(renames across the whole series). A single-cardcopy_2dtest needs a GPU; not added.Changes after the second review
int64_t/uint64_t, pointer out-parameters, include order); details in [WIP][Distributed] gin_overlap 1/3: dynload wrappers, device kernels and the build gate #79967 (Part 2).record_window_mark/tail_markamong the methods that return a slot (they return an error code now, see Part 4 / Part 5).Part 4 (formerly #79969): per-stage submit functions and the 2D-bucket gather
What this part adds (
gin_engine.h/gin_engine.cconly)The host functions that hand one stage of one bucket to its side stream, and the stream / event
plumbing around them.
submit_dense_intra_node_copy(cudaMemcpyBatchAsyncof the column to theother ranks on the node + the arrival write),
submit_dense_intra_node_reduce(+ intra-nodecredit),
submit_dense_inter_node_put,submit_moe_put,submit_dense_inter_node_reduceandsubmit_moe_reduce(+ GIN credit). Each waits only on work this rank has already handed out (acompute-stream mark or a local side-stream event), never on a remote counter.
record_window_mark: records a mark on Paddle's current compute stream into the ringconfigure()created;GIN_COMPUTE_STREAM, the macro that fetches that stream.join_bucket_streams(compute stream waits for a bucket's side-stream events),scale_range,end_step(GIN flush of both groups so the send window can be reused),stats.gather_setup,gather_gate,gather_wait_bucket,gather_mark_ready,gather_put_to_owner(GIN put across nodes, CE + release store within anode),
gather_join. The table-driven entry pointgather_put_entryforwards to the releaselayer and comes with it in Part 5.
Callers after merge: none. The only caller of
submit_*is the release layer (Part 5); theGradCommEnginemethods are called from Python (#79971).What CI can show: on a build with the gate on (see #79967 (Part 2)), that the two files compile. No test
calls them.
How correctness is established: moved from PaddleFleet's extension unchanged. The bitwise
evidence so far is for earlier, out-of-tree states of the engine (4 nodes / 32 cards, 10 steps,
identical to Paddle's
reduce_scatterbaseline; see #79967 (Part 2)), not for this tree. The acceptancegate is the same 10-step AdamW and Muon run on a build of the whole series. The whole
series has been compiled (see #79967 (Part 2)); nothing has been run on a GPU yet.
Changes after the first review
attrs_idx.scale_rangeandgather_put_to_ownercheck their bounds without wrap-around and rejectnumel < 0.cudaErrorNotReadyand a bad argumentcudaErrorInvalidValue(gather_setup,gather_put_to_owner,scale_range).join_bucket_streams/gather_wait_buckettakekind(0 dense, 1 expert), asbucket_readyand the gather table do, instead ofis_dense.record_window_marknow returns an error code.)record_window_markkeeps returning the slot; its declaration now says so (an out-parameter would change the Python binding for no behavior change).reduce_scatter). They need GPUs and come with the acceptance run.Changes after the second review
record_window_marknow returns an error code.)record_window_mark's declaration ingin_engine.hlists its return values: the window mark slot (>= 0), or -1 whenconfigurehas not run, the compute stream cannot be read, orcudaEventRecordfailed (same style asbucket_ready/tail_mark). Comment only, no behavior change.int64_t/uint64_t, pointer out-parameters, include order); details in [WIP][Distributed] gin_overlap 1/3: dynload wrappers, device kernels and the build gate #79967 (Part 2).record_window_mark(int* slot)returns an error code and writes the slot to*slot(was: the slot, or -1 for every failure):cudaErrorNotReadywithoutconfigure, the error ofpaddle_compute_stream, or thecudaError_tofcudaEventRecord. The declaration says so.Part 5 (formerly #79970): the release layer
What this part adds
release.cc: the code that decides, for each bucket and stage, whether it may be handed tothe GPU now, and then calls the
submit_*functions of Part 4. The decision lives here exactly onceand is shared by two releasers that never run at the same time (both take the control mutex;
advance()releases only in IDLE, the tail thread only in RUN):advance()in each window where DeepEP uses neither the network nor theSMs, with the set of stages the caller picked for that window, and returns immediately;
tail_start) and loops over thesame
release_next_*until every bucket is out; the main thread then only reports buckets thatbecame ready, says when all are posted, waits on a condition variable and finishes.
Readiness is read from the host window (credits, intra-node arrivals, GIN signals) on the CPU,
never waited on from the GPU. Also in this file: the tail mark ring,
status/stuck_report/last error for diagnostics, the Muon 2D gather table and its entry point, and the constants table
(
ADVANCE_*/STAGE_*/COUNTER_*/TAIL_*), whose values are defined only here so thePython side reads them by name. The file is
.cubecause it sits with the kernels, but it ishost code.
gin_engine.h/gin_engine.cc: therelease::declarations and the thinGradCommEnginewrappers the Python layer calls (
release_setup,release_step_reset,advance,bucket_ready,release_status,release_stuck_report,release_error,release_activity,tail_*,tail_gather_*,gather_put_entry), plusis_valid_mark(onlyadvanceuses it). They checkarguments and forward. The class comment's "call order" paragraph comes with them.
gin_overlap/CMakeLists.txt:release.ccjoinsGIN_OVERLAP_SRCS.With this the C++ side is complete; it is still not reachable from Python (#79971).
Callers after merge: none. The
GradCommEnginewrappers are called from Python (#79971, and thePaddleFleet scheduler on top of it).
What CI can show: on a build with the gate on (see #79967 (Part 2)), that
release.cccompiles and thelibrary links with every declared
release::function defined. No test runs the release layer;its threading (mutex / condition variable / tail thread) is only exercised by a real multi-node
run.
How correctness is established: moved from PaddleFleet, where the release decision had just
been moved into C++ (one copy shared by the main and the tail thread). That C++ release layer
has not been through the bitwise run yet: the bitwise evidence so far (4 nodes / 32 cards,
10 steps, identical to Paddle's
reduce_scatterbaseline; see #79967 (Part 2)) predates it. The acceptancegate is the 10-step AdamW and Muon run on a build of the whole series. The whole series has been
compiled (see #79967 (Part 2)); nothing has been run on a GPU yet.
Changes after the first review
release::setupchecks every offset in the per-bucket tables and the host-window areas: non-negative, aligned (counters 8 B, data 4 B), inside the window (send_bytes/recv_bytes/host_bytes).setup_gather_tablechecks both arrival-counter offsets of each entry.step_resetclears it).kernels/release.cutogin_overlap/release.cc(it has no kernel).controlis unpublished again when the tail thread cannot be started, sosetupcan be retried.bucket_ready/tail_*/tail_mark/release_activityare documented next to their declarations.Changes after the second review
release::setup(tables_in_bounds) requires the densesend_offset(row[0]) andrecv_offset_intra(row[3]) to be 16-byte aligned and returnscudaErrorInvalidValueotherwise:submit_dense_intra_node_reducepasses both tok_reduce_intra_nodeasfloat4*. The other columns keep their 4-byte (data) / 8-byte (counter) checks. The Python layout already produces these offsets aligned (send ranges and recv slots are 256 B aligned), so a valid table is accepted as before.int64_t/uint64_t, pointer out-parameters, include order); details in [WIP][Distributed] gin_overlap 1/3: dynload wrappers, device kernels and the build gate #79967 (Part 2).tail_mark(int* slot)/release::record_tail_mark(int* slot)return an error code and write the slot to*slot(was: -1 / -2):cudaErrorNotReadywithoutrelease_setup,cudaErrorLaunchOutOfResourceswhen this step's tail mark slots are used up, or the error ofpaddle_compute_stream/cudaEventRecord. The setup checks run before the compute stream is read.Not verified: the precision acceptance run (4 nodes, bitwise) is still in progress, and the changes of this round have not been through it. The behavior changes of this round are the argument check in
release::setupand the error codes ofrecord_window_mark/tail_mark(Python still gets the slot); the cpplint fixes change casts, integer type spellings and reference -> pointer parameters only.是否引起精度变化
否
🤖 Generated with Claude Code