Skip to content

[WIP][Distributed] gin_overlap 2/3: engine and release layer (C++) - #79970

Open
AlAuAu wants to merge 21 commits into
PaddlePaddle:developfrom
AlAuAu:gin_split_5_release
Open

AlAuAu wants to merge 21 commits into
PaddlePaddle:developfrom
AlAuAu:gin_split_5_release

Conversation

@AlAuAu

@AlAuAu AlAuAu commented Nov 20, 2026 •

Copy link
Copy Markdown
Contributor

PR Category

Distributed Strategy

PR Types

New features

Description

[WIP] gin_overlap 2/3: the engine (setup, per-stage submits, 2D-bucket gather) and the release layer of the NCCL-GIN primitive (#79964). Depends on #79967. The branch contains only this PR's changes on top of develop (63c4ef64e6), so Files changed shows only them, with one exception: gin_overlap/CMakeLists.txt is created by #79967 and extended here, so this PR carries the whole final file (about 23 of its lines are #79967's). It does not build until #79967 is merged; it will then be rebased onto develop. CI fails until then, which is expected. #79971 depends on this PR. It replaces the earlier #79968 and #79969, which are closed; their descriptions are merged below.

Part 3 (formerly #79968): engine setup (comms, windows, devcomm, side streams)

What this part adds

The host side that builds everything the per-step code runs on. Nothing is submitted to the GPU
per step yet (Part 4).

  • gin_engine.h: the file header is the one full description of the C++ side: terms (bucket,
    shard, GIN, LSA, devcomm, CE, window, mark, release, credit, owner, tail thread), the two groups,
    the four dense stages / two expert stages / Muon 2D gather, the receive pools, the host-window
    layout, and the constraints every later file relies on. It is added here in full. It is
    organised by concept, not by PR, and the parts about the release layer (Part 5) are single
    bullets or single sentences inside it (e.g. "the release layer reads remote counters on the
    CPU", "only one writer of the release state: advance or the tail thread"); cutting those out
    would mean rewriting the description, so it is not split.
  • gin_engine.h / gin_engine.cc, initialization only: CommGroup state of the two groups
    (sharding / moe_sharding), the CUDA_CHECK / NCCL_CHECK / CUDA_DRIVER_CHECK macros, and the
    GradCommEngine methods called once at startup, in this order: set_comms (reuse Paddle's
    ncclComm_t per ring id, cross-check size / rank), alloc_windows (send / recv / signal as
    NCCL_WIN_COLL_SYMMETRIC), alloc_host_counters (one HOST_NUMA host window per group,
    registered with NCCL_WIN_STRICT_ORDERING), create_dev_comm (ncclDevCommCreate + LSA
    pointers), set_group_topology, configure (side streams, per-bucket events, the window-mark
    ring, credit offsets, host-window LSA pointers, preloading every kernel module). Also
    error_string and the NCCL runtime / build version queries.
  • paddle_comm.cc: the ncclComm_t of Paddle's ProcessGroup by ring id, the way DeepEP gets it
    (ProcessGroupMapFromGid → ProcessGroupNCCL → NCCLCommContext::GetNcclComm).
  • paddle_tensor.cc / .h: Paddle's current compute stream, the zero-copy paddle::Tensor over a
    send-window slice (paddle::from_blob, deleter = nullptr, so the NCCL window keeps owning the
    memory), and the pitched copy_2d through cuMemcpy2DAsync.
  • gin_overlap/CMakeLists.txt: the three .cc files join GIN_OVERLAP_SRCS; DEPS gains
    process_group_nccl (for paddle_comm.cc).

Comments in this part mention methods and files that arrive later (release_setup, the pybind file,
the Python layer); they are kept as they are in #79964.

Callers after merge: none. The methods are called from Python, which is bound in #79971.

What CI can show: on a build with the gate on (see #79967 (Part 2)), that these files compile against
Paddle's headers and that the declarations in gin_engine.h match the definitions (a static
check of declarations, includes and GIN_OVERLAP_SRCS passes for this branch). No test calls
the engine.

How correctness is established: moved from PaddleFleet's extension, same checks and same order
of collective calls; paddle_comm.cu / paddle_tensor.cu became .cc (in-tree they need no
nvcc). The bitwise evidence so far is for earlier, out-of-tree states of the engine (4 nodes /
32 cards, 10 steps, identical to Paddle's reduce_scatter baseline; see #79967 (Part 2)), not for this tree.
The acceptance gate is the same 10-step AdamW and Muon run on a build of the whole series.
The whole series has been compiled (see #79967 (Part 2)); nothing has been run on a GPU yet.

Changes after the first review

  • copy_2d rejects non-contiguous tensors and checks its bounds without unsigned wrap-around; its errors start with copy_2d:. send_window_tensor checks its bounds the same way.
  • A failed configure destroys the streams and events it created, frees its temporary buffer and resets the globals it set, so it can be retried. A failed set_comms clears the groups it set. The steps that cannot be rolled back (alloc_windows, alloc_host_counters, create_dev_comm: registration and devcomm creation are collective) say in their comments that a failure needs a process restart.
  • Window release (P2): not implemented. The constraints in the gin_engine.h header and the Python docstring state that the engine lives as long as the process and that shutdown_process_group / restart_process_group are not supported (using the engine after them is undefined). No Python-side interception was added: restart_process_group rebuilds the comm inside the same group object (same .id), so a cheap Python check cannot see it, and a reliable one needs a hook in ProcessGroupNCCL.
  • alloc_symmetric_window returns the size it registered; CommGroup records recv_bytes too.
  • GIN_CHECK replaces the { rc = launch_x(...); if (rc) return rc; } blocks. The methods that do not return a code in the usual convention say what they return.
  • Deferred: separate error codes for CUresult in CUDA_DRIVER_CHECK, and one name for moe / expert (renames across the whole series). A single-card copy_2d test needs a GPU; not added.

Changes after the second review

Part 4 (formerly #79969): per-stage submit functions and the 2D-bucket gather

What this part adds (gin_engine.h / gin_engine.cc only)

The host functions that hand one stage of one bucket to its side stream, and the stream / event
plumbing around them.

  • The six stage submits: submit_dense_intra_node_copy (cudaMemcpyBatchAsync of the column to the
    other ranks on the node + the arrival write), submit_dense_intra_node_reduce (+ intra-node
    credit), submit_dense_inter_node_put, submit_moe_put, submit_dense_inter_node_reduce and
    submit_moe_reduce (+ GIN credit). Each waits only on work this rank has already handed out (a
    compute-stream mark or a local side-stream event), never on a remote counter.
  • record_window_mark: records a mark on Paddle's current compute stream into the ring
    configure() created; GIN_COMPUTE_STREAM, the macro that fetches that stream.
  • join_bucket_streams (compute stream waits for a bucket's side-stream events), scale_range,
    end_step (GIN flush of both groups so the send window can be reused), stats.
  • Muon 2D-bucket gather to the owner: gather_setup, gather_gate, gather_wait_bucket,
    gather_mark_ready, gather_put_to_owner (GIN put across nodes, CE + release store within a
    node), gather_join. The table-driven entry point gather_put_entry forwards to the release
    layer and comes with it in Part 5.

Callers after merge: none. The only caller of submit_* is the release layer (Part 5); the
GradCommEngine methods are called from Python (#79971).

What CI can show: on a build with the gate on (see #79967 (Part 2)), that the two files compile. No test
calls them.

How correctness is established: moved from PaddleFleet's extension unchanged. The bitwise
evidence so far is for earlier, out-of-tree states of the engine (4 nodes / 32 cards, 10 steps,
identical to Paddle's reduce_scatter baseline; see #79967 (Part 2)), not for this tree. The acceptance
gate is the same 10-step AdamW and Muon run on a build of the whole series. The whole
series has been compiled (see #79967 (Part 2)); nothing has been run on a GPU yet.

Changes after the first review

  • P1: see [WIP][Distributed] gin_overlap 1/3: dynload wrappers, device kernels and the build gate #79967 (Part 2) (the gate now requires CUDA 13.0); the batch attribute index is named attrs_idx.
  • P2: scale_range and gather_put_to_owner check their bounds without wrap-around and reject numel < 0.
  • A wrong state returns cudaErrorNotReady and a bad argument cudaErrorInvalidValue (gather_setup, gather_put_to_owner, scale_range).
  • join_bucket_streams / gather_wait_bucket take kind (0 dense, 1 expert), as bucket_ready and the gather table do, instead of is_dense.
  • (Superseded below: record_window_mark now returns an error code.) record_window_mark keeps returning the slot; its declaration now says so (an out-parameter would change the Python binding for no behavior change).
  • Deferred: the multi-process tests (single node / multi node / owner on the same or another node, bitwise against reduce_scatter). They need GPUs and come with the acceptance run.

Changes after the second review

  • (Superseded below: record_window_mark now returns an error code.) record_window_mark's declaration in gin_engine.h lists its return values: the window mark slot (>= 0), or -1 when configure has not run, the compute stream cannot be read, or cudaEventRecord failed (same style as bucket_ready / tail_mark). Comment only, no behavior change.
  • cpplint fixes in this layer's files (casts, int64_t / uint64_t, pointer out-parameters, include order); details in [WIP][Distributed] gin_overlap 1/3: dynload wrappers, device kernels and the build gate #79967 (Part 2).
  • record_window_mark(int* slot) returns an error code and writes the slot to *slot (was: the slot, or -1 for every failure): cudaErrorNotReady without configure, the error of paddle_compute_stream, or the cudaError_t of cudaEventRecord. The declaration says so.

Part 5 (formerly #79970): the release layer

What this part adds

release.cc: the code that decides, for each bucket and stage, whether it may be handed to
the GPU now, and then calls the submit_* functions of Part 4. The decision lives here exactly once
and is shared by two releasers that never run at the same time (both take the control mutex;
advance() releases only in IDLE, the tail thread only in RUN):

  • the main thread calls advance() in each window where DeepEP uses neither the network nor the
    SMs, with the set of stages the caller picked for that window, and returns immediately;
  • the tail thread takes over after the last combine of the step (tail_start) and loops over the
    same release_next_* until every bucket is out; the main thread then only reports buckets that
    became ready, says when all are posted, waits on a condition variable and finishes.

Readiness is read from the host window (credits, intra-node arrivals, GIN signals) on the CPU,
never waited on from the GPU. Also in this file: the tail mark ring, status / stuck_report /
last error for diagnostics, the Muon 2D gather table and its entry point, and the constants table
(ADVANCE_* / STAGE_* / COUNTER_* / TAIL_*), whose values are defined only here so the
Python side reads them by name. The file is .cu because it sits with the kernels, but it is
host code.

gin_engine.h / gin_engine.cc: the release:: declarations and the thin GradCommEngine
wrappers the Python layer calls (release_setup, release_step_reset, advance, bucket_ready,
release_status, release_stuck_report, release_error, release_activity, tail_*,
tail_gather_*, gather_put_entry), plus is_valid_mark (only advance uses it). They check
arguments and forward. The class comment's "call order" paragraph comes with them.

gin_overlap/CMakeLists.txt: release.cc joins GIN_OVERLAP_SRCS.

With this the C++ side is complete; it is still not reachable from Python (#79971).

Callers after merge: none. The GradCommEngine wrappers are called from Python (#79971, and the
PaddleFleet scheduler on top of it).

What CI can show: on a build with the gate on (see #79967 (Part 2)), that release.cc compiles and the
library links with every declared release:: function defined. No test runs the release layer;
its threading (mutex / condition variable / tail thread) is only exercised by a real multi-node
run.

How correctness is established: moved from PaddleFleet, where the release decision had just
been moved into C++ (one copy shared by the main and the tail thread). That C++ release layer
has not been through the bitwise run yet: the bitwise evidence so far (4 nodes / 32 cards,
10 steps, identical to Paddle's reduce_scatter baseline; see #79967 (Part 2)) predates it. The acceptance
gate is the 10-step AdamW and Muon run on a build of the whole series. The whole series has been
compiled (see #79967 (Part 2)); nothing has been run on a GPU yet.

Changes after the first review

  • P2: release::setup checks every offset in the per-bucket tables and the host-window areas: non-negative, aligned (counters 8 B, data 4 B), inside the window (send_bytes / recv_bytes / host_bytes). setup_gather_table checks both arrival-counter offsets of each entry.
  • P3: a bucket reported twice in a step is rejected in RUN too, from a bitmap only the main thread touches (step_reset clears it).
  • The release layer moved from kernels/release.cu to gin_overlap/release.cc (it has no kernel).
  • The six release positions are named variables, one per stage.
  • control is unpublished again when the tail thread cannot be started, so setup can be retried.
  • The return values of bucket_ready / tail_* / tail_mark / release_activity are documented next to their declarations.
  • Deferred: host-only unit tests of the release layer (credit targets, stage order, state checks) with a fake host window; they need the submit functions to be stubbed out, which is a refactor.

Changes after the second review

  • P3: release::setup (tables_in_bounds) requires the dense send_offset (row[0]) and recv_offset_intra (row[3]) to be 16-byte aligned and returns cudaErrorInvalidValue otherwise: submit_dense_intra_node_reduce passes both to k_reduce_intra_node as float4*. The other columns keep their 4-byte (data) / 8-byte (counter) checks. The Python layout already produces these offsets aligned (send ranges and recv slots are 256 B aligned), so a valid table is accepted as before.
  • cpplint fixes in this layer's files (casts, int64_t / uint64_t, pointer out-parameters, include order); details in [WIP][Distributed] gin_overlap 1/3: dynload wrappers, device kernels and the build gate #79967 (Part 2).
  • tail_mark(int* slot) / release::record_tail_mark(int* slot) return an error code and write the slot to *slot (was: -1 / -2): cudaErrorNotReady without release_setup, cudaErrorLaunchOutOfResources when this step's tail mark slots are used up, or the error of paddle_compute_stream / cudaEventRecord. The setup checks run before the compute stream is read.

Not verified: the precision acceptance run (4 nodes, bitwise) is still in progress, and the changes of this round have not been through it. The behavior changes of this round are the argument check in release::setup and the error codes of record_window_mark / tail_mark (Python still gets the slot); the cpplint fixes change casts, integer type spellings and reference -> pointer parameters only.

是否引起精度变化

否

🤖 Generated with Claude Code

@Paddle-Review-Bot

Paddle-Review-Bot commented Nov 20, 2026 •

Copy link
Copy Markdown
Contributor

Paddle-Bot Review Board (review完成)

序号 位置 优先级 规则来源 状态
1 放行层 setup 参数表偏移校验 P2 默认规则 ✅
2 尾部运行期重复 bucket_ready 未拒绝 P3 默认规则 ✅
3 dense 数据偏移只按 4 字节对齐校验 P3 默认规则 ✅

可维护性建议(总分 84.7 / 100,只给建议,不影响评审结论)

序号 位置 维度 状态
1 引擎返回码约定的例外 契约一致性 ✅
2 放行层源文件位置 包组织结构 ✅
3 阶段进度数组下标 底层优雅度 ✅
4 尾部线程启动失败后的状态 初始化耦合 ✅
5 放行判断的单测覆盖 测试策略 🟡
各维度得分(仅评本 PR 改动,点击展开)
65 测试策略
5
85 初始化耦合
4
85 未完成迁移
85 包组织结构
2
85 跨模块架构
85 依赖健康度
85 AI 生成债务
85 高层优雅度
85 抽象适配度
85 API 表面一致性
85 约定偏离
85 底层优雅度
3
85 类型安全
86 设计一致性
86 中层优雅度
87 契约一致性
1
88 错误处理一致性
88 命名质量
89 逻辑清晰度
— 鉴权一致性

Powered by Nyanpasu claude with Opus 5.5 默认推理级别, please check the suggestions carefully.

Comment on lines +726 to +736
dense_buckets.resize(num_dense_buckets);
for (int i = 0; i < num_dense_buckets; ++i) {
const long long* row = dense_table.data() + 7 * (size_t)i;
dense_buckets[i] =
DenseBucket{row[0], row[1], row[2], row[3], row[4], row[5], row[6]};
}
moe_buckets.resize(num_moe_buckets);
for (int i = 0; i < num_moe_buckets; ++i) {
const long long* row = moe_table.data() + 5 * (size_t)i;
moe_buckets[i] = MoeBucket{row[0], row[1], row[2], row[3], row[4]};
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 setup 不校验每桶参数表里的偏移,但放行判断会在 CPU 上直接按这些偏移读 host window。

  • blocked_dense_inter_node_reduce / blocked_moe_reduce 读 host + signal_offset/8*8,blocked_intra_node_reduce 读 intra_arrive_offset/8 + bucket*16 + l,全程不和 host_bytes 比较。Python 侧的布局一旦算错,结果是训练进程里 CPU 越界读(段错误),拿不到错误码。
  • 偏移不是 8 字节对齐时,/8 会截断到相邻的 word,而 GPU 的 signal 写在原偏移上。这个桶的分片求和会永远等不到目标值,症状是卡住,原因难查。
  • setup_gather_table 校验了 send_offset >= 0 和 shard_numel > 0,没校验 row[8] / row[9] 非负;负数转成 size_t 后就是 GPU 写 owner host window 时用的巨大偏移。

建议在 setup 里对 signal_offset、intra_arrive_offset(含 bucket*16+ranks_per_node 那一段)、各 recv_offset_* / send_offset 校验非负、对齐(signal/arrive 8 字节,数据 4 字节),并检查是否落在对应 window 内(host_bytes / send_bytes);不满足时返回 cudaErrorInvalidValue。setup_gather_table 也补上 row[8]、row[9] 的同类检查。

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

已在当前提交修复:tables_in_bounds 对每桶的 send/recv/signal 偏移、intra_arrive 区和三个 credit 区做了非负、对齐(计数 8 B,数据 4 B)和窗口范围检查,setup_gather_table 也补上了 row[8] / row[9] 的检查。

Comment on lines +876 to +882
if (mark_ < 0 || mark_ >= tail_mark_index) return 2;
if (control->state == DONE)
return kind < 0 ? 0 : 2; // 已全部放完:只换 mark 的消息直接丢弃
if (control->state != RUN) return 2;
control->queue.push_back(ReadyMessage{kind, bucket, mark_, scale});
control->cv_proxy.notify_all();
return 0;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P3 函数注释说「这个桶本步已经 ready 过」时返回 2,但只有 IDLE 分支检查了这一点。RUN 分支把重复的 bucket_ready 照收进队列,run_step 会再把 ready_* 置 1,并用新的 scale 覆盖旧值。如果这个桶的分片求和还没放行,调用方重复上报的错误就会变成静默改系数。

ready_* 在 RUN 期间归尾部线程所有,不能直接读。可以加一份只由主线程维护的「本步已上报」位图(step_reset 清零,IDLE / RUN 两个分支都查它),也可以把注释改成只在 IDLE 下检查重复。

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

已在当前提交修复:新增只由主线程维护的 reported_* 位图,IDLE 和 RUN 两个分支都拒绝重复上报,step_reset 负责清零。

Comment on lines +457 to +472
int bucket_ready(int kind, int bucket, float scale, int mark);
int release_status(std::vector<int>& positions,
std::vector<int>& counts,
std::vector<int>& posted,
int* done);
int release_stuck_report(std::vector<long long>& flat);
int release_error(int* code, int* stage, int* bucket);
int release_activity();
int tail_mark();
int tail_start(int intra_node_reduce_blocks,
int shard_reduce_blocks,
int mark,
int all_posted);
int tail_all_posted();
int tail_wait(double timeout_s);
int tail_abort(double wait_s);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 可维护性建议(契约一致性,不阻塞合入):类注释的「约定」写着返回 int 返回码(0 成功,<10000 = cudaError_t,>=10000 = 10000 + ncclResult_t),但本 PR 新增的几个方法不走这套编码:bucket_ready / tail_all_posted 返回 0/1/2;tail_wait / tail_abort 返回 TAIL_* 状态,其中 TAIL_RUN = 1 与 cudaErrorInvalidValue 数值相同;tail_mark 返回槽号或 -1/-2。调用方套用通用的返回码检查就会误判。建议在「约定」里逐条列出这些例外及其取值,或改成「返回码 + 出参」的形式(同 release_error)。

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

已处理:bucket_ready、tail_all_posted、tail_mark、tail_wait、tail_abort、release_activity 的返回值都在声明旁写明了,类注释也说明了例外方法看各自的声明。

kernels/put.cu
kernels/signal.cu
kernels/scale.cu
kernels/release.cu)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 可维护性建议(包组织结构,不阻塞合入):kernels/release.cu 里没有 __global__,也没有 <<<>>>,内容是 host 侧状态机和 std::thread。这和 kernels/kernels.h 文件头「kernels/ 只放 kernel 的 launch 包装」的约定不符,还会让这段 host 代码多经过一次 nvcc。建议移到 gin_overlap/release.cc,与 gin_engine.cc 并列,并同步更新这里和第 12 行的注释。

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

已处理:放行层已移到 gin_overlap/release.cc,CMake 和各处引用同步更新了。

// - pos_moe_put:专家 put。
// - stage_positions[3]:专家分片求和。
// 一个阶段的待放行队列 = 它的位置到上一阶段位置之间的那些桶。
int pos_dense_copy = 0, stage_positions[4] = {0, 0, 0, 0}, pos_moe_put = 0;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 可维护性建议(底层优雅度,不阻塞合入):六个阶段的进度里,两个是具名变量,另外四个放在 stage_positions[0..3] 里,下标含义只写在上面的注释中。所以 release_next_put、pending_counts、stuck_stage 里出现 stage_positions[1] >= stage_positions[0] 这类表达式,要对着注释才能看懂。建议拆成与 STAGE_* 对应的具名变量(如 pos_dense_intra_node_reduce / pos_dense_inter_node_put / pos_dense_inter_node_reduce / pos_moe_reduce)。step_reset 里的清零逻辑也随之变直白。

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

已处理:六个阶段的进度都改成了具名变量,stage_positions 数组已删除。

Comment on lines +749 to +754
control = new TailControl();
try {
std::thread(thread_main).detach();
} catch (...) {
return (int)cudaErrorUnknown;
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 可维护性建议(初始化耦合,不阻塞合入):control 在尾部线程启动之前就已经发布。std::thread 构造失败时只返回 cudaErrorUnknown,control 留在非空状态,于是 setup 再也不能重试(入口检查 control 非空)。之后 advance 照常可用;tail_start 会把状态改成 RUN,但没有线程处理,tail_wait 只能等到超时。建议在 catch 分支里 delete control; control = nullptr; 后再返回,或者等线程成功启动后再给 control 赋值。

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

已处理:线程启动失败时会 delete control 并置空,setup 可以重试。

Comment on lines +178 to +184
inline unsigned long long credit_target(long long num_buckets,
long long position,
long long depth) {
if (depth < 0) return 0;
const long long target = step * num_buckets + position - depth + 1;
return target > 0 ? (unsigned long long)target : 0ull;
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 可维护性建议(测试策略,不阻塞合入):放行层的判断大多是纯 host 逻辑:credit_target 的跨步递推、blocked_* 对 host window 计数的比较、六个阶段位置的先后约束,以及 IDLE/RUN/DONE/ERR/ABORTED 的状态转换。这些逻辑目前只能靠多机实跑覆盖。建议后续的单测 PR 把这部分和 submit_* 解耦(例如用一块普通内存模拟 host window,用桩函数替代 submit),至少覆盖 credit 目标、阶段推进顺序和 bucket_ready / tail_* 的状态检查。

Comment on lines +394 to +398
if (!in_window(row[0], m * dense.size, 4, 4, dense.send_bytes) ||
!in_window(row[1], 1, 8, 8, dense.host_bytes))
return false;
if (row[5] >= 0 &&
!in_window(row[3], (L - 1) * Nn * m, 4, 4, dense.recv_bytes))

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P3 dense 的 send_offset(row[0])和 recv_offset_intra(row[3])这里只要求 4 字节对齐,但 submit_dense_intra_node_reduce 把 send + send_offset、recv + recv_offset 直接当 float4* 交给 k_reduce_intra_node。这个 kernel 没有标量回退(kernels/reduce.cu 71-111 行),要求 16 字节对齐。

结果是一个 4 字节对齐、但不是 16 字节对齐的偏移能通过 setup,要到 kernel 才报 misaligned address。这是 sticky 错误,进程只能重启,拿不到 cudaErrorInvalidValue。上一轮建议里写的“数据 4 字节”对这两列不够。

建议把这两列的对齐要求改成 16(Python 侧按 256 对齐,不受影响),或者在 submit_dense_intra_node_reduce 里检查两个指针的对齐,不满足时返回 cudaErrorInvalidValue。

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

已在当前提交修复:dense 的 send_offset(row[0])和 recv_offset_intra(row[3])现在按 16 字节对齐校验,与 k_reduce_intra_node 的 float4 访问一致。

@AlAuAu
AlAuAu force-pushed the gin_split_5_release branch from f6bc21c to ae8eee4 Compare November 21, 2026 11:21
@AlAuAu
AlAuAu force-pushed the gin_split_5_release branch from ae8eee4 to f1cb51e Compare November 21, 2026 12:43
@AlAuAu
AlAuAu force-pushed the gin_split_5_release branch from f1cb51e to c10b645 Compare November 21, 2026 14:52
@AlAuAu AlAuAu changed the title [WIP][Distributed] gin_overlap 5/6: the release layer [WIP][Distributed] gin_overlap 1/2: NCCL-GIN gradient-comm primitive (C++) Nov 21, 2026
…treams

Second piece of the move from PaddleFleet's extension: the host side that
builds everything the per-step code runs on. Nothing calls it yet (the
Python binding comes last), and nothing is submitted to the GPU per step
yet (next commit).

* gin_engine.h / gin_engine.cc, initialization only: the CommGroup state of
  the two communication groups (sharding / moe_sharding), the error-check
  macros, and the GradCommEngine methods that the Python layer calls once at
  startup, in this order: set_comms (reuse Paddle's ncclComm_t for each
  ring id and cross-check size / rank), alloc_windows (send / recv / signal
  as NCCL_WIN_COLL_SYMMETRIC windows), alloc_host_counters (one HOST_NUMA
  host window per group, registered with NCCL_WIN_STRICT_ORDERING),
  create_dev_comm (ncclDevCommCreate + LSA pointers), set_group_topology,
  and configure (side streams, per-bucket events, the window-mark ring, the
  credit offsets, the host-window LSA pointers, and preloading every kernel
  so that the first launch of a module does not land on the backward
  critical path). Also error_string and the NCCL runtime / build version
  queries.
* paddle_comm.cc: takes the ncclComm_t out of Paddle's ProcessGroup by ring
  id, the way DeepEP does (ProcessGroupMapFromGid -> ProcessGroupNCCL ->
  NCCLCommContext::GetNcclComm).
* paddle_tensor.cc / .h: Paddle's current compute stream, the zero-copy
  paddle::Tensor over a send-window slice (paddle::from_blob with deleter =
  nullptr, so the memory stays owned by the NCCL window, plus a post-wrap
  self-check), and the pitched copy_2d through cuMemcpy2DAsync.

The file header of gin_engine.h is the one full description of the C++
side (terms, the two groups, the dense / expert / 2D stages, the receive
pools, the host-window layout and the constraints). It is added here in
full: it is organised by concept, not by layer, so the parts about the
per-stage submits and the release layer (next two commits) cannot be
moved out without rewriting it.

paddle_comm.cu and paddle_tensor.cu become .cc: they were .cu only because
an out-of-tree build has to hand any translation unit that includes Paddle
headers to nvcc (thrust); in-tree they are ordinary sources. The pybind
registration that was stranded in paddle_tensor.cu moves to the pybind
layer, so paddle_tensor.h declares the real paddle::Tensor signatures.

The library now also depends on process_group_nccl, for paddle_comm.cc.
Like the kernels, the code is moved, not rewritten: same checks, same
order of collective calls.

Co-Authored-By: Claude Code <noreply@anthropic.com>
Translate every comment, docstring, error message and log line of the
gin_overlap sources (C++, CUDA, CMake, pybind and the Python layer) into
English, and tighten the wording on the way. The terms are defined once,
in the gin_engine.h file header (C++) and in the engine.py module
docstring (Python); every other file uses them as defined there.

Strings only. comment_clean_check.py --ignore-strings passes on every
changed file: C++ / CUDA / CMake token streams are identical except for
string literals (printf conversions inside merged literals keep their
order), and the Python ASTs are identical except for str constants and
the literal parts of f-strings. Log lines all carry the [gin_overlap]
prefix; copy_2d errors now start with "copy_2d:".

Co-Authored-By: Claude Code <noreply@anthropic.com>
…ne API

Argument checks (values come from Python):
- scale_range, gather_put_to_owner, send_window_tensor and copy_2d check
  their bounds without unsigned wrap-around, and reject numel < 0.
  copy_2d also rejects non-contiguous tensors (a stride-0 or transposed
  view passes a numel-based bound but the copy reads contiguous bytes).
- release::setup checks every offset of the per-bucket tables and the
  host window areas: non-negative, aligned (counters 8 B, data 4 B) and
  inside its window. setup_gather_table checks both arrival counter
  offsets of each entry the same way. CommGroup records recv_bytes for
  this, and alloc_symmetric_window returns the size it actually
  registered instead of the caller repeating the 4096 B minimum.
- A wrong state returns cudaErrorNotReady and a bad argument
  cudaErrorInvalidValue (gather_setup, gather_put_to_owner, scale_range).
- bucket_ready rejects a bucket reported twice in a step in the RUN state
  too, from a bitmap only the main thread touches.

Init failures:
- configure destroys the streams and events it created, frees its
  temporary buffer and resets the globals it set when a later step fails,
  so it can be retried. set_comms clears the groups it set on failure.
  release::setup unpublishes its control block if the tail thread cannot
  be started. The steps that cannot be rolled back (window registration,
  devcomm creation, the host window) say so: restart the process.
- The gin_engine.h constraints state that the engine lives as long as
  the process; shutdown_process_group / restart_process_group are not
  supported.

API:
- GIN_CHECK replaces the dozen { rc = launch_x(...); if (rc) return rc; }
  blocks. launch_* / preload_* return cudaError_t.
- launch_put_to_expert_peers takes (send_offset, recv_offset) like
  launch_put_to_node_peers.
- join_bucket_streams / gather_wait_bucket take kind (0 dense, 1 expert),
  as bucket_ready and the gather table do, instead of is_dense.
- The six release positions are named variables, one per stage.
- The methods that do not return a code (record_window_mark, tail_mark,
  bucket_ready, tail_all_posted, tail_wait, tail_abort, release_activity,
  the version getters) document what they return.
- tail_wait / tail_abort release the GIL while they block.

No kernel, reduction order, release condition or credit target changes.

Co-Authored-By: Claude Code <noreply@anthropic.com>
kernels/release.cu has no kernel and no <<<>>>: it is the host-side
release state machine and its std::thread. kernels/ holds only the
kernel launch wrappers, so the file moves next to gin_engine.cc as
release.cc and is compiled by the host compiler instead of nvcc. The
content is unchanged apart from the file name in comments.

Co-Authored-By: Claude Code <noreply@anthropic.com>
Remove gin_overlap/ from the cpplint exclude list and fix every finding:
C-style casts become static_cast / reinterpret_cast, long long and
unsigned long long become int64_t / uint64_t, non-const reference
out-parameters become pointers (call sites updated), and C system headers
come before C++ ones. atomicAdd_system keeps its unsigned long long
prototype through one aliased type. No change to values, types seen by
Python, or arithmetic.

Co-Authored-By: Claude Code <noreply@anthropic.com>
…e slot as an out-parameter

C++: both return 0 or a cudaError_t code and write the slot to *slot:
cudaErrorNotReady without configure / release_setup, the error of
paddle_compute_stream or cudaEventRecord, and cudaErrorLaunchOutOfResources
when this step's tail mark slots are used up (was -1 / -2). The Python
binding still returns the slot and raises RuntimeError with the error text
on failure. CPU test of the binding before setup.

Co-Authored-By: Claude Code <noreply@anthropic.com>
…ther

Third piece of the move from PaddleFleet's extension: the host functions
that hand one stage of one bucket to its side stream, and the stream /
event plumbing around them. Still no caller: the release layer that decides
when each stage may go (next commit) is the only code that calls submit_*.

* The six stage submits: submit_dense_intra_node_copy (cudaMemcpyBatchAsync
  of the column to the other ranks on the node + the arrival write),
  submit_dense_intra_node_reduce (+ intra-node credit),
  submit_dense_inter_node_put, submit_moe_put, submit_dense_inter_node_reduce
  and submit_moe_reduce (+ GIN credit). Each one only waits on work this
  rank has already handed out (a compute-stream mark or a local side-stream
  event), never on a remote counter: whether the remote side is ready is
  decided on the host from the host window before the call.
* record_window_mark: records a mark on Paddle's current compute stream into
  the ring configure() created.
* join_bucket_streams (compute stream waits for a bucket's side-stream
  events), scale_range, end_step (GIN flush of both groups so the send
  window can be reused), stats.
* Muon 2D-bucket gather to the owner: gather_setup, gather_gate,
  gather_wait_bucket, gather_mark_ready, gather_put_to_owner (GIN put across
  nodes, CE + release store within a node) and gather_join. The table-driven
  entry point, gather_put_entry, forwards to the release layer and comes
  with it.

Moved as is from the extension, like the previous commits.

Co-Authored-By: Claude Code <noreply@anthropic.com>
Translate every comment, docstring, error message and log line of the
gin_overlap sources (C++, CUDA, CMake, pybind and the Python layer) into
English, and tighten the wording on the way. The terms are defined once,
in the gin_engine.h file header (C++) and in the engine.py module
docstring (Python); every other file uses them as defined there.

Strings only. comment_clean_check.py --ignore-strings passes on every
changed file: C++ / CUDA / CMake token streams are identical except for
string literals (printf conversions inside merged literals keep their
order), and the Python ASTs are identical except for str constants and
the literal parts of f-strings. Log lines all carry the [gin_overlap]
prefix; copy_2d errors now start with "copy_2d:".

Co-Authored-By: Claude Code <noreply@anthropic.com>
…the gate

Both cudaMemcpyBatchAsync calls use the CUDA 13 signature (8 arguments,
const void** sources). CUDA 12.8 / 12.9 declare a 9-argument version, so
a build with the gate open on those toolkits would not compile. Raise the
gate to CUDA >= 13.0, the only toolkit the code has been built and run
with, instead of adding an untested 12.x branch; gin_dynload.h gets the
matching #error backstop. The batch attribute index is named attrs_idx,
which is what that argument is.

The arch condition reuses ARCH_BIN_CONTAINS_GE_90, computed at the top of
the same file for DeepEP, instead of a second loop over NVCC_ARCH_BIN.
_BUILD_HINT in the Python layer now lists exactly the gate's conditions,
and the CMake comment says to change both together.

Co-Authored-By: Claude Code <noreply@anthropic.com>
…ne API

Argument checks (values come from Python):
- scale_range, gather_put_to_owner, send_window_tensor and copy_2d check
  their bounds without unsigned wrap-around, and reject numel < 0.
  copy_2d also rejects non-contiguous tensors (a stride-0 or transposed
  view passes a numel-based bound but the copy reads contiguous bytes).
- release::setup checks every offset of the per-bucket tables and the
  host window areas: non-negative, aligned (counters 8 B, data 4 B) and
  inside its window. setup_gather_table checks both arrival counter
  offsets of each entry the same way. CommGroup records recv_bytes for
  this, and alloc_symmetric_window returns the size it actually
  registered instead of the caller repeating the 4096 B minimum.
- A wrong state returns cudaErrorNotReady and a bad argument
  cudaErrorInvalidValue (gather_setup, gather_put_to_owner, scale_range).
- bucket_ready rejects a bucket reported twice in a step in the RUN state
  too, from a bitmap only the main thread touches.

Init failures:
- configure destroys the streams and events it created, frees its
  temporary buffer and resets the globals it set when a later step fails,
  so it can be retried. set_comms clears the groups it set on failure.
  release::setup unpublishes its control block if the tail thread cannot
  be started. The steps that cannot be rolled back (window registration,
  devcomm creation, the host window) say so: restart the process.
- The gin_engine.h constraints state that the engine lives as long as
  the process; shutdown_process_group / restart_process_group are not
  supported.

API:
- GIN_CHECK replaces the dozen { rc = launch_x(...); if (rc) return rc; }
  blocks. launch_* / preload_* return cudaError_t.
- launch_put_to_expert_peers takes (send_offset, recv_offset) like
  launch_put_to_node_peers.
- join_bucket_streams / gather_wait_bucket take kind (0 dense, 1 expert),
  as bucket_ready and the gather table do, instead of is_dense.
- The six release positions are named variables, one per stage.
- The methods that do not return a code (record_window_mark, tail_mark,
  bucket_ready, tail_all_posted, tail_wait, tail_abort, release_activity,
  the version getters) document what they return.
- tail_wait / tail_abort release the GIL while they block.

No kernel, reduction order, release condition or credit target changes.

Co-Authored-By: Claude Code <noreply@anthropic.com>
kernels/release.cu has no kernel and no <<<>>>: it is the host-side
release state machine and its std::thread. kernels/ holds only the
kernel launch wrappers, so the file moves next to gin_engine.cc as
release.cc and is compiled by the host compiler instead of nvcc. The
content is unchanged apart from the file name in comments.

Co-Authored-By: Claude Code <noreply@anthropic.com>
Co-Authored-By: Claude Code <noreply@anthropic.com>
Remove gin_overlap/ from the cpplint exclude list and fix every finding:
C-style casts become static_cast / reinterpret_cast, long long and
unsigned long long become int64_t / uint64_t, non-const reference
out-parameters become pointers (call sites updated), and C system headers
come before C++ ones. atomicAdd_system keeps its unsigned long long
prototype through one aliased type. No change to values, types seen by
Python, or arithmetic.

Co-Authored-By: Claude Code <noreply@anthropic.com>
…e slot as an out-parameter

C++: both return 0 or a cudaError_t code and write the slot to *slot:
cudaErrorNotReady without configure / release_setup, the error of
paddle_compute_stream or cudaEventRecord, and cudaErrorLaunchOutOfResources
when this step's tail mark slots are used up (was -1 / -2). The Python
binding still returns the slot and raises RuntimeError with the error text
on failure. CPU test of the binding before setup.

Co-Authored-By: Claude Code <noreply@anthropic.com>
…rd to it

Fourth piece of the move from PaddleFleet's extension: the code that
decides, for each bucket and stage, whether it may be handed to the GPU now,
and then calls the submit_* functions of the previous commit. With this the
C++ side is complete; it is still not reachable from Python (next commit).

kernels/release.cu: the release decision lives here exactly once and is
shared by two releasers that never run at the same time (both take the
control mutex; advance() only releases in IDLE, the tail thread only in
RUN), so there is one writer of the per-step state at any time:
* the main thread, which calls advance() in every window where DeepEP uses
  neither the network nor the SMs, with the set of stages the caller picked
  for that window, and returns right away;
* the tail thread, which takes over after the last combine of the step
  (tail_start) and loops over the same release_next_* until every bucket
  is out; the main thread then only reports buckets that became ready,
  says when all are posted, waits on a condition variable and finishes.
Readiness is read from the host window (credits, intra-node arrivals, GIN
signals), never waited on from the GPU. Also here: the tail mark ring,
status / stuck_report / last error for diagnostics, the Muon 2D gather
table and its entry point, and the constants table (ADVANCE_* / STAGE_* /
COUNTER_* / TAIL_*), whose values are defined only here so the Python side
reads them by name. The file is .cu because it sits with the kernels, but
it is host code (no kernel is defined in it).

gin_engine.h / gin_engine.cc: the release:: declarations and the thin
GradCommEngine wrappers the Python layer calls (release_setup,
release_step_reset, advance, bucket_ready, release_status,
release_stuck_report, release_error, release_activity, tail_*,
tail_gather_*, gather_put_entry). They only check arguments and forward.

kernels/release.cu is appended to GIN_OVERLAP_SRCS. Moved as is from the
extension, like the previous commits.

Co-Authored-By: Claude Code <noreply@anthropic.com>
Translate every comment, docstring, error message and log line of the
gin_overlap sources (C++, CUDA, CMake, pybind and the Python layer) into
English, and tighten the wording on the way. The terms are defined once,
in the gin_engine.h file header (C++) and in the engine.py module
docstring (Python); every other file uses them as defined there.

Strings only. comment_clean_check.py --ignore-strings passes on every
changed file: C++ / CUDA / CMake token streams are identical except for
string literals (printf conversions inside merged literals keep their
order), and the Python ASTs are identical except for str constants and
the literal parts of f-strings. Log lines all carry the [gin_overlap]
prefix; copy_2d errors now start with "copy_2d:".

Co-Authored-By: Claude Code <noreply@anthropic.com>
…ne API

Argument checks (values come from Python):
- scale_range, gather_put_to_owner, send_window_tensor and copy_2d check
  their bounds without unsigned wrap-around, and reject numel < 0.
  copy_2d also rejects non-contiguous tensors (a stride-0 or transposed
  view passes a numel-based bound but the copy reads contiguous bytes).
- release::setup checks every offset of the per-bucket tables and the
  host window areas: non-negative, aligned (counters 8 B, data 4 B) and
  inside its window. setup_gather_table checks both arrival counter
  offsets of each entry the same way. CommGroup records recv_bytes for
  this, and alloc_symmetric_window returns the size it actually
  registered instead of the caller repeating the 4096 B minimum.
- A wrong state returns cudaErrorNotReady and a bad argument
  cudaErrorInvalidValue (gather_setup, gather_put_to_owner, scale_range).
- bucket_ready rejects a bucket reported twice in a step in the RUN state
  too, from a bitmap only the main thread touches.

Init failures:
- configure destroys the streams and events it created, frees its
  temporary buffer and resets the globals it set when a later step fails,
  so it can be retried. set_comms clears the groups it set on failure.
  release::setup unpublishes its control block if the tail thread cannot
  be started. The steps that cannot be rolled back (window registration,
  devcomm creation, the host window) say so: restart the process.
- The gin_engine.h constraints state that the engine lives as long as
  the process; shutdown_process_group / restart_process_group are not
  supported.

API:
- GIN_CHECK replaces the dozen { rc = launch_x(...); if (rc) return rc; }
  blocks. launch_* / preload_* return cudaError_t.
- launch_put_to_expert_peers takes (send_offset, recv_offset) like
  launch_put_to_node_peers.
- join_bucket_streams / gather_wait_bucket take kind (0 dense, 1 expert),
  as bucket_ready and the gather table do, instead of is_dense.
- The six release positions are named variables, one per stage.
- The methods that do not return a code (record_window_mark, tail_mark,
  bucket_ready, tail_all_posted, tail_wait, tail_abort, release_activity,
  the version getters) document what they return.
- tail_wait / tail_abort release the GIL while they block.

No kernel, reduction order, release condition or credit target changes.

Co-Authored-By: Claude Code <noreply@anthropic.com>
kernels/release.cu has no kernel and no <<<>>>: it is the host-side
release state machine and its std::thread. kernels/ holds only the
kernel launch wrappers, so the file moves next to gin_engine.cc as
release.cc and is compiled by the host compiler instead of nvcc. The
content is unchanged apart from the file name in comments.

Co-Authored-By: Claude Code <noreply@anthropic.com>
…fsets in release setup

Co-Authored-By: Claude Code <noreply@anthropic.com>
Remove gin_overlap/ from the cpplint exclude list and fix every finding:
C-style casts become static_cast / reinterpret_cast, long long and
unsigned long long become int64_t / uint64_t, non-const reference
out-parameters become pointers (call sites updated), and C system headers
come before C++ ones. atomicAdd_system keeps its unsigned long long
prototype through one aliased type. No change to values, types seen by
Python, or arithmetic.

Co-Authored-By: Claude Code <noreply@anthropic.com>
…e slot as an out-parameter

C++: both return 0 or a cudaError_t code and write the slot to *slot:
cudaErrorNotReady without configure / release_setup, the error of
paddle_compute_stream or cudaEventRecord, and cudaErrorLaunchOutOfResources
when this step's tail mark slots are used up (was -1 / -2). The Python
binding still returns the slot and raises RuntimeError with the error text
on failure. CPU test of the binding before setup.

Co-Authored-By: Claude Code <noreply@anthropic.com>
@AlAuAu
AlAuAu force-pushed the gin_split_5_release branch from c10b645 to 00624af Compare November 21, 2026 15:14
@AlAuAu AlAuAu changed the title [WIP][Distributed] gin_overlap 1/2: NCCL-GIN gradient-comm primitive (C++) [WIP][Distributed] gin_overlap 2/3: engine and release layer (C++) Nov 21, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants