Repository navigation
[Example] Add Kimi K3 Gated MLA prefill forward - #1970
dlaerbinjwy wants to merge 14 commits into
Conversation
…ation 🤖 Add a fixed-domain AscendC Expert example with independent mathematical references, device boundary tests, CI routing and reproducible benchmarks. Replace elementwise readout tail masking with an aligned vector fill. Validate 100 workspace tests, 25 core and 4 extended packaged tests on 910B3, and document timing, precision and remaining acceptance work.
|
👋 Hi! Thank you for contributing to the TileLang project. Please remember to run We appreciate you taking this step! Our team will review your contribution, and we look forward to your awesome work! 🚀 |
Match the Ruff 0.17.3 format check used by AscendC-PTO CI. The computational Python AST is unchanged in all four files. Ruff format, Ruff lint and git whitespace checks pass locally.
Load the full latent score tile, default to PTO with explicit AscendC support, and add independent attention-core benchmarks and verified 910B3 evidence. Synchronize with the current ascendc_pto runtime baseline.
Add the experimental FP16/BF16 prefill/decode package for default PTO and AscendC, with Cube projections, shared latent cache, packed adapters and independent FP64 references. Register the existing core and extended suites, preserve the legacy decode tests, and add source-derived benchmarks and simulator/model evidence. All 145 core and 22 extended assertions pass locally; the 187.70-second cold core run still exceeds the 120-second acceptance budget. Keep ordinary-call performance and unqualified baseline limits explicit.
|
/re-test |
|
🔄 Re-running failed jobs Original workflow run: View details Only the failed jobs will be re-executed. |
Add independent-process benchmarks with six balanced method orders, separate Event/wall/graph distributions, per-boundary precision qualification and immutable input/source checks. Keep diagnostic comparators unscored and keep the existing numerical implementation and historical protocols unchanged. Register 62 CPU regressions and attach the 36-trial, two-backend/two-dtype performance report, host attribution and four hardware Task snapshots. All candidate initial seed and post-round checks passed; ordinary-call performance limitations and unqualified BF16 references remain explicit. Validation: 155 package CPU tests passed, 43 device cases deselected; strict collection has 229 unique cases. Ruff 0.17.4 lint/format passed. Numerical kernels were unchanged; this is not a new optimization-speedup claim.
…or work Record source-pinned a172-parent comparisons, installed-artifact and hardware evidence. Retain original BF16 model cross-official failures and local 120-second target as explicit boundaries.
…mple Replace the installable package with one example file under examples/, matching how operators are contributed in this repository. The operator code is the same: example_gated_mla_prefill.py is a mechanical merge of the package's _contracts, kernels/vector and kernels/prefill modules, with the two relative imports removed, _contracts inlined whole so no transitive dependency is lost, and one byte-identical EXPERT_PASSES kept once. The generated CPP is byte-identical across both entry points, both backends and both dtypes, in the pipelined and fallback domains alike. The forward runs the seven already-projected inputs through Q absorption, latent attention, value recovery and the per-channel sigmoid gate, returning the result before W_O. The layer is NoPE, so the 64-wide auxiliary cache carries plain key channels; the output gate is enabled in the released Kimi K3 configuration. The attention core keeps two entry points that agree bit for bit. Head-aligned shapes whose cache fits one KV block, at batch one, compile main_pipelined, which skews the Cube loop by one wave so a wave's QK is issued before the previous wave's PV and the Vector softmax between them overlaps a Cube GEMM. Cube time blocked on flags falls from 57.63 to 1.19 us and Vector from 66.95 to 23.28 us, while cube_time 14.66 us and mte1_time 29.07 us are unchanged: only the schedule moved. The batch bound is a correctness one, since C and E are L1-resident under a batch tag; a negative test asserts the entry is not compiled outside it. Single-kernel Task time over nine shapes improves by 1.358x to 1.442x inside the domain, rising monotonically with wave count, and stays within 0.982-1.008x outside it. Core-scope Graph timing reaches 75.07 percent of the official fused_infer_attention_score_v2 on PTO and 82.86 percent on AscendC at B1/H96/Tq=Tk128 FP16; ordinary non-Graph calls are host dominated and do not improve. BF16 has no external reference passing the same accuracy qualification, so no BF16 ratio is reported. The test module covers 22 cases: a budgeted gate against an FP32 reference with exact zeros outside the valid query length, an analytic causal-mask oracle whose expected values are multiples of one half so a mask off by one key moves a row by exactly that, bit-exact batch isolation, and negative gates for invalid shapes, strided operands, wrong dtypes and the compiled domain itself. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…ios, dtype coverage Evidence only. The kernel region of the example is byte-identical to the previous commit (sha256 of the first 1543 lines unchanged), so the CI result already recorded for it still applies. Correctness Add reference_gated_mla_prefill_expanded, a second reference that shares no contraction order with the first: it materialises per-head keys and values from the latent cache and runs ordinary attention, where the existing one folds W_K into the query and contracts against the cache directly. The two are algebraically equal but computed by different code, so their agreement is evidence about the absorbed decomposition rather than about one way of writing it. Six CPU-only cases pin that agreement at 0.37 ULP worst case (22 -> 28 tests), and the kernel tolerance is now justified as sitting well above that floor instead of being tuned to the kernel's own output. Extend __main__ from one configuration to five, each exercising a different property rather than a larger version of the same one, and assert the inactive query tail is exactly zero before applying any tolerance. Performance Replace the single-working-point ratio with eight shapes, both arms qualified against an FP32 reference before either is timed. FIA implements right-down aligned causal masking only, so every cell sets query_start = kv_len - valid_q and both arms compute the same mask; comparing a top-left aligned candidate instead compares two different problems and shows a normalised error near 1.0 at every Tq != Tk. Add a block_n ablation that holds the shape still, which separates the pipeline from the shape: the pipelined entry is worth 2.20x at Tq128 and 2.61x at Tq64, at identical normalised error. It also establishes two limits now recorded in the docs: block_n >= 256 is refused with "unsupported attention tile", so the pipelined entry is structurally unreachable for Tk > 128; and public_prefill_block_n is narrower than what qualifies, leaving 2.61x at B1/H96/Tq64/Tk128. Widening the selector is not done here, because its contract is deliberately "measured profile only" and the newly reachable path carries no test coverage. Add timings for both dtypes on both backends, measured on the file as submitted by NPUGraph capture and replay. BF16 costs 26% (PTO) and 15% (AscendC) more than FP16, and only where the pipeline runs, matching the BF16 P-residual path; the backends converge to within 1.6% off that entry. Note that the Tq64 rows of the two tables are not comparable: the kernel skips cache blocks the mask makes entirely invisible, so query_start = 0 leaves one of two blocks live where query_start = 64 leaves both. Manifest Drop two comments and two blank lines left behind by the removed package and place the example's entry under its own directory header. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…plicate constant design.md Adds the design document the repository's other operators carry, following the ten-section structure of examples_experiment/cumsum_kda/design.md and labelling the programming mode as examples/fused_sigmoid_gating_delta_rule does. It covers what the benchmark documents do not: why Expert mode is required here (the cross-wave skew is hand-ordered, and an automatic CV sync pass inserts exactly the barrier that removes it), the on-chip budget per scope, the pass_configs and the reason each is off, and the mapping onto the Kimi K3 released configuration. Two numbers in it are worth surfacing. The L1 budget is 188 KiB of 512 KiB at the measured profile, and the second C/E slot that would lift the batch == 1 boundary costs 144 KiB, so 332 KiB would still fit -- that boundary is an unverified gap rather than a capacity wall. And block_n >= 256 is refused with "unsupported attention tile", which is what puts a ceiling on Tk > 128 rather than any budget. CI cost and coverage Both benchmark documents now state what the gates cost: 200 s for the 28 tests from a cold JIT cache, which is what CI starts from, and 233 s for the example's __main__. They also state which axes the shipped gates cover and which two are deliberately left to the benchmark tables -- cache lengths past 128 and head counts other than 96 are qualified at 2.1e-04 in the ratio matrix, but no shipped case pins them, because each distinct shape costs a JIT compilation and none of those shapes can reach the pipelined entry. Duplicate constant EXPERT_PASSES was defined twice, byte-identical, left over from merging the package's _contracts.py and kernels/*.py into one file; the second definition silently shadowed the first. Removed, leaving one definition and five uses. This touches the kernel region, so the previous commit's byte-identity argument no longer carries and the operator was re-verified on a 910B3: 28 tests passed in 199.83 s, and the example's five configurations passed on both backends. ruff check and ruff format are clean on all five files. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…validator
Two leftovers from merging the package's _contracts.py and kernels/*.py into a
single file, found by an AST pass over top-level definitions:
* DTYPES was defined twice, byte-identical, the second silently shadowing the
first.
* validate_chain_spans was defined and never called. In the package it was
the chain entry's span check (gmla/_runtime.py called it), but in this file
every check it performs is already done by the stage it delegates to:
q_absorb calls validate_projection_spans, build_cube_attention calls
validate_attention_spans, and each stage validates its own tensors through
_tensor, which is why the four rejection cases pass without it. The one
check unique to it covered an empty-cache output count. Removing it leaves
no orphans: _positive, checked_elements, validate_projection_spans and
validate_attention_spans all have other callers.
1639 -> 1598 lines. No behaviour change, but the kernel region moves, so the
operator was re-verified on a 910B3: 28 tests passed in 192.73 s and the
example's five configurations passed on both backends. ruff check and ruff
format are clean on all five files.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The directory had the design document and the benchmark reports but no entry point, which is the one piece the better-equipped example directories here all carry. This routes rather than repeats: the interface and the seven inputs, how to run the example and the gates, the supported configurations and the domain boundary, and a one-paragraph summary of accuracy and performance that points at design.md and the benchmark reports for the rest. Two things are stated here because a reader hits them first. metadata is a runtime input rather than a compile-time parameter, so one compiled kernel serves different valid lengths; and block_n comes from a host selector that returns 128 only for the measured profile, with block_n >= 256 refused by the compiler, which is what caps Tk > 128. Documentation only; no code changes. ruff check and ruff format are clean on all six files. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
/re-test |
|
🔄 Re-running failed jobs Original workflow run: View details Only the failed jobs will be re-executed. |
|
/re-test |
|
🔄 Re-running failed jobs Original workflow run: View details Only the failed jobs will be re-executed. |
|
/re-test |
|
🔄 Re-running failed jobs Original workflow run: View details Only the failed jobs will be re-executed. |
|
/re-test |
|
🔄 Re-running failed jobs Original workflow run: View details Only the failed jobs will be re-executed. |
|
/re-test |
|
🔄 Re-running failed jobs Original workflow run: View details Only the failed jobs will be re-executed. |
描述
在 tilelang-ascend 实现 Kimi K3 Gated MLA 的 prefill 前向,位于
examples/gated_mla_prefill/。参考模型为 Kimi-K3
modeling_kimi_linear.py(revisionc5d1dd4c)中的KimiMLAAttention。模型源码仅作溯源链接,不随包分发。算子语义
Gated MLA 层的注意力块:从七个已投影输入出发,经 Q 吸收投影、latent 注意力、值恢复与逐通道 sigmoid 门,输出位于 W_O 之前。该层为 NoPE(官方代码
assert self.use_nope、rotary_emb = None),因此 64 宽的辅助 cache 携带普通 key 通道而非旋转位置编码;输出门在发布配置中为mla_use_output_gate: true,是生产路径而非可选变体。计算公式
记 b 为 batch、t 为 query 行、h 为头、k 为 cache 位置(H=96、D=128、R=512、E=64、
scale = (D+E)^-0.5):key
k对 query 行t可见当且仅当k < min(query_start + t + 1, kv_len);超出有效 query 长度的行写出精确零。score、online softmax 状态与z为 FP32;其余同输入 dtype。实现方案(三次 kernel 发射)
三个阶段的形状与计算特性差异很大,不强行融成一次发射。
[128]@[128,512]逐头独立,Cube 分形收益不抵搬运(Z @ W_Vᵀ) * sigmoid(G)融合block_npublic_prefill_block_n决定,不暴露为调参旋钮iterations == 1 && heads % block_m == 0 && batch == 1编译main_pipelined,否则main跨 wave Cube/Vector 软件流水
基线是单缓冲四 flag 握手,Cube 与 Vector 严格交替。同形 PipeUtilization(各核中位):
cube_timemte1_time每个核的等待量接近另一个核的忙碌量,即两者从不重叠;同一采样里 GM→L1 带宽占用 2.5%、vector bank 冲突 1%,既非带宽也非冲突受限。
本 PR 把 Cube 循环错开一个 wave:wave w 的 QK 在 wave w−1 的 PV 之前发射,Cube 做 GEMM 时 Vector 处理夹在中间的 softmax。三个 workspace 加 2 槽维度,四 flag 改为按 workspace 配对的六个 ready/consumed 信号量,消费者在循环前预置一个槽位;两个 scope 在循环后各自排空预置的信号量,不让 cross flag 跨越 launch 存活。协议形式取自仓库内
examples/mha_sink_fwd_bhsd。cube_time与mte1_time一字未变,只有调度变了。适用域是正确性边界,不是调优选择。 C 与 E 按 batch tag 常驻 L1,错开的循环会让 wave w 的 QK 把新 batch 的 C/E 载入待执行 PV 仍在读的缓冲。放开
batch == 1需要第二个 C/E 槽位、约多占 144 KiB L1,必须在硬件上确认预算。测试里有一条负判据钉住它。性能
口径:
msprof op单 kernel Task Duration,每格 3 次独立采集 × 8 次发射、丢弃首次,报中位数与半极差。FP16 / PTO。环境 910B3 / CANN 9.2.0 / torch_npu 2.7.1。加速随 wave 数单调上升(7→1.358 … 39→1.442);wave 数相同的两格(Tk128 与 Tk64,均 20)加速持平(1.418 与 1.409),尽管绝对耗时差近一倍。决定收益的是并行度而非工作量 —— 填充与排空各占一个 wave,开销为
1/waves。后三格在守卫之外,全部落在 1.00× 附近,域外零收益是设计结果。逐步实测(同形):208.16 → 193.96 → 176.98 → 165.86 → 159.58 → 151.66 → 143.50 → 98.55 → 97.80 µs,最后两步即本 PR 的流水与阶段判定进位。前七步为单进程 pilot 快照,后两步为 3 次采集中位数,两者不可直接相减作为严格增量。
两种 dtype、两个后端,在提交状态的文件上实测(NPUGraph 捕获重放,10 次调用一张图、重放 20 次、3 次独立 run,attention 阶段):
BF16 比 FP16 贵、且只在流水跑着的实测 profile 上贵(PTO +26%、AscendC +15%),与 BF16 走 P 残差路径一致;AscendC 在该 profile 领先 PTO 9%(FP16)/17%(BF16),其余各行两后端收敛到 1.6% 以内 —— 两后端只在错拍生效处才有差别。
对官方参考的比例:仅 attention 核心范围,对 absorbed
npu_fused_infer_attention_score_v2(v2 为当前较新接口,非兜底档),FP16 / PTO,NPUGraph 计时,8 个形状,两条臂都先对 FP32 参考过合格判据再计时(候选 1.2–2.4e-04、参考 3.4–6.3e-04,全部合格):整张表独立重跑一遍,每个比例复现在 1.2 个点以内。
FIA 只实现右下对齐 causal(其他模式被设备拒绝:
Only support sparse 0/3/4/9 when ifa mla is enabled),本算子用metadata[b,2](query_start)原生表达同一对齐,故每格设query_start = kv_len - valid_q,两条臂算同一个掩码;否则比的是两个不同问题,Tq != Tk 的形状会显示归一化误差接近 1.0。入口列说明这些行之间不可互比:流水入口要求 cache 一轮覆盖且单 batch,而public_prefill_block_n只对实测过的那一个 profile 返回 128,故 8 行里 7 行走保守回退。固定形状、只动block_n才能把流水与形状分开:block_n=64block_n=128两个入口的归一化误差在所示位数上完全相同(1.2e-04/2.1e-04)—— 它们算的是同一件事。这是本改动的诚实度量;比例表的离散大部分来自选择器与 tile 上限。两条随之而来的结论:
block_n>= 256 被直接拒绝(unsupported attention tile),故 Tk > 128 时流水入口在结构上不可达,Tk256/Tk512 的低比例是上限而非调优缺口。该比例与消融取自本 example 不随包携带的装置;benchmark 文档给出口径、形状、掩码约定与环境以便复现,不宣称本 example 自带它们。
不能把 Graph 比例表述为所有调用域达标:同口径的普通(非 Graph)调用由主机侧开销主导,对 FIA 约 66%,本优化按设计不改善它。
剩余方向:Cube 侧忙碌 89.28 µs 中
cube_time仅 14.66 µs、mte1_time29.07 µs,约 45 µs 既非算也非搬,指令发射与核间同步下发是候选。仓库内examples/flash_attention/fa_opt的阶梯将 60%→72% 归于"优化核间同步下发次数"、72%→80% 归于"用 axpy 减少指令数",两步本 PR 均未实施;A9 之后 Cube 为关键路径(Cube busy 89.28 µs 对 Vector 68.53 µs),前者优先级更高。精度
误差下限:两个独立金标
模块带两个参考实现,不共享任何收缩顺序:吸收式(kernel 实现的那种,把
W_K折进 query 后直接与 latent cache 收缩)与展开式(先把 latent cache 展开成逐头 K/V,再走常规注意力)。两者代数等价(对 latent rank 的内层求和可按任一顺序取),但代码路径完全不同,所以一致性证明的是吸收式分解本身正确,而不只是 kernel 匹配了某一种写法。
CPU 实测(无需 NPU),按输出幅度归一:
最差 0.37 ULP,全部不到一个最低位。这是下限 —— 容差若定得比两个正确参考之间的差距还紧,量的就是参考而不是 kernel。下面的
rtol=2^-9、atol=2^-10明显宽于该下限,是据此选定而非按实现输出凑出来的。判据
test_example_gated_mla_prefill.py共 28 项(其中 6 项为上述 CPU 金标互证),四类判据不同标准:rtol=2^-9、atol=2^-10。覆盖 FP16/BF16 × PTO/AscendC。QL = QA = 0则可见 score 全为 0、可见 key 概率为exp(0);令latent_kv[k,:] = k则 PV 累加为Σ(0..n-1)、除以归一化因子后为(n-1)/2。所有期望值都是 0.5 的整数倍,而 mask 多收或少收一个 key 恰好移动 0.5,故主判据为「四舍五入到最近 0.5 后精确相等」,没有容差可争论;另有一条 8 ULP of FP32 上界防止舍入掩盖漂移。覆盖整块与逐行两条 mask 路径、query_start平移、kv_len钳位。batch ≥ 2与多 KV 块时必须不被编译 —— 若哪天它开始被编译,说明文档所述域边界已变。已知限制
iterations == 1、heads % block_m == 0且batch == 1时编译。前两条是守卫性质,第三条是正确性边界。域外形状走原实现,实测零收益。block_n>= 256 被拒绝(unsupported attention tile),故 Tk > 128 时任何block_n都到不了流水入口;Tk256/Tk512 的低比例是可用 tile 决定的上限。public_prefill_block_n刻意比"能合格的范围"更窄:B1/H96/Tq64/Tk128 上流水block_n=128合格且快 2.61×,但选择器不选它。契约为"仅实测 profile"、新路径无测试覆盖,故本 PR 不放宽,列为第一条性能待办。query_start = kv_len - valid_q;Tq > Tk 的形状无法对比。变更类型
[Bugfix]Bug 修复[Feature]新功能/新算子[Refactor]重构[CI]CI 或构建[Skills]Skill/开发流程[Testing]测试[Example]示例/Benchmark文件
本 PR 共 8 个文件,其中 6 个新增、2 个各改一行:
examples/gated_mla_prefill/README.mdexamples/gated_mla_prefill/example_gated_mla_prefill.pyexamples/gated_mla_prefill/test_example_gated_mla_prefill.pyexamples/gated_mla_prefill/design.mdexamples/gated_mla_prefill/benchmark_gated_mla_prefill.mdexamples/gated_mla_prefill/benchmark_gated_mla_prefill_zh.mdci/operator_test_manifest.yamlpyproject.tomlcann(见下)pyproject.toml这一行的理由:CANN是昇腾工具链的名字,codespell 会把它判成can的拼写错误。不加这一行,
bash format.sh会在仓库原有的 4 处报错(
tilelang/tools/ascend_exception_dump_bin.py三处、tilelang/transform/pass_config.py一处),与本 PR 的文件无关。实测加与不加,本 PR 自己的 5 个新文件都不触发 codespell。
codespell 不在 GitHub Actions 门禁内,只在
format.sh的本地流程里。