Skip to content

[Example] Add Kimi K3 Gated MLA prefill forward - #1970

Open
dlaerbinjwy wants to merge 14 commits into
tile-ai:ascendc_ptofrom
dlaerbinjwy:gated-mla-decode
Open

dlaerbinjwy wants to merge 14 commits into
tile-ai:ascendc_ptofrom
dlaerbinjwy:gated-mla-decode

Conversation

@dlaerbinjwy

@dlaerbinjwy dlaerbinjwy commented Oct 30, 2026 •

Copy link
Copy Markdown
Contributor

描述

在 tilelang-ascend 实现 Kimi K3 Gated MLA 的 prefill 前向,位于 examples/gated_mla_prefill/。

参考模型为 Kimi-K3 modeling_kimi_linear.py(revision c5d1dd4c)中的 KimiMLAAttention。模型源码仅作溯源链接,不随包分发。

算子语义

Gated MLA 层的注意力块:从七个已投影输入出发,经 Q 吸收投影、latent 注意力、值恢复与逐通道 sigmoid 门,输出位于 W_O 之前。该层为 NoPE(官方代码 assert self.use_nope、rotary_emb = None),因此 64 宽的辅助 cache 携带普通 key 通道而非旋转位置编码;输出门在发布配置中为 mla_use_output_gate: true,是生产路径而非可选变体。

计算公式

记 b 为 batch、t 为 query 行、h 为头、k 为 cache 位置(H=96、D=128、R=512、E=64、scale = (D+E)^-0.5):

ql[b,t,h,:] = q_content[b,t,h,:] @ w_k[h]                               # [128] @ [128,512] -> [512]
s[b,t,h,k]  = scale * ( ql[b,t,h,:] · c[b,k,:] + qa[b,t,h,:] · e[b,k,:] )
p[b,t,h,k]  = softmax_k( s[b,t,h,k] )                                    # 仅在可见 k 上
z[b,t,h,:]  = Σ_k p[b,t,h,k] · c[b,k,:]                                  # [512], FP32
y[b,t,h,:]  = ( z[b,t,h,:] @ w_v[h]ᵀ ) * sigmoid( gate_logits[b,t,h,:] )  # [128]

key k 对 query 行 t 可见当且仅当 k < min(query_start + t + 1, kv_len);超出有效 query 长度的行写出精确零。score、online softmax 状态与 z 为 FP32;其余同输入 dtype。

实现方案(三次 kernel 发射)

三个阶段的形状与计算特性差异很大,不强行融成一次发射。

组件 选择 理由
Q 吸收 Vector kernel,FP32 累加后显式舍入回输入 dtype [128]@[128,512] 逐头独立,Cube 分形收益不抵搬运
注意力核心 持久 Cube,固定 20 个物理块;online softmax 状态常驻 UB 块数等于物理核数,摊薄每块序幕
值恢复 + 门 Vector kernel,(Z @ W_Vᵀ) * sigmoid(G) 融合 门是逐通道逐元素,与值恢复同一次遍历
block_n host 侧 public_prefill_block_n 决定,不暴露为调参旋钮 只有实测过的那一个 profile 选 128,其余 64;强制其它组合会编出公开调用走不到的 kernel
掩码 head 对齐时整块 packed mask,否则逐行 整块路径省去逐行比较
双入口 iterations == 1 && heads % block_m == 0 && batch == 1 编译 main_pipelined,否则 main 见下

跨 wave Cube/Vector 软件流水

基线是单缓冲四 flag 握手,Cube 与 Vector 严格交替。同形 PipeUtilization(各核中位):

基线 本 PR
Cube 阻塞在 flag 57.63 / 129.66 µs(44%) 1.19 µs(1%)
Vector 阻塞在 flag 66.95 / 130.55 µs(51%) 23.28 µs(25%)
cube_time 14.66 µs 14.66 µs
mte1_time 29.07 µs 29.07 µs

每个核的等待量接近另一个核的忙碌量,即两者从不重叠;同一采样里 GM→L1 带宽占用 2.5%、vector bank 冲突 1%,既非带宽也非冲突受限。

本 PR 把 Cube 循环错开一个 wave:wave w 的 QK 在 wave w−1 的 PV 之前发射,Cube 做 GEMM 时 Vector 处理夹在中间的 softmax。三个 workspace 加 2 槽维度,四 flag 改为按 workspace 配对的六个 ready/consumed 信号量,消费者在循环前预置一个槽位;两个 scope 在循环后各自排空预置的信号量,不让 cross flag 跨越 launch 存活。协议形式取自仓库内 examples/mha_sink_fwd_bhsd。

cube_time 与 mte1_time 一字未变,只有调度变了。

适用域是正确性边界,不是调优选择。 C 与 E 按 batch tag 常驻 L1,错开的循环会让 wave w 的 QK 把新 batch 的 C/E 载入待执行 PV 仍在读的缓冲。放开 batch == 1 需要第二个 C/E 槽位、约多占 144 KiB L1,必须在硬件上确认预算。测试里有一条负判据钉住它。

性能

口径:msprof op 单 kernel Task Duration,每格 3 次独立采集 × 8 次发射、丢弃首次,报中位数与半极差。FP16 / PTO。环境 910B3 / CANN 9.2.0 / torch_npu 2.7.1。

shape waves 优化前 本 PR 加速
H32 / Tq128 / Tk128 7 56.04 ±0.43 41.28 ±0.53 1.358×
H96 / Tq64 / Tk128 10 76.12 ±0.52 55.29 ±0.25 1.377×
H64 / Tq128 / Tk128 13 96.73 ±0.20 69.70 ±0.19 1.388×
H96 / Tq128 / Tk64 20 73.11 ±0.65 51.87 ±0.22 1.409×
H96 / Tq128 / Tk128 20 140.87 ±0.86 99.35 ±0.99 1.418×
H96 / Tq256 / Tk128 39 263.36 ±0.48 182.68 ±0.83 1.442×
B2 / H96 / Tq128 / Tk128 — 267.22 ±0.61 265.16 ±1.74 1.008×
H96 / Tq128 / Tk256 — 180.85 ±1.32 180.44 ±0.32 1.002×
H48 / Tq128 / Tk128 — 91.45 ±0.62 93.09 ±0.39 0.982×

加速随 wave 数单调上升(7→1.358 … 39→1.442);wave 数相同的两格(Tk128 与 Tk64,均 20)加速持平(1.418 与 1.409),尽管绝对耗时差近一倍。决定收益的是并行度而非工作量 —— 填充与排空各占一个 wave,开销为 1/waves。后三格在守卫之外,全部落在 1.00× 附近,域外零收益是设计结果。

逐步实测(同形):208.16 → 193.96 → 176.98 → 165.86 → 159.58 → 151.66 → 143.50 → 98.55 → 97.80 µs,最后两步即本 PR 的流水与阶段判定进位。前七步为单进程 pilot 快照,后两步为 3 次采集中位数,两者不可直接相减作为严格增量。

两种 dtype、两个后端,在提交状态的文件上实测(NPUGraph 捕获重放,10 次调用一张图、重放 20 次、3 次独立 run,attention 阶段):

dtype 后端 B1/Tq128/Tk128 B1/Tq64/Tk128 B2/Tq64/Tk128
FP16 PTO 92.68 ±0.09 79.29 ±0.27 145.21 ±0.13
FP16 AscendC 84.70 ±0.40 80.33 ±0.26 147.49 ±0.16
BF16 PTO 116.63 ±0.21 89.71 ±0.24 167.87 ±0.53
BF16 AscendC 97.08 ±0.20 90.98 ±0.62 168.91 ±0.79

BF16 比 FP16 贵、且只在流水跑着的实测 profile 上贵(PTO +26%、AscendC +15%),与 BF16 走 P 残差路径一致;AscendC 在该 profile 领先 PTO 9%(FP16)/17%(BF16),其余各行两后端收敛到 1.6% 以内 —— 两后端只在错拍生效处才有差别。

对官方参考的比例:仅 attention 核心范围,对 absorbed npu_fused_infer_attention_score_v2(v2 为当前较新接口,非兜底档),FP16 / PTO,NPUGraph 计时,8 个形状,两条臂都先对 FP32 参考过合格判据再计时(候选 1.2–2.4e-04、参考 3.4–6.3e-04,全部合格):

形状 block_n cache 轮次 入口 本 PR FIA v2 比例
B1 / H96 / Tq128 / Tk128 128 1 流水 92.55 69.10 74.9% ±0.5
B1 / H96 / Tq64 / Tk128 64 2 标量 139.80 49.39 35.3% ±0.2
B1 / H96 / Tq128 / Tk256 64 4 标量 449.65 82.74 18.4% ±0.3
B1 / H96 / Tq64 / Tk256 64 4 标量 264.49 56.80 21.4% ±0.1
B1 / H96 / Tq128 / Tk512 64 8 标量 935.48 119.75 12.8% ±0.2
B1 / H32 / Tq128 / Tk128 64 2 标量 82.44 50.71 61.5% ±0.2
B1 / H64 / Tq128 / Tk128 64 2 标量 140.87 57.88 41.1% ±0.0
B2 / H96 / Tq128 / Tk128 64 2 标量 393.41 96.45 24.5% ±0.1

整张表独立重跑一遍,每个比例复现在 1.2 个点以内。

FIA 只实现右下对齐 causal(其他模式被设备拒绝:Only support sparse 0/3/4/9 when ifa mla is enabled),本算子用 metadata[b,2](query_start)原生表达同一对齐,故每格设 query_start = kv_len - valid_q,两条臂算同一个掩码;否则比的是两个不同问题,Tq != Tk 的形状会显示归一化误差接近 1.0。

入口 列说明这些行之间不可互比:流水入口要求 cache 一轮覆盖且单 batch,而 public_prefill_block_n 只对实测过的那一个 profile 返回 128,故 8 行里 7 行走保守回退。固定形状、只动 block_n 才能把流水与形状分开:

形状 标量 block_n=64 流水 block_n=128 倍数
B1 / H96 / Tq128 / Tk128 204.54 ±0.10 92.98 ±0.09 2.20×
B1 / H96 / Tq64 / Tk128 141.94 ±0.40 54.31 ±0.09 2.61×

两个入口的归一化误差在所示位数上完全相同(1.2e-04/2.1e-04)—— 它们算的是同一件事。这是本改动的诚实度量;比例表的离散大部分来自选择器与 tile 上限。两条随之而来的结论:

  • block_n >= 256 被直接拒绝(unsupported attention tile),故 Tk > 128 时流水入口在结构上不可达,Tk256/Tk512 的低比例是上限而非调优缺口。
  • B1/H96/Tq64/Tk128 上出厂选择器白扔了 2.61×(合格、54.31 µs,而选择器给 141.94 µs)。本 PR 不放宽选择器:其契约刻意为"仅实测 profile",新打开的路径尚无测试覆盖,补齐覆盖需整条验证链重跑。记为第一条性能待办,测量已在手。

该比例与消融取自本 example 不随包携带的装置;benchmark 文档给出口径、形状、掩码约定与环境以便复现,不宣称本 example 自带它们。

不能把 Graph 比例表述为所有调用域达标:同口径的普通(非 Graph)调用由主机侧开销主导,对 FIA 约 66%,本优化按设计不改善它。

剩余方向:Cube 侧忙碌 89.28 µs 中 cube_time 仅 14.66 µs、mte1_time 29.07 µs,约 45 µs 既非算也非搬,指令发射与核间同步下发是候选。仓库内 examples/flash_attention/fa_opt 的阶梯将 60%→72% 归于"优化核间同步下发次数"、72%→80% 归于"用 axpy 减少指令数",两步本 PR 均未实施;A9 之后 Cube 为关键路径(Cube busy 89.28 µs 对 Vector 68.53 µs),前者优先级更高。

精度

误差下限:两个独立金标

模块带两个参考实现,不共享任何收缩顺序:吸收式(kernel 实现的那种,把 W_K 折进 query 后直接与 latent cache 收缩)与展开式(先把 latent cache 展开成逐头 K/V,再走常规注意力)。

吸收式  s = scale * ( (q @ w_k) · c  + qa · e )
展开式  k = w_k @ c ;  s = scale * ( q · k + qa · e )

两者代数等价(对 latent rank 的内层求和可按任一顺序取),但代码路径完全不同,所以一致性证明的是吸收式分解本身正确,而不只是 kernel 匹配了某一种写法。

CPU 实测(无需 NPU),按输出幅度归一:

用例 dtype max_abs 归一 ULP
实测 profile fp16 6.10e-05 1.95e-04 0.20
实测 profile bf16 2.44e-04 7.81e-04 0.10
逐行 mask 形状 fp16 6.10e-05 2.08e-04 0.21
短 query + 钳位 cache + 平移 fp16 3.05e-05 3.07e-04 0.31
两 batch fp16 6.10e-05 2.35e-04 0.24
只有一个可见 key fp16 1.22e-04 3.61e-04 0.37

最差 0.37 ULP,全部不到一个最低位。这是下限 —— 容差若定得比两个正确参考之间的差距还紧,量的就是参考而不是 kernel。下面的 rtol=2^-9、atol=2^-10 明显宽于该下限,是据此选定而非按实现输出凑出来的。

判据

test_example_gated_mla_prefill.py 共 28 项(其中 6 项为上述 CPU 金标互证),四类判据不同标准:

  • 预算判据:整链对模块内 FP32 参考,位置敏感 —— 先要求超出有效 query 长度的行精确为零,再对其余用 rtol=2^-9、atol=2^-10。覆盖 FP16/BF16 × PTO/AscendC。
  • 解析判据:令 QL = QA = 0 则可见 score 全为 0、可见 key 概率为 exp(0);令 latent_kv[k,:] = k 则 PV 累加为 Σ(0..n-1)、除以归一化因子后为 (n-1)/2。所有期望值都是 0.5 的整数倍,而 mask 多收或少收一个 key 恰好移动 0.5,故主判据为「四舍五入到最近 0.5 后精确相等」,没有容差可争论;另有一条 8 ULP of FP32 上界防止舍入掩盖漂移。覆盖整块与逐行两条 mask 路径、query_start 平移、kv_len 钳位。
  • 逐位相同判据:同批内各 batch 与单独计算必须逐位相同;空 batch 不得扰动相邻活跃 batch。这两条用容差会比不测更糟 —— 它们要抓的是 tile 越界写入相邻 batch,产生的是有界且看起来合理的小差值。
  • 负判据:非法形状、非连续(strided)操作数、错误 dtype 必须抛出具体异常并匹配消息片段,不接受笼统异常;以及流水入口在 batch ≥ 2 与多 KV 块时必须不被编译 —— 若哪天它开始被编译,说明文档所述域边界已变。

已知限制

  • 流水入口仅在 iterations == 1、heads % block_m == 0 且 batch == 1 时编译。前两条是守卫性质,第三条是正确性边界。域外形状走原实现,实测零收益。
  • block_n >= 256 被拒绝(unsupported attention tile),故 Tk > 128 时任何 block_n 都到不了流水入口;Tk256/Tk512 的低比例是可用 tile 决定的上限。
  • public_prefill_block_n 刻意比"能合格的范围"更窄:B1/H96/Tq64/Tk128 上流水 block_n=128 合格且快 2.61×,但选择器不选它。契约为"仅实测 profile"、新路径无测试覆盖,故本 PR 不放宽,列为第一条性能待办。
  • BF16 有计时、无比例:已给出 BF16 在两后端的耗时;不给比例是因为该 dtype 下没有能通过同等精度资格的外部参考。
  • 普通(非 Graph)调用无显著提速。
  • 对官方 FIA 的比例覆盖 8 个形状(FP16 / PTO、核心范围、Graph 计时,两臂均先过合格判据);AscendC 与 BF16 有计时无比例;性能表前半报的是优化前后加速比,与比例表回答不同问题。
  • FIA 对比要求两侧同为右下对齐 causal,故比例各格一律设 query_start = kv_len - valid_q;Tq > Tk 的形状无法对比。
  • 不含 decode 路径、cache 追加、packed 布局适配与完整模型集成。

变更类型

  • [Bugfix] Bug 修复
  • [Feature] 新功能/新算子
  • [Refactor] 重构
  • [CI] CI 或构建
  • [Skills] Skill/开发流程
  • [Testing] 测试
  • [Example] 示例/Benchmark

文件

本 PR 共 8 个文件,其中 6 个新增、2 个各改一行:

文件 内容
examples/gated_mla_prefill/README.md 入口:接口、用法、支持范围,并指向下面三份文档
examples/gated_mla_prefill/example_gated_mla_prefill.py 算子实现、两个独立金标与串联入口
examples/gated_mla_prefill/test_example_gated_mla_prefill.py 28 项精度/负向用例(6 项纯 CPU)
examples/gated_mla_prefill/design.md 设计文档:编程模式、片上预算、pass_configs、K3 语义对齐
examples/gated_mla_prefill/benchmark_gated_mla_prefill.md 基准报告
examples/gated_mla_prefill/benchmark_gated_mla_prefill_zh.md 中文版
ci/operator_test_manifest.yaml 登记一行
pyproject.toml codespell 忽略词表加 cann(见下)

pyproject.toml 这一行的理由:CANN 是昇腾工具链的名字,codespell 会把它判成 can
的拼写错误。不加这一行,bash format.sh 会在仓库原有的 4 处报错
(tilelang/tools/ascend_exception_dump_bin.py 三处、tilelang/transform/pass_config.py
一处),与本 PR 的文件无关。实测加与不加,本 PR 自己的 5 个新文件都不触发 codespell。
codespell 不在 GitHub Actions 门禁内,只在 format.sh 的本地流程里。

…ation 🤖

Add a fixed-domain AscendC Expert example with independent mathematical
references, device boundary tests, CI routing and reproducible benchmarks.
Replace elementwise readout tail masking with an aligned vector fill.

Validate 100 workspace tests, 25 core and 4 extended packaged tests on
910B3, and document timing, precision and remaining acceptance work.
@github-actions

Copy link
Copy Markdown

👋 Hi! Thank you for contributing to the TileLang project.

Please remember to run bash format.sh in the root directory of the project to ensure your changes are properly linted and formatted. This will help ensure your contribution passes the format check.

We appreciate you taking this step! Our team will review your contribution, and we look forward to your awesome work!

🚀

Match the Ruff 0.17.3 format check used by AscendC-PTO CI.
The computational Python AST is unchanged in all four files.
Ruff format, Ruff lint and git whitespace checks pass locally.
Load the full latent score tile, default to PTO with explicit AscendC support,
and add independent attention-core benchmarks and verified 910B3 evidence.
Synchronize with the current ascendc_pto runtime baseline.
Add the experimental FP16/BF16 prefill/decode package for default PTO and AscendC, with Cube projections, shared latent cache, packed adapters and independent FP64 references.

Register the existing core and extended suites, preserve the legacy decode tests, and add source-derived benchmarks and simulator/model evidence. All 145 core and 22 extended assertions pass locally; the 187.70-second cold core run still exceeds the 120-second acceptance budget. Keep ordinary-call performance and unqualified baseline limits explicit.
@dlaerbinjwy dlaerbinjwy changed the title [Example] Add experimental Gated MLA decode with prefix mask vectorization [Feature] Add multi-head Gated MLA forward for PTO and AscendC 🤖 Nov 11, 2026
@dlaerbinjwy

Copy link
Copy Markdown
Contributor Author

/re-test

@github-actions

Copy link
Copy Markdown

🔄 Re-running failed jobs

Original workflow run: View details

Only the failed jobs will be re-executed.

dlaerbinjwy and others added 6 commits November 12, 2026 00:18
Add independent-process benchmarks with six balanced method orders, separate Event/wall/graph distributions, per-boundary precision qualification and immutable input/source checks. Keep diagnostic comparators unscored and keep the existing numerical implementation and historical protocols unchanged.

Register 62 CPU regressions and attach the 36-trial, two-backend/two-dtype performance report, host attribution and four hardware Task snapshots. All candidate initial seed and post-round checks passed; ordinary-call performance limitations and unqualified BF16 references remain explicit.

Validation: 155 package CPU tests passed, 43 device cases deselected; strict collection has 229 unique cases. Ruff 0.17.4 lint/format passed. Numerical kernels were unchanged; this is not a new optimization-speedup claim.
…or work

Record source-pinned a172-parent comparisons, installed-artifact and hardware evidence.
Retain original BF16 model cross-official failures and local 120-second target as explicit boundaries.
…mple

Replace the installable package with one example file under examples/, matching
how operators are contributed in this repository. The operator code is the same:
example_gated_mla_prefill.py is a mechanical merge of the package's _contracts,
kernels/vector and kernels/prefill modules, with the two relative imports removed,
_contracts inlined whole so no transitive dependency is lost, and one byte-identical
EXPERT_PASSES kept once. The generated CPP is byte-identical across both entry
points, both backends and both dtypes, in the pipelined and fallback domains alike.

The forward runs the seven already-projected inputs through Q absorption, latent
attention, value recovery and the per-channel sigmoid gate, returning the result
before W_O. The layer is NoPE, so the 64-wide auxiliary cache carries plain key
channels; the output gate is enabled in the released Kimi K3 configuration.

The attention core keeps two entry points that agree bit for bit. Head-aligned
shapes whose cache fits one KV block, at batch one, compile main_pipelined, which
skews the Cube loop by one wave so a wave's QK is issued before the previous
wave's PV and the Vector softmax between them overlaps a Cube GEMM. Cube time
blocked on flags falls from 57.63 to 1.19 us and Vector from 66.95 to 23.28 us,
while cube_time 14.66 us and mte1_time 29.07 us are unchanged: only the schedule
moved. The batch bound is a correctness one, since C and E are L1-resident under
a batch tag; a negative test asserts the entry is not compiled outside it.

Single-kernel Task time over nine shapes improves by 1.358x to 1.442x inside the
domain, rising monotonically with wave count, and stays within 0.982-1.008x
outside it. Core-scope Graph timing reaches 75.07 percent of the official
fused_infer_attention_score_v2 on PTO and 82.86 percent on AscendC at
B1/H96/Tq=Tk128 FP16; ordinary non-Graph calls are host dominated and do not
improve. BF16 has no external reference passing the same accuracy qualification,
so no BF16 ratio is reported.

The test module covers 22 cases: a budgeted gate against an FP32 reference with
exact zeros outside the valid query length, an analytic causal-mask oracle whose
expected values are multiples of one half so a mask off by one key moves a row by
exactly that, bit-exact batch isolation, and negative gates for invalid shapes,
strided operands, wrong dtypes and the compiled domain itself.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@dlaerbinjwy dlaerbinjwy changed the title [Feature] Add multi-head Gated MLA forward for PTO and AscendC 🤖 [Example] Add Kimi K3 Gated MLA prefill forward Nov 13, 2026
dlaerbinjwy and others added 4 commits November 20, 2026 15:23
…ios, dtype coverage

Evidence only. The kernel region of the example is byte-identical to the
previous commit (sha256 of the first 1543 lines unchanged), so the CI result
already recorded for it still applies.

Correctness

  Add reference_gated_mla_prefill_expanded, a second reference that shares no
  contraction order with the first: it materialises per-head keys and values
  from the latent cache and runs ordinary attention, where the existing one
  folds W_K into the query and contracts against the cache directly. The two
  are algebraically equal but computed by different code, so their agreement
  is evidence about the absorbed decomposition rather than about one way of
  writing it. Six CPU-only cases pin that agreement at 0.37 ULP worst case
  (22 -> 28 tests), and the kernel tolerance is now justified as sitting well
  above that floor instead of being tuned to the kernel's own output.

  Extend __main__ from one configuration to five, each exercising a different
  property rather than a larger version of the same one, and assert the
  inactive query tail is exactly zero before applying any tolerance.

Performance

  Replace the single-working-point ratio with eight shapes, both arms
  qualified against an FP32 reference before either is timed. FIA implements
  right-down aligned causal masking only, so every cell sets
  query_start = kv_len - valid_q and both arms compute the same mask;
  comparing a top-left aligned candidate instead compares two different
  problems and shows a normalised error near 1.0 at every Tq != Tk.

  Add a block_n ablation that holds the shape still, which separates the
  pipeline from the shape: the pipelined entry is worth 2.20x at Tq128 and
  2.61x at Tq64, at identical normalised error. It also establishes two
  limits now recorded in the docs: block_n >= 256 is refused with
  "unsupported attention tile", so the pipelined entry is structurally
  unreachable for Tk > 128; and public_prefill_block_n is narrower than what
  qualifies, leaving 2.61x at B1/H96/Tq64/Tk128. Widening the selector is not
  done here, because its contract is deliberately "measured profile only" and
  the newly reachable path carries no test coverage.

  Add timings for both dtypes on both backends, measured on the file as
  submitted by NPUGraph capture and replay. BF16 costs 26% (PTO) and 15%
  (AscendC) more than FP16, and only where the pipeline runs, matching the
  BF16 P-residual path; the backends converge to within 1.6% off that entry.

  Note that the Tq64 rows of the two tables are not comparable: the kernel
  skips cache blocks the mask makes entirely invisible, so query_start = 0
  leaves one of two blocks live where query_start = 64 leaves both.

Manifest

  Drop two comments and two blank lines left behind by the removed package
  and place the example's entry under its own directory header.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…plicate constant

design.md

  Adds the design document the repository's other operators carry, following
  the ten-section structure of examples_experiment/cumsum_kda/design.md and
  labelling the programming mode as examples/fused_sigmoid_gating_delta_rule
  does. It covers what the benchmark documents do not: why Expert mode is
  required here (the cross-wave skew is hand-ordered, and an automatic CV sync
  pass inserts exactly the barrier that removes it), the on-chip budget per
  scope, the pass_configs and the reason each is off, and the mapping onto the
  Kimi K3 released configuration.

  Two numbers in it are worth surfacing. The L1 budget is 188 KiB of 512 KiB at
  the measured profile, and the second C/E slot that would lift the batch == 1
  boundary costs 144 KiB, so 332 KiB would still fit -- that boundary is an
  unverified gap rather than a capacity wall. And block_n >= 256 is refused
  with "unsupported attention tile", which is what puts a ceiling on Tk > 128
  rather than any budget.

CI cost and coverage

  Both benchmark documents now state what the gates cost: 200 s for the 28
  tests from a cold JIT cache, which is what CI starts from, and 233 s for the
  example's __main__. They also state which axes the shipped gates cover and
  which two are deliberately left to the benchmark tables -- cache lengths past
  128 and head counts other than 96 are qualified at 2.1e-04 in the ratio
  matrix, but no shipped case pins them, because each distinct shape costs a
  JIT compilation and none of those shapes can reach the pipelined entry.

Duplicate constant

  EXPERT_PASSES was defined twice, byte-identical, left over from merging the
  package's _contracts.py and kernels/*.py into one file; the second definition
  silently shadowed the first. Removed, leaving one definition and five uses.

  This touches the kernel region, so the previous commit's byte-identity
  argument no longer carries and the operator was re-verified on a 910B3:
  28 tests passed in 199.83 s, and the example's five configurations passed on
  both backends. ruff check and ruff format are clean on all five files.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…validator

Two leftovers from merging the package's _contracts.py and kernels/*.py into a
single file, found by an AST pass over top-level definitions:

  * DTYPES was defined twice, byte-identical, the second silently shadowing the
    first.

  * validate_chain_spans was defined and never called. In the package it was
    the chain entry's span check (gmla/_runtime.py called it), but in this file
    every check it performs is already done by the stage it delegates to:
    q_absorb calls validate_projection_spans, build_cube_attention calls
    validate_attention_spans, and each stage validates its own tensors through
    _tensor, which is why the four rejection cases pass without it. The one
    check unique to it covered an empty-cache output count. Removing it leaves
    no orphans: _positive, checked_elements, validate_projection_spans and
    validate_attention_spans all have other callers.

1639 -> 1598 lines. No behaviour change, but the kernel region moves, so the
operator was re-verified on a 910B3: 28 tests passed in 192.73 s and the
example's five configurations passed on both backends. ruff check and ruff
format are clean on all five files.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The directory had the design document and the benchmark reports but no entry
point, which is the one piece the better-equipped example directories here all
carry. This routes rather than repeats: the interface and the seven inputs, how
to run the example and the gates, the supported configurations and the domain
boundary, and a one-paragraph summary of accuracy and performance that points
at design.md and the benchmark reports for the rest.

Two things are stated here because a reader hits them first. metadata is a
runtime input rather than a compile-time parameter, so one compiled kernel
serves different valid lengths; and block_n comes from a host selector that
returns 128 only for the measured profile, with block_n >= 256 refused by the
compiler, which is what caps Tk > 128.

Documentation only; no code changes. ruff check and ruff format are clean on
all six files.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@dlaerbinjwy

Copy link
Copy Markdown
Contributor Author

/re-test

@github-actions

Copy link
Copy Markdown

🔄 Re-running failed jobs

Original workflow run: View details

Only the failed jobs will be re-executed.

@dlaerbinjwy

Copy link
Copy Markdown
Contributor Author

/re-test

@github-actions

Copy link
Copy Markdown

🔄 Re-running failed jobs

Original workflow run: View details

Only the failed jobs will be re-executed.

@dlaerbinjwy

Copy link
Copy Markdown
Contributor Author

/re-test

@github-actions

Copy link
Copy Markdown

🔄 Re-running failed jobs

Original workflow run: View details

Only the failed jobs will be re-executed.

@dlaerbinjwy

Copy link
Copy Markdown
Contributor Author

/re-test

@github-actions

Copy link
Copy Markdown

🔄 Re-running failed jobs

Original workflow run: View details

Only the failed jobs will be re-executed.

@dlaerbinjwy

Copy link
Copy Markdown
Contributor Author

/re-test

@github-actions

Copy link
Copy Markdown

🔄 Re-running failed jobs

Original workflow run: View details

Only the failed jobs will be re-executed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants