perf(ds4): extend sparse prefill scheduling to 10K

[dusterbloom]

September 10 regression fix — 64463db

Partial fix; keep draft. The observed adaptive-acceptance regression is fixed.
The previous head's full prefill speed is not preserved, and the original
prose-quality gate remains unqualified. This is not merge-ready or a Vulkan
parity claim.

What changed

• HIP F32 prefill now shares KV loads across eight attention heads, with
  quad key loads, while preserving compact attention's sequential dot
  products, softmax tree, and ascending-row value sums.
• F32 fixtures now require byte-identical outputs against the compact
  reference, including all 64 model heads at a late-context tail. The old
  tolerance missed this regression.
• The adaptive controller, draft cap, fused verification, verifier F16 KV,
  expert count, weights, quantization, and CUDA/F16 attention policies are
  unchanged. Eight attention heads does not mean eight drafted tokens.
• No extra global scratch buffer or experimental profiler code is included.

Measured result on lucebox8

Same frozen 9,133-token five-key retrieval prompt, one wide prefill frontier,
six experts, 32K allocation, cold caches, concurrency one, two requests per
arm, natural 59-token outputs (cap 128). Adaptive DSpark is enabled with cap
four, not fixed width. Rates use full API stage time.

Mode	Previous-head prefill	Fixed prefill	Previous-head decode	Fixed decode
AR	269.103	253.780	19.976	19.993
Adaptive DSpark	261.522	249.882	31.820	34.423

All rates are tok/s. Previous-head values are earlier same-day production
receipts for 8c9faa374, retained below; they are not a fresh main or Vulkan
benchmark. A fresh matched normal/ROCprof capture also reproduced that
head's 90% acceptance / 16 rounds.

The fix restores 98% acceptance / 15 rounds in both repeats, versus
90% / 16 before. All final AR and adaptive responses pass strict 5/5
retrieval and the fixed response hash. AR decode is preserved.
Prefill is still about 4–6% below the previous head; do not combine the
old prefill rate with the new decode rate to claim a full win.

At 4K chunks, the same HIP kernel before the CUDA-scope/documentation cleanup
measured 228.754 prefill / 34.622 decode, also 98% / 15 rounds.
The cleaned AR 4K check measured 228.883 / 19.909.

Verification and remaining gates

• 533/533 selected CTests pass on gfx1151, including DS4, MMQ, grouped
  MMID, fused combine, maskless F16/F32, dispatch controls, and server units.
• F32 attention is byte-identical to compact at 10,240 tokens and a 129-token
  tail starting at 122,880, with three submissions and 16/64-head coverage.
  These are kernel fixtures, not full-model 128K qualification.
• Six focused tests pass again after restoring the final build; its HIP
  library is byte-identical to the artifact used for the full suite and final
  workflow checks:
  660e95f3c2f4649740a74f842a75f9cee1b68ac9e1c6228bde1d4e96230c2476.
• All completed model arms obey the unchanged 4 GiB headroom / 262144-page
  swap-out guard and 600-second timeout. No failed arm is counted as a gain.
• Tiled-score, transposed-key, 16-head, and barrier-reduction experiments
  were discarded because they did not improve the combined result. Their
  code and receipts remain retained outside the PR.
• Original prose quality is still an open gate: KLD 0.011536 versus frozen
  limit 0.011216. The original four-domain frozen tapes/harness have not
  been located; kernel/retrieval tests do not replace them.
• New-head CI has started and is not yet fully green. No merge, deployment,
  article publication, broad-workflow win, or new matched Vulkan claim.

Commit: 64463db74a861b4cf8428c65e88cec1b27627715.
Evidence: LB8 /data/pr684-regression-20260910.YPAP3H/; local
/private/tmp/pr684-regression-20260910.eA2qDz/REGRESSION-RESULTS.md.
The LuceGraph baseline has complete/zero-drop v1 output and native DS4 scope
attribution of 312077/312414 GPU dispatches, all on gfx1151 Agent 2.
Legacy DS4 emits no SeqEngine rounds, so the generic host HTML join is
unavailable; native GPU reports are retained without fabricated round IDs.

Previous-head results and original PR history (retained unchanged)

September 10 follow-up — current head 8c9faa374

Conflicts are resolved against main 7ad10a703; GitHub reports MERGEABLE, but this PR remains draft, not approved for merge. Fresh-head CI: run 34527436416, in progress at this update.

Fixes delivered

• Preserve both sides of the CI conflict: maskless/fused-combine tests and main's draft-top-K coverage.
• Share byte-identical immutable F32 mask inputs within their layer-major graph cache. The reusable owner-local pool removes per-layer quadratic host duplication without changing mask values, GPU tensor identities, or lifetime boundaries.
• Preserve main's F32 current-row prefill KV; extend the existing wave32 streaming kernel to F32 input instead of introducing an extra F16 rounding step. The verifier's independent F16 path is unchanged.
• Retain the already-integrated perf(ds4): fuse expert-major MoE route combine #658/perf(ds4): accelerate long context and add PFlash #664/perf(ds4): accelerate ROCm decode and long-context selection #667/perf(ds4): default mixed ROCmFP MMQ prefill on gfx1151 #683 correctness and performance work. No expert-count, weight-format, adaptive-controller, machine, or service change. No profiler instrumentation is in this PR.

Commits: 76e44ecc5, 4095c738f, 8c9faa374. Normal non-force push to the actual fork branch dusterbloom/lucebox-hub:perf/ds4-sparse-prefill-wide.

Fresh LB8 validation

ROCm 7.2.4, physical Radeon 8060S Strix Halo gfx1151, wave32; not R9700. Release HIP, graphs ON, affine OFF, four build workers.

533/533 selected CTests pass with no CTest skips. Five new immutable-pool tests also pass with ASan/UBSan. F32 compact/streamed oracle maximum errors: 8.3446503e-7 at 10,240 tokens and 4.1723251e-7 at a 129-token tail after position 122,880. Masked/analytical streaming outputs are byte-identical across three submissions. Tolerances are unchanged. These are kernel fixtures, not full-model 128K qualification.

Same-task model results

Same installed ROCmFPX MIX target and Q4RMFP4 DSpark draft, native six experts, 32K allocation, cold prefix/prefill caches, concurrency 1. Same frozen 9,133-token five-key retrieval prompt; two measured requests per arm, each naturally stops at 59 output tokens. Adaptive width is enabled with cap 4, not fixed width. Rates use total API stage tokens / total API stage time.

Current-head configuration	Prefill tok/s	Decode tok/s	Acceptance / rounds
AR, 4K chunks	242.206	19.931	—
AR, one wide batch	269.103	19.976	—
Adaptive, 4K chunks	242.519	32.131	90% / 16
Adaptive, one wide batch	261.522	31.820	90% / 16
Adaptive, compact control, 4K	216.720	34.598	98% / 15
Adaptive, compact control, wide	231.869	34.509	98% / 15
Adaptive, precise-exp control, 4K	236.467	32.036	90% / 16

4K means 4,096 + 4,096 + 941 tokens. Wide requests chunk 10,240 but actually processes one 9,133-token batch. Compact changes only GGML_CUDA_MLA_STREAM_TOPK=0; precise-exp changes only GGML_CUDA_MLA_STREAM_FAST_EXP=0. All seven arms exit zero without triggering the unchanged 4 GiB memory guard. All 14 measured responses pass strict 5/5 retrieval and the same fixed output hash.

The reproduced wide-batch memory failure is fixed for this task. Minimum MemAvailable is 5.495 GiB for wide AR (zero swap-out), 4.951 GiB for wide adaptive (18,377 swap-out pages), and 4.888 GiB for wide compact (34,972 pages). The previous candidate's same-task wide AR/spec attempts crossed the guard before producing output. No limit was lowered. This does not qualify every 10K or long-context memory shape.

Default adaptive performance is not yet fixed. The retained earlier-main 2f0eff051 receipt used 98% acceptance / 15 rounds and 33.900 tok/s; it is not a fresh timing of main 7ad10a703. Preserving F32 inputs alone or disabling fast-exp does not recover acceptance. F32 + compact attention does recover 98% / 15 rounds at both tested chunk sizes, but gives back prefill throughput. No default was changed to hide this tradeoff. Do not combine one configuration's prefill with another's decode and call it a single win.

Profiling and evidence boundary

A separate fresh LuceGraph/ROCprof build of merge-only 76e44ecc5+profile attributes 311,995 / 312,332 dispatches (99.89%) to five native DS4 scopes, all on verified gfx1151. One measured prefill contains 21.885 s GEMM and 7.499 s attention; one measured decode scope contains only ~44 ms attention. The old cache retained 1,486,880,768 bytes of F32 inputs at a 4K frontier, motivating immutable mask sharing.

Legacy DS4 produces complete/zero-drop JSONL but no SeqEngine scheduler rounds. Native DS4 trace scopes use trace-order IDs, not JSONL-correlated IDs; a generic host-round HTML join is unavailable. The valid reports are kernels.json/kernels.md and native Perfetto. Normal-run timings, not ROCprof overhead, supply throughput.

An initial incremental build reused stale graph/test objects after timestamp-preserving transfer. No model run used that build. Explicitly rebuilding all changed units produced the passing suite; both failed and successful build/test logs are retained.

Full receipts and exact runner on LB8:
/data/pr684-followup-20260910.emWWWR/.
Local report:
/private/tmp/pr684-followup-20260910.oPVHd0/FOLLOWUP-RESULTS.md.

Still required before merge/publication

• Resolve or qualify the default streaming/adaptive acceptance tradeoff across representative tasks.
• Requalify the historical frozen prose gate: KLD 0.011536 versus 0.011216. Its original frozen tapes/harness were not located in the inspected evidence; kernel/retrieval checks do not replace it. The failure remains recorded.
• Fresh matched Vulkan and full-model 128K+ measurements before any broad Vulkan-win or full-model byte-identity claim.
• Complete new-head CI/review. No merge or article publication was performed.

Earlier PR description and historical receipts — superseded by the current-head results above

Summary

• integrate current main 2f0eff051 including perf(ds4): fuse expert-major MoE route combine #658/perf(ds4): accelerate long context and add PFlash #664/perf(ds4): accelerate ROCm decode and long-context selection #667/perf(ds4): default mixed ROCmFP MMQ prefill on gfx1151 #683, retaining fused-combine alignment admission, MLA scratch barriers, saved verifier rows, runtime RoPE positions, and model-local mixed-MMQ policy
• keep main's already-merged fused MoE combine and its existing disable/trace controls; this is not an additional gain over current main
• extend current main's DS4 prefill schedule from two to five independent
  2,048-token numerical bands, admitting one physical frontier up to 10,240
  tokens
• derive long ratio-4 causal visibility in the HIP attention path instead of
  retaining and uploading its quadratic score mask
• pass authoritative long-prefill top-k rows directly, keep selected
  raw/compressed KV transport in F16, and stream each selected latent row once
  across sixteen wave32 heads
• preserve native top-6 routing, selected-row order, per-row reduction order,
  independent 2K attention bands, and chronological state commit
• preserve mrciffa's authorship on the three extracted streaming-attention
  commits from perf(ds4): default mixed ROCmFP MMQ prefill on gfx1151 #683
• preserve cheese-cakee's authorship on the fused-combine implementation and
  correctness commits from perf(ds4): fuse expert-major MoE route combine #658

Routing, expert matrix multiplies, sparse/exact prefill admission, and state
commit retain main's policies. Maskless ratio-4 prefill selects streaming
attention automatically when eligible, while other indexed shapes keep the
existing opt-in policy. Existing streaming, FP32 staging, fast-exp and fused
combine overrides are preserved; no new serving flag or private BWS policy
is added. GGML_CUDA_MLA_STREAM_TOPK=0 retains the compact maskless fallback.

September 10 integration fix

The old PR head predates important merged correctness fixes. This update
retains them, removes the duplicate fused-combine CMake target produced by
the overlap, and adds permanent HIP regression coverage:

• compact versus streamed attention at 10,240 tokens and a 129-token tail
  starting at position 122,880, with pinned inputs and three submissions;
• explicit-mask versus analytical-mask outputs and numerical tolerance
  against the compact reference, with no loosened tolerances;
• default streaming dispatch and the explicit disable switch;
• rejection of runtime-position or oversized (>512) top-k contracts on the
  fixed-position maskless path; masked verification keeps its wider support;
• fused-combine alignment and new maskless tests included in both HIP CI jobs.

Fresh integrated LB8 validation: Release HIP gfx1151, graphs ON, 504/504
selected CTests passed. At 10,240 tokens the compact/streamed maximum
absolute error is 1.2218952e-6; at the 129-token tail starting at 122,880 it
is 3.4272671e-7. Explicit-mask and analytical-mask streaming outputs are
byte-identical on all three submissions of each fixture. These are kernel
fixtures, not full-model long-context measurements.

Wide-batch memory qualification did not pass. On the same 9,133-token
retrieval task with the current main optimizations and installed target,
candidate AR and adaptive runs at requested chunk 10,240 both crossed the
unchanged 4 GiB host-headroom guard before producing output. Candidate AR at
chunk 8,192 also crossed the guard while handling the tail. The runner
terminated these attempts; no actual OOM was observed. Their partial timings
are not counted as gains. The safe-chunk regression check is reported
separately, not substituted for 10K qualification.

The older model timings and quality results below are historical receipts,
not measurements of this merged head. The recorded prose quality miss and
wider-batch memory qualification remain open. Keep this PR draft; no matched
Vulkan parity or full-model byte-identity claim is made.

Fresh matched 4K-chunk model regression

Same 9,133-token cold-prefix retrieval task, installed ROCmFPX MIX target,
six experts, same input token IDs, 32K context allocation, caches off, two
measured requests per arm. Both use effective chunks 4,096 + 4,096 + 941.
Rates use API-reported full stage time, not client decode windows.

Metric (tok/s)	Current main	Fixed #684	Change
AR prefill	225.073	246.246	+9.41%
AR decode	19.956	19.950	-0.03%
Adaptive prefill	223.143	247.476	+10.90%
Adaptive decode	33.900	31.779	-6.26%

All eight responses pass strict 5/5 retrieval, contain 59 tokens, and have
the same output hash. Mean request wall time improves from 43.541 to 40.054
seconds in AR and from 42.680 to 38.779 seconds in adaptive mode.

Adaptive decode is not preserved in this check. Adaptive width is enabled
with q cap 4. Main uses 15 rounds with 98% draft acceptance; the candidate
uses 16 with 90% in both repeats. Round wall times are similar, but the extra
round lowers decode throughput. The precise numerical cause of the acceptance
change is not isolated. This regression, wider-batch memory headroom, and the
historical prose-quality miss remain open before merge-readiness. No fixed
high-acceptance or unmatched Vulkan win is claimed.

Fix head 431721f1e; engine/test code 1b91332f6 (last commit is documentation
only), integrated main 2f0eff051. Full receipts on LB8:
/data/pr684-fix-20260910.HsTxSi/, including model-summary.json, the four
completed arms, three aborted candidate attempts, regression-tests.log,
and the exact scripts. The initial main/adaptive arm requested chunk 10,240;
main's 4,096 cap gives the same effective shape as the candidate's explicit
4,096 setting.

Historical fused-combine discriminator (already merged in #658)

A frozen F-C-F-C process bracket used the same full model, prompts, runtime
shape, and machine state. Four warm observations per arm produced:

Prompt	Legacy combine	Fused combine	Change
2,563 tokens	194.594 tok/s	202.578 tok/s	+4.103%
9,133 tokens	239.201 tok/s	250.925 tok/s	+4.901%

The exact same-binary comparison measured +3.270% at 2K and +4.923% at 8K.
The final flag-free build independently reached 204.044 and 252.362 tok/s.
Its server SHA-256 is
cc21d9f4257492f5ae46758088baaa0b459f43c0dcd3cfe280c13bd7cbf9c0dd.

The focused HIP microbenchmark improved from 1.876 to 0.240 ms at 401 rows
(7.807x), and from 10.374 to 1.158 ms at 2,048 rows (8.962x). Decode remained
within noise, so no decode gain is claimed.

That historical A/B used #658's selector before the old branch removed it.
The current integration retains main's selector and trace support.

Historical production workflow (before current-main integration)

Strix Halo gfx1151, 98.3 GB ROCmFPX MIX artifact, target-only sparse
prefill, native top-6 routing, fused autoregressive decode, Q4_0 K/V cache,
caches off, 9,133-token SWE-bench-shaped prompt, and up to 128 generated
tokens:

Build	Physical prefill	Warm prefill median	Warm decode median	Matched 128-token warm wall
Previous PR head 619413c	8,192 + 941	231.365 tok/s	19.005 tok/s	46.319 s
Scheduling-only receipt d6471f1	one 9,133-token step	252.175 tok/s	19.220 tok/s	42.797 s
Change	tail removed	+8.99%	+1.13%	-7.60%

The final two warm prefill samples were 252.09 and 252.26 tok/s. The matched
128-output-token warm sample saved 3.522 seconds. Decode code is unchanged;
its small movement is reported, not claimed as a decode optimization. Sparse
replay produced 47 tokens in one warm sample and 128 in the other, so whole
wall is compared only between matched 128-token samples.

The earlier clean-head 7,680-token receipt remains 254.921 tok/s, nominally
0.335% above the cited 254.07 tok/s Vulkan result at the same token count. The
prompts, artifacts, and quality protocols are not proven identical, so this PR
does not claim a cross-runtime record from that comparison.

Memory and execution shape

The ratio-4 analytic path removes retained masks from 21 attention layers:

• 6.562 GiB at 8,192 tokens
• 8.157 GiB at 9,133 tokens

That memory headroom permits the five-band frontier without a runtime memory
guard. The four full-vocabulary quality processes reported zero swaps; the
end-to-end runs had no OOM, managed-memory cliff, or system-manager failure.

Failure found by the real workflow

The first maskless implementation passed units but segfaulted on the real
9,133-token request. A GDB hardware watchpoint found the generic ring-mask
writer overwriting the layer-2 graph header: after omitting the layer-major
score mask, control incorrectly fell through to old-row logic that assumes a
saved ring unavailable in layer-major prefill.

The root-cause correction restricts that branch to
causal_batch && !layer_major_batch. It adds no fallback, allocation, flag,
or defensive guard. The corrected source crosses the former crash point and
passes the real workflow.

Historical qualification (before current-main integration)

Release HIP gfx1151, build concurrency capped at four. The full pinned-Halo
test_deepseek4_unit passes, including the ratio-4 boundary formula and a
focused explicit-mask versus analytic-mask GPU oracle with bit-identical
streaming-attention output. The fused-combine qualification also passes all
2x4 top-k/row/shared-output HIP cases, meta allocation/layout contracts, CPU/HIP
parity, legacy differential tolerance (1e-6), and a byte-exact signed-zero
case. test_ggml_meta_split_layout passes 17/17 checks.

The unchanged Q2048/Q8192/Q8192/Q2048 ABBA harness used four frozen real-domain
token tapes, 128 teacher-forced full-vocabulary rows, and 32 greedy rows per
arm:

Domain	Cross mean KLD	Same-arm noise	Cross top-1	Greedy prefix	Result
Prose/reasoning	0.011536	0.008973	92.97%	2	KL-only miss
Code/kernel	0.001257	0.001329	98.44%	25	pass
Tool/agent	0.001293	0.001353	97.66%	32	pass
Multilingual/structured	0.003537	0.003923	97.66%	21	pass

Code, tool, and multilingual passed the preregistered limits. Prose passed its
top-1 and greedy rules but measured KLD 0.011536 against the frozen
max(0.006, 1.25 * noise) limit of 0.011216, a 0.000320 miss. This is
consistent with the inherited sparse replay noise, but the sample remains
reported as failed: no threshold was changed and it was not rerun until
passing. The focused attention-kernel oracle is bit-identical.

Known limitation

Sparse prefill remains approximate and nondeterministic across identical
same-arm replays. State and logit hashes differ even within equal-width
controls. These results do not claim bitwise full-model determinism.

The 9,133-token timing fixture carries an agent-shaped coding transcript but
no actual tool schema and only a 128-token output budget. It directly exercises
the changed path; it is not a complete agent-task quality proof.

Scope

This PR contains public scheduling, selected-row dataflow, F16 transport,
generic HIP attention-kernel work, tests, and the narrow ring-mask root-cause
fix. It contains no private Behavioural Working Set mask, calibration, expert
reuse, slab, or placement policy. Temporary attribution telemetry was removed
before the final commit.

[dusterbloom]

Clean public-head validation for da67e47f:

• Built from a git archive export, excluding all private/staging working-tree edits.
• HIP Release, gfx1151, -j4; test_deepseek4_unit passed on the Halo device.
• Strix Halo performance/high policy, CPU boost on.
• 7,680 prompt tokens, sparse Q8192, Q8 K / Q4 V, fused decode, prefix/prefill caches disabled.
• Cold topology-build run: 33,256.8 ms.
• Warm runs: 30,122.4 and 30,131.6 ms.
• Warm midpoint: 254.921 tok/s.

This is +0.851 tok/s / +0.335% over the cited 254.07 tok/s Vulkan result at the same token count. All three clean one-token runs emitted The.

Caveat: matched token count and Strix Halo class, but not a proven identical prompt or model artifact. Sparse prefill itself remains approximate; this proves the performance gate, not full behavioral parity.