Skip to content

perf(EPv2/gfx1250): retune EP4 topk-9 fp4 combine; un-shadow the topk-9 dispatch row - #737

Merged
jhchouuu merged 8 commits into
ROCm:mainfrom
jhchouuu:jiahzhou/gfx1250-topk9-tuning
Oct 12, 2026
Merged

perf(EPv2/gfx1250): retune EP4 topk-9 fp4 combine; un-shadow the topk-9 dispatch row#737
jhchouuu merged 8 commits into
ROCm:mainfrom
jhchouuu:jiahzhou/gfx1250-topk9-tuning

Conversation

@jhchouuu

@jhchouuu jhchouuu commented Oct 10, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator

Summary

Tuning tables and HIP lookup key for gfx1250 EP4 topk 9: DeepSeek-R1 with its shared expert dispatched as a ninth column.

  1. Every topk-9 table swept. Each bucket is set to the fewest blocks within 3% of the best geometry under both random and grouped routing, the rule the topk-6 rows already follow. That leaves the CUs a kernel does not need free for whatever runs beside it. 256 blocks remain only where nothing smaller comes within 5%:
    • fp4 dispatch: 64x8 to 512, 128x8 to 1024, 192x8 to 1536, 256x8 to 2048, 192x24 to 8192, 192x32 above
    • fp4 push combine: 64x8 to 128, 192x8 to 512, 256x8 to 1024, 192x16 to 1536, 256x16 to 2048, 192x24 above
    • bf16 pull combine (untuned until now, 64x8 throughout): 64x4 to 64, 64x8 to 128, 128x8 to 256, 256x8 to 1024, 192x8 above
  2. HIP's lookup is keyed by what dispatch transports.
    • HIP tunes dispatch and combine in separate tables. The dispatch half is keyed by the dispatch dtype, the combine half by quant_type.
    • The op layer used to hand HIP EpDispatchCombineConfig.dtype_str, which is FlyDSL's key. It names both halves ("fp4_disp_bf16_comb"), so it reads as if HIP's dispatch geometry depended on the combine dtype.
    • The dispatch table keys are now "bf16" / "fp8" / "fp4". This covers every row, including the mi355x EP8 and gfx1250 EP8/EP16 ones.
    • The op layer passes the new EpDispatchCombineConfig.dispatch_dtype_str. dtype_str stays FlyDSL's key.
    • lookup() still accepts "fp4_disp_bf16_comb" as an alias, for callers that pass dtype_str (aiter's MegaMoE).
    • The rename changes no lookup. Passing dtype_str and passing dispatch_dtype_str give the same result for 700 lookups: 5x5 dispatch/combine dtypes x 7 (device, shape) keys x 4 quant types, 0 differ.
    • Against main, every lookup outside gfx1250 EP4 topk 9 returns what it did before. That covers 112 lookups, with main given dtype_str and this PR given dispatch_dtype_str or the alias.

bench_ep.py also gets ROUTE=grouped: group-limited routing (DeepSeek's EXPERT_GROUPS=8, TOPK_GROUPS=4), with SHARED_EXPERTS=1 trailing column pinned to the token's own rank. Under ROUTE=ring with topk >= world, every token reaches all 4 ranks. Under grouped, a token reaches 2-3 ranks and the ranks receive unevenly: 3072/4608/4608/3072 rows at 1536 tokens. The ring comment no longer describes it as a serving placement.

Lookup: main vs this PR (topk 9, gfx1250 EP4)

Geometry is block x warp; "to N" is the bucket's upper token bound.

lookup main this PR
fp4 dispatch ("fp4"; main's key "fp4_disp_bf16_comb") 64x8 to 512, 64x16 to 1024, 128x16 64x8 to 512, 128x8 to 1024, 192x8 to 1536, 256x8 to 2048, 192x24 to 8192, 192x32
fp4 push combine (quant_type="fp4_blockwise") 64x8 to 256, 64x16 to 2047, 64x24 to 2048, 128x24 to 16384, 64x16 64x8 to 128, 192x8 to 512, 256x8 to 1024, 192x16 to 1536, 256x16 to 2048, 192x24
bf16 pull combine (quant_type="none") 64x8 64x4 to 64, 64x8 to 128, 128x8 to 256, 256x8 to 1024, 192x8

The bf16/fp8 dispatch rows are unchanged, and so are the fp8 push combine table, topk 6, topk 8, EP8 and EP16.

Measurements

Setup:

  • 4x gfx1250 on one host (EP4), mori 76d0b301 + this PR, bench_ep.py, HIP graph.
  • hidden 7168, topk 9, 65 experts a rank, fp4 dispatch.
  • Both halves are pinned through DBN/DWPB/CBN/CWPB for every point, since an unpinned half falls back to the single-shot default. Each leg is timed with hip events inside the graph.
  • 4 to 16384 tokens a rank. Every point passed CHECK=1.
  • Random routing over every geometry (16 dispatch, 12 fp4 combine, 10 bf16 combine), and grouped routing over the candidates.
  • bf16 combine ran under random routing twice; the two rounds agree within ~1%.
  • Bold is the geometry the table picks for that row.

Worst slowdown of a pick against the best geometry, 16+ tokens a rank:

  • fp4 dispatch: 5.4% (16384, grouped); 2.9% under random routing
  • fp4 combine: 4.6% (1536, grouped); 1.0% under random routing
  • bf16 combine: 7.6% (16, random; ~1 us). No one geometry is good at 4, 16 and 64 together, and 64x4 is the best at 4 and 64.

fp4 dispatch, us (random / grouped), combine pinned at 256x16:

tokens/rank 64x8 128x8 192x8 256x8 192x24 192x32 256x32
512 28.9 / 21.4 29.3 / 21.6 29.0 / 21.7 28.9 / 21.7 35.7 / 32.8 35.8 / 32.9 35.1 / 33.5
1024 49.5 / 47.9 33.3 / 24.6 33.2 / 24.8 33.2 / 24.8 38.7 / 35.0 39.0 / 36.0 39.4 / 35.3
1536 57.3 / 47.6 52.3 / 49.7 37.5 / 27.7 37.5 / 27.7 42.0 / 30.2 42.8 / 37.6 43.4 / 38.4
2048 60.0 / 47.3 53.6 / 50.1 56.3 / 52.6 40.7 / 30.5 42.6 / 34.5 45.5 / 37.1 46.5 / 40.0
4096 108.3 / 90.8 68.6 / 52.5 72.0 / 54.4 68.3 / 56.3 55.7 / 47.5 57.6 / 46.2 57.3 / 46.4
8192 200.1 / 184.5 124.7 / 98.3 116.6 / 100.3 98.5 / 75.6 91.3 / 78.2 92.7 / 85.4 88.3 / 75.5
16384 372.1 / 369.8 234.1 / 196.2 202.6 / 171.3 187.6 / 146.1 161.7 / 137.6 157.9 / 131.0 153.4 / 124.5

fp4 push combine, us (random / grouped), dispatch pinned at 128x16:

tokens/rank 64x8 192x8 256x8 192x16 256x16 192x24
128 14.6 / 14.0 14.5 / 13.9 14.6 / 13.9 15.2 / 14.6 15.4 / 14.5 21.0 / 19.5
512 21.1 / 18.7 17.8 / 15.8 17.9 / 15.8 18.6 / 16.4 18.6 / 16.2 24.5 / 23.4
1024 35.3 / 30.1 25.9 / 21.2 23.8 / 19.7 25.8 / 19.6 25.0 / 19.4 30.2 / 26.0
1536 50.7 / 40.2 31.7 / 26.1 31.8 / 25.7 31.2 / 25.0 30.9 / 23.9 37.1 / 30.5
2048 67.5 / 52.8 39.4 / 32.2 36.6 / 29.9 39.5 / 31.7 36.5 / 29.0 39.8 / 34.7
4096 122.7 / 95.5 77.8 / 61.9 70.5 / 58.3 75.9 / 53.5 74.7 / 53.2 60.5 / 51.5
16384 451.0 / 317.7 297.6 / 189.8 273.7 / 168.8 285.1 / 171.9 283.3 / 161.7 158.5 / 132.6

bf16 pull combine, us (random / grouped), dispatch pinned at 128x16. 192x8 is slow at 256-1024 under both routings, which keeps 256x8 there:

tokens/rank 64x4 64x8 128x8 256x8 192x8
4 14.4 / 13.6 21.5 / 15.5 20.1 / 14.5 20.2 / 14.7 20.2 / 14.8
16 15.5 / 14.6 15.7 / 14.7 14.6 / 13.9 23.1 / 15.9 14.4 / 13.8
64 14.8 / 14.1 15.8 / 14.7 16.6 / 15.1 16.6 / 15.2 20.9 / 16.1
256 29.1 / 28.2 24.8 / 18.4 21.3 / 17.5 22.3 / 18.1 41.0 / 29.6
512 46.5 / 45.7 33.8 / 31.0 31.4 / 23.2 28.1 / 22.6 58.3 / 42.9
1024 79.6 / 80.9 53.0 / 50.2 44.5 / 36.5 45.2 / 33.8 47.9 / 36.2
1536 112.2 / 116.1 71.8 / 69.0 59.0 / 48.2 60.7 / 44.4 58.2 / 43.7
16384 1048.0 / 1121.6 599.2 / 609.7 471.0 / 369.5 479.1 / 388.8 485.6 / 338.3

Test status

black / ruff (repo versions) and pre-commit pass. lookup() was evaluated offline for every key above.

🤖 Generated with Claude Code

jhchouuu and others added 3 commits October 10, 2026 07:01
@jhchouuu @claude
perf(EPv2/gfx1250): retune EP4 topk-9 fp4 combine; un-shadow the topk…
29ab84d 
…-9 dispatch row

ROCm#725 and ROCm#726 each added a (4, 7168, 9, None) key to _DISPATCH_TABLE, so the
later dict entry silently replaced the earlier one: ROCm#725's block-256 schedule
never took effect, and a bf16-keyed lookup for topk 9 fell back to the 64x16
single-shot default. Merge the two into one entry (keeping ROCm#725's row as the
default) and drop the same duplicate from _COMBINE_TABLE.

fp4 dispatch at topk 9 now uses 256x8 up to 4096 tokens and 128x16 past it.
The fp4 push combine at topk 9 moves to 256x16 up to 2048 tokens and 256x24
up to 16384: in MegaMoE nothing runs beside the combine, so the whole grid
pays (1536 tokens: 47.8 -> 28.8 us; 16384: 243.4 -> 156.1 us).

bench_ep gains ROUTE=grouped, ATOM's --fake-eplb placement for group-limited
routing (DeepSeek) with a shared column pinned to the own rank; ROUTE=ring only
matches models without group-limited routing.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@jhchouuu @claude
perf(EPv2/gfx1250): 192x8 bucket for topk-9 fp4 dispatch up to 1536; …
064bb77 
…spell out ROUTE=grouped

192x8 holds one token a warp up to 1536 tokens, the decode batch, and beats
256x8 there (33.5 vs 35.3 us, grouped routing); past 1536 it would give warps a
second token, so 256x8 keeps 1537..4096.

ROUTE=grouped now names every quantity after ATOM's own comments and reads
EXPERT_GROUPS / TOPK_GROUPS / SHARED_EXPERTS; the routing it builds is
unchanged (checked element for element against the previous version).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@jhchouuu @claude
refactor(EPv2): key HIP dispatch tuning by the dispatch dtype alone
8262296 
The HIP dispatch and combine tables are independent, yet the dispatch table was
keyed by EpDispatchCombineConfig.dtype_str, which names both halves
("fp4_disp_bf16_comb"). On HIP the combine half is always bf16 rows (its wire
format is quant_type's table), so the key now reads "fp4". lookup() maps the
old string to it, so the op layer and out-of-op callers that pass dtype_str get
the same schedules (72 lookups across every device/key/quant compared equal).
FlyDSL's table, which does need both halves, keeps its keys.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@jhchouuu

jhchouuu commented Oct 10, 2026

Copy link
Copy Markdown
Collaborator Author

/ci-rerun

@github-actions

github-actions Bot commented Oct 10, 2026

Copy link
Copy Markdown
Contributor

Re-running MoRI CI test and CCO CI test against the current main.

jhchouuu and others added 5 commits October 10, 2026 15:08
@jhchouuu @claude
refactor(EPv2): pass HIP's tuning lookup the dispatch dtype, not dtyp…
5c35680 
…e_str

HIP tunes dispatch and combine in separate tables: the dispatch half is keyed
by what dispatch transports, the combine half by quant_type. The op layer
still handed it EpDispatchCombineConfig.dtype_str, FlyDSL's key, which names
both halves ("fp4_disp_bf16_comb") and reads as if HIP's dispatch geometry
depended on the combine dtype.

Add EpDispatchCombineConfig.dispatch_dtype_str ("fp4" / "fp8" / "bf16") and
use it for the HIP lookup; dtype_str stays FlyDSL's key. lookup() documents
dtype as the dispatch wire and keeps "fp4_disp_bf16_comb" only as an alias
for callers that pass dtype_str (aiter's MegaMoE). Every HIP lookup returns
the same schedule as before: 5x5 dispatch/combine dtypes x every table key x
every quant type, 450 lookups, 0 differ.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@jhchouuu @claude
perf(EPv2/gfx1250): sweep every topk-9 table: fp4 dispatch, fp4 and b…
b0943c3 
…f16 combine

Re-swept on main + this PR (4x gfx1250, EP4, hidden 7168, topk 9, 65 experts a
rank): bench_ep.py, HIP graph, per-leg hip events, both halves pinned, 4 to
16384 tokens a rank, random and grouped (ATOM fake-EPLB) routing; every point
passed CHECK. Every bucket below is within ~3% of the best under both routings.

- fp4 dispatch: 64x8 to 512, 256x8 to 2048, 256x24 to 4096, 256x32 above
  (16384: 153.4/124.5 us random/grouped, against 128x16's 164.0/124.3).
- fp4 push combine: 256x8 to 1024, 256x16 to 2048, 192x24 above (16384:
  158.5/132.6 us, against 256x24's 167.8/133.2).
- bf16 pull combine, untuned until now (64x8 throughout): 128x8 to 256, 256x8
  to 4096, 192x8 above (1536: 60.6/44.4 us against 71.8/69.0; 16384:
  484.7/338.3 against 599.0/609.7).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@jhchouuu @claude
perf(EPv2/gfx1250): topk-9 tables take the fewest blocks within 3%
cd3cd3b 
Re-picked every topk-9 bucket by the rule the topk-6 rows already follow: the
fewest blocks within 3% of the best under both random and grouped routing, so
the CUs a kernel does not need stay free for whatever runs beside it. 256 blocks
remain only where nothing smaller comes within 5%. Swept 192x24/192x32 dispatch,
128x8/192x8 fp4 combine and 192x16 bf16 combine as well, and the lower-block
candidates under grouped routing.

- fp4 dispatch: 64x8 / 128x8 / 192x8 to 512 / 1024 / 1536, 256x8 to 2048,
  192x24 to 8192, 192x32 above (1536 takes 192x8, which ties 256x8).
- fp4 push combine: 64x8 to 128, 192x8 to 512, 256x8 to 1024, 192x16 to 1536,
  256x16 to 2048, 192x24 above.
- bf16 pull combine: 64x4 to 64, 64x8 to 128, 128x8 to 256, 256x8 to 1024,
  192x8 above.

Worst slowdown against the best geometry, 16+ tokens: dispatch 5.4% (16384,
grouped), fp4 combine 4.6% (1536, grouped), bf16 combine 6.5% (16, grouped).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@jhchouuu @claude
Merge origin/main into jiahzhou/gfx1250-topk9-tuning
d719696 
Conflicts in hip_tuning_configs.py, resolved to this branch's tables:
- topk-9 fp4 dispatch: this branch's sweep (main kept ROCm#726's row under
  "fp4_disp_bf16_comb"; main also un-shadowed the ROCm#725 row, as this branch did).
- topk-6 fp4 dispatch: same buckets, keyed "fp4" as on this branch.
- the combine-table comment: main's (EP8 and EP16).

Main's new EP16 dispatch row is re-keyed from "fp4_disp_bf16_comb" to "fp4",
since lookup() now reads that spelling as "fp4"; under the old key the row
would never match. Checked against main over every table key, dispatch dtype
and quant type (112 lookups): every key but topk 9 returns what main returns
for "bf16", "fp8" and "fp4_disp_bf16_comb".

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@jhchouuu @claude
docs(EPv2): drop measurements from the topk-9 table comments and ATOM…
65dac86 
… from bench_ep

hip_tuning_configs.py: the topk-9 rows keep their geometry; the sweep tables and
the reasoning around them leave the comments. The bf16 combine row's comment
says what the row is for, since the old one ("mirrors topk 8, only the dispatch
half is retuned") no longer holds.

bench_ep.py: ROUTE=ring and ROUTE=grouped describe the placement they generate,
without naming the serving stack it came from.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@jhchouuu
jhchouuu marked this pull request as ready for review October 12, 2026 05:21
@jhchouuu
jhchouuu requested a review from zhangfei829 October 12, 2026 06:28
@jhchouuu
jhchouuu merged commit f39fa76 into ROCm:main Oct 12, 2026
15 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@zhangfei829 zhangfei829 Awaiting requested review from zhangfei829

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@jhchouuu