Skip to content

[DistMuon] Simplify layout planning and matrix views - #4809

Closed
weifengpy wants to merge 4 commits into
gh/weifengpy/72/basefrom
gh/weifengpy/72/head
Closed

weifengpy wants to merge 4 commits into
gh/weifengpy/72/basefrom
gh/weifengpy/72/head

Conversation

@weifengpy

@weifengpy weifengpy commented Sep 20, 2026

Copy link
Copy Markdown
Contributor

Stack from ghstack (oldest at bottom):

DistMuon carries overlapping prepared-layout records and separate paths for
native matrices and row-concatenated matrix batches. Resolve layouts once and
build explicit matrix views after local compute ownership is known. Dense
Muon helpers consume ordinary tensors and views; optimizer orchestration
handles sharding and redistribution.

Represent each view with shape, element strides, and a relative offset. Store
the final views by parameter name and rebuild them after checkpoint loading.

This keeps parameter normalization, the scalar BlockShard API, Kimi layouts,
ownership rules, Newton-Schulz batching, and adjusted-LR application order
unchanged. Variable block sizes and per-matrix scaling are introduced by the
next PR in the stack.

Validation:

  • CPU FlexShard: 8 existing tests passed.
  • GPU DistMuon: 2 tests passed with the original numerical assertions intact.
  • 160 native/uniform matrix cases produced exactly the same directions and
    final updates as the original implementation across FP32/BF16 and LR modes.
  • All 72 builder/direct-constructor input cases match the original base,
    including parameter-name types, generator handling, and group copying.
  • pre-commit run --all-files passed.
  • A 30-step Kimi debug run with FSDP=8/EP=8, --debug.seed=42, and
    --debug.deterministic matched the original base 6c2dadb at every loss,
    grad_norm, and learning-rate value. Both runs used PyTorch git1807a24.
    Maximum difference was zero; final loss was 2.655120611190796 and final
    grad_norm was 1.0288628339767456.

[ghstack-poisoned]
@meta-cla meta-cla Bot added the CLA Signed This label is managed by the Meta Open Source bot. label Sep 20, 2026
[ghstack-poisoned]
[ghstack-poisoned]
[ghstack-poisoned]
@weifengpy weifengpy closed this Sep 21, 2026
submartingales pushed a commit to botcanlearn/torchtitan-upstream that referenced this pull request Sep 21, 2026
DistMuon carries overlapping prepared-layout records and separate paths for
native matrices and row-concatenated matrix batches. Resolve layouts once and
build explicit matrix views after local compute ownership is known. Dense
Muon helpers consume ordinary tensors and views; optimizer orchestration
handles sharding and redistribution.

Represent each view with shape, element strides, and a relative offset. Store
the final views by parameter name and rebuild them after checkpoint loading.

This keeps parameter normalization, the scalar BlockShard API, Kimi layouts,
ownership rules, Newton-Schulz batching, and adjusted-LR application order
unchanged. Variable block sizes and per-matrix scaling are introduced by the
next PR in the stack.

Validation:

- CPU FlexShard: 8 existing tests passed.
- GPU DistMuon: 2 tests passed with the original numerical assertions intact.
- 160 native/uniform matrix cases produced exactly the same directions and
  final updates as the original implementation across FP32/BF16 and LR modes.
- All 72 builder/direct-constructor input cases match the original base,
  including parameter-name types, generator handling, and group copying.
- pre-commit run --all-files passed.
- A 30-step Kimi debug run with FSDP=8/EP=8, --debug.seed=42, and
  --debug.deterministic matched the original base 6c2dadb at every loss,
  grad_norm, and learning-rate value. Both runs used PyTorch git1807a24.
  Maximum difference was zero; final loss was 2.655120611190796 and final
  grad_norm was 1.0288628339767456.
ghstack-source-id: 9a4a64a
Pull-Request: pytorch/torchtitan#4809
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CLA Signed This label is managed by the Meta Open Source bot.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant