Conversation
submartingales
pushed a commit
to botcanlearn/torchtitan-upstream
that referenced
this pull request
Sep 21, 2026
DistMuon carries overlapping prepared-layout records and separate paths for native matrices and row-concatenated matrix batches. Resolve layouts once and build explicit matrix views after local compute ownership is known. Dense Muon helpers consume ordinary tensors and views; optimizer orchestration handles sharding and redistribution. Represent each view with shape, element strides, and a relative offset. Store the final views by parameter name and rebuild them after checkpoint loading. This keeps parameter normalization, the scalar BlockShard API, Kimi layouts, ownership rules, Newton-Schulz batching, and adjusted-LR application order unchanged. Variable block sizes and per-matrix scaling are introduced by the next PR in the stack. Validation: - CPU FlexShard: 8 existing tests passed. - GPU DistMuon: 2 tests passed with the original numerical assertions intact. - 160 native/uniform matrix cases produced exactly the same directions and final updates as the original implementation across FP32/BF16 and LR modes. - All 72 builder/direct-constructor input cases match the original base, including parameter-name types, generator handling, and group copying. - pre-commit run --all-files passed. - A 30-step Kimi debug run with FSDP=8/EP=8, --debug.seed=42, and --debug.deterministic matched the original base 6c2dadb at every loss, grad_norm, and learning-rate value. Both runs used PyTorch git1807a24. Maximum difference was zero; final loss was 2.655120611190796 and final grad_norm was 1.0288628339767456. ghstack-source-id: 9a4a64a Pull-Request: pytorch/torchtitan#4809
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stack from ghstack (oldest at bottom):
DistMuon carries overlapping prepared-layout records and separate paths for
native matrices and row-concatenated matrix batches. Resolve layouts once and
build explicit matrix views after local compute ownership is known. Dense
Muon helpers consume ordinary tensors and views; optimizer orchestration
handles sharding and redistribution.
Represent each view with shape, element strides, and a relative offset. Store
the final views by parameter name and rebuild them after checkpoint loading.
This keeps parameter normalization, the scalar BlockShard API, Kimi layouts,
ownership rules, Newton-Schulz batching, and adjusted-LR application order
unchanged. Variable block sizes and per-matrix scaling are introduced by the
next PR in the stack.
Validation:
final updates as the original implementation across FP32/BF16 and LR modes.
including parameter-name types, generator handling, and group copying.
--debug.deterministic matched the original base 6c2dadb at every loss,
grad_norm, and learning-rate value. Both runs used PyTorch git1807a24.
Maximum difference was zero; final loss was 2.655120611190796 and final
grad_norm was 1.0288628339767456.