Conversation
weifengpy
force-pushed
the
pr/dist-muon-stacked-shard0
branch
from
September 21, 2026 05:27
35e26a8 to
90ed00f
Compare
weifengpy
force-pushed
the
pr/dist-muon-assignment-validation
branch
from
September 21, 2026 05:27
5152b20 to
0e81a9f
Compare
weifengpy
force-pushed
the
pr/dist-muon-stacked-shard0
branch
from
September 21, 2026 05:33
90ed00f to
a13c8f7
Compare
weifengpy
force-pushed
the
pr/dist-muon-assignment-validation
branch
from
September 21, 2026 05:33
0e81a9f to
55f9e03
Compare
weifengpy
force-pushed
the
pr/dist-muon-assignment-validation
branch
from
September 21, 2026 07:23
55f9e03 to
4fd21a0
Compare
weifengpy
force-pushed
the
pr/dist-muon-stacked-shard0
branch
from
September 21, 2026 07:23
a13c8f7 to
0a2a8a8
Compare
weifengpy
force-pushed
the
pr/dist-muon-assignment-validation
branch
from
September 21, 2026 19:43
4fd21a0 to
aa37c86
Compare
A DistMuon selection regex can match some parameters while silently leaving others with compute layouts assigned to AdamW. The existing parameter-coverage check accepts that configuration because every parameter still has an optimizer. Require every locally trainable parameter with a DistMuon compute layout to be assigned to DistMuon. Otherwise, raise ValueError naming the parameter and its assignment before constructing optimizers for that model part. Also catch mapped parameters omitted from every group, including when no DistMuon group exists. Canonicalize checkpoint-wrapped names and ignore frozen parameters and entries for other pipeline stages. The optimizer container collects assignments and dispatches registered validators; the DistMuon-specific rule lives beside DistMuon. Keep existing string identifiers, taking DistMuon's identifier from `DistMuon.__name__`. Stacked on #4811 to review the optimizer-container validation separately from Kimi's parameter selection and Shard(0) compute changes. Validation: - All 23 existing optimizer parameter-group tests pass on the parent and this change. No test files are modified. - Standalone checks reject the old w[123] regex, metadata with only AdamW groups, and mapped unassigned parameters before optimizer construction. Wrapped valid assignments and frozen/off-stage entries are accepted. - Ten seed-42 deterministic CPU steps with mixed Adam/AdamW match the parent bitwise for loss, gradient norm, parameters, and optimizer state. - A previous eight-GPU deterministic C4 comparison of the valid Muon configuration with and without this startup validation matched all 10 full-precision training losses, gradient norms, and seven held-out losses. - Full pre-commit reports the same 30 existing Pyrefly errors; all other hooks pass. Focused type checks pass.
weifengpy
force-pushed
the
pr/dist-muon-assignment-validation
branch
from
September 21, 2026 19:46
aa37c86 to
b31e4bf
Compare
weifengpy
force-pushed
the
pr/dist-muon-stacked-shard0
branch
from
September 21, 2026 19:46
11dd361 to
074332c
Compare
weifengpy
added a commit
that referenced
this pull request
Sep 21, 2026
…#4811) w13 didn't get any fqn match after splitting w123 into w13 and w2. this PR fixes it: * #4535 * https://github.com/pytorch/torchtitan/pull/4676/changes#r4058499709 * same problem first raised in #4531 by @WhoisZihan the next follow up is adding validation prevent future cases like above in optimizer init: #4816 Co-authored-by: Zihan Yang <whois.zihan.yang@gmail.com> Co-authored-by: Zihan Yang <whois.zihan.yang@gmail.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A DistMuon selection regex can match some parameters while silently leaving others with compute layouts assigned to AdamW. The existing parameter-coverage check accepts that configuration because every parameter still has an optimizer.
Require every locally trainable parameter with a DistMuon compute layout to be assigned to DistMuon. Otherwise, raise ValueError naming the parameter and its assignment before constructing optimizers for that model part. Also catch mapped parameters omitted from every group, including when no DistMuon group exists. Canonicalize checkpoint-wrapped names and ignore frozen parameters and entries for other pipeline stages.
The optimizer container collects assignments and dispatches registered validators; the DistMuon-specific rule lives beside DistMuon. Keep existing string identifiers, taking DistMuon's identifier from
DistMuon.__name__.Stacked on #4811 to review the optimizer-container validation separately from Kimi's parameter selection and Shard(0) compute changes.
Validation: