Skip to content

[Kimi K3] Add packed MXFP4 import and MX QAT recipe - #4791

Draft
elfiegg wants to merge 3 commits into
pytorch:mainfrom
elfiegg:pr/kimi-k3-mx-qat-4269
Draft

elfiegg wants to merge 3 commits into
pytorch:mainfrom
elfiegg:pr/kimi-k3-mx-qat-4269

Conversation

@elfiegg

@elfiegg elfiegg commented Sep 18, 2026

Copy link
Copy Markdown
Contributor

Summary

  • load packed Hugging Face tensor/scale pairs as virtual BF16 tensors through DCP without gathering all experts on one rank
  • validate the released checkpoint's quantization_config at load time and derive the packed parameter policy from its target/ignore metadata rather than hard-coding w1/w2/w3
  • add a generic Kimi grouped-expert converter and debug recipe using TorchAO's stateless MXFP4-weight/MXFP8-activation grouped-MM primitive
  • keep BF16 nn.Parameter masters and enable full-state DCP checkpoints for QAT continuation

Part of #4269. This draft depends on pytorch/ao#4917 for the model-independent numerical primitive.

Validation

  • 10 packed-reader metadata, aligned/unaligned slice, codec, and error-path tests
  • 3 Kimi adapter tests covering metadata-derived policy and the unquantized reader path
  • grouped-expert test verifies BF16 optimizer-visible masters, no packed/scales in state, TorchAO grouped-MM engagement, and finite STE gradients
  • linked PR verification passed on GB200 in Ptyche job 2851977
  • one-node EP4 packed-load/QAT/DCP fresh-process resume passed in job 2849711
  • two-node, 8-rank EP8 packed-load/QAT/DCP fresh-process resume passed in job 2849804; a post-refactor rerun is being prepared
  • the released 1.5 TiB checkpoint manifest and nine early/middle/late tensor decodes matched TorchAO exactly in Lyris job 3095771

Intentionally not claimed by this draft

  • distributed repacking/export to HF-compatible safetensors
  • the issue's 200-step convergence and compiled-performance budgets
  • completion of the currently queued 128-GPU released-checkpoint training evaluation (3100759)

Those remain follow-ups before #4269 can be closed.

@meta-cla meta-cla Bot added the CLA Signed This label is managed by the Meta Open Source bot. label Sep 18, 2026
@pytorch-bot

pytorch-bot Bot commented Sep 18, 2026

Copy link
Copy Markdown

The following ciflow label(s) have been added but CI has not been triggered yet because the workflows are awaiting approval:

  • ciflow/rl

Once a maintainer approves the workflows (scroll to the bottom of the PR page), the corresponding CI jobs will be triggered automatically. Please ping one of the reviewers if you do not have access to approve and run workflows.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ciflow/rl CLA Signed This label is managed by the Meta Open Source bot.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant