Add FastWAM RoboDojo evaluation adapter - #1311
Closed
q6y6y6 wants to merge 310 commits into
Closed
Conversation
Co-authored-by: gushiqiao <975033167>
Co-authored-by: gushiqiao <975033167>
Co-authored-by: yihuiwen <yihuiwen@sensetime.com>
This PR adds kernel skills. --------- Co-authored-by: helloyongyang <yongyang1030@163.com>
Added news about the release of new models.
## Summary
This PR fixes the LTX-2.3 config checkpoint paths.
## Changes
- fix invalid `dit_original_ckpt` paths in:
- `configs/ltx2/ltx2_3.json`
- `configs/ltx2/ltx2_3_offload.json`
- `configs/ltx2/ltx2_3_distill_offload.json`
- `configs/ltx2/ltx2_3_distill_upsample_offload.json`
- fix invalid `upsampler_original_ckpt` path in:
- `configs/ltx2/ltx2_3_distill_upsample_offload.json`
## Why
The previous LTX-2.3 config paths were missing a `/` in the checkpoint
location, which would cause path validation / model loading to fail.
Co-authored-by: shihaobai <1798930569@qq.com>
Complete The LightX2V's Support To Matrix-Game-3. --------- Co-authored-by: Yang Yong (雍洋) <yongyang1030@163.com>
…odelTC#1028) Co-authored-by: Yang Yong (雍洋) <yongyang1030@163.com>
Co-authored-by: yihuiwen <yihuiwen@sensetime.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
Co-authored-by: yihuiwen <yihuiwen@sensetime.com>
Co-authored-by: yihuiwen <yihuiwen@sensetime.com>
This pull request introduces several new configuration files and significant updates to the disaggregated (disagg) connection logic and workload orchestration for the LightX2V project. The main focus is on supporting distributed inference with improved network handling, chunked data transfer, and dynamic workload simulation. The changes enhance reliability, configurability, and usability for running and testing disaggregated video inference pipelines. **Key changes:** ### New configuration and workload simulation * Added four new configuration files for disaggregated controller, encoder, transformer, and decoder modes, each specifying model parameters, quantization settings, RDMA protocol details, and distributed ranks. [[1]](diffhunk://#diff-440d18d9304ad8fbc166e4de7ec8269aa7219d6a6e5a88462cf0a716634bb1a5R1-R58) [[2]](diffhunk://#diff-d572d987f2b9dbc6221429d35d2ca572a229e075ad8ae71157ff4775216b6a7cR1-R58) [[3]](diffhunk://#diff-6fc0f74620b78b925329ce7d5642aa6c36646cd38186d876151ae3d6f6f6658bR1-R58) [[4]](diffhunk://#diff-a7a24a5c453ee5f04be754b15e016adb89aba716ab856b91a1489ebc303c1b0fR1-R58) * Introduced a workload staging configuration (`wan22_i2v_workload_stages.json`) to define warmup and change phases for dynamic load testing. * Added `run_user.py` example script to simulate dynamic user workloads, sending requests to the controller based on stage specifications and supporting configurable request rates. ### Disagg connection reliability and protocol improvements * Implemented `_normalize_loopback_host` to ensure consistent use of IPv4 loopback addresses, controlled by the `DISAGG_FORCE_IPV4_LOOPBACK` environment variable, improving local and mixed-protocol deployments. [[1]](diffhunk://#diff-a5e5778ac7adc9b8f2c175153e932db47158abebba325b47298001bc80e89ba7R85-R92) [[2]](diffhunk://#diff-a5e5778ac7adc9b8f2c175153e932db47158abebba325b47298001bc80e89ba7L544-R583) * Enhanced error handling in the transfer loop and status synchronization, logging exceptions and preventing crashes during data transfer and status updates. * Added support for chunked data transfer in `send_data`, controlled by the `MOONCAKE_TRANSFER_CHUNK_BYTES` environment variable, to handle large tensors more efficiently and robustly. ### Protocol and metadata updates * Updated ZMQ communication to include `receiver_engine_rank` in multipart messages for both encoder and transformer threads, ensuring correct routing and status updates in distributed settings. [[1]](diffhunk://#diff-a5e5778ac7adc9b8f2c175153e932db47158abebba325b47298001bc80e89ba7R364-R383) [[2]](diffhunk://#diff-a5e5778ac7adc9b8f2c175153e932db47158abebba325b47298001bc80e89ba7R392-R393) [[3]](diffhunk://#diff-a5e5778ac7adc9b8f2c175153e932db47158abebba325b47298001bc80e89ba7R439-R458) [[4]](diffhunk://#diff-a5e5778ac7adc9b8f2c175153e932db47158abebba325b47298001bc80e89ba7R467-R468) [[5]](diffhunk://#diff-a5e5778ac7adc9b8f2c175153e932db47158abebba325b47298001bc80e89ba7R600) * Improved local IP detection logic in `mooncake.py` to better select a non-loopback IPv4 address for outbound connections, enhancing compatibility in multi-host environments. ### New libs * locust These changes collectively improve the flexibility, reliability, and scalability of the disaggregated inference pipeline, making it easier to configure, test, and deploy in distributed environments.
Improve Hunyuan3D inference performance with reduced transformer sync overhead, FlashInfer MoE, fused QK RMSNorm, FP8 DiT support, and fused self-attention QKV projection. MoE FP8 remains disabled to avoid accuracy risk.
--------- Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> Co-authored-by: Yang Yong (雍洋) <yongyang1030@163.com>
Fix missing dense max_seqlen_q/kv propagation in the new Ulysses path, which caused some attention backends to fail after the sequence-parallel refactor.
Add MLU configs and launch scripts for single-card and 8-card S2V inference. Bind each distributed worker before CNCL initialization, make Gemma attention and scheduler RNG device configurable, and reduce the text feature masking memory peak. Co-authored-by: wushuo1 <540295877@example.com>
audio_mel_cpu_offload
1. mlu sdpa Nan 2. support replace by platform attn ops in gemma3 text encoder --------- Co-authored-by: wushuo1 <540295877@example.com>
ModelTC#1300) --------- Co-authored-by: liuhongda <liuhongda@sensetime.com>
handle single masked InfiniteTalk multi-role input
- Add Phased DMD and SGMD trainers with High/Low Student, Fake, and dual-model inference support. - Refactor the monolithic DMD trainer into separate config, role management, math, checkpoint, and trainer modules. - Extract CDM, Diversity, IDA, and RealDataFake into reusable Trainer Tricks. - Support independent Fake experts, backward-compatible checkpoint recovery, and new training configs. --------- Co-authored-by: Charles2530 <2569337619@qq.com> Co-authored-by: Cursor <cursoragent@cursor.com>
Contributor
|
Caution The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased. |
q6y6y6
marked this pull request as ready for review
July 30, 2026 02:25
Contributor
|
Caution The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
20 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changed
lightx2v_ros/src/robodojoament Python package with a reusable RoboDojo/XPolicyLab observation adapter forFastWAMPolicyWhy
This makes the native LightX2V FastWAM runtime directly reusable from the RoboDojo/XPolicyLab evaluation server, without falling back to the upstream FastWAM runtime. It also preserves the released action horizon/replanning behavior and supports the split asset layout used by the evaluation environment.
Evaluation status
The integration has completed end-to-end RoboDojo evaluation. Its current success rate is close to the upstream baseline: layout 0 succeeds, while layout 1 still fails. We suspect this layout-dependent behavior comes from the released official checkpoint; the current evidence is not sufficient to attribute the layout 1 result to the LightX2V integration.
Validation
python -m compileallfor the changed FastWAM modules and new adapter packagepython -m json.tool configs/fastwam/robotwin_i2va.jsonpython setup.py checkfor the new ament Python packagegit diff --checkpre-commitwas not available in the local environment, so repository hooks were not run.