[Backport release/3.0.0] Add TorchRL environment wrapper for isaaclab_rl (#7502)

[StafaH]

Description

Backports #7502 to release/3.0.0.

This adds isaaclab_rl.torchrl.IsaacLabTorchRLWrapper, TorchRL PPO training and playback entrypoints, the torchrl dependency extra, Cartpole agent configuration, API and workflow documentation, and the original regression coverage and changelog fragments. It also preserves the terminal-observation fix for converted multi-agent environments.

The merged source commit 5c91e9b1e3c79933b3e62c494fb596ab7b7031dd was applied onto release commit be270ed88bbc0ea46201ce5cdd4cd0a502971cbc. The only conflict was in uv.lock; it was resolved using the package versions from the release manifests while retaining the new TorchRL dependency graph. uv lock --check passes.

Type of change

• New feature (non-breaking change which adds functionality)
• Documentation update

Release backport

This pull request targets release/3.0.0 directly; no further backport is requested.

Validation

• TorchRL wrapper integration and actor tests: 5 passed across manager-based and direct Cartpole tasks, including finite- and infinite-horizon behavior.
• Focused MARL, TorchRL actor, entrypoint, and typed preset tests: 19 passed.
• uv lock --check: passed.
• Release-based changelog validation: passed for all three touched source packages.
• uv run isaaclab -f: all formatting, lint, metadata, and repository hooks passed. The changelog hook was run separately against release/3.0.0 because its default base is develop.
• Warning-as-error documentation build: passed with uv run --isolated --extra dev -- make -C docs current-docs. The release branch places Sphinx in the dev extra rather than the test extra.
• git diff --check: passed.

Screenshots

Not applicable.

Checklist

• I have read and understood the contribution guidelines
• I have run the pre-commit checks
• I have made corresponding changes to the documentation
• My changes generate no new warnings
• I have preserved tests that prove the feature works
• I have included a changelog fragment for every touched source package
• The original contributor is included in CONTRIBUTORS.md

[StafaH]

run-ci

[greptile-apps]

Confidence Score: 4/5

The PR should not merge until TorchRL training either implements checkpoint restoration or rejects checkpoint-bearing unified requests before they are forwarded to Hydra.

Findings

1.  Checkpoint Resume Is Unsupported ▶
2.  TorchRL Badge Is Missing ▶

Summary

This backport adds TorchRL as an optional reinforcement-learning backend, including a TorchRL EnvBase adapter, PPO reference runner, unified train/play entrypoints, Cartpole configurations, dependency metadata, terminal-observation support for converted multi-agent environments, tests, and documentation.

• Preserves batched observations, separate termination signals, and terminal observations across the Isaac Lab/TorchRL boundary.
• Registers TorchRL with unified dispatch and Cartpole task metadata.
• Adds an optional TorchRL dependency graph and corresponding CI coverage.
• The training path still needs checkpoint-resume handling to match the unified request contract.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart LR
    User[Unified train/play request] --> Dispatch[RL backend dispatch]
    Dispatch --> TorchEntry[TorchRL train or play entrypoint]
    TorchEntry --> Env[Isaac Lab vector environment]
    Env --> Wrapper[IsaacLabTorchRLWrapper]
    Wrapper --> TD[TensorDict observations and transitions]
    TD --> PPO[TorchRL actor, critic, GAE, PPO]
    PPO --> Checkpoint[Actor checkpoint]
    Checkpoint --> TorchEntry

Loading

_{Reviews (1) · Last reviewed commit: "Add TorchRL environment wrapper for isaa..."}

[greptile-apps]

Checkpoint resume is unsupported

The unified TrainingRequest now accepts backend="torchrl" and forwards its documented checkpoint field as --checkpoint, but this parser does not register that option. The unknown argument is therefore forwarded to Hydra, while train_ppo always creates a fresh actor, critic, optimizer, and collector. A TorchRL resume request cannot restore the requested checkpoint and may fail during configuration processing instead of resuming training. Please implement checkpoint restoration, including the intended iteration and optimizer state, or explicitly reject checkpoint-bearing TorchRL requests before Hydra processes them.

Knowledge Base Used: Reinforcement learning integrations

[greptile-apps]

TorchRL badge is missing

These generated rows advertise TorchRL support, and the browser exposes it in the selector and generates the correct command. However, the RL capability-symbol table has no torchrl entry, so Cartpole task cards omit the TorchRL capability badge. This is a non-blocking visual inconsistency that makes the new integration less discoverable; add TorchRL to capabilitySymbolSets.

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

[isaaclab-review-bot]

Isaac Lab Review Bot

The TorchRL wrapper, PPO runner, unified train/play integration, Cartpole configuration, terminal-observation conversion, documentation, and tests are coherently connected. One dependency-resolution issue remains: adding the optional TorchRL extra upgrades the workspace-wide TensorDict version for base installations.

• Design and architecture: TorchRL-specific spec conversion, auto-reset handling, and terminal-observation substitution remain isolated in the new backend, while the shared MARL adapter exposes concatenated final observations without mutating per-agent extras. However, the optional integration currently changes a shared base dependency through the lockfile, expanding its compatibility impact beyond TorchRL users.
• API: The additive public surfaces are consistent: the backend literal and dispatch registrations, lazy package exports, Cartpole agent entry points, optional extra, API documentation, and changelog fragments align. The remaining concern is dependency compatibility rather than the exposed TorchRL API.
• Implementation: The wrapper and training/playback paths align with Isaac Lab’s in-step reset semantics, including enabling and consuming terminal observations. Regression coverage exercises timeout handling, environment specs, MARL final-observation conversion, and unbatched action bounds. Before merge, constrain the TorchRL dependency graph to preserve the release branch’s TensorDict version or explicitly validate the existing base stack against TensorDict 0.14.2.

Minor fixes needed. Posted 1 actionable finding inline.

Automated review; human maintainers own approval decisions.

[isaaclab-review-bot]

🟡 Warning · Design Architecture — Optional extra upgrades shared tensordict version

Resolving torchrl>=0.13 pulls torchrl 0.14.0, and the lock moves tensordict from 0.13.0 to 0.14.2 for the entire workspace. tensordict is used on base install paths (the RSL-RL wrapper returns TensorDict observations and its tests import it unconditionally), so users who never select --extra torchrl get the upgrade on a release branch. Bound the extra (for example >=0.13,<0.14) or confirm the base stack against tensordict 0.14.