[4A/10] Share native resources through the clone lifecycle

docs/source/concepts/backend_architecture.rst

@@ -120,6 +120,34 @@ path without knowing which physics engine owns the state. See
 :doc:`/source/developer-tools/scene_data_providers` for the complete
 data-flow model.
 
+Sharing Newton resources
+~~~~~~~~~~~~~~~~~~~~~~~~
+
+Newton renderers and visualizers declare their native configuration with ``physics_cfg``:
+
+.. code-block:: python
+
+   physics_cfg = NewtonCfg(solver_cfg=MJWarpSolverCfg())
+   renderer_cfg = NewtonWarpRendererCfg(physics_cfg=physics_cfg)
+   visualizer_cfg = NewtonGLVisualizerCfg(physics_cfg=physics_cfg, max_visible_envs=4)
+
+Within one simulation and clone plan, equal concrete configuration types and values reuse one
+registered model, state, and control. Different values create independent native resources.
+Cloning populates builders; initialization finalizes each distinct resource once before consumers
+borrow its handles.
+
+Derived SDP/query bindings use ``invalidate_on=PhysicsEvent.STOP`` when registering resources.
+The registry releases those bindings once when physics stops; initialization binds the new native
+layout. Independent render models and existing viewer sessions are retained.
+
+When ``physics_cfg`` is omitted, configuration composition supplies the active Newton configuration,
+or Newton defaults under another physics backend. An explicit configuration always takes precedence.
+This resolution happens before consumer construction, not during rendering.
+
+Visible-world selection belongs to the visualizer: it keeps the complete shared model and filters
+Newton's rigid-shape render batches. It does not reduce physics work or guarantee partial updates
+for particle and debug visualization paths.
+
 Native engine access boundary
 -----------------------------
 

docs/source/developer-tools/scene_data_providers.rst

@@ -87,10 +87,16 @@ The transforms are returned as :class:`SceneDataFormat.Transform` (Warp ``transf
 so consumers that want this format get them zero-copy.
 
 Newton-native consumers (Newton visualizer, Rerun, Viser, Newton Warp renderer, OVRTX renderer)
-also need a Newton ``Model``/``State`` to render against. To provide that,
-:class:`~isaaclab_newton.physics.NewtonManager` builds a **shadow Newton model** from the USD
-stage on first access and updates its ``body_q`` from the PhysX backend each render frame.
-When the scene has PhysX or OVPhysX deformables, the shadow model also allocates
+also need a Newton ``Model``/``State`` to render against. They share a registry-owned
+``NewtonBackend``, populated by ``NewtonReplicateContext`` from the clone plan. This resource owns
+native handles and clone metadata, not a simulation context, physics policy, or consumer callbacks.
+The separately registered ``NewtonSceneQueries`` consumer shares SDP bindings, material writers,
+BVHs, and query graphs across cameras and viewers. Physics prepares kinematics before queries;
+the query consumer never calls the physics manager. Reading the model does not build another model
+or discover the completed stage. The existing transport copies
+PhysX transforms through :meth:`SceneDataProvider.get_transforms` into its retained ``body_q``
+buffer; OVRTX retains its renderer-side matrix conversion.
+When the scene has PhysX or OVPhysX deformables, the shared model also allocates
 ``particle_q`` render slots for soft/cloth meshes, syncs simulation nodal positions through
 :meth:`SceneDataProvider.get_points` with ``allow_passthrough=False`` into a separate
 sim-sized buffer, and remaps or copies those positions into the render-sized ``particle_q``
@@ -99,8 +105,9 @@ barycentric sim-to-visual remap so Newton Warp and OVRTX render the paired visua
 than tet simulation topology. The shadow deformable registry exposes render-slot offsets and
 ``particles_per_body`` counts for OVRTX point bindings.
 
-This is hidden behind :meth:`NewtonManager.get_model` / :meth:`NewtonManager.get_state`, so
-renderers don't need to know which physics backend is active.
+The existing :meth:`~isaaclab_newton.physics.NewtonManager.get_model` and
+:meth:`~isaaclab_newton.physics.NewtonManager.get_state` accessors resolve the same registered
+resource. The scene-data format and transport interfaces are unchanged.
 
 Newton backend
 --------------
@@ -112,8 +119,11 @@ model and state, and the provider exposes that state as :class:`SceneDataFormat.
 Data requirements
 ------------------
 
-Visualizers and renderers declare what they need from the scene data path. This is resolved at
-simulation-context construction time and is what triggers the shadow-model build for PhysX:
+Consumers are constructed from cfg before cloning and initialized afterward. Clone planning routes
+required Newton representations alongside asset ``cloning_contexts``. The simulation registry owns
+the shared Newton runtime, PhysX tensor view, and OVPhysX runtime/stage; these owners are colocated
+with their physics managers. Clone contexts build resources, while physics managers retain stepping
+and backend setup. Foreign Newton bindings are prepared at ``PHYSICS_READY`` before consumers initialize.
 
 .. list-table::
    :header-rows: 1

scripts/demos/newton_viewer_dominoes.py

@@ -25,7 +25,7 @@
 args_cli = parser.parse_args()
 
 import torch
-from isaaclab_newton.physics import NewtonCfg, NewtonManager, NewtonShapeCfg, XPBDSolverCfg
+from isaaclab_newton.physics import NewtonCfg, NewtonShapeCfg, XPBDSolverCfg
 
 from pxr import Gf, UsdGeom
 
@@ -147,9 +147,6 @@ def main() -> None:
         sim.set_camera_view(eye=(0.0, -18.0, 15.0), target=(0.0, 0.0, 0.0))
         _scene = InteractiveScene(DominoSceneCfg(num_envs=1, env_spacing=1.0))
         _apply_display_colors()
-        if NewtonManager._builder is None:
-            NewtonManager.instantiate_builder_from_stage()
-        NewtonManager._builder.rigid_gap = 0.001
         sim.reset()
         print(
             f"[INFO]: Setup complete with {len(LOGO_DOMINO_POSES)} green dominoes. "

source/isaaclab/changelog.d/shared-backend-lifecycle.major.rst

@@ -0,0 +1,28 @@
+Changed
+^^^^^^^
+
+* Clone planning automatically included the Newton representation required by selected renderers
+  and visualizers under other physics backends. Explicit asset physics routing was preserved.
+  ``replicate_physics=False`` skipped the active physics context while still dispatching other
+  declared contexts; custom contexts should now be omitted from asset ``cloning_contexts`` when
+  they should not run.
+* **Breaking:** Configured visualizers propagated construction and initialization failures, like
+  explicitly requested CLI visualizers, instead of logging and continuing. Remove unavailable
+  visualizers from the configuration. Call ``SimulationContext.clear_instance()`` before retrying
+  after a constructor failure. Teardown closed pending and initialized visualizers and
+  attached cleanup failures to the original exception when one was already being handled.
+* Registered renderer resources in their constructors before cloning and initialized consumers
+  afterward. Existing scene-data transport interfaces remained unchanged.
+* Added configuration-value identity to the backend registry. Pass ``cfg=`` to share equal concrete
+  configurations independently of constructor arguments; omitting it retained type-only identity.
+  Consumer physics dependencies were composed before construction and excluded from active-physics
+  launcher overrides. Hard resets rebuilt registered native resources and rebound existing consumers.
+  ``invalidate_on=PhysicsEvent.STOP`` scoped derived bindings to the current physics layout while
+  preserving independent native models. Explicit removal also released the registry's event subscription.
+
+Deprecated
+^^^^^^^^^^
+
+* Deprecated ``REQUIRES_STAGE_AND_MODEL`` and the informational ``requires_newton_model`` flag.
+  Consumers declared cfg-owned clone contexts directly instead of relying on visualizer type-name
+  inference. USD consumers continued to set ``requires_usd_stage`` before cloning.

source/isaaclab/isaaclab/app/sim_launcher.py

@@ -33,6 +33,7 @@
 from isaaclab.renderers.renderer_cfg import RendererCfg
 from isaaclab.sensors.camera.camera_cfg import CameraCfg
 from isaaclab.utils._device import set_cuda_device
+from isaaclab.visualizers.visualizer_cfg import VisualizerCfg
 
 logger = logging.getLogger(__name__)
 
@@ -246,10 +247,11 @@ def _refresh_physics_scan_flags(config_scan: Scan, concrete_physics_cfgs: list[P
 def scan(cfg, launcher_args: argparse.Namespace | dict | None = None) -> Scan:
     """Walk *cfg* once, collecting all launch signals and applying ``--physics``.
 
-    When the ``physics`` key is present in *launcher_args*, every physics config is
+    When the ``physics`` key is present in *launcher_args*, every active physics config is
     replaced by the requested backend (see :func:`make_physics_cfg`): nested configs
     in place, a root config via :attr:`Scan.effective_cfg` (it cannot be mutated in
-    place). Automatic PhysX configurations and RTX
+    place). Renderer and visualizer ``physics_cfg`` dependencies do not select active
+    physics and are preserved. Automatic PhysX configurations and RTX
     renderer placeholders (``renderer_type="auto_rtx"``) are also resolved
     at this stage using the full *launcher_args* context.
 
@@ -306,6 +308,8 @@ def visit(node, parent, key):
         except TypeError:
             return
         for name, child in children.items():
+            if name == "physics_cfg" and isinstance(node, (RendererCfg, VisualizerCfg)):
+                continue
             if child is None or isinstance(child, (int, float, str, bool)):
                 continue
             visit(child, node, name)

source/isaaclab/isaaclab/assets/asset_base_cfg.py

@@ -58,6 +58,8 @@ class InitialStateCfg:
     requests no explicit physics context.
     :class:`~isaaclab.cloner.UsdReplicateContext` is still added automatically when ``spawn``
     is set and Kit is available; listing it explicitly forces USD replication even without Kit.
+    A selected Newton renderer or visualizer also adds its rendering context under another physics
+    backend. This does not override explicit routing when Newton is the active physics backend.
     """
 
     prim_path: str = MISSING

source/isaaclab/isaaclab/cloner/clone_plan.py

@@ -221,17 +221,20 @@ def _context_rows(
     populated_rows: set[int],
     global_paths: tuple[str, ...] = (),
 ) -> dict[type[object], tuple[int, ...]]:
-    """Route plan rows to the clone contexts registered for this simulation."""
+    """Route environment cfgs to registered contexts; every cfg must have a row mapping."""
     sim = sim_utils.SimulationContext.instance()
     if sim is None:
         return {}
 
     physics_context = sim.physics_manager.clone_context_type
     if physics_context is not None and not isinstance(physics_context, type):
         raise TypeError("PhysicsManager.clone_context_type must be a context class.")
-    rows_by_context: dict[type[object], set[int]] = (
-        {} if physics_context is None or not global_paths else {physics_context: set()}
-    )
+    spawn_contexts = (UsdReplicateContext,) if has_kit() else ()
+    render_contexts = {type(context) for context in sim.render_context.clone_contexts} - {physics_context}
+    spawn_contexts += tuple(render_contexts)
+    rows_by_context: dict[type[object], set[int]] = {
+        context: set() for context in (physics_context, *render_contexts) if context is not None and global_paths
+    }
 
     for cfg in cfgs:
         rows = cfg_rows[id(cfg)]
@@ -241,8 +244,8 @@ def _context_rows(
             contexts = () if physics_context is None else (physics_context,)
         else:
             contexts = tuple(string_to_callable(value) if isinstance(value, str) else value for value in references)
-        if isinstance(fields.get("spawn"), sim_utils.SpawnerCfg) and has_kit():
-            contexts = tuple(dict.fromkeys((*contexts, UsdReplicateContext)))
+        if isinstance(fields.get("spawn"), sim_utils.SpawnerCfg):
+            contexts += spawn_contexts
         for context_type in contexts:
             if not isinstance(context_type, type):
                 raise TypeError(f"{type(cfg).__name__}.cloning_contexts must contain only context classes.")
@@ -253,7 +256,7 @@ def _context_rows(
     return {
         context_type: tuple(sorted(rows & populated_rows))
         for context_type, rows in rows_by_context.items()
-        if rows & populated_rows or context_type is physics_context and bool(global_paths)
+        if rows & populated_rows or global_paths and context_type in (physics_context, *render_contexts)
     }
 
 
@@ -297,10 +300,13 @@ def make_clone_plan(
 
     cfgs = tuple(cfgs)
     global_paths = _minimal_roots(global_paths)
+    sim = sim_utils.SimulationContext.instance()
 
     # 1) Build per-group records: (cfg, spawn_cfg, destination_template, num_variants).
     groups: list[tuple[Any, Any, str, int]] = []
     for cfg in cfgs:
+        if sim is not None and "renderer_cfg" in vars(cfg):
+            sim.render_context.get_renderer(cfg.renderer_cfg)
         matched = match(cfg.prim_path, env_template)
         count = num_spawn_variants(cfg.spawn)
         if count <= 0:
@@ -460,6 +466,8 @@ def clone_plan_from_env_0(
         if spawn is not None and num_spawn_variants(spawn) != 1:
             raise ValueError("clone_plan_from_env_0 requires single-variant spawners.")
         records.append((cfg, prim_path, matched, spawn))
+        if "renderer_cfg" in vars(cfg):
+            sim.render_context.get_renderer(cfg.renderer_cfg)
 
     env_cfgs = tuple(cfg for cfg, _, matched, _ in records if matched is not None)
     global_paths = _minimal_roots(prim_path for _, prim_path, matched, _ in records if matched is None)

source/isaaclab/isaaclab/cloner/cloner_cfg.py

@@ -77,8 +77,8 @@ class CloneCfg:
     replicate_physics: bool = True
     """Whether physics replication clones each environment. Default is True.
 
-    If False, cloning is USD-only: the physics engine parses the per-env USD prims directly
-    instead of replicating env_0's parsed structure. Applied by :func:`~isaaclab.cloner.replicate`.
+    If False, the active physics clone context is skipped; USD and other declared scene
+    representations are still built. Applied by :func:`~isaaclab.cloner.replicate`.
     """
 
 

source/isaaclab/isaaclab/cloner/replicate_session.py

@@ -17,7 +17,6 @@
 from .clone_plan import make_clone_plan
 from .cloner_cfg import DEFAULT_ENV_TEMPLATE
 from .cloner_strategies import sequential
-from .usd import UsdReplicateContext
 
 if TYPE_CHECKING:
     from .clone_plan import ClonePlan
@@ -26,29 +25,31 @@
 def replicate(plan: ClonePlan, *, replicate_physics: bool = True) -> None:
     """Dispatch the active fully routed clone plan.
 
-    Planning derives routing from the input cfgs; dispatch does not rediscover or reshape that mapping.
+    Planning derives asset and renderer routing from the input cfgs before dispatch.
     Every context is owned by the active :class:`~isaaclab.sim.SimulationContext` and receives
     only ``plan``.
 
     Args:
         plan: Replication layout to dispatch.
-        replicate_physics: Whether physics replication clones each environment. If False,
-            cloning is USD-only; an asset whose contexts are all physics-based is not cloned.
+        replicate_physics: Whether the active physics context clones each environment.
+            If False, other declared contexts still build the required scene representations.
     """
     sim = SimulationContext.instance()
     if sim is None:
         raise RuntimeError("Clone-plan replication requires an active SimulationContext.")
     if sim.get_clone_plan() is not plan:
         raise ValueError("replicate() requires the active SimulationContext's ClonePlan.")
-    context_types = tuple(
-        context_type for context_type in plan.context_rows if replicate_physics or context_type is UsdReplicateContext
-    )
-    missing = [context_type for context_type in context_types if context_type not in sim._backend_registry]
+    missing = [context_type for context_type in plan.context_rows if context_type not in sim._backend_registry]
     if missing:
         names = ", ".join(f"{context_type.__module__}.{context_type.__qualname__}" for context_type in missing)
         raise RuntimeError(f"Clone contexts must be registered before plan dispatch: {names}.")
 
-    contexts = [sim._backend_registry[context_type] for context_type in context_types]
+    contexts = [
+        context
+        for context_type in plan.context_rows
+        for _, context, _ in sim._backend_registry[context_type]
+        if replicate_physics or context is not sim.physics_manager._clone_context
+    ]
     for context in sorted(contexts, key=lambda item: item.replicate_priority):
         context.replicate(plan)
 

source/isaaclab/isaaclab/physics/physics_manager.py

@@ -87,6 +87,7 @@ class PhysicsManager(ABC):
     _callback_id: ClassVar[int] = 0
     views: ClassVar[dict[tuple[type, str], Any]] = {}
     clone_context_type: ClassVar[type[object] | None] = None
+    _clone_context: ClassVar[object | None] = None
 
     supports_anim_recording: ClassVar[bool] = False
     """Whether this backend can service ``--anim_recording_enabled`` (OVD Recorder).

source/isaaclab/isaaclab/physics/physics_manager_cfg.py

@@ -7,7 +7,7 @@
 
 from __future__ import annotations
 
-from dataclasses import MISSING
+from dataclasses import MISSING, fields
 from typing import TYPE_CHECKING, Any
 
 from isaaclab.utils import configclass
@@ -62,6 +62,14 @@ class PhysxAutoCfg(PhysicsCfg):
     """Concrete OvPhysX configuration, or ``None`` when OvPhysX is unsupported."""
 
 
+def _compose_physics_cfg(cfg: object, physics_cfg: PhysicsCfg | None) -> None:
+    """Resolve a consumer's omitted native configuration before constructing the consumer."""
+    for cfg_field in fields(cfg):
+        if "physics_cfg_type" in cfg_field.metadata and getattr(cfg, cfg_field.name) is None:
+            cfg_type = cfg_field.metadata["physics_cfg_type"]
+            setattr(cfg, cfg_field.name, physics_cfg if isinstance(physics_cfg, cfg_type) else cfg_type())
+
+
 def _resolve_physx_auto_cfg(physics_cfg: PhysicsCfg, use_isaac_sim: bool) -> PhysicsCfg:
     """Resolve a :class:`PhysxAutoCfg` to a concrete backend."""
     if not isinstance(physics_cfg, PhysxAutoCfg):