[4A/10] Share native resources through the clone lifecycle

[ooctipus]

Description

PR 4A of the lifecycle/SDP refactor, following #7608. This narrows the existing PR to native resource ownership and construction/cloning/initialization order. The atomic SDP transport cutover is deferred to 4B.

• The simulation registry keys native resources by backend type and the full configuration's concrete type and values. Equal Newton configurations share one model/state/control; explicitly different configurations create independent resources.
• Newton camera/viewer configs declare physics_cfg. Composition fills omitted values from the active Newton config, or Newton defaults under foreign physics; explicit values win. A viewer's visible-world subset uses the same full native model.
• NewtonReplicateContext owns the cfg-specific builder during cloning. Initialization constructs a complete NewtonBackend from cfg, builder, device, and geometry. The backend stays in newton_manager.py, owns native handles/geometry only, and retains neither SimulationContext nor a builder.
• Shared NewtonSceneQueries binds realized SDP layouts at construction. Registry-owned STOP invalidation discards derived bindings once; consumers reacquire them after physics rebuilds while independent render models remain alive. Viewer initialization and hard reset use one model-binding path without recreating the viewer session.
• Renderers and visualizers construct through cfg.class_type(cfg) before cloning. Planning includes their declared clone contexts alongside asset cloning_contexts; dispatch does not infer routing from the finished stage.
• PhysX and OVPhysX native handles are registry-owned in their existing manager modules. PhysX creates one native tensor view instead of two. Pending and initialized visualizers have deterministic ownership and cleanup.

Deliberately unchanged in 4A: SDP format/transport APIs, foreign-state copying, renderer-side matrix conversions, existing guarded FK, and Fabric transport. 4B will replace these atomically with publication, cached conversion/passthrough, and SDP-owned Fabric. Mutable-geometry discovery and remaining visualizer planning are not claimed complete here.

No dependency or lockfile changes. Against the develop merge base f75767005f: 87 files, +3,298 / −3,319, net −21 lines. Production Python: −570; tests: +435; documentation/examples/release notes: +114. The latest cfg-identity follow-up is −3 production lines relative to 2934579d8; removing its duplicated orchestration cut 85 production lines from the initial implementation.

Behavior changes

• replicate_physics=False skips the active physics clone context, while other declared representations still build. Omit a custom context from asset cloning_contexts when it should not run.
• Breaking: cfg-declared visualizer construction/initialization failures propagate instead of silently continuing. Remove unavailable visualizers from cfg. Constructor errors propagate directly without automatic rollback; callers recovering in-process must call SimulationContext.clear_instance() before retrying. Explicit teardown closes partial consumers and preserves the original failure.

Validation

Latest cfg-identity and orchestration follow-up (553575bfa):

• 432 focused CPU tests passed, 3 skipped, with 32 GPU cases deselected. The shared camera/raycaster graph test then passed both cases on GPU 0. Twelve additional marker backend-selection/viewer tests passed on CPU.
• Live GPU-0 checks covered shared Newton physics/camera/viewer state, visible worlds [1, 3] within one four-world model, viewer-preserving hard reset, and explicitly different Newton configurations producing exactly two native resources.
• OVPhysX hard reset preserved the existing Newton render model and finalized it only once. Reordered source-layout tests verified fresh mappings, retained native buffers, and no leaked event subscriptions across repeated resets.
• Repository formatting/changelog checks and the warning-as-error documentation build passed.
• These are correctness/lifecycle checks, not a new matched performance comparison. The timing tables below retain their measured commit labels, including the unresolved runtime regression signal.

Earlier committed validation (before this follow-up):

• Resource/query boundary follow-up: 208 Newton physics/VBD tests and 18 live raycaster/camera tests passed on GPU 0; 182 consumer/material/visualizer/OVRTX tests passed with 3 skips on CPU, plus the shared-state/particle-only regression tests. The architecture gate failed before and passed after.

• Covered first initialization before solver binding, hard reset, in-place dirty state, native state swaps, retained foreign buffers/mappings, particle-only models with no rigid-body buffer, one material writer for shared consumers, and task teardown.

• Native SDP transform publication is now side-effect-free. Camera and physics-sensor callers prepare kinematics before executing queries, including same-step rerenders. Existing compatibility state accessors retain their guarded FK behavior.

• Live 4,096-environment Kuka Camera and two-camera Cartpole under Isaac PhysX and OVPhysX completed reset/stepping/teardown with finite, nonconstant RGB. Two camera groups shared one native resource and query runtime.

• Formatting/changelog checks and the warning-as-error documentation build passed again.

• Constructor-handler removal: all 52 simulation-context tests and 44 visualizer orchestration tests passed. The new propagation/explicit-cleanup regression failed before the edit and passed afterward; formatting/changelog checks passed again.

• 261 core clone/SDP/render/visualizer tests passed; 2 skipped.

• GPU 0: 207 Newton manager/VBD, 55 Kit/PhysX lifecycle, and 6 existing Fabric tests passed.

• 72 contrib coupling tests and 141 OVPhysX/OVRTX contract tests passed; 8 CUDA-dependent OV cases skipped in the CPU invocation.

• Additional Newton clone, material, and foreign-state binding tests passed, including identity/reordered geometry layouts and rebinding after native views change.

• GPU 0 Cartpole Camera: OVPhysX→Newton, Isaac PhysX→Newton, and Newton→Isaac RTX completed reset and stepping with finite, nonconstant RGB in all 32 environments. Isaac PhysX→Newton soft Franka completed reset plus 5 warm-up and 5 measured steps at 4 environments.

• Repository formatting/changelog checks and warning-as-error documentation build passed.

Matched warm startup and runtime

Isaac-Lift-KukaAllegro-Camera, 4,096 environments, Newton MJWarp physics, default 64×64 RGB, seed 0, no interactive visualizer. GPU 0: RTX 5090. 4A dbeccd280f versus its exact develop base f75767005f.

Renderer	Warm startup 4A / develop	Runtime ms/step 4A / develop	Environment FPS 4A / develop
Newton renderer	28.63 / 29.27 s	53.47 / 53.09	76,603 / 77,157
OVRTX	46.28 / 46.38 s	89.43 / 89.20	45,801 / 45,919

One fresh-process measured sample per branch and renderer, after a separate cache warm-up run. Startup measures script entry through the first completed step, including imports, cfg, application launch, construction, and reset; it excludes the outer uv/shell launcher. Runtime discards the first 50 steps and measures 200 full environment steps, synchronizing GPU work around the block. FPS means environment transitions/s, including action sampling, rendering, observations, and automatic resets, not display FPS.

Resolved cfgs and dependencies matched exactly; both measured checkouts were clean. Every sample verified finite, nonconstant RGB in all 4,096 environments. One-second monitoring found no competing GPU-0 compute job, apart from the unchanged NX desktop process. GPU 1 was used separately on the shared host. No profiler or rendering-quality reduction was used.

Interpretation: approximately performance-neutral in these samples. Throughput was 0.7% lower for Newton and 0.3% lower for OVRTX; this single-pair check does not establish a meaningful regression or speedup. These are 4A results, not the full future refactor. Cold startup was not measured.

Raw measurements, resolved cfgs, invocations, source-path checks, and GPU monitoring are retained locally under /tmp/pr4a-benchmark-20260920/; the local driver is /tmp/pr4a-bench.py. No benchmark tooling is added to this PR.

Resource/query follow-up timing

Final reserved-GPU-0 reverse-order comparison: committed follow-up 2934579d8 first, then parent d1006f829. Both tracked worktrees were clean. Same local measurement script, 4,096 Kuka Camera environments, Newton physics/rendering, 50 discarded steps and 200 measured steps:

Metric	Parent d1006f829	Follow-up 2934579d8	Change
Warm startup	28.40 s	28.29 s	−0.4%
Runtime ms/step	55.61	57.86	+4.1%
Environment FPS	73,654	70,786	−3.9%

Resolved cfg files were byte-identical; dependency versions and environment settings matched. Recorded module paths verified the intended checkout. One-second monitoring detected no overlapping GPU-0 compute job in either run. All 4,096 cameras produced finite, nonconstant 64×64 RGB; both processes completed teardown successfully.

Startup remained approximately neutral, but the runtime slowdown repeated: the earlier precommit pair showed +3.8% step time, and this clean committed reverse-order pair showed +4.1%. Treat this as an unresolved regression signal requiring investigation, not verified runtime neutrality. Both runs emitted roughly 820,000 lines of MJWarp line-search-limit warnings, adding substantial logging/I/O noise; shared CPU/I/O contention was not controlled. These limitations do not dismiss the repeated slowdown. No solver settings, rendering quality, or production code were changed for this check.

Startup excludes outer uv/CLI bootstrap; runtime includes action sampling, full environment stepping/rendering/observations, and resets. These are warm measurements, not cold startup. The older develop comparison above is explicitly for commit dbeccd280f, not this follow-up.

The earlier two-camera Isaac PhysX→Newton diagnostic sample was 4.10 ms/step before versus 3.88 ms after (32 environments; 20 discarded/100 measured steps). It verifies this path remains operational and showed no slowdown in that sample, not a statistically established speedup.

Local artifacts: /tmp/pr4a-query-kuka-matched-{before_reserved,after_reserved}/ and /tmp/pr4a-query-reserved.log. Earlier overlap-contaminated confirmation attempts were discarded. No benchmark scripts are committed.

Remaining CI

Full CI still needs to run on this narrowed diff. Earlier Kit visualizer golden-image mismatches are not claimed resolved by these local checks; no goldens or thresholds were relaxed.

Type of change

• Refactor / bug fix
• Breaking visualizer failure-policy change
• Documentation update

Checklist

• Read the contribution guidelines
• Ran repository formatting and changelog checks
• Updated documentation and package changelog fragments
• Added or updated focused regression tests
• Confirmed no new warnings across the complete backend matrix
• Full CI passed

[ooctipus]

run-ci

[ooctipus]

Added matched startup and steady-state FPS benchmarks to the description and marked ready for review. Kuka Camera, 4,096 envs, GPU 0; two samples per side, first 50 steps excluded and next 200 timed. Newton renderer: PR/develop 28.30/29.97 s and 78,624/76,593 env FPS. OVRTX: 45.35/45.87 s and 45,528/46,402 env FPS (1.9% lower FPS in this small sample, explicitly reported). The Isaac PhysX→Isaac RTX smoke also passed on retry. Visualizer CI remains unresolved: one stale factory-identity assertion and three tiled-image mismatches, detailed in the description. No production changes, golden replacements, or threshold relaxations were made for this benchmark pass.

[greptile-apps]

Confidence Score: 4/5

The PR appears safe to merge from a correctness perspective, with a non-blocking multi-camera performance regression worth addressing.

Findings

1.  Repeated Camera Scene Sync ▶

Summary

This PR moves rigid-transform publication and conversion into the Scene Data Provider, shares clone-built Newton resources across physics and rendering consumers, and updates Isaac RTX, OVRTX, Newton renderers, and visualizers to consume the new lifecycle.

• Adds dirty-generation transform caching, format conversion, remapping, Fabric output, and authored-scale preservation.
• Moves Newton model/state/control ownership into the simulation backend registry and builds foreign-physics visualization models from clone plans.
• Updates backend state writers, renderer initialization, visualizer construction, documentation, changelogs, and focused tests.
• The main remaining concern is repeated full renderer synchronization for every camera in a single physics step.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart LR
    P[Physics backend] -->|native transforms + dirty latch| SDP[Scene Data Provider]
    SDP -->|native or cached conversion| N[Newton resource]
    SDP -->|TransposedMatrix44d| O[OVRTX]
    SDP -->|FabricMatrix44| F[Isaac RTX / Kit]
    CP[Clone plan] --> N
    N --> NW[Newton Warp renderer]
    N --> V[Newton / Rerun / Viser]
    F --> C[Camera and viewport output]
    O --> C
    NW --> C

Loading

_{Reviews (1) · Last reviewed commit: "Route renderer rigid transforms through ..."}

[greptile-apps]

Repeated Camera Scene Sync

Removing the per-physics-step guard makes every camera fetch synchronize the full set of registered renderers again. Eager scene updates can also perform this synchronization before the camera fetches. SDP deduplicates transform conversions, but it does not prevent renderer geometry work—such as OVRTX deformable copies—from repeating for every camera in the same physics step. This is a non-blocking performance concern for multi-camera scenes; preserve same-step dirty-publication handling without rerunning every renderer update per camera.

Knowledge Base Used:

• Simulation, rendering, and sensors
• Rendering and visualization extensions

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

[isaaclab-review-bot]

Isaac Lab Review Bot

The SDP publication cutover and clone-context ownership model are coherent, but three actionable issues remain: removed public NewtonManager methods lack deprecation or migration support, OVPhysX now passes a shape-incompatible alias to native binding reads, and PhysX dirty-state invalidation does not survive CUDA graph replay.

• Design and architecture: Building shared Newton render resources from the clone plan and owning them in NewtonReplicateContext removes the lazy stage-walking shadow model cleanly. However, the publication architecture relies on host-side dirty latches, which do not capture replayed device-side writes without an explicit graph-aware invalidation mechanism.
• API: The new SceneDataPublication and request_transforms contract is documented and changelogged. The deletion of the public NewtonManager.update_visualization_state and sync_transforms_to_usd classmethods nevertheless violates the repository requirement for prior deprecation and migration guidance; compatibility shims or an explicit supported migration are needed.
• Implementation: OVPhysX previously required binding-shaped float32 destinations, but now supplies a one-dimensional transformf alias to read_into; the alias should retain the binding's (N, 7) float32 shape while sharing storage. PhysX mask writers also set the dirty latch only in Python during graph capture, so later graph replays can leave SDP conversions and pre-render kinematics stale; this needs graph-replay-aware invalidation analogous to the Newton path.

Minor fixes needed. Posted 3 actionable findings inline.

Automated review; human maintainers own approval decisions.

[isaaclab-review-bot]

🟡 Warning · Api — Removed public NewtonManager methods undocumented

NewtonManager.update_visualization_state and sync_transforms_to_usd are public classmethods on an exported class and are deleted here, yet this fragment states physics accessors remained supported and gives no migration path. Repository rules require a deprecation and migration guidance for removed public APIs. Either keep deprecated shims delegating to NewtonReplicateContext.update_transforms / UsdReplicateContext._update_fabric, or document the removal and replacement here.

[isaaclab-review-bot]

🟡 Warning · Implementation — Binding read alias changes shape and dtype

The deleted docstring stated the staging/merged split existed because the wheel writes into dst only when dst.shape == binding.shape, so a slice of the merged buffer cannot be read into directly. This alias is (row_count,) transformf while the binding is (N, 7) float32, and read_into is the same call the old code used with the matching staging buffer. Alias each binding as (row_count, 7) float32 over the same storage.

[isaaclab-review-bot]

🟡 Warning · Implementation — Host dirty latch misses graph-replayed writes

Transforms were previously re-gathered on every read; they are now gated on a Python boolean set by the asset writers. The mask writers are documented as the graphed-pipeline path, so their assignment runs at capture time only, and a replayed pose or joint write followed by a render leaves both the cached transforms and pre_render's kinematics refresh skipped. Newton handles this with _transforms_may_change_on_graph_replay; OVPhysX shares the same gap.

[ooctipus]

run-ci

[ooctipus]

run-ci

[ooctipus]

run-ci

[ooctipus]

run-ci

[ooctipus]

run-ci

[ooctipus]

run-ci

[ooctipus]

run-ci

[ooctipus]

run-ci