Isaac Sim Robot Navigation — Runtime

Purpose

Drive mobile robots at runtime with RL policies, trajectory followers, and physics/baked/per-frame stage strategies while avoiding common Kit 110 pitfalls.

Prerequisites

• Built Isaac Sim ($ISAAC_SIM_DIR or _build/linux-x86_64/release).
• NVIDIA GPU with a current driver (nvidia-smi).
• Shell env contract from isaac-sim-orchestrator: $ISAAC_SIM_DIR, $ISAAC_LAB_DIR, $WORKSPACE_DIR.

Limitations

• Targets Isaac Sim 6 / Kit 110 unless a section states otherwise.
• Does not replace official NVIDIA documentation for unsupported edge cases.

Troubleshooting

Runtime navigation specialization. Driving a robot through a USD scene live, in your own Isaac Sim script.

Read These Skills First

• navigation-primitives — occupancy maps, A* planning, differential/holonomic kinematics, robot footprints (Spot Z=0.69, etc.), look-at chase camera math, common gotchas. This skill assumes you know that substrate.
• isaac-sim-rendering — RT2, persistent session, ACES, frame validation
• isaac-sim-troubleshooting — Kit 110 hang/freeze reference

When To Use This Skill (vs siblings)

Runtime APIs

For shared primitives (omap, A*, kinematics, look-at), see navigation-primitives.

Physics Simulation on Kit 110 (CRITICAL)

Timeline MUST be playing for Physics simulation

import isaacsim.core.experimental.utils.app as app_utils
app_utils.play(commit=True)  # WITHOUT THIS, PHYSICS NEVER STEPS

Strip non-robot physics from heavy stages

On 33K+ prim stages with multiple physic bodies, simulation hangs. Strip ALL physics from non-robot prims:

from pxr import Usd, UsdPhysics

for p in Usd.PrimRange(stage.GetPseudoRoot()):
    if str(p.GetPath()).startswith("/World/Robot"):
        continue
    if p.HasAPI(UsdPhysics.RigidBodyAPI):
        p.RemoveAPI(UsdPhysics.RigidBodyAPI)
    if p.HasAPI(UsdPhysics.CollisionAPI):
        p.RemoveAPI(UsdPhysics.CollisionAPI)

Kit 110 Exec Model (CRITICAL — 2026-03-15)

• Synchronous code executes immediately in --exec scripts. No async needed for scene setup.
• asyncio.ensure_future() needs explicit event loop subscription:

  omni.kit.app.get_app().get_update_event_stream().create_subscription_to_pop(lambda e: None)
  asyncio.ensure_future(main())
  

• capture_viewport_to_file() returns MultiAOVFileCapture — use .wait_for_result(completion_frames=8), NOT .wait().
• Pattern: sync scene build → async render scheduling → event loop keeps it alive
• Scene load + 100 asset placement + 3-camera render in ~40s total

Camera Chase Cam

The look-at math and standard camera offsets (chase/overhead/POV) live in navigation-primitives. Runtime considerations on top of that:

• Camera collision avoidance (TODO): chase camera clips into rack geometry when robot enters narrow aisles. Need raycast from chase eye → robot; if occluded, raise camera or shift laterally. Not yet implemented.

Failure Modes & Recovery

FAILURE: VkResult: ERROR_OUT_OF_DEVICE_MEMORY during navigation render

• SYMPTOMS: Vulkan crash partway through capture loop on heavy stage
• CAUSE: Approach B (baked timeSamples) + RT2 + 50K+ prims
• FIX: Switch to approach C (per-frame transform)

FAILURE: Robot floats above ground or falls through floor

• CAUSE: Wrong Z offset
• FIX: See navigation-primitives Robot Footprint table

FAILURE: Physics never steps after kill+relaunch

• CAUSE: Kill+relaunch bug, dirty GPU state
• FIX: Wait 30+ seconds, verify zero Kit processes, or reboot

FAILURE: A* path looks fine on omap but robot clips a rack at render

• CAUSE: Skipped step 5/6 of the validation pipeline (see navigation-primitives)
• FIX: Always validate every smoothed point is in navigable space

Integration Points

• RECEIVES from: navigation-primitives — omap, A*, kinematics, robot specs, camera math
• RECEIVES from: occupancy-map — map.yaml or runtime grid
• RECEIVES from: urdf-mjcf-to-usd-conversion — robot USD
• PRODUCES for: isaac-sim-rendering — animated scene ready for RT2 capture and frame sequences of nav runs