Isaac Sim 6 Troubleshooting — Large USD Scene Hangs

Purpose

Diagnose Isaac Sim startup hangs, stage-load stalls, MDL/shader compilation freezes, physics stepping blocks, Replicator/Hydra issues, Nucleus latency, and GPU OOM crashes.

Prerequisites

• Built Isaac Sim ($ISAAC_SIM_DIR or _build/linux-x86_64/release).
• NVIDIA GPU with a current driver (nvidia-smi).
• Shell env contract from isaac-sim-orchestrator: $ISAAC_SIM_DIR, $ISAAC_LAB_DIR, $WORKSPACE_DIR.

Limitations

• Symptom lists are heuristic; root causes can overlap across subsystems.
• Some fixes require local Kit builds or NVIDIA internal-only packages.

Troubleshooting

Diagnosis and resolution for Isaac Sim 6.0 (Kit 110) hangs, freezes, and perf degradation on large USD scenes (10K-150K+ prims, factory/warehouse twins).

Upstream debugging docs: docs/isaacsim/utilities/debugging/ (profiling, Python debugging tutorials). Pair this skill with profile-isaac-sim for Tracy-based diagnosis.

Quick Diagnosis Flowchart

Isaac Sim hangs/freezes
    |
    +-- During startup (before stage load)?
    |       -> Section 1: Startup Hangs
    |
    +-- During shutdown / SimulationApp.close()?
    |       -> Section 1a: Shutdown Hangs (Kit Framework Deadlock)
    |
    +-- During stage open / USD loading?
    |       -> Section 2: Stage Loading Hangs
    |
    +-- During shader/material compilation?
    |       -> Section 3: MDL/Shader Compilation
    |
    +-- During physics stepping / `SimulationManager` setup or timeline play?
    |       -> Section 4: Physics Hangs
    |
    +-- During Replicator / sensor capture?
    |       -> Section 5: Replicator Hangs
    |
    +-- During rendering (viewport frozen)?
    |       -> Section 6: Hydra/Rendering Hangs
    |
    +-- Nucleus/remote asset fetch?
    |       -> Section 7: Nucleus Network Hangs
    |
    +-- OOM crash or memory spike?
    |       -> Section 8: Memory Issues
    |
    +-- Slow but not frozen?
            -> Section 9: Performance Optimization

Section 1: Startup Hangs

Common Causes and Fixes

1. Extension conflict (circular import)

./isaac-sim.sh --/app/extensions/exclude='["problematic.extension"]'

2. Renderer initialization freeze

./isaac-sim.sh --vulkan         # Force Vulkan
./isaac-sim.sh --reset-user     # Reset user settings

3. Thread over-subscription on high-core-count CPUs

./isaac-sim.sh \
    --/plugins/carb.tasking.plugin/threadCount=16 \
    --/plugins/omni.tbb.globalcontrol/maxThreadCount=16

simulation_app = SimulationApp({"headless": False, "limit_cpu_threads": 16})

4. Nucleus login popup hang — Force-kill Isaac Sim and restart. Complete login before proceeding.

Section 1a: Shutdown Hangs (Kit Framework Deadlock)

Symptom

SimulationApp.close() never returns. Process hangs indefinitely (or until CI kills it at 1500s). Stack trace shows threads blocked in futex_wait inside carb.tasking thread pool teardown during shutdown_and_release_framework() or unload_all_plugins().

Root Cause

shutdown_and_release_framework() calls app->shutdown() then framework->unloadAllPlugins(). These are C extension calls that hold the GIL while internally deadlocking on native thread joins. A Python-side threading.Event.wait() watchdog cannot fire because it needs the GIL to check its timeout and call os.kill() — it starves.

Fix: SIGALRM Watchdog

SIGALRM is kernel-delivered regardless of GIL state. Set an alarm before the blocking C call; if it deadlocks, the alarm handler SIGKILLs the process unconditionally.

Pattern (generalized — apply to any potentially-deadlocking native teardown):

1. Save the previous SIGALRM handler.
2. Install a handler that calls os.kill(os.getpid(), signal.SIGKILL).
3. signal.alarm(timeout_seconds) — 120s for full shutdown, 30s for plugin unload.
4. Call the blocking native function.
5. In finally: cancel the alarm (signal.alarm(0)) and restore the previous handler.

When to Suspect This

• CI job hangs after all test assertions pass (shutdown phase).
• strace shows the main thread stuck in futex_wait inside Kit's plugin system.
• Python-based timeout mechanisms (threading, asyncio) silently fail to trigger.

Key Insight

Any time a Python watchdog must guard a C extension call that can hold the GIL indefinitely, threading.Event/threading.Timer will be starved. Use signal.alarm() (Linux) or os.alarm() as the only reliable GIL-independent timeout mechanism in CPython.

Reference

• File: source/extensions/isaacsim.simulation_app/isaacsim/simulation_app/simulation_app.py

Section 2: Stage Loading Hangs

Diagnosis

Check layer count:

for layer in stage.GetUsedLayers():
    print(f"  {layer.GetDisplayName()} ({layer.GetFileFormat().formatId})")

If layer count > 1,000, this is the primary bottleneck.

Common Causes

1. Excessive layer count (>1,000 layers)

Fix: Package assets into library layers (consolidate per-asset files into a small number of library .usd/.usdc layers; see usd-composition-architecture for the layered-asset pattern).

2. Missing/broken references — Cause USD to try every resolver including network fallbacks. Fix: Audit references:

from pxr import Sdf
for layer in stage.GetUsedLayers():
    for ref in layer.GetExternalReferences():
        if not os.path.exists(ref):
            print(f"MISSING: {ref}")

Section 3: Shader/MDL Compilation

• First launch always compiles MDL shaders — can take 5-15 minutes, normal
• Subsequent launches use shader cache
• Cache location: ~/.nvidia-omniverse/data/Kit/Isaac-Sim/shader_cache/
• Force rebuild: delete cache folder

Section 4: Physics Hangs

Physics setup / first-play hangs (SimulationManager.setup_simulation, timeline.play(), or the legacy world.reset() flow — see Renaming Extensions to migrate off omni.isaac.core.World): usually caused by:

• PhysicsScene not yet defined when physics starts.
• Too many contact pairs on the first step.
• GPU dynamics enabled with too many rigid bodies (>100K).

Fix:

# Ensure PhysicsScene exists before reset
if not stage.GetPrimAtPath("/World/PhysicsScene"):
    ps = UsdPhysics.Scene.Define(stage, "/World/PhysicsScene")
    
# Reduce initial contact storm
px_scene.CreateEnableStabilizationAttr().Set(True)

Section 5: Replicator Hangs

• rep.orchestrator.run() hangs: ensure the timeline is playing (omni.timeline.get_timeline_interface().play() or isaacsim.core.experimental.utils.app.play(commit=True)) before invoking.
• step_async() never completes: use await rep.orchestrator.step_async() correctly inside an async context.
• Frame capture hangs: call app_utils.update_app() (or a simulation_app.update()) once before capture so the renderer has a fresh frame.

Section 6: Rendering (Hydra) Hangs

• Viewport black and frozen: Check GPU memory (nvidia-smi)
• RTX renderer OOM: Reduce texture streaming budget:

  carb.settings.get_settings().set("/rtx/resourcemanager/textureMipCountBudget", 256)
  

Section 7: Nucleus Network Hangs

• Set timeout: carb.settings.set("/omni/client/timeout_seconds", 10.0)
• Use local assets when possible: Copy needed USDs to local disk
• Disable Nucleus: Launch with --/omni/client/enabled=false for local-only work

Section 8: Memory Issues

• Prims > 150K: Enable instancing (UsdGeom.PointInstancer)
• Textures: Use texture atlases, reduce resolution
• Monitor: watch -n1 'nvidia-smi --query-gpu=memory.used,memory.free --format=csv'

Section 9: Performance Optimization

• Layer count: Package into library layers (see Section 2)
• Mesh instancing: Use PointInstancer for repeated assets
• Payload loading: Mark large sub-scenes as unloaded payloads
• Physics: Reduce collision mesh complexity for non-critical objects