Isaac Sim Orchestrator

Purpose

Turn an established goal or contract into a validated library, Kit integration, or runnable simulation by mapping capabilities and executing specialist skills in order. A request such as "the forklift demo is scoped and approved; build it and give me the video" enters the four phases below directly. Use isaac-sim-workflow first only when the deliverable or acceptance criteria still need scoping.

Specialist ownership

A focused obstacle-aware arm demo belongs to motion-generation, including pick-and-place around a pole with physics and video. If this orchestrator is loaded first, hand off to motion-generation before implementation and keep it as the primary skill. Orchestration may support composition and delivery.

When a native skill invocation tool is available, invoke the specialist with that tool; shell reads of its instructions or scripts do not replace invocation. When the runtime only supports file-based skill loading, use that mechanism and state the trace limitation accurately.

Prerequisites

• For Kit integration: built Isaac Sim, a supported NVIDIA GPU, and the shell environment below.
• For a standalone library foundation: a configured source/libraries build or installed wheel/SDK; Kit and a GPU are not universal requirements.

Limitations

• Kit-specific guidance targets Isaac Sim 6 / Kit 110 unless a section states otherwise.
• Does not replace official NVIDIA documentation for unsupported edge cases.

Troubleshooting

Environment contract

Kit-routed skills use these variables; standalone library work uses its configured build/install environment instead:

Run nvidia-smi before GPU-backed simulation or rendering to size num_envs and pick RT2 vs PathTracing. Do not require it for a library-only CPU workflow or hardcode a GPU class.

Execution mode

Choose the owning surface before running code:

Extension-to-library rewrite sequence

When an established contract moves reusable behavior out of an extension:

1. Inventory and classify the old API, implementation, adapters, packaging, dependencies, tests, imports, and version surfaces.
2. Keep one implementation in source/libraries; preserve supported imports and prove its public facade without Kit.
3. Stage the built wheel and SDK in a thin Kit extension that retains only Kit lifecycle or translation behavior.
4. Validate library and Kit routes independently with isaac-sim-validator, including their separate version decisions.

Launch a server-enabled sim using the launch procedure in isaac-sim-remote; do not duplicate its command here. Before Kit controller iteration, require a successful health response and then send scene setup, tuning, and validation probes through the client. Retain their responses as execution evidence. If the server cannot be made available, report that workflow validation is blocked; do not silently switch to repeated SimulationApp launches or claim workflow completion from a successful video. For graceful shutdown after live iteration, invoke the remote skill's shutdown helper through its localhost client.

Task decomposition

For any request, run all four phases. The specific steps inside each phase depend on the goal; identify capabilities first, then verify each in isolation before combining.

Phase 1 — Verify foundations

1a. Feature/skill mapping (before any code):

• If a demo still lacks a deliverable or acceptance criteria, load isaac-sim-workflow to define them, then return to these phases. If the request already fixes the subject, behavior, constraints, and output (for example, a collision-free aisle drive delivered as a 20-second video), use that contract directly; do not repeat the scoping pass.
• Cross-check the request against documented Isaac Sim features and APIs.
• For each capability, look up an existing skill:
  • Skill exists -> load it, follow its procedure.
  • Skill missing -> build one inline. Mark its frontmatter status: draft, flag it HIGH PRIORITY in skill-distillation, tell the user upfront, and shorten iteration cycles (share intermediate results, ask targeted questions early).
• Write the feature -> skill mapping into the task WORKLOG.md before 1b.

1b. Foundation verification (capability by capability):

• List every capability the task needs (assets, physics, robot control, sensors, rendering, ...).
• Verify each in isolation: does it load, does it behave correctly on its own.
• Do not move on until each foundation passes.

Phase 2 — Incremental integration

Combine verified foundations one at a time. Re-run stability/correctness checks after each addition. Every failure has exactly one new variable.

Phase 3 — Polish & deliver

• Validate output visually or programmatically. Task success, not just script completion.
• Add output-specific requirements (writers, annotations, video capture, DR).
• Package and hand off with a short summary.

Phase 4 — Distill (mandatory)

• List iterations, failures, workarounds.
• Record user corrections.
• Classify each lesson: new skill, skill update, procedure fix, or MEMORY.md fact.
• Follow skill-distillation to decide whether a reusable lesson warrants a skill change. Apply scoped edits when the user has authorized them; otherwise record or propose the patch before writing. If there is no reusable lesson, record that conclusion.

See skill-distillation for the full procedure. Phase 4 is not optional, even when no file change is warranted.

Sub-agent rules

• Each phase can run as a sub-agent with its own WORKLOG.md.
• Sub-agents commit at logical checkpoints, never half-done.
• Large script generation: write incrementally to files. Do not try to produce 200+ lines in one turn.
• If a sub-agent times out, WORKLOG.md survives for the next pickup.

Routed skills

Multi-robot fleet reference

Sample robots

Scene setup

• sim_warehouse_v4.usda pattern: shell + lights + racks + PhysicsScene + ground collision.
• Shell (sm_warehouse_mega.usd) is in cm; robots and equipment in meters.
• Strip physics from environment assets offline. Runtime stripping core-dumps on large stages.

PhysicsScene

from pxr import UsdPhysics, PhysxSchema

physics_scene = UsdPhysics.Scene.Define(stage, "/World/PhysicsScene")
physics_scene.CreateGravityDirectionAttr().Set((0, 0, -1))
physics_scene.CreateGravityMagnitudeAttr().Set(9.81)

physx_scene = PhysxSchema.PhysxSceneAPI.Apply(physics_scene.GetPrim())
physx_scene.CreateEnableCCDAttr().Set(True)
physx_scene.CreateEnableStabilizationAttr().Set(True)
physx_scene.CreateSolverTypeAttr().Set("TGS")
physx_scene.CreateTimeStepsPerSecondAttr().Set(60)
physx_scene.CreateGpuMaxNumPartitionsAttr().Set(8)   # 10+ robots

Grid placement

import math

def place_robots_grid(stage, robot_usd_path, prefix, count, spacing=3.0, start_z=0.0):
    cols = math.ceil(math.sqrt(count))
    robots = []
    for i in range(count):
        row, col = divmod(i, cols)
        x, y = col * spacing, row * spacing
        prim_path = f"/World/Robots/{prefix}_{i}"
        ref = stage.OverridePrim(prim_path)
        ref.GetReferences().AddReference(robot_usd_path)
        from pxr import UsdGeom, Gf
        xform = UsdGeom.Xformable(ref)
        xform.ClearXformOpOrder()
        xform.AddTranslateOp().Set(Gf.Vec3d(x, y, start_z))
        robots.append(prim_path)
    return robots

Separation

• Mobile robots: minimum 2 m between centers.
• Articulated arms: 1.5x reach radius minimum.
• Aerial: stagger altitudes by >= 2 m.

Collision groups

from pxr import PhysxSchema

def create_collision_group(stage, group_path, robot_paths):
    group = PhysxSchema.PhysxCollisionAPI.Apply(stage.DefinePrim(group_path))
    for path in robot_paths:
        prim = stage.GetPrimAtPath(path)
        collision_api = PhysxSchema.PhysxCollisionAPI.Apply(prim)
        collision_api.GetCollisionGroupsRel().AddTarget(group_path)

Scaling limits (by VRAM)

Limits scale approximately linearly with available VRAM. Beyond these thresholds risks CUDA OOM.

Optimization:

• make_instanceable: true in URDF config.yaml (shared mesh data).
• LOD switching for distant robots.
• Disable physics on robots outside the active zone.

Navigation

Differential drive kinematics:

vL = (vx - omega * tw/2) / wheel_r
vR = (vx + omega * tw/2) / wheel_r

PD steering defaults: KP=2.5, KD=1.2, MAX_W=1.5, waypoint tolerance 4.0 m. Out-of-bounds: |Z| > 50 or |X|/|Y| > 500 -> mark dead.

Camera

Chase camera: 12 m behind, min height 2.5 m, clamped inside warehouse bounds, smooth interpolation alpha = min(1.0, DT*2.0). Dynamically raise camera to avoid rack intrusion.

View modes (cycle every 4 s): chase, overhead (z=50 m), aisle (eye-level), wide (z=20 m, yaw=0).

Lighting (warehouse default)

Rendering

• RayTracedLighting (RT2), 1920x1080.
• 320x240 window with hideUi=1 to save GPU.
• Always set DISPLAY=:0; headless viewport init fails for complex regions.
• maxBounces=7, aovs=none.

Timing

DT = 1/60
Settle: 200-500 frames after timeline.play()
Capture: every 4th step (15 fps)

Workflow: multi-robot sim

1. Receive request (e.g. "6 Novas in a warehouse with 100 racks").
2. Compose scene: load warehouse USD, position robots via place_robots_grid.
3. Create collision groups if needed.
4. Use usd-pipeline to validate mesh scale and shaders.
5. Run in a persistent session: iterate live via isaac-sim-remote (Python server), or isaac-sim.sh --exec script.py for a standalone/handoff run.
6. Use a render-pulse loop every 100 steps.
7. Validate the render via isaac-sim-validator.
8. Deliver video and final scene.

Workflow: Physical AI end-to-end pipeline

Route chain for tasks that span asset import, physics, sensors, and validation (e.g. "import a Franka arm, simulate grasping, capture LiDAR + RGB, validate").

Pipeline stages

┌─────────────────────────┐     ┌──────────────────────┐     ┌───────────────────┐     ┌─────────────────────┐
│ 1. Asset Import         │────▶│ 2. Physics Setup     │────▶│ 3. Sensor Attach  │────▶│ 4. Validate & Ship  │
│                         │     │                      │     │                   │     │                     │
│ urdf-mjcf-to-usd-conv  │     │ physics-simulation   │     │ isaac-sim-sensor  │     │ isaac-sim-validator │
│ usd-pipeline            │     │ usd-articulation     │     │ isaac-camera      │     │ isaac-sim-rendering │
│ usd-composition-arch    │     │                      │     │ isaac-sim-remote  │     │                     │
└─────────────────────────┘     └──────────────────────┘     └───────────────────┘     └─────────────────────┘

Stage 1 — Asset import

Handoff to Stage 2: USD file(s) on disk, prim paths known, make_instanceable: true for RL workloads.

Stage 2 — Physics setup

Prerequisites from Stage 1:

• Robot USD must have IsaacRobotAPI on root (applied by importer).
• Static environment prims need CollisionAPI only (no RigidBodyAPI).
• PhysicsScene is always created here even if the importer applied per-joint attrs.

Handoff to Stage 3: Sim plays without crashes, robot holds pose under gravity for 200 frames.

Stage 3 — Sensor attachment

Mount-point convention: sensors attach to Xform prims under robot links. If the imported URDF lacks a mount link, create one:

mount = UsdGeom.Xform.Define(stage, f"{robot_path}/{link_name}/sensor_mount")

Handoff to Stage 4: At least one frame of non-zero sensor data (depth > 0, point cloud non-empty, IMU reports gravity).

Stage 4 — Validate and deliver

The validator gates delivery but does not cover runtime physics/sensor correctness. For runtime checks, verify:

1. RigidBodyAPI.GetVelocityAttr() stays finite across the sim window.
2. Sensor annotator data shape matches configured resolution/channels.
3. Articulation joint positions stay within drive limits.

End-to-end procedure

1. Phase 1a — Map request features to the pipeline stages above.
2. Phase 1b — Verify each stage in isolation:
   • Import: USD loads without errors, prim count reasonable for VRAM.
   • Physics: Robot holds pose, timeline.play() + 200 frames stable.
   • Sensors: One sensor produces valid data on a simplified scene.
3. Phase 2 — Integrate incrementally (import → physics → sensors), re-checking stability after each addition.
4. Phase 3 — Run isaac-sim-validator, capture final output, deliver.
5. Phase 4 — Distill lessons per skill-distillation.

Debug protocol

Rendering

Asset loading

Training

Operating rules

• Never delete work folders; reuse and branch.
• Always save the .usd file; never assume it lives only in memory.
• Validate every render; never deliver black frames.
• Every long-running process must be killable (pkill or pidfile).
• No bare ~/ in produced scripts; expand to $HOME or $ENV_VAR.
• make_instanceable: true for all RL robots.
• Call simulation_app.update() 5x after a camera switch.
• Never import torch before timeline.play().
• Log GPU memory every epoch.
• Lazy-load HoD assets (decompress on demand).
• Run skill-distillation (step 5 of the request loop) at task end. Capture lessons in the relevant SKILL.md, not in scratch memory.