Isaac Sim Orchestrator

Purpose

Turn an established goal or demo contract into a runnable simulation by mapping capabilities, executing specialist skills in order, and validating output before delivery. Use isaac-sim-workflow first when the deliverable and acceptance criteria still need to be scoped.

Prerequisites

• Built Isaac Sim ($ISAAC_SIM_DIR or _build/linux-x86_64/release).
• NVIDIA GPU with a current driver (nvidia-smi).
• Shell env contract from isaac-sim-orchestrator: $ISAAC_SIM_DIR, $ISAAC_LAB_DIR, $WORKSPACE_DIR.

Limitations

• Targets Isaac Sim 6 / Kit 110 unless a section states otherwise.
• Does not replace official NVIDIA documentation for unsupported edge cases.

Troubleshooting

Environment contract

Every routed skill assumes these variables; set them in the agent config or shell:

Run nvidia-smi at session start to size num_envs and pick RT2 vs PathTracing. Do not hardcode GPU class.

Execution mode

Prefer live iteration: when a sim is running, push code over the Python server via isaac-sim-remote (port 8226). Write a standalone script only for handoff/repro; avoid python.sh except to test that script.

Launch a server-enabled sim using the launch procedure in isaac-sim-remote; do not duplicate its command here.

Task decomposition

For any request, run all four phases. The specific steps inside each phase depend on the goal; identify capabilities first, then verify each in isolation before combining.

Phase 1 — Verify foundations

1a. Feature/skill mapping (before any code):

• If the request is a demo, load isaac-sim-workflow first: it defines the deliverable type, acceptance criteria, validation needs, and routes back through these phases with that context.
• Cross-check the request against documented Isaac Sim features and APIs.
• For each capability, look up an existing skill:
  • Skill exists -> load it, follow its procedure.
  • Skill missing -> build one inline. Mark its frontmatter status: draft, flag it HIGH PRIORITY in skill-distillation, tell the user upfront, and shorten iteration cycles (share intermediate results, ask targeted questions early).
• Write the feature -> skill mapping into the task WORKLOG.md before 1b.

1b. Foundation verification (capability by capability):

• List every capability the task needs (assets, physics, robot control, sensors, rendering, ...).
• Verify each in isolation: does it load, does it behave correctly on its own.
• Do not move on until each foundation passes.

Phase 2 — Incremental integration

Combine verified foundations one at a time. Re-run stability/correctness checks after each addition. Every failure has exactly one new variable.

Phase 3 — Polish & deliver

• Validate output visually or programmatically. Task success, not just script completion.
• Add output-specific requirements (writers, annotations, video capture, DR).
• Package and hand off with a short summary.

Phase 4 — Distill (mandatory)

• List iterations, failures, workarounds.
• Record user corrections.
• Classify each lesson: new skill, skill update, procedure fix, or MEMORY.md fact.
• Update the skill files; re-read to confirm a fresh agent can follow them.

See skill-distillation for the full procedure. Phase 4 is not optional.

Sub-agent rules

• Each phase can run as a sub-agent with its own WORKLOG.md.
• Sub-agents commit at logical checkpoints, never half-done.
• Large script generation: write incrementally to files. Do not try to produce 200+ lines in one turn.
• If a sub-agent times out, WORKLOG.md survives for the next pickup.

Routed skills

For the modern Kit 110 public API surface (bootstrap, stage/app utilities, common calls) used across these skills, see references/api-cheatsheet.md.

Multi-robot fleet reference

Sample robots

Scene setup

• sim_warehouse_v4.usda pattern: shell + lights + racks + PhysicsScene + ground collision.
• Shell (sm_warehouse_mega.usd) is in cm; robots and equipment in meters.
• Strip physics from environment assets offline. Runtime stripping core-dumps on large stages.

PhysicsScene

from pxr import UsdPhysics, PhysxSchema

physics_scene = UsdPhysics.Scene.Define(stage, "/World/PhysicsScene")
physics_scene.CreateGravityDirectionAttr().Set((0, 0, -1))
physics_scene.CreateGravityMagnitudeAttr().Set(9.81)

physx_scene = PhysxSchema.PhysxSceneAPI.Apply(physics_scene.GetPrim())
physx_scene.CreateEnableCCDAttr().Set(True)
physx_scene.CreateEnableStabilizationAttr().Set(True)
physx_scene.CreateSolverTypeAttr().Set("TGS")
physx_scene.CreateTimeStepsPerSecondAttr().Set(60)
physx_scene.CreateGpuMaxNumPartitionsAttr().Set(8)   # 10+ robots

Grid placement

import math

def place_robots_grid(stage, robot_usd_path, prefix, count, spacing=3.0, start_z=0.0):
    cols = math.ceil(math.sqrt(count))
    robots = []
    for i in range(count):
        row, col = divmod(i, cols)
        x, y = col * spacing, row * spacing
        prim_path = f"/World/Robots/{prefix}_{i}"
        ref = stage.OverridePrim(prim_path)
        ref.GetReferences().AddReference(robot_usd_path)
        from pxr import UsdGeom, Gf
        xform = UsdGeom.Xformable(ref)
        xform.ClearXformOpOrder()
        xform.AddTranslateOp().Set(Gf.Vec3d(x, y, start_z))
        robots.append(prim_path)
    return robots

Separation

• Mobile robots: minimum 2 m between centers.
• Articulated arms: 1.5x reach radius minimum.
• Aerial: stagger altitudes by >= 2 m.

Collision groups

from pxr import PhysxSchema

def create_collision_group(stage, group_path, robot_paths):
    group = PhysxSchema.PhysxCollisionAPI.Apply(stage.DefinePrim(group_path))
    for path in robot_paths:
        prim = stage.GetPrimAtPath(path)
        collision_api = PhysxSchema.PhysxCollisionAPI.Apply(prim)
        collision_api.GetCollisionGroupsRel().AddTarget(group_path)

Scaling limits (by VRAM)

Limits scale approximately linearly with available VRAM. Beyond these thresholds risks CUDA OOM.

Optimization:

• make_instanceable: true in URDF config.yaml (shared mesh data).
• LOD switching for distant robots.
• Disable physics on robots outside the active zone.

Navigation

Differential drive kinematics:

vL = (vx - omega * tw/2) / wheel_r
vR = (vx + omega * tw/2) / wheel_r

PD steering defaults: KP=2.5, KD=1.2, MAX_W=1.5, waypoint tolerance 4.0 m. Out-of-bounds: |Z| > 50 or |X|/|Y| > 500 -> mark dead.

Camera

Chase camera: 12 m behind, min height 2.5 m, clamped inside warehouse bounds, smooth interpolation alpha = min(1.0, DT*2.0). Dynamically raise camera to avoid rack intrusion.

View modes (cycle every 4 s): chase, overhead (z=50 m), aisle (eye-level), wide (z=20 m, yaw=0).

Lighting (warehouse default)

Rendering

• RayTracedLighting (RT2), 1920x1080.
• 320x240 window with hideUi=1 to save GPU.
• Always set DISPLAY=:0; headless viewport init fails for complex regions.
• maxBounces=7, aovs=none.

Timing

DT = 1/60
Settle: 200-500 frames after timeline.play()
Capture: every 4th step (15 fps)

Workflow: multi-robot sim

1. Receive request (e.g. "6 Novas in a warehouse with 100 racks").
2. Compose scene: load warehouse USD, position robots via place_robots_grid.
3. Create collision groups if needed.
4. Use usd-pipeline to validate mesh scale and shaders.
5. Run in a persistent session: iterate live via isaac-sim-remote (Python server), or isaac-sim.sh --exec script.py for a standalone/handoff run.
6. Use a render-pulse loop every 100 steps.
7. Validate the render via isaac-sim-validator.
8. Deliver video and final scene.

Workflow: Physical AI end-to-end pipeline

Route chain for tasks that span asset import, physics, sensors, and validation (e.g. "import a Franka arm, simulate grasping, capture LiDAR + RGB, validate").

Pipeline stages

┌─────────────────────────┐     ┌──────────────────────┐     ┌───────────────────┐     ┌─────────────────────┐
│ 1. Asset Import         │────▶│ 2. Physics Setup     │────▶│ 3. Sensor Attach  │────▶│ 4. Validate & Ship  │
│                         │     │                      │     │                   │     │                     │
│ urdf-mjcf-to-usd-conv  │     │ physics-simulation   │     │ isaac-sim-sensor  │     │ isaac-sim-validator │
│ usd-pipeline            │     │ usd-articulation     │     │ isaac-camera      │     │ isaac-sim-rendering │
│ usd-composition-arch    │     │                      │     │ isaac-sim-remote  │     │                     │
└─────────────────────────┘     └──────────────────────┘     └───────────────────┘     └─────────────────────┘

Stage 1 — Asset import

Handoff to Stage 2: USD file(s) on disk, prim paths known, make_instanceable: true for RL workloads.

Stage 2 — Physics setup

Prerequisites from Stage 1:

• Robot USD must have IsaacRobotAPI on root (applied by importer).
• Static environment prims need CollisionAPI only (no RigidBodyAPI).
• PhysicsScene is always created here even if the importer applied per-joint attrs.

Handoff to Stage 3: Sim plays without crashes, robot holds pose under gravity for 200 frames.

Stage 3 — Sensor attachment

Mount-point convention: sensors attach to Xform prims under robot links. If the imported URDF lacks a mount link, create one:

mount = UsdGeom.Xform.Define(stage, f"{robot_path}/{link_name}/sensor_mount")

Handoff to Stage 4: At least one frame of non-zero sensor data (depth > 0, point cloud non-empty, IMU reports gravity).

Stage 4 — Validate and deliver

The validator gates delivery but does not cover runtime physics/sensor correctness. For runtime checks, verify:

1. RigidBodyAPI.GetVelocityAttr() stays finite across the sim window.
2. Sensor annotator data shape matches configured resolution/channels.
3. Articulation joint positions stay within drive limits.

End-to-end procedure

1. Phase 1a — Map request features to the pipeline stages above.
2. Phase 1b — Verify each stage in isolation:
   • Import: USD loads without errors, prim count reasonable for VRAM.
   • Physics: Robot holds pose, timeline.play() + 200 frames stable.
   • Sensors: One sensor produces valid data on a simplified scene.
3. Phase 2 — Integrate incrementally (import → physics → sensors), re-checking stability after each addition.
4. Phase 3 — Run isaac-sim-validator, capture final output, deliver.
5. Phase 4 — Distill lessons per skill-distillation.

Debug protocol

Rendering

Asset loading

Training

Operating rules

• Never delete work folders; reuse and branch.
• Always save the .usd file; never assume it lives only in memory.
• Validate every render; never deliver black frames.
• Every long-running process must be killable (pkill or pidfile).
• No bare ~/ in produced scripts; expand to $HOME or $ENV_VAR.
• make_instanceable: true for all RL robots.
• Call simulation_app.update() 5x after a camera switch.
• Never import torch before timeline.play().
• Log GPU memory every epoch.
• Lazy-load HoD assets (decompress on demand).
• Run skill-distillation (step 5 of the request loop) at task end. Capture lessons in the relevant SKILL.md, not in scratch memory.