Isaac Sim Remote

Purpose

Drive an already-running Isaac Sim over the code-editor TCP socket so agents can iterate on stages, prims, physics, and screenshots without restarting the app.

Security

This skill is a local developer control plane, not a remote-execution service.

• Bind: clients default to 127.0.0.1:8226. Do not publish that port off-host.
• Trust boundary: equal to the OS user who launched Isaac Sim on this machine.
• Design (intentional): isaacsim_send.py delivers Python source to the local code-editor server; execute_command.py dispatches any name already present in omni.kit.commands.get_commands() after an identifier check and audit log. An allowlist would defeat live Kit iteration — treat registry-wide dispatch as the product, not a hole.
• Mitigations: loopback-only default; --dry-run prints payloads without send; CLI --arg values use ast.literal_eval (literals only).

Prerequisites

• Built Isaac Sim ($ISAAC_SIM_DIR or _build/linux-x86_64/release).
• NVIDIA GPU with a current driver (nvidia-smi).
• Shell env contract from isaac-sim-orchestrator: $ISAAC_SIM_DIR, $ISAAC_LAB_DIR, $WORKSPACE_DIR.

Limitations

• Requires Isaac Sim launched with the Python code-editor server enabled.
• Remote scripts cannot import arbitrary repo modules unless paths are injected.

Troubleshooting

Related: debug-with-local-kit (when behavior depends on a Kit-from-source build), profile-isaac-sim (to attach Tracy to the running process), isaac-sim-validator (final QA gate on any rendered output).

Upstream isaac-sim-ui (menu/widget OmniUIQuery automation) and isaac-sim-recording (cursor tracking, tutorial video capture) are not imported. The inline UI patterns here (Play-button click via OmniUIQuery, full-app vs viewport screenshots) cover the common cases.

Every helper in scripts/ is invoked through scripts/isaacsim_send.py. From agent runtimes that expose skill execution helpers, call e.g. run_script("scripts/app_screenshot.py", args=["--help"]); from a built tree run the file with ./python.sh (Linux) / python.bat (Windows) under _build/*/release. Command recipes live in references/workflows.md.

Available Scripts

Launching Isaac Sim

cd _build/linux-x86_64/release
bash isaac-sim.sh --no-window --no-ros-env \
    --enable isaacsim.code_editor.python_server

Wait for app ready. TCP server: 127.0.0.1:8226. Use --enable isaacsim.code_editor.python_server (not carb /enabled=true). Full launch variants and health/asset/send/screenshot/control workflows: references/workflows.md. Pitfalls: references/pitfalls.md. APIs: references/api-reference.md.

Important Notes

• Lighting in headless mode: No lights = black image. Always add a DomeLight with intensity 1000–5000.
• Renderer warm-up: After new stage, call await app_utils.update_app_async(steps=120) before rendering.
• Connection errors: Ensure Isaac Sim shows app ready and python_server is --enabled.
• Server hangs: Crashed examples can wedge the server. Restart Isaac Sim. Use health_check.py to verify.
• Assets not found: Configure S3 fallback with set_asset_root.py.