Data Collection Simulation

Purpose

Produce annotated static-scene synthetic data with Replicator writers (RGB, depth, segmentation, pose, Kitti/COCO variants) in headless batch runs.

Prerequisites

• Built Isaac Sim ($ISAAC_SIM_DIR or _build/linux-x86_64/release).
• NVIDIA GPU with a current driver (nvidia-smi).
• Shell env contract from isaac-sim-orchestrator: $ISAAC_SIM_DIR, $ISAAC_LAB_DIR, $WORKSPACE_DIR.

Limitations

• Targets Isaac Sim 6 / Kit 110 unless a section states otherwise.
• Does not replace official NVIDIA documentation for unsupported edge cases.

Troubleshooting

Build headless SDG pipelines using Isaac Sim 6.0 Replicator. Outputs annotated frames (RGB, depth, segmentation, bbox, pose) to disk via writers.

Available Scripts

Running scripts

From agent runtimes that expose skill execution helpers, invoke helpers with run_script():

run_script("scripts/minimal_sdg_pipeline.py", args=["--help"])

From a built Isaac Sim tree, run the same file with ./python.sh (Linux) or python.bat (Windows) from _build/*/release, or execute shell helpers directly when they do not require the simulator.

Related Skills

• mobility-gen — mobile-robot trajectory recording then replay+render (two-phase, robot-mounted sensors)
• isaac-sim-sensor — sensor primitives (camera, LiDAR, IMU, contact)
• occupancy-map — produces occupancy maps for spawn placement

Architecture

Config (YAML/JSON) → SimulationApp (headless) → Scene Setup → Randomizers → Capture Loop → Writer → Disk

Required Imports

from isaacsim import SimulationApp
simulation_app = SimulationApp({"renderer": "RealTimePathTracing", "headless": True})

import carb.settings
import omni.replicator.core as rep
import omni.usd
import isaacsim.core.experimental.utils.stage as stage_utils
from isaacsim.core.experimental.utils.semantics import add_labels, remove_all_labels
from isaacsim.storage.native import get_assets_root_path

stage_utils.open_stage() / stage_utils.add_reference_to_stage() / stage_utils.define_prim() / stage_utils.get_current_stage() are the Kit 110 replacements for the legacy isaacsim.core.utils.stage.* and isaacsim.core.utils.prims.create_prim flow. Keep omni.replicator.core for SDG primitives.

Migration: for the full omni.isaac.* → isaacsim.* mapping, see Renaming Extensions.

Writers

Resolve writers via rep.WriterRegistry.get(name) (or rep.writers.get(name), same registry).

omni.replicator.core built-in writers (omni.replicator.core/scripts/writers_default/):

isaacsim.replicator.writers adds Isaac-specific writers:

Deprecated and not for new work: DOPEWriter, YCBVideoWriter, PytorchWriter, PytorchListener (also OgnPose node). They will be removed in a future major release.

Writer initialization patterns

Two equivalent patterns are supported. Recent (Isaac Sim 6.0+) examples favor the explicit-backend form:

backend = rep.backends.get("DiskBackend")
backend.initialize(output_dir="/tmp/sdg_output")
writer = rep.writers.get("BasicWriter")
writer.initialize(backend=backend, rgb=True, bounding_box_2d_tight=True)
writer.attach(rp)

Legacy short form (still works; backend created implicitly from output_dir):

writer = rep.WriterRegistry.get("BasicWriter")
writer.initialize(output_dir="/tmp/sdg_output", rgb=True, bounding_box_2d_tight=True)
writer.attach(rp)

Annotators

Core annotators available via rep.annotators.get(name):

• rgb — RGBA uint8
• distance_to_image_plane — depth float32
• semantic_segmentation — per-pixel class labels
• instance_segmentation — per-pixel instance IDs
• bounding_box_2d_tight — tight 2D bboxes
• bounding_box_2d_loose — loose 2D bboxes
• bounding_box_3d — 3D bboxes in world coords
• camera_params — intrinsics, extrinsics, resolution
• occlusion — visibility ratio per instance
• normals — surface normals
• pointcloud — 3D point cloud from depth

Minimal Pipeline

run_minimal_sdg_pipeline(output_dir, num_frames, rt_subframes) — open a warehouse stage, tag a prop with semantic labels, create a camera, attach a BasicWriter, and run the capture loop.

See scripts/minimal_sdg_pipeline.py.

Full warehouse pipeline

For a config-driven, headless warehouse capture (YAML config, --num-frames, --output-dir), see scripts/warehouse_sdg.py. Validate the produced dataset with scripts/validate_sdg_output.sh:

bash scripts/validate_sdg_output.sh <output_dir> [expected_frames]

Domain Randomization

Use rep.functional API for randomization each frame:

# Scatter objects on surface
rep.functional.randomizer.scatter_2d(prims=objects, surface_prims=plane, check_for_collisions=True, rng=rng)

# Randomize camera pose
rep.functional.modify.pose(cam, position_value=rng.uniform(pos_min, pos_max),
                           look_at_value=target_prim, look_at_up_axis=(0, 0, 1))

# Randomize lights via OmniGraph events
rep.utils.send_og_event(event_name="randomize_lights")

Headless Execution

The canonical Isaac Sim Python launcher is python.sh (use python.bat on Windows). isaac-sim.sh launches the full editor app and is not the right entry point for standalone SDG scripts.

# $ISAAC_SIM_DIR is either the install root or <repo>/_build/linux-x86_64/release
"$ISAAC_SIM_DIR/python.sh" "$WORKSPACE_DIR/data_collection.py" --config config.yaml

Headlessness is controlled by SimulationApp({"headless": True}) inside the script, not by a launcher flag.

Or via the Isaac Lab runner ($ISAAC_LAB_DIR is your Isaac Lab checkout):

"$ISAAC_LAB_DIR/isaaclab.sh" -p data_collection.py --config config.yaml

Configuration Pattern

Use YAML config to parameterize everything:

resolution: [1280, 720]
rt_subframes: 32
num_frames: 100
headless: true
env_url: "/Isaac/Environments/Simple_Warehouse/full_warehouse.usd"
writer: BasicWriter
output_dir: /tmp/sdg_output
annotations:
  rgb: true
  bounding_box_2d_tight: true
  semantic_segmentation: true
  distance_to_image_plane: true
  bounding_box_3d: true
objects:
  - url: "/Isaac/Props/YCB/Axis_Aligned/003_cracker_box.usd"
    label: cracker_box
    count: 5
  - url: "/Isaac/Props/YCB/Axis_Aligned/008_pudding_box.usd"
    label: pudding_box
    count: 3

Validation Checklist

1. Output directory contains expected number of frames
2. RGB images are non-black (mean RGB > 30)
3. Annotation files match frame count
4. Semantic labels appear in segmentation maps
5. Bounding boxes have non-zero area
6. No NaN in depth maps

Key Rules

• Set rep.orchestrator.set_capture_on_play(False) for manual step control.
• rt_subframes: render the same frame multiple times to reduce ghosting from large pose deltas and to let materials/textures converge. Tune for your renderer: small (4-8) is often enough for RTX Real-Time + DLSS Quality; 16-32 is typical for path tracing or scenes with heavy material streaming.
• DLSS Quality: carb.settings.get_settings().set("rtx/post/dlss/execMode", 2). Recommended for SDG; default Performance mode can produce edge artifacts below ~600x600.
• Tag every prim you want annotated via add_labels(...) (taxonomy-aware) or rep.functional.modify.semantics(...). Both write the UsdSemantics.LabelsAPI schema.
• For static-scene SDG, pass delta_time=0.0 to rep.orchestrator.step so the timeline does not advance between captures.
• Call rep.orchestrator.wait_until_complete() before cleanup so the background backend has flushed everything to disk.
• Use rng = np.random.default_rng(seed) and rep.set_global_seed(seed) for reproducible randomization.
• Performance knobs for high-throughput captures (see sdg_getting_started_05.py):
  • rep.orchestrator.step(wait_for_render=False) decouples capture from render completion. Data may correspond to a previous frame; only use when strict frame-to-data correspondence is not required.
  • carb.settings.get_settings().set("/exts/omni.replicator.core/enableWriteToFabric", True) writes randomization deltas directly to Fabric instead of going through USD first. Faster, but transient — changes are not persisted in the USD stage.

Sibling SDG workflows

Reference examples

Paths relative to $ISAAC_SIM_DIR (install root or <this-repo> for a source build) and $ISAAC_LAB_DIR:

• API examples: $ISAAC_SIM_DIR/source/standalone_examples/api/isaacsim.replicator.examples/sdg_getting_started_0[1-5].py, sdg_workflow_0[12].py, multi_camera.py, motion_blur_raytracing.py, motion_blur_pathtracing.py, cosmos_writer_simple.py, simready_assets_sdg.py, sdg_deformables.py, sdg_geomsubset.py, subscribers_and_events.py, custom_event_and_write.py, custom_fps_writer_annotator.py.
• Scene-based SDG: $ISAAC_SIM_DIR/source/standalone_examples/replicator/scene_based_sdg/
• Object-based SDG: $ISAAC_SIM_DIR/source/standalone_examples/replicator/object_based_sdg/
• Augmentation pipelines: $ISAAC_SIM_DIR/source/standalone_examples/replicator/augmentation/
• Cosmos warehouse writer: $ISAAC_SIM_DIR/source/standalone_examples/replicator/cosmos_writer_warehouse.py
• Infinigen SDG: $ISAAC_SIM_DIR/source/standalone_examples/replicator/infinigen/
• Grasping SDG: $ISAAC_SIM_DIR/source/standalone_examples/api/isaacsim.replicator.grasping/grasping_workflow_sdg.py
• Isaac Lab imitation learning: $ISAAC_LAB_DIR/scripts/imitation_learning/