Action and Event Data Generation

Purpose

Route a synthetic data generation (SDG) request to the right Action and Event Data Generation (AEDG) skills, and carry the context every one of them assumes: the launcher, the extension stack, the env-var contract, and the order the stages compose in.

Use this whenever the goal is to do SDG with simulated humans and robots moving with behaviors, physical events and incidents (spills, fires, toppling), procedurally placed or packed objects, camera coverage, and scene captions — plus the ground-truth annotations that come with them.

Prerequisites

• Isaac Sim with the AEDG app ($ISAAC_SIM_DIR) — launch with isaac-sim.action_and_event_data_generation.sh.
• NVIDIA GPU with a current driver (nvidia-smi) for any actual run; the offline config generators need neither GPU nor simulator.
• Shell env contract from isaac-sim-orchestrator: $ISAAC_SIM_DIR, $WORKSPACE_DIR.
• $NVIDIA_API_KEY only for the LLM-backed features (captioning, behavior-tree generation).

Limitations

• This skill routes and supplies shared context; it does not itself run a simulation. Every concrete workflow lives in a sub-skill below.
• The AEDG extensions are authored in the metrosim repo and consumed here as exact registry pins. These skills drive them; API changes belong upstream.
• Config version rules differ per extension and are not interchangeable — see Version rules below. This is the single most common cause of a config being rejected.

Troubleshooting

The stack

Eight extensions. isaacsim.exp.action_and_event_data_generation.base.kit holds the exact version each one resolves to — read it there rather than from any skill, which would go stale the moment the app moves:

Shared substrate: omni.metropolis.pipeline (OMP) and omni.metropolis.utils (OMU).

How the stages compose

Per the architecture in the product docs — object simulation defines the static scene, events and actors add dynamics, then sensors and captioning capture it:

   Object Simulation (IRO)          ← static environment, procedural placement
            │
            ├── Event Generation (IRI)      ← spills / fires / toppling
            └── Actor Simulation (IRA)      ← people + robots with behaviors
                     │
                     │  (IRA calls ISP itself — see below)
                     ▼
   Sensor Placement (ISP) → Calibration (ISC)   ← where cameras go, and their intrinsics
            │
            ▼
   VLM Scene Captioning (IRC)      ← scene graphs + captions over the rendered views

ISP is not always a separate step. An IRA config can place its own cameras, in which case the standalone placement skills are unnecessary — see the next section.

ISP is optional: IRA places cameras from its own config

Each sensor.groups.<name> in an actor config carries a placement block, and IRA's scene assembly calls straight into isaacsim.sensors.rtx.placement while building the scene (scene_assembly/sensor_loader.py imports CameraPlacementManager and CircularCameraPlacement). So for a config-driven actor run, camera placement is already handled:

  sensor:
    root_prim_path: /World/Cameras     # default
    groups:
      ceiling_cameras:
        num: 6
        aim_at_targets:                # or: maximum_coverage
          height_range: [7.0, 10.0]
          look_down_angle_range: [30.0, 45.0]
          distance_range: [5.0, 10.0]
          focal_length_range: [10.0, 15.0]

Two consequences worth knowing:

• Placement is opt-out, not opt-in. SensorGroup.placement has a default factory, so a group with only num: still runs aim_at_targets. You get ISP whether or not you asked for it.
• Typos fail fast. The group model is extra="forbid" and the placement key is extracted before that check, so aim_at_target (singular) is rejected rather than silently falling back to the default.

So when do you need the standalone camera skills?

Calibration (ISC) is genuinely a separate stage: IRA places cameras but never calibrates them.

I want to… → use this

Adjacent, outside AEDG: isaac-camera for hand-authoring one camera's intrinsics and AOVs, and isaac-sim-remote for the python_server socket several of these skills send payloads over.

Launch

# from the Isaac Sim build root
./isaac-sim.action_and_event_data_generation.sh

Runs kit/kit apps/isaacsim.exp.action_and_event_data_generation.full.kit. Panels appear under Tools > Action and Event Data Generation (Actor / Object / Incident) and Tools > Sensors (Camera Placement / Calibration).

Several sub-skills drive a running instance over the python_server socket rather than the GUI. Add the extension to expose it:

./isaac-sim.action_and_event_data_generation.sh --enable isaacsim.code_editor.python_server

Headless, config-driven actor runs use the bundled script. Before running, explain that the configuration and all referenced assets, including USD layers and scripts, can execute Python with the user's permissions. Obtain explicit acknowledgement that they are trusted; this is not a sandbox. Do not add --allow-trusted-scripts automatically or run without that acknowledgement.

./python.sh tools/actor_sdg/actor_sdg.py --config_file my_scene.yaml --allow-trusted-scripts

Version rules (read before authoring any config)

Each extension validates its config version differently. Copying a sample and editing it is the usual way to get rejected:

Always derive the value from the installed build — never hardcode it, and never copy a version out of a skill or a sample config. The sub-skills' generator scripts do this with --from-ext.

Env vars expand in the shell, not in YAML

Configs are read with plain yaml.safe_load, so $VAR and ${VAR} stay literal inside a config file. Substitute before running — in a shell command that writes the resolved path, via envsubst < in.yaml > out.yaml, or by hand.

Cost

• Runtime scales with duration, frame rate, camera count, and render mode; path tracing is far slower than rasterization. Short warehouse runs are minutes, large multi-camera runs are hours.
• GPU required. The first captured frame on a cold app pays one-time shader compilation.
• NIM credits are consumed by captioning and behavior-tree generation. Scene-graph-only captioning needs no key.

Product documentation

docs/isaacsim/action_and_event_data_generation/ — index.rst for the extension table and architecture, plus per-extension tutorials (tutorial_replicator_agent.rst, tutorial_replicator_object.rst, tutorial_replicator_incident.rst, tutorial_replicator_caption.rst, tutorial_sensors_rtx_placement.rst, tutorial_behavior_tree_gen.rst, tutorial_telemetry.rst) and the worked example example_event_reactive_actors.rst.