Behavior Tree Generation

Purpose

Turn a natural-language scenario into behavior-tree output using an LLM-driven planner. This is a focused sub-skill of Action and Event Data Generation, packaged as omni.ai.behavior_tree_gen.core (scripted pipeline + API) and omni.ai.behavior_tree_gen.bridge (Kit UI).

Prerequisites

• Installed Isaac Sim with the Action and Event Data Generation app ($ISAAC_SIM_DIR).
• NVIDIA GPU with a current driver (nvidia-smi) for an actual run (offline helper scripts need neither).
• Shell env contract from isaac-sim-orchestrator: $ISAAC_SIM_DIR, $WORKSPACE_DIR.
• $NVIDIA_API_KEY for the chat/embedding models (prepare_runtime fails without it).

Limitations

• Requires a valid NVIDIA API key; prepare_runtime() does not fall back to a local model.
• Strict call order — setup_workspace() → prepare_runtime() (must return success=True) → generate_behavior_tree().
• Bundled example actions (e.g. MoveTo) are transitional: they demonstrate extensibility, not production quality, and can misbehave.
• This generates a behavior tree from text; it is distinct from the hand-authored actor behavior_tree JSON consumed by an Actor SDG (isaacsim.replicator.agent) config.

Available Scripts

Running scripts

From agent runtimes that expose skill execution helpers, invoke with run_script():

run_script("scripts/starter_context.py", args=["--help"])

Turn a natural-language scenario into behavior-tree output using an LLM-driven planner. Part of Isaac Sim's Action and Event Data Generation feature (launch the app with isaac-sim.action_and_event_data_generation.sh). This skill covers only the behavior-tree generation workflow.

When to use (vs siblings)

Use to turn a natural-language scenario into a behavior tree (LLM pipeline). Distinct from the hand-authored actor behavior_tree config that an Actor SDG (isaacsim.replicator.agent) group consumes — this generates the tree.

Environment

Follows the library env-var contract (see isaac-sim-orchestrator): $ISAAC_SIM_DIR, $WORKSPACE_DIR. Needs $NVIDIA_API_KEY for the chat/embedding models (prepare_runtime fails without it). Write outputs to $WORKSPACE_DIR/bt instead of a hardcoded path.

Not the same as the actor behavior_tree config key. An Actor SDG (isaacsim.replicator.agent) group consumes a hand-authored JSON behavior tree to drive a character/robot group. This skill generates a behavior tree from a text scenario via an LLM pipeline. The output of this workflow can seed the trees the actor skill runs, but the two are different systems.

Extensions

Run (UI)

1. Enable omni.ai.behavior_tree_gen.bridge (it pulls in .core).
2. Open Tools > Behavior Tree Gen.
3. Optional: Window > Examples > Behavior Tree Gen Examples → load the bundled Basic Scene or Warehouse Scene. This loads a demo stage and pre-fills the workflow panels.
4. In Behavior Tree Gen: confirm the Context Cache Files (context JSON, node catalogs, metadata schemas), the Network Config (NVIDIA API key + model JSON), and the Output Settings folder; enter the scenario text in the Planner panel; click Run Pipeline.

Output behavior-tree files are written under the selected output folder; planner/RAG cache goes under the derived cache directory.

Run (scripted API)

The UI is a thin wrapper over three public calls in omni.ai.behavior_tree_gen.core.api, used in this exact order — each prepares state the next consumes:

import os
from pathlib import Path
from omni.ai.behavior_tree_gen.core import api as core_api

OUTPUT_DIR = Path(os.environ["WORKSPACE_DIR"]) / "bt"   # not "Your/Output/Folder/Path"

session = core_api.setup_workspace(          # 1. sync — build the reusable PlannerSession
    cache_dir=str(OUTPUT_DIR / "planner_cache"),
    output_dir=str(OUTPUT_DIR),
    context_data_paths=actor_context_paths + object_context_paths,
    node_catalog_paths=node_catalog_paths,
    actor_schema_path=actor_schema_path,
    object_schema_path=object_schema_path,
)

runtime = await core_api.prepare_runtime(    # 2. async — configure LLM/embeddings/RAG/Action IR
    session,
    api_key=API_KEY,                          # NVIDIA API key (UI, carb setting, or NVIDIA_API_KEY)
    model_selection_config_path=model_selection_config_path,
)
if not runtime.success:
    raise RuntimeError(runtime.message)

result = await core_api.generate_behavior_tree(session, SCENARIO)   # 3. async — emit the tree
if not result.success:
    raise RuntimeError(result.error_message)
print(result.behavior_tree_folder_path)

setup_workspace() is synchronous; prepare_runtime() and generate_behavior_tree() are coroutines. In Script Editor, wrap all three in one async def and asyncio.ensure_future(run()). See references/api-and-inputs.md for the full parameter list, return fields, and the required-inputs breakdown.

Required inputs (minimum)

• Scenario text — the natural-language goal.
• Output folder — writable; holds generated trees + reusable cache.
• NVIDIA API key — needed by prepare_runtime() for NVIDIA-hosted chat/embedding models (from the UI, a carb setting, or the NVIDIA_API_KEY env var).
• Context JSON — actor + object instances (ActorInfo / InteractableObjectInfo).
• Node-catalog JSON — the behavior-tree nodes the planner may use.
• Metadata schemas — actor/object JSON Schemas that give metadata fields meaning.

Authoring context/schema: prefer the bundled example files under the bridge's data/example/context_info/ (and .../schemas/) as your reference — they match the current build. As an optional offline quick-start you can also generate a starter context + schema pair (then edit them):

python3 scripts/starter_context.py --entity object --id Table > table_context.json
python3 scripts/starter_context.py --emit-schema object > object_metadata_schema.json

Verify it worked

# result.behavior_tree_folder_path is the authoritative location; it lives under the output_dir
# you passed to setup_workspace ($WORKSPACE_DIR/bt).
ls "$WORKSPACE_DIR/bt" 2>/dev/null && echo "tree written" || echo "no tree — check NVIDIA_API_KEY + that prepare_runtime returned success"

A successful run sets result.success and writes tree files under the output folder; failures are almost always a missing $NVIDIA_API_KEY or prepare_runtime not returning success before generate_behavior_tree.

Integration points

• Consumes: actor/object context JSON + node-catalog JSON + metadata schemas + a scenario string; an $NVIDIA_API_KEY.
• Produces: behavior-tree output files that can seed the behavior_tree key of an Actor SDG (isaacsim.replicator.agent) group.

Troubleshooting

• Call order — setup_workspace → prepare_runtime → generate_behavior_tree. prepare_runtime() must return success=True before generate_behavior_tree() works.
• Missing API key — prepare_runtime() fails without a valid NVIDIA API key; it does not fall back to a local model.
• Context vs schema — context supplies instance data; the schema defines the metadata structure. Base fields (id, semantic_description, supported_interactions, entity_type) stay top-level; schema-defined fields go under metadata. Required by the shipped schemas: actors need metadata.prim_path + metadata.actor_type; objects need metadata.prim_path + metadata.interactable_type.
• Stale workspace — after editing a tracked input file (context, catalog, schema, model config), reload the workspace so the typed models rebuild.
• Example actions are transitional — bundled custom actions (e.g. MoveTo) can misbehave (paths overlapping the target); they demonstrate extensibility, not production quality.