USD Asset Pipeline

Purpose

Discover assets, measure bounds and shaders, swap placeholders, correct offsets, and validate headless shader compatibility before rendering.

Prerequisites

• Built Isaac Sim ($ISAAC_SIM_DIR or _build/linux-x86_64/release).
• NVIDIA GPU with a current driver (nvidia-smi).
• Shell env contract from isaac-sim-orchestrator: $ISAAC_SIM_DIR, $ISAAC_LAB_DIR, $WORKSPACE_DIR.

Limitations

• Targets Isaac Sim 6 / Kit 110 unless a section states otherwise.
• Does not replace official NVIDIA documentation for unsupported edge cases.

Troubleshooting

Available Scripts

Running scripts

From agent runtimes that expose skill execution helpers, invoke helpers with run_script():

run_script("scripts/articulation_builder.py", args=["--help"])

From a built Isaac Sim tree, run the same file with ./python.sh (Linux) or python.bat (Windows) from _build/*/release, or execute shell helpers directly when they do not require the simulator.

When to use

• Catalog USD assets from a folder tree (sizes, shaders, prim counts).
• Replace placeholder geometry (cubes/spheres/bboxes) with real assets.
• Build set-dressed scenes from modular libraries.
• Validate headless rendering compatibility (MDL vs UsdPreviewSurface).
• Cube-prototype to real-asset swap workflows.

Core Concepts

The Placeholder-to-Asset Pipeline

Real-world USD scene building follows this pattern:

1. Prototype with cubes — Layout spatial zones using colored UsdGeom.Cube meshes
2. Catalog assets — Measure every candidate USD asset (bbox, shaders, prim count)
3. Map blocks to assets — Match placeholder types to appropriately-sized real assets
4. Place with offset correction — Reference assets at block positions, correcting for asset bbox center offset
5. Validate renders — Vision model + domain expert scoring
6. Iterate — Fix overshoot, corridor intrusion, scale mismatches

Why Bbox Offset Correction Matters

Most USD assets are NOT centered at origin. A rack asset might have its bbox center at (46.7, 104.6, 4.5) — if you place it at the target position (112.0, 20.0, 0.0) without correction, it lands 46.7m east and 104.6m north of where you want it.

Formula:

translate_x = target_x - asset_bbox_center_x
translate_y = target_y - asset_bbox_center_y
translate_z = -asset_bbox_min_z  (puts asset base on ground plane)

Phase 1: Asset Discovery & Measurement

Script Pattern (Kit Python — No Renderer Needed)

measure_asset(path) — open a USD file, compute bbox, detect UsdPreviewSurface shader, count prims. Returns dict with width/depth/height/center/mpu/prims/dual_shader/shader_tag. catalog_assets(root_dir, extensions) — recursively find and measure all USD assets.

See scripts/measure_asset.py.

Key Learnings

• Use Usd.Stage.Open() not Sdf.Layer.FindOrOpen() — Sdf fails silently on binary .usd crate files, returns default mpu=1.0
• Always check mpu — Some assets use cm (mpu=0.01), some use meters (mpu=1.0). Scale measurements accordingly.
• Invalid bbox (min > 1e30) means the asset didn't compose — usually missing references or payloads
• Kit Python (kit/python/bin/python3) is fastest for measurement — no renderer startup needed
• For bbox in Kit runtime (SimulationApp), define a temp prim with reference, update a few frames, then compute bbox — more reliable for complex compositions

Phase 2: Shader Compatibility Check

The MDL Problem on Headless arm64

Rule: For headless rendering pipelines, ONLY use assets with UsdPreviewSurface fallback (dual-shader) or native UsdPreviewSurface.

Xvfb Discovery

Running Isaac Sim under DISPLAY=:99 (Xvfb virtual framebuffer) instead of the locked real display :0 produces non-black renders for some MDL assets. Not fully reliable but worth trying:

# Start Xvfb
Xvfb :99 -screen 0 1920x1080x24 &>/dev/null &

# Run with virtual display
DISPLAY=:99 isaac-sim.sh --exec script.py

Identifying Dual-Shader Assets

Look for these patterns in USD:

• info:id = "UsdPreviewSurface" on any Shader prim → headless-safe
• info:mdl:sourceAsset without UsdPreviewSurface sibling → MDL-only, headless-unsafe
• Lightspeed-processed assets typically = MDL-only
• "Collected" Dematic assets often = dual-shader

Phase 3: Placeholder-to-Asset Mapping

Strategy

1. Group placeholder cubes by name prefix (e.g., CvL001→CvL, BRk045→BRk)
2. For each prefix, find the best-fit asset by:
   • Similar function (racks→rack assets, conveyors→conveyor assets)
   • Compatible size (asset shouldn't massively overshoot the placeholder zone)
   • Dual-shader compatibility (headless rendering requirement)
3. Document the mapping table before building

Mapping Table Format

| Block Prefix | Count | Placeholder Size | Asset | Asset Size | Shader | Notes |
|---|---|---|---|---|---|---|
| BRk | 352 | 3.5×1.2×5.0m | ASRS_Racks_Center | 2.36×2.82×8.29m | dual | Per-block, no scaling |
| CvL | 190 | 2.0×4.0×1.5m | Conveyor_09 | 0.94×6.72×1.66m | dual | Roller conveyor |

Size Philosophy

Use natural asset sizes, NOT scaled-to-cube. Scaling assets to match cube dimensions destroys visual density and realism. Place at the block's XY position with the asset's natural dimensions.

Exception: If an asset is dramatically larger than its zone (e.g., 83m assembly in a 20m zone), use smaller modular pieces instead.

Phase 4: Placement Script Pattern

SimulationApp Runtime Placement

get_asset_bbox(stage, asset_path, app) — reference asset temporarily to get accurate bbox center. collect_blocks(stage) — group visible Cube prims by name prefix with world positions. place_assets(stage, blocks, asset_map, asset_bboxes, module_name) — place assets at block positions with bbox-center offset correction; hides original cubes.

See scripts/place_assets.py.

Hierarchy Convention

/World/
  Module1/          # Racks
    VNA/
      VNA001        # Individual asset reference
      VNA002
    BRk/
      BRk001
  Module2/          # Conveyors, sorters
    CvL/
      CvL001
    Sort/
      Sort001
  Module3/          # Safety, humans

Phase 5: Animation & Articulation

Animation Baking

Use bake_waypoints() from spatial-reasoning skill to animate robots, humans, or objects along paths.

bake_waypoints(xform_op, waypoints, speed_mps, fps, mpu) — keyframe an XformOp along a list of (x,y,z) waypoints. Defined in the spatial-reasoning skill at ../spatial-reasoning/scripts/spatial.py.

Articulation Builder

For robot USD:

build_forklift_articulation(output_path) — create chassis + fixed-joint mast + prismatic lift joint; includes _create_box, _create_fixed_joint, _create_prismatic_joint helpers.

See scripts/articulation_builder.py.

UV Mapping & Materials

For textures:

• Use sphereUV mapping for global assets
• Use linear for flat planes (floors, walls)
• Always use UsdPreviewSurface as fallback
• Never use MDL-only materials for headless

Phase 6: Validation

Color Key for Placeholder Flow

Format Validation Checklist

• All assets have .usd, .usda, or .usdc extension
• All paths use forward slashes /
• No .. relative paths
• mpu = 1.0 for all assets (script validates scale)
• No ./ or ../ syntax in references
• Topology: artifacts must not be nested under Xform if empty

Asset Recommendation (Based on Scan)

First, scan all assets with catalog_assets(), then:

• Filter: dual_shader is True
• Sort: prims < 5000
• Assign: Match bbox dimensions within 20% tolerance of placeholder cube
• Reject: All blockpallet_a*, palletstack_a* — these are not real assets

Hard-Won Lessons

1. Never scale assets to match cube dimensions — destroys visual density. Use natural sizes.
2. Large assemblies (>20m) rarely fit block clusters — use smaller modular pieces instead.
3. Always correct for bbox center offset — most assets aren't origin-centered.
4. Lightspeed-processed assets = MDL-only = BLACK on headless arm64. Only use "Collected" dual-shader variants.
5. Tote_01, Pallet_Pile, AMR_Table = MDL-only — they'll place but render black. Use Scissor_Lift or Electrical_Panel as functional placeholders.
6. Kill ALL kit processes before new Isaac Sim launch — zombie processes cause 90-170s cold starts.
7. Clean /dev/shm/carb-* between restarts to prevent SIGKILL.
8. SimulationApp headless requires explicit DomeLight + DistantLight — GUI adds viewport lights automatically, headless does NOT.
9. Xvfb (DISPLAY=:99) can improve MDL rendering over locked desktop (:0), but not a universal fix.