3D Spatial Reasoning

Purpose

Apply coordinate math, bounding-box analysis, collision-free layouts, look-at placement, and lightweight path/grid helpers when composing USD scenes.

Prerequisites

• Built Isaac Sim ($ISAAC_SIM_DIR or _build/linux-x86_64/release).
• NVIDIA GPU with a current driver (nvidia-smi).
• Shell env contract from isaac-sim-orchestrator: $ISAAC_SIM_DIR, $ISAAC_LAB_DIR, $WORKSPACE_DIR.

Limitations

• Targets Isaac Sim 6 / Kit 110 unless a section states otherwise.
• Does not replace official NVIDIA documentation for unsupported edge cases.

Troubleshooting

Core spatial math for placing objects correctly in OpenUSD scenes.

Available Scripts

Running scripts

From agent runtimes that expose skill execution helpers, invoke helpers with run_script():

run_script("scripts/spatial.py", args=["--help"])

From a built Isaac Sim tree, run the same file with ./python.sh (Linux) or python.bat (Windows) from _build/*/release, or execute shell helpers directly when they do not require the simulator.

Coordinate System

• Stage: Z-up, metersPerUnit=1.0 (meters)
• All placement coordinates are in meters
• Rotations: always AddRotateXYZOp() — never individual axis ops

metersPerUnit Conversion (CRITICAL)

Assets have their own metersPerUnit. Common values:

• 1.0 = meters (robots, environments)
• 0.01 = centimeters (most KION lightspeed assets, SimModelAnimation, compositions)

Rule: Before referencing any asset, read its metersPerUnit:

asset_stage = Usd.Stage.Open(asset_path)  # Must use Usd.Stage, NOT Sdf.Layer
mpu = UsdGeom.GetStageMetersPerUnit(asset_stage)

Sdf.Layer.FindOrOpen() FAILS SILENTLY on binary .usd crate files — always returns None. Never use it for mpu detection.

Getting real-world size of an asset:

bbox_cache = UsdGeom.BBoxCache(Usd.TimeCode.Default(), [UsdGeom.Tokens.default_])
raw_range = bbox_cache.ComputeWorldBound(default_prim).ComputeAlignedRange()
raw_size = raw_range.GetMax() - raw_range.GetMin()
real_size_meters = [raw_size[i] * mpu for i in range(3)]

Transform Matrix Order (CRITICAL)

When placing an asset with scale + rotation + translation:

# CORRECT: Translate * Rotate * Scale
# Scale shrinks asset to meters, Rotate orients it, Translate positions it
scale_mat = Gf.Matrix4d().SetScale(Gf.Vec3d(mpu, mpu, mpu))
rot_mat = Gf.Matrix4d().SetRotate(Gf.Rotation(Gf.Vec3d(0,0,1), heading_deg))
trans_mat = Gf.Matrix4d().SetTranslate(Gf.Vec3d(x, y, z))
xf.MakeMatrixXform().Set(trans_mat * rot_mat * scale_mat)

WRONG: scale * translate scales the translation vector too — a 3m offset becomes 0.03m for mpu=0.01.

Placement Helper

def place(stage, prim_path, asset_path, x, y, z=0, rot_z=0, mpu=0.01):
    """Place an asset with correct metersPerUnit scaling."""
    prim = stage.DefinePrim(prim_path, "Xform")
    prim.GetReferences().AddReference(asset_path)
    xf = UsdGeom.Xformable(prim)
    S = Gf.Matrix4d().SetScale(Gf.Vec3d(mpu, mpu, mpu))
    R = Gf.Matrix4d().SetRotate(Gf.Rotation(Gf.Vec3d(0,0,1), rot_z))
    T = Gf.Matrix4d().SetTranslate(Gf.Vec3d(x, y, z))
    xf.MakeMatrixXform().Set(T * R * S)
    return prim

Look-At Camera Math

def look_at_rotation(cam_pos, target_pos):
    """Compute XYZ Euler rotation for camera to look at target. Z-up stage."""
    dx = target_pos[0] - cam_pos[0]
    dy = target_pos[1] - cam_pos[1]
    dz = target_pos[2] - cam_pos[2]
    horiz = math.sqrt(dx*dx + dy*dy)
    pitch = math.degrees(math.atan2(horiz, -dz))
    yaw = math.degrees(math.atan2(dx, -dy))
    return Gf.Vec3f(pitch, 0, yaw)

Grid Layout

def grid_positions(n_items, spacing, origin=(0,0)):
    """Generate grid positions for n items with given spacing."""
    cols = min(8, math.ceil(math.sqrt(n_items * 1.5)))
    positions = []
    for i in range(n_items):
        row, col = divmod(i, cols)
        x = origin[0] + col * spacing
        y = origin[1] + row * spacing
        positions.append((x, y))
    return positions, cols

Baked Waypoint Animation

See bake_waypoints() in scripts/spatial.py (28 lines).

Zone Boundary Check

def in_zone(x, y, zone):
    """Check if (x,y) is within a zone dict with 'x':[min,max], 'y':[min,max]."""
    return zone['x'][0] <= x <= zone['x'][1] and zone['y'][0] <= y <= zone['y'][1]

Shell vs Interior Coordinate Alignment (CRITICAL)

The KION mega warehouse shell at mpu=0.01 is ~170×100m in stage meters. When placing interior cubes/assets at specific zone coordinates (e.g., racks at X=30-60), the camera must be positioned WITHIN the layout zone, not at the building origin.

Hero camera rule: Place camera at (zone_start_x + 3, zone_center_y, 2.5) looking INTO the zone — never at (0, y, z) which will be inside the shell wall geometry.

Top-down camera rule: Place at (layout_center_x, layout_center_y, max(W,D)*0.9) with focal 12mm. This is the money shot for layout validation — always render this first.

Camera-to-layout match: Before rendering, verify camera pos is inside the bounding box of your placed content, not just inside the shell.

Render File Size Validation

If all renders within a stage have identical byte counts, the camera path switch didn't take effect. Fix: create a fresh camera prim for each shot (remove + redefine), not just change the path.

KION Asset Scale Map (CRITICAL — 2026-03-14)

The KION asset tree has MIXED units. ALWAYS validate with Usd.Stage.Open() + UsdGeom.BBoxCache:

Rule: ONLY Shell + Lighting need 0.01. ALL other KION assembly assets are in meters.

Placing meter-scale assets at mpu=0.01 makes them 100× too small (invisible). This bug silently breaks scenes — assets "exist" in USD but render as sub-centimeter specs.

Validation command:

stage = Usd.Stage.Open(asset_path)
bbox = UsdGeom.BBoxCache(0, [UsdGeom.Tokens.default_]).ComputeWorldBound(stage.GetPseudoRoot()).ComputeAlignedRange()
size = bbox.GetMax() - bbox.GetMin()
# If any dimension > 1000 → centimeters (use 0.01)
# If all dimensions < 100 → meters (use 1.0)

Common Gotchas

0a. Shell interior bounds ≠ shell bbox — sm_warehouse_mega bbox is (-13.2,-5.2)→(172.6,136.8) but Corner module starts at X=0.6 and Center (floor) at X=11.6. Actual usable interior: X=3-170, Y=0-134. Always check module children bboxes, not just root. 0b. UsdGeom.Cube extent is [-1,1]³ (size 2) — scale by w/2, d/2, h/2 NOT w, d, h. Using full dimensions makes every block 2× its intended size. This caused 14,570 phantom clashes in uber warehouse v2. 0b. SetTranslateOnly() WIPES scale from Gf.Matrix4d — if you set mat[0][0]=0.01 then call mat.SetTranslateOnly(...), the scale reverts to identity. Build the matrix with explicit 16 floats: Gf.Matrix4d(sx,0,0,0, 0,sy,0,0, 0,0,sz,0, tx,ty,tz,1). This caused a 100× shell scaling bug. 0c. World delta != child local delta — to move a child by a world vector while keeping its authored local rotation/scale, map the vector through the parent's inverse rotation/scale, then write only the translation row. Adding a world delta straight to the local translation applies the parent's rotation/scale to it (wrong motion whenever the parent isn't identity/translation-only).

1. Sdf.Layer.FindOrOpen fails on binary .usd crate files — use Usd.Stage.Open
2. Matrix order: T * R * S not S * R * T
3. Cameras default look along -Z in camera space = straight down in Z-up stage (no rotation needed for top-down)
4. xformOp:translate already exists in xformOpOrder → use MakeMatrixXform() instead of AddTranslateOp()
5. DHGen humans already have xformOps defined — must use MakeMatrixXform() which clears and replaces
6. Shell is ~170×100m — layout area is usually a subset. Camera must be inside the layout area, not at origin.
7. Isaac Sim vp.camera_path sometimes silently fails between rapid renders — always create fresh camera prim with stage.RemovePrim() + UsdGeom.Camera.Define() for each shot
8. RT2 needs 200+ settle frames for convergence. Less = noisy/incomplete renders.

Lessons Learned — GSRC Integration (2026-03-12)

Scaling Awareness

• When placing large equipment (50m+ footprint), the facility MUST grow proportionally
• A 58×83m module in a 120×80m warehouse = impossible overlap
• Always calculate: equipment_area / facility_area — if > 30%, facility needs to grow

Collision Prevention

• Before placing a new zone, check all existing zone bounding boxes
• Zone overlaps are the #1 failure mode when scaling up quickly (V7/V8 had overlaps)
• Use coordinate-based collision detection: if new_x_range overlaps existing_x_range AND new_y_range overlaps existing_y_range → collision

Variation Workflow

• Build ONE variation at a time — render — show — get feedback — iterate
• Don't batch multiple variations without showing intermediate results
• Each variation should have a distinct spatial philosophy, not just moved zones

Occupancy Map as Validation

• Generate occupancy map after each variation as a spatial sanity check
• If free_space% < 40%, the layout is too dense for robot navigation
• If free_space% > 80%, the layout has too much wasted space
• Sweet spot for warehouses: 55-70% free space

Advanced Topics

See advanced.md for details.