Validation Difference GIFs

Purpose

Animate pixel differences between validation captures and golden images to quickly localize benchmark visual regressions.

Limitations

• Sensitive to tolerance settings and driver/GPU differences in captures.
• Requires existing golden image directories from a benchmark run.

Troubleshooting

Generate per-camera GIF animations showing the pixel-wise difference between captured benchmark images and their golden references. Useful for debugging validation tolerance failures.

Available Scripts

Running scripts

From agent runtimes that expose skill execution helpers, invoke helpers with run_script():

run_script("scripts/generate_diff_gifs.sh", args=[])

From a built Isaac Sim tree, run the same file with ./python.sh (Linux) or python.bat (Windows) from _build/*/release, or execute shell helpers directly when they do not require the simulator.

Prerequisites

• ImageMagick (composite, convert) must be available on PATH.
• A completed validation capture run with matching directory structure to the golden data.

Paths

Captures and golden data live under the Isaac Sim build release directory:

standalone_examples/benchmarks/validation/captures/<run_name>/
standalone_examples/benchmarks/validation/golden_data/<benchmark_name>/

Both share a parallel directory tree (e.g. Robots/Robot_0/.../front_hawk/left/camera_left/rgb/).

Usage

Find the latest capture

ls -td standalone_examples/benchmarks/validation/captures/benchmark_robots_nova_carter_ros2_*/ | head -1

Run the script

The bundled script is resolved via ${CLAUDE_SKILL_DIR}; falls back to the canonical repo path skills/validation-diff-gifs/scripts/:

bash "${CLAUDE_SKILL_DIR:-skills/validation-diff-gifs}/scripts/generate_diff_gifs.sh" \
    <captured_run_dir> \
    <golden_benchmark_dir> \
    [amplify] [fps]

Example

LATEST=$(ls -td standalone_examples/benchmarks/validation/captures/benchmark_robots_nova_carter_ros2_*/ | head -1)
GOLDEN="standalone_examples/benchmarks/validation/golden_data/benchmark_robots_nova_carter_ros2"

bash "${CLAUDE_SKILL_DIR:-skills/validation-diff-gifs}/scripts/generate_diff_gifs.sh" "$LATEST" "$GOLDEN"

Output

For each rgb/ directory under the capture, a diff_animation.gif is written alongside the captured PNGs:

<captured_run_dir>/Robots/.../front_hawk/left/camera_left/rgb/diff_animation.gif

• Black pixels = identical between captured and golden
• Brighter pixels = larger difference (amplified by the amplify factor)

SDG Golden-Set Tolerance Diffs

When production_capture_qa.py (from the data-collection-sim skill) produces a failing qa_report.json, use the SDG script to generate diffs only for the frames that tripped tolerance thresholds:

bash "${CLAUDE_SKILL_DIR:-skills/validation-diff-gifs}/scripts/sdg_tolerance_diff_gifs.sh" \
    <sdg_output_dir> \
    <sdg_golden_dir> \
    [qa_report.json] [amplify] [fps]

SDG Example (CI integration)

# After a production_capture_qa.py run exits non-zero:
SDG_OUT="/data/sdg_production_run"
SDG_GOLDEN="/data/sdg_golden/warehouse_baseline"
QA_REPORT="$SDG_OUT/qa_report.json"

bash "${CLAUDE_SKILL_DIR:-skills/validation-diff-gifs}/scripts/sdg_tolerance_diff_gifs.sh" \
    "$SDG_OUT" "$SDG_GOLDEN" "$QA_REPORT"

# Outputs:
#   /data/sdg_production_run/rgb/diff_animation.gif
#   /data/sdg_production_run/rgb/diff_failed_frames.txt
#   /data/sdg_production_run/distance_to_image_plane/diff_animation.gif

SDG Output Structure

<sdg_output_dir>/
├── rgb/
│   ├── rgb_0000.png ... rgb_0199.png
│   ├── diff_animation.gif          ← animated diff (failed frames only when filtered)
│   └── diff_failed_frames.txt      ← frame indices that tripped thresholds
├── distance_to_image_plane/
│   ├── distance_to_image_plane_0000.npy ...
│   └── diff_animation.gif          ← depth diff (grayscale)
└── qa_report.json                   ← from production_capture_qa.py

Prerequisites (SDG script)

• ImageMagick (composite, convert)
• jq (only when passing a qa_report.json)
• Python 3 with numpy (for depth .npy diffing); Pillow optional but recommended

Interpreting results

• Uniform low-level noise across the frame → rendering non-determinism (likely acceptable)
• Bright regions concentrated on object edges → sub-pixel movement differences
• Entire frame bright → wrong timestamp match or completely different camera pose
• One camera consistently worse than others → possible per-camera issue (tick rate, initialization)
• SDG: clustered failures at specific frame indices → seed-dependent pose or lighting issue