WBench Evaluation Pipeline

Score one model across 22 metrics / 5 dimensions. Reads work_dirs/<model>/videos/case_<id>_combined.mp4, writes per-metric per-case JSON to work_dirs/<model>/evaluation/<metric>/case_<id>.json and an aggregated work_dirs/<model>/evaluation/report.json.

Entry point: main.py (repo root). cd into the checkout first.

Three phases (run in order; --phase all does all four)

precompute must finish before gpu/vlm — the GPU metrics depend on the masks, depth and poses it produces.

Workflow

1. Confirm the videos are in place

ls work_dirs/<model>/videos/*.mp4 | wc -l

The pipeline auto-detects navi vs non-navi per case (is_navi_case) and only runs navi-only metrics (navigation_trajectory, spatial_consistency, scene/subject adherence) on applicable cases — a navi-only model (158 videos) is fine, missing non-navi cases are simply skipped, not errored.

2. Pick free GPUs

nvidia-smi --query-gpu=index,memory.used,utilization.gpu --format=csv,noheader

Pass them with --gpus 0,1,2,3 (default: all visible GPUs).

3. Precompute (background, slowest phase)

mkdir -p logs
nohup python main.py --model <model> --phase precompute --gpus 0,1,2,3 \
  > logs/eval_<model>_precompute.log 2>&1 &

4. GPU metrics

nohup python main.py --model <model> --phase gpu --gpus 0,1,2,3 \
  > logs/eval_<model>_gpu.log 2>&1 &

Scope with --metrics: dimension names (quality, consistency, interaction, setting, physical) or individual metrics (--metrics aesthetic_quality,segment_continuity). Re-running skips cases whose case_<id>.json already has a score (--phase gpu is incremental). visual_plausibility runs separately via tools/run_visual_plausibility.py (needs the wbench-vp env).

5. VLM metrics

nohup python main.py --model <model> --phase vlm --vlm_workers 8 \
  > logs/eval_<model>_vlm.log 2>&1 &

6. Report

python main.py --model <model> --phase report

Prints a Full/Navi table and writes work_dirs/<model>/evaluation/report.json.

Single-video debug

python main.py --video work_dirs/<model>/videos/case_1_combined.mp4 \
  --case data/cases/case_1.json

Gotchas

• Conda env: most metrics use the main env; visual_plausibility needs wbench-vp (vLLM). Use absolute python paths — conda activate doesn't persist in subshells. See project-conda-envs memory / repo CLAUDE.md.
• Turn splitting is uniform: per-turn VLM metrics split the clip by total_frames // n_turns; the pipeline does not read turns.json. For models with equal-length turns (e.g. kling) this is exact; for non-uniform turn lengths it misaligns per-turn metrics. Whole-clip metrics are unaffected.
• CPU thrash: GPU workers already cap threads (OMP/MKL/torch = 4). Don't launch multiple models in parallel — models are meant to run serially.
• Failed cases write {"score": null, "error": ...} and are counted in report.json's n_errors; they don't abort the run.

WBench Video Generation

Run a registered model over the dataset cases and produce one combined multi-turn clip per case at work_dirs/<model>/videos/case_<id>_combined.mp4.

Entry point: generate.py (repo root). Paths are relative to the repo root, so cd into the checkout first.

Registered models

wan, kling, seedance (text-conditioned), camera_preview, action_preview (reference camera/action demos). List them anytime:

python -c "from src.models import list_models; print(list_models())"

To add your own, register_model(name, cls) in src/models/__init__.py; subclass ConditionedVideoModel (camera/action) or BaseVideoModel (text). See src/models/{camera,action}/example_model.py.

Which cases to cover (by model type)

A case is "navigation" if it has ≥1 W/A/S/D/arrow action. Camera/action models must be restricted to the 158 navi cases — passing all 289 wastes compute and produces videos for cases the model can't be scored on.

Workflow

1. Pick GPU (if the model needs one) and set API creds (text models)

Text models call a video API:

export VIDEO_API_URL="https://your-video-api.com"
export VIDEO_API_KEY="your-key"

2. Smoke test on one case first

python generate.py --model <model> --cases data/cases/case_1.json

Confirm work_dirs/<model>/videos/case_1_combined.mp4 exists and plays before launching the full run.

3. Full run (background for anything > a few minutes)

mkdir -p logs
nohup python generate.py --model <model> --resume \
  > logs/generate_<model>.log 2>&1 &

• --resume skips cases that already have a video — safe to re-run after an interruption.
• --limit N caps the number of cases (quick sanity passes).
• --cases f1.json f2.json ... restricts to specific cases.
• For camera/action models, pass only the navi cases via --cases (glob the navi id list from data/cases/).

4. Verify output

ls work_dirs/<model>/videos/ | wc -l          # expected count (289 or 158)
python -c "import cv2,glob; \
  [print(p, int(cv2.VideoCapture(p).get(7))) for p in glob.glob('work_dirs/<model>/videos/*.mp4')[:3]]"

Gotchas

• The combined clip must contain all turns concatenated in order — the multi-turn logic (build prompt → infer → take last frame → next turn) is handled by generate_multi_turn; don't emit per-turn files.
• Video filename uses the JSON id field, not the source filename (e.g. case_210_scratch.json whose JSON id is 211 → case_211_combined.mp4).
• Frames-per-turn differ per model and matter for submission turns.json — see the wbench-submit skill.

WBench Submission Packaging

Assemble a submission bundle and (optionally) push it to a HuggingFace dataset. Full spec: docs/SUBMISSION.md.

Bundle layout

<model_name>/
├── meta.json                    # required
├── report.json                  # required for path A (self-eval), omit for path B
├── turns.json                   # optional but recommended (non-uniform turns)
└── videos/
    └── case_<id>_combined.mp4

<id> is the case's JSON id field (e.g. 1, e_5, ps_3), never the filename.

Two submission paths

• A — self-evaluation (default): include report.json (run the wbench-evaluate skill first). We re-run a random subset to confirm reproducibility.
• B — we evaluate (fallback): omit report.json, submit videos only; we run the 22 metrics. Batched.

Required case coverage

1. meta.json

{
  "model_name": "mymodel",
  "type": "text",                 // text | camera | action
  "display_name": "My Model 1.0",
  "org": "My Lab",
  "sampled_by": "...",            // who generated the videos
  "contact": "you@example.com",
  "num_videos": 289,
  "split": "full"                 // full | navi
}

2. turns.json (recommended)

Per-turn metrics need per-turn frame ranges. Models allot a non-uniform number of frames per turn, so provide the boundaries explicitly:

{
  "fps": 24,
  "cases": {
    "1":   { "turn_frames": [0, 57, 97, 137, 177] },
    "e_5": { "turn_frames": [0, 60, 120, 180] }
  }
}

• turn_frames = start frame of each turn plus a final sentinel; turn i spans [turn_frames[i], turn_frames[i+1]).
• N turns → N+1 boundaries, strictly increasing, start at 0, end ≤ actual frame count.
• Omitting it → uniform split (total_frames / n_turns). Fine for equal-length turns (kling), but understates per-turn metrics for non-uniform turns (most autoregressive world models).

Generate programmatically from your model's frames-per-turn rule + the per-turn chunk_length exposed by case_to_poses / case_to_actions in src/models/{camera,action}. Validate before shipping:

import cv2, json
turns = json.load(open("work_dirs/<model>/turns.json"))
for cid, t in turns["cases"].items():
    tf = t["turn_frames"]
    assert tf[0] == 0 and tf == sorted(tf) and len(set(tf)) == len(tf), cid
    nf = int(cv2.VideoCapture(f"work_dirs/<model>/videos/case_{cid}_combined.mp4").get(7))
    assert tf[-1] <= nf, f"{cid}: sentinel {tf[-1]} > {nf} frames"
print("turns.json OK")

3. Upload to HuggingFace (private dataset)

The reference bundles live under meituan-longcat/WBench-examples (folders hyworld1.5/, kling3/). If your network needs a proxy to reach huggingface.co, export https_proxy / http_proxy first.

# export https_proxy=http://<your-proxy>:<port> http_proxy=http://<your-proxy>:<port>
export HF_TOKEN=<write-scoped token>     # from https://huggingface.co/settings/tokens

REPO=meituan-longcat/WBench-examples

# create once (skip if it exists)
hf repo create "$REPO" --repo-type dataset --private

# upload a bundle (local dir → repo subfolder). Resumable: re-run to retry.
hf upload "$REPO" work_dirs/<model> <model> --repo-type dataset

Large bundles (multi-GB) — run in the background and watch the log:

mkdir -p logs
nohup bash -c 'export HF_TOKEN=<token>; \
  hf upload meituan-longcat/WBench-examples work_dirs/<model> <model> --repo-type dataset' \
  > logs/hf_upload_<model>.log 2>&1 &

Gotchas

• meta.json model_name must match the bundle/repo folder name (e.g. kling3, hyworld1.5).
• mp4 files are auto-tracked as LFS by hf upload — no manual git lfs track.
• hf upload is resumable: an interrupted upload re-runs and skips files already pushed.
• A HuggingFace dataset link with this exact structure can be submitted instead of raw files (videos are large).
• docs/SUBMISSION.md mentions a validate_submission.py — it is not yet in the repo; validate turns.json with the snippet above for now.