Add Model Prep

Goal

Prepare external assets and the shared parity-test environment for a FastVideo model port. Stop before writing conversion scripts, model components, pipeline code, registry entries, or executable parity tests.

Ask First

Ask once, then proceed if the HF token is already exported:

Before prep: (1) official reference repo or Diffusers pipeline URL, (2) HF repo
id or local weights path and whether it has a root model_index.json, (3) target
model_family, (4) workload types, (5) which token env var is exported:
HF_TOKEN, HUGGINGFACE_HUB_TOKEN, or HF_API_KEY, (6) may I stage clone and
weights under the FastVideo repo root, and (7) may I install official reference
dependencies into the current FastVideo conda/env for parity tests?

Useful optional inputs: pipeline_class, reference_dir, hf_revision, official_revision, reuse_hints, download_scope.

Rules

• Follow ../add-model/shared/common_rules.md for token/auth safety, state files, escape hatches, and skip/pass semantics.
• Run from the FastVideo repo root.
• Use repo-relative defaults: <ReferenceDir>/, official_weights/<model_family>/, converted_weights/<model_family>/.
• Install official reference deps into the current FastVideo environment, not a new venv/conda env, so parity tests run both implementations with one shared numeric stack.
• If the reference is a Diffusers class/package instead of a cloneable repo, record import path and version instead of cloning.
• Prep may create only the local-test README and PORT_STATUS.md skeletons; executable .py parity tests belong to ../add-model-02-parity/SKILL.md.

Escape Hatches

Follow ../add-model/shared/common_rules.md. Prep-specific ask cases include overwriting an existing clone or weight directory, installing untrusted/private deps, choosing between incompatible official references, large downloads outside the agreed scope, or missing gated-repo auth setup by env var name.

Workflow

1. Verify the repo:

git rev-parse --show-toplevel

Expected markers: fastvideo/, scripts/checkpoint_conversion/, scripts/huggingface/download_hf.py, fastvideo/registry.py.

2. Inspect HF or local weight layout:

python ".agents/skills/add-model-01-prep/scripts/inspect_hf_layout.py" \
    "Org/Model" \
    --revision "<revision>" \
    --json

For a local path, replace Org/Model with /path/to/weights. Record source_layout, needs_conversion, model_index_class, and components_seen.

3. Download HF weights if needed:

python ".agents/skills/add-model-01-prep/scripts/download_hf_weights.py" \
    "Org/Model" \
    "official_weights/<model_family>" \
    --revision "<revision>"

For selected files, repeat --file-name. For partial snapshots, repeat --allow-pattern or --ignore-pattern. If the user provided a local path, record it instead of copying large weights by default.

4. Clone the official reference repo if applicable:

python ".agents/skills/add-model-01-prep/scripts/clone_reference_repo.py" \
    "<official_repo_url>" \
    "<ReferenceDir>" \
    --branch "<tag-or-branch>" \
    --commit "<commit-sha>" \
    --update-gitignore

Omit --branch, --commit, or --update-gitignore when not needed. The helper refuses to overwrite existing paths and prints remote/HEAD instead.

5. Keep prep assets ignored. Ensure .gitignore includes relevant entries:

/<ReferenceDir>/
/official_weights/
/converted_weights/

6. Follow the official repo's setup instructions in the current environment. Inspect dependency files and README install docs before installing anything:

• README*, install docs, or model-card instructions.
• requirements*.txt, pyproject.toml, setup.py, environment.yml.

Use the current FastVideo conda/env. Do not create a new env even if upstream docs recommend one; translate the needed install commands into the active env. Prefer editable/no-deps first so the official source is importable without changing shared pins:

uv pip install --no-deps -e ./<ReferenceDir>

Then install only missing official deps needed for parity imports. Stop before installing requirements that would change FastVideo's core stack. If upstream requires private/non-PyPI deps, record that parity needs a local stub helper rather than pretending setup is complete.

7. Create the model-family local test skeleton and top-level port state file:

mkdir -p tests/local_tests/<model_family>
cp ".agents/skills/add-model-01-prep/templates/local_tests_readme.md" \
    tests/local_tests/<model_family>/README.md
cp ".agents/skills/add-model-01-prep/templates/port_status.md" \
    tests/local_tests/<model_family>/PORT_STATUS.md

Edit every placeholder in the README and PORT_STATUS.md. The README gives later review agents enough information to reproduce the shared environment and run/review parity work:

• official code URL or import path, local clone path, and commit/version;
• HF URL or local weight path, revision, access notes, and token env var name only;
• commands already run and any blocked official dependency installs;
• shared-env install commands to re-run without changing core pins;
• expected local parity test paths and pytest commands;
• private-dependency stubs or known setup gaps;
• PR/review notes explaining which parity tests are required before handoff.

Do not include raw tokens, absolute cache paths that are not repo-reproducible, or large generated outputs. If prep is blocked before imports work, still create the README with official_env_status=blocked and the exact blocker.

PORT_STATUS.md must follow ../add-model/contracts/port_state.md. Record open questions and prep issues immediately, using stable IDs such as Q001 and I001. Keep resolved questions/issues in the table with a resolution instead of deleting them.

Handoff

End with the canonical prep handoff contract from ../add-model/contracts/prep_handoff.md and update the shared state files before handoff.

Helper Scripts

• scripts/inspect_hf_layout.py: classify HF/local layout.
• scripts/download_hf_weights.py: download HF snapshot or selected files.
• scripts/clone_reference_repo.py: clone reference repo safely.

Add Model Parity

Goal

Create parity tests as early as possible in a FastVideo port. The first pass can land before conversion or component implementation as an executable scaffold; handoff is blocked until the same tests become non-skip PASS with real weights.

When To Run

Follow ../add-model/shared/common_rules.md for token/auth safety, state files, escape hatches, and skip/pass semantics.

Run immediately after /add-model Phase 1 has identified:

• official component classes and call signatures;
• FastVideo target component buckets/classes/configs;
• local reference clone or import path from add-model-01-prep;
• local raw or Diffusers weight path;
• official_env_status=imports_ok, or private deps that will be stubbed locally in tests;
• local_tests_readme documenting setup and planned review/test commands;
• expected component inputs and output tensors.

Do not wait for all FastVideo components to be implemented. Write the tests first, then let component-porting subagents make them pass.

Outputs

• One component parity test per required component, including reused components: tests/local_tests/<bucket>/test_<family>_<component>_parity.py.
• Optional helper for upstream private deps: tests/local_tests/helpers/<family>_upstream.py.
• Pipeline parity is owned later by ../add-model-09-pipeline/SKILL.md after all component parity tests pass non-skip.
• A parity status block for the /add-model parity verification phase.

Early Scaffold Rules

• A scaffold may skip while the FastVideo class, converted weights, or official import is missing.
• A scaffold must already encode the real official load path, FastVideo load path, deterministic inputs, expected output extraction, and tolerance target.
• Each parity test must declare its coverage scope in the file docstring or a module constant: production_loader, implementation_subcomponent, or both. Implementation/subcomponent parity may bypass production loaders deliberately, but final handoff still needs production-loader coverage somewhere before the pipeline depends on that component.
• Official reference imports must run in the current FastVideo environment; do not create or assume a separate upstream venv/conda env.
• A scaffold is not evidence of correctness. It becomes evidence only after a local non-skip PASS.
• Prefer env-var path overrides with repo-relative defaults.
• Keep tests local-only under tests/local_tests/; package/CI quality tests are added later.
• Update shared state files as described in ../add-model/shared/common_rules.md whenever adding or activating parity tests.

Component Template

Copy templates/component_parity_test.py and fill every TODO marker. The template is distilled from:

• tests/local_tests/transformers/test_ltx2.py
• tests/local_tests/transformers/test_gamecraft_parity.py
• tests/local_tests/encoders/test_ltx2_gemma_parity.py
• tests/local_tests/vaes/test_oobleck_vae_parity.py
• tests/local_tests/sd35/test_sd35_component_parity.py

The template supports three states:

Subagent Dispatch Pattern

After Phase 1, dispatch one parity subagent per component before or alongside component implementation:

Create a local parity test scaffold for <family> <component>.

Use the prep handoff:
- official_ref_dir/import: <...>
- local_weights_dir: <...>
- source_layout: <...>
- needs_conversion: <yes/no>
- official_env_status: <imports_ok | private_deps_need_stubs>
- local_tests_readme: tests/local_tests/<model_family>/README.md
- port_state_file: tests/local_tests/<model_family>/PORT_STATUS.md
- official_definition_files: <paths + classes/functions>
- official_instantiation_files: <paths + factory/pipeline/config call sites + args>
- concerns_or_unknowns: <known ambiguous inputs, outputs, deps, or args>

The complete per-component packet must match
`../add-model/contracts/component_context.md`.

Read the official component call path and the planned FastVideo component API.
Add tests/local_tests/<bucket>/test_<family>_<component>_parity.py based on
add-model-02-parity/templates/component_parity_test.py.

The scaffold must load the official model with real weights when available,
load the FastVideo model through the standardized config/class/loader path when
available, create deterministic inputs, compare concrete outputs, and skip only
when a dependency is genuinely missing. Do not make an unconditional skip or a
shape-only test.

FastVideo Load Patterns

Pick the narrowest load path that matches the component:

For early scaffolds, an import of the planned FastVideo class may be inside a helper that calls pytest.skip if the class does not exist yet. Replace that skip with a real import once the component PR adds the class.

Direct class/config construction is allowed for implementation or subcomponent parity, such as connector-only encoder checks or official monolithic-checkpoint mapping tests. Label that scope explicitly and add separate production-loader coverage when converted component dirs are available.

Official Load Patterns

• Clone/reference repo path: add its source dir to sys.path before imports.
• HF/Diffusers reference: import only inside the test, not production code.
• Private deps: add a helper under tests/local_tests/helpers/ to install stubs before importing upstream modules; do not rely on an external upstream environment.
• Gated HF repos: resolve HF_TOKEN, HUGGINGFACE_HUB_TOKEN, or HF_API_KEY under the token rules in ../add-model/shared/common_rules.md.

Non-Skip Activation Checklist

Before /add-model handoff, each scaffolded test must be activated:

[ ] Official side imports and loads real weights.
[ ] FastVideo side imports and loads the converted or original weights.
[ ] Test executes at least one real forward call on both sides.
[ ] Test compares output tensors, not only shapes or state-dict keys.
[ ] Local pytest output contains PASSED, not SKIPPED or XFAIL.
[ ] Tolerance is justified for the component scope and kernel alignment.

Component Parity Details

Reference imports:

• Import from official_ref_dir or the recorded package/import path.
• If upstream has private deps, add a helper under tests/local_tests/helpers/<family>_upstream.py that installs minimal stubs before importing upstream modules.
• Common stubs: identity compile/op-registration decorators, CP world size set to 1, identity scatter/gather, and test-friendly custom-op kernels.
• Stub decorators that register torch.ops.<ns>.<op> must preserve the torch.library registration side-effect. Identity decorators alone are not enough.
• Delete stub helpers and every install_stubs() call as soon as the real deps become required installs. No-op shims are dead code.

Kernel and wrapper pitfalls:

• If parity routes flash-attn GQA through SDPA, expand KV heads manually on the SDPA side with repeat_interleave along the head axis.
• If upstream VAE decode() denormalizes internally but FastVideo/Diffusers expects pre-denormalized latents, apply z = z * std + mean only on the FastVideo side in the parity test.
• Per-channel VAE latents_mean / latents_std must be reshaped explicitly, e.g. .view(1, z_dim, 1, 1, 1) for 5D video latents.

Tolerance guide:

Element-wise assert_close alone is not enough for deep full-DiT parity. Also log global abs-mean drift and per-modality summaries.

When a non-skip component parity run is numerically red after weight/input checks, invoke ../add-model-08-trace/SKILL.md before adding bespoke forward hooks. Use docs/contributing/activation_trace.md to keep FASTVIDEO_TRACE_LAYERS, FASTVIDEO_TRACE_STATS, and FASTVIDEO_TRACE_STEPS identical across FastVideo and upstream traces.

Useful local commands:

pytest tests/local_tests/<bucket>/test_<family>_*parity*.py -v -s
pytest tests/local_tests -k "<family> and parity" -v -s

Escape Hatches

Follow ../add-model/shared/common_rules.md. Parity-specific ask cases include private dependency approval, choosing between incompatible official references, accepting a shape-only substitute, or loosening required tolerances.

Pipeline Parity

Pipeline parity is later than component parity because it needs stages, presets, registry wiring, converted weights, and green component parity. Record official pipeline call notes in local_tests_readme, but do not treat pipeline parity as owned by this skill.

Use ../add-model-09-pipeline/SKILL.md and its templates/pipeline_parity_test.py for pipeline parity scaffolding and debugging. Compare denoised latents or decoded media, not just successful generation.

Handoff Status Block

Return ../add-model/contracts/parity_status.md to /add-model and update the shared state files before handoff.

Add Model Port DiT

Goal

Prototype or parity-debug one diffusion transformer in FastVideo-native code. This skill is for one component only; do not work on the VAE, encoders, pipeline, or unrelated conversion code unless the current component cannot load without a minimal fix there.

Inputs

Follow ../add-model/shared/component_skill_common.md and require the complete packet from ../add-model/contracts/component_context.md.

DiT-specific packet fields:

• component: transformer or DiT name.
• parity_test: tests/local_tests/<bucket>/test_<family>_<component>_parity.py.
• weights: converted transformer dir or local official path.
• target_files: fastvideo/models/dits/<family>.py and fastvideo/configs/models/dits/<family>.py.

Modes

Use the common prototype and parity-debug modes from ../add-model/shared/component_skill_common.md.

DiT-specific prototype concerns include ambiguous official flags, shape mismatches, missing FastVideo layer equivalents, and dedicated output heads.

Reuse Proof

Apply the shared reuse proof. DiT-specific comparison must include attention algorithm, positional embeddings, RoPE/patching, timestep/guidance embeddings, scaling constants, dtype casts, state-dict names, and every output head.

Existing FastVideo Patterns

• Base class: fastvideo/models/dits/base.py::BaseDiT.
• Config bases: DiTConfig and DiTArchConfig in fastvideo/configs/models/dits/base.py.
• Use the matching DiT config bucket. Wrong bucket inheritance can typecheck but fail during pipeline wiring.
• Config export: add the config to fastvideo/configs/models/dits/__init__.py.
• Registry discovery: set EntryClass = <ClassName> in the model file.
• Loader path: TransformerLoader reads transformer/config.json, calls dit_config.update_model_arch(config), resolves _class_name through ModelRegistry, and constructs the class with config and hf_config.
• Reference examples: stable_audio.py, wanvideo.py, sd3.py, longcat.py, and ltx2.py.
• Layer guidance: fastvideo/layers/AGENTS.md.

Implementation Rules

• Use FastVideo-native layers by default: ReplicatedLinear for DiT hot-path linears, DistributedAttention for standard full-sequence attention, and LocalAttention for local/window attention or simple single-GPU parity paths.
• Raw SDPA is acceptable for cross-modality flat streams when no FastVideo distributed equivalent exists; document the SP gap in the module docstring.
• Mirror official tensor contracts exactly: latent packing, patch ordering, timestep embedding scale, RoPE/positional embedding, guidance embedding, cross-attention context order, output head order, and dtype casts.
• Preserve all output heads that the official DiT emits. Do not silently drop audio, depth, pose, mask, or auxiliary heads.
• Put architecture fields on DiTArchConfig; keep inference steps, CFG scales, FPS, flow shift, and sampling defaults out of the arch config.
• Define _fsdp_shard_conditions, _compile_conditions, param_names_mapping, and reverse_param_names_mapping where needed.
• Follow the production import boundary in ../add-model/shared/common_rules.md.

Prototype Checks

Follow the shared prototype success criteria. A useful one-off check is:

python - <<'PY'
# Import the target config/class, instantiate with random weights, and print
# state_dict names/shapes for the conversion mapping.
PY

Parity-Debug Loop

Run the shared parity-debug loop. The component test command is:

pytest <parity_test> -v -s

For numerical drift, use ../add-model-08-trace/SKILL.md before writing bespoke hooks. Start with FastVideo's activation trace (fastvideo/hooks/activation_trace.py; docs/contributing/activation_trace.md) and a block-level regex such as FASTVIDEO_TRACE_LAYERS="^block\.layers\.[0-9]+$". Only fall back to custom per-block hooks if the needed boundary or statistic is not exposed by FASTVIDEO_TRACE_STATS.

Escape Hatches

Follow ../add-model/shared/common_rules.md and the component-specific guidance in ../add-model/shared/component_skill_common.md. DiT-specific ask cases include dropping an output head/modality, accepting an unsupported kernel/private op, or choosing between incompatible official transformer definitions.

Handoff

Return ../add-model/contracts/component_skill_handoff.md following the common handoff rules in ../add-model/shared/component_skill_common.md.

Add Model Port VAE

Goal

Prototype or parity-debug one VAE or autoencoder in FastVideo-native code. This skill covers video, image, and audio VAEs.

Inputs

Follow ../add-model/shared/component_skill_common.md and require the complete packet from ../add-model/contracts/component_context.md.

VAE-specific packet fields:

• component: VAE or autoencoder name.
• parity_test: tests/local_tests/vaes/test_<family>_<component>_parity.py.
• weights: converted VAE dir, HF subfolder, or local official path.
• target_files: fastvideo/models/vaes/<arch_or_family>.py and fastvideo/configs/models/vaes/<arch_or_family>.py.

Modes

Use the common prototype and parity-debug modes from ../add-model/shared/component_skill_common.md.

VAE-specific prototype concerns include latent normalization, stochastic posterior behavior, tiling incompatibility, temporal/spatial/audio layout, and decode output containers.

Reuse Proof

Apply the shared reuse proof. VAE-specific comparison must include latent layout, temporal/spatial/audio compression, scaling factor, mean/std normalization, posterior behavior, encode/decode output objects, tiling flags, and cropping.

Existing FastVideo Patterns

• Shared tiling wrapper: fastvideo/models/vaes/common.py::ParallelTiledVAE.
• Config bases: VAEConfig and VAEArchConfig in fastvideo/configs/models/vaes/base.py.
• Use the matching VAE config bucket. Wrong bucket inheritance can typecheck but fail during pipeline wiring.
• Config export: add the config to fastvideo/configs/models/vaes/__init__.py.
• Registry discovery: set EntryClass = <ClassName> in the model file.
• Loader path: VAE loaders resolve _class_name through ModelRegistry and load converted component weights from the VAE subdir.
• Reference examples: oobleck.py, autoencoder_kl.py, wanvae.py, ltx2vae.py, and gamecraftvae.py.
• Layer guidance: fastvideo/layers/AGENTS.md.

Implementation Rules

• Name reusable VAE architectures by architecture (oobleck.py, autoencoder_kl.py); name family-specific VAEs by family.
• Match official encode/decode contracts exactly: input layout, latent layout, temporal/spatial/audio compression, scaling factor, mean/std normalization, posterior sampling behavior, decode output object, and frame/sample cropping.
• Compare deterministic outputs in parity: decode outputs, encode mean/mode, or round-trip tensors. Do not compare stochastic samples unless the RNG path is explicitly controlled.
• Use FastVideo tiling only when it preserves official numerics for the tested shape; disable it in config for audio or unsupported dimensions.
• Put architecture constants on VAEArchConfig; put load_encoder, load_decoder, tiling, dtype, and pretrained path fields on VAEConfig.
• Follow the production import boundary in ../add-model/shared/common_rules.md.

Prototype Checks

Follow the shared prototype success criteria.

Parity-Debug Loop

Run the shared parity-debug loop. The component test command is:

pytest <parity_test> -v -s

For numerical drift, check normalization, latent scaling, posterior mode vs sample, channel order, and temporal/spatial/audio cropping before changing layers.

Escape Hatches

Follow ../add-model/shared/common_rules.md and the component-specific guidance in ../add-model/shared/component_skill_common.md. VAE-specific ask cases include dropping an encode/decode path, accepting an unsupported private op, or choosing between incompatible official VAE definitions.

Handoff

Return ../add-model/contracts/component_skill_handoff.md following the common handoff rules in ../add-model/shared/component_skill_common.md.

Add Model Port Encoder

Goal

Prototype or parity-debug one encoder or encoder-like conditioner in FastVideo-native code. Use this for text encoders, image encoders, audio encoders, and compound conditioners that fit the encoder config/loader bucket.

Inputs

Follow ../add-model/shared/component_skill_common.md and require the complete packet from ../add-model/contracts/component_context.md.

Encoder-specific packet fields:

• component: encoder or encoder-like conditioner name.
• parity_test: tests/local_tests/encoders/test_<family>_<component>_parity.py.
• weights: converted encoder dir, HF subfolder, or external HF id.
• target_files: fastvideo/models/encoders/<arch_or_family>.py and fastvideo/configs/models/encoders/<arch_or_family>.py.

Modes

Use the common prototype and parity-debug modes from ../add-model/shared/component_skill_common.md.

Encoder-specific prototype concerns include tokenizer kwargs, hidden-state extraction, output packing, connector order, and external/passthrough weight needs.

Reuse Proof

Apply the shared reuse proof. Encoder-specific comparison must include tokenizer contracts, hidden-state extraction, masks, positional IDs, output packing, connector/projection ordering, passthrough paths, and returned dataclass shape.

Existing FastVideo Patterns

• Base classes: TextEncoder and ImageEncoder in fastvideo/models/encoders/base.py.
• Output type: BaseEncoderOutput.
• Config bases: TextEncoderConfig, ImageEncoderConfig, TextEncoderArchConfig, and ImageEncoderArchConfig in fastvideo/configs/models/encoders/base.py.
• Use the matching encoder config bucket. Wrong bucket inheritance can typecheck but fail during pipeline wiring.
• Config export: add the config to fastvideo/configs/models/encoders/__init__.py.
• Registry discovery: set EntryClass = <ClassName> or a list of class names in the model file.
• Reference examples: native t5.py, clip.py, siglip.py, llama.py, qwen2_5.py, gemma.py, and compound stable_audio_conditioner.py.
• Layer guidance: fastvideo/layers/AGENTS.md.

Implementation Rules

• Reuse tokenizers and pure data utilities when needed, but do not add runtime third-party model-class imports as a placeholder for a component that owns weights or numerical behavior.
• For LLM-style encoders, follow existing tensor-parallel patterns such as QKVParallelLinear, MergedColumnParallelLinear, RowParallelLinear, VocabParallelEmbedding, and RMSNorm when matching native examples.
• Match official hidden-state extraction exactly: layer index, pooled output, attention mask dtype, padding side, truncation, special tokens, final norm, output_hidden_states, and returned tuple/dataclass shape.
• For connector or conditioner modules, preserve sub-conditioner order and the exact packing of cross-attention tokens, masks, and global conditioning.
• Put tokenizer kwargs and architecture constants on the arch config when they affect numerical behavior.
• If an external HF encoder is explicitly accepted as a lazy wrapper, keep it isolated, document why it is not a native port, and still require parity for the wrapper's output contract.

Hybrid external-HF encoder checklist:

• Put external model folders in passthrough subfolders such as text_encoder/<external_name>/, or record a root model_index.json path field that the loader resolves to a local directory.
• Keep external model parameters out of the FastVideo-owned state-dict surface when the external model is loaded lazily from its own HF files.
• Convert and strict-check only the FastVideo-owned connector/projection weights; document external model weights as passthrough.
• Add parity for the wrapper's final output contract and, when useful, a narrower connector-only parity test that labels its scope as implementation_subcomponent.
• Verify the production loader resolves the same external path used by the pipeline, not just the direct class used in the parity test.

Prototype Checks

Follow the shared prototype success criteria.

Parity-Debug Loop

Run the shared parity-debug loop. The component test command is:

pytest <parity_test> -v -s

For numerical drift, check tokenization, masks, hidden-state selection, positional IDs, dtype/autocast, and output packing before changing layers.

Escape Hatches

Follow ../add-model/shared/common_rules.md and the component-specific guidance in ../add-model/shared/component_skill_common.md. Encoder-specific ask cases include accepting private model-code execution, choosing between incompatible tokenizer/encoder references, or dropping a required conditioning stream.

Handoff

Return ../add-model/contracts/component_skill_handoff.md following the common handoff rules in ../add-model/shared/component_skill_common.md.

Add Model Port Generic

Goal

Prototype or parity-debug one scheduler, conditioner, upsampler, vocoder, adapter, preprocessor, or unknown component in FastVideo-native code.

Inputs

Follow ../add-model/shared/component_skill_common.md and require the complete packet from ../add-model/contracts/component_context.md.

Generic-component packet fields:

• component: component name.
• component_type: scheduler, conditioner, upsampler, vocoder, adapter, preprocessor, or unknown.
• parity_test: tests/local_tests/<bucket>/test_<family>_<component>_parity.py.
• weights: converted component dir, HF subfolder, or none.
• target_files: matching fastvideo/models/ and fastvideo/configs/models/ bucket files when applicable.

Modes

Use the common prototype and parity-debug modes from ../add-model/shared/component_skill_common.md.

Generic-component prototype concerns include stateless/stateful ambiguity, missing loader buckets, source prefixes, mutable scheduler state, and output container shape.

Reuse Proof

Apply the shared reuse proof. Generic-component comparison must include mutable state, scaling constants, scheduler/conditioner semantics, output containers, and whether the component owns state or is stateless.

Existing FastVideo Patterns

• Schedulers live under fastvideo/models/schedulers/ and expose EntryClass.
• Upsamplers use fastvideo/models/upsamplers/ plus configs under fastvideo/configs/models/upsamplers/; see hunyuan15.py.
• Vocoders and audio-specific modules can live under fastvideo/models/audio/ with configs under fastvideo/configs/models/audio/; see ltx2_audio_vae.py.
• Compound conditioners may fit the encoder bucket when the pipeline loader uses ConditionerLoader; see stable_audio_conditioner.py.
• Registry discovery uses EntryClass; config bucket exports are required when pipeline configs import them by bucket.
• Use the narrowest matching config bucket. Wrong bucket inheritance can typecheck but fail during pipeline wiring.
• Layer guidance: fastvideo/layers/AGENTS.md.

Bucket Decision

• If the component is a transformer/DiT, stop and use add-model-03-port-dit.
• If the component is a VAE/autoencoder, stop and use add-model-04-port-vae.
• If the component is a text/image/audio encoder or encoder-like conditioner, stop and use add-model-05-port-encoder unless the loader requires a different bucket.
• Otherwise choose the narrowest existing bucket. Add a new bucket only when no existing loader/config shape can represent the component without misleading names or unsafe runtime behavior.

Implementation Rules

• Match official behavior, not just shapes: constructor args, default values, runtime flags, RNG use, dtype/autocast, scaling constants, masks, and output containers all matter.
• Keep the implementation minimal and native. Do not keep a runtime import of the official implementation as the production component.
• For schedulers, compare timesteps, sigmas/noise levels, step outputs, shift handling, prediction type, and any mutable internal state.
• For upsamplers, compare resize mode, align_corners, residual branches, causal padding, normalization, and exact target-shape behavior.
• For vocoders/audio components, compare waveform shape, sample-rate contract, channel order, hop length, normalization, and dtype.
• If private upstream deps are required only for tests, keep stubs under tests/local_tests/helpers/ and do not import them from production code.

Prototype Checks

Follow the shared prototype success criteria.

Parity-Debug Loop

Run the shared parity-debug loop. The component test command is:

pytest <parity_test> -v -s

For numerical drift, add targeted intermediate comparisons in the test to identify the first divergent operation.

Escape Hatches

Follow ../add-model/shared/common_rules.md and the component-specific guidance in ../add-model/shared/component_skill_common.md. Generic-component ask cases include creating a new loader bucket, accepting an unsupported private op, choosing between incompatible official definitions, or dropping a required component.

Handoff

Return ../add-model/contracts/component_skill_handoff.md following the common handoff rules in ../add-model/shared/component_skill_common.md.

Add Model Conversion

Goal

Convert official weights into a FastVideo-loadable component layout after Phase 4 native prototypes exist. The conversion script owns parameter mapping, component splitting, passthrough assets, config emission, and strict-load verification.

Inputs

Follow ../add-model/shared/common_rules.md for token/auth safety, state files, escape hatches, production boundaries, and skip/pass semantics.

Require the initial request from ../add-model/contracts/conversion_request.md.

If the FastVideo key/shape dump is missing, return to /add-model Phase 4. Do not write a final mapping against an unimplemented component.

For Phase 6 retry requests from component skills, also require the retry shape from ../add-model/contracts/conversion_request.md.

Output

• scripts/checkpoint_conversion/<family>_to_diffusers.py.
• converted_weights/<family>/ with model_index.json and per-component subfolders.
• Updated tests/local_tests/<model_family>/README.md with conversion command, source layout, output path, and strict-load status.
• Updated tests/local_tests/<model_family>/PORT_STATUS.md with conversion state, retry history, open questions, and issues/blockers.

Reference Scripts

• scripts/checkpoint_conversion/convert_ltx2_weights.py: component prefix splitting, metadata config extraction, passthrough Gemma/tokenizer assets, and optional component-only output.
• scripts/checkpoint_conversion/stable_audio_to_diffusers.py: monolithic model.safetensors split into transformer/VAE/conditioner, plus copied passthrough subfolders. Use this shape for single-checkpoint official repos.
• scripts/checkpoint_conversion/convert_gamecraft_full.py: separate official sources for transformer, VAE, text encoders, tokenizers, scheduler, and root model_index.json.
• scripts/checkpoint_conversion/longcat_to_fastvideo.py: fused QKV/KV split, renamed native transformer weights, and copied existing Diffusers components.
• scripts/checkpoint_conversion/pt_to_safetensors.py: simple .pt extraction helper for nested checkpoint dictionaries.

Source Layout Decision

Choose exactly one primary layout:

Monolithic checkpoints need explicit prefix ownership. For example, Stable Audio uses one model.safetensors with DiT, pretransform/VAE, and conditioner keys; the converter splits those keys into FastVideo component subfolders and writes per-component configs.

Script Shape

Start from templates/family_to_diffusers.py or the closest reference script. Keep the script explicit and reviewable:

• COMPONENT_SPECS or COMPONENT_PREFIXES declares component ownership.
• PARAM_NAME_MAP declares key renames.
• SKIP_PATTERNS declares intentionally dropped training-only keys.
• tensor split/fuse helpers are named by operation, e.g. split_qkv.
• build_component_configs(...) writes loader-compatible config files. Most model components use config.json; schedulers use scheduler_config.json.
• build_model_index(...) writes a root model_index.json matching the target FastVideo pipeline and component classes.
• verification reports missing, unexpected, skipped, unchanged, renamed, and shape-mismatched keys.

model_index.json library tokens must match FastVideo loaders:

• standard native DiT/VAE/audio/vocoder/upsampler components loaded by existing Diffusers-style loaders usually use "diffusers" with a FastVideo _class_name in the component config.json;
• text encoders, tokenizers, image encoders, processors, and feature extractors usually use "transformers";
• conditioner currently expects "fastvideo";
• use fully qualified "fastvideo.<module>" only when intentionally relying on the custom fastvideo-library escape path;
• do not write bare "fastvideo" for transformer, VAE, or other loaders that expect "diffusers" unless the loader explicitly expects it.

Mapping Rules

• Use Phase 4 key/shape dumps to derive mappings. Do not guess from official key names alone.
• Preserve each component's official file paths, parity test path, and prototype concerns in comments or structured constants near the mapping that uses them.
• Every official inference parameter should be mapped, copied through, or listed as intentionally skipped with a reason.
• Every FastVideo prototype parameter should receive a tensor or be listed as an intentional external/passthrough parameter.
• Shape matches are necessary but not sufficient; check semantic pairing for Q/K/V, gate/up/down, norm scale/bias, LoRA/base, and modality-specific heads.
• If official and FastVideo fuse or split tensors differently, convert tensors in the script rather than changing production code to match checkpoint quirks.

Verification

Run conversion locally, then verify before returning to Phase 6. For retry requests, update the mapping, rerun conversion, and refresh only the implicated converted component when safe; otherwise rerun the full conversion.

python scripts/checkpoint_conversion/<family>_to_diffusers.py \
    --src <official_weights> \
    --revision <hf_revision> \
    --dst converted_weights/<model_family>

Omit --revision for local sources or when prep recorded default / none.

Minimum output layout:

converted_weights/<family>/
  model_index.json
  transformer/config.json
  transformer/*.safetensors
  vae/config.json
  vae/*.safetensors
  scheduler/scheduler_config.json as needed
  text_encoder/... as needed

Required checks:

• model_index.json exists and lists every required component.
• Each converted component has the config filename its loader expects and safetensors weights when it owns weights. Scheduler dirs require scheduler_config.json; most other native model dirs use config.json.
• Weight filenames may vary by loader: transformer and VAE loaders glob all *.safetensors; text encoders may load *.safetensors, *.bin, and sometimes *.pt; conditioner currently expects diffusion_pytorch_model.safetensors. Use the loader's actual accepted layout rather than assuming one global filename.
• Passthrough components are copied or referenced deliberately.
• Each emitted component config validates through the same path production loaders use. Instantiate the relevant config and call update_model_arch(...) or update_model_config(...) with the emitted JSON so unknown keys fail during conversion, not at pipeline load time.
• Record production loader strictness for every stateful component. If the loader intentionally uses non-strict loading, add explicit missing/unexpected-key assertions in the parity test and document exactly which keys are allowed.
• Each new FastVideo component strict-loads converted weights where its production loader is strict. If strict loading is impossible, record the exact allowed missing/unexpected keys and why they are not inference weights.
• Retry fixes include the original component parity evidence and the new strict-load result in local_tests_readme so the component subagent can resume without rediscovering context.
• local_tests_readme records the command, output directory, and strict-load result.

Do not chase numerical parity in this skill except to identify a conversion mapping bug. Long parity-debug loops belong to /add-model Phase 6.

Escape Hatches

Follow ../add-model/shared/common_rules.md. Conversion-specific ask cases include selecting between incompatible official checkpoints, publishing/uploading weights, overwriting an existing converted repo not created by this run, accepting non-strict missing inference weights, or dropping a component/output from scope.

Handoff

Return ../add-model/contracts/conversion_handoff.md and update the shared state files before handoff.

Add-Model Trace

Manual Invocation

Load this skill when /add-model Phase 6 component parity has failed and the root cause requires layer-by-layer divergence analysis. This skill is not auto-fired. The calling subagent (DiT, VAE, encoder, or generic port skill) loads it when its standard parity-debug loop hits a wall and cannot isolate the divergence from end-to-end tensor comparisons alone.

Do not load this skill for first-pass parity failures. Try weight-diff and end-to-end tensor comparison first. Load this skill only when those do not isolate the cause.

Goal

Find the first numerical divergence point between FastVideo's port and the official reference, layer by layer, by instrumenting both sides at matching tensor boundaries. The investigation must leave zero source residue in production code when it closes.

When To Run

After a component parity test FAILS at a bf16-noise-realistic tolerance AND the calling subagent's first-pass debug (weight-diff, end-to-end tensor compare) does not isolate the cause.

Required inputs before starting:

• A working FastVideo loader for the component under investigation.
• A working official loader, typically via tests/local_tests/helpers/<family>_upstream.py::load_upstream_<component>.
• Shared deterministic test inputs (same tensors on both sides).
• The component parity test file path and its current failure output.

Primary Path: FastVideo Activation Trace

Use FastVideo's first-class activation trace before writing custom hooks: fastvideo/hooks/activation_trace.py, documented in docs/contributing/activation_trace.md.

Pipeline runs attach trace to the transformer during pipeline initialization. Component-only parity harnesses may call attach_activation_trace(model) from local test/debug code; do not add trace calls to production model code.

Prefix the failing parity command with a tight layer regex:

FASTVIDEO_TRACE_ACTIVATIONS=1 \
FASTVIDEO_TRACE_LAYERS="^block\.layers\.[0-9]+$" \
FASTVIDEO_TRACE_STATS="abs_mean,sum,max,shape" \
FASTVIDEO_TRACE_STEPS="0" \
FASTVIDEO_TRACE_OUTPUT="/tmp/opencode/fv_trace.jsonl" \
pytest tests/local_tests -k "parity" -v -s

Match the layer regex to the actual model.named_modules() names. Empty or broad regexes are expensive; prefer block-level names first, then narrow to submodules after the first divergent block is known.

Trace Compare Contract

One JSONL file per side. FastVideo output should use FASTVIDEO_TRACE_OUTPUT; the upstream harness should emit the same JSONL shape:

{"module":"block.layers.0","tensor":"out","step":0,"abs_mean":0.0123,"sum":1.0,"max":0.5,"shape":[1,16,32]}

Compare rows by (module, step, tensor). The first row whose shape, abs_mean, or max diverges beyond the component tolerance is the first broken boundary. Keep FASTVIDEO_TRACE_LAYERS, FASTVIDEO_TRACE_STATS, and FASTVIDEO_TRACE_STEPS identical between sides; if row order differs, sort or normalize before diffing.

Drill-Down Loop

Initial run: trace every top-level block (^block\.layers\.[0-9]+$ or the family's equivalent). Identify the first block index where abs_mean or max drifts beyond tolerance while earlier blocks match.

Drill run: tighten FASTVIDEO_TRACE_LAYERS to submodules inside the first divergent block: attention output, MLP projections, norm outputs, modality adapters, or other named boundaries exposed by named_modules().

Iterate: if the first divergent operation is a free function or tensor op not visible as an nn.Module, use the fallback instrumentation hierarchy below.

The loop ends when the first divergent submodule or operation is identified with a file:line citation in the official source.

Fallback Instrumentation Hierarchy

Use these only when activation trace cannot observe the needed boundary or statistic.

(1) Custom forward hooks

module.register_forward_hook(...) and register_forward_pre_hook(...). Always within try/finally with handle.remove(). Zero source residue.

(2) Runtime monkey-patch

module.attr = wrapped_func or cls.method = wrapped_method, restored via try/finally (save original first). Use for free functions and non-Module sites such as activation functions (swiglu, apply_rotary_emb).

(3) Source edits in FastVideo's own code

Only when (1) and (2) are insufficient. Track all edits within a single named git stash boundary OR a temporary branch. Run git diff before closing the investigation to confirm cleanup.

(4) Source edits in official repo source

Allowed only when hook and monkey-patch approaches cannot capture the site. For git-tracked or editable official clones, use git diff in the clone path to verify cleanup. For non-editable site-packages, back up the target file before editing and restore it before handoff.

Hypothesis Toggles

Use env-var-gated monkey-patches to A/B test suspect implementations without source edits. Pattern: <FAMILY>_DEBUG_PATCH_<HYPOTHESIS>=1.

Example from the magi-human investigation:

MAGI_DEBUG_PATCH_LINEAR=1

This patched PackedExpertLinear.forward to mirror upstream's _BF16ComputeLinear explicit-cast pattern, isolating a dtype-cast difference as the root cause.

Document all toggles in the script docstring. Each toggle must:

• save the original before patching;
• restore the original in a try/finally block;
• print a [debug] Patched <ClassName>.<method> line to stdout when active.

Cleanup Gate

The calling agent MUST report [cleanup-gate] PASS on all five items before handoff. Do not hand off with any item unresolved.

1. git diff in the FastVideo repo: empty. No stray prints, hooks, or monkey-patches in production code.
2. git diff in the official-repo clone (if used): empty. For non-editable site-packages installs: diff original.py original.py.trace-backup is empty OR pip install --force-reinstall <pkg> succeeded and the installed file matches the original.
3. git stash list: only the named investigation stash (or empty). No unnamed stashes left from this session.
4. No new untracked files outside /tmp/opencode/ (logs) and the existing debug script directory (tests/local_tests/transformers/ or equivalent).
5. mypy clean on any production files touched during the investigation.

Escape Hatches

Escalate to the calling bucket skill when:

• A forward hook on an official module raises because of a custom forward signature or varlen handler args that the hook closure cannot satisfy. The bucket skill has component-specific knowledge to work around this.
• The first divergent layer is block[0], meaning the divergence is in the adapter, modality dispatcher, coordinate embedding, or packing step before any block runs. Check those sites first; the bug is not in attention or MLP.
• Per-block drift is never zero anywhere across all blocks. This usually means the inputs are not bit-identical between sides. Verify with a state-dict compare (weight-diff script) AND confirm the input tensors are the same object or have identical values before the forward call.

Handoff

Return to the calling subagent with:

• FastVideo trace JSONL path and upstream trace JSONL path.
• Trace settings used: FASTVIDEO_TRACE_LAYERS, FASTVIDEO_TRACE_STATS, and FASTVIDEO_TRACE_STEPS.
• The first divergent (module, step, tensor) row and observed drift.
• The upstream file:line citation where the divergence originates.
• Fallback hook/patch verdict if activation trace could not observe the boundary.
• Hypothesis verdict if an A/B toggle was used, for example PATCH_LINEAR=1.
• Cleanup-gate status: [cleanup-gate] PASS or a list of unresolved items.

The calling agent uses this to scope the production fix in the FastVideo component file.

References

• docs/contributing/activation_trace.md for canonical activation-trace env vars, JSONL output, cost model, and troubleshooting.
• fastvideo/hooks/activation_trace.py for the implementation and attach_activation_trace(model) entry point.
• templates/block_trace_debug.py in this skill directory: fallback custom-hook template when activation trace cannot observe the needed boundary or stat.
• tests/local_tests/transformers/_debug_magi_human_block_parity.py in the FastVideo3 repo: historical worked example for custom hook/patch debugging.
• add-model/SKILL.md Phase 6: the calling context for this skill.
• add-model-03-port-dit/SKILL.md, add-model-04-port-vae/SKILL.md, add-model-05-port-encoder/SKILL.md, add-model-06-port-generic/SKILL.md: bucket-specific debug language and component-specific escape-hatch knowledge.

Changelog

Add Model Pipeline

Goal

Implement and verify the end-to-end FastVideo pipeline after the native components and converted weights have passed non-skip component parity. This skill owns pipeline class/stage wiring, pipeline configs, presets, registry entries, examples, smoke tests, and pipeline parity-debug.

FastVideo has one pipeline architecture: stage-based composition through ComposedPipelineBase. Add or specialize stages only when existing stages cannot represent the official behavior safely.

Hard Gate

Do not start pipeline work until every required component, including reused components, has a non-skip local parity PASS.

If any component row is missing, skipped, red, or blocked, return to /add-model Phase 6. Pipeline parity cannot distinguish stage wiring mistakes from broken component numerics when component parity is still unresolved.

Inputs

Follow ../add-model/shared/common_rules.md for token/auth safety, state files, escape hatches, production boundaries, and skip/pass semantics.

Require a complete packet matching ../add-model/contracts/pipeline_context.md.

The packet must include:

• official pipeline files and official call/default sources;
• workload types, input/output modalities, and output contract;
• converted or source model_index.json path;
• component parity rows, all non_skip_pass;
• target FastVideo pipeline/config/preset/registry/example/test paths;
• local_tests_readme and port_state_file paths.

Outputs

• Pipeline package under fastvideo/pipelines/basic/<family>/.
• Pipeline config under fastvideo/configs/pipelines/<family>.py or a documented family-local config file when that matches existing project style.
• Presets under fastvideo/pipelines/basic/<family>/presets.py.
• Registry updates in fastvideo/registry.py.
• Basic example under examples/inference/basic/basic_<family>*.py.
• Local smoke and parity tests under tests/local_tests/pipelines/.
• Updated tests/local_tests/<model_family>/README.md.
• Updated tests/local_tests/<model_family>/PORT_STATUS.md.
• Handoff matching ../add-model/contracts/pipeline_handoff.md.

Mode: Pipeline Definition

Use this mode first.

1. Read the official pipeline call path before editing FastVideo code.
2. Compare official defaults against the planned FastVideo config and presets: steps, CFG scales, secondary CFG, flow shift, schedulers, sigmas, seed/RNG, resolution, frames, FPS, duration, VAE scaling, decode slicing, negative prompt defaults, and output heads.
3. Create or update the pipeline class with _required_config_modules matching the emitted model_index.json and ComposedPipelineBase.load_modules. Runtime pipeline resolution is exact: model_index.json["_class_name"] must match a registered EntryClass.__name__, or a wrapper/alias class in EntryClass. Registry detectors do not select the executable pipeline class.
4. Add new public generation kwargs to fastvideo/api/sampling_param.py before examples or presets use them. SamplingParam.update() ignores unknown keys except for logging, and preset defaults apply only to declared fields. Add CLI args when the option should be available from command-line entrypoints.
5. Put loader-time changes in load_modules() or earlier, not initialize_pipeline(). ComposedPipelineBase.__init__ loads modules before post_init() calls initialize_pipeline(), so process-global flags, loader path rewrites, dtype overrides, and tokenizer path changes needed for loading cannot be introduced there.
6. Use self.get_module("transformer_2", None) and similar optional accessors for truly optional modules. Do not hard-require optional modules by accident.
7. Avoid mutating class-level _required_config_modules in custom code. If a pipeline needs dynamic modules, copy the list to an instance-owned value or pass required_config_modules explicitly so one pipeline instance cannot leak module requirements into another.
8. Create the stage chain in official execution order. Prefer existing shared stages for standard text encoding, timestep preparation, latent preparation, denoising, and decoding.
9. Add model-specific stages only for family-specific behavior that does not fit the shared stage contracts.
10. Add pipeline config classes for wiring and runtime defaults. Do not duplicate component architecture fields unless a loader requires them in the subconfig. Family-local config files such as fastvideo/pipelines/basic/<family>/pipeline_configs.py are valid only when fastvideo/registry.py imports and registers the classes explicitly.
11. Add InferencePreset objects with model_family, name, version, defaults, optional validation-only stage_schemas, and an ALL_PRESETS tuple. stage_schemas validates user-facing stage_overrides names; it does not drive create_pipeline_stages() execution.
12. Register config classes and presets in fastvideo/registry.py: add register_configs(...), import the family's ALL_PRESETS, and append it to _register_presets(). Detectors should cover HF paths and _class_name strings for config/preset lookup, but not as a replacement for exact pipeline class-name resolution.
13. Add a basic example with a user-story docstring and normal file-path inputs for image, audio, or video references. Keep orchestration glue in the pipeline or a helper, not in the example.
14. Add a separate smoke test tests/local_tests/pipelines/test_<family>_pipeline_smoke.py that proves imports, EntryClass, registry, presets, config defaults, and at least one real load/generate path when weights are local. Older local tests sometimes colocate smoke checks in parity files; new ports should use the separate file convention.
15. Add or update pipeline parity test scaffolding with templates/pipeline_parity_test.py.
16. Update local_tests_readme and port_state_file with commands, statuses, default sources, decisions, and blockers.

Production import boundaries are defined in ../add-model/shared/common_rules.md.

Mode: Pipeline Parity Debug

Run after pipeline definition and after smoke can execute far enough to load the pipeline. Loop until pipeline parity is a non-skip PASS or a precise blocker is returned.

Mandatory order:

pytest tests/local_tests/pipelines/test_<family>_pipeline_smoke.py -v -s
DISABLE_SP=1 pytest tests/local_tests/pipelines/test_<family>_pipeline_parity.py -v -s
python examples/inference/basic/basic_<family>.py

Pipeline parity must compare real outputs, not only successful generation:

• denoised latents when decode parity is expensive or nondeterministic;
• decoded videos/images when visual output should be deterministic enough;
• decoded waveform or audio features for audio pipelines;
• separate video and audio targets for joint AV pipelines unless a validated joint metric exists.

Debug pipeline drift in this order:

1. Confirm both sides use the same component weights and component parity PASS results are still valid.
2. Align official and FastVideo call arguments, presets, and default values.
3. Align scheduler timesteps, sigmas/noise levels, prediction type, flow shift, guidance math, and secondary-guidance branches.
4. Align RNG: initial latents/noise, generator device, seed, per-step noise, VAE sampling, and any official +1 frame or crop/slice behavior.
5. Align conditioning: prompt templates, negative prompts, masks, image/audio preprocessing, modality packing, text truncation, and dtype/autocast.
6. Align decode: latent scaling, per-channel mean/std, tiling flags, output channel order, sample rate, FPS, and final slicing.
7. Add targeted stage-level diagnostics to identify the first divergent stage.

If stage diagnostics show the first bad stage is transformer/denoising or a mid-DiT block, enable activation trace before adding ad hoc pipeline prints; see docs/contributing/activation_trace.md and ../add-model-08-trace/SKILL.md. Keep FASTVIDEO_TRACE_LAYERS, FASTVIDEO_TRACE_STATS, and FASTVIDEO_TRACE_STEPS identical across reruns so pipeline parity traces diff one-to-one.

If the first divergence belongs to component implementation, strict loading, or conversion mapping, stop pipeline edits and return next_step=return_to_phase_6 with the exact failing evidence. Do not patch conversion from this skill.

Stage And Variant Rules

• Canonical video T2V order: InputValidationStage, TextEncodingStage, ConditioningStage, TimestepPreparationStage, LatentPreparationStage, DenoisingStage, DecodingStage.
• Canonical video I2V delta adds image loading/encoding and image VAE encoding in the official order, commonly: TextEncodingStage, ImageEncodingStage, ConditioningStage, TimestepPreparationStage, LatentPreparationStage, ImageVAEEncodingStage, DenoisingStage, DecodingStage.
• Treat ConditioningStage as default-present for Wan-style pipelines, but still follow the reference if another family truly skips or replaces it.
• T2V video pipelines usually use validation, text encoding, conditioning, timestep preparation, latent preparation, denoising, and decoding.
• I2V adds image loading/encoding and image-latent preparation according to the official pipeline, not by assuming CLIP or Wan-specific branches.
• Pick image, audio, and video encoders from the reference. Do not assume CLIP or any other common encoder unless the reference uses it.
• Cross-attention class names are not prescribed; match the family style and preserve the official tensor contract.
• WorkloadType currently has no T2A, A2A, or AV values. Until that enum is extended, audio-only pipelines may register with WorkloadType.T2V and preset workload_type="t2v" as a compatibility shim, but must document the rationale in code and PORT_STATUS.md.
• Audio-only pipelines should not force real video semantics into presets. Use minimal video-shaped placeholders such as small height/width and num_frames=1 only when shared VideoGenerator/validation paths require them, and document that the real output is audio.
• Record modality-specific shape knobs and output contract in the pipeline handoff: video uses height, width, num_frames, and fps; audio uses audio_seconds and sampling_rate; joint AV records both plus whether output is muxed or paired files.
• Use sibling pipeline classes/configs when required modules, HF repo layout, stage chains, or inputs differ materially.
• Use one kwargs-driven pipeline class only when variants share weights, modules, stage chain, and safe call semantics.
• Split later if components diverge, workload tags require separate discovery, signatures become unsafe, or stage branches become substantial.
• If the DiT branches on added_kv_proj_dim, document the T2V/I2V split.
• If the reference uses transformer_2, boundary_ratio, guidance_scale_2, or DMD step lists, keep those on config, presets, or stages deliberately.
• Support every official output head in scope. If a head is out of scope, record explicit user approval in PORT_STATUS.md.

Pipeline verification order:

pytest tests/local_tests/pipelines/test_<family>_pipeline_smoke.py -v -s
DISABLE_SP=1 pytest -v -s tests/local_tests/pipelines/test_<family>_pipeline_parity.py
python examples/inference/basic/basic_<family>.py

Smoke tests prove loadability only. They are not a substitute for numerical component or pipeline parity.

Escape Hatches

Follow ../add-model/shared/common_rules.md. Pipeline-specific ask cases include dropping a public mode, modality, or output head; adding a new workload enum; changing official defaults for user-facing behavior; accepting a known pipeline parity blocker; running GPU-heavy quality work outside the agreed scope; or publishing/uploading generated references or converted weights.

Handoff

Return ../add-model/contracts/pipeline_handoff.md and update the shared state files before handoff.

Do not hand back a green pipeline if smoke or parity skipped locally. A skip is a setup gap, not a pass.

References

• fastvideo/pipelines/composed_pipeline_base.py for module loading and stage execution.
• fastvideo/pipelines/basic/wan/ for standard video T2V/I2V/DMD variants.
• fastvideo/pipelines/basic/stable_audio/ for audio-specific stage composition.
• fastvideo/configs/pipelines/stable_audio.py and fastvideo/pipelines/basic/stable_audio/presets.py for config/preset shape.
• fastvideo/registry.py for register_configs(...) and preset registration.
• tests/local_tests/pipelines/test_gamecraft_pipeline_parity.py for latent parity structure.
• tests/local_tests/pipelines/test_stable_audio_pipeline_parity.py for audio parity structure.
• tests/local_tests/pipelines/test_stable_audio_pipeline_smoke.py for no-GPU import/registry/preset preflight shape.

Add-Model PR Review

Use this skill when a reviewed PR appears to add, port, or substantially modify a FastVideo model family, model variant, first-class model component, checkpoint conversion, model pipeline, or local parity coverage.

This is a review skill, not an implementation workflow. Do not run /add-model or start writing missing port code during review. Use the add-model skill stack as a rubric for findings.

Trigger Paths

Trigger this skill if git diff --name-only <base>...HEAD includes any of:

• fastvideo/models/dits/, fastvideo/configs/models/dits/
• fastvideo/models/vaes/, fastvideo/configs/models/vaes/
• fastvideo/models/encoders/, fastvideo/configs/models/encoders/
• fastvideo/models/schedulers/, fastvideo/configs/models/schedulers/
• fastvideo/models/upsamplers/, fastvideo/configs/models/upsamplers/
• fastvideo/models/audio/, fastvideo/configs/models/audio/
• fastvideo/pipelines/basic/, fastvideo/configs/pipelines/
• fastvideo/registry.py, fastvideo/api/sampling_param.py
• scripts/checkpoint_conversion/
• examples/inference/basic/
• tests/local_tests/, especially component or pipeline parity tests
• fastvideo/tests/ssim/ or other quality-regression tests for generated media

Also trigger when the PR title/body claims a new model, model variant, VAE, encoder, scheduler, conditioner, pipeline, conversion script, or generated-media quality baseline even if the path list is incomplete.

Review Inputs

Read these add-model references as review checklists:

• ../add-model/SKILL.md: phase gates and final handoff requirements.
• ../add-model/shared/common_rules.md: token/auth safety, production import boundaries, state files, and skip/pass semantics.
• ../add-model/contracts/final_handoff.md: final evidence expected from a complete port.
• ../add-model/contracts/component_context.md and ../add-model/contracts/component_skill_handoff.md: component evidence and parity-debug expectations.
• ../add-model/contracts/conversion_request.md and ../add-model/contracts/conversion_handoff.md: conversion evidence, strict-load status, config validation, and retry context.
• ../add-model/contracts/pipeline_context.md and ../add-model/contracts/pipeline_handoff.md: pipeline class/stage/config/ preset/registry/example evidence.

Then read only the satellite skill(s) that match touched areas:

• DiT/transformer changes: ../add-model-03-port-dit/SKILL.md.
• VAE changes: ../add-model-04-port-vae/SKILL.md.
• Encoder/conditioner changes: ../add-model-05-port-encoder/SKILL.md.
• Scheduler/upsampler/vocoder/other components: ../add-model-06-port-generic/SKILL.md.
• Component parity tests: ../add-model-02-parity/SKILL.md.
• Checkpoint conversion: ../add-model-07-conversion/SKILL.md.
• Pipeline/config/presets/registry/examples: ../add-model-09-pipeline/SKILL.md.
• Prep/state docs: ../add-model-01-prep/SKILL.md.

Required Review Lanes

For a full model-family or model-variant PR, cover all lanes. For a component-only PR, cover the component, conversion/parity as applicable, and the documented downstream consumer.

1. Scope and source-of-truth lane: Verify the PR clearly identifies the official reference, weights/revision, supported variants, modalities, output heads, and any approved scope cuts.

2. Component lane: Verify each required component is FastVideo-native or has a documented and accepted lazy-wrapper exception. Check bucket/config inheritance, EntryClass, state-dict surface, reused-component evidence, and output heads.

3. Conversion lane: Verify mappings are derived from prototype key/shape dumps, source layout is supported, skipped keys are intentional, emitted configs validate through production paths, component strict-load status is recorded, model_index.json library tokens match loaders, and revisions are pinned when converting from HF.

4. Component parity lane: Verify local parity tests exist for every required component, including reused components. Scaffolds may skip in CI, but the PR must provide local non-skip PASS evidence or an explicit accepted blocker.

5. Pipeline lane: Verify stage order, required modules, _class_name / EntryClass.__name__ resolution, config defaults, presets, SamplingParam fields, registry registration, examples, smoke tests, and pipeline parity.

6. Quality and evidence lane: Verify media quality regression is added or explicitly deferred, examples run, generated outputs are non-corrupt, tests/local_tests/<family>/README.md and PORT_STATUS.md are current, and final blockers are surfaced in the review.

Findings To Prioritize

Prioritize review findings in this order:

• Missing or skipped required component parity without accepted blocker.
• Pipeline parity/smoke/example missing or skipped for a pipeline PR.
• Conversion emits unloadable or unvalidated configs/weights.
• Wrong model_index.json _class_name, component library token, or registry class resolution.
• Runtime diffusers/transformers model-class imports for components that own weights or numerical behavior.
• Dropped modalities, output heads, variants, or conditioning streams without explicit approval.
• Reused FastVideo component lacks exact definition/instantiation proof or non-skip parity.
• Public generation kwargs/preset defaults missing from SamplingParam.
• Tests only check shapes, importability, or successful generation without numerical/media comparison.
• Tokens, credentials, reference clones, staged weights, or generated bulk assets committed to the PR.

Output Format

Write normal code-review findings first, ordered by severity. Include file and line references from the PR diff when possible.

Use this phrasing for missing add-model evidence:

This PR does not satisfy the add-model <component|conversion|pipeline|final>
gate because <specific required evidence> is missing. The risk is <runtime load,
numerical parity, dropped output, registry resolution, etc.>.

Keep the summary short. Mention which lanes were reviewed and which could not be verified because assets, GPU time, or external credentials were unavailable.

Add Model

Manual Invocation

This skill is for explicit /add-model use only. Do not auto-start it from a casual model-port mention. The setup-only workflow is ../add-model-01-prep/SKILL.md.

Goal

Port a new FastVideo model family, model variant, or first-class reusable component so it can be loaded through FastVideo's native model, config, stage, registry, preset, and test infrastructure.

FastVideo has one pipeline architecture: stage-based composition via ComposedPipelineBase. Vary the stages and modules, not the architecture.

Scope Shapes

Use this skill for either shape:

If upstream ships many variants, lock scope before coding. "Base model" means checkpoint variant, not a modality subset. If the base checkpoint produces audio, pose, depth, masks, or other output heads, either support those outputs or get explicit user agreement to drop them.

Required Input

Start from an add-model-01-prep handoff, or equivalent fields matching contracts/prep_handoff.md.

Before Phase 0, read the shared rules and all relevant schemas:

• shared/common_rules.md
• contracts/prep_handoff.md
• contracts/port_state.md
• contracts/escape_hatch.md
• contracts/component_context.md
• contracts/parity_status.md
• contracts/conversion_request.md
• contracts/conversion_handoff.md
• contracts/component_skill_handoff.md
• contracts/pipeline_context.md
• contracts/pipeline_handoff.md
• contracts/final_handoff.md

Hard Rules

• Follow shared/common_rules.md for token/auth safety, state files, escape hatches, production import boundaries, and skip/pass semantics.
• If the prep handoff is missing or ambiguous, stop and run ../add-model-01-prep/SKILL.md.
• If a needed component is not ported, do not ship the pipeline that needs it.
• Wan is grandfathered for missing local parity; do not copy its missing-test precedent for new work.

Escape Hatches

Follow shared/common_rules.md and contracts/escape_hatch.md. The main orchestrator should ask only when no phase skill can safely continue under the shared rules.

Files Map

Phase 0: Scope And Handoff Gate

1. Validate every required handoff field.
2. Resolve needs_conversion=unknown before component work:

python ".agents/skills/add-model-01-prep/scripts/inspect_hf_layout.py" \
    "<hf-or-local-path>" \
    --json

3. List first-PR scope across both axes:
   • Variant axis: base, distill, SR/refine, causal, DMD, I2V, V2V, etc.
   • Modality axis: video, image, audio, pose, depth, masks, text, etc.
4. For component-only work, explicitly name the downstream full-pipeline PR or planned consumer.
5. Confirm official_env_status is imports_ok or private_deps_need_stubs. If it is blocked, return to ../add-model-01-prep/SKILL.md before parity scaffolding.
6. Confirm local_tests_readme exists and records official setup, HF weights, dependency changes, and planned parity commands for reviewers.
7. Confirm port_state_file exists, follows contracts/port_state.md, and has rows for open questions/issues found during prep.
8. If there are multiple official implementations, choose the one whose architecture matches the published weights. A blessed library port can be a better parity reference than a highly configurable research repo; document the choice in tests.

Phase 1: Reference And Architecture Study

Read the official pipeline call path before writing code.

Record:

• Required modules from model_index.json or equivalent: transformer, VAE, text encoders, tokenizers, scheduler, image encoders, audio VAE, vocoder, conditioners, upsamplers.
• Input/output modalities and every dedicated DiT output head.
• Text/image/audio encoding flow, latent shape, dtype, scaling, packing, scheduler/timestep math, guidance math, VAE normalization, and decode flow.
• Whether the official code relies on private deps, custom ops, or special kernels that parity tests must stub.

Arch config rule:

• ArchConfig fields must match the emitted per-component config, especially transformer/config.json, one-to-one.
• Pipeline knobs do not belong on the DiT arch config: inference steps, CFG scales, flow shift, FPS, VAE stride, text target length, data-proxy knobs, eval defaults, and sampling defaults go on PipelineConfig, presets, or stages.
• If the HF repo is raw or has empty configs, synthesize transformer/config.json from the official Python model-config class, not from data/eval config classes.

Phase 2: Early Parity Scaffolding

Create component parity tests before or alongside implementation. Use ../add-model-02-parity/SKILL.md and its templates/component_parity_test.py. The official reference must import in the current FastVideo environment, or the prep handoff must identify private deps that will be stubbed locally for tests. Use local_tests_readme as the reviewer-facing source for setup commands and update its planned test table as parity scaffolds are added.

This phase is early by design:

• Official loading can be implemented from the reference study.
• FastVideo loading can target planned standardized class/config/loader paths.
• Tests may initially skip because the FastVideo class or converted weights do not exist yet.
• The scaffold must still contain real official loading, deterministic inputs, output extraction, and concrete tensor comparisons. No unconditional skips, no shape-only tests.

Use subagents here: dispatch one parity-test subagent per required component, including components that may be reused. Their output becomes the red/skip target that porting or reuse-verification subagents make pass later.

Phase 3: Reuse Gate And Component Dispatch

Build a component inventory before implementation:

Reuse is allowed only from the checked-out FastVideo tree. Do not wait for or depend on an open PR adding a native class; add the native port directly in this PR if the current tree cannot be reused.

Reuse decision:

1. Record exact official definition and instantiation evidence for every component.
2. If an existing FastVideo class and config match both definition and instantiation, pass that reused target to the bucket-specific skill in mode=prototype and require reuse evidence plus key/shape dumps.
3. If either definition or instantiation differs, port the component directly as FastVideo-native code through the bucket-specific skill.
4. Reused components still require non-skip component parity against the exact official instantiation used by the target pipeline.

Porting subagent dispatch:

• Dispatch one subagent per component after Phase 2 parity scaffolds exist.
• Use ../add-model-03-port-dit/SKILL.md for DiTs/transformers.
• Use ../add-model-04-port-vae/SKILL.md for VAEs.
• Use ../add-model-05-port-encoder/SKILL.md for text, image, audio, or compound encoders/conditioners that fit the encoder config bucket.
• Use ../add-model-06-port-generic/SKILL.md for schedulers, upsamplers, vocoders, adapters, preprocessors, or unknown components.
• Each subagent owns one component only and must loop on that component's local parity test until it produces a non-skip PASS or returns a precise blocker.

Every component subagent must receive a complete packet matching contracts/component_context.md. If any required path is unknown, pass unknown plus the exact search already performed. Do not silently omit ambiguous official files or prototype concerns.

Bucket, layer, and attention rules live in the bucket-specific skills and fastvideo/layers/AGENTS.md.

Phase 4: Native Component Prototype

Conversion needs a FastVideo state-dict surface. Use the Phase 3 bucket-specific skill in mode=prototype for every required component, including reused components.

Prototype success criteria:

• the FastVideo-native or reused class/config can import and instantiate with the exact official architecture args;
• official and FastVideo key/shape dumps exist for every stateful component;
• local_tests_readme and port_state_file record prototype status and concerns;
• the returned handoff matches contracts/component_skill_handoff.md.

Do not chase numerical parity in Phase 4. Prototype mode ends when conversion has the key/shape surface it needs, or when the component skill returns a precise blocker or escape hatch.

Phase 5: Param Mapping And Weight Conversion

Use ../add-model-07-conversion/SKILL.md after Phase 4 prototypes exist. Send a request matching contracts/conversion_request.md; consume the returned contracts/conversion_handoff.md update before Phase 6.

Use the prep handoff's needs_conversion value:

• no: verify the source already has the component layout FastVideo loaders can consume, then record any passthrough components.
• yes: write scripts/checkpoint_conversion/<family>_to_diffusers.py and output converted_weights/<family>/.
• unknown: return to Phase 0.

The conversion skill owns source-layout handling, mapping derivation, config and model_index.json emission, passthrough assets, strict-load verification, and Phase 6 retry requests. Component skills must not patch conversion scripts or converted weights ad hoc.

Phase 6: Component Parity Debug

This is the expected expensive loop. Dispatch one subagent per required component, including reused components, using the bucket-specific skill in mode=parity-debug.

Each subagent gets:

• the complete component context packet from Phase 3/4;
• updated conversion mapping notes and strict-load result from Phase 5;
• any prototype concerns or unknowns that were not resolved before conversion.

The bucket-specific skills own parity-debug tactics. If a failure belongs to conversion, route it through ../add-model-07-conversion/SKILL.md with a retry request matching contracts/conversion_request.md, then resume the component skill with the updated conversion handoff.

When a component failure narrows to layer-by-layer numerical drift, load ../add-model-08-trace/SKILL.md before writing custom hooks. It uses fastvideo/hooks/activation_trace.py; canonical env vars and JSONL format are documented in docs/contributing/activation_trace.md.

Phase 6 ends only when every required component handoff reports parity_status=non_skip_pass, or when a precise blocker or escape hatch is recorded in port_state_file.

Phase 7: Pipeline, Stages, And Variants

Do not start Phase 7 until every required component, reused or ported, has a non-skip local parity PASS from Phase 6. If any component parity test is still scaffold_skip, debug_red, blocked, or missing, resume Phase 6 first.

Use ../add-model-09-pipeline/SKILL.md for pipeline definition and parity-debug. Send a complete packet matching contracts/pipeline_context.md; consume the returned contracts/pipeline_handoff.md before moving to quality regression or final handoff.

The pipeline skill owns:

• pipeline class, stage chain, and optional model-specific stages;
• pipeline config, presets, registry updates, and examples;
• official args/defaults/presets comparison before setting FastVideo defaults;
• pipeline smoke and parity tests;
• continuous pipeline parity-debug until non-skip PASS or precise blocker;
• updates to local_tests_readme and port_state_file.

The pipeline handoff must explicitly cover stage order, variants, modality and output-head handling, config/preset/registry/example status, smoke/parity tests, and any return-to-Phase-6 evidence.

Phase 8: PipelineConfig, Presets, Registry, Examples

This phase is implemented through ../add-model-09-pipeline/SKILL.md after the Phase 7 component-parity gate passes. Accept the pipeline handoff only if it covers configs, presets, registry detection/exact class resolution, examples, new SamplingParam fields for public kwargs/defaults, and local smoke/parity status. Detailed rules live in ../add-model-09-pipeline/SKILL.md.

Phase 9: Parity Activation And Local Verification

Local parity is author-run, not CI-enforced. CI may only run package-level quality tests later. Before handoff, Phase 2 scaffolds must be activated into non-skip PASS results.

Order is mandatory:

1. Run conversion if needed.
2. Run component parity for every required component, including reused ones.
3. Run pipeline smoke.
4. Run pipeline parity.
5. Run the basic example.

If pipeline smoke or parity points back to component implementation, strict-load, or conversion mapping, return to Phase 6 or Phase 5 rather than patching around the issue in the pipeline.

Skip policy:

• Follow shared/common_rules.md: a committed local test may skip for absent clones/weights, but a local skip is not a verified pass.

Use the commands and tolerance guidance from ../add-model-02-parity/SKILL.md for component checks and from ../add-model-09-pipeline/SKILL.md for pipeline smoke, pipeline parity, and examples. Record exact commands, status, and blockers in local_tests_readme and port_state_file.

Phase 10: Quality Regression

Video outputs:

• Add fastvideo/tests/ssim/test_<family>_similarity.py when output video quality must be preserved.
• Seed references through seed-ssim-references after the test exists.

Audio outputs:

• SSIM does not apply. Use an audio-specific regression metric such as mel-spectrogram L1, multi-resolution STFT, CLAP cosine, or a project-approved learned metric.
• Document the metric and hardware/runtime assumptions in the test.

Joint AV outputs:

• Keep video and audio regression checks separate unless there is a validated joint metric.

Phase 11: Post-Parity Review And Handoff

After parity is green, run a hot-path review before handoff:

• Hoist constant tensor allocations out of sampler/denoising loops.
• Replace per-step randn_like churn with preallocated buffers plus .normal_() when safe.
• Move torch.backends.* flag changes to one-shot setup/load paths.
• Delete batch.extra writes that nothing reads.
• Derive magic constants from configs when possible.

Pre-handoff checklist:

[ ] Prep handoff is complete and committed nowhere with token values.
[ ] Conversion was run if needed and output loads with real weights.
[ ] Every required component, reused or newly ported, has a non-skip local parity PASS.
[ ] `local_tests_readme` lists every component parity test, command, status, and blocker if any.
[ ] `port_state_file` has every open question/issue either resolved or listed as an explicit blocker.
[ ] Any `next_step=ask_user` has a matching `escape_hatch` block and `E###` row.
[ ] Pipeline smoke has a non-skip local PASS.
[ ] Pipeline parity has a non-skip local PASS against the official reference.
[ ] Basic example runs and writes a non-corrupt output.
[ ] Video SSIM or audio-specific quality regression is added or explicitly deferred.
[ ] Runtime production code has no diffusers/transformers model-class imports.
[ ] Production comments are WHY-focused; examples have user-story docstrings.
[ ] Post-parity hot-path pass is complete.

Ask before deleting any reference clone or staged weights created by add-model-01-prep. Leave .gitignore entries so future parity assets stay untracked. Never commit the clone, weights, .env, credentials, or anything matching *secret*.

References

• ../add-model-01-prep/SKILL.md for user-input collection, HF inspection, weight staging, reference cloning, and setup handoff.
• contracts/ for canonical handoff schemas used by prep, parity, conversion, component porting, escape hatches, and final handoff.
• ../add-model-02-parity/SKILL.md for early component parity scaffolds and activation templates.
• ../add-model-07-conversion/SKILL.md for Phase 5 mapping, conversion scripts, monolithic checkpoint splitting, and strict-load checks.
• ../add-model-03-port-dit/SKILL.md, ../add-model-04-port-vae/SKILL.md, ../add-model-05-port-encoder/SKILL.md, and ../add-model-06-port-generic/SKILL.md for component subagent implementation and parity-debug loops.
• ../add-model-09-pipeline/SKILL.md for pipeline definition, config/preset/ registry/example wiring, smoke tests, and pipeline parity-debug.
• fastvideo/layers/AGENTS.md for native layer selection and state-dict surface guidance.
• docs/contributing/coding_agents.md for narrative context.
• docs/design/overview.md for pipeline/config/registry architecture.
• fastvideo/pipelines/basic/wan/ for standard T2V/I2V/DMD/Causal variants.
• fastvideo/pipelines/basic/ltx2/ for non-standard stages and audio/video patterns.
• tests/local_tests/pipelines/test_gamecraft_pipeline_parity.py for pipeline parity shape.
• tests/local_tests/transformers/test_ltx2.py, tests/local_tests/vaes/test_ltx2_vae.py, and tests/local_tests/encoders/test_ltx2_gemma_parity.py for component parity.
• scripts/checkpoint_conversion/convert_ltx2_weights.py for modern conversion script shape.
• scripts/checkpoint_conversion/wan_to_diffusers.py for legacy regex mapping reference only.

Changelog

Decompose Pipeline PR

Purpose

When a PR adds a new pipeline (or first-class component port) and crosses ~3,000 LOC, single-shot review converges to rubber-stamping. This skill decomposes such a PR into a stack of independently-reviewable PRs without disturbing main.

It is the inverse of add-model: where add-model walks adding a new pipeline as a fresh PR, this skill walks decomposing an existing oversized pipeline PR.

Worked example: PR #1280 (daVinci-MagiHuman, 9,812 LOC, 56 files) → 2 prerequisite PRs off main + 8-PR stack:

• #1293 will/activation-trace (prerequisite)
• #1294 will/loader-infra (prerequisite)
• #1295 (1/8) housekeeping
• #1296 (2/8) t5gemma encoder
• #1297 (3/8) DiT
• #1298 (4/8) pipeline stages
• #1299 (5/8) pipeline orchestrator
• #1300 (6/8) provenance (AGENTS.md, JOURNAL.md, lessons)
• #1301 (7/8) conversion scripts
• #1302 (8/8) registry activation

Prerequisites

• Open PR number on hao-ai-lab/FastVideo (or any FastVideo fork)
• gh CLI authenticated against the target remote
• Local git worktree support (git worktree)
• Git config user.name / user.email set
• Pre-commit installed (pre-commit install --hook-type pre-commit --hook-type commit-msg)
• The target PR's branch fetched locally as origin/<feature-branch>

Inputs

Steps

1. Verify ground truth (do not trust gh pr diff --name-only)

gh pr diff <N> --name-only has been observed to emit phantom file entries. Always cross-check against the authoritative git diff:

mkdir -p .agents/tmp/decompose-<N>
git fetch origin pull/<N>/head:<feature-branch>
git diff origin/main..origin/<feature-branch> --name-status \
  > .agents/tmp/decompose-<N>/files.txt
git diff origin/main..origin/<feature-branch> --stat

Use the --name-status output as the authoritative file list. If it disagrees with gh pr diff --name-only, trust the git diff.

2. Tier the diff by blast radius

Classify every changed file into one of four tiers:

FastVideo Tier 2 protected paths:

fastvideo/utils.py
fastvideo/pipelines/composed_pipeline_base.py
fastvideo/models/loader/component_loader.py
fastvideo/configs/models/dits/__init__.py
fastvideo/configs/models/encoders/__init__.py
fastvideo/configs/models/vaes/__init__.py
fastvideo/envs.py
fastvideo/fastvideo_args.py
fastvideo/distributed/**
fastvideo/layers/**
fastvideo/attention/**
fastvideo/registry.py    # treat as Tier 3 if change is the activation

Tier 3 detection (mechanical):

git diff origin/main..origin/<feature-branch> -- fastvideo/registry.py | \
  grep -E "^\+.*register_configs\("

If registry.py only contains register_configs additions, treat it as Tier 3. If it modifies existing behavior, treat it as Tier 2 (rare).

3. Identify reusable Tier-1 components

Within Tier 1, look for sub-trees that are not model-specific and could land separately:

• Encoders matching a known multi-model base (T5/T5-Gemma/Llama/Gemma/CLIP variants)
• New stage classes that subclass shared bases without referencing the new model
• Hook/profiler/debug infra under fastvideo/hooks/
• New helpers that have no model-specific dependencies

These get split into their own PRs (e.g. PR 4 t5gemma-encoder in the MagiHuman example).

4. Hunt for missing test coverage on Tier 2 changes

For every Tier-2 file modified, check whether the original PR added unit tests for the new behavior:

for f in <list-of-tier-2-files>; do
  echo "=== Tests for $f ==="
  git diff origin/main..origin/<feature-branch> -- \
    "$(echo $f | sed 's|fastvideo/|fastvideo/tests/|; s|\.py|*|')"
done

If a Tier-2 PR has no accompanying tests, emit a "must-add tests" list with a sketch of the case grid. Tier-2 PRs do not ship without those tests.

The MagiHuman example required this for PR-B (utils.py): the original PR shipped no test_utils_loader.py, so the decomposition added 9 unit-test cases covering the umbrella-detector boundary, the optional-component-dirs relaxation, and regression coverage on every existing 2-segment HF id.

5. Build the dependency DAG and topo-sort

Edges:

• Tier 2 infra → Tier 1 code that imports it
• Reusable Tier 1 components → model-specific Tier 1 code that uses them (encoder before DiT before pipeline)
• Tier 1 → Tier 3 (activation always last)
• Tier 0 has no dependents (lands first as a freebie)

Topo-sort produces the stack ordering. Pull Tier-2 PRs out of the stack when they have no model-specific dependency — they should land off main with their own focused review, not buried in a model port.

Render as a tree (markdown):

main
 ├─ <prereq-A>
 │   └─ <prereq-B>
 │       ├─ <stack-01-housekeeping>
 │       │   └─ <stack-02-encoder>
 │       │       └─ <stack-03-dit>
 │       │           └─ ...
 │       │               └─ <stack-N-activate>
 │       └─ (parallel) <skill-pr> off main

6. Detect mis-shelved docs and debug scratch

Two categories to flag:

• Mis-shelved docs: Markdown files under tests/local_tests/ are journals, not tests. Flag for relocation to the package dir as JOURNAL.md.
• Debug scratch: files starting with _debug_, _scratch_, or _explore_. Flag for drop (do not carry into any output PR).

For MagiHuman: tests/local_tests/magi-human.md → relocate. Two _debug_magi_human_*.py files → drop.

7. Author the AGENTS.md manifest skeleton

For the new pipeline package, generate a 6-section AGENTS.md scaffold with the file table pre-populated from the diff:

1. Manifest — file table by role
2. Parity invariants — load-bearing rules with one-paragraph each + lesson refs
3. Cross-refs — "If you change X, re-run Y" matrix
4. Run book — single pytest command + prereqs (HF tokens, GPU, wall-time)
5. Open questions — known issues (e.g. tolerance carve-outs)
6. Provenance — PR table with branch names and source SHA

The provenance section is filled incrementally during stack execution and finalized in the activation PR.

8. Extract lessons from the PR body

Scan the PR body for sections titled "Key implementation work", "Bug hunt", "Lessons", or sentences with patterns like "took N waves to localize", "silent regression", "investigation revealed". Each becomes a candidate .agents/lessons/<YYYY-MM-DD>_<slug>.md draft.

Lessons MUST follow the existing template in .agents/lessons/README.md:

• YAML frontmatter: date, experiment, category, severity
• Sections: What Happened, Root Cause, Fix / Workaround, Prevention
• Filename: <YYYY-MM-DD>_<short-slug>.md

Lessons co-locate with the code they concern: a conversion-script lesson lands in the same PR as the conversion script, not in the docs PR.

9. Emit the commit-footer convention

Every commit in the stack ends with:

<Feature>-Stack: N/M

E.g. Magi-Stack: 5/8. Use the package directory name as the feature key. After all PRs squash-merge, git log --grep='^<Feature>-Stack:' reconstructs the lineage even if PR numbers later get renumbered.

10. Produce the worktree bootstrap

Generate a runnable bash script:

#!/bin/bash
set -euo pipefail

REPO=/home/<user>/FastVideo
WORKTREE=/home/<user>/FastVideoMagi   # NB: directory name must be a valid
                                       # Python identifier (no hyphens) so
                                       # mypy doesn't choke
SOURCE_PR=<N>
SOURCE_BRANCH=will/<feature>
SOURCE_SHA=$(git -C "$REPO" rev-parse "origin/$SOURCE_BRANCH")

git -C "$REPO" fetch origin main:main
git -C "$REPO" fetch "origin/$SOURCE_BRANCH"
git -C "$REPO" worktree add "$WORKTREE" origin/main

# Capture baseline for provenance. Everything under .agents/tmp is transient
# and ignored by git.
OUTPUT_DIR="$REPO/.agents/tmp/decompose-$SOURCE_PR"
mkdir -p "$OUTPUT_DIR"
cat > "$OUTPUT_DIR/<feature>-baseline-${SOURCE_SHA:0:8}.txt" <<EOF
Source PR: <repo>#$SOURCE_PR
Source SHA: $SOURCE_SHA
Authoritative file count: $(git -C "$REPO" diff origin/main..origin/$SOURCE_BRANCH --name-only | wc -l)
Date captured: $(date -u +%Y-%m-%dT%H:%M:%SZ)
EOF

11. Author preserve via git checkout, not cherry-pick

For each stack PR:

git -C "$WORKTREE" switch -c <new-branch> <base-branch>
git -C "$WORKTREE" checkout origin/<source-branch> -- <file1> <file2> ...
git -C "$WORKTREE" commit -m "[<scope>]: <subject>

<body>

<Feature>-Stack: N/M"
git -C "$WORKTREE" push -u origin <new-branch>
gh pr create --base <base-branch> --head <new-branch> --title "..." --body "$(cat <<EOF ... EOF)"

Notes:

• git checkout origin/<source> -- <files> extracts only the named files, preserving the diff. The original PR's author is not preserved on the new commit (it's authored by whoever runs the script). Reference the original PR + source SHA in every commit body and PR description for authorship attribution.
• Never use git cherry-pick for this workflow — cherry-pick applies whole commits, which mixes concerns across PR boundaries.

Outputs

The skill produces all transient planning artifacts under .agents/tmp/decompose-<pr>/:

1. A markdown decomposition plan (plan.md)
2. A proposed branch graph
3. A worktree-bootstrap script (bootstrap.sh)
4. Per-PR file allocation lists (under stack/)
5. AGENTS.md scaffolds for any new pipeline packages
6. Draft lesson files (placed alongside the PR that owns the code they concern)
7. A finalized provenance table for the package AGENTS.md

Anti-Patterns

The skill should warn against:

• "Just rebase the megaPR into smaller commits." Doesn't help review; reviewer still sees one PR.
• Co-locating tests under the new package. FastVideo's convention is by-kind under fastvideo/tests/ and tests/local_tests/<family>/. Don't invent a new layout per pipeline.
• Splitting Tier 2 changes into "one file per PR." Tier 2 PRs are about semantic units (e.g., "loader umbrella + optional component dirs" together because they jointly define the new diffusers-format contract), not file-count.
• Landing the activation switch first ("just register, the code can be empty"). The skill enforces activation-last so every intermediate state is dead code, not broken code.
• Trusting gh pr diff --name-only. Cross-check against git diff origin/main..origin/<feature-branch> --name-status — gh's output has been observed to include phantom entries.
• Worktree dir names with hyphens. mypy interprets them as invalid Python package names and refuses to run. Use CamelCase or underscores.
• Skipping the lesson-extraction step. PR bodies contain the most expensive learnings of the original implementation. Losing them to a squash-merge is the silent decay of institutional knowledge.

Example Usage

User: split PR 1280
Agent: [invokes decompose-pipeline-pr]
       → produces .agents/tmp/decompose-1280/plan.md with:
         - tiered file table (56 files: 3 tier-0, 35 tier-1, 9 tier-2,
           9 tier-3)
         - branch graph (PR-A + PR-B + 8-PR stack)
         - worktree bootstrap script
         - per-PR file lists
         - AGENTS.md scaffold for fastvideo/pipelines/basic/magi_human/
         - 3 draft lessons extracted from the PR body
       → asks user to confirm before opening branches

References

• The MagiHuman decomposition (worked example): fastvideo/pipelines/basic/magi_human/AGENTS.md (after PR #1302 merges)
• Existing skill: .agents/skills/add-model/SKILL.md (the inverse — adding a new pipeline as a fresh PR)
• Lesson template: .agents/lessons/README.md
• Skill template: .agents/skills/SKILL_TEMPLATE.md

dreamverse-deploy — redeploy migrated Dreamverse on a chosen GPU

Scope: project (lives in this repo at .agents/skills/dreamverse-deploy/)

When to use: you want to (re)launch the migrated apps/dreamverse/ backend and frontend on this dev node, pinned to a specific physical GPU. Tears down any existing deploy on the same ports first, then boots fresh and waits for both /readyz and the FE root to return 200.

Prerequisites

• Working tree containing apps/dreamverse/
• dreamverse-server installed from this checkout; if missing, run uv pip install -e ".[dreamverse]"
• Local conda env at ~/miniconda3/envs/fv-main/ with flashinfer-python, cerebras-cloud-sdk, openai installed (override the default path with DREAMVERSE_PYTHON=/path/to/python)
• ~/.env exporting CEREBRAS_API_KEY, GROQ_API_KEY, etc.
• npm available in $PATH (or set NPM=/path/to/npm)
• gcc-13 + g++-13 at /usr/bin/ (workaround for nvcc gcc-15 rejection)
• Recommended: native ffmpeg at $HOME/opt/ffmpeg-native/bin/ffmpeg, built via bash apps/dreamverse/scripts/install_native_ffmpeg.sh. The deploy detects that binary directly and exports it for the backend. The installer's generated apps/dreamverse/scripts/ffmpeg-env.sh is for manual launches. When the binary is missing, the deploy falls back to system ffmpeg with a warning. Set DREAMVERSE_REQUIRE_NATIVE_FFMPEG=true to make the missing binary a hard failure.

If any required prereq is missing, the script fails fast with a clear message.

Usage

# Deploy on GPU 4 with the current web port. The legacy helper default remains
# 5274, so pass 5299 explicitly. Torch compile and warmup are both off.
./.agents/skills/dreamverse-deploy/scripts/dreamverse-deploy.sh 4 8009 5299

# Deploy on GPU 6 with custom ports
./.agents/skills/dreamverse-deploy/scripts/dreamverse-deploy.sh 6 8089 5275

# Deploy on GPU 0 with warmup enabled
./.agents/skills/dreamverse-deploy/scripts/dreamverse-deploy.sh --warmup 0 8009 5299

# Deploy with torch.compile enabled (max-autotune; first segment ~3-4min,
# subsequent segments save ~3s — only worth it for benchmarking)
./.agents/skills/dreamverse-deploy/scripts/dreamverse-deploy.sh --torch-compile 4 8009 5299

# Deploy with both warmup AND torch.compile enabled
./.agents/skills/dreamverse-deploy/scripts/dreamverse-deploy.sh --warmup --torch-compile 4 8009 5299

# Flags can appear before, between, or after positional args
./.agents/skills/dreamverse-deploy/scripts/dreamverse-deploy.sh 4 8089 5275 --warmup

Arguments

Flags

Flags can appear in any position relative to the positional args. Explicit flag values always win over env-var defaults.

Environment variables (used when no flag is given)

What it does

1. Validates prereqs.
2. Kills any process on the target backend/frontend ports + waits for the target GPU to release memory (allows up to 30s for cleanup).
3. Sources ~/.env.
4. Exports the env recipe required for boot:
   • CUDA_VISIBLE_DEVICES=<gpu>
   • FASTVIDEO_ENABLE_DEVTOOLS=1
   • FASTVIDEO_ENABLE_STARTUP_WARMUP=<DREAMVERSE_WARMUP>
   • FASTVIDEO_GPU_COUNT=1
   • ENABLE_TORCH_COMPILE=<0|1 derived from DREAMVERSE_TORCH_COMPILE>
   • CC=/usr/bin/gcc-13 CXX=/usr/bin/g++-13 CUDAHOSTCXX=/usr/bin/g++-13
   • NVCC_PREPEND_FLAGS="-ccbin /usr/bin/gcc-13 -allow-unsupported-compiler"
   • FASTVIDEO_FFMPEG_BIN=$HOME/opt/ffmpeg-native/bin/ffmpeg + FASTVIDEO_VIDEO_CODEC=<libx264|h264_nvenc> (when the native binary exists)
5. Launches the installed dreamverse-server console command in a detached setsid session and captures its PID.
6. Polls /readyz until 200. The budget is 5 minutes by default, 8 minutes with one startup optimization enabled, and 15 minutes with both warmup and torch.compile enabled.
7. Launches the devtools frontend through npm in a detached session and captures its PID.
8. Polls FE / until 200 (max 60s).
9. Prints URLs, PIDs, and log paths.

What it does NOT do

• Does not modify ~/.env or the FastVideo .venv.
• Does not push code or commit anything.
• Does not run Playwright. Use the e2e wrapper separately:

  cd apps/dreamverse/web
  PLAYWRIGHT_SKIP_WEBSERVER=1 BACKEND_HOST=127.0.0.1 BACKEND_PORT=8009 \
    PLAYWRIGHT_BASE_URL=http://127.0.0.1:5299 \
    NEXT_PUBLIC_INCLUDE_DEVTOOLS=1 \
    npm exec -- playwright test
  

  The standard suite runs by default; the long-running two-segment audio-continuation spec is gated behind PLAYWRIGHT_LONG_RUNNING=1 (see below).

Long-running e2e (paired with --warmup --torch-compile)

apps/dreamverse/web/e2e/long-running-segments.spec.ts drives a real two-segment session through the FE, captures every WS frame, and asserts segments 1 AND 2 both reach media_segment_complete with at least one binary fMP4 chunk per segment. It guards against the BrokenPipe regression previously caused by dropped LTX-2 audio continuation kwargs. Skipped by default. Enable with:

./.agents/skills/dreamverse-deploy/scripts/dreamverse-deploy.sh \
    --warmup --torch-compile 4 8009 5299

cd apps/dreamverse/web
PLAYWRIGHT_SKIP_WEBSERVER=1 \
  BACKEND_HOST=127.0.0.1 \
  BACKEND_PORT=8009 \
  PLAYWRIGHT_BASE_URL=http://127.0.0.1:5299 \
  NEXT_PUBLIC_INCLUDE_DEVTOOLS=1 \
  PLAYWRIGHT_LONG_RUNNING=1 \
  npm exec -- playwright test e2e/long-running-segments.spec.ts

Expected runtime: ~7-9 minutes on a B200 (torch.compile max-autotune warm-up dominates the cold start; per-test timeout is 900s). The spec hard-fails on any WS error/step_error frame so the BrokenPipe regression surfaces with the actual ffmpeg/audio diagnostics rather than an opaque "test timed out".

Teardown

Stop both services without redeploying:

# Stop services on default ports (port-pattern based)
./.agents/skills/dreamverse-deploy/scripts/dreamverse-deploy.sh --stop

# Stop AND nuke any process holding GPU N
./.agents/skills/dreamverse-deploy/scripts/dreamverse-deploy.sh --stop 4

The redeploy path (<GPU> mode) automatically nukes any process holding the target GPU before launching — including orphan multiproc_executor worker subprocesses left over from a parent backend that was killed without grace. This was the failure mode of an earlier naive port-only kill: parent dies, children survive, GPU stays full, next deploy OOMs.

Notes

• The installed dreamverse-server console command enters apps/dreamverse/dreamverse/server_entry.py, which loads the current Dreamverse runtime from apps/dreamverse/dreamverse/.
• The B200 / sm_100a NVCC flags are mandatory on this dev node because the conda toolchain ships gcc-15, which nvcc rejects. The script requires the configured gcc-13 and g++-13 binaries during preflight.

Deployment boundary

This skill is for a local checkout on a directly attached GPU. For a container image, use apps/dreamverse/docker/README.md. For Modal, follow apps/dreamverse/scripts/modal/README.md; do not adapt this process-killing workflow to a remote deployment.

Re-seed Performance Baseline

Purpose

Replace or advance the rolling performance baseline for a single (model_id, gpu_type) pair in the HF dataset FastVideo/performance-tracking.

Performance comparison uses the median of up to the last 5 successful records for the same model and GPU. Failed records are useful audit history, but they do not move the future baseline because compare_baseline.py loads records with successful_only=True.

This skill now reseeds from a reviewed batch of one or more source performance JSONs. It uploads one new success=true record per accepted source JSON; it does not blindly replicate one measurement into 3 or 5 records. The effective reseed size is therefore dynamic and equals the number of provided, validated, internally consistent source JSONs.

If the operator provides fewer than 3 records, call out that the last-5 rolling median may not move immediately. If the operator provides 3 consistent shifted records, the rolling median usually moves immediately. If the operator provides 5 consistent shifted records, the last-5 window is effectively reset to the new runtime profile.

These records are intentional operator-approved baseline resets, not ordinary independent main-branch persistence. Mark them clearly with provenance fields so the HF history remains auditable.

Use this skill when a performance test fails for an intentional and reviewed reason, such as a torch/runtime/container upgrade that legitimately increases peak memory or changes timings. This is the performance equivalent of reseed-ssim-references: backup first, scope tightly, require explicit human approval, then upload reviewed accepted baseline records.

When to use

• A PR or main run failed the rolling performance comparison by more than the allowed regression threshold, and maintainers agree the shift is caused by an intentional runtime, dependency, hardware image, or benchmark environment change rather than a FastVideo logic regression.
• One or more shifted source result JSONs have been reviewed and accepted, and the operator wants to use those exact reviewed results to advance the rolling baseline.
• The source batch is internally consistent: no provided source JSON regresses against the batch median by more than the configured tolerance.

When not to use

• The benchmark failure might be a real code regression. Fix or investigate the code path first.
• The fixed benchmark thresholds in .buildkite/performance-benchmarks/tests/*.json are too low. Those are a separate gate from the rolling HF baseline and may need a code review change.
• There is no clear source run, commit, and rationale. Baseline history is a production signal; do not edit it without provenance.
• The provided source JSONs disagree materially with each other. Rerun or investigate instead of uploading a noisy reseed batch.

Inputs

Hardcoded defaults:

• HF repo: FastVideo/performance-tracking (HF_REPO_ID override is supported by the code, but use the default unless the user explicitly asks).
• Local sync root: /tmp/perf-tracking (PERFORMANCE_TRACKING_ROOT override is supported).
• Backup root: /tmp/performance_reseed_backup.
• Download scratch root for source artifact URLs: /tmp/performance_reseed_source.
• Baseline window: last 5 success=true records for the same (model_id, gpu_type).
• Reseed count: dynamic. Upload exactly one accepted seed record per validated source JSON.

Steps

1. Validate the target and source results

Normalize source_results to a list. If the user passes a single source_result, treat it as a one-element source_results list and report that a single record may not move the last-5 median immediately.

If any source result is a Buildkite artifact URL, download it first into a local scratch directory under /tmp/performance_reseed_source/ and use the downloaded JSON path for the rest of the workflow. If the agent cannot access the artifact because Buildkite authentication is missing, ask the user to download the artifact manually and provide the local path.

Prefer the normalized Buildkite artifact emitted by compare_baseline.py:

perf_reports/results/normalized_perf_*.json

That file is already in the HF tracking schema. Load each normalized JSON directly:

import json

with open(source_result, encoding="utf-8") as f:
    record = json.load(f)

Stop if any normalized record's model_id or gpu_type does not match the requested model_id and gpu_type.

The source records may have success: false when they came from failed rolling baseline comparisons. That is expected; only the reviewed reseed records become new success: true baseline records after explicit approval.

Sort validated source records by their original timestamp ascending before preparing the seed records. If a source timestamp is missing or unparsable, preserve input order for those records and print a warning. This makes the fresh reseed timestamps deterministic and makes it clear which records enter the last-5 window when more than 5 source JSONs are provided.

Check that HF_API_KEY is exported. The sync path may be public, but the upload path requires write access.

1a. Check source batch consistency

Before syncing or preparing uploads, reject source batches that are internally inconsistent. Use the same metric direction as compare_baseline.py:

• Lower is better: latency, memory, text_encoder_time_s, dit_time_s, vae_decode_time_s.
• Higher is better: throughput.

For each metric with at least two non-null source values:

1. Compute the source batch median.
2. For lower-is-better metrics, compute (source_value - batch_median) / batch_median.
3. For throughput, compute (batch_median - source_value) / batch_median.
4. Stop if any source record regresses against the batch median by more than max_intra_batch_regression.

Default max_intra_batch_regression to PERF_MAX_REGRESSION when set, otherwise 0.05. Print a table with per-source values, batch median, and worst intra-batch regression.

This check prevents uploading a mixed batch where one JSON is materially slower or faster than the others. If the batch fails this check, ask the user to provide a cleaner batch or explicitly investigate the variance. Do not silently drop outliers unless the user gives a concrete reviewed reason and a new source list.

1b. How to obtain source results from CI

The performance CI exports normalized source results for failed rolling baseline comparisons when compare_baseline.py ran. The preferred artifacts come from:

perf_reports/results/normalized_perf_*.json

The normal operator flow is:

1. Open the failed Buildkite performance job or several reruns of the same benchmark after the accepted environment shift.
2. Download the normalized_perf_*.json artifacts for the target benchmark.
3. Pass all reviewed local paths or artifact URLs as source_results.

Do not scrape the Markdown performance summary to reconstruct JSON. The normalized JSON artifacts are the only supported source of truth for reseed metrics and provenance. Raw fastvideo/tests/performance/results/perf_*.json artifacts are not accepted by this skill. If no normalized JSON artifact is present, that run is not a valid source for baseline reseeding.

2. Sync and back up existing HF records under /tmp

Use fastvideo/tests/performance/hf_store.py helpers directly. Do not use compare_baseline.py as a sync shortcut; on full main runs it can persist records, while this step must only fetch and back up existing history.

The sync command pattern is:

export PERFORMANCE_TRACKING_ROOT="${PERFORMANCE_TRACKING_ROOT:-/tmp/perf-tracking}"
export HF_REPO_ID="${HF_REPO_ID:-FastVideo/performance-tracking}"
PYTHONPATH=fastvideo/tests/performance python -c 'from hf_store import sync_from_hf; import os; sync_from_hf(os.environ["PERFORMANCE_TRACKING_ROOT"], strict=True)'

Then back up only the sanitized model directory under /tmp:

SHORT_COMMIT=$(git rev-parse --short=12 HEAD)
TIMESTAMP=$(date -u +%Y%m%d_%H%M%S)
MODEL_SAFE=$(PYTHONPATH=fastvideo/tests/performance python - <<'PY'
from hf_store import sanitize
print(sanitize("<model_id>"))
PY
)
BACKUP_DIR="/tmp/performance_reseed_backup/${TIMESTAMP}_${SHORT_COMMIT}_${MODEL_SAFE}"
mkdir -p "$BACKUP_DIR"
cp -R "${PERFORMANCE_TRACKING_ROOT}/${MODEL_SAFE}" "$BACKUP_DIR/" 2>/dev/null || true

Write provenance next to the backup:

cat > "$BACKUP_DIR/PROVENANCE.txt" <<EOF
model_id: <model_id>
gpu_type: <gpu_type>
source_results:
  - <source_result_1>
  - <source_result_2>
reseed_record_count: <len(source_results)>
max_intra_batch_regression: <threshold>
head_commit: $(git rev-parse HEAD)
timestamp_utc: $(date -u +%FT%TZ)
reason: <intent_rationale>
EOF

If the backup has no prior records, this is not a destructive reseed; it is a first baseline seed. Continue, but report that baseline history was empty.

3. Compute old baseline and candidate shift

Load the last 5 successful records for the target:

from hf_store import load_records_for_model

records = load_records_for_model(
    "/tmp/perf-tracking",
    "<model_id>",
    "<gpu_type>",
    last_n=5,
    successful_only=True,
)

Print a small table showing old medians, source batch medians, candidate medians after appending the proposed seed records, and source batch spread for:

• latency
• throughput
• memory
• text_encoder_time_s
• dit_time_s
• vae_decode_time_s

Also print how many successful old records exist. Make clear:

• 1 seed record usually does not move a last-5 median by itself.
• 3 consistent seed records usually move the last-5 median immediately.
• 5 consistent seed records effectively reset the last-5 window.
• The records are intentional approved baseline resets and must be labeled that way.

4. Confirm intent

Require an explicit confirmation phrase before preparing the upload:

About to RE-SEED performance baseline for <model_id> on <gpu_type>. This will upload <N> new success=true records to FastVideo/performance-tracking/<sanitize(model_id)>/, one per accepted source JSON.

Reason: <intent_rationale> Source results: <source_results> Reseed record count: <N> Max intra-batch regression: <threshold> Note: these records come from a reviewed source batch and are intended to move the rolling median to the accepted runtime profile. They are not ordinary main-branch persistence. HEAD: <git rev-parse --short=12 HEAD> Backup: <BACKUP_DIR>

Reply confirm performance reseed to proceed, anything else to abort.

Do not continue unless the user types exactly confirm performance reseed.

5. Create the accepted seed records

Create one seed record from each normalized source result. Do not copy the source JSON wholesale.

Infer the baseline field allowlist from all existing HF records for the target (model_id, gpu_type) after syncing, including both success=true and success=false records. Use the union of non-provenance keys present in those target records, preserving only fields that also exist in the normalized source record or are explicitly set by the reseed workflow. Always include model_id, timestamp, and success because the upload path and baseline loader depend on them. Always set timestamp to a fresh reseed timestamp and success to true. Do not include unrelated source-only fields that are absent from existing HF records.

Exclude existing provenance or operator metadata from the inferred baseline field allowlist. At minimum, exclude keys prefixed with baseline_reseed and any fields known to be local-only audit metadata.

If there are no previous HF records for the target model/GPU, fall back to this default baseline field list:

• model_id
• timestamp
• commit_sha
• gpu_type
• latency
• throughput
• memory
• text_encoder_time_s
• dit_time_s
• vae_decode_time_s
• success

Do not upload extra fields from the source artifact.

Optional provenance fields are allowed and useful:

• baseline_reseed: true
• baseline_reseed_reason
• baseline_reseed_source_result
• baseline_reseed_source_timestamp
• baseline_reseed_source_success
• baseline_reseed_batch_size
• baseline_reseed_batch_index
• baseline_reseed_operator
• baseline_reseed_max_intra_batch_regression

Use a fresh reseed timestamp for each seed record, not the original source result timestamp. This is required because load_records_for_model(..., last_n=5) keeps the last records after loading the model directory; stale filenames/timestamps may not enter the last-5 window and therefore may not move the median. Preserve the original source timestamp in baseline_reseed_source_timestamp.

Use the existing filename convention from _write_tracking_record(): <sanitize(timestamp)>_<sanitize(commit_sha)>.json under the sanitized model directory, but include a deterministic suffix such as _reseed_01, _reseed_02, and so on before .json so multiple records from the same batch do not overwrite each other.

If a source record already exists on HF with success=false, do not edit it in place unless the user explicitly asked for an audit-preserving correction. Prefer uploading new accepted seed records so failed history remains visible.

6. Pause before upload

Print:

• Backup directory path under /tmp.
• Prepared local record paths under PERFORMANCE_TRACKING_ROOT.
• HF paths that will receive the new records.
• Old rolling medians.
• Source batch medians, source batch spread, reseed count, and candidate medians.
• Rationale.

Ask the user to reply exactly upload. Anything else aborts and leaves the prepared records plus backup on disk.

7. Upload only the scoped records

Use the shared storage helper so the path and repo type match CI:

from hf_store import upload_record

upload_record("<local_record_path>", record, strict=True)

Run it once per prepared record. Each upload goes to:

FastVideo/performance-tracking/<sanitize(model_id)>/<record_filename>.json

Never bulk upload the whole tracking root. Never modify another model's directory in the same operation.

8. Report outcome and offer cleanup

Report:

• Uploaded HF paths.
• Backup directory under /tmp.
• Local tracking root, usually /tmp/perf-tracking.
• Old baseline window count and medians.
• Source batch medians, source batch spread, reseed count, and candidate medians.
• Expected effect based on reseed count.
• Any separate threshold changes still needed in .buildkite/performance-benchmarks/tests/*.json.

Include the intent_rationale in the PR or follow-up comment so reviewers can distinguish an accepted baseline shift from a hidden regression.

After the upload is verified, ask whether the user wants to clear temporary local state. Explain what each directory is for:

• PERFORMANCE_TRACKING_ROOT, usually /tmp/perf-tracking: local synced mirror of FastVideo/performance-tracking plus the prepared local seed records used for scoped upload.
• /tmp/performance_reseed_backup/<...>: local backup of the target model's pre-reseed HF history plus PROVENANCE.txt, kept so a bad reseed can be audited or corrected.
• /tmp/performance_reseed_source/<...> when used: downloaded source JSON artifacts from Buildkite URLs.

Ask:

Reseed succeeded. Do you want me to delete the local temp tracking mirror, source downloads, and reseed backup under /tmp? These files are local safety/audit artifacts only; HF already has the uploaded records.

Reply cleanup reseed temp to delete them, anything else to keep them.

Do not delete anything unless the user replies exactly cleanup reseed temp. If cleanup is requested, remove only the specific directories created for this reseed. Never remove unrelated /tmp contents.

Failure modes and handling

• HF_API_KEY unset. Stop before upload. Do not create an untracked process that appears to have reseeded but never reached HF.
• Source result does not match target. Stop. The wrong benchmark or GPU would poison a separate baseline.
• Source batch is internally inconsistent. Stop if any source regresses against the source batch median by more than max_intra_batch_regression. Ask for cleaner sources or a reviewed explanation before continuing.
• Too few source records to move the median. Continue only after making clear that one or two records may not immediately move the last-5 median.
• The source results are noisy or suspicious. Stop. Reseeding amplifies those measurements into the baseline, so they must be reviewed first.
• HF sync fails. Stop for destructive reseeds. A stale or empty sync can make the old baseline look missing.
• Candidate still violates fixed thresholds. Report that this skill only handles the rolling HF baseline; update benchmark JSON thresholds in code review if maintainers accept the new absolute limit.
• The user aborts at either confirmation. Leave the backup and prepared records on disk. Nothing should be uploaded.
• The user declines cleanup. Keep /tmp/perf-tracking, the source download directory if any, and /tmp/performance_reseed_backup/<...> in place for audit/debugging.
• A bad seed was uploaded. Use the backup and HF history to identify the uploaded file, then remove or supersede it with an explicitly reviewed corrective record. Do not silently rewrite unrelated history.

References

• .agents/skills/reseed-ssim-references/SKILL.md — safety pattern for intentional baseline replacement.
• fastvideo/tests/performance/compare_baseline.py — normalization, rolling median comparison, and persistence rules.
• fastvideo/tests/performance/hf_store.py — HF sync, record loading, sanitize(), and upload_record().
• fastvideo/tests/performance/test_inference_performance.py — source result JSON schema.
• .buildkite/performance-benchmarks/tests/*.json — fixed absolute benchmark thresholds, separate from rolling baseline comparisons.

Changelog

Re-seed SSIM Reference Videos

Purpose

Replace the existing SSIM reference videos for a single (test_file, model_id) pair on the HF dataset (FastVideo/ssim-reference-videos). This is destructive on HF — the old refs are overwritten — so the skill always:

1. Confirms intent with a one-liner the user has to type.
2. Downloads the existing refs as a local, timestamped backup.
3. Regenerates on Modal L40S (same code path that CI uses).
4. Pauses for a side-by-side eyeball of backup vs new mp4s.
5. Uploads with --force, scoped to the single --model-id.
6. Reminds the user to keep the backup until the PR lands.

Pairs with seed-ssim-references, which is the inverse (first-time seeding only, refuses to overwrite). Re-seeding is intentionally a separate, more ceremonial operation because mistakenly clobbering production refs is much harder to recover from than failing closed.

When to use

• An intentional code change (model port fix, kernel upgrade, attention backend swap, hyperparameter change in the test itself) has shifted the expected SSIM output and the existing refs no longer represent the new ground truth.
• A test is failing in CI for the right reason (the new code is correct, the old refs are stale).

When not to use

• A test is failing for the wrong reason (the port is buggy, not the refs). Fix the port; re-seeding hides the bug.
• A brand-new test that has no refs on HF yet. Use seed-ssim-references.
• "Just to clean up drift" without a concrete code change to point at. The PR description has to justify why refs changed; without a concrete change, there's nothing to write.

Inputs

Hardcoded:

• Modal GPU: L40S (matches CI; re-seeding from another SKU produces refs that L40S CI cannot match).
• Quality tier: default. full_quality is a separate, deliberate operation.
• HF repo: FastVideo/ssim-reference-videos (override via FASTVIDEO_SSIM_REFERENCE_HF_REPO).
• Device folder: L40S_reference_videos.

Prerequisites

The user has confirmed:

• modal CLI authenticated.
• hf CLI authenticated, and HF_API_KEY (or HUGGINGFACE_HUB_TOKEN / HF_TOKEN) exported with write access to FastVideo/ssim-reference-videos.
• The current branch's code is the change that motivated the re-seed (i.e. git rev-parse HEAD is the commit that intentionally invalidated refs).

Fail fast if any of these are missing.

Steps

1. Validate inputs and confirm intent

• Verify test_file exists and matches fastvideo/tests/ssim/test_*_similarity.py.

• Grep the file for *_MODEL_TO_PARAMS and assert model_id is one of its keys. If the file has only a single hardcoded model, accept that model id as the only valid value.

• Print the rationale and ask the user to type confirm reseed (not just y — make it deliberate):

  About to RE-SEED references for model <model_id> from test <test_file>. This will OVERWRITE existing refs on FastVideo/ssim-reference-videos/reference_videos/default/L40S_reference_videos/<model_id>/ after backup + Modal regen + eyeball.

  Reason: <intent_rationale> HEAD: <git rev-parse --short=12 HEAD>

  Reply confirm reseed to proceed, anything else to abort.

  Stop until the user types exactly confirm reseed. Anything else aborts with no side effects.

2. Back up existing refs

Always required. The backup is the only graceful path back if anything goes wrong later.

SHORT_COMMIT=$(git rev-parse --short=12 HEAD)
TIMESTAMP=$(date -u +%Y%m%d_%H%M%S)
MODEL_SAFE=$(echo "<model_id>" | tr '/' '_')
BACKUP_DIR="ssim_reseed_backup/${TIMESTAMP}_${SHORT_COMMIT}_${MODEL_SAFE}"
mkdir -p "$BACKUP_DIR"

hf download \
    --repo-type dataset FastVideo/ssim-reference-videos \
    --include "reference_videos/default/L40S_reference_videos/<model_id>/**" \
    --local-dir "$BACKUP_DIR"

mp4_count=$(find "$BACKUP_DIR" -name "*.mp4" | wc -l)
echo "Backup mp4 count: $mp4_count"
[ "$mp4_count" -gt 0 ] || {
    echo "ERROR: backup is empty for <model_id>. Either the model id is wrong"
    echo "or there are no existing refs (use seed-ssim-references instead)."
    exit 1
}

# Provenance — used in the PR description
cat > "$BACKUP_DIR/PROVENANCE.txt" <<EOF
test_file: <test_file>
model_id: <model_id>
head_commit: $(git rev-parse HEAD)
timestamp_utc: $(date -u +%FT%TZ)
reason: <intent_rationale>
EOF

If the hf download produces zero mp4s, abort — the user has either picked a non-existent model_id or there are no refs yet (in which case seed-ssim-references is the right tool).

3. Regenerate on Modal L40S

Mirror CI's exact env recipe so the regenerated refs are byte-comparable to what CI will produce on the same commit. Two differences from CI:

1. Pass the same env prefix CI uses (IMAGE_VERSION, BUILDKITE_*) — see .buildkite/pipeline.yml:1-3 and .buildkite/scripts/pr_test.sh:62-83. Without this, ssim_test.py:17-18 resolves a different GHCR image tag (default is latest, CI is py3.12-latest), and ssim_test.py:38-46 bakes different values into the image's frozen env block. Mismatched image or env is the most common source of SSIM drift between reseed and CI runs.
2. Do not pass --skip-reference-download. Letting the test fetch the existing refs and run the full SSIM compare gives "before" SSIM numbers for the PR description, and the test still produces the new mp4s regardless of whether the comparison passes or fails.

SUBDIR="${TIMESTAMP}_${SHORT_COMMIT}"

IMAGE_VERSION="py3.12-latest" \
BUILDKITE_REPO="$(git config --get remote.origin.url)" \
BUILDKITE_COMMIT="$(git rev-parse HEAD)" \
BUILDKITE_PULL_REQUEST="${BUILDKITE_PULL_REQUEST:-false}" \
modal run fastvideo/tests/modal/ssim_test.py \
    --git-repo="$(git config --get remote.origin.url)" \
    --git-commit="$(git rev-parse HEAD)" \
    --hf-api-key="$HF_API_KEY" \
    --test-files="<test_file>" \
    --sync-generated-to-volume \
    --generated-volume-subdir="$SUBDIR" \
    --no-fail-fast

Capture the printed modal volume get ... hint — its <SUBDIR> matches $SUBDIR and is needed for step 4. Capture the SSIM numbers from the test output (or from the JSON next to the generated mp4) for the PR description.

4. Download generated videos

modal volume get --force hf-model-weights \
    ssim_generated_videos/default/"$SUBDIR"/generated_videos \
    ./generated_videos_modal/default

After this, the new mp4s live at:

./generated_videos_modal/default/generated_videos/L40S_reference_videos/<model_id>/<backend>/<prompt>.mp4

--force is required when ./generated_videos_modal/default already exists from a prior run; safe on the first run too.

5. PAUSE — user reviews quality side-by-side

Print the diff and the comparison:

echo "=== File list diff (backup vs new) ==="
diff -u \
    <(find "$BACKUP_DIR/reference_videos/default/L40S_reference_videos/<model_id>" -name "*.mp4" \
        | sed "s|$BACKUP_DIR/reference_videos/default/L40S_reference_videos/||" | sort) \
    <(find ./generated_videos_modal/default/generated_videos/L40S_reference_videos/<model_id> -name "*.mp4" \
        | sed "s|./generated_videos_modal/default/generated_videos/L40S_reference_videos/||" | sort) \
    || true

echo
echo "=== SSIM numbers from this run (paste into PR) ==="
find ./generated_videos_modal/default/generated_videos/L40S_reference_videos/<model_id> -name "*_ssim.json" -exec cat {} \;

Then stop and tell the user:

Old refs backed up to $BACKUP_DIR. New videos in ./generated_videos_modal/default/generated_videos/L40S_reference_videos/<model_id>/.

Open both in a video player. Confirm the new videos:

1. Look correct (no obvious artifacts, no black/static frames).
2. Are intentionally different from the backup in the way described in <intent_rationale> (e.g. slight numerical drift only, not a different scene / different motion / corrupted output).

Reply upload to overwrite HF, anything else to abort. Aborting leaves the backup and new videos on disk for inspection — nothing on HF changes.

Do not proceed until the user types exactly upload. If they abort, leave everything on disk and stop here.

6. Copy into the local reference layout

Same as seed-ssim-references step 5:

python fastvideo/tests/ssim/reference_videos_cli.py copy-local \
    --quality-tier default \
    --device-folder L40S_reference_videos \
    --generated-dir ./generated_videos_modal/default/generated_videos/L40S_reference_videos

Result: fastvideo/tests/ssim/reference_videos/default/L40S_reference_videos/<model_id>/<backend>/<prompt>.mp4.

7. Upload with --force, scoped to --model-id

The --force flag is what makes this skill different from seed-ssim-references. Always pair it with --model-id so a typo cannot accidentally overwrite a neighboring model's refs.

python fastvideo/tests/ssim/reference_videos_cli.py upload \
    --quality-tier default \
    --device-folder L40S_reference_videos \
    --model-id "<model_id>" \
    --force

The CLI's overwrite guard refuses without --force; with --force it overwrites only files under reference_videos/default/L40S_reference_videos/<model_id>/.

8. Report success and retention guidance

Print:

• The HF path that was overwritten (<repo>/reference_videos/default/L40S_reference_videos/<model_id>/).

• The local backup directory path.

• The new SSIM numbers from step 5.

• This restore command, in case the PR review surfaces a problem after upload:

  python fastvideo/tests/ssim/reference_videos_cli.py upload \
      --quality-tier default \
      --device-folder L40S_reference_videos \
      --model-id "<model_id>" \
      --reference-dir "$BACKUP_DIR/reference_videos/default/L40S_reference_videos" \
      --force
  

• This PR-description checklist (see fastvideo/tests/ssim/AGENTS.md → Updating Reference Videos):

  1. Source commit that produced the new refs (HEAD at re-seed time).
  2. Test command and GPU SKU (L40S).
  3. Before/after SSIM numbers.
  4. The <intent_rationale> from step 1.
  5. A note that the backup lives at $BACKUP_DIR and should be retained until CI on the PR is green.

Do not auto-rerun the SSIM test — the user does that as part of the PR.

Failure modes and how to handle them

• HF_API_KEY unset. Stop before step 2.
• Backup is empty (zero mp4s). Stop before step 3 — the model id is wrong or the refs don't exist yet (use seed-ssim-references).
• Modal run fails before generation. No mp4s on the volume. Don't upload. Investigate the failure (test crash, OOM, partition exhaustion), fix, then retry from step 3. Backup is still intact.
• Quality regressed (visual or metric). User aborts at step 5. Backup retained. New videos retained on disk for inspection. Nothing on HF changed. Either fix the underlying code change or abandon the re-seed.
• User confirmed upload but later realized the new refs are wrong. Run the restore command from step 8 with the backup --reference-dir. This is exactly why the backup exists.
• Multi-model test, only one model is being re-seeded. Run the skill once per model id. The --model-id scope on upload guarantees the others are untouched.

Design notes (for future skill maintainers)

• Per-model_id scope is mandatory. The dataset houses many model subtrees; re-seeding the wrong one is hard to undo without backup.
• default tier only; full_quality is a separate, deliberate operation with different params and ~doubled runtime, and isn't what CI gates on.
• The skill deliberately does not pass --skip-reference-download to Modal so we get pre-reseed SSIM numbers for the PR. The seed-skill passes it because no refs exist yet; for re-seed, refs do exist and exposing the comparison is informative.
• The two-token confirm (confirm reseed, then upload) is intentional. Re-seeding is high-blast-radius and should not be one-keystroke.
• The backup directory is plain mp4s + PROVENANCE.txt. No HF metadata is preserved; the restore path uses reference_videos_cli.py upload --reference-dir which doesn't need it.

References

• .agents/skills/seed-ssim-references/SKILL.md — the first-time seed skill this one parallels. Read it for the Modal flag rationale shared between the two flows.
• fastvideo/tests/ssim/AGENTS.md — directory rules, including the PR expectations for any reference-video change (rationale, before/after SSIM, source commit/model/backend).
• fastvideo/tests/ssim/reference_videos_cli.py — copy-local, upload (with --model-id, --force), download. The overwrite guard at upload_reference_videos is the safety net this skill leans on.
• fastvideo/tests/modal/ssim_test.py — Modal orchestrator; --sync-generated-to-volume, --generated-volume-subdir, --skip-reference-download, --no-fail-fast.

Changelog

Seed SSIM Reference Artefacts (mp4 or pt)

Purpose

A brand-new SSIM test in fastvideo/tests/ssim/ fails forever until its reference artefacts exist on the HF dataset (FastVideo/ssim-reference-videos). The dataset hosts two kinds of artefacts side-by-side per (model_id, backend, prompt):

• .mp4 — pixel ground-truth for tests that call run_text_to_video_similarity_test / run_image_to_video_similarity_test in inference_similarity_utils.py. Compared via SSIM.
• .pt — pre-VAE latent bundle (fp16 full latent + fp32 slice + metadata + slice_spec + format_version) for tests that call run_text_to_latent_similarity_test in latent_similarity_utils.py. Compared via cosine distance on the slice and the full tensor.

This skill:

1. Detects which artefact type the test produces (pixel vs latent).
2. Runs the test on Modal's L40S pool to generate the artefacts.
3. Downloads them to the local repo via modal volume get.
4. Pauses so the user can verify quality:
   • mp4: visual eyeball in a video player.
   • pt: numerics dump (shape, slice stats, NaN/Inf check, metadata).
5. Uploads only the new test's files to HF, with a guard that refuses to overwrite anything already present.

The skill is run manually, once per new test. Before invoking it, the user has already sanity-tested the new test locally — it launches VideoGenerator and writes an artefact without crashing (the missing-reference assertion at the end is expected). The skill does not re-test locally; it goes straight to Modal L40S (which is what CI uses).

When to use

• A new test_*_similarity.py file has been added in fastvideo/tests/ssim/ and the HF dataset has no reference_videos/default/L40S_reference_videos/<model_id>/ subtree for it yet.

When not to use

• Regular CI runs — once refs exist, pytest fastvideo/tests/ssim/ downloads them automatically.
• Re-seeding an existing test. That requires --force on the upload step, and is out of scope here; treat as a separate, deliberate operation.

Inputs

The skill has one required input: the path to the new SSIM test file. Prompt the user for it if they didn't supply it.

Everything else is fixed:

• Modal runner GPU: L40S (hardcoded in fastvideo/tests/modal/ssim_test.py).
• Device folder: L40S_reference_videos.
• Quality tier: default (the tier CI runs). The full_quality tier is not seeded by this skill.
• HF repo: FastVideo/ssim-reference-videos (dataset).
• Multi-model test files: all model ids in *_MODEL_TO_PARAMS are seeded together; the Modal run produces one mp4 per (model, prompt, backend) and the upload scopes by --model-id, looping if there is more than one.

Prerequisites

The user has confirmed:

• modal CLI authenticated.
• HF_API_KEY (or HUGGINGFACE_HUB_TOKEN / HF_TOKEN) exported with write access to FastVideo/ssim-reference-videos.
• The test file runs locally end-to-end (generates an mp4; SSIM assertion failure due to missing reference is expected and fine).

Fail fast if the token env var is missing.

Steps

1. Ask for the test file, then detect artefact type

If the user didn't name one, ask: "Which SSIM test file do you want to seed references for? (e.g. fastvideo/tests/ssim/test_ltx2_similarity.py)".

Validate:

• Path exists and matches fastvideo/tests/ssim/test_*_similarity.py.
• File defines a *_MODEL_TO_PARAMS dict — grep it to extract the set of model ids. Those ids drive step 5.

Detect artefact type by inspecting the file's imports / helper call:

• latent (.pt) — file imports run_text_to_latent_similarity_test from fastvideo.tests.ssim.latent_similarity_utils (or any other helper that ends with _latent_similarity_test).
• pixel (.mp4) — file imports run_text_to_video_similarity_test / run_image_to_video_similarity_test from fastvideo.tests.ssim.inference_similarity_utils, OR uses the legacy custom-inline helper pattern (see test_gamecraft, test_longcat, etc.). Default to pixel when both heuristics fail.

Record ARTEFACT_TYPE ∈ {pixel, latent} for use in step 4. Steps 2, 3, 5, and 6 are artefact-type-agnostic — _iter_reference_files, copy_generated_to_reference, and upload_reference_videos already walk both .mp4 and .pt (see reference_videos_cli.py).

If either check fails, stop and tell the user what's wrong.

2. Run the test on Modal L40S

Pick a subdir name so repeated runs don't collide:

SHORT_COMMIT=$(git rev-parse --short=12 HEAD)
TIMESTAMP=$(date -u +%Y%m%d_%H%M%S)
SUBDIR="${TIMESTAMP}_${SHORT_COMMIT}"

Then launch the Modal run. The IMAGE_VERSION and BUILDKITE_* env-prefix must match what CI exports in .buildkite/scripts/pr_test.sh, otherwise fastvideo/tests/modal/ssim_test.py resolves a different GHCR image tag (default is latest, CI is py3.12-latest) and bakes different values into the image's frozen env block (ssim_test.py:17-18, 38-46). Mismatched image or env produces SSIM drift that doesn't show up until the same commit runs in CI.

IMAGE_VERSION="py3.12-latest" \
BUILDKITE_REPO="$(git config --get remote.origin.url)" \
BUILDKITE_COMMIT="$(git rev-parse HEAD)" \
BUILDKITE_PULL_REQUEST="${BUILDKITE_PULL_REQUEST:-false}" \
modal run fastvideo/tests/modal/ssim_test.py \
    --git-repo="$(git config --get remote.origin.url)" \
    --git-commit="$(git rev-parse HEAD)" \
    --hf-api-key="$HF_API_KEY" \
    --test-files="<test_file>" \
    --sync-generated-to-volume \
    --generated-volume-subdir="$SUBDIR" \
    --skip-reference-download \
    --no-fail-fast

Env prefix rationale (parity with CI; see .buildkite/pipeline.yml:1-3 and .buildkite/scripts/pr_test.sh:62-83):

• IMAGE_VERSION=py3.12-latest: pins the Modal image tag to the same one CI uses. The published py3.12-latest and latest tags point at Python 3.12 / CUDA 12.6.3 / cu126; py3.12-cuda12.6.3-latest is the explicit alias for the same image. CUDA 13 / cu130 is available under the explicit py3.12-cuda13.0.0-latest tag. This tag policy comes from infra-build-image.yml; the unparameterized docker/Dockerfile build itself still defaults to CUDA 13 / cu130.
• BUILDKITE_REPO/BUILDKITE_COMMIT/BUILDKITE_PULL_REQUEST: mirror what Buildkite exports. ssim_test.py:38-46 bakes these into the image's .env(...) block; mismatched values can perturb in-container code paths that branch on PR-vs-non-PR. false for BUILDKITE_PULL_REQUEST matches Buildkite's "non-PR build" sentinel.

Flag rationale:

• --skip-reference-download: no refs exist yet, so conftest must not try to pull them.
• --no-fail-fast: lets the test finish generation before _assert_similarity raises FileNotFoundError: Reference video folder does not exist. The expected failure is what we want — the mp4 has already been written.
• --sync-generated-to-volume + --generated-volume-subdir: copies the generated mp4s to the hf-model-weights Modal volume under ssim_generated_videos/default/<SUBDIR>/generated_videos/ so we can pull them locally.

The Modal run will end with a nonzero exit (expected) and print a modal volume get hf-model-weights ssim_generated_videos/default/<SUBDIR>/generated_videos ./generated_videos_modal/default command. Capture that <SUBDIR> — you need it for step 3.

3. Download generated videos locally

modal volume get --force hf-model-weights \
    ssim_generated_videos/default/"$SUBDIR"/generated_videos \
    ./generated_videos_modal/default

--force is required when the parent ./generated_videos_modal/default already exists; without it, modal volume get errors with [Errno 21] Is a directory. Safe to pass on the first run too.

After this, the mp4s live at ./generated_videos_modal/default/generated_videos/L40S_reference_videos/<model_id>/<backend>/<prompt>.mp4. The extra generated_videos/ level comes from the volume layout in _sync_generated_videos_to_volume (ssim_test.py) — the command copies <repo>/fastvideo/tests/ssim/generated_videos/<tier> to ssim_generated_videos/<tier>/<SUBDIR>/generated_videos/, and modal volume get preserves that trailing generated_videos/ segment.

4. PAUSE — user reviews quality

Type-aware verification.

For ARTEFACT_TYPE = pixel — list the downloaded mp4s and ask the user to open them in a video player:

"Generated videos downloaded to ./generated_videos_modal/default/generated_videos/L40S_reference_videos/. Please open them and confirm the quality looks correct. Reply upload to continue, or anything else to abort."

For ARTEFACT_TYPE = latent — .pt files are not human-watchable. Print a numerics dump for each .pt so the user can sanity-check shape, distribution, and metadata:

import torch
from pathlib import Path
ROOT = Path("./generated_videos_modal/default/generated_videos/L40S_reference_videos")
for p in sorted(ROOT.rglob("*.pt")):
    d = torch.load(p, map_location="cpu", weights_only=False)
    s = d["expected_slice"]
    L = d["latent"].float()
    print(f"=== {p.relative_to(ROOT)} ===")
    print(f"  format_version: {d['format_version']}")
    print(f"  shape:          {d['shape']}")
    print(f"  dtype_original: {d['dtype_original']}")
    print(f"  slice_spec:     {d['slice_spec']}")
    print(f"  slice  shape={tuple(s.shape)}  mean={s.mean():+.4f}  std={s.std():.4f}  min={s.min():+.4f}  max={s.max():+.4f}")
    print(f"  latent shape={tuple(L.shape)}  mean={L.mean():+.4f}  std={L.std():.4f}  min={L.min():+.4f}  max={L.max():+.4f}")
    print(f"  finite: latent NaN={torch.isnan(L).any().item()} Inf={torch.isinf(L).any().item()}; "
          f"slice NaN={torch.isnan(s).any().item()} Inf={torch.isinf(s).any().item()}")
    print(f"  metadata: {d['metadata']}\n")

Sanity criteria:

• format_version == 1 (matches LATENT_REFERENCE_FORMAT_VERSION).
• shape matches what the model produces (e.g. LTX-2 distilled = [1, 128, T_lat, H_lat, W_lat]; Stable Audio Open 1.0 = [1, 64, 1024]).
• slice_spec.kind matches a registered kind (corner_3x3_first_frame for video, audio_first_8_timesteps for audio).
• No NaN/Inf. mean ≈ 0, std ≈ 1 (denoised latents stay close to the initial Gaussian distribution; very wide deviations suggest numerical drift).
• metadata.prompt matches the test's prompt.

Then ask:

"Numerics look right? Reply upload to continue, or anything else to abort."

Do not proceed until the user explicitly says upload. If they abort, leave everything on disk so they can inspect further — no cleanup.

5. Copy into the local reference layout

Scoped copy — only the new test's artefacts. Single command works for both artefact types because _iter_reference_files walks .mp4 and .pt:

python fastvideo/tests/ssim/reference_videos_cli.py copy-local \
    --quality-tier default \
    --device-folder L40S_reference_videos \
    --generated-dir ./generated_videos_modal/default/generated_videos/L40S_reference_videos

(The --generated-dir points at the device-folder root inside the downloaded tree; copy-local walks all <model>/<backend>/*.{mp4,pt} underneath it. Since the Modal run was scoped to a single test file via --test-files, only that test's model(s) are present — so the copy is implicitly per-test.)

Result for pixel: fastvideo/tests/ssim/reference_videos/default/L40S_reference_videos/<model_id>/<backend>/<prompt>.mp4. Result for latent: same path with .pt extension.

6. Upload to HF — scoped per model_id, with overwrite guard

For each <model_id>:

python fastvideo/tests/ssim/reference_videos_cli.py upload \
    --quality-tier default \
    --device-folder L40S_reference_videos \
    --model-id "<model_id>"

The upload command:

• Uploads only reference_videos/default/L40S_reference_videos/<model_id>/.
• Refuses if any file already exists at that path on HF (this is the guard — seeding a new test should never clobber existing refs). To override, the user must re-run with --force. If the guard fires, stop and report exactly which files exist; do not silently --force.

Reads the HF token from HF_API_KEY / HUGGINGFACE_HUB_TOKEN / HF_TOKEN.

7. Report success

List what was uploaded (paths in repo) and remind the user to push any related code changes. Do not auto-verify by re-running Modal — the user can run pytest fastvideo/tests/ssim/<test_file> later to confirm end-to-end; it will auto-download the refs they just uploaded.

Failure modes and how to handle them

• HF_API_KEY unset. Stop before step 2. The Modal run needs it (passed via --hf-api-key), and step 6 needs it for upload. If the user ran hf auth login instead of exporting an env var, read the cached token via huggingface_hub.get_token() and forward it to Modal as --hf-api-key="$CACHED_TOKEN".
• Modal run fails before generation. No artefacts on the volume — nothing to download. Fix the test locally (pytest fastvideo/tests/ssim/<test_file>) and retry from step 2.
• ./generated_videos_modal/default/L40S_reference_videos/ missing after modal volume get. The run didn't produce artefacts (most likely the test crashed before writing, or REQUIRED_GPUS exceeded the partition capacity — see Modal logs).
• Latent test crashed with FSDP / inference_mode error (RuntimeError: Inference tensors do not track version counter). The test must pass init_kwargs_override={"use_fsdp_inference": False} when sp_size == 1 — see test_stable_audio_similarity.py for the pattern. Fix in the test, push, retry.
• Upload guard fires (files already exist). The test name / model id collides with something already on HF. Verify the user actually wants to replace existing refs; if so, re-run the upload with --force. If not, rename the model id in *_MODEL_TO_PARAMS and re-seed.
• Quality looks wrong in step 4. Abort. The artefacts stay on disk for inspection. The fix is usually in the test's params (resolution, steps, seed) — edit the test, then re-run the skill.
  • For latent: also check slice_spec.kind matches the latent rank (corner_3x3_first_frame requires 5-D, audio_first_8_timesteps requires 3-D); a rank/kind mismatch raises in _extract_expected_slice.

Design notes (for future skill maintainers)

• The skill deliberately runs on Modal, not locally, because the CI runner is L40S. Seeding from a different GPU SKU produces refs that CI's L40S runs can't match (pixel SSIM drifts across SKUs; latent cosine has tighter cross-SKU bf16 drift but the configured tolerances assume same-SKU seed → same-SKU verify).
• The skill is default-tier only. full_quality refs are seeded by a separate, deliberate operation — they double runtime and aren't what CI gates on.
• The overwrite guard in reference_videos_cli.py upload is default-on specifically because this skill exists. Re-seeding is a distinct operation that requires explicit --force.
• Both artefact types share the same Modal flow: the orchestrator sets --skip-reference-download + --no-fail-fast, runs pytest, the test's helper writes the artefact (.mp4 via imageio for pixel, save_latent_reference → torch.save for latent) BEFORE the missing-reference assertion raises. _sync_generated_videos_to_volume in ssim_test.py does a shutil.copytree of the whole generated_videos/ tree, picking up .mp4, .pt, and the *_ssim.json / *_latent.json metric files alongside.

References

• fastvideo/tests/modal/ssim_test.py — Modal orchestrator; see --sync-generated-to-volume, --generated-volume-subdir, --skip-reference-download, --no-fail-fast.
• fastvideo/tests/ssim/reference_videos_cli.py — copy-local, upload (with --model-id, --force), download, ensure subcommands. Extension allowlist is REFERENCE_EXTENSIONS = VIDEO_EXTENSIONS + LATENT_EXTENSIONS (.pt).
• fastvideo/tests/ssim/README.md — reference layout, HF repo conventions.
• fastvideo/tests/ssim/inference_similarity_utils.py — pixel helpers (run_text_to_video_similarity_test, run_image_to_video_similarity_test, build_init_kwargs).
• fastvideo/tests/ssim/latent_similarity_utils.py — latent helper (run_text_to_latent_similarity_test), slice spec dispatch (_extract_expected_slice), reference schema (save_latent_reference / load_latent_reference), LATENT_REFERENCE_FORMAT_VERSION.

Changelog