cvpr-artifact-evaluation
GitHub指导为CVPR论文打包代码、模型或数据集，涵盖匿名评审期内的密封补充材料规范（如去除隐私信息、禁用外链）及录用后的发布要求。提供构建可复现环境、清理元数据及编写REPRODUCE.md的最佳实践，确保评审者能快速验证结果。
Trigger Scenarios
Install
npx skills add brycewang-stanford/Awesome-Journal-Skills --skill cvpr-artifact-evaluation -g -y
SKILL.md
Frontmatter
{
"name": "cvpr-artifact-evaluation",
"description": "Use when packaging code, models, datasets, or demo videos for a CVPR paper at either review time or release time, covering anonymous supplement packaging under the external-link ban, the dataset-release-by-camera-ready rule, model-weight and license decisions, and making a vision artifact runnable by a skeptical stranger."
}
CVPR Artifact Evaluation
CVPR has no badge-granting artifact committee; the "evaluation" of your artifacts is done informally, twice, by two different audiences. At review time, reviewers poke at an anonymous supplement while deciding whether to trust your tables. After acceptance, the entire field — 2026 drew 44,011 authors alone — decides whether your method becomes a baseline or a citation nobody reproduces. Package for both audiences deliberately.
Review-time artifacts: sealed-box rules
Everything must ride inside the uploaded supplement, because the 2026-verified policy bans external links that expand content or subvert review. That kills the usual anonymous-repo-link workflow many venues tolerate. Practical consequences:
- Code goes in the supplement archive, scrubbed: no
.githistory, no usernames in paths, no institutional cluster hostnames, no wandb entity names in configs. - Model weights usually exceed sane archive sizes; ship configs + training commands + exact dependency pins instead, and say which checkpoints will be released.
- Demo videos are first-class CVPR evidence (see
cvpr-supplementary) but strip container metadata — video files carry author tags more often than PDFs do. - The 2026 supplementary size cap was not verifiable at check time (待核实); design the package assuming a strict cap, then confirm on the current page.
# Build a clean review artifact from a working repo
git archive --format=tar HEAD | tar -x -C /tmp/artifact # no .git, no untracked junk
cd /tmp/artifact
grep -rniE "$(whoami)|<lab-name>|<cluster-host>|wandb\.(ai|entity)" . | head # identity scan
pip-compile --quiet requirements.in # pin, don't approximate
zip -r ../supplement_code.zip . -x '*.ckpt' '*.pth' # weights out, recipe in
The runnable-by-a-stranger bar
A reviewer who opens your archive gives it minutes, not an afternoon. One entry point, one config, one expected number:
| Artifact layer | Minimum viable | Gold standard |
|---|---|---|
| Environment | pinned requirements.txt + CUDA/driver note |
Dockerfile reproducing the CRF hardware row |
| Inference | script + 5 sample images + expected outputs | notebook rendering figure-quality results |
| Training | full config + command + seed | resumable run with logged curves |
| Evaluation | script that recomputes one main-table row | all-tables harness with dataset download stubs |
| Data | loader + split files + provenance note | checksummed archive or release plan with license |
The single highest-leverage file is REPRODUCE.md mapping each table/figure in the
paper to one command. It converts a suspicious reviewer into a supportive one faster
than any rebuttal sentence.
Release-time artifacts: promises come due
Two clocks start at acceptance. First, the verified dataset clause: a dataset claimed as a contribution must be public by the camera-ready deadline — hosting, license, and consent scrubbing included. Second, the softer but real credibility clock: the gap between "code coming soon" in the README and actual code is measured publicly at CVPR scale.
Release decisions to make explicitly rather than by default:
- License: weights and data need one (CC-BY-SA? research-only? non-commercial?); "no license" means legally unusable for the industrial half of the CVPR audience.
- Weights: which checkpoints — final only, or the ablation grid reviewers saw?
- Provenance: for scraped or generated data, document sources and filtering; for human data, document consent posture. Vision datasets attract scrutiny years later.
- Benchmarks: if you release an evaluation, freeze the metric code and version it — silent metric fixes fork the leaderboard.
Weights and hosting decisions
Model releases have their own failure modes at vision scale:
- Host durability: lab web servers die with graduations. Prefer archival or platform hosting (institutional repositories, model hubs) with a checksum published in the README; the CVF open-access page will outlive your URL, so choose hosts with the same life expectancy.
- Checkpoint provenance: name each released checkpoint after the paper's table row
it reproduces (
table2_row4_vitb.pth), and state which exact code commit evaluates it to the printed number. - Safety posture for generative models: decide before release what you ship — weights, LoRA deltas, or inference-only API — and write the model card's misuse paragraph yourself rather than letting the first misuse write it for you.
- Version pinning at release: tag the repo at camera-ready (
v1.0-cvpr) so later refactors don't silently break the commands printed in the paper's supplement.
Anti-patterns with CVPR-specific cost
- A project-page URL in the submission "for videos" — that is the banned external link, not a convenience.
- Supplement code that hard-codes
/home/<name>/data— an anonymity leak and proof nobody ran it elsewhere. - Claiming a dataset contribution, then releasing it months after camera-ready — a verified policy breach, not just bad manners.
- Releasing training code without evaluation code: at a benchmark-driven venue the evaluation is the artifact people actually reuse.
After release: the maintenance tail
A used artifact generates issues, and the field reads your issue tracker. Budget a
small maintenance window post-conference: pin the environment against dependency rot,
answer the first wave of "can't reproduce Table 2" issues (usually environment
mismatches — point to the Dockerfile), and keep a RESULTS.md of community
reproductions. Six months of light maintenance is what turns a CVPR paper into the
baseline the next cycle's papers must cite.
Reverify each cycle
- Supplement size/format caps and whether anonymized-repo links are (dis)allowed — the external-link wording can shift by year.
- The dataset-release clause wording.
- Any new artifact, checklist, or reproducibility fields on the OpenReview form.
Output format
[Artifact stage] review-supplement / camera-ready-release
[Identity scan] clean / hits: <files>
[Runnability] entrypoint · pins · expected-output map present?
[Weights & data plan] <ship now / release plan + license>
[Dataset clause] n/a / due at camera-ready: <status>
[Gaps] <ordered fixes>
Version History
- 9f86f09 Current 2026-07-19 14:41
