artifact-evaluation-prep
GitHub用于准备研究制品包以通过会议评估、复现性审查或发布。提供安装指南、复现命令、Docker环境检查及数据打包，确保评审者能重现论文主张，区别于通用代码发布。
Trigger Scenarios
Install
npx skills add NeverSight/learn-skills.dev --skill artifact-evaluation-prep -g -y
SKILL.md
Frontmatter
{
"name": "artifact-evaluation-prep",
"description": "Prepare a research artifact package for conference artifact evaluation, reproducibility review, badges, supplementary material, or post-acceptance artifact release. Use this skill whenever the user needs install instructions, reviewer-facing reproduction commands, Docker or environment checks, data\/checkpoint packaging, hardware\/runtime estimates, anonymized or public artifact metadata, artifact evaluation forms, or a claim-to-artifact reproducibility audit for ML\/AI venues.",
"allowed-tools": "Read, Write, Edit, Bash, Glob, WebSearch, WebFetch",
"argument-hint": "[project-dir] [--venue <venue>] [--mode audit|package|instructions|smoke-test]"
}
Artifact Evaluation Prep
Prepare a paper's code, data, checkpoints, scripts, and instructions so an external artifact reviewer can reproduce the paper-facing claims with minimal ambiguity.
Use this skill when:
- a venue requires or offers artifact evaluation, reproducibility badges, or artifact appendices
- the user needs reviewer-facing install, quickstart, demo, or reproduction instructions
- a camera-ready or accepted paper needs an artifact package handoff
- code, data, checkpoints, models, Docker images, or external services must be packaged
- runtime, hardware, random seeds, expected outputs, or troubleshooting notes need to be made explicit
- claims in the paper need to be mapped to runnable scripts or released artifacts
Do not use this skill as a general code-release skill. Use release-code for public repository hygiene, licensing, CITATION files, tags, and GitHub releases. Use this skill for reviewer-facing artifact execution and claim reproduction.
Pair this skill with:
camera-ready-finalizerto recover accepted-paper obligations and final claim/evidence staterelease-codeto prepare public repository hygiene after artifact obligations are clearreproducibility-auditwhen environment, data, or execution drift needs a broader auditrun-experimentfor generating or testing reproduction commandsfigure-results-reviewwhen artifact outputs must match paper figures or tablescitation-auditwhen artifact metadata cites datasets, code, or prior artifactsresearch-project-memorywhen artifact status, blockers, and reviewer-facing instructions should persist
Skill Directory Layout
<installed-skill-dir>/
├── SKILL.md
└── references/
├── artifact-audit.md
├── memory-writeback.md
├── package-manifest.md
├── report-template.md
└── reviewer-instructions.md
Progressive Loading
- Always read
references/artifact-audit.md,references/package-manifest.md, andreferences/reviewer-instructions.md. - Read
references/report-template.mdbefore writing a saved artifact evaluation report. - Read
references/memory-writeback.mdwhen the project hasmemory/, component.agent/folders, or the user asks for persistent state. - If venue rules matter, verify current official artifact evaluation instructions before asserting deadlines, badge names, anonymity rules, upload fields, page limits, or required formats.
Core Principles
- Artifact evaluation is a reviewer workflow, not just a code dump.
- The artifact must reproduce the paper's important claims at an acceptable cost, or clearly document what it cannot reproduce.
- Prefer one reliable quickstart and one complete reproduction path over many fragile commands.
- Every command should state expected runtime, hardware, input, output, and success criteria.
- Package only redistributable data, checkpoints, and dependencies; document restricted assets precisely.
- Keep anonymity, licensing, and external-service assumptions explicit.
- Treat smoke tests as required. An untested instruction file is not an artifact package.
Step 1 - Recover Evaluation Context
Collect:
- venue and artifact evaluation track, if known
- official artifact instructions, badge criteria, anonymity policy, and upload mechanism
- accepted or submitted paper, appendix, supplementary material, and checklist
- code repository, commit hash, branches, and worktrees
- datasets, checkpoints, pretrained models, generated outputs, and external dependencies
- hardware expectations: CPU/GPU type, memory, disk, runtime, network access
- paper claims, figures, tables, and experiments that the artifact should support
- constraints: private data, license limits, large files, cloud dependencies, nondeterminism, or reviewer time budget
If no venue is specified, produce a venue-agnostic artifact package but mark venue-specific fields as unresolved.
Step 2 - Map Claims to Artifact Paths
For each paper-facing claim or result, record:
- claim or result ID
- paper location
- script, notebook, config, or command that supports it
- input data or checkpoint
- expected output file, metric, table, or figure
- approximate runtime and hardware
- deterministic tolerance or expected variance
- reviewer priority: quickstart, core, optional, or not reproducible in package
Do not imply full reproducibility if only a smoke test or cached output is provided.
Step 3 - Build the Artifact Manifest
Read references/package-manifest.md.
Create or update a manifest that lists:
- repository URL or archive path
- exact commit, tag, or checksum
- directory layout
- environment files and Docker images
- data and checkpoint locations
- reproduction scripts and configs
- expected generated outputs
- license and citation metadata
- known limitations and unsupported claims
Prefer small, stable names such as ARTIFACT.md, REPRODUCE.md, or docs/artifact_evaluation.md unless the venue requires a specific filename.
Step 4 - Write Reviewer Instructions
Read references/reviewer-instructions.md.
Provide:
- setup commands
- quick smoke test under a short runtime budget
- core reproduction commands for main paper claims
- expected outputs and how to compare them with the paper
- troubleshooting for common failures
- hardware, storage, network, and time requirements
- contact policy or anonymous support channel if allowed
- limitations and optional extended runs
Instructions should be copy-pasteable and should not require the reviewer to infer hidden paths or environment variables.
Step 5 - Smoke Test the Artifact
When allowed by the user and environment, run at least:
- environment creation or dependency resolution
- import or CLI sanity check
- quickstart command
- one representative data/checkpoint load
- one expected-output comparison
If commands are too expensive, record the exact reason and create a minimal substitute test.
Step 6 - Handle Packaging Risks
Audit:
- anonymization vs public release state
- licenses for code, data, pretrained weights, and third-party assets
- large-file strategy and checksums
- private paths, credentials, API keys, and machine-specific assumptions
- random seeds and nondeterminism
- version pinning and dependency conflicts
- reviewer time budget and failure recovery
Route public release issues to release-code; route environment drift to reproducibility-audit if available.
Step 7 - Write the Artifact Evaluation Report
Read references/report-template.md.
If saving to a project and no path is given, use:
docs/submission/artifact_evaluation_prep_YYYY-MM-DD.md
The report must include:
- readiness decision
- blocking issues
- claim-to-artifact map
- package manifest summary
- smoke-test status
- reviewer instruction status
- risks, limitations, and reviewer-facing caveats
- handoff to release, camera-ready, or memory
Step 8 - Write Back to Project Memory
Read references/memory-writeback.md when memory exists.
Update artifact status, reproduction commands, blockers, claim support, release actions, and final handoff notes without copying full command logs into memory.
Final Sanity Check
Before finalizing:
- every important paper claim is either reproducible, smoke-tested, cached with explanation, or explicitly out of scope
- quickstart instructions have expected outputs and runtime
- hardware, data, checkpoints, licenses, and anonymity state are clear
- package paths and links are stable
- reviewer-facing failure modes are documented
- public-release and camera-ready obligations are routed
- project memory records artifact readiness and open blockers
Version History
- e0220ca Current 2026-07-05 21:34
