sigmod-artifact-evaluation
GitHub指导作者为SIGMOD论文准备代码和数据以参与ARI计划，涵盖徽章申请、HotCRP注册及评估标准。提供打包规范、硬件声明建议及避免常见失败点（如链接失效、文档缺失）的实操指南。
Trigger Scenarios
Install
npx skills add brycewang-stanford/Awesome-Journal-Skills --skill sigmod-artifact-evaluation -g -y
SKILL.md
Frontmatter
{
"name": "sigmod-artifact-evaluation",
"description": "Use when preparing a SIGMOD paper's code and data for the Availability & Reproducibility Initiative (ARI), covering the post-acceptance opt-in, HotCRP artifact registration, the Artifacts Available \/ Artifacts Evaluated \/ Results Reproduced badges, evaluator criteria, and the Best Artifact award path."
}
SIGMOD Artifact Evaluation
SIGMOD runs a named, long-standing artifact program: the Availability &
Reproducibility Initiative (ARI) at reproducibility.sigmod.org. It is
optional, happens after acceptance, and is decoupled from the accept/
reject decision — but its badges are embedded into the paper's PDF in the ACM
Digital Library, so the payoff is permanent and public. Recent editions
registered artifacts through a dedicated HotCRP site (e.g.,
sigmod25ari.hotcrp.com); confirm the current edition's site and dates
before promising a timeline.
The badge ladder
| Badge | What evaluators must confirm | Typical blocker |
|---|---|---|
| Artifacts Available | Code, data, scripts, notebooks reachable at a stable public location | Link rot; "email us for the dataset" |
| Artifacts Evaluated | Package is exercisable and well documented | Undocumented cluster assumptions |
| Results Reproduced | Key results of the paper independently regenerated | Figures that need hand-tuned steps |
Aim explicitly at one rung. A package engineered for Results Reproduced looks different from one that merely publishes source: it names which figures and tables constitute the "key results" and drives each one end to end.
What ARI evaluators grade
The initiative's stated criteria are coverage (how much of the paper the artifact backs), ease of reproducibility (how little effort a rerun takes), flexibility (can parameters, workloads, and datasets be varied), and portability (does it run beyond the authors' exact machine). Database artifacts fail these in predictable ways:
- Coverage: the artifact rebuilds microbenchmarks but not the headline end-to-end comparison against the competing engine.
- Ease: a 40-step README where a driver script should be.
- Flexibility: scale factors and thread counts hard-coded into binaries.
- Portability: kernel-version, NUMA-layout, or GPU assumptions that only hold on the lab machine; no container or VM escape hatch.
Hardware honesty
Data-management experiments often need big machines. ARI evaluations have historically accommodated this via cloud credits or author-provided access in some editions — but the current edition's policy must be checked, not assumed. Regardless of policy:
- State minimum and recommended hardware in the README's first screen.
- Provide a reduced-scale mode (smaller scale factor, fewer threads) that preserves every qualitative trend, and say which absolute numbers will differ from the paper.
- Wall-clock estimates per experiment; evaluators budget time, and an unannounced 30-hour run is how evaluations stall.
Packaging pattern that passes
artifact/
README.md # claims map: Fig/Table -> script -> expected output
LICENSE
Dockerfile # or VM image link; pinned OS + dependency versions
data/fetch.sh # pulls public datasets by checksum; sizes stated
run_all.sh # full reproduction, prints per-step ETA
run_small.sh # laptop-scale variant, trends preserved
experiments/
fig7_throughput/ # one directory per paper artifact, self-contained
tab3_latency/
plot/ # regenerates the exact PDF figures from raw logs
The claims map is the single highest-leverage file: a table from paper artifact to command to expected output, with tolerances ("within 10% on different hardware; ordering of systems preserved").
Incentives beyond the badge
SIGMOD confers a Best Artifact award through the initiative, and reproducibility reports from past editions are themselves published in the ACM DL — meaning strong artifacts earn citable recognition. For an engine or index paper, the artifact also becomes the de facto baseline implementation future papers must compare against, which compounds citations for years.
Questions evaluators ask that READMEs rarely answer
- Which exact figure numbers count as the paper's key results, and where is that stated?
- What should I see on screen when a run succeeds — and what does a known benign warning look like, so I don't abort a healthy run?
- How much disk does the full dataset expand to after decompression?
- Can steps be resumed after a failure, or does every retry start from data fetch?
- Which numbers are hardware-sensitive, and how much drift is acceptable before I should suspect a real problem?
- Who do I contact (anonymity no longer applies) if a step fails, and how fast do authors respond during the evaluation window?
Answer all six in the README's first two screens and the evaluation's most common failure mode — silent stall and abandonment — mostly disappears.
Sequencing with the paper lifecycle
- During review: keep the anonymized package coherent (see
sigmod-supplementary); it becomes the ARI seed. - At camera-ready: publish and tag the release matching published numbers
(see
sigmod-camera-ready). - At ARI registration: submit the tagged version via the edition's HotCRP, then leave it frozen — evaluate-time pushes create version skew.
- After badging: keep the archive stable; the badge points at the DL record forever.
Output format
[Target badge] Available / Evaluated / Results Reproduced
[Claims map] Fig/Table -> script coverage, gaps listed
[Criteria audit] coverage / ease / flexibility / portability findings
[Hardware plan] full-scale needs, reduced-scale mode, runtimes
[Registration] ARI edition site + dates confirmed or 待核实
[Fix queue] ordered work before artifact submission
Version History
- 9f86f09 Current 2026-07-19 17:37
