sigir-reproducibility
GitHub用于增强SIGIR论文可复现性或准备Reproducibility Track投稿，涵盖固定检索管线、种子与方差、文档版本记录及复现研究结构。
Trigger Scenarios
Install
npx skills add brycewang-stanford/Awesome-Journal-Skills --skill sigir-reproducibility -g -y
SKILL.md
Frontmatter
{
"name": "sigir-reproducibility",
"description": "Use when strengthening the reproducibility of a SIGIR paper or preparing a SIGIR Reproducibility track submission — pinning the retrieval pipeline, seeds and variance for neural rankers, documenting collection versions and index settings, and structuring a reproduction study of published IR results with honest divergence analysis."
}
SIGIR Reproducibility
Reproducibility is unusually load-bearing at SIGIR for two reasons. First, the venue runs a dedicated Reproducibility track (its own track in 2026, split out of the former combined Resource & Reproducibility track — budget and dates 待核实 on the current page), so reproduction studies are publishable first-class work. Second, the field's own literature documents how often reported IR gains fail to replicate under matched tuning — reviewers of regular papers therefore read reproducibility signals as a proxy for whether the gains are real.
Why IR results drift: the usual suspects
| Drift source | Typical symptom | Pin it by |
|---|---|---|
| Collection version | "MS MARCO" numbers off by points | Exact version/split ids, ir_datasets identifiers, checksums |
| Index-time analysis | BM25 baseline differs across papers | Scripted index build; record stemmer, stopwords, k1/b |
| Doc processing for neural models | Recall@k shifts | Max length, stride, title concatenation recorded as config |
| Truncated vs judged pools | Inflated dense-retrieval scores | State pooling; report judged@k alongside nDCG |
| Seeds and nondeterminism | ±0.005 nDCG run-to-run | Multiple seeds; report mean ± sd, not the best run |
| Eval tool discrepancies | MAP differs at 4th decimal | One canonical scorer (trec_eval/ir_measures) with flags recorded |
| Hyperparameter asymmetry | Baselines lose by under-tuning | Equal tuning budget, documented per system |
A reproducibility-strong SIGIR paper closes each row with an artifact, not a promise: the config file is the documentation.
Minimum reporting block for any empirical SIGIR paper
Put this in the paper (it fits in ~0.3 page and pre-empts three review objections):
- Collections with versions; train/dev/test usage; any filtering.
- Index/build settings for every system, baselines included.
- Tuning protocol: search space, budget, selection metric, dev split — symmetric across systems.
- Seeds: how many, and whether tables show mean, sd, and the significance test.
- Compute: hardware, wall-clock for index+train+retrieve, and query latency setup.
- Pointer to the repository with run files (see
sigir-artifact-evaluation).
# config-as-artifact: one file per reported system, committed to the repo
system: ours-dense-v2
collection: msmarco-passage/dev/small # ir_datasets id
index: {tokenizer: bert-base-uncased, max_len: 256, stride: 128}
train: {seeds: [13, 42, 71], batch: 64, lr: 2e-5, epochs: 3}
eval: {tool: ir_measures, metrics: [nDCG@10, RR@10, R@1000], qrels: official}
significance: {test: paired-t, correction: bonferroni, alpha: 0.05}
Writing a Reproducibility track paper
A reproduction study is not a re-run; it is an investigation. The track rewards:
- Target selection with stakes — results the community builds on (a widely cited ranker, a standard baseline configuration, a claimed efficiency win).
- Faithful reimplementation first: reproduce with the original artifacts where they exist; document every forced deviation and why.
- Divergence analysis as the contribution: when numbers differ, isolate the cause (collection version? tuning? eval tool?) with controlled toggles — the drift table above is your experimental design.
- Generalization probes: does the original conclusion survive new collections, matched tuning, or current baselines? "Holds, but the margin halves under equal tuning" is a publishable, community-serving finding.
- Respectful register: the genre convention is scientific, not gotcha — criticize configurations, not authors, and give original authors' artifacts credit where due.
Anti-patterns the track's reviewers flag: reproducing only the headline number while skipping ablations; declaring "failure to reproduce" without exhausting configuration space; and shipping a reproduction whose own pipeline is unpinned (the irony reject).
Decay planning: the three-year audience
A SIGIR paper's reproducibility has two audiences with different failure modes: the reviewer this spring (needs the 10-minute runnable path) and the researcher in three years (meets link rot, dataset takedowns, deprecated toolkit APIs, and vanished model checkpoints). Plan for the second audience explicitly:
- Cite collections by stable identifiers (
ir_datasetsids, TREC track names, dataset DOIs), never by lab-server URLs. - Freeze an archival copy of your own artifacts (Zenodo/institutional DOI) at camera-ready; GitHub is a working mirror, not an archive.
- Record toolkit versions in the paper text, not only in the lockfile — the paper outlives the repository more often than authors expect.
- If a resource you depend on has restrictive terms (query logs, commercial APIs), state what a future reproducer can do without it: which tables survive, which die.
- Leave the qrels/run checksums in the repo README; three years later they are the only way to prove a re-download matches the evaluated state.
Reproducibility as review defense for regular papers
- Close comparisons without variance are the most common SIGIR experimental criticism; three seeds with mean ± sd is cheap insurance for neural systems.
- An honest "we could not reproduce baseline X's published number; we report our best faithful configuration (details in repo)" reads as strength, not weakness — the community knows the drift problem intimately.
- Never copy baseline numbers across collections or eval setups from other papers' tables without saying so; mixed-provenance tables are a known reject trigger.
Quick self-audit before submission
- Every number in every table regenerates from a shipped run file plus one documented command.
- Every system row has a config file; diffs between systems are visible as config diffs, not prose.
- The seeds behind each neural row are enumerable, and the shipped run is identified (which seed, or the per-topic mean).
- A colleague outside the project reproduced Table 1 from the repo README without asking questions (the strongest cheap test available).
- The paper's reporting block and the repo's configs agree — reviewers diff them when suspicious.
Output format
[Mode] hardening a regular paper / Reproducibility track study
[Drift audit] rows closed with artifacts: <k>/7 (collection/index/processing/pool/seed/tool/tuning)
[Reporting block] present in paper y/n; missing items <list>
[For repro studies] target + stakes / faithfulness log / divergence causes isolated
[Variance] seeds <n>, mean±sd shown y/n, test named y/n
[Biggest residual risk] <the one unpinned thing a reviewer will find>
Version History
- 9f86f09 Current 2026-07-19 17:36
