kdd-reproducibility
GitHub针对KDD论文复现性加固，提供三级诚实模型（可重跑/可重建/已验证）及数据流水线披露清单。涵盖种子、划分、基线调优和计算资源报告，确保结果透明可信，满足会议对复现性的评分要求。
Trigger Scenarios
Install
npx skills add brycewang-stanford/Awesome-Journal-Skills --skill kdd-reproducibility -g -y
SKILL.md
Frontmatter
{
"name": "kdd-reproducibility",
"description": "Use when hardening the reproducibility story of a KDD paper, where reproducibility is an explicit decision factor for area chairs. Covers claim-to-evidence tiers for mining pipelines, seeds and splits on large graphs and streams, baseline-tuning disclosure, compute reporting, and honest limits for ADS deployment results."
}
KDD Reproducibility
Use this before the paper freezes. The KDD 2026 CFP lists reproducibility of results among the factors area chairs weigh in acceptance recommendations — at this venue it is a scored dimension, not a virtue. The bar is shaped by what KDD papers claim: pipelines over large, messy, sometimes proprietary data, where the reader must be able to tell exactly which parts they can rerun, which they can rebuild, and which they must take on documented trust.
Three-tier honesty model
Declare every result in the paper as one of:
- Rerunnable — public data, shipped code, fixed seeds; a reader regenerates the number. This should cover the headline comparison table.
- Rebuildable — method fully specified (pseudocode + hyperparameters + data schema), but data is proprietary or too large; a reader can reconstruct the pipeline on their own data.
- Attested — post-launch/production measurements that no outsider can repeat; reproducibility here means the measurement protocol is fully specified.
An ADS paper is usually tier 2-3; a Research Track paper claiming tier 1 while shipping tier 2 is what burns trust in review.
Mining-pipeline disclosure checklist
The stages that silently decide results in data-mining papers, and what to pin:
| Pipeline stage | What must be pinned | Classic KDD reviewer catch |
|---|---|---|
| Data acquisition | Version/date of each dataset, filtering rules, dedup | "Which snapshot of the graph?" |
| Preprocessing | Feature construction code, normalization, leakage guards | Target leakage via time-travel features |
| Splits | Split logic (temporal vs random), exact seeds, cold-start handling | Random splits on temporal data inflating results |
| Negative sampling | Ratio, distribution, per-epoch resampling or not | Baselines run with a different sampling scheme |
| Baselines | Source of implementation, tuning budget per baseline | Own method tuned for weeks, baselines run at defaults |
| Hardware/compute | Machines, memory, wall-clock per experiment | Throughput claims with no hardware context |
Equal-tuning-budget disclosure matters more at KDD than almost anywhere: the venue's history is full of boosting/embedding/GNN comparisons decided by tuning asymmetry.
Config-as-artifact discipline
Every reported number should trace to a committed config, and the mapping should be mechanical:
# repro/manifest.py - emitted next to every result file
import json, subprocess, time
def write_manifest(cfg, dataset_stats, out="results/manifest.json"):
manifest = {
"config_file": cfg.path, # one config per table row
"seed": cfg.seed, # and the full seed list for repeats
"dataset": dataset_stats, # {"name":..., "rows":..., "edges":..., "snapshot":...}
"git_commit": subprocess.check_output(
["git", "rev-parse", "HEAD"]).decode().strip(),
"wall_clock_sec": cfg.elapsed,
"hardware": cfg.hardware, # e.g. "1x A100-80GB, 256GB RAM"
"timestamp": time.strftime("%Y-%m-%dT%H:%M:%SZ"),
}
json.dump(manifest, open(out, "w"), indent=2)
Tables generated from manifests cannot drift from the artifact — the failure mode where the PDF says 0.847 and the repo reproduces 0.831 is a rebuttal-phase disaster, and KDD rebuttals cannot even link to a corrected artifact.
Stochasticity on large data
- Repeat runs are expensive at KDD scale; the honest compromise is repeats on the small/medium datasets plus a variance statement, and single seeded runs at the largest scale with the seed published.
- State what is nondeterministic even under a fixed seed (parallel reductions, GPU atomics, hash-partitioned sampling), so a near-miss reproduction reads as expected rather than as failure.
- Never present a best-of-k run as a typical run; if model selection used validation metrics, say k and the selection rule.
ADS attested-results protocol
For post-launch numbers, reproducibility means specification: metric definitions, measurement window, traffic allocation, ramp schedule, guardrails, and any seasonal confounders in the window. A reader should be able to audit the measurement, even though they cannot repeat it. Blur only what confidentiality forces (absolute denominators can become relative lifts), and say explicitly what was blurred and why.
Vignette: the tier audit that changed a claim
A recommendation paper reports wins on two public datasets and one "large industrial dataset." The tier audit finds: public results are rerunnable (good); the industrial result is attested but the paper's abstract says "we release all code and data" — written before the legal review pulled the industrial set. Left uncorrected, that sentence is the kind of paper-vs-reality contradiction a practitioner reviewer catches in minutes and generalizes from ("what else is oversold?"). The repair:
- Abstract sentence rewritten to "code and the two public datasets are released; the industrial pipeline is fully specified in Section 4."
- The industrial section gains the schema, size, and collection-window spec that makes it rebuildable-in-principle.
- The headline claim re-anchored on the public results, with the industrial run as corroboration rather than as the primary evidence.
Reproducibility review at KDD is largely consistency review: the checklist is the paper against itself.
Pre-freeze reproducibility gate
Run this list the week before the paper freezes, while fixes are still cheap:
- Every number in the abstract traces to a table, and every table row traces to a committed config.
- Tier labels (rerunnable/rebuildable/attested) assigned per result and reflected in the availability sentence — no tier inflation anywhere in the text.
- Split logic re-derived from the code, not from memory; temporal data confirmed time-ordered end to end.
- Baseline provenance table complete: implementation source, version, tuning budget.
- The manifest of the single largest run (hardware, wall-clock, dataset cardinality) exists — it anchors both the scale claim and the compute disclosure.
- Repository smoke-tested from a clean clone by someone who did not write it.
Output format
[Tier map] <headline table: rerunnable/rebuildable/attested, per major result>
[Pipeline pins] <acquisition/preprocessing/splits/sampling/baselines/compute status>
[Tuning symmetry] <our budget vs baseline budget, disclosed where>
[Variance story] <repeats + seeds at which scales>
[ADS measurement spec] complete / gaps: <...> / N-A
[Trust risks] <paper-artifact drift, leakage, undeclared selection>
Version History
- 9f86f09 Current 2026-07-19 16:56
