cikm-reproducibility
GitHub指导CIKM论文增强可复现性：通过固化流水线各阶段参数、处理不可发布数据、规范GenAI披露及环境捕获，解决组件集成导致的隐性结果漂移问题。
Trigger Scenarios
Install
npx skills add brycewang-stanford/Awesome-Journal-Skills --skill cikm-reproducibility -g -y
SKILL.md
Frontmatter
{
"name": "cikm-reproducibility",
"description": "Use when hardening the reproducibility of a CIKM paper — pinning the pipeline stages where IR, mining, and knowledge-management results silently diverge, documenting KGs and enterprise data that cannot be released, keeping the GenAI disclosure consistent with how code and data were produced, and preparing the post-acceptance release."
}
CIKM Reproducibility
Reproducibility at CIKM has a venue-specific difficulty: the typical paper chains components from different communities — an index, a graph, a model, a ruleset — and each link has its own silent-divergence habits. A reader who cannot rebuild the chain cannot attribute the result, and a blended review panel contains someone able to notice each weak link.
Divergence map for chained pipelines
| Chain link | How results silently drift | Pin |
|---|---|---|
| Text preprocessing / indexing | Tokenizer versions, stopword lists, index-time defaults differ across toolkits | Name toolkit + version + config file in the artifact |
| KG snapshot | Public KGs (Wikidata-class) change daily; entity counts drift | Freeze and state the dump date; ship the extracted subgraph if licensable |
| Candidate generation | Recall stage caps and thresholds rarely reported | Report every cutoff; they bound the final metrics |
| Training | Seeds, hardware nondeterminism, early-stopping criteria | Seed policy + selection rule in the protocol paragraph |
| Evaluation | Metric implementations disagree at tie-breaking and cutoffs | Name the evaluation library version; never hand-roll silently |
| LLM components | Model version/API drift; prompts unlogged | Pin model identifiers and dates; log prompts verbatim in the artifact |
The discipline: for each link, either the artifact pins it or the paper states it. A link pinned nowhere is where a failed replication will land.
Unreleasable data, releasable knowledge
CIKM's KM lane routinely involves enterprise corpora, clickstreams, or proprietary KGs that cannot ship. The venue-honest pattern:
- Describe the unreleasable data statistically (size, schema, class balance, collection window) at a level where a reader could construct a synthetic analog — then actually provide that analog generator when feasible.
- Run the public-data variant of every headline experiment, even if the effect is smaller; a result that exists only on invisible data asks the panel for faith.
- State the release position explicitly in the paper ("logs cannot be released; the sampling script and schema are in the artifact") rather than leaving the reader to discover the gap.
GenAI disclosure as a reproducibility document
CIKM 2026's mandatory GenAI Usage Disclosure covers code and data, not just prose (source map, 2026-07-08). Treat it as part of the methods record: if evaluation scripts, synthetic data, prompts, or labels were generated with AI assistance, the disclosure plus the artifact should together let a reader judge what that implies for the result. A disclosure that says "AI used for coding" while the artifact contains unexplained generated labels is an inconsistency automated compliance checks — which the conference reserves — or reviewers can catch.
Environment capture
Chained pipelines multiply environment surface, so capture it in layers:
| Layer | Capture mechanism |
|---|---|
| OS + system libraries | Container image or a documented base image tag |
| Language environments | Lockfiles (exact versions), not loose requirement ranges |
| Toolkit configs | The actual config files, committed — not "default settings" prose |
| Data inputs | Checksums + download scripts, or the frozen extraction (see KG row) |
| Hardware assumptions | GPU/CPU class and memory floor stated where results are timed |
The test is transferability: a lab-mate on a clean machine, without the authors in the room, reaches the headline table. Running that internal replication before submission is the single highest-yield reproducibility exercise — it finds the unpinned link while it can still be pinned.
Where reproducibility pays at this venue
Three concrete CIKM payoffs beyond principle. First, the blended panel: whichever
lane doubts the result will probe its own link of the chain, so pinning every link
is defense in all three directions. Second, resource-track reviewers and readers
judge adoptability, which is reproducibility wearing its public face
(cikm-artifact-evaluation). Third, follow-up work: CIKM's back catalog shows
methods becoming standard baselines (DRMM, BERT4Rec); papers get that afterlife
only when third parties can run them — the reproducible version of a method is the
one that accumulates citations as a baseline.
Release timeline
Anonymized review artifact during submission (see cikm-supplementary for what the
budget permits); public repository at camera-ready, with license, versioned release
tag, and the exact commit that produced the proceedings numbers. The 2026
notification-to-camera-ready window is thirteen days — build the release during
the review wait (cikm-workflow Mode A), not inside that window.
Honest-failure disclosure
Chained pipelines rarely reproduce perfectly, and the venue-credible move is to say so first: a REPRODUCING.md that states which numbers regenerate exactly, which vary within a stated tolerance (GPU nondeterminism, sampling), and which depend on restricted inputs and therefore only regenerate in public-analog form. Declared tolerance reads as competence; discovered variance reads as concealment. The same document is where to state known environment sensitivities ("results verified on CUDA X; version Y shifts Table 3 by ±0.2") — the sentence that saves a replicator a week is the sentence that earns the citation.
One-command bar
# The replication target for a CIKM chained pipeline:
git clone <repo> && cd <repo>
make setup # pinned environment, data download or synthetic analog
make table2 # rebuilds the headline table end-to-end from the frozen inputs
If make table2 cannot exist because data is restricted, the repo must say so at
the top and offer the public-variant target instead. Silent partiality — a repo that
looks complete but is not runnable — costs more reviewer goodwill than an honest
scope statement.
Output format
[Chain audit] <link → pinned where (paper / artifact / nowhere)>
[Data position] <releasable / described+analog / public-variant-only>
[Disclosure consistency] <GenAI section vs. artifact contents>
[Release plan] <review artifact state → camera-ready repo state, dated>
[Weakest link] <the divergence a replicator would hit first>
Version History
- 9f86f09 Current 2026-07-19 14:39
