uai-reproducibility
GitHub针对UAI论文强化可复现性证据，涵盖随机种子、采样器诊断、ELBO轨迹、数据集披露及计算报告。提供确定性账本代码与披露清单，确保结果可验证且符合会议对代码数据公开的强烈建议。
Trigger Scenarios
Install
npx skills add brycewang-stanford/Awesome-Journal-Skills --skill uai-reproducibility -g -y
SKILL.md
Frontmatter
{
"name": "uai-reproducibility",
"description": "Use when hardening reproducibility evidence for a UAI paper, including seeds, sampler convergence diagnostics, ELBO and calibration traces, dataset and hyperparameter disclosure, compute reporting, and honest code-availability statements, since UAI strongly encourages released code and data and reviews whether claims are convincingly backed."
}
UAI Reproducibility
UAI's 2026 CFP did not impose a formal reproducibility checklist (one may appear later — 待核实 each cycle), but it strongly encouraged code and data availability and instructed reviewers to judge whether claims are backed up convincingly. At this venue "convincing" has a technical meaning: stochastic-inference results carry diagnostics, not just point estimates. This skill turns that norm into an audit.
The UAI-specific bar
Reproducibility questions at a probabilistic-inference venue go one level deeper than "can I rerun the script":
- A sampler that reproduces the same posterior mean but different tail quantiles has not reproduced the paper — report and check the diagnostics that detect this (R-hat, ESS, divergent transitions where relevant).
- A variational result is reproducible when the ELBO trajectory and the selected restart are recoverable, not merely the final metric; multi-restart selection rules must be stated.
- Calibration claims reproduce only if the binning scheme, the split used for recalibration, and the α grid are all pinned down; empirical coverage moves with all three.
- Causal-discovery results depend on graph generation as much as on the algorithm: publish the SCM sampler, noise families, and intervention protocol.
Determinism ledger
Record every randomness source once, in code, and cite it from the paper:
# repro/ledger.py — imported by every experiment entry point
import json, platform, random, numpy as np
def fix_and_log(seed: int, path: str = "run_manifest.json"):
random.seed(seed)
np.random.seed(seed)
manifest = {
"seed": seed,
"python": platform.python_version(),
"numpy": np.__version__,
"chains": 4, "warmup": 1000, "draws": 2000, # sampler config lives here
"elbo_restarts": 10, "restart_rule": "best final ELBO",
}
json.dump(manifest, open(path, "w"), indent=2)
return manifest
The manifest style matters more than the specific fields: one machine-readable file per run, checked into the artifact, lets a reviewer reconcile the paper's Table 3 with an actual execution.
Disclosure map
| What must be recoverable | Where it lives at UAI | Common omission |
|---|---|---|
| Model and assumption set | Main part, stated with each theorem/method | Assumptions distributed across three sections |
| Sampler / optimizer settings | Appendix (unlimited, same PDF) | "Default settings" without library version |
| Hyperparameter search space and selection rule | Appendix table | Only the winning configuration reported |
| Seeds and number of repeats | Appendix + artifact manifest | Single-run results with no variance |
| Dataset versions, splits, preprocessing | Appendix + loader script in ZIP | Preprocessing "as in [12]" where [12] is ambiguous |
| Compute (hardware, runtime, memory) | Appendix | Runtime reported only for the proposed method, not baselines |
| Code/data availability statement | Main part or appendix | Silence, which reviewers read as "unavailable" |
Diagnostic quick reference
What "reported convincingly" tends to mean per inference family — as conventions of the field, not venue mandates:
| Family | Minimum reported | Stronger version |
|---|---|---|
| MCMC | R-hat per parameter block, ESS, chain count/length | Rank plots; comparison against a long-run gold standard |
| Variational | Final ELBO, restart count and rule | ELBO traces; posterior-quality check on a tractable case |
| SMC / particle | Particle count, resampling scheme, ESS trajectory | Variance of the marginal-likelihood estimate over repeats |
| Conformal / intervals | Split sizes, α grid, empirical coverage | Conditional coverage slices; width distribution |
| Calibration | Binning scheme, ECE definition used | Reliability diagrams with confidence bands over seeds |
| Causal discovery | Graph generator, noise family, SHD/SID per seed | Sensitivity to assumption violations (unfaithfulness, confounding) |
If a row's "minimum" column is missing for your method family, expect the backing criterion to absorb the damage.
Honesty over completeness
- If code cannot be released (industrial constraints, licensed data), say so in the paper and compensate: fuller pseudocode, exact hyperparameters, synthetic surrogates for private datasets. The encouraged-not-mandatory wording gives room for honesty, not for vagueness.
- Report failure modes you observed — initializations that collapse, chains that need longer warmup on one dataset. Probabilistic-ML reviewers trust papers that know where their method breaks.
- Never let variance disappear in the retelling: if three of ten seeds underperform, the aggregate table must reflect it (mean ± sd over all ten, or a stated, principled selection rule).
Where reproducibility evidence lives
Split by tier deliberately: the availability statement and diagnostic summaries in the reviewed PDF (body or appendix), where they count toward backing; manifests, loaders, and per-run logs in the ZIP, where they support spot-checks. Never leave the only mention of seeds or repeat counts inside the optional archive — reviewers grade what the PDF says.
Pre-submission reproducibility drill
- Clone your own artifact onto a machine that never ran the project; follow only the README.
- Reproduce the smallest headline number end to end, including the diagnostic that validates it.
- Diff the regenerated figure against the paper's; investigate any visible deviation before a reviewer does.
- Grep the paper for every "we observe/we find/consistently" and confirm each maps to a logged run.
- Write the availability statement last, describing what is actually in the ZIP — not what was planned in January.
Output format
[Repro grade] turnkey / recoverable with effort / under-specified
[Diagnostics reported] <R-hat/ESS/ELBO/coverage/SHD... as applicable>
[Determinism ledger] present / partial / absent
[Disclosure gaps] <items from the map still missing>
[Availability statement] drafted / needs honesty pass / missing
Version History
- 9f86f09 Current 2026-07-19 17:51
