OSDI Reproducibility

Make the numbers survivable. OSDI-specific hooks below (artifact timing, badge scope, open-access exposure) are 2026-cycle facts verified 2026-07-08; the provenance discipline is venue-independent engineering.

Why the bar is high at OSDI specifically

Two venue mechanics raise the stakes beyond generic good practice:

• USENIX proceedings are open access from day one. Every reader on the internet — including the teams whose systems you outperformed — gets the free PDF immediately and can attempt your numbers. Errors get found publicly and fast.
• Artifact evaluation happens after acceptance (May 8, 2026, 8:59 pm PDT in the '26 cycle), when the experiments are months old. If provenance was not recorded while runs happened, the artifact reconstructs a memory, not an experiment.

Reproducibility at OSDI is therefore a recording problem during the project, not a packaging problem at the end. Packaging is osdi-artifact-evaluation's job; this skill makes packaging possible.

The provenance ledger

Maintain one machine-readable ledger, committed beside the code, updated by the run scripts themselves — never by hand after the fact:

# runs/2025-11-14-recovery-scale/ledger.yaml (written by the harness, per experiment)
experiment: recovery-vs-cluster-size      # maps to RQ in the experiment matrix
commit: 4f2c9e1 (system) / 8a11d02 (harness)
hardware: 64x c6525-25g (CloudLab), 25 GbE, NVMe model+fw recorded per node
os_kernel: Ubuntu 22.04, 5.15.0-91; mitigations=on; governor=performance
baseline_versions: replayfs v2.3.1 (tag), ckptstore rebuilt from paper (SHA)
workload: trace block-2025-w2, reconstruction script + source documented
runs: 10 per point; seeds 1..10; outliers kept, plotted as distribution
raw_output: s3://bucket/runs/2025-11-14/... (checksummed)
figures: fig7 <- plot_recovery.py @ 8a11d02 on raw_output

The last line is the anti-drift rule: every figure in the paper regenerates from checksummed raw output by a committed script. If a figure cannot name its script and input, the number it shows is unverifiable — by the AE committee and by you in June.

What systems papers must pin down

Hardware access is the honest limit of systems reproducibility: a result needing 64 specific machines will not rerun on a laptop. The discipline is disclosure plus graceful degradation — document the full testbed, and provide a scaled-down configuration that exercises every code path even if it cannot reproduce headline magnitudes.

Determinism where it is cheap, honesty where it is not

Distributed systems are not bitwise-reproducible; do not pretend otherwise. Seed what can be seeded (workload generation, placement decisions, fault-injection schedules), report distributions over repeated runs for what cannot, and state which class each reported number belongs to. A paper that says "recovery time varies ±8% run to run; we report 10-run distributions" pre-empts the reviewer who reruns and gets a different point value.

Traces, data, and the licensing wall

Workload provenance is where systems reproducibility most often dies quietly:

• Production-derived traces usually cannot be redistributed. Decide the release posture before the evaluation depends on them: either obtain redistribution rights early, or build a documented reconstruction pipeline (statistical profile → generator → validation against the original) and treat the generator as part of the artifact.
• Public traces still need version pinning and checksums — public archives reorganize, and "the standard trace" is not an identifier.
• Sensitive measurements (multi-tenant clusters, user-facing services) need the anonymization step documented as code, because it changes distributions and a reproducer must know how.
• Whatever the posture, the paper states it in one honest sentence; discovering an unreleasable dataset during artifact evaluation reads far worse than declaring it in the submission.

The shared smoke checker in ../../resources/code/README.md catches structural gaps (missing README/manifest/license) but none of the above — licensing and provenance are judgment calls only the authors can make.

Timing against the 2026 cycle

• During experiments (autumn) — the ledger above, enforced by the harness.
• Silent review window (Dec–Mar) — freeze the testbed image, trace archives, and environment; the '26 Call for Artifacts encouraged preparing artifacts while the paper was under consideration, and a conditional accept may demand new runs on the frozen setup (osdi-author-response).
• After notification (Mar 26) — packaging sprint to the May 8 artifact deadline; in 2026 the badge evaluated was Artifacts Available, so permanent archiving is the floor — but a ledger-backed artifact is what makes the optional two-page Artifact Appendix in the final paper worth writing (osdi-camera-ready).

The handoff test

The standing acceptance test for all of the above: a new group member, given only the repository and the ledger, regenerates one paper figure on the scaled-down configuration without asking anyone anything. Run it quarterly and before each gate (submission, artifact deadline, final paper). Every question they are forced to ask is a missing ledger entry; every mismatch they hit is drift between paper and artifact that an AE evaluator — or a public reproducer holding the open-access PDF — would have found later, with an audience.

Output format

[Ledger] exists + harness-written? gaps: <dimensions from the table>
[Figure regeneration] all figures script+input traceable? failures: <list>
[Determinism statement] seeded vs distributional numbers classified? yes/no
[Testbed freeze] image/trace archive frozen for the review window? yes/no
[AE readiness] distance from ledger to packageable artifact: <low/med/high>