asplos-reproducibility
GitHub用于增强ASPLOS论文结果的可复现性。通过捕获软硬件全栈状态（硅片、固件、OS、工具链等）并生成清单，确保实验可独立重复。提供脚本化采集方法和针对硬件访问受限的三级可用性声明策略，以符合ACM徽章评估要求。
Trigger Scenarios
Install
npx skills add brycewang-stanford/Awesome-Journal-Skills --skill asplos-reproducibility -g -y
SKILL.md
Frontmatter
{
"name": "asplos-reproducibility",
"description": "Use when hardening an ASPLOS paper's results for independent repetition — pinning simulator versions and configs, recording kernel\/firmware\/BIOS state, packaging FPGA bitstreams and RTL, documenting hardware dependencies an evaluator may lack, and writing availability statements that match what the ACM badges will later require."
}
ASPLOS Reproducibility
Systems results decay fast: a kernel update, a microcode revision, or a silently
changed simulator default can move numbers by more than the paper's claimed margin.
Reproducibility work at ASPLOS is therefore state capture — recording the full
machine, model, and toolchain state behind every figure — done while the experiments
run, not reconstructed at camera-ready time. It also front-loads artifact
evaluation: the badge criteria (asplos-artifact-evaluation) are exactly a demand
that this state capture exists and works.
The state ledger
Maintain one ledger row per experimental platform, committed alongside results:
| Layer | Capture | Why it moves numbers |
|---|---|---|
| Silicon | CPU model + stepping, memory config/topology, device (e.g. CXL expander) firmware | Steppings differ in errata and prefetch behavior |
| Firmware/BIOS | Microcode revision; SMT, turbo, prefetcher, C-state, NUMA settings | Any one knob can swamp a 10% effect |
| OS | Kernel version + full config, relevant sysctls, mitigations state | Speculation mitigations alone shift syscall-heavy results |
| Toolchain | Compiler + flags, libraries, runtime versions | -O level and allocator choice are classic silent variables |
| Simulator | Exact commit, all config files, region/checkpoint method, warm-up length | Defaults change across releases without notice |
| FPGA | Board, toolchain version, constraints, bitstream hash, achieved clock | Re-synthesis at a different clock is a different experiment |
| Workloads | Suite versions, input sets, trace provenance and preprocessing | "SPEC" without input class is unrepeatable |
| Randomness | Seeds for any stochastic component + run counts | Needed for the dispersion numbers to mean anything |
Scripted capture beats remembered capture
Run at the start of every measurement session; store output next to the data:
#!/bin/sh
# state-capture.sh — commit this file and its output with each result set
uname -a; cat /proc/cmdline
grep -m1 'model name' /proc/cpuinfo; grep microcode /proc/cpuinfo | sort -u
cat /sys/devices/system/cpu/vulnerabilities/* 2>/dev/null | sort -u
cat /sys/devices/system/cpu/smt/control 2>/dev/null
numactl --hardware 2>/dev/null | head -5
cc --version | head -1
git -C "$SIM_DIR" rev-parse HEAD 2>/dev/null # simulator commit
sha256sum "$BITSTREAM" 2>/dev/null # FPGA bitstream identity
The hardware-access problem, named honestly
ASPLOS artifacts often need hardware an independent evaluator will not have. The honest pattern is a three-tier availability statement drafted at submission time:
- Repeatable anywhere: simulator experiments and analysis scripts — full configs and one command per figure.
- Repeatable with named hardware: the exact platform requirements (board, expander, CPU family), plus what to expect if the evaluator's part differs.
- Not independently repeatable: results on lab-only or pre-production hardware — say so, and provide either supervised access, raw logs with the analysis pipeline, or a scaled-down proxy. Silence here reads as concealment; a stated limitation reads as engineering.
Claim-preservation, not number-worship
State which conclusions should survive environmental drift and which are environment-specific: "the ordering of policies is stable across kernels 6.6-6.9; absolute runtimes are not." This single sentence pattern prevents the most common failed-reproduction dispute — an evaluator matching your ordering but not your absolute numbers and calling it a failure.
Timing across the ASPLOS cycle
- Before September 9: ledger current; capture script in the repo; availability tiers drafted (they inform the paper's own text).
- Response window: the ledger is your defense when a reviewer doubts a number — you can state the exact conditions instead of hand-waving.
- Major Revision: re-run under the captured original state where possible; where the environment has drifted, disclose the drift in the change note.
- After acceptance: the ledger becomes the Artifact Appendix's dependency section nearly verbatim; AE calendars for 2027 were 待核实 at pack-check time, so confirm dates when notified.
One command per figure
The internal gold standard that makes everything downstream cheap: every figure and table in the paper regenerates from a single committed command that reads raw results and emits the exact plot. It catches stale-figure bugs before submission, turns response-window questions into lookups, and becomes the Reproducible-badge run script with a rename. Institute it at the first result, when it costs minutes — retrofitting it at camera-ready costs days.
Trace and dataset provenance
Workload inputs decay independently of code. For each trace or dataset, record origin (public suite version, generated-by script + seed, or production source), preprocessing steps as scripts rather than prose, and a checksum of the exact bytes used. Production traces that cannot be released need a characterization (rate, skew, working-set curves) plus a matched synthetic generator committed to the repo — this is also the anonymity-safe form for submission, since a raw trace can identify its owner.
When numbers drift between submission and revision
The Major Revision window arrives months after the original runs, and environments drift. Protocol:
- Re-run a sentinel subset (three representative experiments) under the captured original state before starting revision work; if the sentinels reproduce, extend confidently.
- If they do not, bisect the ledger — kernel, microcode, simulator commit — until the moved variable is found; the ledger exists for exactly this moment.
- Disclose in the change note which results were re-collected and under what changed conditions, and re-state the claim-preservation sentence for the new environment. Silent regeneration of all numbers invites a reviewer to ask which version was real.
Output format
[Ledger coverage] platforms with complete rows: N/N · gaps listed
[Capture automation] script committed + outputs stored with data: Y/N
[Simulator pinning] commit + configs + region method + warm-up recorded: Y/N
[Availability tiers] anywhere / named-hardware / not-repeatable — each populated
[Claim preservation] drift-stable vs environment-specific conclusions stated: Y/N
[Badge readiness] which badges the current package could already earn
Version History
- 9f86f09 Current 2026-07-19 14:19
