ieeesp-reproducibility
GitHub用于在IEEE S&P论文提交前强化证据可复现性。涵盖环境锁定、概率攻击的试验记录、系统快照及伦理发布声明，确保实验结果在审查期间不失效。
Trigger Scenarios
Install
npx skills add brycewang-stanford/Awesome-Journal-Skills --skill ieeesp-reproducibility -g -y
SKILL.md
Frontmatter
{
"name": "ieeesp-reproducibility",
"description": "Use when hardening the reproducibility of an IEEE S&P (Oakland) paper's evidence before submission, including environment pinning for exploits and side channels, seed and trial reporting for probabilistic attacks, measurement snapshotting of live systems, and honest availability statements under ethical release limits."
}
IEEE S&P Reproducibility
Use this while the evidence is being produced — months before
ieeesp-artifact-evaluation packages it. Security results decay in ways ML
results do not: targets get patched, infrastructure changes, and an attack
that "worked in March" can be unreproducible by review time. Reproducibility
at S&P is first about freezing the world you measured.
Freeze the target, not just the code
| Evidence type | What must be pinned | Decay risk if not |
|---|---|---|
| Software exploit | Target version, build flags, distro, patch level | Silent fix ships mid-review |
| Microarchitectural attack | CPU model + stepping, microcode, OS mitigations state | Microcode update changes timing |
| Network measurement | Scan dates, vantage points, target-list snapshot | Internet moved; sample unrecoverable |
| Web/API study | Crawl date, client fingerprint, geographic origin | Server-side behavior shifts |
| ML attack/defense | Model weights hash, dataset version, threat-budget ε | "Same" model retrained differently |
Record these in the paper, not only in lab notes — a reviewer asking "does
this still work after
Determinism ledger for probabilistic evidence
Attacks with success probabilities, fuzzing campaigns, and randomized defenses need trial discipline:
- Report trials, not anecdotes: success rate over n runs with n stated, plus dispersion — a "9/10 successful" exploit and a "measured once" exploit are different claims.
- Fuzzing comparisons follow the field's known pitfalls: equal CPU-time budgets, multiple campaigns, identical seed corpora — state all three.
- Randomized defenses (ASLR-like, moving-target): evaluate across the randomness, not one lucky layout.
- Timing measurements: report the noise floor and the machine's quiescence conditions (isolated cores, frequency pinning) or the numbers will not transfer.
Reproducibility ledger (one row per experiment in the paper):
exp_id | claim it supports | target snapshot (ver/date/hw) |
trials & seeds | dispersion reported? | rerun cost (time/hw/$) |
rerunnable by outsider? (yes / gated / world-dependent)
The last column becomes the availability statement and the honest badge target later.
When ethics limits release, say exactly what and why
S&P reviewers accept withheld material when the reasoning is specific:
- "Exploit for CVE-pending issue withheld until the fix ships; released to reviewers via the chairs on request" beats a silent gap.
- User-derived datasets: describe schema and collection so others can re-collect ethically, even when raw data cannot ship.
- Never fabricate openness — a promised-but-empty repository found during review or after publication is a reputation event, and at this venue reviewers do check.
Cheap habits that pay at rebuttal time
- One
env.lockper experiment directory: container digest, package list, kernel and microcode versions, dumped automatically by the run script. - Raw outputs archived before aggregation; the rebuttal question is always about a number two steps upstream of the figure.
- A
regenerate_figures.shthat goes from archived raw data to every figure — this is also the artifact-evaluation core later. - Date-stamped disclosure and measurement logs, because ethics questions in
review are answered with timelines (
ieeesp-author-response).
What this venue does not require
Keep effort calibrated: S&P has no submission-time reproducibility checklist in the verified 2026/2027 materials (待核实 each cycle), artifact evaluation is post-acceptance and optional, and appendices are explicitly not guaranteed reader attention. The reproducibility work above is therefore aimed at three audiences in order: your own rebuttal, the shepherd, and the AE committee — not at a submission-form requirement.
Output format
[Ledger status] <n>/<total> experiments with snapshot + trials + dispersion
[World-dependence] <which results cannot be re-run by anyone, ever — flagged in text?>
[Release plan] open / gated (reason) / withheld (reason) — per component
[Rebuttal readiness] raw data archived ✓/✗ · env locks ✓/✗ · figure regen ✓/✗
[Gaps to close before registration week] <ordered list>
Version History
- 9f86f09 Current 2026-07-19 16:00
