Agent Skills
› brycewang-stanford/Awesome-Journal-Skills
› usenixsec-reproducibility
usenixsec-reproducibility
GitHub指导USENIX Security论文撰写开放科学附录，规范代码、数据共享与 withheld 决策，确保实验结果可复现及独立验证。
Trigger Scenarios
撰写USENIX Security论文的开放科学附录
决定安全研究数据的共享范围与理由
构建可复现的安全实验环境
Install
npx skills add brycewang-stanford/Awesome-Journal-Skills --skill usenixsec-reproducibility -g -y
SKILL.md
Frontmatter
{
"name": "usenixsec-reproducibility",
"description": "Use when strengthening reproducibility for a USENIX Security Symposium paper — writing the mandatory Open Science appendix, deciding what can and cannot be shared (exploit code, vulnerable-device data, human-subjects material), and making measurement, attack, and defense results independently regenerable."
}
USENIX Security Reproducibility
Since the '26 cycle, USENIX Security has made openness a structural requirement: every submission carries an Open Science appendix stating where the artifacts behind the paper live, and acceptance is later conditioned on that availability verifying (Phase-1 AE). This skill covers the appendix itself and the underlying engineering that makes the statement true. Policy text is per-cycle — reread the current CFP section before relying on wording.
The Open Science appendix is a contract, not a caption
Reviewers read it during evaluation; the AEC enforces it after acceptance. It should answer, concretely:
- What exists: code, datasets, configurations, analysis notebooks, hardware designs — enumerated, not gestured at.
- Where: resolvable locations (anonymized mirrors at submission; permanent, ideally DOI-backed archives in the final paper).
- What is withheld and why: the policy accepts justified omission, and security work legitimately generates a lot of it.
\section*{Open Science}
All scanner source code, the analysis pipeline, and the aggregated measurement
tables (Sections 4--6) are available at \url{<anonymized-archive>} and will be
deposited with a DOI upon publication. The raw scan captures contain
per-endpoint identifying data for still-unpatched hosts and are withheld;
Appendix~B documents the aggregation procedure so the tables can be
regenerated from an independent scan. The 214 disclosure emails are withheld
as human-subjects correspondence; templates are included.
Withholding decisions that hold up
| Material | Default posture | Acceptable justification pattern |
|---|---|---|
| Attack/PoC code | Share, targeted at a bundled testbed | Weaponizable against unpatched population → share after patch window, state the date |
| Vulnerability details | Share post-disclosure | Coordinated disclosure incomplete → embargo with timeline in the Ethical Considerations appendix |
| Scan/measurement raw data | Share aggregated | Raw data identifies vulnerable hosts or users → aggregate + publish the aggregation code |
| User-study transcripts | Withhold; share instruments | IRB/consent scope → release codebooks, surveys, and quantitative summaries |
| Malware corpora | Hashes + provenance | Redistribution illegal or dangerous → document acquisition path others can follow |
| Vendor-provided datasets | As NDA allows | Contract limits → say so and provide synthetic or public-subset substitutes |
Two disciplines make these defensible: decide them at experiment time (retrofit justifications read as excuses), and always ship the procedure even when the data stays closed — an independent team with their own vantage point should be able to re-derive your tables.
Determinism engineering for security experiments
Security results are unusually time- and environment-coupled. Pin what you can and timestamp what you cannot:
- The internet moves. Record scan dates, target-list snapshots, and blocklist versions; a reproduction in 2027 measures a different network than 2026 did.
- Fuzzing and probabilistic attacks: fix seeds where the harness allows, and report distributions over repeated campaigns (count, median, IQR) rather than a best run; state the compute budget per campaign.
- ML-based pipelines (classifiers, LLM-assisted analysis): pin model versions and weights; a hosted-API dependency is a reproducibility hole — snapshot outputs and say so.
- Version-sensitive exploits: record exact kernel/firmware/browser builds, and keep the vulnerable version archived in the artifact since upstream will patch.
# Capture the environment fingerprint alongside every experiment run
{ date -u +%FT%TZ; uname -srmo; git rev-parse HEAD;
sha256sum targets.txt config.yaml; pip freeze | sort; } > runs/$(date +%s).env
Paper-side reporting that reviewers check
- Every headline number traces to a script in the artifact; the claims table in
the artifact README (see
usenixsec-artifact-evaluation) is the index. - Ranges and repetition counts for stochastic results; single-run numbers flagged as such.
- Dataset construction fully specified: collection window, filters, dedup rules, ground-truth labeling procedure and inter-rater agreement where humans labeled.
- Negative-result honesty: configurations where the attack fails or the defense costs too much belong in the paper, and reviewers at this venue notice absence.
Reverify each cycle
- Exact Open Science appendix requirements and page allowance in the current CFP.
- Whether availability verification remains acceptance-conditional ('27 待核实).
- Current AEC guidance on acceptable archives and anonymized hosting.
Output format
[Inventory] artifacts enumerated: code / data / configs / instruments
[Open items] each withheld artifact + justification + procedure substitute
[Determinism] seeds, environment fingerprints, time-coupling documented
[Trace] headline claims ↔ regeneration scripts: n/m covered
[Appendix draft] Open Science text ready: yes / gaps listed
Version History
- 9f86f09 Current 2026-07-19 17:52
