asplos-artifact-evaluation
GitHub指导作者为ASPLOS会议准备Artifact附录，撰写ae.tex，明确Available/Functional/Reproducible三种徽章要求。提供符合评估委员会标准的代码库结构、依赖说明及与评估者协作的策略。
Trigger Scenarios
Install
npx skills add brycewang-stanford/Awesome-Journal-Skills --skill asplos-artifact-evaluation -g -y
SKILL.md
Frontmatter
{
"name": "asplos-artifact-evaluation",
"description": "Use when preparing an ASPLOS artifact for the post-acceptance evaluation committee — writing the ae.tex Artifact Appendix with software\/hardware\/dataset dependencies, targeting the Available \/ Functional \/ Reproducible badges, archiving on a public repository, and planning the collaborative back-and-forth with evaluators."
}
ASPLOS Artifact Evaluation
Artifact evaluation at ASPLOS is a post-acceptance, opt-in, collaborative process: an independent committee works with authors to validate the paper's key results, and successful artifacts carry badges on the published paper (AE pages, checked 2026-07-08). It is also a tradition the venue itself highlights — systems readers increasingly treat an unbadged systems paper as a weaker citation. Treat AE as part of the publication, budgeted like a small sixth section.
The three badges and what each actually demands
| Badge | 2027 criterion (paraphrased from the AE pages) | Practical bar |
|---|---|---|
| Available | Artifact placed on a publicly accessible archival repository | A DOI-issuing archive (institutional or Zenodo-class); a GitHub URL alone is not archival |
| Functional | Evaluators can prepare and run the artifact; they document the steps they followed | Clean-machine install + a smoke experiment that completes in minutes, not hours |
| Reproducible | Evaluators validate the paper's key results | Per-claim run scripts whose output maps visibly onto specific figures/tables |
Evaluators assign scores per requested badge and record what they could and could not reproduce — so the artifact's job is to make their success path short and their failure modes diagnosable.
The Artifact Appendix is the contract
ASPLOS 2027 expects an Artifact Appendix built from the provided ae.tex template
(or equivalent sections) covering: all software, hardware, and dataset
dependencies; the key results to be reproduced; and how to prepare, run,
and validate the experiments. Write it as if the evaluator is competent, busy,
and using different hardware than yours:
- Dependencies include the awkward ones: kernel versions, privileged access, BIOS
settings, board models, expander firmware — everything from the state ledger in
asplos-reproducibility. - "Key results" means a selected subset: pick the claims that define the paper, not all 40 bars of every figure. Ambition here creates failure reports.
- Validation must be decidable: state the expected output and the tolerance within which the claim holds ("ordering preserved; absolute times ±15%").
Package layout that evaluators can navigate blind
artifact/
README.md # 10-minute quick start + full map
APPENDIX.pdf # the ae.tex appendix as submitted
env/ # container/VM recipe OR exact install script
hardware.md # tiered requirements + what to do without them
run/
smoke.sh # minutes-scale end-to-end sanity check
claim1_fig6.sh # one script per key result, named for its figure
claim2_tab3.sh
expected/ # reference outputs + tolerance statement per claim
data/ or data.md # datasets, or archival pointers + checksums
Hardware-dependent claims: give evaluators a path
The recurring ASPLOS AE failure is a paper whose headline number needs silicon the committee lacks. Acceptable mitigations, in descending order of strength:
- Provide remote access to the platform for the evaluation window (with an anonymity-safe access route if the process requires it).
- Ship the simulator-backed subset as the reproducible core, and mark the silicon results as demonstrably-run (logs + analysis pipeline included).
- Offer a scaled-down proxy (smaller FPGA, reduced workload) with an explicit argument for why the proxy's behavior transfers.
Say which mitigation applies in the appendix, per claim — evaluators score against what you requested, so calibrated requests outperform hopeful ones.
Collaboration protocol
- Expect rounds: evaluators report blockers, authors fix and respond. Reserve maintainer time in the weeks after camera-ready (exact 2027 AE dates: 待核实 — confirm at notification).
- Fix-forward, do not re-argue: an evaluator's confusion is a defect in the README.
- Keep the artifact frozen at a tagged version during evaluation; hotfixes go on a branch the evaluators are told about.
Common evaluator blockers, pre-empted
Field experience across systems AE committees converges on a short list of first-hour failures, all preventable:
- Undeclared credentials or licenses — a workload, simulator model, or dataset that needs a registration the evaluator lacks; declare it in the appendix and provide an alternative path.
- Hidden network assumptions — builds that fetch from internal mirrors or rate-limited hosts; vendor the dependencies or provide the container image.
- Root-only steps without warning — kernel-module or BIOS-adjacent steps must be flagged up front so the evaluator can pick a sacrificial machine.
- Hour-scale first feedback — if the smoke test takes an evening, the first blocker report costs a full round trip; minutes-scale smoke tests keep the collaboration inside the calendar.
- Output the evaluator must interpret — raw logs with no comparator; every claim script should end by printing PASS/FAIL against the tolerance.
Anonymity boundary
AE runs after acceptance, so evaluator-facing materials need not be anonymous — but any artifact pointer placed in the submission itself (an appendix teaser, a footnoted repository) falls under the double-blind rules and must be anonymized end to end: repository owner, commit author strings, container registry paths, and dataset hosting all leak identity. The clean pattern is to keep the submission's artifact story descriptive ("we will submit an artifact covering claims 1-3") and materialize the links only in the Artifact Appendix after notification.
Dry-run protocol
Before submission to the AEC, have a colleague who did not build the artifact execute the README on a clean machine, timing each stage and noting every question they had to ask. Their questions are defects; fix the README, not the colleague. Two such passes typically halve the evaluation rounds.
Output format
[Badges requested] available / functional / reproducible — with rationale
[Appendix status] dependencies / key results / prepare-run-validate all drafted: Y/N
[Smoke test] clean-environment runtime: N min · passes: Y/N
[Claim scripts] one per key result, mapped to figure/table: list
[Hardware path] per silicon-dependent claim: access / sim-subset / proxy
[Archive] DOI-issuing repository chosen + deposit dry-run done: Y/N
Version History
- 9f86f09 Current 2026-07-19 14:19
