uist-reproducibility
GitHub指导UIST论文实现可复现性，涵盖构建（软硬件参数）、测量（协议与自动化评估）及研究（用户实验细节）三大维度。要求精确记录硬件、代码版本、环境条件及原始日志，确保其他实验室能重建系统并复现结果。
Trigger Scenarios
Install
npx skills add brycewang-stanford/Awesome-Journal-Skills --skill uist-reproducibility -g -y
SKILL.md
Frontmatter
{
"name": "uist-reproducibility",
"description": "Use when making a UIST paper's results replicable — reporting implementation parameters and measurement protocols so a lab could rebuild the system, specifying hardware down to parts and calibration, logging technical evaluations deterministically, and writing honest availability statements for interface systems."
}
UIST Reproducibility
Reproducibility at UIST means something different from rerunning a training script: the question is whether a competent lab could rebuild the artifact and reproduce its measured behavior. That decomposes into three ledgers — build, measurement, and study — and most UIST papers under-specify a different one than they think. UIST posts no reproducibility checklist (none found for 2026 — 待核实), so this discipline is self-imposed and reviewer-enforced.
The build ledger: could they make one?
Everything the system's behavior depends on, pinned:
- Software: exact framework and driver versions, OS, lockfiles; for anything learned, model checkpoints and training data provenance.
- Hardware: part numbers (not "an IMU" but which IMU), mechanical tolerances that matter, firmware version, and the calibration routine with expected outputs.
- Environment: the physical conditions the system assumes — lighting range, acoustic environment, mounting geometry, surface materials.
- Magic numbers: every threshold, filter coefficient, debounce window, and gain, with how each was set (tuned by hand? on which data?). These constants are where re-implementations actually fail.
The measurement ledger: could they get your numbers?
A latency or accuracy figure is reproducible only with its protocol:
| Reported number | Must be pinned |
|---|---|
| Latency | Measurement boundary (sensor-to-photon? software-only?), instrument, event count, load conditions |
| Recognition accuracy | Dataset splits, per-user vs pooled, session separation, chance level |
| Tracking error | Ground-truth apparatus and its own accuracy, spatial sampling grid |
| Throughput / bitrate | Task, phrase set or corpus, session and rest structure |
| Power / weight / cost | Configuration measured, currency and date for cost |
Automate the protocol: a measurement harness checked into the supplement converts
"trust me" into "run this" (see uist-artifact-evaluation for packaging).
# eval/rerun.sh — regenerate every reported number from raw logs
python analyze_latency.py logs/latency_10k.jsonl --out tables/table1.csv
python analyze_accuracy.py logs/study/ --split per-user --seed 17 --out tables/table2.csv
diff -u tables/table1.csv paper_tables/table1.csv # drift check against the PDF
Log at the event level with timestamps and raw sensor values, not just computed
outcomes — future you, rebuttal you (see uist-author-response), and replicating
labs all consume the same logs.
The study ledger: could they rerun the human part?
For any user evaluation: full task instructions and stimuli, counterbalancing scheme, practice/rest structure, apparatus placement (photograph it), inclusion/exclusion criteria, compensation, and the analysis scripts from raw logs to reported statistics. Share instruments even when raw human data cannot leave the IRB envelope — protocol transparency and data availability are separable, and saying so precisely is the honest move.
The availability statement
Write one even though UIST does not require a template, and make it specific:
GOOD: "Firmware, PCB design files, BOM, and the measurement harness are at
<archive-DOI> (tag: as-published). The gesture corpus (14 of 16
participants consented to release) is included; per-participant raw
video is withheld under IRB #—. The two commercial tracking SDKs
required are named in BUILD.md with tested versions."
BAD: "Code available upon reasonable request."
Papers with learned components inherit ML reporting norms too — seeds, training
configs, compute — and can borrow the shared kit's checklists (see
../../resources/code/README.md).
Learned components inherit ML norms
When the system embeds recognition or generation models, the ML reporting conventions stack on top of the systems ledgers:
- Seeds, splits, and training configs pinned; per-user vs pooled evaluation made explicit (interaction data is brutally user-dependent, and pooled splits inflate accuracy).
- For third-party or hosted models: exact model identifiers and versions, the full prompt set, decoding parameters, and dated transcripts — hosted models drift, so the logged behavior is the only permanent record of what reviewers saw.
- Report the interactive costs alongside accuracy: per-inference latency on the deployment hardware, and per-session cost for metered APIs; a technique that is reproducible but unaffordable to run is only half-replicable.
- The shared kit's checklists cover this lane; the adapter in
resources/code/README.mdscopes what it can and cannot check.
The one-page REPRO.md
Compress the three ledgers into a single file at the archive root:
# Reproducing <SystemName> (paper §6-7)
## Rebuild parts: BOM.csv · firmware: v2.3 · calibration: docs/calib.md
## Environment tested: indoor 200-800 lux, 18-26°C · assumes: mounted per Fig 4
## Constants thresholds in config.yaml — tuned on pilot data (n=4), §5.2
## Remeasure eval/rerun.sh regenerates Tables 1-2 from logs/ (or raw capture)
## Study instruments/ · counterbalancing: latin square, §7.1 · IRB #—
## Known drift accuracy drops outdoors (§9); p95 latency sensitive to BLE stack
The "known drift" line is the credibility multiplier: it tells replicators you know where the envelope ends before they find out.
Replication drift and its uses
When a rebuilt system misses the paper's numbers, the causes rank: unstated environmental assumptions, hand-tuned constants, part substitutions, then genuine bugs. Pre-empt the first two by stating the envelope and the tuning story in the paper body — it costs three sentences and buys the paper years of credibility (and Lasting Impact eligibility runs on decade-scale credibility).
Output format
[Build ledger] pinned / gaps: <software · hardware · environment · constants>
[Measurement ledger] protocols pinned for <k>/<n> reported numbers
[Study ledger] instruments · counterbalancing · analysis scripts — present?
[Availability statement] drafted? honest about withholdings?
[Top drift risk] <the unstated assumption most likely to break replication>
Version History
- 9f86f09 Current 2026-07-19 17:51
