Agent Skills
› brycewang-stanford/Awesome-Journal-Skills
› corl-reproducibility
corl-reproducibility
GitHub指导CoRL机器人学习论文实现可复现性，区分软件（训练/仿真）与硬件部分。要求固定版本、记录配置、发布数据权重及评估脚本，并详细文档化硬件设置以支持审计和重建实验环境。
Trigger Scenarios
用户需要复现或发表机器人学习领域的学术论文
用户希望整理并发布模型检查点、训练配置或演示数据
Install
npx skills add brycewang-stanford/Awesome-Journal-Skills --skill corl-reproducibility -g -y
SKILL.md
Frontmatter
{
"name": "corl-reproducibility",
"description": "Use when making a CoRL robot-learning paper reproducible — pinning simulator and driver versions, releasing training configs, demonstration data and checkpoints, documenting hardware setups that cannot be rerun, seed policy, evaluation scripts, and honest availability statements for code, data, and robot platforms."
}
CoRL Reproducibility
Robot-learning papers have a split reproducibility problem: the training half is software and can in principle be rerun anywhere, while the hardware half is a physical setup nobody else owns. Strong CoRL papers treat these halves differently — the software half is made rerunnable, the hardware half is made auditable — and say plainly which is which.
The rerunnable half: simulation and training
The silent reproducibility killers in this field are version-shaped:
- Simulator versions change physics. Contact solvers, default damping, and collision margins shift between releases of MuJoCo, Isaac, PyBullet, and friends; a policy's success rate is a function of the simulator build. Pin the exact version and any physics-relevant flags.
- Environment wrappers drift. Task-suite repositories (benchmark forks, custom reward shims) move under you; record the commit hash of every env repo, not just your own.
- GPU nondeterminism. cuDNN autotuning and atomics make bit-identical training runs unrealistic — so define reproducibility at the distribution level: same configs + fresh seeds should land inside your reported seed spread.
# repro-manifest.yaml — ship at the repo root; one block per results table
table_3:
code_commit: "" # your repo @ exact hash
env_suite: {repo: "", commit: "", sim: "mujoco==X.Y.Z", physics_flags: []}
training: {config: "configs/table3.yaml", seeds: [0,1,2,3,4], gpu: "1xA100-80GB", hours: 14}
data: {demos: "dataset-v2 (1,204 episodes)", url_or_status: "", license: ""}
checkpoints: {released: true, path: "ckpts/table3/", selection_rule: "last epoch, no eval peeking"}
evaluation: {script: "eval/run.py", episodes_per_task: 50, init_state_list: "eval/states.json"}
expected_tolerance: "per-task success within ±4 pts of Table 3 mean"
The selection_rule line matters more than it looks: checkpoint selection via
test-set peeking is the field's quiet irreproducibility engine, and stating the
rule is the cheapest credibility purchase available.
The auditable half: hardware
Nobody will re-run your robot, so the goal is that a skeptical expert could verify the experiment happened as described and rebuild an equivalent rig:
| What to document | Why an auditor needs it |
|---|---|
| Robot model, end-effector, firmware/driver versions | Behavior differs across firmware, not just robots |
| Sensor models, mounting poses, calibration procedure | Camera extrinsics silently dominate visuomotor results |
| Control interface: frequency, action space, safety filters | "30 Hz end-effector deltas" vs "torque control" are different papers |
| Deployed compute + inference latency | Policy behavior is latency-dependent |
| Scene inventory: objects (make/size), fixtures, lighting | Enables an equivalent-rig rebuild and honest comparison |
| Raw episode logs and unedited evaluation video | The audit trail for every printed success rate |
Log every evaluation episode at capture time (timestamped video plus a CSV of outcomes); retrofitting an audit trail after reviews ask for it is impossible.
Data and checkpoint release
- Demonstration datasets are results: release episode counts, collection method (teleop rig, scripted, crowdsourced), operator count, and filtering rules.
- Released checkpoints let others reproduce evaluation even when training is too expensive to repeat — for large policies this is often the highest-value artifact you can ship.
- If data or weights cannot be released (proprietary platform, human-subject footage), say so in the paper with the reason, and release what remains: configs, eval scripts, sim environments, and metrics logs. A precise partial-release statement outperforms a vague "code available upon request."
During review vs after acceptance
- During review everything must be anonymous (see
corl-submission): anonymized repo mirrors, no lab-identifying video, no cloud buckets with named projects. - The 2026 camera-ready pipeline has a sharp constraint: PMLR does not accept video as supplementary material, so post-acceptance videos, code, and data live on external hosting (project site, GitHub, archive), linked from the main text (corl.org author instructions, read 2026-07-08). Plan the public homes of artifacts before the October camera-ready deadline, and prefer DOI-stamped archives for anything you cite as permanent.
- No formal reproducibility checklist was verified for the 2026 cycle in this pack (待核实) — but reviewer expectations enforce one informally; use the manifest above regardless of what the form requires.
Availability statement patterns
Strong: "Code, training configs, evaluation scripts, and the 1,204-episode
teleop dataset: <URL>. Checkpoints for Tables 2-4: <URL>. Hardware
evaluations are documented in Appendix C (rig spec, logs, uncut video);
the platform itself cannot be redistributed."
Weak: "Code will be released upon acceptance." (unverifiable promise)
Broken: "Results reproducible with standard settings." (no artifact at all)
Pre-submission audit
[ ] repro-manifest present; one block per headline table
[ ] Simulator/env/driver versions + commits pinned everywhere
[ ] Seed policy stated; checkpoint selection rule stated
[ ] Hardware rig spec complete enough for an equivalent rebuild
[ ] Episode-level logs + uncut eval video archived internally
[ ] Data/checkpoint release plan with named blockers, if any
[ ] Anonymous during review; external hosting plan ready for camera-ready
Verify the live cycle's supplementary and camera-ready rules at https://www.corl.org/contributions/instruction-for-authors before promising any artifact channel — PMLR-side constraints and CoRL-side forms both change yearly.
Version History
- 9f86f09 Current 2026-07-19 14:57
