Agent Skills
› aiming-lab/AutoResearchClaw
› experimental-design
experimental-design
GitHub提供可复现的机器学习实验设计规范，涵盖基线选择、随机种子使用、消融实验设计及变量控制等最佳实践。
Trigger Scenarios
规划机器学习实验
设计消融研究
设置对照组
Install
npx skills add aiming-lab/AutoResearchClaw --skill experimental-design -g -y
SKILL.md
Frontmatter
{
"name": "experimental-design",
"metadata": {
"author": "researchclaw",
"version": "1.0",
"category": "experiment",
"priority": "2",
"references": "Bouthillier et al., Accounting for Variance in ML Benchmarks, MLSys 2021",
"trigger-keywords": "experiment,ablation,baseline,control,hypothesis,reproducib",
"applicable-stages": "9,10,12"
},
"description": "Best practices for designing reproducible ML experiments. Use when planning ablations, baselines, or controlled experiments."
}
Experimental Design Best Practice
- ALWAYS include meaningful baselines (not just random):
- At least one classical method baseline
- At least one recent SOTA method baseline
- A simple-but-strong baseline (e.g., linear probe, k-NN)
- Use MULTIPLE random seeds (minimum 3, ideally 5)
- Report mean +/- std across seeds
- Design ablations that isolate EACH key component:
- Remove one component at a time
- Each ablation must be meaningfully different from baseline
- Control variables: change only ONE thing per comparison
- Use standard splits (train/val/test) — never test on training data
- Report wall-clock time and memory usage alongside accuracy
Version History
- e2e23c9 Current 2026-07-25 07:48
