Agent Skills › Prismer-AI/Prismer › ml-experiment

ml-experiment

GitHub

用于设计、实施和评估机器学习实验，涵盖模型训练、基准测试、消融研究及论文复现。支持使用Jupyter执行代码、统计分析及结果可视化，确保实验可复现性与严谨性。

docker/templates/cs-researcher/skills/ml-experiment/SKILL.md Prismer-AI/Prismer

Trigger Scenarios

用户希望训练机器学习模型需要对比不同算法性能运行消融研究以分析组件贡献评估ML模型在特定任务上的表现复现学术论文的实验结果

Install

npx skills add Prismer-AI/Prismer --skill ml-experiment -g -y

More Options

Non-standard path

npx skills add https://github.com/Prismer-AI/Prismer/tree/main/docker/templates/cs-researcher/skills/ml-experiment -g -y

Use without installing

npx skills use Prismer-AI/Prismer@ml-experiment

指定 Agent (Claude Code)

npx skills add Prismer-AI/Prismer --skill ml-experiment -a claude-code -g -y

安装 repo 全部 skill

npx skills add Prismer-AI/Prismer --all -g -y

预览 repo 内 skill

npx skills add Prismer-AI/Prismer --list

SKILL.md

Frontmatter

{
    "name": "ml-experiment",
    "description": "Design and run machine learning experiments with proper evaluation using jupyter_execute, including training, benchmarking, and ablation studies. Use when the user wants to train models, compare algorithms, run ablation studies, evaluate ML performance, or reproduce paper results."
}

ML Experiment Skill

Description

Design, implement, and evaluate machine learning experiments with reproducible workflows, proper baselines, and statistical analysis.

Tools Used

jupyter_execute - Execute ML code in Python (auto-switches to Jupyter)
jupyter_notebook - Manage experiment notebooks
update_notebook - Set up experiment cells
update_latex - Write experiment results to papers
latex_compile - Compile CS conference papers (auto-switches to LaTeX)
arxiv_to_prompt - Read related work from arXiv papers
update_notes - Write experiment logs and analysis summaries

Capabilities

Experiment Design

Proper train/validation/test splits
Cross-validation and bootstrap confidence intervals
Ablation study design
Hyperparameter search (grid, random, Bayesian)

Implementation

PyTorch and TensorFlow model building
Data loading and augmentation pipelines
Training loops with logging and checkpointing
Distributed training setup

Evaluation

Standard metrics per task (accuracy, F1, BLEU, mAP, etc.)
Statistical significance testing (paired t-test, bootstrap)
Comparison with baselines
Error analysis and visualization

Usage Patterns

Run an Experiment

When user says: "Train a model for [task]"

Clarify dataset, metrics, and baselines
Implement data loading and preprocessing
Build model architecture
Train with proper logging
Evaluate and compare to baselines
Report results with confidence intervals

Reproduce a Paper

When user says: "Reproduce [paper title/arXiv ID]"

Fetch paper using arxiv_to_prompt
Extract key method details
Implement core algorithm
Run experiments matching paper setup
Compare results to reported numbers

Tool Examples

Train and evaluate a classifier

# via jupyter_execute
import torch
from sklearn.model_selection import train_test_split
from sklearn.metrics import classification_report

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
# ... train model ...
print(classification_report(y_test, predictions))

Run ablation study

# via jupyter_execute
configs = [
    {"name": "full", "use_augmentation": True, "use_dropout": True},
    {"name": "no_aug", "use_augmentation": False, "use_dropout": True},
    {"name": "no_dropout", "use_augmentation": True, "use_dropout": False},
]
results = {c["name"]: train_and_eval(**c) for c in configs}