Agent Skills
› NeverSight/learn-skills.dev
› data-pipeline-manager
data-pipeline-manager
GitHub管理机器学习数据集流水线，涵盖数据获取、预处理、划分及版本控制。用于审计数据质量、防止训练测试污染，确保实验有效性与可复现性，为模型训练提供可靠的数据基础。
Trigger Scenarios
需要获取、预处理或拆分数据集
设计或审计训练/验证/测试划分协议
检查数据质量或训练测试集污染风险
需要记录数据决策以保证实验可复现性
Install
npx skills add NeverSight/learn-skills.dev --skill data-pipeline-manager -g -y
SKILL.md
Frontmatter
{
"name": "data-pipeline-manager",
"description": "Manage ML dataset pipelines before training. Use when the user needs to acquire, preprocess, split, or version datasets, design train\/val\/test protocols, audit data quality, check for train\/test contamination, or make data decisions that affect experimental validity and reviewer trust.",
"allowed-tools": "Read, Write, Edit, Bash, Glob, WebSearch, WebFetch",
"argument-hint": "[project-dir] [--mode audit|split|quality|contamination|versioning|plan] [--dataset <name>]"
}
Data Pipeline Manager
Make deliberate, documented data decisions before training. This skill prevents the most common experimental validity failures: undocumented preprocessing, leaky splits, contaminated test sets, and irreproducible data pipelines.
Use this skill when:
- deciding how to acquire, access, or download a dataset
- designing or auditing train/val/test split protocols
- preprocessing choices need documentation before they affect results
- data quality issues could corrupt baselines or ablations
- contamination between training and evaluation data is a reviewer risk
- dataset versions, seeds, and pipeline steps need to be pinned for reproducibility
- a paper claim depends on a specific data protocol and the protocol is not yet written down
Do not use this skill for running experiments — use run-experiment once the pipeline is validated. Do not use this skill to write result sections — use experiment-story-writer after results exist.
Pair this skill with:
experiment-design-plannerto ensure the split protocol matches the experimental design's evaluation criteriabaseline-selection-auditwhen dataset choice or protocol affects which baselines are comparableresearch-project-memoryto record data decisions as claims, evidence, and risksresult-diagnosiswhen surprising results may trace to a data artifact rather than a method failurepaper-evidence-boardwhen data protocol details need to appear in the paper's methods and evidence slotsinit-python-projectwhen the project'sdata/layout and pipeline scripts need to be scaffolded
Skill Directory Layout
<installed-skill-dir>/
├── SKILL.md
├── references/
│ ├── split-protocols.md
│ └── contamination-guide.md
└── templates/
└── pipeline-plan.md
Progressive Loading
- Always read
references/split-protocols.mdwhen designing or auditing split protocols. - Read
references/contamination-guide.mdwhen handling LLM evaluation, pre-trained model probing, retrieval benchmarks, or any setting where the model may have seen test-adjacent data. - Read
templates/pipeline-plan.mdbefore writing a pipeline plan. - Read
memory/claim-board.mdandmemory/evidence-board.mdwhen data decisions need to be linked to paper claims. - Read
code/.agent/benchmark-plan.mdor equivalent when a benchmark protocol already exists.
Core Principles
- Treat data decisions as experimental controls, not implementation details.
- Document every preprocessing step before it runs, not after results appear.
- Design splits from the evaluation goal backward: what question must the test set answer?
- Never let the model, hyperparameter search, or any paper-motivated decision touch the test set.
- Contamination is the paper's responsibility: assume reviewers will ask, and have an answer.
- Version the processed dataset alongside the code that created it.
- Seed-fix every shuffle, sampling step, and augmentation that affects reproducibility.
- Data quality issues found after training are ten times more expensive than those found before.
Step 1 — Gather Context
Locate:
- project root: look for
data/,datasets/,code/data/, or user-provided path - existing pipeline scripts:
data_utils.py,preprocess.py,scripts/prepare_data.sh, or equivalent - existing documentation:
docs/data/,data/README.md, or any dataset description - experiment design:
code/.agent/benchmark-plan.md,memory/claim-board.md - paper methods draft:
paper/sections/experiments.texor equivalent when present
Record:
- what dataset(s) are in use or planned
- whether a split already exists
- what preprocessing is already applied or planned
- whether any data quality audit has been done
- whether contamination has been checked
If nothing exists, produce a draft pipeline plan and mark uncertain fields as TBD.
Step 2 — Select Pipeline Mode
Choose based on the user's request or the current gap:
audit: review an existing pipeline for completeness, reproducibility, and reviewer risksplit: design or validate the train/val/test split protocolquality: run or plan a data quality audit (missing values, label noise, distribution shifts, class imbalance)contamination: check for data leakage between splits and pre-training / retrieved dataversioning: pin dataset version, hashes, download scripts, and processing seedsplan: create a full pipeline plan covering all of the above for a new dataset
Default: use audit when a pipeline exists, plan when none exists.
Step 3 — Design or Audit the Split Protocol
Read references/split-protocols.md.
For every dataset split, record:
- Split: train | val | test | held-out
- Size: N examples (% of total)
- Stratification: class-stratified | time-ordered | random | grouped | domain-separated
- Overlap policy: none | explicit-allowed (state why)
- Leakage controls: [list controls]
- Assignment seed: <integer>
- Purpose: what hypothesis or metric this split answers
- Forbidden use: what must NOT touch this split before final evaluation
Flag any split designed after the method was partially tuned — this is a reviewer-visible risk. Flag any split where the validation set was used as a proxy test set.
Step 4 — Data Quality Audit
Check:
- Missing values: are they meaningful (masked) or errors? What is the imputation or exclusion policy?
- Label noise: known label error rate, labeling protocol, inter-annotator agreement if applicable
- Class imbalance: imbalance ratio, whether baselines handle it equivalently
- Distribution shift: between train and test, or between this dataset and cited prior work
- Duplicate or near-duplicate examples: deduplication policy and its effect on split boundaries
- Outliers: what is excluded and on what criteria — is this reproducible?
- Data leakage from preprocessing: normalization fit on train only; no augmentation at test time
For each finding, classify as:
blocker: will cause invalid results or comparison failuresrisk: should be disclosed in the paper's limitations or experiment settingsnote: worth tracking but not blocking
Step 5 — Contamination Check
Read references/contamination-guide.md.
For LLM evaluation, retrieval, or pre-trained model probing:
- Check whether benchmark test examples appear in common pretraining corpora (e.g., via n-gram overlap, exact hash, or known contamination audits)
- Check whether model selection used test performance directly or indirectly
- Check whether retrieved context at test time could contain test-set answers
- Check whether data augmentation generated test-adjacent examples
For standard supervised tasks:
- Confirm no sample appears in both train and test after deduplication
- Confirm that grouping or stratification respects natural data boundaries (e.g., speaker IDs, document IDs, patient IDs)
- Confirm that any held-out reserve set was never used for threshold selection or model comparison
For every contamination finding, record the severity and the available mitigation.
Step 6 — Version and Reproducibility Record
For reproducible experiments, the pipeline plan must include:
- Dataset name and version: <name>==<version> or commit/hash
- Download source: <url or repo> (verified checksum or hash)
- Raw data hash: sha256=<hash>
- Processing script: <path/to/script> at git commit <sha>
- Processing seed: <integer> for every random step
- Processed data hash: sha256=<hash> of final splits
- Storage location: <path or remote>
- Excluded samples: <rule and count>
- Known issues: <list>
If any of these fields cannot be filled, mark as TODO and add to memory/action-board.md.
Step 7 — Write the Pipeline Plan
Use templates/pipeline-plan.md.
Save to:
code/.agent/data-pipeline-plan.md
If no code/ directory exists, save to:
.agent/data-pipeline-plan.md
When updating, preserve stable decisions and add a compact change note.
Step 8 — Memory Writeback
After completing a pipeline plan or audit:
- Write data decisions that affect claims to
memory/claim-board.md - Write contamination risks to
memory/risk-board.md - Write unresolved data quality issues as actions to
memory/action-board.md - Write evidence about dataset characteristics to
memory/evidence-board.mdwhen they are cited in the paper - Update
memory/current-status.mdwhen the dataset version or split protocol changes
Step 9 — Route Follow-Ups
experiment-design-planner: align experimental design with the finalized split protocolbaseline-selection-audit: recheck baseline comparability after split or preprocessing changesresult-diagnosis: diagnose surprising results that may trace to a data issuepaper-evidence-board: add dataset protocol details to the paper's evidence slotsinit-python-project: scaffolddata/layout and pipeline scripts
Final Sanity Check
Before finalizing:
- every split has a documented purpose and a forbidden-use policy
- every preprocessing step is reproducible with a fixed seed and a version-pinned script
- data quality findings are classified as blockers, risks, or notes
- contamination checks are explicitly done or explicitly deferred with a risk entry
- dataset version, hashes, and download source are recorded or marked TODO
- memory writeback covers affected claims, risks, and actions
- next action is clear
Version History
- e0220ca Current 2026-07-05 21:34
