Agent Skills
› NeverSight/learn-skills.dev
› compute-budget-planner
compute-budget-planner
GitHub在ML实验提交前估算GPU计算预算，用于规划训练成本、设计烟雾测试、评估消融实验矩阵及优化资源配置，防止算力浪费并确保实验符合可用资源限制。
Trigger Scenarios
计划训练运行前的成本未知
需要为烟雾测试设定合理规模
评估消融实验矩阵的整体开销
寻找更经济的模型或数据替代方案
受限计算资源下的任务优先级排序
Install
npx skills add NeverSight/learn-skills.dev --skill compute-budget-planner -g -y
SKILL.md
Frontmatter
{
"name": "compute-budget-planner",
"description": "Estimate GPU compute budget before running ML experiments. Use when planning how much compute an experiment, ablation matrix, or sweep will cost, sizing smoke tests, finding cheaper alternatives, or deciding whether a planned run fits available resources.",
"allowed-tools": "Read, Write, Edit, Bash, Glob",
"argument-hint": "[project-dir] [--mode estimate|smoke|ablation|compare] [--model <model-size>]"
}
Compute Budget Planner
Estimate and control compute spend before jobs are submitted. This skill prevents GPU-hour waste from under-scoped smoke tests, over-specified ablations, and jobs that could have been half the cost with a smarter design.
Use this skill when:
- a planned training run's cost is unknown before submission
- a smoke test needs to be sized to give signal without wasting a full GPU-day
- an ablation matrix needs to be costed before scheduling all variants
- a cheaper model variant, shorter sequence, or smaller dataset could answer the same question
- available compute is limited and job priority needs to be set
- a paper's experimental budget needs to be disclosed or defended to reviewers
Do not use this skill to submit jobs — use run-experiment after sizing is done. Do not use this skill for experiment design — use experiment-design-planner first.
Pair this skill with:
experiment-design-plannerto align experiment scope with compute budget before design is lockedrun-experimentto submit the sized job with the correct resource specbaseline-selection-auditwhen baseline replication costs need to be included in the total budgetresearch-project-memoryto record compute decisions as evidence or risk items
Skill Directory Layout
<installed-skill-dir>/
├── SKILL.md
└── references/
└── estimation-guide.md
Progressive Loading
- Always read
references/estimation-guide.md. - Read existing job configs in
jobs/,configs/, orexperiments/when a previous run exists. - Read
memory/evidence-board.mdwhen the compute estimate is tied to a paper claim.
Core Principles
- Estimate before submitting, not while waiting for the job to time out.
- A smoke test that uses less than 5% of the full job cost should be the default first step.
- Compute cost compounds: one model × one dataset × one seed is the atom; ablations multiply.
- Cheaper alternatives should always be explored before committing to expensive runs.
- Reviewer trust is higher when compute is disclosed and reproducibility is feasible on standard hardware.
- Underestimating compute is a submission risk: a result that cannot be reproduced because the hardware is unavailable is a reviewer concern.
Step 1 — Gather Training Context
Collect:
- model name and parameter count (or approximate FLOPs if known)
- dataset size: number of training examples, sequence length or image size
- planned training duration: epochs or steps
- batch size and gradient accumulation steps
- hardware: GPU type (A100, H100, V100, RTX 3090), VRAM, number of GPUs
- framework and mixed precision: FP16, BF16, or FP32
- existing throughput: samples/second or tokens/second from a prior run (if available)
If no prior run exists, use references/estimation-guide.md to estimate throughput.
Step 2 — Estimate Full Run Cost
Compute:
total_steps = ceil(dataset_size / batch_size) × epochs
or planned_steps (if step-based)
gpu_hours = total_steps × seconds_per_step / 3600 × num_gpus
Report:
- estimated GPU-hours (single GPU equivalent)
- wall-clock time with the planned number of GPUs
- whether the run fits in the available time limit (e.g., SLURM 24h, RunAI policy)
If throughput is unknown, provide a range using the estimation guide's reference values for the model class and hardware.
Step 3 — Size the Smoke Test
A smoke test should:
- use ≤ 5% of the full run's compute (ideally 1–2%)
- complete within 10–30 minutes on one GPU
- produce a loss curve with a clear downward trend (proves the code runs correctly)
- exercise the full data pipeline, model forward/backward, checkpointing, and logging
Recommend:
Smoke test steps: max(100, 0.01 × total_steps)
Smoke dataset: subsample 1% of training data, or use 1 batch per shard
Expected smoke wall-clock: < 30 min on 1× GPU
Flag if the smallest feasible smoke run exceeds 30 minutes. This is a sign the setup overhead is too high.
Step 4 — Cost the Ablation Matrix
For each ablation dimension, record:
| Variant | Changed variable | Cost multiplier | GPU-hours | Priority |
|---|---|---|---|---|
| Baseline | — | 1× | required | |
| Ablation A | [removed component] | 1× | required | |
| Ablation B | [changed config] | 0.5× | should-have | |
| Sweep C | [hyperparameter] | 3× | optional |
Classify each variant as:
required: paper claim depends on this comparisonshould-have: reviewer will likely ask; defensible to omit with a reasonoptional: nice-to-have; drop first if compute is limited
Report:
- total required GPU-hours
- total should-have GPU-hours
- total optional GPU-hours
- recommended minimum run set
Step 5 — Identify Cheaper Alternatives
Check whether any of the following can answer the same question at lower cost:
- Smaller model / fewer parameters: Can a 125M model prove the concept before scaling to 1B?
- Shorter sequences / smaller images: Does the method work at half resolution?
- Fewer training steps: Can early stopping or learning rate warmup give a valid comparison at 50% of steps?
- Smaller dataset: Can a 10% subsample reproduce the trend with ±5% relative error?
- Lower-rank approximation: Can LoRA or adapter training substitute for full fine-tuning for this ablation?
- Faster hardware: Can an A100 replace 4× V100s and finish in one-quarter the wall-clock time?
For each alternative, record the expected signal quality and the risk of a misleading result.
Step 6 — Write the Budget Plan
Save to:
code/.agent/compute-budget.md
The plan should include:
- full run estimate (GPU-hours and wall-clock)
- smoke test spec (steps, data fraction, expected wall-clock)
- ablation matrix with priority and cost
- cheaper alternatives considered and rejected or accepted
- available resource envelope and whether budget fits
- recommended submission order: smoke → required ablations → should-have → optional
Memory Writeback
- If compute cost affects a paper claim's feasibility, add to
memory/risk-board.md - If a cheaper alternative is accepted and changes the experimental design, update
memory/decision-log.md - Update
memory/evidence-board.mdwhen compute details need to be cited in the paper (hardware, training duration, GPU-hours)
Final Sanity Check
Before finalizing:
- smoke test spec is under 30 minutes on 1 GPU
- full run estimate fits available time limit
- ablation matrix has priorities assigned
- cheaper alternatives were considered
- required GPU-hours are within the available resource envelope or a risk entry is filed
Version History
- e0220ca Current 2026-07-05 21:34
