Agent Skills
› aiming-lab/AutoResearchClaw
› distributed-training
distributed-training
GitHub提供基于PyTorch DDP的多GPU分布式训练最佳实践，涵盖进程组初始化、数据采样、同步批归一化及梯度累积等核心模式。
Trigger Scenarios
需要扩展模型训练以使用多个GPU
配置PyTorch分布式环境
优化大规模数据并行训练
Install
npx skills add aiming-lab/AutoResearchClaw --skill distributed-training -g -y
SKILL.md
Frontmatter
{
"name": "distributed-training",
"metadata": {
"author": "researchclaw",
"version": "1.0",
"category": "tooling",
"priority": "7",
"references": "PyTorch DDP Tutorial, pytorch.org; Goyal et al., Accurate Large Minibatch SGD, 2017",
"trigger-keywords": "distributed,multi-gpu,parallel,ddp,scale",
"applicable-stages": "10,12"
},
"description": "Multi-GPU and distributed training patterns with PyTorch DDP. Use when scaling training across GPUs."
}
Distributed Training Best Practice
- Use DistributedDataParallel (DDP) over DataParallel for multi-GPU
- Initialize process group: dist.init_process_group(backend='nccl')
- Use DistributedSampler for data sharding
- Synchronize batch norm: nn.SyncBatchNorm.convert_sync_batchnorm()
- Only save checkpoint on rank 0
- Scale learning rate linearly with world size
- Use gradient accumulation for effectively larger batch sizes
Version History
- e2e23c9 Current 2026-07-25 07:48
