Agent Skills
› aiming-lab/AutoResearchClaw
› data-loading
data-loading
GitHub提供PyTorch DataLoader高效加载最佳实践，优化数据管道以防止GPU饥饿。涵盖workers配置、内存锁定、持久化工作进程及大数据集处理策略，帮助提升训练性能。
Trigger Scenarios
设置DataLoader参数
数据预处理优化
解决GPU等待问题
Install
npx skills add aiming-lab/AutoResearchClaw --skill data-loading -g -y
SKILL.md
Frontmatter
{
"name": "data-loading",
"metadata": {
"author": "researchclaw",
"version": "1.0",
"category": "tooling",
"priority": "6",
"references": "PyTorch Data Loading Tutorial, pytorch.org",
"trigger-keywords": "data,loading,dataloader,dataset,preprocessing,augmentation",
"applicable-stages": "10"
},
"description": "Optimize data loading pipeline to prevent GPU starvation. Use when setting up DataLoader or data preprocessing."
}
Efficient Data Loading Best Practice
- Use num_workers = min(8, os.cpu_count()) for DataLoader
- Enable pin_memory=True when using GPU
- Use persistent_workers=True to avoid re-spawning
- Pre-compute and cache transformations when possible
- For image data: use torchvision.transforms.v2 (faster)
- For large datasets: consider memory-mapped files or WebDataset
- Profile with torch.utils.bottleneck to find I/O bottlenecks
Version History
- e2e23c9 Current 2026-07-25 07:48
