Agent Skills
› aiming-lab/AutoResearchClaw
› nlp-pretraining
nlp-pretraining
GitHub提供NLP模型预训练与微调的最佳实践，涵盖优化器配置、参数高效微调方法及评估指标，用于生成或审查相关训练代码。
Trigger Scenarios
生成NLP训练代码
审查NLP训练代码
Install
npx skills add aiming-lab/AutoResearchClaw --skill nlp-pretraining -g -y
SKILL.md
Frontmatter
{
"name": "nlp-pretraining",
"metadata": {
"author": "researchclaw",
"version": "1.0",
"category": "domain",
"priority": "3",
"references": "Devlin et al., BERT, NAACL 2019; Hu et al., LoRA, ICLR 2022",
"trigger-keywords": "language model,pretraining,fine-tuning,bert,gpt,llm,transformer,nlp,text",
"applicable-stages": "9,10"
},
"description": "Best practices for language model pretraining and fine-tuning. Use when generating or reviewing NLP training code."
}
NLP Pretraining/Fine-tuning Best Practice
Fine-tuning recipe:
- Use pre-trained checkpoints (HuggingFace hub)
- AdamW optimizer, lr=2e-5 to 5e-5
- Linear warmup (6% of total steps) + linear decay
- Batch size: 16-32 (use gradient accumulation for larger effective batch)
- 3-5 epochs for classification, 1-2 for generation
- Weight decay: 0.01
Parameter-efficient methods:
- LoRA: r=8-64, alpha=16-128, apply to q/v projections
- Prefix tuning: 10-20 prefix tokens
- Adapters: bottleneck dimension 64-256
Evaluation:
- Classification: accuracy, F1 (macro for imbalanced)
- Generation: perplexity, BLEU/ROUGE, human evaluation
- Use multiple seeds and report mean +/- std
Version History
- e2e23c9 Current 2026-07-25 07:48
