GEM Training: How Meta Doubled the Efficiency of Its LLM-Scale Ads Foundation Model

Gem Training: How Meta Doubled the Efficiency of Its LLM-Scale Ads Foundation Model

摘要

Meta的GEM模型通过定制化内核与超低精度训练，将计算效率翻倍至20-25% MFU，同时采用拓扑感知的5D并行与SM-free通信，在12个月内将训练FLOPs扩展4倍，成功解决了推荐系统与LLM混合架构的规模化挑战。