GEM训练：Meta如何将其LLM规模广告基础模型的效率提升一倍

Meta’s Generative Ads Recommendation Model (GEM), the foundation model behind ads recommendations across Instagram and Facebook, now trains at LLM scale on several thousand of the latest-generation GPUs. This post goes into the details on how we achieved: doubling end-to-end (E2E) training efficiency to 20–25% Model FLOPs Utilization (MFU) while scaling training FLOPs 4x in 12 months, by co-designing kernels, precision, parallelism, networking, and memory together.
Meta的生成式广告推荐模型（GEM），作为Instagram和Facebook广告推荐背后的基础模型，现已在数千块最新一代GPU上以LLM规模进行训练。本文详细介绍了我们如何实现这一目标：通过协同设计kernel、精度、并行策略、网络和内存，将端到端（E2E）训练效率翻倍至20–25%的模型FLOPs利用率（MFU），同时在12个月内将训练FLOPs提升了4倍。
Training GEM presents unique engineering challenges at the intersection of recommendation systems and LLMs as the model combines a hybrid architecture plus recommendations-domain data properties that are unlike typical LLM workloads.
训练 GEM 在推荐系统和 LLM 的交叉领域带来了独特的工程挑战，因为该模型结合了混合架构以及不同于典型 LLM 工作负载的推荐领域数据特性。
AI infrastructure optimized for LLM training (kernels, parallelism, low precision recipes etc.) does not directly transfer, requiring significant innovation and hardware/software co-design to reach LLM-scale training for recommendation models efficiently.
为 LLM 训练优化的 AI 基础设施（内核、并行、低精度配方等）无法直接迁移，需要重大的创新和软硬件协同设计，才能高效实现推荐模型的 LLM 规模训练。
We tackled these challenges through complementary compute efficiency and scaling efficiency innovations:
- Compute efficiency: Achieved through a customized recommendation kernel library — Jagged Flash Attention (JFA), Generalized Dot-Product Attention (GDPA), BlockAttention, etc. — and mixed ultra-low precision training (including MXFP8 attention and MLP) optimized for recommendation workloads, purpose-built to exploit latest generation GPU’s architecture.
  - Scaling efficiency: Topology-aware 5D parallelism with Streaming Multiprocessor (SM)-free collectives — 2D FSDP + Expert Parallelism for dense parameters, combined with Fully Sharded 2D Model Parallelism for sparse parameters — co-designed with Meta’s multi-tiered network hierarchy to redu...