华为机器翻译模型训练推理加速实践

如果无法正常显示，请先停止浏览器的去广告插件。

1. 华为机器翻译架构和模型加速魏代猛华为2012 / 机器翻译实验室

3. 个人简介 - 华为高级技术专家，机器翻译算法负责人，产品落地华为云、HMS、华为手机等 - 北京大学硕士，研究方向：机器翻译、同传翻译、语义理解等 - 带领团队参加 WMT20/21/22 news、biomedical、efficiency等赛道多项第一， IWSLT 22 多项第一， WAT20比赛多项第一 - 在AAAI，ACL，EMNLP，ICASSP等发表论文30+

4. 大纲 • 机器翻译简介 • 模型推理问题 • 端测推理加速 • 华为机器翻译 • 总结

5. 机器翻译简介 1、主流的机器翻译模型包含Encoder和Decoder两部分，Encoder将原文整个序列编码成一个多维向量，Decoder将原文序列的向量解码成译文。 2、Attention模型记录原文和译文的词对齐关系，指导机器翻译在解码译文某个词时，应该更关注与原文的哪一个部分，以提升长句翻译质量。

6. 机器翻译简介 Encoder：6 Decoder：6 Hidden size：1024 参数量：2亿大小：800M Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser and Illia Polosukhin. “Attention is All you Need” Neural Information Processing Systems (2017).

7. 模型推理问题 Transformer模型在GPU，CPU，ARM运行典型值耗时 GPU/T4 CPU/Intel ARM 45 ms/token 150 ms/token - 模型太大，计算量太大端侧最具挑战

8. 模型推理问题让模型变小大小存储计算量质量质量速度

9. 大纲 • 机器翻译简介 • 模型推理问题 • 端测推理加速 • 华为机器翻译 • 总结

10. 端侧推理加速小模型 → 质量变差计算量小模型存储小小模型 → 高质量？ Geoffrey E. Hinton, Oriol Vinyals and Jeffrey Dean. “Distilling the Knowledge in a Neural Network” arXiv: Machine Learning (2015): n. pag. Yoon Kim and Alexander M. Rush. “Sequence-Level Knowledge Distillation” Empirical Methods in Natural Language Processing (2016). Markus Freitag, Yaser Al-Onaizan and Baskaran Sankaran. “Ensemble Distillation for Neural Machine Translation” arXiv: Computation and Language (2017): n. pag.

11. 端侧推理加速知识蒸馏 96% 小模型 & 高质量 Tiny Bert Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. TinyBERT: Distilling BERT for Natural Language Understanding. EMNLP 2020

12. 端侧推理加速小模型：更小空间，更快的推理 ------模型压缩，低精度推理增加量化层，FP32->Int8->FP32 ， E2E训练 Benoit Jacob, Skirmantas Kligys, Bo Chen, Menglong Zhu, Matthew Tang, Andrew Howard, Hartwig Adam and Dmitry Kalenichenko. “Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference” Computer Vision and Pattern Recognition (2018).

13. 端侧推理加速小模型：更小空间，更快的推理 ------模型压缩，低精度推理 Log0值问题 4 bitlog 4bit 后期前期 Alham Fikri Aji and Kenneth Heafield. 2020. Compressing Neural Machine Translation Models with 4-bit Precision. In Proceedings of the Fourth Workshop on Neural Generation and Translation, pages 35–42, Online. Association for Computational Linguistics.

14. 端侧推理加速小模型：更小空间，更快的推理 ------模型压缩，低精度推理模型 32 bit 8 bit 4 bit Log 4 bit( 后期 ) BLEU(WMT14) 26.5 26.4(-0.1) 24.3(-2.2) 25.1(-1.4) Parameter Size 260M 66M 35M 35M Log 4 bit( 前期 ) 26.2(-0.3) 35M 直接4bit 影响大，log 4 bit前期介入量化训练很关键

15. 端侧推理加速小模型：更小空间，更快的推理 ------模型压缩，低精度推理 Int8推理模型中计算量最大的是矩阵运算(GEMM) Int8推理：用整型运算代替浮点型运算提速处理好量化和反量化是提速的关键华为Noah高性能推理实验室 https://github.com/huawei-noah/bolt Operator FC Transpose Eltwise MatMul Softmax LayerNorm Reshape Activation Embedding Slice Multiply Total FP32 4.806 0.4158 0.198 0.1134 0.063 0.063 0.027 0.0216 0.018 0.009 0.0054 5.94 INT8 1.53 0.228 0.13 0.068 0.040 0.036 0.013 0.011 0.010 0.003 0.001 2.1 Daya Shanker Khudia, Jianyu Huang, Protonu Basu, Summer Deng, Haixin Liu, Jongsoo Park and Mikhail Smelyanskiy. “FBGEMM: Enabling High-Performance Low-Precision Deep Learning Inference..” arXiv: Learning (2021): n. pag.

16. 端侧推理加速小模型：更强的能力 ------结构优化，参数共享，多语言模型 tgt_t3 能否变小？ src_t1, src_t2, … tgt_t1, tgt_t2 Encoder 25-40层，decoder3层 Qiang Wang, Bei Li, Tong Xiao, Jingbo Zhu, Changliang Li, Derek F. Wong and Lidia S. Chao. “Learning Deep Transformer Models for Machine Translation..” Meeting of the Association for Computational Linguistics (2019).

17. 端侧推理加速小模型：更强的能力 ------结构优化，参数共享，多语言模型相邻层共享最好

18. 端侧推理加速小模型：更强的能力 ------结构优化，参数共享，多语言模型 10个语种放一个模型 30个语种模型容量不够 Melvin Johnson, Mike Schuster, Quoc V. Le, Maxim Krikun, Yonghui Wu, Zhifeng Chen, Nikhil Thorat, Fernanda B. Viégas, Martin Wattenberg, Greg S. Corrado, Macduff Hughes and Jeffrey Dean. “Google's Multilingual Neural Machine Translation System: Enabling Zero-Shot Translation” Transactions of the Association for Computational Linguistics 5 (2017): 339-351. Naveen Arivazhagan, Ankur Bapna, Orhan Firat, Dmitry Lepikhin, Melvin Johnson, Maxim Krikun, Mia Xu Chen, Yuan Cao, George Foster, Colin Cherry, Wolfgang Macherey, Zhifeng Chen and Yonghui Wu. “Massively Multilingual Neural Machine Translation in the Wild: Findings and Challenges” arXiv: Computation and Language (2019): n. pag.

19. 小结策略知识蒸馏量化推理模型结构参数共享多语言 ShortList Decoder 结构质量 √ - √ √ √ 速度 √ √ √ - - 专注于减少计算量大小 - √ - √ √

20. 端侧推理加速 h: 512 voc_size: 32000 Encoder voc_size ShortList优化 h Decoder 每个词：100候选 16词→300候选(去重) input 10 x FFN 512x32000 → 512x300 词对齐可以用Fastalign，每个词75候选

21. 端侧推理加速 Decoder结构 푆 :� 2 � � � → � � � � Yann N. Dauphin, Angela Fan, Michael Auli and David Grangier. “Language modeling with gated convolutional networks” International Conference on Machine Learning (2017). Biao Zhang, Deyi Xiong and Jinsong Su. “Accelerating Neural Transformer via an Average Attention Network” Meeting of the Association for Computational Linguistics (2018).

22. 端侧推理加速 Decoder结构 LSTM SRU SRU++ Tao Lei, Yu Zhang, Sida I. Wang, Hui Dai, Yoav Artzi, Simple Recurrent Units for Highly Parallelizable Recurrence, EMNLP 2017 Tao Lei. 2021. When Attention Meets Fast Recurrence: Training Language Models with Reduced Compute. Association for Computational Linguistics 2021.

23. 端侧推理加速 Decoder结构 SRU++ AASRU Yann N. Dauphin, Angela Fan, Michael Auli and David Grangier. “Language modeling with gated convolutional networks” International Conference on Machine Learning (2017). Hengchao Shang, Ting Hu, Daimeng Wei, HW-TSC’s Submission for the WMT22 Efficiency Task. In Proceedings of the Seventh Conference on Machine Translation (WMT), pages 677–681, ACL 2022

24. 小结策略知识蒸馏量化推理模型结构参数共享多语言 shortlist Decoder 结构质量 √ - √ √ √ - - 速度 √ √ √ - - √ √ 大小 - √ - √ √ - -

25. 小结 WMT22 Efficiency Task Hengchao Shang, Ting Hu, Daimeng Wei, Zongyao Li, Xianzhi Yu, Jianfei Feng, Ting Zhu, Lizhi Lei, Shimin Tao, Hao Yang, Ying Qin, Jinlong Yang, Zhiqiang Rao, and Zhengzhe Yu. 2022. HW-TSC’s Submission for the WMT22 Efficiency Task. In Proceedings of the Seventh Conference on Machine Translation (WMT), pages 677–681, ACL 2022 Kenneth Heafield, Biao Zhang, Graeme Nail, Jelmer Van Der Linde, and Nikolay Bogoychev. 2022. Findings of the WMT 2022 Shared Task on Efficient Translation. In Proceedings of the Seventh Conference on Machine Translation (WMT), pages 100–108, Abu Dhabi, United Arab Emirates (Hybrid). ACL 2022.

26. 大纲 • 机器翻译简介 • 模型推理问题 • 端测推理加速 • 华为机器翻译 • 总结

27. 华为机器翻译

28. 华为机器翻译

29. 总结 1. 以业务为轴心，按场景优化 2. 立足深度学习不断变化的大背景 3. 结合自身优势，不断迭代策略知识蒸馏量化推理模型结构参数共享多语言 shortlist Decoder 结构 GPU - √ √ - √ - - CPU - √ √ - √ √ - ARM √ √ √ √ √ √ √

30.

31.