解锁效率和性能：在Slack上导航Spark 3和EMR 6升级之旅

Slack Data Engineering recently underwent data workload migration from AWS EMR 5 (Spark 2/Hive 2 processing engine) to EMR 6 (Spark 3 processing engine). In this blog, we will share our migration journey, challenges, and the performance gains we observed in the process. This blog aims to assist Data Engineers, Data Infrastructure Engineers, and Product Managers who may be considering migrating to EMR 6/Spark 3.

Slack数据工程最近将数据工作负载从AWS EMR 5（Spark 2/Hive 2处理引擎）迁移到EMR 6（Spark 3处理引擎）。在这篇博客中，我们将分享我们的迁移之旅、遇到的挑战以及我们在这个过程中观察到的性能提升。本博客旨在帮助考虑迁移到EMR 6/Spark 3的数据工程师、数据基础设施工程师和产品经理。

In Data Engineering, our primary objective is to support internal teams—such as Product Engineering, Machine Learning, and Data Science—by providing essential datasets and a reliable data infrastructure to facilitate the creation of their own datasets. We ensure the reliability and timeliness of critical billing and usage data for our clients. Maintaining Landing Time SLAs (Service Level Agreements) serves as a measure to keep up these promises**.**

在数据工程中，我们的主要目标是支持内部团队，如产品工程、机器学习和数据科学，通过提供基本数据集和可靠的数据基础设施来促进他们创建自己的数据集。我们确保为客户提供可靠和及时的关键计费和使用数据。保持着陆时间 SLA（服务级别协议）是衡量我们遵守这些承诺的标准。

Over time, the rapid expansion of our data volume frequently led to the violation of our critical data pipeline’s SLAs. As we sought alternatives to Spark 2 and Hive 2, Spark 3 emerged as a compelling solution for all our data processing needs, notably due to its Adaptive Query Execution (AQE) feature that could improve performance for some of our skewed datasets. We embarked on this EMR 6/Spark 3 migration due to enhanced performance, enhanced security—with updated log4j libraries—and the potential for significant cost savings.

随着数据量的快速扩大，我们的关键数据流水线的SLA经常被违反。在寻找替代Spark 2和Hive 2的方案时，Spark 3成为了我们所有数据处理需求的一个引人注目的解决方案，尤其是由于其自适应查询执行（AQE）功能，可以提高一些倾斜数据集的性能。我们进行了这次EMR 6/Spark 3迁移，以提升性能、增强安全性（使用更新的log4j库）并实现显著的成本节约。

This year-long project consisted of two major phases:

这个为期一年的项目包括两个主要阶段：

Phase 1: Upgrade EMR from 5.3x to 6.x.
阶段 1：将 EMR 从 5.3x 升级到 6.x。
Phase 2: Upgrade from Hive ...