Uber: 通过GC调优提升Presto的可靠性

Uber uses open-source Presto to query nearly every data source, both in motion and at rest. Presto’s versatility empowers us to make intelligent, data-driven business decisions. We operate around 20 Presto clusters spanning over 10,000 nodes across two regions. We have about 12,000 weekly active users running approximately 500,000 queries daily, which read about 100 PB from HDFS. Today, Presto is used to query various data sources like Apache Hive, Apache Pinot, AresDb, MySQL, Elasticsearch, and Apache Kafka, through its extensible data source connectors.

Uber使用开源的Presto查询几乎所有的数据源，包括运动和静止的数据。Presto的多功能性使我们能够做出智能的、数据驱动的业务决策。我们在两个地区拥有约20个Presto集群，跨越10,000个节点。我们每周大约有12,000个活跃用户，每天运行大约500,000个查询，从HDFS读取约100 PB的数据。如今，Presto用于查询各种数据源，如Apache Hive，Apache Pinot，AresDb，MySQL，Elasticsearch和Apache Kafka，通过其可扩展的数据源连接器。

Figure 1

图1

Our selection of cluster types can accommodate any request, whether for interactive or batch purposes. Interactive workloads cater to dashboards/desktop users waiting for the results, and batch workloads are scheduled jobs that run on a predefined schedule. Each of our clusters is classified based on their machine type. Most of our clusters comprise bigger machines, which are equipped with more than 300 GB of heap memory, while other clusters have smaller machines that are equipped with less than 200 GB of heap memory, and we have modified the concurrency of each cluster depending on its size and type of the machines that make it up.

我们选择的集群类型可以适应任何请求，无论是交互式还是批处理。 交互式 工作负载适用于等待结果的仪表板/桌面用户，而 批处理 工作负载是在预定时间表上运行的计划任务。我们的每个集群都根据其机器类型进行分类。我们的大多数集群由更大的机器组成，配备超过300 GB的堆内存，而其他集群由配备少于200 GB堆内存的较小机器组成，并且我们根据其大小和机器类型修改了每个集群的并发性。

On a weekly basis, memory fragmentation optimization activity is carried out across all production clusters. Even though we were constantly improving fragmentation, we still suffered from constant Full Garbage collections (very long pauses) and sporadically a few out-of-memory errors. Just to give a sense of the problem, let me show you Presto Full GC, cumulative count:

每周，我们在所有生产集群上进...