Uber计划将其批量数据分析和机器学习训练堆栈迁移到Google Cloud Platform（GCP）。他们将使用HiveSync和Hudi库来实现在两个区域之间保持数据湖同步，并将本地数据湖的数据复制到云端数据湖和对应的Hive Metastore。迁移后，他们将在GCP上为YARN和Presto集群提供新的IaaS，并通过现有的数据访问代理将流量路由到云端堆栈。迁移过程中可能会面临性能、成本管理、非分析/机器学习应用使用HDFS和未知挑战等问题，但他们计划通过改进开源连接器、利用云的弹性、迁移其他文件存储用例以及积极解决问题来解决这些挑战。

文章讨论了在工作粒度方面的昂贵性以及提高性能和减少存储成本的方法。通过放弃工作粒度要求，可以显著提高读取吞吐量。

Data categorization–the process of classifying data based on its characteristics and essence–is a foundational pillar of any privacy or security program. The effectiveness of fine-grained data…

In the past few years, the Machine learning (ML) adoption and impact at Uber have accelerated across all business lines. Today, ML plays a key role in Uber’s business, being used to make business…

Quality and testing go hand in hand, and in 2023 we took on a new and exciting challenge to change how we test our mobile applications. Specifically, we are training ML models to test our applications…

The financial accounting services platform at Uber operates at an internet scale– approximately 1.5 billion journal entries (JEs) per day and 120 million transactions per day via ETL and data…

Uber Eats’ mission is to make eating effortless, at any time, for anyone. The Uber Eats home feed is an important tool for fulfilling this goal, as it aims to provide a magical food browsing…

Rendering large lists in React can be a challenging task for developers. As the size of the list grows, the DOM (Document Object Model) tree also grows, leading to performance issues like slow…

Uber uses open-source Presto to query nearly every data source, both in motion and at rest. Presto’s versatility empowers us to make intelligent, data-driven business decisions. We operate around 20…

The Uber Michelangelo feature store, called Palette, is a database of Uber-specific curated and internally crowd-sourced features that are easy to use in machine learning projects. It comes to solve…

As a marketplace-based, consumer-facing app, Uber encounters a multitude of sources of fraud across its platform. In one of the most common cases of fraud, bad actors use various methods to attempt to…

Discover real-time query analytics and governance with DataCentral: Uber’s big data observability powerhouse, tackling millions of queries in petabyte-scale environments.

Uber’s mission is to reimagine the way the world moves for the better and provide earning opportunities globally through its marketplace. One effective approach to bring the Uber brand and marketplace…

Docstore is Uber's in-house, distributed database built on top of MySQL®. Storing tens of PBs of data and serving tens of millions of requests/second, it is one of the largest database engines at Uber…

Uber is a global business, and has a customer base that’s spread throughout the world. Uber’s customer base is divided into many user personas, predominantly riders, drivers, eaters, couriers, and…

If you were to ask a veteran SOC (Security Operations Center) analyst about Network IDS (Intrusion Detection Systems) or IPS (Intrusion Prevention Systems), the response would probably contain phrases…