Learn how we achieved diff-time E2E testing for thousands of microservices at Uber.

Discover how Uber's in-house ETL framework helps standardize modular ETL development, improving developer productivity and ensuring reliable data pipelines.

Learn all about our journey of successfully upgrading our MySQL fleet at Uber from v5.7 to v8.0, enhancing performance and reliability.

Learn about how the Storage team at Uber significantly reduced costs and improved speed for backups of its Petabyte-scale, MyRocks-based distributed databases by devising a Differential Backups…

Uber's data lake is migrating to the cloud! Learn how they're tackling security challenges and scaling the system to handle massive amounts of data while ensuring a seamless transition for users.

What follows is the story of how we started designing for surfaces outside the app, the engineering problems we had to solve along the way, and ultimately how we measurably improved the experience of riders and drivers.

The Odin platform aims to provide a unified operational experience by encompassing all aspects of managing stateful workloads. These aspects include host lifecycle, workload scheduling, cluster management, monitoring, state propagation, operational user interfaces, alerting, auto-scaling, and automation. Uber deploys stateful systems at global, regional, and zonal levels, and Odin is designed to manage these systems consistently and in a technology-agnostic manner. Moreover, Odin supports co-location to increase hardware cost efficiency. All stateful workloads must be fully containerized, a relatively novel and controversial concept when the platform was created.

This blog post is the first of a series on Uber’s stateful platform. The series aims to be accessible and engaging for readers with no prior knowledge of building container platforms and those with extensive expertise. This post provides an overview of Odin’s origins, the fundamental principles, and the challenges encountered early on. The next post will explore how we have safely scaled operational throughput, significantly improving our handling of large-scale, fleet-wide operations and up-leveling runbooks through workflows. Stay tuned for more posts in the series.

Large Language Models (LLMs) have emerged as pivotal instruments in the tech industry, unlocking new avenues for innovation and progress across various sectors. At Uber, the impact of LLMs is…

Uber提出了Kafka分层存储方案，用于解决Kafka集群存储的扩展性、效率和操作成本问题。该方案通过引入本地和远程两个存储层，实现了存储的扩展性和长期存储的可行性。远程存储层可以使用不同的扩展存储，并具有更长的数据保留期。分层存储方案减轻了Kafka代理的本地存储负担，降低了操作成本。此外，该方案通过引入RemoteLogManager和RemoteStorageManager等组件，实现了远程日志的复制、获取和删除操作，并提供元数据的生命周期管理。分层存储方案的架构图展示了本地和远程日志的复制过程，以及远程日志的清理过程。跟随者副本需要复制可在领导者本地存储中找到的段，并在开始从领导者获取任何消息之前构建辅助数据。

In collaboration with CLP’s developers, we have developed an innovative end-to-end system to manage unstructured logs across Uber’s various data and ML platforms.

这篇文章介绍了Uber如何将现有的Cassandra集群从单区容错转换为单区故障容错。Uber通过添加新节点环、配置节点分布和分组、复制数据到新环、切换客户端连接等步骤实现了这一目标。他们还改进了驱动程序，实现了动态切换流量而无需重新启动客户端。为了实现无缝的流量切换，Uber创建了一个微服务来发布Cassandra集群的联系信息，并要求客户端选择与其所在区域相同的初始节点和协调器节点。此外，他们还解决了多机架设置中的挑战，确保了各个机架的容量均衡。

这篇文章主要介绍了基于微服务架构的软件开发过程中的调试选项，包括远程调试和本地调试。通过SLATE环境中的远程调试，可以在生产基础设施上进行只读调试，但存在动态修改的限制。为了填补本地调试环境和生产环境之间的差距，SLATE Attach引入了一个SLATE代理，将测试请求重定向到开发人员的本地开发机器进行调试和开发。文章还提到了SLATE Sniffer用于通过在生产环境中监控调试问题。

文章介绍了Uber在个性化营销方面的工作。他们使用层次化配置合并和覆盖不同地理区域的策略，以提供个性化的推荐和营销。除了送货业务外，Uber还计划将个性化能力扩展到其他业务领域。

Testopedia是Uber内部的一个测试工具，用于识别和处理测试中的不稳定性问题。它可以根据测试历史记录和运行结果，自动创建问题票据并将相关信息发送到JIRA。Testopedia还支持自定义配置，用户可以自定义分组规则、票据类型、优先级和描述模板。Uber还计划将GenAI等先进技术整合到系统中，以自动生成测试修复方案，并通过AI分析故障模式，将测试失败自动分类到更具体的子类别中，提高故障排查和解决的效率。通过在Uber的各个Monorepo中推广使用Testopedia，已经显著减少了不稳定测试的数量，提高了CI的可靠性。

Uber计划将其批量数据分析和机器学习训练堆栈迁移到Google Cloud Platform（GCP）。他们将使用HiveSync和Hudi库来实现在两个区域之间保持数据湖同步，并将本地数据湖的数据复制到云端数据湖和对应的Hive Metastore。迁移后，他们将在GCP上为YARN和Presto集群提供新的IaaS，并通过现有的数据访问代理将流量路由到云端堆栈。迁移过程中可能会面临性能、成本管理、非分析/机器学习应用使用HDFS和未知挑战等问题，但他们计划通过改进开源连接器、利用云的弹性、迁移其他文件存储用例以及积极解决问题来解决这些挑战。

文章讨论了在工作粒度方面的昂贵性以及提高性能和减少存储成本的方法。通过放弃工作粒度要求，可以显著提高读取吞吐量。