控制大规模 Monorepo 变更的发布

At Uber, we rely on continuous deployment to ensure that changes to thousands of microservices reach production automatically and without human monitoring. This blog explores one of the particularly interesting problems we faced as continuous deployment was adopted across Uber. The problem is summarized as follows: When thousands of services can be changed with a single commit to one of our monorepos, for example, upgrading the RPC library used by virtually every Go service at Uber, how do we minimize the blast radius of a bad change?

在 Uber，我们依赖持续部署来确保对数千个微服务的变更能够自动、无需人工监控地进入生产环境。这篇博客探讨了我们在 Uber 全面采用持续部署过程中遇到的一个特别有趣的问题。问题可以概括如下：当一次对某个 monorepo 的提交就能影响数千个服务时（例如，升级 Uber 几乎所有 Go 服务都在使用的 RPC 库），我们如何将不良变更的影响范围降到最低？

As we introduced more deployment automation, we found a clear opportunity to improve our processes in this respect. Our existing safety mechanisms were focused on evaluating the effect of a commit before it’s merged into the code base and on its effect on individual services’ health during deployment.

随着我们引入更多部署自动化，我们发现了在这方面改进流程的明确机会。我们现有的安全机制侧重于在提交合并到代码库之前评估其影响，以及在部署过程中评估其对单个服务健康的影响。

So, in response to this question, we introduced cross-cutting service deployment orchestration to our continuous deployment system. This gives us the ability to prevent issues from reaching our most important services by using signals from other services, thereby minimizing the risk associated with automation rolling out changes impacting many services.

因此，针对这一问题，我们在持续部署系统中引入了跨服务部署编排。这使我们能够利用来自其他服务的信号，防止问题波及到我们最重要的服务，从而将与自动化推出影响众多服务的变更相关的风险降至最低。

At Uber, code is organized into a few large monorepos: One for each of our core programming languages. Each of these repositories is the source of hundreds or thousands of microservices, serving the numerous needs of our business. Development is trunk-based and all production releases of these services are built from the main branch.

在 Uber，代码被组织成几个大型 monorepos：每种核心编程语言对应一个。每个仓库都是数百甚至数千个微服务的来源，满足我们业务的众多需求。开发采用主干开发模式，所有这些服务的生产版本均从主分支构建。

As we detailed in a previous blog post on our cont...