Uber进行的不稳定测试的全面改革

A few years ago, we started tackling flaky tests in an effort to stabilize CI experience across our monorepos. The project first debuted in our Java monorepo and received good results in driving down frictions in developers’ workflow. However, as we evolved our CI infrastructure and started onboarding it to our largest repository with the most users, Go Monorepo, the stop-gap solution became increasingly challenging to scale to the scope.

几年前，我们开始着手解决CI中的不稳定测试，以稳定我们的Monorepo中的CI体验。该项目首次在我们的Java Monorepo中亮相，并在降低开发人员工作流程中的摩擦方面取得了良好的结果。然而，随着我们的CI基础设施的发展，并将其引入到我们最大的存储库和最多用户的Go Monorepo中，这种权宜之计的解决方案在规模上变得越来越具有挑战性。

The legacy service had an analyzer built-in, which categorizes tests based on a window of historical test runs. However, most of the time it works in a sandbox with little visibility into details, like what history it has examined for a test, what the reason was behind a decision, or additional information about a test. Thus, often some tests were miscategorized but we didn’t know why and had to manually recategorize.

传统服务内置了一个分析器，根据历史测试运行的窗口对测试进行分类。然而，大部分时间它在一个具有很少细节可见性的沙盒中工作，比如它检查了一个测试的哪些历史、决策背后的原因或有关测试的其他信息。因此，经常会出现一些测试被错误分类的情况，但我们不知道原因，必须手动重新分类。

It also has little extensibility of supporting different strategies to categorize tests. It only supports the sliding window strategy to categorize tests. The legacy test model is specifically tailored to Java, assuming inputs like test suites, parameters, annotations, etc., which are not always available in other languages.

它还很难支持不同的策略来对测试进行分类。它只支持滑动窗口策略来对测试进行分类。传统的测试模型专门针对Java进行了定制，假设输入包括测试套件、参数、注释等，这在其他语言中并不总是可用。

The “serial” and “parallel” concepts add additional logic on each monorepo’s CI side to respond differently. Also, because it encapsulates many scenarios–categorize, transition, recover in CI, notifications, etc. – complexity greatly increases when it needs to be both generic enough to accommodate each repo and effective enough to not miss any flakiness.

“串行”和“并行”概念在每个单体库的CI端添加了额外的逻辑，以不同方式响应。此外，由于它封装了许多场景-分类、过渡、恢复在CI中、通知等-当它需要既足够通用以适应每个库，又足够有效以不错过任何不稳定...