规模化地处理不稳定的测试。自动检测和抑制
At Slack, the goal of the Mobile Developer Experience Team (DevXp) is to empower developers to ship code with confidence while enjoying a pleasant and productive engineering experience. We use metrics and surveys to measure productivity and developer experience, such as developer sentiment, CI stability, time to merge (TTM), and test failure rate.
在Slack，移动开发者体验团队(DevXp)的目标是让开发者能够放心地发布代码，同时享受愉快而富有成效的工程体验。我们使用指标和调查来衡量生产力和开发人员的体验，如开发人员的情绪、CI稳定性、合并时间（TTM）和测试失败率。
The DevXp team has continuously invested in test infrastructure and CI improvements, but one of the biggest impediments to our progress is the high — and ever increasing — rate of test failures due to flaky tests. Before we automatically handled flaky tests, the main branch stability was hovering around a 20% pass rate. When looking closely into the 80% of failing builds we found:
DevXp团队不断投资于测试基础设施和CI的改进，但阻碍我们进步的最大因素之一是高的--而且是不断增加的--由于不稳定的测试造成的测试失败率。在我们自动处理不稳定的测试之前，主分支的稳定性一直徘徊在20%左右的通过率。当仔细研究这80%的失败构建时，我们发现。
- 57% failed due to test job failures consisting of flaky and failing automated tests
- 57%的失败是由于测试工作的失败，包括易碎和失败的自动测试
- 13% failed due to developer errors, CI issues, or test infra issues
- 13%的失败是由于开发人员的错误、CI问题或测试信息问题造成的
- 10% failed due to merge conflicts
- 10%由于合并冲突而失败
After replacing the manual approach of handling test failures with an automated flaky test suppression system, we were able to drop test job failures from 57% to less than 5%.
在用一个自动化的故障测试抑制系统取代人工处理测试故障的方法后，我们能够将测试工作的故障率从57%降到5%以下。
This post describes the path we have taken to minimize the number of flaky tests through an approach of automated test failure detection and suppression. This is not a new problem that we are trying to solve; many companies have published articles on systems created for handling flaky tests. This article outlines how test flakiness is an increasing problem at scale and how we got it under control at Slack.
这篇文章描述了我们采取的方法，通过自动测试失败检测和抑制的方法，最大限度地减少故障测试的数量。这并不是我们要解决的新问题；许多公司已经发表了关于为处理不稳定测试而创建的系统的文章。这篇文章概述了测试故障是如何在规模上成为一个日益严重的问题，以及我们如何在Slack中控制它。