测试预算。时间受限的CI反馈
At Shopify we run more than 170,000 tests in our core monolith. Naturally, we're constantly exploring ways to make this faster, and the Test Infrastructure team analyzed the feasibility of introducing a test budget: a fixed amount of time for tests to run. The goal is to speed up the continuous integration (CI) test running phase by accepting more risk. To achieve that goal we used prioritization to reorder the test execution plan in order to increase the probability of a fast failure. Our analysis provided insights into the effectiveness of executing prioritized tests under a time constraint. The single most important finding was that we were able to find failures after we had run only 70% of the test-selection suite.
在Shopify，我们在我们的核心单体中运行超过170,000个测试。自然，我们在不断探索如何使其更快，测试基础设施团队分析了引入测试预算的可行性：测试运行的固定时间。其目的是通过接受更多的风险来加快持续集成（CI）测试运行阶段的速度。为了实现这一目标，我们使用优先级来重新安排测试执行计划，以增加快速失败的概率。我们的分析提供了在时间限制下执行优先级测试的有效性的洞察力。最重要的一个发现是，在我们只运行了70%的测试选择套件之后，我们就能够发现故障。
The Challenge
挑战
Shopify’s codebase relies on CI to avoid regressions before releasing new features. As the code submission rate grows along with the development team size, so does the size of the test pool and the time between code check-ins and test result feedback. As seen in the figure below developers will occasionally get late CI feedback while other times the CI builds complete in under 10 minutes. This non-normal cadence of receiving CI feedback leads to more frequent context switches.
Shopify的代码库依靠CI来避免在发布新功能前出现退步。随着代码提交率与开发团队规模的增长，测试池的规模以及代码签入和测试结果反馈之间的时间也在增长。如下图所示，开发人员偶尔会收到较晚的 CI 反馈，而其他时候 CI 构建在 10 分钟内完成。这种接收 CI 反馈的非正常节奏导致了更频繁的上下文切换。
The feedback time varies
反馈时间不同
Various techniques exist to speed up CI such as running tests in parallel or reducing the number of tests to run with test selection. Balancing the cost of running tests against the value of running them is a fundamental topic in test selection. Furthermore, if we think of the value as a variable then we can make the following observations for executing tests: