避免容器化环境中的CPU节流现象

Avoiding CPU Throttling in a Containerized Environment

At Uber, all stateful workloads run on a common containerized platform across a large fleet of hosts. Stateful workloads include MySQL®, Apache Cassandra®, ElasticSearch®, Apache Kafka®, Apache HDFS™, Redis™, Docstore, Schemaless, etc., and in many cases these workloads are co-located on the same physical hosts.

在Uber，所有有状态的工作负载都在一个共同的容器化平台上运行，跨越大量的主机。有状态的工作负载包括MySQL®、Apache Cassandra®、ElasticSearch®、Apache Kafka®、Apache HDFS™、Redis™、Docstore、Schemaless等，在很多情况下，这些工作负载都是在同一台物理主机上共存的。

With 65,000 physical hosts, 2.4 million cores, and 200,000 containers, increasing utilization to reduce cost is an important and continuous effort. Until recently efforts were blocked due to CPU throttling, which indicates that not enough resources have been allocated.

拥有65,000台物理主机，240万个内核和20万个容器，提高利用率以降低成本是一项重要而持续的努力。直到最近，由于CPU节流，表明没有分配到足够的资源，所以努力受阻。

It turned out that the issue was how the Linux kernel allocates time for processes to run. In this post we will describe how switching from CPU quotas to cpusets (also known as CPU pinning) allowed us to trade a slight increase in P50 latencies for a significant drop in P99 latencies. This in turn allowed us to reduce fleet-wide core allocation by up to 11% due to less variance in resource requirements.

事实证明，问题在于Linux内核是如何为进程分配运行时间的。在这篇文章中，我们将描述如何从CPU配额切换到cpusets（也被称为CPU pinning），使我们能够以P50延迟的轻微增加换取P99延迟的显著下降。这反过来又使我们能够将整个舰队的核心分配减少11%，因为资源需求的变化较小。

Cgroups, Quotas, and Cpusets

C组，配额，和Cpusets

CPU quotas and cpusets are Linux kernel scheduler features. The Linux kernel implements resource isolation through cgroups, and all container platforms are based on this. Typically a container maps to a cgroup, which controls the resources of any process running in the container.

CPU配额和cpusets是Linux内核的调度器功能。Linux内核通过cgroup实现了资源隔离，所有的容器平台都是基于此的。通常，一个容器映射到一个cgroup，它控制着在容器中运行的任何进程的资源。

There are two types of cgroups (controllers in Linux terms) for performing CPU isolation: CPU and cpuset. They both control how much CPU a set of processes is allowed to use, but in two different ways: through CPU time quo...