推进我们的厨师基础设施

At Slack, we manage tens of thousands of EC2 instances that host a variety of services, including our Vitess databases, Kubernetes workers, and various components of the Slack application. The majority of these instances run on some version of Ubuntu, while a portion operates on Amazon Linux. With such a vast infrastructure, the critical question arises: how do we efficiently provision these instances and deploy changes across them? The solution lies in a combination of internally-developed services, with Chef playing a central role. In this blog post, I’ll discuss the evolution of our Chef infrastructure over the years and the challenges we encountered along the way.

在Slack，我们管理着数以万计的托管各种服务的EC2实例，包括我们的Vitess数据库、Kubernetes工作节点和Slack应用程序的各个组件。其中大部分实例运行在某个版本的Ubuntu上，而一部分则在Amazon Linux上运行。在这样庞大的基础设施中，关键问题是：我们如何高效地为这些实例提供配置并在它们之间部署更改？解决方案是一组内部开发的服务与Chef发挥了核心作用。在本博客文章中，我将讨论多年来我们的Chef基础设施的演变以及我们遇到的挑战。

A journey down memory lane: our previous process

回忆之旅：我们以前的流程

In the early days of Slack, we relied on a single Chef stack. Our Chef infrastructure included a set of EC2 instances, an AWS application load balancer, an RDS cluster, and an AWS OpenSearch cluster. As our fleet of instances started to grow, we scaled up and out this stack to match our increasing demand.

在 Slack 的早期阶段，我们依赖于单个 Chef 堆栈。我们的 Chef 基础设施包括一组 EC2 实例、一个 AWS 应用负载均衡器、一个 RDS 集群和一个 AWS OpenSearch 集群。随着实例群的增长，我们扩展了这个堆栈，以满足不断增长的需求。

We had three environments on this Chef stack: Sandbox, Dev, and Prod. Nodes from each environment in our fleet were mapped to one of these environments. Slack has two types of cookbooks: cookbooks we download from the Chef Supermarket and the ones that are internally created. When these cookbooks are modified, we upload them using a process called DishPig. When a change is merged to our repo, we triggered a CI job to look for any changes to the cookbooks or roles and built an artifact with this. Then this artifact gets uploaded to a S3 bucket and then from this S3 bucket, we notify an SQS queue. We triggered the DishPig process on...