Timestone。Netflix的高吞吐量、低延迟的优先级排队系统，内置支持不可并行的工作负载

by Kostas Christidis

作者 Kostas Christidis

Introduction

简介

Timestone is a high-throughput, low-latency priority queueing system we built in-house to support the needs of our media encoding platform, Cosmos. Over the past 2.5 years, its usage has increased, and Timestone is now also the priority queueing engine backing our general-purpose workflow orchestration engine (Conductor), and the scheduler for large-scale data pipelines (BDP Scheduler). All in all, millions of critical workflows within Netflix now flow through Timestone on a daily basis.

Timestone是一个高吞吐量、低延迟的优先级队列系统，我们在内部建立了这个系统来支持我们的媒体编码平台Cosmos的需求。在过去的2年半里，它的使用量不断增加，现在Timestone也是支持我们通用工作流协调引擎（Conductor）的优先级队列引擎，以及大规模数据管道的调度器（BDP Scheduler）。总而言之，Netflix内部数以百万计的关键工作流程现在每天都在通过Timestone流动。

Timestone clients can create queues, enqueue messages with user-defined deadlines and metadata, then dequeue these messages in an earliest-deadline-first (EDF) fashion. Filtering for EDF messages with criteria (e.g. “messages that belong to queue X and have metadata Y”) is also supported.

Timestone客户端可以创建队列，用用户定义的截止日期和元数据排队，然后以最早截止日期优先（EDF）的方式排队。还支持用标准过滤EDF消息（例如，"属于队列X且元数据为Y的消息"）。

One of the things that make Timestone different from other priority queues is its support for a construct we call exclusive queues — this is a means to mark chunks of work as non-parallelizable, without requiring any locking or coordination on the consumer side; everything is taken care of by the exclusive queue in the background. We explain the concept in detail in the sections that follow.

使得Timestone与其他优先级队列不同的一点是，它支持一种我们称之为独占队列的结构--这是一种将工作块标记为不可并行的手段，不需要在消费者一方进行任何锁定或协调；一切都由后台的独占队列来处理。我们在后面的章节中详细解释这个概念。

Why Timestone

为什么选择Timestone

When designing the successor to Reloaded — our media encoding system — back in 2018 (see “Background” section in The Netflix Cosmos Platform), we needed a priority queueing system that would provide queues between the three components in Cosmos (Figure 1):

早在2018年设计Reloaded--我们的媒体编码系统--的继任者时（见《Netflix Cosmos平台》中的 "背景 "部分），我们需要一个优先排队系统，在Cosmos的三个组件之间提供队列（图1）。

1. the API fr...