介绍 Netflix 的键值数据抽象层

Vidhya Arvind, Rajasekhar Ummadisetty, Joey Lynch, Vinay Chella

Vidhya Arvind，Rajasekhar Ummadisetty，Joey Lynch，Vinay Chella

Introduction

介绍

At Netflix our ability to deliver seamless, high-quality, streaming experiences to millions of users hinges on robust, global backend infrastructure. Central to this infrastructure is our use of multiple online distributed databases such as Apache Cassandra, a NoSQL database known for its high availability and scalability. Cassandra serves as the backbone for a diverse array of use cases within Netflix, ranging from user sign-ups and storing viewing histories to supporting real-time analytics and live streaming.

在Netflix，我们能够向数百万用户提供无缝、高质量的流媒体体验，关键在于强大的全球后端基础设施。这个基础设施的核心是我们使用多个在线分布式数据库，如Apache Cassandra，这是一个以高可用性和可扩展性而闻名的NoSQL数据库。Cassandra在Netflix内部支持各种用例，从用户注册和存储观看历史记录到支持实时分析和直播。

Over time as new key-value databases were introduced and service owners launched new use cases, we encountered numerous challenges with datastore misuse. Firstly, developers struggled to reason about consistency, durability and performance in this complex global deployment across multiple stores. Second, developers had to constantly re-learn new data modeling practices and common yet critical data access patterns. These include challenges with tail latency and idempotency, managing “wide” partitions with many rows, handling single large “fat” columns, and slow response pagination. Additionally, the tight coupling with multiple native database APIs — APIs that continually evolve and sometimes introduce backward-incompatible changes — resulted in org-wide engineering efforts to maintain and optimize our microservice’s data access.

随着引入新的键值数据库和服务所有者推出新的用例，我们在数据存储的误用方面遇到了许多挑战。首先，开发人员难以理解在跨多个存储中进行复杂全球部署时的一致性、持久性和性能问题。其次，开发人员不得不不断重新学习新的数据建模实践和常见但关键的数据访问模式。这些包括处理尾延迟和幂等性的挑战，管理包含许多行的“宽”分区，处理单个大型“胖”列以及缓慢的响应分页。此外，与多个本地数据库API的紧密耦合——这些API不断演进，有时引入不兼容的变化——导致了整个组织的工程努力来维护和优化我们微服务的数据访问。

To overcome these challenges, we developed a holistic approach that builds upon our Data Gateway Platform. This approach led to th...