大规模构建服务拓扑：架构、挑战与经验教训

Featured

精选

By Parth Jain, Rakesh Sukumar, Yingwu Zhao, Renzo Sanchez-Silva & Nathan Fisher*
A deep dive into the engineering challenges of building a real-time service dependency map at Netflix scale: from streaming architectures and distributed aggregation pipelines to time-travel queries and the methodology that made it work.*

作者：Parth Jain、Rakesh Sukumar、Yingwu Zhao、Renzo Sanchez-Silva & Nathan Fisher*
深入探讨在 Netflix 规模下构建实时服务依赖图的工程挑战：从流式架构和分布式聚合管道，到时间旅行查询以及使其成功落地的方法论。*

Introduction

引言

In our first post, we introduced the problem: engineers at Netflix needed a unified, real-time view of service dependencies to troubleshoot faster, understand blast radius, and navigate our distributed architecture. We described our multi-source approach, combining eBPF network flows, IPC metrics, and distributed tracing into physically separate graph layers that can be queried independently or merged into a comprehensive view.

在我们的第一篇文章中，我们介绍了这个问题：Netflix的工程师需要一个统一的、实时的服务依赖视图，以便更快地进行故障排除、了解爆炸半径并导航我们的分布式架构。我们描述了我们的多源方法，将eBPF网络流、IPC指标和分布式追踪结合到物理上独立的图层中，这些图层可以独立查询或合并为全面的视图。

That post explained what we built and why. This post is about how, the engineering reality of building this system at Netflix scale.

上一篇文章解释了我们构建了什么以及为什么。这篇文章是关于如何构建的，即在Netflix规模下构建该系统的工程现实。

Here’s the truth: the first version worked perfectly… in our local environment. Production was a different story. Kafka consumers fell behind. Instances ran out of memory. Some nodes received 100x the traffic of others. Garbage collection pauses consumed more CPU than actual business logic.

事实是这样的：第一个版本在我们的本地环境中运行得非常完美……但在生产环境中却是另一回事。Kafka 消费者处理落后。实例内存耗尽。某些节点接收到的流量是其他节点的 100 倍。垃圾回收暂停消耗的 CPU 比实际业务逻辑还要多。

What you’ll learn in this post isn’t a success story, it’s a learning journey. We’ll walk through the architecture decisions that enabled scale, the production challenges that tested those decisions, the optimization methodology that guided us through, and the lessons that apply to any distributed system. Along the way, we’ll share the innovations that made it possible to proces...