100倍更快：我们如何为Netflix Maestro的工作流引擎加速
[
[
By Jun He, Yingyi Zhang, Ely Spears
由 Jun He，Yingyi Zhang，Ely Spears 提供
TL;DR
总结
We recently upgraded the Maestro engine to go beyond scalability and improved its performance by 100X! The overall overhead is reduced from seconds to milliseconds. We have updated the Maestro open source project with this improvement! Please visit the Maestro GitHub repository to get started. If you find it useful, please give us a star.
我们最近升级了Maestro引擎，不仅提升了可扩展性，还将其性能提高了100X！整体开销从秒级减少到毫秒级。我们已经更新了Maestro开源项目以包含这一改进！请访问Maestro GitHub仓库以开始使用。如果您觉得有用，请给我们一个星星。
Introduction
介绍
In our previous blog post, we introduced Maestro as a horizontally scalable workflow orchestrator designed to manage large-scale Data/ML workflows at Netflix. Over the past two and a half years, Maestro has achieved its design goal and successfully supported massive workflows with hundreds of thousands of jobs, managing millions of executions daily. As the adoption of Maestro increases at Netflix, new use cases have emerged, driven by Netflix’s evolving business needs, such as Live, Ads, and Games. To meet these needs, some of the workflows are now scheduled on a sub-hourly basis. Additionally, Maestro is increasingly being used for low-latency use cases, such as ad hoc queries, beyond traditional daily or hourly scheduled ETL data pipeline use cases.
在我们之前的博客文章中，我们介绍了Maestro作为一个水平可扩展的工作流协调器，旨在管理Netflix的大规模数据/机器学习工作流。在过去的两年半里，Maestro实现了其设计目标，并成功支持了数十万作业的大规模工作流，每天管理数百万次执行。随着Maestro在Netflix的采用增加，新的用例应运而生，受到Netflix不断变化的业务需求的驱动，例如直播、广告和游戏。为了满足这些需求，一些工作流现在按子小时调度。此外，Maestro越来越多地用于低延迟用例，例如临时查询，超越传统的每日或每小时调度的ETL数据管道用例。
While Maestro excels in orchestrating various heterogeneous workflows and managing user end-to-end development experiences, users have experienced noticeable speedbumps (i.e. ten seconds overhead) from the Maestro engine dur...