Lyft市场中流媒体管道的演变
The journey of evolving our streaming platform and pipeline to better scale and support new use cases at Lyft.
我们的流媒体平台和管道不断发展，以更好地扩展和支持Lyft的新用例的旅程。
Background
背景介绍
In 2017, Lyft’s Pricing team within our Marketplace organization was using a cronjob-based Directed Acyclic Graph (DAG) to compute dynamic pricing for rides. Each unit in the DAG would run at the top of every minute, fetch the data from the previous unit, compute the result, and store it for the next unit. This paradigm had several minutes of inherent latency.
2017年，Lyft的定价团队在我们的市场组织中使用一个基于cronjob的有向无环图（DAG）来计算乘车的动态定价。DAG中的每个单元将在每分钟的顶部运行，从上一个单元获取数据，计算结果，并为下一个单元存储。这种模式有几分钟的固有延迟。
As product iteration speed increased over time, this older infrastructure became unable to support the faster development cycles, primarily because all the ML feature generation required writing custom logic which would take multiple weeks to develop.
随着产品迭代速度的增加，这个旧的基础设施变得无法支持更快的开发周期，主要是因为所有的ML功能生成都需要编写自定义逻辑，而这需要多个星期的时间来开发。
The team needed better infrastructure to make the dynamic pricing system more reactive for the following reasons:
该团队需要更好的基础设施，以使动态定价系统更具反应性，原因如下。
- Decrease end-to-end latency that would make the system more reactive to marketplace imbalances.
- 减少端到端的延迟，这将使系统对市场的不平衡作出更多反应。
- Decrease development time and increase product iteration speed.
- 减少开发时间，提高产品迭代速度。
MVP
MVP
After much deliberation, we decided that streaming engines would be a better fit for our requirements and selected Apache Beam. Since this was an entirely new framework, we had to come up with a pipeline design that ensured functional parity with the existing system. The very first version (see Figure 1) was designed to consume events, convert data to ML features, orchestrate model executions, and sync decision variables to their respective services.
经过深思熟虑，我们决定流媒体引擎将更适合我们的要求，并选择了Apache Beam。由于这是一个全新的框架，我们必须想出一个管道设计，以确保与现有系统的功能一致。第一个版本（见图1）被设计为消费事件，将数据转换为ML特征，协调模型的执行，并将决策变量同步到各自的服务。
Figure 1: Initial pipeline
图1：初始管道
Implementing the pipeline was straightforward and required little custom logic since stream...