Our journey to scale incentive campaigns to unlock high experiment velocity for Growth.

This is the second post in the series on how we scaled our development practices at Lyft in the face of an ever-increasing number of developers and services.

This post will focus on how we brought a great development experience right to the laptop to allow for super fast iteration.

Late in 2018, Lyft engineering completed decomposing our original PHP monolith into a collection of Python and Go microservices. A few years down the road, microservices had been largely successful in allowing teams to operate and ship services independently of one another. Separation of concerns that microservices brought about enabled us to experiment and deliver features faster–deploying hundreds of times each day–and provided us with the flexibility to use different programming languages where they work best, have stricter or looser requirements based on service criticality, and much more. However, as the number of engineers, services, and tests all increased, our development tooling struggled to keep up with an explosion of microservices, eroding much of the productivity gains we had strived for.

This four-part series will walk through the development environments that served Lyft’s engineering team as it grew from 100 engineers and a handful of services to 1000+ engineers and hundreds of services. We’ll discuss the scaling challenges that caused us to pivot away from most of those environments, as well as a testing approach based predominantly on heavy integration tests (often approaching end-to-end), in favor of a local-first approach centered on testing components in isolation.

• Part One: History of development and test environments (this post)
• Part Two: Optimizing for fast local development
• Part Three: Extending the service mesh in staging with overrides
• Part Four: Gating deployments with automated acceptance tests

In Q2 of 2021, Lyft served 17.1 million active riders through our suite of mobile applications. At this scale, every crash, frozen frame, or hiccup can translate to thousands of hours of wasted time and frustration. Given the potential to impact millions of Lyft’s customers, we doubled our efforts in 2020 to improve our applications’ speed and stability by launching an internal mobile performance initiative.

This is the first in a series of stories detailing our journey to materially improve the speed and stability of our applications. We hope that this can be used by other companies as a reference and inspiration for their own investments in this space.

In this post, we will explore both how and why we started investing in mobile performance at Lyft, and we will dig into our team’s philosophy — everything from how we select projects to how we evaluate success.

How we built at Lyft a gradient boosted tree package to predict travel time more accurately.

One of the Security Team’s projects this year has been to make it easy to generate reports and dashboards from Cartography, Lyft’s security intelligence graph. Cartography links together various entities like compute, permissions, Github repositories, users, etc. and has powerful query capabilities, but it does not integrate with our analytics tools out of the box. To remedy this we’ve leveraged Lyft’s data infrastructure to build an ETL solution that extracts data from Cartography and transforms it into something that can be consumed by our analytics tools. Our solution improves on older approaches by being both significantly easier to work with and more powerful.

The Android engineering team at Lyft exclusively uses IntelliJ to develop new Android features (Android Studio is on the horizon, but that’s for another article). The IntelliJ platform offers a powerful SDK that enables developers to build plugins that enrich the IDE’s feature set, and a few years ago the team decided to take advantage of this SDK by building a plugin.

How Lyft improves on device location tracking with map data.

即使是一个精心制作的交接文件也不能取代你与开发之间良好的语言交流。这应该与开始、移交和书面文档一起使用。你越让开发了解你的设计，还原的结果就越接近实际发布的产品。

A/B tests, controlled experiments to measure performance differences between groups, are ubiquitous tools in data science and represent the gold standard in business measurement. At Lyft, A/B tests are found in nearly every product, from measuring the performance of marketplace algorithms and app designs, to engineering infrastructure changes. The units of analysis for most A/B tests are users, and sometimes markets or times of day. In this post, we are going to describe a new type of A/B test we are performing on our hardware team at Lyft. We will also illustrate a couple of A/B tests which we have done to safely and confidently improve the user experience, lower costs, and provide the best possible service for the cities in which we operate.

在之前的一篇博文中，我们讨论了Feature Service的架构，它在Lyft管理机器学习（ML）的特征存储和访问。在这篇文章中，我们将讨论LyftLearn的架构，这是一个建立在Kubernetes上的系统，负责管理ML模型训练以及批量预测。

从在地图上回放共享骑行，到实时解决物理安全问题，到管理成千上万的自行车和滑板车车队，再到查看自动驾驶汽车的轨迹，Lyft的前端服务支持大量不同的用例。

随着Lyft在过去十年的发展，我们的业务需求也变得越来越复杂。然而，满足这些不同需求的一个核心要求是必须要有高性能的网络应用。

我们与熟悉Lyft前端工程历史的高级工程师和员工坐下来，了解我们是如何应对性能挑战，并建立一个生态系统来支持需要高性能网络应用的无数使用案例的。通过回顾我们在Lyft构建100多个高性能前端微服务的历程，我们已经将我们的经验提炼成我们所说的网络性能需求层次，这是一个可以战略性地识别构建网络应用的组织中最有影响的性能需求的系统。

今天Lyft很高兴地宣布Flyte的开源，Flyte是一个结构化编程和分布式处理平台，用于高并发、可扩展和可维护的工作流。Flyte在Lyft服务生产模型训练和数据处理已有三年多的时间，成为定价、地点、预计到达时间（ETA）、地图、自驾（L5）等团队事实上的平台。事实上，Flyte在Lyft管理着超过7000个独特的工作流程，每月总执行量超过10万次，100万个任务，1000万个容器。在这篇文章中，我们将向您介绍Flyte，概述它所解决的问题类型，并提供如何利用它来满足您的机器学习和数据处理需求的例子。

机器学习构成了Lyft应用的骨干。Lyft的一些ML例子包括决定匹配司机和乘客的最佳方式，决定如何为乘车定价，向乘客和司机分发优惠券和奖励，检测欺诈，路线规划和自动支持。这些用例的ML模型需要通过数据仓库上的批处理作业或通过事件流计算的特征。此外，无论这些特征是以何种方式计算的，它们都需要通过批处理查询来进行模型训练，并通过低延迟的在线推理来提供。

前端软件开发是一个复杂的解决方案空间，需要考虑很多因素，比如可扩展性、弹性、一致的用户体验、异构的客户端功能、实时体验，代码要同时在服务器和浏览器环境下运行。

在Lyft，我们使用一个内部特性服务来存储ML模型使用的批处理和流式处理特性，使它们可以在离线模式（用于训练）和在线模式（用于推理）下访问。该服务将这些特征复制到Elasticsearch中，实现高级查询。对于增长型产品，我们使用Elasticsearch来支持内部工具，比如我们的客户数据平台，它使团队能够根据无数特征定义用户群。

2020年，新的健康与安全和实时计划对我们以前的Elasticsearch SLA发起了挑战。随着每周不同用例的涌入，我们很快就超出了现有的集群，延迟增加，AWS成本飙升。在这篇文章中，我们将分享我们测量和改变与Elasticsearch交互的各种方法，从而最终降低了延迟和成本。