How Lyft’s ML Platform Saves Time and Money on Big Data/ML Workloads.

Recommendation plays an important role in Lyft’s understanding of its riders and allows for customizing app experiences to better fulfill their needs. At times, recommendations are also leveraged to manage the marketplace, making sure there’s a healthy balance between ride demand and driver supply. This allows ride requests to be fulfilled with more desirable dispatch outcomes such as matching riders with the best driver nearby.

This blog post focuses on the scope and the goals of the recommendation system, and explores some of the most recent changes the Rider team has made to better serve Lyft’s riders.

We know what you’re thinking — testing in production is one of the cardinal sins of software development. However, at Lyft we have come to realize that load testing in production is a powerful tool to prepare systems for unexpected bursty traffic and peak events. We’ll explore why Lyft needed a custom performance testing framework that worked in production, how we built a cross-functional solution, and how we’ve continued to improve this testing platform since its launch in 2016.

What exactly do we mean by “Load Testing”? In the context of this article we mean any tool that creates traffic to stress test systems and see how they perform at the limits of their capacity.

Graph learning methods can reveal interesting insights that capture the underlying relational structures. Graph learning methods have many industry applications in areas such as product or content recommender systems and network analysis.

In this post, we discuss how we use graph learning methods at Lyft to generate embeddings — compact vector representation of high-dimensional information. We will share interesting rideshare insights uncovered by embeddings of riders, drivers, locations, and time. As the examples will show, trained embeddings from graphs can represent information and patterns that are hard to capture with traditional, straightforward features.

Across the past couple of years, different mobile app teams across Lyft have been moving to Server Driven UI (SDUI) for three main reasons:

• To deal with business complexity
• To increase release velocity
• To be more flexible in how we staff and build features

This post is about Lyft Bikes and Scooters’ journey to SDUI, why we’ve gone down this path, and what’s worked well for us.

The health of Lyft’s marketplace depends on how riders and drivers are distributed across space and time. Within the complex rideshare space, it is not easy to define typical marketplace concepts like “market efficiency” and “supply-demand balance”. A simple question such as “Do we have enough drivers right now?” has different answers depending on context:

• Are there enough drivers in the right places to maintain good service levels?
• Are there enough drivers system-wide, assuming a ride request will be accepted no matter how far away it is?
• Are there enough to maintain an attractive earning rate?

Each question leads in a different direction. Being able to answer such questions is the interesting (and challenging!) part of operating a healthy two-sided marketplace.

Hundreds of millions of real-time decisions are made each day at Lyft by online machine learning models. These model-based decisions include price optimization for rides, incentives allocation for drivers, fraud detection, ETA prediction, and innumerable others that impact how riders move and drivers earn.

Lyft hosts a dynamic marketplace connecting millions of people to a robust transportation network. In order to offer high value and quality service for both riders and drivers we need to make complex optimization decisions in near-real time. The environment can change quickly with traffic, events and weather, making these decisions even more challenging.

We have employed multi-arm bandits (MAB) algorithms, a common machine learning method for decision making using long-term rewards, to improve our real-time decision making capability. MABs allow us to not only iterate at a faster cadence and lower cost, but also allow for dynamic user experiences and responsive marketplace systems. We will walk through some of our most impactful MAB applications in UI optimization and personalized messaging, concluding with applications in our marketplace algorithms.

Monitoring mobile performance and resource consumption at Lyft.

Over the past 2 years, we’ve built a comprehensive vulnerability management program at Lyft. This blog post will focus on the systems we’ve built to address OS and OS-package level vulnerabilities in a timely manner across hundreds of services run on Kubernetes. Along the way, we’ll highlight the technical challenges we encountered and how we eliminated most of the work required from other engineers. In this first of two posts, we describe our graph approach to finding where a given vulnerability was introduced — a key building block that enables automation of most of the patch process.

Using Envoy as an Explicit CONNECT and Transparent Proxy.

Feature flags are everywhere in modern software development: They’re a great tool for running A/B experiments, slowly rolling out changes to users, and even turning off problematic codepaths during incidents. When an engineer implements a new feature, it’s practically second-nature to gate it behind a feature flag.

While this practice is largely beneficial for the most part, incidents are occasionally caused when a feature flag enables a buggy codepath and causes a crash or an otherwise degraded user experience. A feature flag that causes a crash immediately upon app launch is particularly painful because even if the feature flag is disabled remotely after an engineer identifies the issue, once an app has the bad configuration it will continue to crash before it’s able to successfully fetch the corrected configuration.

We’ve experienced this issue a few times at Lyft over the years. When a crash on launch was introduced by turning on a feature flag or changing other remote configurations, we usually had to ship a hotfix to get users out of infinite crash loops since we had no way of pushing configuration updates to the app when it was crashing so early in its lifecycle. This inevitably resulted in disappointed users, fewer rides, and lost revenue.

To help mitigate these crash loops, we created Safe Mode.

Pain Points of Traditional Automated UI Tests.

The journey of evolving our streaming platform and pipeline to better scale and support new use cases at Lyft.

In the spring of 2020 we started the journey to improve the performance of Lyft’s mobile applications, initially focusing on app start time (also known as Time to Interact or TTI). There was a great deal of opportunity for improvement in the TTI space at Lyft and we were confident that with a small investment, we would be able to add meaningful impact. The success of this project helped pave the way for further investment in Mobile Performance at Lyft.

Envoy Mobile is an ambitious open source initiative to bring the power of Envoy Proxy to mobile apps, leading to unparalleled observability, cutting edge technologies, control, and consistency in the mobile networking space.