Feature flags are everywhere in modern software development: They’re a great tool for running A/B experiments, slowly rolling out changes to users, and even turning off problematic codepaths during incidents. When an engineer implements a new feature, it’s practically second-nature to gate it behind a feature flag.

While this practice is largely beneficial for the most part, incidents are occasionally caused when a feature flag enables a buggy codepath and causes a crash or an otherwise degraded user experience. A feature flag that causes a crash immediately upon app launch is particularly painful because even if the feature flag is disabled remotely after an engineer identifies the issue, once an app has the bad configuration it will continue to crash before it’s able to successfully fetch the corrected configuration.

We’ve experienced this issue a few times at Lyft over the years. When a crash on launch was introduced by turning on a feature flag or changing other remote configurations, we usually had to ship a hotfix to get users out of infinite crash loops since we had no way of pushing configuration updates to the app when it was crashing so early in its lifecycle. This inevitably resulted in disappointed users, fewer rides, and lost revenue.

To help mitigate these crash loops, we created Safe Mode.

Pain Points of Traditional Automated UI Tests.

The journey of evolving our streaming platform and pipeline to better scale and support new use cases at Lyft.

In the spring of 2020 we started the journey to improve the performance of Lyft’s mobile applications, initially focusing on app start time (also known as Time to Interact or TTI). There was a great deal of opportunity for improvement in the TTI space at Lyft and we were confident that with a small investment, we would be able to add meaningful impact. The success of this project helped pave the way for further investment in Mobile Performance at Lyft.

Envoy Mobile is an ambitious open source initiative to bring the power of Envoy Proxy to mobile apps, leading to unparalleled observability, cutting edge technologies, control, and consistency in the mobile networking space.

Our journey of enabling user privacy rights started all the way back with the California Consumer Privacy Act (“CCPA”) going live in 2020. Recently, new state laws have begun going into effect, including upcoming changes in California with the California Privacy Rights Act, and our early work set us up for success. Lyft is positioned to seamlessly handle changes thanks to the choices we made with the federated design of our architecture. In this blog post, we’ll share an overview of some technical strategies Lyft uses to provide important privacy rights like those in CCPA and describe how we implemented user data export and deletion in our online systems.

To start off, let’s review some of the key challenges we faced in developing our response to user requests regarding privacy rights.

In our last blog, we discussed how managing our business effectively comes down to, in large part, making causally valid forecasts based on our decisions. Such forecasts accurately predict the future while still agreeing with experiment (e.g. increasing prices by X will decrease conversion by Y). With this, we can optimize our decisions to yield a desirable future.

But there remains a gap between theory and the implementation that makes it a reality. In this blog, we will discuss the design of software and algorithms we use to bridge this gap.

In a microservice world, significant route configuration changes to the front proxy are often required to keep up with an evolving business. Making these route configuration changes easily manageable and testable quickly becomes a challenging issue with the number of developers needing to make changes and the frequency of these changes.

Pricing forms the backbone of the Lyft’s marketplace system, which is core in the journey of improving people’s lives with the world’s best transportation. In general, the goal of price optimization is to find the price that balances supply and demand while covering the expenses necessary to provide an easy-to-use rideshare platform, where drivers have tremendous influence on the rider prices through their entry and exit on the platform.

Continuous Deployment (CD) is when software changes, such as new features and fixes, are automatically deployed to our customers as quickly and safely as possible. At Lyft, we pride ourselves on “making it happen”, so in 2019 we set out to move from our manual deployment process to CD. The microservice architecture at Lyft made our path to CD challenging, as adoption required changing how each team maintained and deployed their services. Not only did we have to make configuration changes to each service, but most importantly we had to drive cultural changes in how each team approached and performed their deployments. We recently hit a new milestone of 90+% of our approximately 1,000 services using CD to deploy to production. We would like to share the journey, technical details, and challenges we faced.

Efficiently managing our marketplace is a core objective of Lyft Data Science. That means providing meaningful financial incentives to drivers in order to supply affordable rides while keeping ETAs low under changing market conditions — no easy task!

Lyft’s tool chest contains a variety of market management products: rider coupons, driver bonuses, and pricing, to name a few. Using these efficiently requires a strong understanding of their downstream consequences — everything from counts of riders opening the Lyft app (“sessions”) to financial metrics.

To complicate the science further, our data is heavily confounded by our previous decisions, so a merely correlational model would fail us. Sifting out causal relationships is the only option for making smart forward looking decisions.

Machine Learning models at Lyft make millions of high stakes decisions per day from physical safety classification to fraud detection to real-time price optimization. Since these ML model based actions impact the real world experiences of our riders and drivers as well as Lyft’s top and bottom line, it is critical to prevent models from degrading in performance and alert on malfunctions.

However, identifying and preventing model problems is hard. Unlike problems in deterministic systems whose errors are easier to spot, models’ performance tends to gradually decrease, which is more difficult to detect.

Safety is fundamental to Lyft. Thus, as part of the Safety & Customer Care Web team, we strive to provide the safest experience for our users. Our focus is to bring safety and support features to our riders’ web experience (ride.lyft.com); this allows riders to request a ride from their browser without needing the mobile app. In doing so, we also maintain consistency with the mobile app experience.

Android applications such as Lyft’s apps are developed by a large number of contributors. This means that the codebase grows and changes very quickly. Features are constantly being added or improved, and all these modifications can potentially impact the performance of the application. Thus, it is important to understand how the application consumes CPU resources and to see the dynamics of such metrics across product releases.

We believe that relevant messaging is at the heart of our mission to improve people’s lives with the world’s best transportation.

In a previous blog post, we discussed LyftLearn’s infrastructure built on top of Kubernetes. In this post, we will focus on the compute layer of LyftLearn, and will discuss how LyftLearn solves some of the major pain points faced by Lyft’s machine learning practitioners.