Recovering from Crashes with Safe Mode

Recovering from Crashes with Safe Mode

摘要

Feature flags are everywhere in modern software development: They’re a great tool for running A/B experiments, slowly rolling out changes to users, and even turning off problematic codepaths during incidents. When an engineer implements a new feature, it’s practically second-nature to gate it behind a feature flag.

While this practice is largely beneficial for the most part, incidents are occasionally caused when a feature flag enables a buggy codepath and causes a crash or an otherwise degraded user experience. A feature flag that causes a crash immediately upon app launch is particularly painful because even if the feature flag is disabled remotely after an engineer identifies the issue, once an app has the bad configuration it will continue to crash before it’s able to successfully fetch the corrected configuration.

We’ve experienced this issue a few times at Lyft over the years. When a crash on launch was introduced by turning on a feature flag or changing other remote configurations, we usually had to ship a hotfix to get users out of infinite crash loops since we had no way of pushing configuration updates to the app when it was crashing so early in its lifecycle. This inevitably resulted in disappointed users, fewer rides, and lost revenue.

To help mitigate these crash loops, we created Safe Mode.