Building Lyft’s Next Emblem — Glow
By: Avneet Oberoi, Michael Vernier, Phoenix Li, Masroor Ahmed
Introduction
Long time riders might remember the original fuzzy, pink Carstache emblem that made Lyft universally recognizable. Over the years, the emblem dropped the fuzz for pink lights in the Glowstache and later evolved with more colors as the beloved Amp, which has been in active use for over seven years! Recently, Lyft has introduced its brighter, bolder next generation emblem — Glow. Glow provides a daytime visible, auto-dimmable display showing rider customizable colors and new animations to help them find their ride faster. Glow also has enhanced GPS and IMU sensors for improved driver location accuracy.
Prior to Glow, Lyft has IoT development experience with not only the Amp but also with Bikes and Scooters, Halo ad displays, and even Autonomous research vehicles. Each of these were built with bespoke solutions for their use case, which were hard to retrofit for new device types. Observing similar functionality across these siloed systems motivated us to collaborate with these teams, where feasible, to build new IoT middleware services which could provide a unified framework for managing a variety of devices.
High Level Overview
Just like for the Amp, we continue to use Bluetooth Low Energy (BLE) as the communication mechanism between the Glow device and the driver’s smartphone, instead of opting for a cellular chip in the device. We leverage the driver’s phone as an “IoT gateway” which serves as a communication link between the Glow device and the Lyft backend.
The Lyft backend consists of services which act as the brain of the whole operation, determining everything from whether the driver is eligible to use a Glow, to controlling every aspect of the driver’s Glow.
The Glow device itself is controllable via a well defined request-response based command framework that was developed in-house based on our requirements.
This post will detail a few simplified foundational components that make up the IoT system for the Glow including:
- Provisioning and Authentication
- Control and Communication
- State Management including the firmware update process
Lastly, we’ll briefly discuss what’s next for the Glow program.
Provisioning and Authentication
For any IoT device, provisioning refers to the process of creating a unique identifier for each device and registering the device in a central Device Registry, so that the state of each one in the fleet can be tracked and managed.
Common provisioning processes typically involve:
- Flashing a bootstrap URL onto the device during manufacturing and then letting the device self-identify by connecting to the URL when activated.
- Having the end user manually register their device through an app or website when they start using the device.
After a device has been provisioned into the Device Registry, future communication with the device requires authentication to verify the device is genuine before it engages with the IoT system. Authentication mechanisms such as verification of on-device certificates, symmetric or asymmetric key cryptography, or more sophisticated hardware solutions can be employed depending on the sensitivity of the data being transferred to and from the device.
For the Amp, we simply relied on treating the MAC address of the Amp device as its unique identifier, which would be read by the driver’s phone and relayed to the backend. Phone manufacturers though can choose to anonymize MAC addresses, providing an obfuscated and random hash of the MAC address every time it is read. Additionally, we had no concept of a dedicated Device Registry then and were simply associating each device’s MAC address with a driver record. Since each device could report multiple identities, we faced obvious challenges with device state management and accurate user-to-device association and tracking.
In order to circumvent the problems faced with the Amp, we designed the following on-the-fly provisioning process for the Glow:
- Each Glow device is assigned a unique serial number during the manufacturing process. This number is stored within the Glow’s internal flash storage and on a barcode which is affixed to the device packaging.
- As part of shipping a device to a driver, the above barcode is scanned and mapped to a specific Lyft driver in the Device Registry service. Once this is complete, the Glow is shipped out to its new owner.
- When the driver receives their Glow device and tries connecting it to their phone, the Lyft Driver app will leverage a pre-shared key and the serial number of the Glow to authenticate the device before any communication can begin.
The diagram below summarizes the different states that a Glow device transitions through, as described above.
The above flow not only helps provide detailed device metadata and accountability for devices issued to drivers, but is also meant as a security measure. Unlike the Amp, a Glow device is made to only illuminate when it has successfully authenticated with the Lyft backend services and when the driver is actively driving for Lyft. This prevents malicious parties from impersonating Lyft drivers with a Glow device obtained from someone or somewhere other than Lyft.
Device Control and Communication
Device Command Framework
Each IoT device needs to be controllable by means of pre-determined operating instructions. These can be as trivial as turning the device on or off, or more involved, like playing a certain animation file on the LED display, such as for the Glow. A device is made remotely controllable through its firmware i.e. the on-device software responsible for deciphering and executing instructions to make the device operate as required.
The instruction set aka the “command set” built for the Glow is simple, yet powerful. It supports instructions to individually control various functionalities such as animation file playback, adjusting device brightness, uploading file to device storage and even monitoring on-device sensors. These types of fine grained device commands are also made highly configurable which enables robust, predictable, and easily extensible behavior from the Glow device.
As an example, the instruction for file uploads to the device unlocks the ability to add more animation options for riders to choose from in the future. The alternative would have been to bundle the additional files as part of a new firmware update, which would be much slower to build and roll out. Additionally, firmware resourcing is often more limited, delaying launch of even simple functionality like the above.
Device Control Flow
The Lyft backend is responsible for controlling the device using the command set described above. We built a backend service, “Device Controller”, which listens for different types of relevant event triggers to determine appropriate instructions for the driver’s Glow device. For example, when the service receives a trigger indicating that the driver is in close proximity to the rider’s pickup location, it issues an instruction for the Glow to play the rider selected pickup animation.
The mobile client acts as the communication gateway, helping with essential flows such as command relay and BLE protocol translation as well as buffering for data transfers when needed.
Each server generated command is delivered to the mobile client by means of a server streaming component which exists in Lyft’s infrastructure. The client simply needs to subscribe for the commands once a Glow is successfully connected to it, in order to receive real-time data from the server.
The entire command set has been divided into two categories, based on how the command is to be handled by the client:
- Passthrough commands: These are simple commands which do not require any pre-processing or state management by the mobile client and can be forwarded directly to the Glow device via BLE. Examples include updating the device’s brightness or updating the animation being played on the device.
- Complex commands: These typically require the client to pre-process payload data, download files from the Lyft backend, buffer data, or maintain some sort of state. Transferring files to the Glow is an action that requires complex commands. A single command is sent from the backend to the client. The client is responsible for first downloading the new files to internal storage, splitting the file into smaller chunks, and sending each chunk to the Glow in a separate command.
Some of the more nuanced details about command ordering/prioritization, data encoding, retry mechanism and encryption mechanism between the client and Glow have been omitted in order to avoid complicating the overall flow we’re trying to depict here.
All commands include a unique identifier to enable tracking of the control flow at any given time. The Glow will send a response for every command message it receives. This response will include whether the action succeeded or failed, an error message describing the failure, or any data that might have been requested by the command. All responses are transmitted to the Lyft backend so that they can be used to diagnose or debug issues.
The end to end flow has been summarized in the image below.
Device Data Streams
While the above section details communication directed to the Glow, we talk briefly here about data originating from the device itself.
The Glow streams data to the client so that neither the server nor the client has to continuously request new data from it. There are 4 types of messages that the Glow generates:
- Sensor readings: Data generated from on device sensors which includes internal device temperature, display brightness, inertial measurements from gyroscopes and accelerometers, position and speed of the vehicle.
- Diagnostic information: The Glow provides information regarding its performance and any errors it encounters during its normal operation. This information is necessary for software maintenance and to keep the Glow device running in optimal condition.
- Warning messages: These are triggered by the Glow when it is operating beyond the bounds of its normal operation. For example, a temperature sensor on the Glow warns the client that the Glow device is running abnormally hot. In such cases, the device will self trigger a sleep command, which will render it inoperable for a duration of time before it can be restarted again.
- Heartbeat messages: This periodic stream of data reflects the current device state such as the installed firmware version, device assets, etc. Such state is explicitly managed as is detailed in the section below.
State Management
An IoT device’s state is a set of properties that represent its current configuration. This state should be updatable for device management and readable by the backend for monitoring and debugging issues with the device. A device can either report its state asynchronously by pushing periodic updates to the backend or synchronously by responding to a request for it.
A “Device Shadow” service maintains a mirror of each device’s current state using the information received from the device. The service also manages the desired state of the device, which can be updated by systems integrated with the service. The Device Shadow service will detect the difference between the current and desired state state and issue appropriate commands required to transition the device to the desired state. This is done in an eventually consistent manner, thereby guaranteeing synchronizing a device to its desired state irrespective of its current connectivity status.
The state information for the Glow includes the current firmware version, a list of files stored on the device, the display brightness and configuration parameters for various device functions. The Glow broadcasts its state information asynchronously to the client to forward to the backend. A simplified flow depicting this process is shown below.
Lastly, we discuss how device firmware is updated via the Device Shadow service.
Firmware Update Process
During the lifetime of an IoT device, it’s almost guaranteed that the firmware will need to be upgraded to add new functionality, address bugs, or patch security vulnerabilities. Some devices can be manually updated by means of a human operator, but most need to support remote over-the-air (OTA) updates. Without a human operator, the firmware update strategy needs to be robust to internal and external problems such as firmware not initializing properly or an update that was only partially installed due to a power interruption. If the system is not designed properly, a device can become inoperable, often referred to as being “bricked.”
A new “OTA Manager” service was created to manage the firmware update lifecycle of devices, including Glow as well as other IoT devices developed by teams within Lyft. The OTA Manager service is wired to an internal developer UI which allows configuring the desired firmware version, the S3 location of the firmware file, the rollout percentage and a set of attribute filters which allow fine grained control of which devices the firmware update should be applied to.
The Device Shadow service interacts with OTA Manager to check for any differences between the installed firmware image on the device, as read from the most recent heartbeat message, and compares it with the desired firmware version. If a difference is detected, a new complex command for file upload will be issued by the server and managed by the client as described earlier in the post, to send the new firmware file to the Glow device.
As the Glow receives the new firmware image, the contents are saved to a different memory location than the currently running version. Once the full image is received, the Device Shadow service commands the device to reboot.
On power on, the Glow runs a small piece of software called a Boot Manager to detect a new pending firmware update. When one is detected, the following happens:
- The previous firmware image is copied from internal to external storage and vice versa for the new firmware image.
- The new firmware image is booted and several checks are performed before confirming that the new firmware image is functioning properly. For example, the device needs to connect to the mobile client via Bluetooth and successfully authenticate with Lyft backend services.
- If any fault occurs, the device will reboot and revert back to the previously working firmware image. The Glow will also attempt to transmit the reason for failure in its next heartbeat message.
- If anything goes wrong during this revert process, a recovery firmware image will be used, which was flashed onto the device as part of the manufacturing process and cannot be changed.
All of these mechanisms combine to help mitigate the risk of bricking a Glow device.
What’s Next ?
The Glow is actively rolling out in markets across the US, with over 30,000 devices already live!
The systems built above are standing strong, but we’re keeping an eye out for any room for improvements in our systems and are already sharing the knowledge gained with other teams at Lyft.
Lyft is hiring! If you’re passionate about building software, visit Lyft Careers to see our openings.
