Crafting Seamless Journeys with Live Activities
Building on our previous journey through design with, “That little island changes everything,” we now invite you to a technical rendezvous. If you haven’t explored the design aspects, we recommend a quick detour to understand the visual symphony that laid the groundwork. In this edition, we transition from pixels to code, exploring the technical side that orchestrates Live Activities at Lyft.
Our primary focus will be on the client-side aspects, with a brief overview of the server functionalities. To maintain brevity, we’ll refrain from providing an exhaustive description of the inner workings. It’s worth mentioning that we won’t delve into the basics, such as widget extension setup or presentation styles and methods, as these can be readily found in the Apple Documentation. Our aim here is to offer insight into three main areas: how we manage lifecycle events, create adaptable UI elements, and seamlessly integrate remote image support.
Lifecycle
Our activities’ lifecycle is server-driven, which provides more flexibility than a client-driven approach. There are a few key benefits: it allows updating behavior for all app versions at once, we can roll out even more specific experiences over time depending on the ride state and offer, and we avoid struggling with the long tail of different behavior in old versions. To support remote updates, it’s essential to send the server tokens, live activities’ start / end conditions, and content to be shown.
With that in mind, we created endpoints for publishing, updating, and invalidating token states, as well as one for generating the active content for the live activity. The ability to control content from our endpoint gave a few advantages:
- By checking the data present in a response, we control whether a live activity should be started based on the current app state.
- By using content from a response, we can initialize the state for a Live Activity start and keep building Live Activity content consistently — with all the generation logic living in one place. This also ensures consistency with content for the similar feature on Android (but this won’t be covered in this article).
Starting the activity:
- Request content state
- If response contains data, request Live Activity from iOS with initial content we received from server and start it
- If activity started, observe push token generation and updates, and send it to server
- Mark live activity as started
- Process any further local or remote updates
Let’s take a look at the scenarios of how live activity could be finished:
- Scenario one: Manual action by the user (swiped left on lock screen).
- Scenario two: If the current local app state is ineligible for a live activity or the time-to-live (TTL) for the activity’s content is reached.
- Scenario three: explicitly by remote update (event=end) if server decides that it’s no longer needed.
The main thing that should be done on the client side when an activity finishes is to send a token invalidation request to the server. A token could be already invalidated on the server if ended by remote update, but we always send an invalidation request to simplify and acknowledge invalidation from the client side.
So far we’ve considered success scenarios, but what could go wrong?
- Request Live Activity from OS could fail. The most frequent reason for this is that the application is in the background, but there could also be others. We mitigated this with a retry mechanism that caches the active ride ID when the live activity initiates, and restarts if the latest ride ID does not match.
- An activity started, but remote updates are not received in reasonable time.
Here stale_date shines (added in iOS 16.2+). We updated our minimum app version support for Live Activities to 16.2+ because of two important improvements: frequent updates and stale_date.
Every content state (local or remote) should have some reasonable stale_date.
We use stale_date to detect outdated content and to kill the activity when applicable. As for analytics, each content contains a property to differentiate whether it was updated from the app/locally or by remote update via APNs.
With local content, an activity could be killed in three cases:
- Token was not generated, so no remote updates could be received. We see a pretty low volume of this (less than 0.1% of all started activities), but it should still be handled properly to not confuse users with the initial state of an activity that was never updated.
- Token generated but failed to send because of connectivity or internal server issues.
- Token successfully posted to our server, but the update was not received within the X minutes limit. Possible issues include: throttling by Apple, delivering APNs failed, or our service is down.
Our default time limit, after which the system considers an initial content of the started Live Activity to be out of date, is 3 minutes, which is configurable on the server.
Showing up-to-date content and handling outdated content should be a number one priority for any app that supports Live Activities.
In addition to stale_date on the client, we have Freshness Monitor on our server service.
Freshness Monitor is a mechanism that continuously monitors all Live Activities push tokens and timestamps. It classifies all live instances into three buckets:
- Fresh: these are instances of Live Activity from which a TTL is in the future, or past by less than our defined time limit.
- Stale but recoverable: these are instances when we didn’t send any updates after the TTL, but there is still a chance that we may recover them. For such instances, we do several high priority retries to send an update and move them to the Fresh bucket.
- Severely stale: if we come to this point, it’s better to just kill an instance of Live Activity to prevent displaying outdated information. An end event will be sent to that instance and the token will be invalidated on successful delivery.
If you’re eager to explore the server intricacies of the Live Activity feature further, feel free to drop a comment. Your engagement will guide us in crafting a separate article dedicated specifically to those details!
Dynamic UI
Live Activities is a feature with high exposure to our users and should always show up-to-date and valid information. We operate like our activity is a separate mini app on top of our main one. At Lyft, many teams work on features in parallel in our main app and run dozens of experiments, which affects the live activity as well. It’s very easy for engineers to forget to update Live Activity content for some new state or keep everything in sync between iOS and Android.
Keeping that in mind, we decided to build live activities content to be fully server driven using server-driven user interface (SDUI). We already use SDUI frameworks for some screens in our apps that overlap with the Live Activity, so reusing basic models and familiar patterns helps us move faster.
We’ve cloned and simplified all existing, required SDUI models to stay within the limited payload size (4KB) for content updates. We also use protobuf which additionally reduces payload size.
Knowing all inputs, let’s dive into covering mechanics and structure of how to support all the possible Live Activity states by UI components.
Accessibility
Our main SDUI building block is the RichText component that can render a formatted text, image, or timer. Additionally, our RichText component has an accessibilityText field to specify a custom description for VoiceOver. We use it to guarantee an exceptional experience for accessibility users, something we continuously focus on while developing the Lyft app.
For example, we prepend the VoiceOver label with “Lyft status: …” for each Dynamic Island, Lock Screen, and Expanded Live Activities primary heading so users know that all subsequent info is about their Lyft.
We also support Dynamic Type (text scaling) by ranking all text shown in Live Activities and truncate the least important labels first.
How it looks in code:
/// Text container for displaying strings with optional timers or icons
public struct RichText: Codable, Hashable {
public let elements: [Component]
public let accessibilityText: String
}/// Component of the rich text. Each component represents a single string/icon/timer
public struct Component: Hashable, Codable {
/// Type of the text component
public let type: ComponentType
/// Color style of the text component
public let color: ColorStyle
}
/// Available rich text component types
public enum ComponentType: Hashable, Codable {
/// Text element with associated string
case text(String)
/// Icon element with associated icon type
case icon(Icon)
/// Timer element with associated timer type and duration in seconds
case timer(TimerType, TimeInterval)
}
/// Available color styles for the rich text
public enum ColorStyle: Hashable, Codable {
/// Text is rendered using solid colors appropriate to context
case solid
/// Text is rendered using a gradient
case gradient
}Multiple Layouts for Each State
At the top level, our content model has three layout properties:
- minimal: has one field of RichText type
- compact: has leading and trailing content of RichText type
- expanded: is more complex and has several layers of nesting, with RichText as a main building block
Minimal and compact fields are responsible for Dynamic Island’s representation of corresponding states.
Expanded is responsible for rendering an activity on the Lock Screen and the expanded state of the Dynamic Island.
Discrete properties for each layout state’s content give us full server flexibility of content in both Dynamic Island and Lock Screen.
Progress Bar
Another important core component we use at Lyft is the progress bar.
Server calculates the exact position of the car on the bar and the stop positions for that ride.
It was important to build a solution that could support any use case — like finishing a previous ride, stops throughout a ride, or different modes—and specify progress mechanics dynamically (see Progress Mechanics section in our previous article).
This is how we represent the progress bar state:
/// Model containing information required for rendering progress bar
public struct ProgressBarDetails: Codable, Hashable {
/// Point describing current position on the progress bar
/// allowed range: [0, 100]
public let currentPosition: Int
/// Array of route points on the bar
public let routePoints: [RoutePoint]
}
public struct RoutePoint: Hashable, Codable {
/// Type of the route point
public enum PointType: Hashable, Codable {
case stop
case waypoint
}
/// Type of the route point
public let type: PointType
/// Position of the point. Allowed range: [0, 100].
public let position: Int
}Finally, the last component we utilize is a RemoteImage that consists of URL and raw data when available. We will discuss working with remote images in more detail in the next section.
In summary, by simply using three basic components: RichText, ProgressBar, and RemoteImage, we could build all the necessary states of our Ride State Live Activity.
Real-life Example
Let’s examine the UI elements comprising the “Accepted” ride state (when the Driver is on the way to the Rider’s location):
For the image above, the color-coded frames represent the following:
- Green—RichText components
- Orange—RemoteImage components
- Purple—ProgressBar
- Blue—static Lyft logo configured on the client
- Gray—a container for all trailing components
Now knowing how the UI is structured and which components are used, let’s review an example of content payload for the “matching” ride state (when a Rider is waiting to be matched with a Driver). This is our simplest case:
Payload:
{
"ride_content": {
"minimal": {
"content": { "icon": 2 },
"a11y_text": "Lyft status: Looking for nearby drivers"
},
"compact": {
"leading": { "icon": 2 },
"trailing": {
"icon": 3,
"a11y_text": "Lyft status: Looking for nearby drivers"
}
},
"expanded": {
"scheduled": {
"main_content": {
"heading": {
"strings": [
{ "text": "Ride requested" }
],
"a11y_text": "Lyft status: Ride requested"
},
"subheading": {
"strings": [
{ "text": "Looking for nearby drivers" }
]
}
},
"bottom_content": {
"strings": [
{ "icon": 7 },
{ "text": "Details in 1—3 min" }
]
},
"show_shimmer": true
}
}
},
"stale_date": "2024-05-01T10:09:39.347Z"
}Remote Images
Our main use case for remote images is showing the driver’s profile photo and car as soon as the rider is matched. These should also be updated in any case of rematching. We start live activity immediately after requesting a ride, so we don’t know anything about the driver at that stage.
We live in a world of constraints (at least for iOS 17 and below), where each Live Activity runs in its own sandbox, and unlike a widget, it can’t access the network or receive location updates. Because of that, using AsyncImage to render the imageUrl isn’t an option.
In the Ask Apple Live Activity session, Apple encourages handling images by using App Groups to share files through the group’s container. You can find how to enable App Groups for your app in the official documentation.
With this approach, images can be downloaded and shown only when an app is active, or is backgrounded and has remaining background processing time. If the system or user terminates the Lyft app during the “matching” state, the image won’t load until the user opens the Lyft app. Without getting into too much detail, we have not found a way to reliably and quickly display two simultaneous images (driver picture and car). We agreed that the driver’s profile photo is more crucial to see first from UX perspective, and we can show it without any delays by sending base64 image data directly into the APNs update payload. It sounds pretty simple but one has to remember that there is a 4KB payload limit!
Another complexity comes from the widget extension updates. Those don’t share any in-memory state, so if an image should be shown in every update we either need to send image data in every update or send image data only once, mark it on server as sent and cache it on disk from the Widget extension. We do the latter.
The last highlight before diving into implementation details is how to fit an image into such a limited payload. Our images are usually bigger than ~2.5KB of available free space. So we apply several transformations to the image which will be sent as data but leave the URL to the original image.
Driver’s profile photo transformations:
First, we crop it to a square and downsize to 72x72 px (acceptable tradeoff between quality and size). It doesn’t require transparency, so we convert to .jpg format if needed (smaller than .png). These two transformations should yield images smaller than 3KB. If the size of image + content data is still more than 4KB, we downsize the image more and reevaluate total payload side. Then, if the payload is still too big, we drop image data and send content without it, which guarantees successful delivery of any updates. So despite the fact that we have achieved the ability to show an image immediately on update, image quality is lower than the original one. Because of that, the last step of this flow is downloading the original image from the app when possible and replacing the thumbnail with it.
Let’s summarize the client’s steps on each state update:
Widget Extension:
- Search for image objects in payload. Exit if no images.
- Check image data in received payload. Save it to shared storage if it has no cached data for that URL.
- Create an image from base64 data if it exists.
Main App (Observe state updates):
- Search for image objects in payload. Exit if no images.
- Use image URL as a key to check shared cache for original image data. Download the original image and save its data to shared storage if it does not exist. Low quality image data received via APNs updates in extension will be replaced.
- Call update method of Live Activity with current state when image is downloaded. All remote images in Live Activities rely on the shared image cache so an update call is needed to refetch the cached image by URL.
- Remove all cached images’ data when activity is dismissed.
For payload content we use the following model:
struct LiveActivityRawImage {
let url: URL
let rawEncodedContent: String?
}And we have `LiveActivityImageHandlingService` to save image data to the App Group’s shared container.
Model:
struct LiveActivityImage: Codable, Cacheable, Equatable {
enum ImageSource: Codable, Equatable {
/// Low quality image is obtained from APNs update in Widget Extension.
/// Used for showing thumbnail without delays.
case remoteUpdate
/// image is downloaded from the main app
case mainApp
}
let rawData: Data
let url: URL
let source: ImageSource
}Service API:
public protocol LiveActivityImageService {
/// Saves an image if necessary (if image is not present or lower quality one is cached)
///
/// - parameter image: Live activity image, with indication of its source
///
/// - returns: True on success, false if image is already saved or failed to save
@discardableResult
func saveIfNeeded(image: LiveActivityImage) -> Bool
/// Retrieves a live activity image if present
///
/// - parameter url: original URL for the image
///
/// - returns: cached Live Activity image, if present.
func getImage(for url: URL) -> LiveActivityImage?
/// Removes all saved images from the cache
func pruneAllImages()
}Given the straightforward nature of implementing this protocol, we’ll refrain from delving further into its implementation details.
Conclusions
Achieving a balance between flexibility, reliability, and reusability necessitates careful consideration of implementation tradeoffs. For example, having static content on a client and building UI solely from data received from the server could speed up development, but limit further updates or make it difficult to experiment with content on the fly. Not showing the driver profile image right away when available could have limited the scope, but we valued providing the best experience to our users.
Our implementation of live activities offers a dynamic enhancement to the Lyft app. We were able to support easy adoption of new use cases and quick iteration, while keeping the live activity and app in sync. The shared server infrastructure and endpoints enable a similar feature on Android too. Altogether, this solution for live activities was a success for Lyft and scales to millions of users per week!
Your insights matter. Whether it’s about client details or any aspect related to Live Activity or mobile development in general at Lyft, we’re excited to receive your feedback. Your comments will steer our content creation process and ensure we cover topics that resonate with you.
Lyft is hiring! If you’re passionate about developing state of the art machine learning/optimization models or building the infrastructure that powers them, read more about them on our blog and join our team.
