YAML Generator for Funnel YAML Files: Streamlining the Mobile Data Workflow Process
At Uber, real-time mobile analytics events—generated by button taps, page views, and more—form the backbone of the mobile data workflow process.
To process these events, our Mobile Data Platform Team designed and developed the Fontana library, which converts the nearly-one-million-QPS (queries per second) volume of events into easily digestible and useful analytics for Uber engineers. As part of this process, funnel YAML files are key config files that are used to define sequences of events for analysis. To this end, our team has also designed and developed the SuperFlurry application, which aids in creating and managing these files.
However, SuperFlurry still required users to create and modify funnel YAML files by editing raw YAML files directly. This was a significant pain point for Uber engineers and PMs aiming to set up their own funnels, especially for those unfamiliar with the format or the specific structure of funnel YAML files, as subtle syntactical errors are easy to make and switching back and forth between file creation and documentation is time-consuming. To remedy this issue, we designed and developed YAML Generator, an application that provides a comprehensive set of options for creating a funnel YAML file alongside a clean and intuitive UI.
The newly developed YAML Generator application builds onto the SuperFlurry application, streamlining the creation of these funnel YAML files. Through the development of this application, the importance of simplifying user-side input and the importance of continuous feedback are highlighted.
Background
The Fontana library is a framework that allows for the processing of large numbers of mobile analytic events. These events are emitted from all Uber Apps in real time, and are converted through funnel YAML files into specific analytics, one of the most important being conversion rate (e.g., percentage of users that purchase a product). Using these analytics, engineers are able to make informed decisions that improve the flow of their applications.
Funnel YAML files
Funnel YAML files describe the specific steps that users take as part of a flow. For example, in order to obtain an Uber Pass, a user might be shown a page offering the pass (Landing Page), tap a button to get the pass (Get Pass), get sent to a purchase screen (Purchase Screen), tap a button to purchase the pass (Purchase), then be shown a screen indicating that the pass was successfully purchased (Purchase Success). Passing this sequence of events as a funnel YAML file into the Fontana library, an engineer could see the conversion rate throughout the entire flow.
Figure 1: Sequence of events for purchasing an Uber Pass.
In addition to simple sequences of steps like the one above, these funnel YAML files can potentially describe more complicated, nested step structures, which might include sequences of steps that are optional or repeated.
SuperFlurry
SuperFlurry is the one-stop shop for creating and managing funnel YAML files, allowing engineers to create new files, as well as update existing files in production. The application also includes several quality-of-life features, such as the ability to look up mobile analytics event names on demand, as well as linking to the funnels dashboard, which displays the various funnels that exist across teams. YAML Generator builds upon the SuperFlurry application to provide engineers an alternative to writing funnel YAML files in raw YAML.
Why YAML Generator?
YAML Generator provides engineers with a discrete and comprehensive set of options as they build their funnel YAML files. This contrasts with writing raw YAML files, where there is no set list of options.
Reduction of Human Error
YAML Generator guarantees that a semantically and functionally correct funnel YAML file will be generated. In addition, it features built-in validation that catches invalid inputs as they occur. This contrasts with raw YAML files, where it’s very easy to write an invalid funnel YAML file and mistakes may be difficult to catch.
Increase in Productivity
YAML Generator streamlines the process through an easy-to-understand UI and provides the user with all the options and info needed to create a funnel, reducing the need to flip back and forth between writing a funnel YAML file and looking at the guide on how to create one. In addition, features such as the clear separation and organization of steps reduce cognitive load for engineers, leading to greater efficiency.
With YAML Generator, engineers can spend less time writing funnel YAML files and more time analyzing the data.
YAML Generator Architecture
We use Fusion.js, Uber’s open source, universal web framework, to create the application UI and Apollo to handle API calls through a GraphQL server. In addition, we use Base Web, Uber’s open source framework for React web components, to style the UI and provide consistency with other Uber applications.
Backend: Apache® Thrift™
We use Apache Thrift to define our APIs and Golang to implement them. For the purposes of this project, we created an API that provides full validation of the funnel YAML files and leveraged an existing API to fetch mobile analytics event names.
Design Progression
Over the lifetime of the YAML Generator project, the application UI has undergone many changes and improvements thanks to feedback from fellow engineers, shown below:
Original Mockup
Figure 2: Non-functional design created for this project’s design review document.
Early-Stage Design
Figure 3: Initial design focused on complete functionality over style.
Mid-Stage Design
Figure 4: Restyled and reorganized design inspired from feedback.
Design Released for Beta Testing
Figure 5: Frontend engineers were enthusiastic about the experimental “YAML Generator Playground,” which allowed users to see both the YAML Generator and raw YAML update in real time. To this end, we migrated the entire application to the playground and released that in beta in SuperFlurry.
Final Design
Figure 6: Inspired by user feedback, several new features were added, including drag-and-drop steps and copy-to-clipboard.
Special Features of the YAML Generator
For funnel YAML files, steps are the building blocks of their functionality. To that end, the application makes it simple to visualize and arrange these steps with drag-and-drop lists, powered by Base Web.
Figure 7: Organization of steps in the final project design.
Lists of Possible Attributes
In our aim to reduce complexity, we worked to provide engineers with a set list of options, rather than requiring them to know what they want beforehand. With that in mind, we created drop-down lists of possible attributes and options for every applicable field.
Figure 8: Options given to users when adding a new step.
Quality-of-Life Improvements
To further improve ease of use and efficiency for engineers, we’ve launched an additional set of quality-of-life features:
- Providing key info for engineers, reducing the need for external lookups
- Autocomplete functionality for applicable fields
- Highlighting invalid input for fields
- Complete editing suite, featuring functionality such as undo, copy-to-clipboard, etc.
Figure 9: The funnel name ‘invalid+name’ includes the invalid character ‘+,’ so the field is highlighted red.
Results and Learnings
This project was launched with the aim of alleviating the issue of complex input: specially formatted YAML files were time-consuming to create, and prone to user error. Through the YAML Generator application, we successfully provide a user-friendly interface that provides a streamlined, error-free method of producing funnel YAML files, which saves time and reduces errors. We hope that this application inspires other engineers to think critically about their workflows and simplify their own inputs wherever possible.
Importance of the Revising Process and Feedback
From draft to final product, there were many key changes and revisions. Some were powered through personal preference, but the most impactful and significant revisions were brought about through feedback. In the early to mid stages of the project, we frequently asked engineers specializing in frontend design for UI-related feedback, guiding the trajectory of the overall design. Then, during the late stages of the project, we launched YAML Generator in beta, allowing anyone to test the application and give additional feedback, inspiring further improvement. Without feedback, our application would not be where it is today.
Conclusion
Designing an application that addresses user pain points can be a very challenging task. The process of creating YAML Generator highlighted the importance of both simplifying input in improving user experience and feedback in making key incremental changes. With the rollout of YAML Generator, we’ve transformed the mobile data workflow process, making funnel creation more accessible and streamlined for all engineers at Uber.
If you are interested in building leading edge applications to process incredible volumes of mobile data, please consider joining our team!
We would like to provide big thanks to Xuan He, Wugang Zhao, An Yang, and Ricardo Villarreal. We also would like to thank the Mobile Data Platform team for providing support, and all engineers who helped test the application.
Apache®, Apache Thrift, and Thrift are either registered trademarks or trademarks of the Apache Software Foundation in the United States and/or other countries. No endorsement by The Apache Software Foundation is implied by the use of these marks.
Lead image by Tumisu from Pixabay.