Micro Frontends: Deep Dive into Rendering Engine (Part 2)

Zalando's Fashion Store has been running on top of microservices for quite some time already. This architecture has proven to be very flexible, and project Mosaic has extended it – although partially – to the frontend, allowing HTML fragments from multiple services to be stitched together, and served as a single page.

Fragments in Mosaic can be seen as the first step towards a Micro Frontends architecture. With the ambitions of the Interface Framework as presented in the first blog post, we did not want to just stop at serving multiple HTML pieces, we wanted more:

• Implemented once, works anywhere - UI blocks should work in different contexts and be context-aware, not context-bound.
• Declarative data dependencies - Components get the data they need but do not re-implement data fetching over and over.
• Simplified A/B Testing - Zalando's decisions are data driven, so experimentation is at the core of our decision making. Running an A/B test that spans multiple pages and user flows should be possible with minimal alignment and zero delivery interruption.
• Feels like Zalando - We want a consistent and accessible look and feel for all user journeys and ability to experiment with design fast, across multiple user flows.
• Power to the engineers - Any developer should be able to contribute to all the Fashion Store experience. This means universal tooling and setup, first-class React integration, easy testing (also for work-in-progress code), and continuous integration.

That's how Renderers came to be.

Introducing Renderers

A Renderer is a self-contained Javascript module that runs inside the Rendering Engine framework. It fully relies on the framework to encapsulate all the implementation details like data fetching and layout composition.

A Renderer declares its data dependencies using GraphQL queries and, based on that data, provides a visual representation of a single Entity type (check Part 1 for a detailed explanation on Entities).

This visual representation is a React component, but data management and layout composition is handled solely by the Rendering Engine framework.

So, Renderers are visualisation components for Entities.

The mapping of Entities to Renderers is one-to-many, since different visual representations may exist for a given entity type. A Product Entity, for example, can be represented as a detailed product page, or as a compact card component in collection view. Each Renderer, on the other hand, corresponds to one specific entity type only.

All Renderers share some important properties:

1. Renderers are composable. A Renderer is able to embed other Renderers as children, or be embedded by other Renderers.
2. Renderers are declarative. They specify their dependencies and behaviour but delegate all implementation to the Rendering Engine, the framework that runs them.
3. Renderers are self-sufficient. A Renderer can visualise its Entity no matter on which page or in which context it appears. This ensures that the choice and arrangement of Renderers remains as flexible as possible.

Enabling dynamic content for Zalandos’ mobile apps

Project Mosaic was solely focused on the web. However, Zalando offers its Fashion Store as two experiences: the Web and the Native Apps. Since they share most parts of the user journey, it was natural to explore if the Apps could benefit from a system based on Entities and Renderers, too.

We knew it would be too much of a stretch for Mosaic fragments. But there's literally nothing that binds Renderers specifically to the Web!

In the Zalando app, we had already implemented server-side layout steering for some parts of the application experience such as the main App landing page. Instead of relying on hardcoded views, the app would receive layouts from a remote Zalando server over the network. The preferred format here would be JSON, but otherwise the same challenges were present: we wanted dynamic, personalizable UIs with declarative data dependencies.

If Renderers were able to output JSON instead of HTML, we could reuse the same rendering core as for the web with the same benefits. Our Renderers relied on React for their output. To cover the app-specific use case, we added a custom React reconciler that consumed custom React elements, and output app-compatible JSON instead of HTML. Now, web developers are able to contribute Native apps features by reusing the same set of APIs as they were used to deliver web experiences and bring the web and native apps experiences closer together. All the existing tools, infrastructure support, and the constantly evolving platform APIs are now shared.

The life of a Renderer

So, how does it look under the hood?

We decided to organise the Renderers API as a set of so-called life cycle methods, each accepting a function declaring Renderer's behaviour for a given context or case. All Renderers are implemented using TypeScript.

Let’s have a look at a simplified version of a collection carousel Renderer:

import{MOVE}from"@tracking/event-names"; import{SimpleCarousel}from"@dx/react-carousel-tile"; import{tile,ViewTracker}from"@if/rendering-engine/api"; import*asReactfrom"react"; import*asqueryfrom"./query.graphql"; exportdefaulttile() .withQueries(({entity:{id}})=>({ carousel:{query,variables:{id}}, })) .withProcessDependencies(({data})=>{ if(data===null){ return{action:"error",message:"No collection data found."}; } return{ action:"render", data, tiles:{entities:getCollectionEntities(data)}, }; }) .withRender((props)=>{ const{ data:{collection}, tiles:{entities}, tools, }=props; return( <ViewTracker> <SimpleCarousel {...collection} onNextClickCarousel={()=>{ tools.tracking.track({name:MOVE}); }} > {entities} </SimpleCarousel> </ViewTracker> ); });

Renderers are implemented using the fluent interface approach. By calling the tile() function of the Rendering Engine API, we are setting up a Renderer that defines various lifecycle methods. Each method receives a function that encapsulates the associated behaviour and has fully typed interfaces. Since renderers are declarative, they do not execute any of the lifecycle methods themselves. Instead, the Rendering Engine framework runs all of them, in due order and context, fetches data and dependencies, and passes the output down to other methods when necessary.

The most important lifecycle methods are:

withQueries

Declares a data dependency via a GraphQL query. Data is fetched automatically by the framework and is available when the other life cycle methods are called.

withProcessDependencies

Based on data delivered by withQueries, defines further action (render, error etc.) and allows data pre-processing, which is then passed to the withRender method. The chosen action tells the Rendering Engine that the Renderer should redirect, or be displayed in an error state.

This life cycle method is also responsible for specifying child entities of the current Renderer. In this example we want to display the collection entities as outfit or product cards based on their entity type. It is important to note that a given renderer does not know which renderers will be used for its child entities

withRender

Returns the root React component to be used as the Renderer output.

For the Web, this is transformed into HTML and rendered on the server (SSR). Later on, the markup is hydrated on the client side with the data. For the Apps, we use a custom React reconciler and custom (non-Web) components to output JSON instead of HTML. However, most of the data flow, dev tooling and infrastructure remain the same for both use cases.

There are more advanced features by using Renderers:

• Progressive Hydration: we can mark specific renderers to be hydrated early, i.e. kicking off their React hydration as fast as possible on the client-side, and thus making its content interactive before its parent renderer.
• Code Splitting: we only load and parse the Renderers needed on a given, personalised page which gives us a good performance out of the box.
• Renderer State: Renderers have access to a local Renderer State. The concept is similar to React’s setState. It enables you to re-run renderer lifecycle methods for example to fetch additional data, and re-render the updated child entities. The "classical" React state can still be used via React Hooks.

Data sharing

Renderers are not intended to share data with each other that is based on the client side state. We want to avoid unwanted data coupling and allow Renderers to be reused in other contexts with minimal risks.

Renderers have access to Zalando’s GraphQL Mutation APIs which allows remote data to be modified. Since all Renderers use the same data schema for their data dependencies, they can subscribe to changes in the schema to limit the need for cross-renderer communication.

Rendering Engine

Rendering Engine is the framework powering the Renderers. It is a backend service written in TypeScript and running in NodeJS coupled to a client-side Javascript module that runs in the browser.

Rendering Engine encapsulates all the complexity and implementation details for the declarative Renderers. It processes incoming customer requests, matches Entities to Renderers, fetches data and other dependencies such as A/B testing assignments, asynchronously renders the response and delivers it back to the Web and Native App clients.

The following sections describe the main responsibilities of Rendering Engine.

UI Composition

All layouts in Interface Framework are represented as trees of nested entities that are visualized using the matching Renderers. The mapping of Entities to Renderers is fully described by a set of rendering rules.

In computer science terms, Rendering Engine recursively and asynchronously transforms a tree of entities into a tree of UI elements. On each step, it takes an entity node and its metadata as input, outputs a UI node plus zero or more child entity nodes, and then recurs over children.

The page rendering always starts with an Entity. We call it the Root Entity since it typically defines what the page is about. After the Rendering Engine receives a request, it extracts the root Entity from the request headers and looks up a matching Renderer. Once a Renderer is found, the Rendering Engine runs the Renderer lifecycle methods to fetch data. In case there are any child entities associated with this Renderer, the same resolution process happens recursively. Thus, each Renderer may "suggest" which entities should be rendered as its children, but has no control over the actual renderer choice. That choice is based exclusively on the Rendering Rules.

The important part here is that we do not block the resolution process. As soon as the entity is matched to a Renderer and the data resolved, the Rendering Engine kicks off the rendering process and starts streaming the HTML content to the client.

Data Fetching

The Rendering Engine takes care of fetching the GraphQL queries from the Fashion Store API. It uses an implementation of Perron, a data client with built-in support for circuit breakers, error handling and retries.

All queries to FSA are batched and cached based on a DataLoader implementation. This prevents duplicate calls to backends during the same request.

Universal Rendering

Zalando being an e-commerce platform, our typical web page would have a prevalence of static content with islands of interactivity and we aim at serving content as fast as possible. This is why Rendering Engine was built from the ground up with full Server-Side Rendering (SSR) support. Each Renderer first generates its markup on the server and the Rendering Engine stitches it all together and streams the HTML to the client which then hydrates the components using our runtime module.

For the Web use case, we provide additional Zalando-specific APIs which add interactivity, mutate data if necessary, lazy-load extra contents etc. For the Native app, the Rendering Engine only serves the JSON markup and the actual rendering happens in App clients for iOS and Android.

Mosaic backward compatibility

We knew that the migration from Mosaic to Interface Framework would not happen in a day. Our Mosaic codebase was extensive and actively maintained. Therefore, the Rendering Engine allowed Mosaic fragments to be used directly inside Renderers.

This made our migration path very smooth. In fact, we now view Mosaic fragments as a powerful API our framework supports, and we still use them sometimes. In addition, this opened up extra integration and observability benefits for the legacy implementations.

Monitoring and Tracing

Improved observability is yet another benefit of the integrated platform. The Rendering Engine automatically collects and reports Web Vitals so that we can correlate performance variations with code changes. A number of custom client-side metrics are also collected. All this happens automatically, so developers who contribute to Renderers can focus on the customer experience We also integrate a variety of common enterprise tools for logging aggregation, Open Tracing and client-side error monitoring, with zero-integration time for the Renderer developers.

Developer Experience

Rendering Engine focuses on providing a great developer experience with the following features:

• Local Development Environment: the framework provides an integrated development server and an on-demand compilation of Renderers. It only builds the Renderers that are shown on the current page. This ensures fast build times even when more and more Renderers are added to the application.
• Multiple version support: Rendering Engine uses the Zalando Design System as a UI component library. The UI components are defined as dependencies for each particular Renderer. To allow greater flexibility, it supports using multiple versions including convenient tools and hooks to simplify the version maintenance.
• Continuous Integration & Deployment: New code changes get tested and built automatically with specific performance reports for every page. These reports include bundle sizes and Lighthouse metrics. The deployments to Kubernetes happen continuously in preview and production environment.
• Automatic Persisted Queries: all GraphQL queries to the Fashion Store API are persisted on the server side together with a unique identifier. It helps reduce the request size, since the Rendering Engine client runtime sends the identifier instead of the whole query string.
• Localization: Rendering Engine supports localized bits of text inside Renderers.

Page Rendering Explained

Let’s have a look at what happens in Interface Framework on a high-level when you visit a page on the Zalando website. In this example, the user visits an outfit view by choosing one from Zalando’s Get the Look page.

The request gets picked up by Skipper, which is an HTTP router and reverse proxy for service composition. Skipper identifies the matching route and forwards the request to the Rendering Engine along with the entity parameters:

entity-type:"outfit" entity-id:"ern:outfit::4NXOAez0Qti"

The Rendering Engine gets the request with the entity above, that is called the root entity. The root entity defines the main content of the page. Based on the Rendering Rules, a matching Renderer is selected for this root entity.

For the outfit page, the set of Rendering Rules looks like the following:

exportconstoutfitViewRule:RenderingRule={ selector:{entity:"outfit"}, renderer:"outfit_view", children:[ { selector:{entity:"outfit"}, renderer:"outfit_highlight-b", children:[ { selector:{entity:"product"}, renderer:"product_horizontal-highlight-product-card", }, ], }, { selector:{entity:"collection"}, renderer:"collection_simple-carousel", children:[ { selector:{entity:"outfit"}, renderer:"outfit_outfit-card", }, ], }, ], };

The Renderer for the root entity is the Outfit View Renderer. We can refer to it as the top-level or root Renderer for the request. The Renderer has a data dependency in the form of the following GraphQL query.

{ outfit(id:"4NXOAez0Qti"){ id creator{ variant{ name } } relevantEntities(first:2){ edges{ node{ id } } } } }

The query is executed in the Fashion Store API and various parts of the query go through different resolvers depending on the fields that are present. Each of the resolvers then calls one or many microservices that provide data.

In our example, we ask for the creator’s name of the outfit together with two relevant entities. One resolver will call the Recommendation System to get the relevant entities for this outfit. Here, our relevant entities are a collection with other outfits from the same creator and a collection with outfits that look similar.

Each Renderer decides which relevant entities appear as its children and adds placeholders for them. This is achieved via the withProcessDependencies lifecycle method. The Rendering Engine picks up all relevant entities and determines matching Renderers. For each of these nested Renderers, the process repeats recursively until no more nested entities must be rendered.

After all the Renderers and their data dependencies are collected, the Rendering Engine renders the React components of each Renderer and streams the content to the client. The next picture shows a sketch of the outfit page that is divided into the corresponding Renderers. Each Renderer is responsible for one part of the page.

Conclusion

We have presented a deep dive into Rendering Engine with all its key functionalities. The final part of this blog series will cover a comparison between Mosaic and Interface Framework and what we have learned during the migration.

Update 2023/07: See Rendering Engine Tales: Road to Concurrent React for an update on Rendering Engine and how we integrated React Concurrent features as part of our upgrade to React 18.

We're hiring! Do you like working in an ever evolving organization such as Zalando? Consider joining our teams as a Frontend Engineer!