A Data Scientist’s Guide To Measuring Product Success

If you’re a data scientist on a product team, much of your work involves getting a product ready for release. You may conduct exploratory data analyses to understand your product’s market, or build the data models and pipelines needed to power a new product feature, or design a machine learning model to unlock new product functionality. But your work doesn’t end once a product goes live. After a product is released, it’s your job to help identify if your product is a success.

When building a product, data scientists focus on measuring a product’s performance metrics, such as a:

• Data model’s freshness
• Data pipeline’s speed
• Machine learning model’s accuracy

In contrast, when measuring product success, we need to focus on the product’s impact. In other words: Is the product doing what it set out to do? Now translating the nebulous concept of “product success” into concrete and relevant metrics is no easy task. At Shopify, our data scientists oversee various product releases every year and we’ve learned some best practices along the way for measuring product success.

Below, we’ll walk you through a framework we’ve created for outlining and measuring product success metrics. This framework will help you identify more approachable questions for measuring success, and enable you to turn your metrics into an effective measurement plan.

Shopify’s product success metrics framework

1. Start At The End (User)

When measuring product success, it can be tempting to jump immediately to hashing out metrics based on product goals. But it’s better to start by zooming out and considering your user’s goals, since products succeed when they help their end users succeed.

When looking to identify your end user’s goals, the key question to ask yourself is “What is the end user trying to achieve within the product’s domain?” There are many ways to go about answering this question:

• Talk with your product team to learn how they understand your end user’s goals
• Interview a sample of end users and ask about their objectives
• Put yourself in the place of your end user and think about what your top priorities would be

In all likelihood, you will identify a number of things that your end user is trying to achieve within your domain. In this case, we find it helpful to establish a hierarchy of goals by distinguishing between the end user’s main goal, and the various subgoals that contribute to the achievement of that goal.

For example, this year Shopify launched a new contextual analytics experience that surfaces inventory data in a merchant’s Products page, helping them make decisions about their products right within their workflow. Our first step in defining this product’s success was to think about merchants—our end users—and their goals in the inventory domain. After discussing with our product team, we defined our merchant’s main goal as “improving the health of their product catalog”. A healthy product catalog is one that is appropriately stocked, priced, and marketed. Under this main goal we identified several subgoals:

• Decreasing inventory costs
• Increasing sales and margin
• Increasing traffic and conversion
• Identifying which products to promote

Your end user’s goals may be far removed from the particular product you’re releasing. What we mean by this is that your product may only influence your end user’s main goal indirectly, and may not even touch on several of their subgoals. But that’s okay! Ideally, your product will promote these goals, and not interfere with them. What’s most important is that you keep these goals in mind when defining your product goals, as it grounds the product’s success in what truly matters: the end user’s success.

2. Define Product Goals

Now it’s time to define the goals of the product you’re launching, factoring in the specific ways in which your product aims to support the end user in their goals. The key question here is “How does the product aim to support your end user in their goals?” Determining the answer to this question should be done in coordination with your product team, as they have the most context on what the product is intended to do.

As in the case of defining end user goals, we find it helpful here to define both a main product goal and various subgoals. But whereas end user subgoals are more like specific ways to achieve the main goal, product subgoals are more like steps that progressively build up to the main goal. In other words, product subgoals are the things that need to happen for the product’s main goal to be achieved.

Looking at our analytics example, our end user’s main goal was to improve the health of their product catalog. Our product aimed to support this goal by encouraging merchants to use data to inform the decisions they make about their product catalog decisions, specifically with respect to how they manage their inventory. With this is mind, we defined our main product goal as “to drive merchants to leverage data in their inventory management decisions”, which we broke down into four subgoals:

1. Make merchants data-aware: Merchants will be familiar with the key product inventory metrics and data points relevant to the task of inventory management
2. Make merchants data-driven: Merchants will regularly look at and explore inventory metrics and data points when making inventory management decisions
3. Make merchants data-attentive: Merchants will fill in the gaps in their product inventory data to arrive at higher quality insights
4. Make merchants data-savvy: Merchants’ inventory metrics will improve based on the data-driven inventory management decisions they make

These subgoals are essential in order for us to hit our product goal and get merchants to leverage their data.

3. Translate Goals Into Questions

With your product goals and subgoals defined, you should now define your primary questions. These are the questions that determine whether you’re achieving your product’s subgoals, and define the success of your product.

For example, our analytics product’s first subgoal was to make merchants aware of the data that the product was highlighting. Since merchants had to know about our product in order to be aware of the data it was surfacing, our corresponding primary question was, “Are merchants discovering our product?”

Often, primary questions follow a similar pattern:

This pattern is a useful guide for defining a product’s success, because for any product to succeed it must first be adopted, then repeatedly used before it will help end users achieve their goals.

4. Answer Your Questions With Metrics

With your primary questions defined, you are ready to turn your attention to defining your product success metrics. For each question, you should define one or more metrics that are concrete and will answer your primary question. There is always a certain degree of art to defining appropriate metrics, but ideally metrics should be:

For example, to measure whether merchants were adopting our analytics product, we decided to look at the number of shops interacting with its main component—a navigation bar displaying key inventory metrics at the top of their Products page. This metric is easy to explain and to instrument, and clearly tells us how many merchants are actively noticing the experience. Furthermore, it’s suitably directional: if this number is lower than expected, it tells us that the analytics bar is not sufficiently discoverable or interesting to merchants.

Finally, in addition to your success metrics, you should define any tripwires, or “anti-success metrics”. These are metrics that indicate when your product is actually making things worse and interfering with your end user’s goals. While your product’s goal is to improve the user’s experience, it’s good to have a concrete way of measuring any adverse impacts so that you can observe and address any issues early on. In the case of our analytics product, our main tripwire metric was a change in the amount of time that merchants spend on their Products page. We didn’t want our experience to interfere with a merchants’ existing workflows, so any change in time spent on this page would alert us to any instances of this happening.

5. Turn Metrics Into A Measurement Plan

With your product success metrics now defined, you may think your work is done. However, a data scientist’s job doesn’t stop there. You also need to define how you’ll be measuring and interpreting these metrics. To this end, we find it helpful to organize our product success metrics into a measurement plan. A measurement plan is a table showing:

• What we are measuring
• When we are measuring it
• What we are comparing those measurements to
• What questions we are then equipped to answer

Sample measurement plan

As you can see from the above example, we divide our table into distinct time periods, which are further divided into distinct product subgoals that can be measured during that period. For each subgoal, we define one or more distinct metrics, which are all paired with comparison metrics and benchmarks.

There are several things to keep in mind when laying out your measurement plan and deciding how you’ll be measuring your metrics:

• Metrics in isolation aren’t informative: To be informative, metrics need something to be compared to, and an indication of what a significant comparison would be. These can be as simple as comparing product metrics before and after launch. What’s important is that you think about and define these comparisons up front, as you will need them in order to contextualize your metrics when you actually measure them.
• Comparisons without benchmarks aren’t interpretable: You need to know what a significant comparison between your success metric and its comparison metric would be. This is why you should also specify benchmarks for all your comparisons. These benchmarks answer the question “What does good look like?” for each metric. You don’t need to have complete confidence in all your benchmark numbers, but having at least some indication of the kind of comparison you’re looking for will allow you to more quickly interpret your metrics as you begin to measure them.
• Metrics may behave differently for different types of end users: Only looking at your metrics across all end users won’t always show the complete picture. This is why you should specify how your metrics can be segmented. For example, at Shopify we may segment our merchants by plan type, order volume or sales band.
• Metrics typically can’t all be measured within the same time period: Some metrics require months of data before they’ll see potential movement, while others will see movement right away. This is why you should divide your measurement plan into distinct time periods: short-term, mid-term, and long-term. We typically think of these as 30-day periods (0 to 30, 30 to 60, 60 to 90 days) after the product launches. Dividing up your measurement plan in this way ensures that you’re measuring the right things at the right time, and also gives your team a clear agenda for your post-launch analysis work.

Key takeaways

To quickly summarize, the most important takeaways are:

• Your product success metrics should be informed by your product’s goals, which should themselves be informed by your end user’s goals
• Break your product’s overall goal into distinct subgoals, and use your success metrics to measure whether you are achieving those subgoals
• Remember to define tripwires or “anti-success metrics”, alongside your success metrics
• Provide comparisons and benchmarks for all your success metrics, so you can define “What does good look like?”
• Indicate how you can segment your success metrics to capture any effects that may not be visible in aggregate
• Divide your success metrics into distinct time periods, according to when you can and should be measuring them

And that’s all there is to our framework! By following these steps you’ll be able to capture all the events you need in order to measure your success metrics.

Willie Costello is a Data Scientist on the Insights team, where he helps build products that empower merchants to be more data-driven in their decision-making. He lives in rural Ontario and spends his spare time kayaking, camping, and cross-country skiing. Connect with Willie on LinkedIn or Twitter.

If you’re passionate about solving complex problems at scale, and you’re eager to learn more about Shopify Data, we're hiring! Reach out to us or apply on our careers page.