Estimating Incremental Lift in Customer Value (Delta CV) using Synthetic Control
How we measure the impact of user actions and product adoptions at PayPal
In today’s competitive digital landscape, understanding user interactions with your products is essential for driving revenue and building lasting customer relationships. At PayPal, our Data Science teams use causal inference to evaluate the impact of key customer actions, such as adopting a new product or adding a credit card to their wallet, on engagement (measured by Transactions per Account, or TPA), revenue, and margin to help make data-driven strategic decisions.
The direct profit from a product adoption or a user action on the app could be $0 if viewed in isolation. However, that does not necessarily mean that these events are not driving engagement and monetization across the PayPal ecosystem; they can change the user engagement with other PayPal products in such a way that the user starts generating more profit.
To measure the overall impact of user actions or product adoption, we introduced Delta CV (delta in Customer Value), and we defined it as a customer’s incremental profit margin in the first year after adoption of a new product or completing an action. For example, if the average Delta CV for adoption of Crypto is $20, we expect customers who adopt Crypto for the first time to bring an additional $20 in margin on average in the next 12 months after the adoption. We define Delta Revenue (or TPA) in the same manner except we calculate the incremental lift in revenue (or TPA) instead of profit margin.
The concept of Delta CV is very different from customer life-time value (CLV) which estimates the total profit generated by a customer over the course of their relationship with PayPal. Delta CV gives us a wholistic view on how new adoptions affect the engagement and value of an existing PayPal user.
Adoption of a new product or completing an action can increase the customer’s value in a few ways:
- The product itself may be a profit generating (Direct effect). For example, paying with PayPal when checking out at a merchant’s online website drives direct margin for PayPal.
- Actions do not necessarily lead to profit directly, but they facilitate the usage of other products that generate profit (Halo effect). For instance, a customer adding a credit card to PayPal’s digital wallet can reduce future frictions or result in a higher conversion rate for that user paying with PayPal in the future.
- Adoption of a product can increase the user engagement with PayPal, leading to adoption or increased usage of other products that generate profit (Halo effect). With Delta CV, we measure the cumulative lift in customer value considering all PayPal transactions of a user.
Today we estimate Delta CV for 40+ products (or actions) in multiple regions. Having Delta CV for different products helps us in a variety of areas such as strategic decision making, calculating the return on investment (ROI) of campaigns, opportunity sizing for new campaign efforts, in-app product ranking and placement, making trade-off between resource allocations, making ramp decisions on product launches, and so on.
Methodology
We measure Delta CV using causal inference and synthetic control. For each product, our treatment group are the adopters of the product for the first time in each quarter. To create a synthetic control group, we focus on users who never adopted the product of interest. Then we find matches for the treatment users inside the control group based on a set of transactional features calculated over 12month pre-adoption. Since we are building a synthetic control and our target variable is CV, we should always match on CV in pre-period. The remaining of our matching features are important covariates of CV. They capture user characteristics and are our best predictors of users’ CV response to external and internal changes.
The synthetic control group acts as a counterfactual, meaning that we assume in the absence of an intervention, control and treatment group would change similarly over time. Therefore, if we introduce a change to the treatment group but not to the control group, the difference in the profit margin of the two groups measures the impact of the intervention.
We select the synthetic control group by matching on our group of features using KNN (K nearest neighbors) algorithm. Every user in treatment will have a synthetic control that is the average of up to 10 users from control. We define a threshold for the Euclidean distance between the treatment and control units, and we remove the matches that exceed this threshold to ensure a high quality of matching. The validity of synthetic control group selection can be checked by a bias analysis.
Interpretation of Delta CV and Caveats
- The incremental lift by adoption of a new product or completing an action can be a highly skewed metric. Some users may bring in significantly more value than others after adoption of new products. Median delta CV is more in line with the expected incremental CV from a typical user, while mean delta CV reflects the financial lift that is expected at scale on average per user.
- Delta CV estimations are subject to variance and bias. It is important to consider the accuracy of estimations when making trade-off decisions based on the model’s output.
- Delta CV model estimates the historical impact of adoptions, so there is always a lag. We generally measure the impact over 12 months after adoption, so there will be a 12-month lag between our estimation and the quarter in which adoption occurred. Sometimes we use delta CV to understand past behavior, and other times we use it as our best estimation for what happens in the future. We can reduce the measurement period from 12 months to say three months for quick readouts, but we know that the early lift can be skewed from novelty or immediate use cases of customers and the CV gap between the treatment and control groups reduces over time.
- In an ideal causal inference scenario, the treatment group undergoes an intervention or change, while the control group does not. However, in our setup, both treatment and control groups may adopt other products during the post-period. It is tempting to assume that other adoptions in the treatment and control groups occur randomly and that their impacts cancel each other out. However, our data shows that in the treatment group, there is a phenomenon of chain adoptions, where certain products are adopted more frequently together than in the control group. Not every user in the treatment or control group adopts multiple products but certain product adoptions are more prevalent in the treatment group. Therefore, Delta CV measures the cumulative effect of a product adoption along with its frequent preceding and subsequent adoptions at scale within the same quarter.
Note that excluding users who adopt other products during adoption period or in the post-period results in very specific, unusual treatment and control groups who do not adopt any product or complete any action within 12 months and do not represent our user base. We have no interest in limiting the Delta CV estimation scope to these specific users. - Delta CV is not an additive metric meaning that adoptions of two products during the same period does not result in a total Delta CV that equals the sum of Delta CV of both. As mentioned earlier, Delta CV is not an immaculate metric; it captures the lift due to other frequent product adoptions by a portion of the treatment users during the same period as well. But more importantly, user engagement does not linearly increase with each new product adoption; therefore, Delta CV cannot be treated as an additive metric.
- Sometimes we cannot find high quality matches for the treatment. The control condition of “never adopted the product” is very limiting for some of our products, especially in markets where we have many high-engaged users. This results in a small synthetic control group size and low quality of matching. We flag the reliability of Delta CV when the average difference between CV of treatment and synthetic control in pre period is larger than $1. This is an important piece of information that is provided along with Delta CV for every product.
End Note
PayPal products are rapidly evolving to provide the best value and experience to customers. While checkout was PayPal’s first product, we now offer an extensive variety of financial products including peer-to-peer payments, debit card, credit card, rewarding shopping experiences with cashback, and much more, all within the PayPal App. Delta CV has been an integral part of strategic decision making in PayPal. Adding new products to the scope of Delta CV, as well as continuously adjusting the matching methodology, is an ongoing effort. Reducing the estimation biases by improving the selection of matching features is another area for improvement.
