Estimating Potential Audience Size of an Ad at Pinterest
Chanheum (Sean) Cho | ML Engineer, Ads Intelligence; Ruixin Qiang |ML Engineer, Ads Intelligence; Keshava Subramanya |Engineering Manager, Ads Intelligence
Introduction
Understanding the size of the potential audience of an ad is an important consideration for an advertiser. It enables advertisers to estimate the total population who might be interested in the products or services they advertise and plan their budgets ahead of time. The Ads Intelligence team at Pinterest provides a service called Potential Audience Size in the Ads Manager, so the advertisers can understand their target audience size while they configure their ad groups. The service updates the estimate in real time as the audience targeting is updated.
In this article, we’ll provide an overview of how the Potential Audience Size feature works, then take a detailed look at each piece. Now let’s dive in!
How it’s Made
In a high level overview, the Potential Audience Size is composed of three different parts: sampling ad requests to build an index, aggregating the number of ad requests in the index that match the targeting specs in Ads Manager, and extrapolating the estimate to get a potential monthly audience size. These steps are comparable to reversing the ads serving process as shown in Figure 2. In ads serving, multiple ad groups are taken as inputs per ad request to retrieve winning ad groups whereas in Audience Sizing, user attributes in ad requests, such as their demographics or interests, are taken as inputs to retrieve the number of matching ad requests.
Like the ads serving infrastructure, Potential Audience Sizing leverages an internal search engine known as Muse.
Creating Targeted User Index
In order to search users who have matching targeting criteria such as demographics or interests, we need to build a search index of users and their information. This allows us to quickly retrieve users with a particular targeting specification so we can provide the real-time updates on the Ads Manager UI.
Creating the index starts with pulling historical ad request logs and building intermediary objects called ad queries that hold the user context in the request. This allows us to map between a particular user and their targeting information by parsing an ad query. This information is used to compute the potential audience size and is not displayed externally.
The mappable data stored in ad queries are translated into documents that Muse can understand. These documents consist of keys such as user ids, demographics, and interests, as shown in Figure 3. These documents, when converted into a search index, enable us to query any of the fields to retrieve matching user ids. The documents are fed to Muse’s index building pipeline to power downstream Planning service for Potential Audience Size, which we will cover in the next section.
The inverted index is built fresh daily with the new data we gathered the day before. This is because some of the user traits in the targeting specifications, such as the user interests, are from Pinterest’s machine learning model outputs. When a model update changes its prediction, the search retrieval would be impacted, which leads to a change in our estimation. For example, if a model’s prediction of a user’s interest changes from “camping” to “camping” and “travel,” we now have more users associated with “travel” than before. Updating the index daily allows us to manage such system-wise updates as well as provide the advertisers the most up-to-date estimation.
Planning Service — Request
The second component of Potential Audience Size is the Planning service. This service is responsible for translating target specifications in the Ads Manager, as shown in Figure 4, to a request to the backend, so the Potential Audience Size is computed and returned appropriately.
Underneath the UI, Ad Manager is sending planning requests to the backend service. The targeting specifications in planning requests are translated into a query Muse understands, named squery, where each target category is joined accordingly. As an example, multiple age groups are chosen in logical disjunction where age groups and languages are joined in conjunction, as shown in Figure 5.
On top of the targeting specifications, an advertiser can create a shortlist of users called Audience list. Explaining Audience lists in depth is out of scope of this article, so we won’t explain it much further than that they can be thought of as an advertiser defined user list. The important consideration here is that there are a lot of them, as each advertiser creates their own lists and each Audience list contains many user ids.
In Potential Audience Size, we utilize a bloom filter for each Audience list to keep track of the user ids associated with the Audience list as well as their total user counts. This allows us to effectively manage the memory and storage usage.
The Planning service is a multi-layered service where translating the advertiser specified targeting specifications to squery is only one part. The Planning service consists of four different layers, where each layer has a different logic for passing the request to the lower layer and the response to the upper layer. The layers are shown in Figure 6.
In the context of sending a request from the Ads Manager to Muse, the rewrite and retrieval layers perform notable tasks. The rewrite layer modifies keywords and interests in the planning request such as stemming keywords, fetching derived interests, and converting interests to their canonical form according to an internal reference. At last, the retrieval layer translates the updated request into squery and submits the search request to the Muse cluster.
Planning Service — Response
The other side of the Planning service and the last piece in Potential Audience Size is the handling of the response from the Muse cluster. In short, this is going from Muse to the sanitize layer as shown in Figure 6. The updated response from the sanitize layer finally gets to be shown in the Ads Manager.
The retrieval layer receives the response from the Muse cluster, which includes the aggregated user number. If one or more Audience lists were configured in the targeting, the retrieval layer applies an adjustment based on the false positive rate of the Audience list’s bloom filter, as shown in Figure 7.
Lower and upper bounds are estimated from the estimated user count. This range helps us express our confidence in our estimates more appropriately.
Once the retrieval layer returns the estimated audience size range, further adjustments are performed in the forecast layer and the sanitize layer. The forecast layer extrapolates the daily user count range to monthly or any given date range if the advertiser specified the date range in the Ads Manager. The sanitize layer, as the last step, applies additional business logic to further refine our estimates.
Once the estimate is complete, the sanitize layer sends the final user size estimate back to Ads Manager, and Ads Manager displays Potential Audience Size to the advertiser.
Outro
In this article, we went over how we are providing Potential Audience Size estimates for the advertisers so they can plan ahead a successful ad campaign. It included how we build a search index to find users based on their traits and how we create Planning service where planning requests pass through multiple layers of modifications to retrieve users that match the targeting specifications. We also looked at how the response from Muse is adjusted to extrapolate daily user estimates to monthly as well as taking account of the usage of Audience lists.
The Ads Intelligence team at Pinterest is always looking to build features that help the advertisers reach their goal efficiently and effortlessly. The Potential Audience Size service is just one of such features. If you’re interested in other such features we built, take a look at Advertiser Recommendation Systems or Campaign Budgets. If you’re excited to help businesses thrive at Pinterest by providing them super powers, come and join our team!
Acknowledgements:
We would like to thank the following cross-functional team (not an exhaustive list) for contributions -
Ads Intelligence team: Flavio Bossolan, Chanheum (Sean) Cho, Tianyuan Cui, Yu Hao, Danilo Nunes Dos Santos, Perrye Ogunwole, Ruixin Qiang, Yeming Shi, Melanie Stam, Keshava Subramanya, Xue Yang
Advertiser Solutions: Kelvin Jiang, Mao Ye
Ads Targeting team: Jacob Gao, Paul Nunez, Scott Zou
Advertiser Experience team: Argun Alparslan, Dani Gnibus, Robbie Holmes, Frannie Huang, Michael Lam, Leo Lam, Priyanka Patil, Meera Srinivasan
To learn more about engineering at Pinterest, check out the rest of our Engineering Blog, and visit our Pinterest Labs site. To view and apply to open opportunities, visit our Careers page
