The Field Guide to Non-Engagement Signals

Leif Sigerson | Sr. Data Scientist; Wendy Matheny | Sr. Lead Public Policy Manager

User engagement is a critical signal used by Pinterest and other online platforms to determine which content to show users. However, it is widely known that optimizing purely for user engagement can surface content that is low-quality (e.g., “clickbait”), or even harmful. Our CEO, Bill Ready, explains that if we’re not careful, content ranking can surface the “car crash we can’t look away from”. On the other hand, “if you ask somebody after they saw the crash, ‘you want to see another one?’, the vast majority of people will say ‘Goodness no’”.

In this blog, we will discuss Non-Engagement Signals, a critical component to ensure we don’t optimize for “the car crash we can’t look away from.” We’ll look at how companies can leverage the new “Field Guide to Non-Engagement Signals”, published by authors from Pinterest and other institutions, to benefit both their users and their business. Ultimately, we hope more companies will adopt Non-Engagement Signals and work together collaboratively to build a more inspired Internet.

What are Non-Engagement Signals and Why Are They Important?

Non-Engagement Signals usually come from two sources:

In-app surveys, where users can directly tell us about the platform (see the example survey from Pinterest below)
Independent assessments of content quality, usually generated by manual labeling

Beyond providing an important balance to engagement signals in our content ranking, Non-Engagement Signals help Pinterest put our values into action. For example, our industry-leading inclusive product work has relied heavily on Non-Engagement Signals. When a user tells us the body type, hair pattern, or skin tone they want to prioritize in their feed, Pinterest can adjust what they see first.

Figure 1: A real survey on Pinterest

Why Make a Field Guide to Non-Engagement Signals?

As the founding signatory of the Inspired Internet Pledge, Pinterest is committed to “sharing best practices, key research findings, and creative solutions across the industry to make the internet a healthier place for everyone.” Our team is approaching this commitment from a variety of angles — including product, policy, strategic giving, and thought leadership.

From an engineering and product angle, we felt that Non-Engagement Signals were an excellent place to start sharing best practices across the industry. Although Non-Engagement Signals can help protect users and provide value to the business, platforms may struggle to use them in content ranking for a couple reasons:

Limited scale: While platforms can log billions of data points about user engagement, Non-Engagement Signals typically require manually labeling content (either by users in surveys or by content experts paid by companies).
Rewards take time to pay off: Although Non-Engagement Signals have been found to benefit long-term user retention, in the short-term they can actually inhibit user engagement, e.g. by removing clickbait.

These barriers mean that companies, especially small and medium-sized companies without resources to invest in long-running experiments, might be missing out on opportunities to benefit their users and their business by using Non-Engagement Signals in their content ranking. Thus, we worked with colleagues across the industry to put together a definitive, industry-leading “Field Guide” to Non-Engagement Signals.

How We Made the Field Guide

Pinterest is proud to have partnered with experts from UC Berkeley and the Integrity Institute to organize the Field Guide. This interdisciplinary coalition started by collecting all the research we could find that has been published about Non-Engagement Signals. This included high quality work by both academia and industry. We then organized this research into a set of “propositions” (AKA ‘wins and learns’) about Non-Engagement Signals.

Once the research was in order, we ran a daylong workshop with experts from seven social media platforms with hands-on experience using Non-Engagement Signals. In this workshop, we discussed the propositions and pressure-tested their applicability across platforms.

The 19 propositions that held up under our pressure test became the Field Guide, published by 11 authors across four institutions.

Why Call it a Field Guide?

We call this resource a Field Guide for two primary reasons:

It provides information, not prescription

The goal is simply to help people at platforms make informed decisions about how to use Non-Engagement Signals, not to tell them what to do.

2. It’s intended as a reference

There’s no single thesis or simple takeaway from the Field Guide. Instead, it has many different applications (more on that below!).

And as an added bonus…

It’s more authoritative than anything else that’s been published.

The Field Guide is based on more than 140 public resources plus hands-on expertise from people at 7 social media platforms. As one of our authors put it, “I wish I’d been able to read this years ago.”

How the Field Guide can be Applied Directly to Product Decisions

Because the Field Guide is rooted in practical industry knowledge, it has a number of actionable product applications. Here are a few applications we’re particularly excited about:

Application 1: How to Tune for Emotional Well-Being

Principle 1 of the Inspired Internet Pledge calls for companies to “Tune for Emotional well-being” — understanding which actions and content correlate with users’ well-being outcomes on a platform. At Pinterest we think the Internet would be a better place if all platforms were able to do this at scale in their content ranking, but that is easier said than done. Luckily, the Field Guide provides some high-quality, actionable guidance on what to do, and what not to do, when tuning for well-being.

Platforms can zero-in on specific types of content that help or harm user well-being. In proposition 6.1 of the Field Guide, we note that “Ranking changes often significantly affect self-reports of content exposure.” With some careful research into which types of content to promote to support users’ well-being, this gives platforms a powerful opportunity to benefit their users.

Conversely, as each person’s well-being depends on many factors in their life, it’s probably not a good idea to try to optimize users’ general well-being, but rather zero-in on specific contributors to well-being. In Proposition 6.3 of the Field Guide, we note: “Broad measures such as general well-being, life satisfaction, polarization, or attitudes toward the company usually do not show statistically significant changes due to ranking even with relatively large and long-running experiments.” Optimizing for general well-being is an appealing idea (especially for those of us who get excited about finding “just the right metric”), but it simply doesn’t work.

Application 2: Using Generative AI to Scale Content Quality Signals

The Field Guide makes it clear that content quality signals are beneficial for both users and the business:

They provide a critical complement to engagement by identifying engaging, but low-quality content. (proposition 4.2)
They can improve retention when used in content ranking (proposition 4.1)

While content quality signals can be extremely helpful, they can also be expensive and hard to scale, especially in cases where platforms need humans to manually label content. The experts in our workshop agreed that Generative AI may help address issues of cost and scale, though it’s important to do this carefully and thoughtfully.

As noted in the Field Guide, “There was consensus that generative AI is likely to provide a cheaper, faster option, which may allow more comprehensive content monitoring. Though participants agreed on the potential impact of GenAI in this area, there were significant open questions on how this could be done, such as the need to keep human involvement in individual content moderation decisions.”

We think the use of GenAI to scale content quality signals could be a powerful opportunity for platforms to build on existing (human) talent and resources to benefit both their users and their businesses. But, it’s critical that this is done intentionally, anchored in human judgment and values.

Application 3: Improving User Retention

The Field Guide focuses heavily on long-term user retention because:

We believe user retention is the metric that’s most important to most platforms (as MAU is reported in every earnings call).
It is especially hard for platforms to understand what causes user retention; definitive conclusions about user retention generally require long-running experiments or very robust analyses.

We hope the Field Guide can give platforms some insights into what causes retention and help them pursue this further themselves.

In the Field Guide, we affirm that ranking by user engagement does indeed benefit retention (proposition 3.1): “Ranking by predicted engagement causes significantly higher time-spent and retention compared to chronological ranking.” More specifically, it appears that more active user engagement (e.g., Repins) is more valuable for user retention (Proposition 3.3).

However, Non-Engagement Signals can provide additional benefits to user retention:

“A positive ranking-weight on quality metrics often significantly increases long-term retention” (proposition 4.1).
“There is evidence that using item-level survey responses in ranking helps retention” (proposition 5.4).

Given that quality metrics and item-level survey responses can be more challenging to scale, we think these two signals may be an undiscovered opportunity for many platforms.

I Work at a Content-Ranking Platform — How Can I Use the Field Guide?

We wrote the Field Guide to help platforms use Non-Engagement Signals, so we hope it’s useful for you!

The propositions in the Field Guide are based on a lot of evidence, but they may not apply in every case. If the field guide contradicts a high-quality internal analysis, disregard the field guide! We suggest relying on the field guide in areas where you don’t have high-quality analyses — where you currently have to rely on intuition, anecdotes, or more limited analyses.

For example, you could fill out this table, noting where the Field Guide adds new information for your platform:

Conclusion

As you can probably tell, we’re passionate about Non-Engagement Signals. We rely on them to benefit our users and our business, and we think the internet would be a better place if all content ranking platforms could more easily apply them. That’s why we supported this Field Guide, and that’s why we’re sharing this information here.

We hope the Field Guide is helpful for you — please reach out if it is (or even if it isn’t!)

This work wouldn’t have been possible without the collaboration and support of many folks, at Pinterest and elsewhere, including:

Our co-organizers: Tom Cunningham, Sana Pandey, and Jonathan Stray
Our co-authors: Jeff Allen, Bonnie Barrilleaux, Ravi Iyer, Smitha Milli, Mohit Kothari, and Behnam Rezaei.
Our supporters: Aidan Crook, Kathy Gu, Daron Sharps and Crystal Espinosa.

And lastly, if you’re interested in joining in on this work, let us know and consider signing the Inspired Internet Pledge.

To learn more about engineering at Pinterest, check out the rest of our Engineering Blog and visit our Pinterest Labs site. To explore and apply to open roles, visit our Careers page.