How Pinterest Fights Spam Using Machine Learning
Vishwakarma Singh | Trust and Safety Machine Learning Lead
Hundreds of millions of people visit Pinterest each month to discover ideas among billions of Pins. Inspiration is a high bar and so we must be vigilant in ensuring Pinners don’t see spam, harmful content, or misinformation. To enforce our community policies, we use machine learning to build automated systems that detect and act against spam.
Our anti-spam system consists of both reactive and proactive components to counter abusers. Our proactive system consists of sophisticated machine learning models. The reactive system includes both rules executed in a real-time rules engine and lightweight machine learning models. We iterate on these models by adding new data and exploring new technical breakthroughs to improve performance over time.
One tactic malicious actors use is misusing a Pin’s image and linking to a malicious external website. In these cases, our models detect spam vectors — like Pin links — as well as users engaging in spammy behaviors. We limit the distribution of Pins with spam links and take action against those identified to be engaging in spammy behavior. We then perform a manual review to limit false positives and notify users.
Machine Learning Models
Spam Domain Model
We identify spam links using a Deep Neural Network classifier (shown in Figure 1). To maximize impact, our model learns to classify a domain as spam rather than a link. We apply the same enforcement to all Pins with links belonging to the same domain. This model is trained on manually-labeled domains to achieve a higher recall and lower false positive rate. We use features created from links, web page text and media, user-domain interactions, and user behavior as inputs. For each domain, we sample links and web pages to create features. We split links into semantic tokens and use only frequent tokens as features. We analyze outlying patterns in user actions over time to create behavioral features. This model is periodically batch-inferred at scale by a PySpark job using Tensorflow, Spark SQL, and a UDF.
Spam User Model
Identifying users engaging in spam activities is the ultimate solution for fighting spam, but it is hard to achieve. And so, we leverage supervised and unsupervised models to build an effective spam user identification system.
Classification Model
Our spam user classification model is a Deep Neural Network (shown in Figure 2) and is part of our proactive system. It’s trained using synthetically-labeled data generated with minimal human supervision to ensure quality. We use features created from user attributes and their past behaviors as inputs. We also use user-domain interaction, summarized as a domain scores distribution, for each user where domain scores are reused from the spam domain model, as input. This model is periodically batch-inferred to score millions of Pinners by a PySpark job using Tensorflow, Spark SQL, and a UDF.
Clustering
We have developed lightweight clustering models for the early detection of suspicious users and bots. This technique also addresses gaps in our classification models (which are unaware of emerging patterns unless re-trained with fresh labeled data). We cluster users on attributes that can isolate suspicious groups with high accuracy. Experts identify these attributes by exploring the behavior of suspicious users and their use of resources for creating spammy content. We implement this model using PySpark and SparkSQL, and execute it daily.
Spam User-Domain Model
Interactions of users with domains are explicitly captured by a heterogeneous bipartite graph as shown in Figure 3. We represent users and domains as nodes in the graph and create an edge between a user and a domain if the user has created or saved a Pin with the domain’s link. This graph facilitates the simultaneous identification of spam users and domains using semi-supervised learning. We use a small set of labeled users and domains to run a label propagation algorithm and learn scores for the unlabeled users and domains. We implement this iterative algorithm in Spark and run it periodically.
Measurement
We measure spam prevalence by computing the number of Pin impressions that either have spam links or were created by users engaging in spammy activities. We periodically sample and manually review both impressed Pins and users. We scaled our measurement by sampling and reviewing from highly-impressed head domains, and then extended the coverage to tail domains. These samples are used for measuring overall spam prevalence as well as training our machine learning models.
We strive to protect Pinners’ experiences by acting against malicious users and spam content with our array of machine learning models. We’ll continue investing in evolving our community guidelines and technology to address inevitable threats.
Acknowledgments
Thanks to Yuanfang Song, Omkar Panhalkar, Rundong Liu, Qinglong Zeng, Attila Dobi, Abhijit Mahabal, Alok Singhal, Maisy Samuelson, and the rest of the Trust and Safety team for their contributions to developing machine learning models for spam. Thanks to Harry Shamansky for helping with the publication of the blog post.
To learn more about Engineering at Pinterest, check out our Engineering Blog, and visit our Pinterest Labs site. To view and apply to open opportunities, visit our Careers page.
