Resource Management with Apache YuniKorn™ for Apache Spark™ on AWS EKS at Pinterest

Resource Management with Apache YuniKorn™ for Apache Spark™ on AWS EKS at Pinterest

摘要

Monarch, Pinterest’s Batch Processing Platform, was initially designed to support Pinterest’s ever-growing number of Apache Spark and MapReduce workloads at scale. During Monarch’s inception in 2016, the most dominant batch processing technology around to build the platform was Apache Hadoop YARN. Now, eight years later, we have made the decision to move off of Apache Hadoop and onto our next generation Kubernetes (K8s) based platform.