Efficient Resource Management at Pinterest’s Batch Processing Platform

Efficient Resource Management at Pinterest’ s Batch Processing Platform

摘要

Pinterest’s Batch Processing Platform, Monarch, runs most of the batch processing workflows of the company. At the scale shown in Table 1, it is important to manage the platform resources to provide quality of service (QoS) while achieving cost efficiency. This article shares how we do that and future work.