从事实与指标到媒体机器学习：Netflix数据工程职能的演变

By Dao Mi, Pablo Delgado, Ryan Berti, Amanuel Kahsay, Obi-Ike Nwoke, Christopher Thrailkill, and Patricio Garza

由 Dao Mi, Pablo Delgado, Ryan Berti, Amanuel Kahsay, Obi-Ike Nwoke, Christopher Thrailkill, 和 Patricio Garza

At Netflix, data engineering has always been a critical function to enable the business’s ability to understand content, power recommendations, and drive business decisions. Traditionally, the function centered on building robust tables and pipelines to capture facts, derive metrics, and provide well modeled data products to their partners in analytics & data science functions. But as Netflix’s studio and content production scaled, so too have the challenges — and opportunities — of working with complex media data.

在Netflix，数据工程一直是使业务能够理解内容、推动推荐和驱动业务决策的关键职能。传统上，该职能集中于构建稳健的表和管道，以捕捉事实、推导指标，并向分析和数据科学职能的合作伙伴提供良好建模的数据产品。但随着Netflix的工作室和内容制作的扩展，处理复杂媒体数据的挑战和机遇也随之增加。

Today, we’re excited to share how our team is formalizing a new specialization of data engineering at Netflix: Media ML Data Engineering. This evolution is embodied in our latest collaboration with our platform teams, the Media Data Lake, which is designed to harness the full potential of media assets (video, audio, subtitles, scripts, and more) and enable the latest advances in machine learning, including latest transformer model architecture. As part of this initiative, we’re intentionally applying data engineering best practices — ensuring that our approach is both innovative and grounded in proven methodologies.

今天，我们很高兴分享我们的团队如何在 Netflix 正式化一种新的数据工程专业：媒体 ML 数据工程。这一演变体现在我们与平台团队的最新合作中，即 媒体数据湖，旨在充分利用媒体资产（视频、音频、字幕、脚本等）的潜力，并支持机器学习的最新进展，包括最新的变换器模型架构。作为这一倡议的一部分，我们有意应用数据工程最佳实践——确保我们的方法既创新又基于经过验证的方法论。

The Evolution: From Traditional Tables to Media Tables

演变：从传统表格到媒体表格

Traditional data engineering at Netflix focused on building structured tables for metrics, dashboards, and data science models. These tables were primarily structured text or numerical fields, ideal for business intelligence, analytics and statistical modeling.

Netflix的传统数据工程专注于构建用于指...