为片段建立背景效果

Last September, Slack released Clips, allowing users to capture video, audio, and screen recordings in messages to help distributed teams connect and share their work. We’ve continued iterating on Clips since its release, adding thumbnail selection, background blur, and most recently, background image replacement.

去年9月，Slack发布了Clips，允许用户在消息中捕捉视频、音频和屏幕记录，以帮助分布式团队连接和分享他们的工作。自Clips发布以来，我们一直在对其进行迭代，增加了缩略图选择、背景模糊，以及最近的背景图片替换。

This blog post provides a deep dive into our implementation of background effects (background blur and background image replacement) for browsers and the desktop client. We’ve used a variety of web technologies, including WebGL and WebAssembly, to make background effects as performant as possible on our desktop platforms.

这篇博文深入介绍了我们对浏览器和桌面客户端的背景效果（背景模糊和背景图片替换）的实现。我们使用了各种网络技术，包括WebGL和WebAssembly，使背景效果在我们的桌面平台上尽可能地执行。

Pipeline introduction

管线介绍

Because of the nature of real-time video processing, we need to ensure that video frames are rendered with minimal latency and without interruption. To achieve this, we’re leveraging experimental technologies to offload the processing workload to a worker thread. This makes the rendering pipeline less prone to frame drops due to background activity, such as message processing, in the main thread.

由于实时视频处理的性质，我们需要确保视频帧以最小的延迟和不中断的方式进行渲染。为了实现这一目标，我们正在利用实验性技术将处理工作负载卸载到工作线程中。这使得渲染管道不容易因主线程中的后台活动（如消息处理）而出现丢帧现象。

Diagram of video processing pipeline

We read from webcam and screen capture video feeds using the Media Streams API. Individual video frame data can be read directly from the stream using the experimental Insertable Streams API, which exposes the media stream as a readable stream of video frames.

我们使用媒体流API从网络摄像头和屏幕捕捉的视频资料中读取。可以使用实验性的可插入流API直接从流中读取单个视频帧数据，它将媒体流暴露为可读的视频帧流。

Six stages of the processing pipeline

After reading data from the input streams, a render loop copies the video into a manipulatable source such as a canvas or ImageBitmap. We resize the image down to 256x144px, to make it suitable for the ML model we use for segmentation. Inference is performed by a WebAssembly module, to produce a segmentation alpha mask. The...