Challenges and Best Practices of FFmpeg/Gstreamer

1.

2. Challenges and Best Practices of FFmpeg/Gstreamer Xu Guangxin Zhao Juan Intel * Intel disclaims all express and implied warranties, including without limitation, the implied warranties of merchantability, fitness for a particular purpose, and noninfringement, as well as any warranty arising from course of performance, course of dealing, or usage in trade.

3. Out Line • FFmpeg/Gstreamer Introduction • Build Your Optimized Pipeline • DevOps with FFmpeg/Gstreamer • Embracing new technologies

4. FFmpeg/Gstreamer Introduction

5. What is FFmpeg is the leading multimedia framework, able to decode, encode, transcode, mux, demux, stream, filter and play pretty much anything that humans and machines have created. It supports the most obscure ancient formats up to the cutting edge. No matter if they were designed by some standards committee, the community or a corporation. It is also highly portable: FFmpeg compiles, runs, and passes our testing infrastructure FATE across Linux, Mac OS X, Microsoft Windows, the BSDs, Solaris, etc. under a wide variety of build environments, machine architectures, and configurations.

6. What is GStreamer provides •an API for multimedia applications •a plugin architecture •a pipeline architecture •a mechanism for media type handling/negotiation •a mechanism for synchronization •over 250 plug-ins providing more than 1000 elements •a set of tools

7. Typical GStreamer pipeline

8. FFmpeg/GStreamer Ecosystem • Adopted opensource software • x264,x265, av1d,av1e, SVTAV1… • Adopted by opensource software • • • • • • Chrome, Firefox(FFmpeg) Handbrake, OBS (FFmpeg) VLC,Mplayer (FFmpeg) OpenCV (FFmpeg, GStreamer) Open WebRTC Toolkit (FFmpeg,GStreamer) …

9. Build Your Optimized Pipeline

10. Software codec pipeline optimization • Compile stage optimization • Enable O3, and AVX • Using better compiler, eg ICC • Using dynamic library vs static library • Profile stage • • • • • Test video clips CPU utilization Hotspots list Memory Access analyze Memory Usage analyze • Hotspot optimization • Handwriting assembly code. X264 transcoding process

11. Hardware Acceleration • Step 1. Know your hardware capabilities • Step 2. Select best software package • Step 3. Choose FFmpeg and GStreamer plugins

12. Pipeline Level Optimization • Know you bottleneck • • • • CPU Different GPU engines Memory bandwidth Synchronization overhead • Latency profile • First frame to show out • The balance between hardware resource and Latency • Distributed transcoding

13. Take an example • Why accessing decode output is too slow？ – Integrated graphics card uses tiled memory to store decode output, there’s no cache for read. – Discrete graphics card has a dedicated memory, CPU need use PCI bus to read Cache less Integrated GPU Main Memory GPU CPU GPU Memory

14. Why accessing decode output is too slow • Solutions: • Do not read the GPU memory. • Use OpenCL/OpenGL/Vulkan to access hardware decoder output. • eg: background aware subtitles • Copy memory to a linear memory using vpp. • hw decoder + sw encoder • Use AVX2/AVX512 to read the cache less memory.

15. Best known method to build a high performance HW pipeline • Pay attention to resource bottle neck. o Shift workload to other threads, if one core is full. o Shift vpp workload to EU, if fix function is fully loaded. • Do not wait for GPU job done. GPU frequency Render/EU used by 3D, OpenCL and VPP Blitter to copy data from/to GPU Decoder and encoder hw usage Fix function Video Post Process hw usage

16. DevOps with FFmpeg/Gstreamer

17. Upstream & Downstream • Upstream • https://git.ffmpeg.org/gitweb/ffmpeg.git • https://gitlab.freedesktop.org/gstreamer • Downstream • Any release or commit you forked to build your service

18. Why upstream • If you use the upstream, you will get • More bug fix • Optimized performance • More features • If you maintain a local fork, you will • not get upstream fix, back port fixes may not possible. • queue too many local patches, they are hard to upstream.

19. Pain points for downstream FFmpeg/GStreamer development • There are 50+ commits every week for Ffmpeg/GStreamer, how to make sure it’s no regression for our components? • We maintained 5 generations of hardware. How to make sure they are well tested. • Community review is slow, how to make sure our customer get updated hw features. • How to make sure the pending patches are well tested and reduce the rebase efforts.

20. Open Source Development Process

21. CI system & Cartwheel

22. A line of code’s real life Upstream review Urgent or customer specific features Internal review coding Upstream mirror Upstream Upstream Release As submodule Downstream (Cartwheel) Product Delivery CI covered

23. Embracing New Technologies

24. The value of 5G to media framework LAN, SDI Companion PC (GPU) RTMP Stitching, Encoding, Upstreaming Media Server CDN Transcoding Encoding, Upstreaming Caching, Forwarding, Distributing Cloud HTTP Viewport Pulling, Decoding, Rendering, [Conventional CDN/HTTP Based Solution] RTP, SRT Distribution Edge Ingestion Edge Stitching, Encoding, Forwarding Cloud + 5G Edge [5G Edge Based Solution] Viewport Pushing Distributing RTP/RTCP

25. The value of AI to Media • Video analysis • Pixel analysis • Visual quality enhancement. • • • • • • AI denoise Super resolution Frame interpolation ROI Beautify, smooth skin Per tile encode optimization

26. FFmpeg FFmpeg Deep Neuro Network DNN - Support AI through OpenVino Tensorflow and Native mode libavcodec - Video Enhancement: Derain, Super Resolution libavdevice - Video Analysis: Object Detection libavformat libavfilter/ vf_sr.c vf_derain.c vf_dnn_processing.c dnn_filter_common.h/c dnn_interface.h vf_dnn_detect.c vf_dnn_classify.c (in code review) DNN interface libswscale libavfilter/dnn/ libswresam ple TensorFlow backend Native backend OpenVINO backend libavutil TensorFlow C library (libtensorflow.so) OpenVINO C library (libinference_engine_c_api.so) computing device such as CPU, GPU, etc.

27.