美团内容智能分发的算法实践

如果无法正常显示，请先停止浏览器的去广告插件。

1. 美团技术沙龙080期：美团内容智能分发的算法实践 Aligning Vision and Language on Temporal Dimension: Cross-modal Video Content Retrieval 视频-文本跨模态时序对齐与内容检索徐行研究员电子科技大学未来媒体研究中心&计算机科学与工程学院 2023年12月23日 1

2. Introduction  Widely spread of massive untrimmed videos in our daily life Untrimmed videos are usually redundant, but contain rich and informative fragments.  Using natural language text to retrieve key information from untrimmed videos is a promising solution.  Videos on Internet Mobile Video Apps Video Surveillance 2

3. Introduction  Applications of Cross-Modal Video Content Retrieval  Searching related video content with given text query on single long-term video or video galleries .  Retrieved video content can be entire video file or partial video clips . (a) Cross-modal Video File Retrieval (c) Video Content Retrieval in Video Gallery (b) Video Content Retrieval in Single Video 3

4. Introduction  Our joint work with Meituan on Cross-modal Video Content Retrieval  Video Content Retrieval in Single Video • SDN: Semantic Decoupling Network for Temporal Language Grounding. IEEE Transactions on Neural Networks and Learning Systems (TNNLS), 2022. (中科院JCR 1区） • GTLR: Graph-based Transformer with Language Reconstruction for Video Paragraph Grounding, IEEE International Conference on Multimedia Expo, 2022 (最佳学生论文奖) • Semi-supervised Video Paragraph Grounding with Contrastive Encoder, IEEE International Conference on Computer Vision and Pattern Recognition, CVPR, 2022 (CCF A) • Faster Video Moment Retrieval with Point-Level Supervision. ACM International Conference on Multimedia (ACM MM), 2023. (CCF A)  Video Content Retrieval in Video Gallery  Joint Searching and Grounding: Multi-Granularity Video Content Retrieval. ACM International Conference on Multimedia (ACM MM), 2023. (CCF A) Code available: https://github.com/CFM-MSG 5

5. Introduction  Previous Methods  Proposal-based Methods: two-stage paradigm that generates candidate moments then conducts cross-modal alignments.  Proposal-free Methods: one-stage paradigm that predicts latent target moments directly. Proposal-based method: CTRL [Gao et. al. ICCV 2017] Proposal-free method: LGI [Mun et. al. CVPR 2020] [1] Gao, Jiyang and Sun, Chen and Yang, Zhenheng and Nevatia, Ram. "Tall: Temporal activity localization via language query." ICCV. 2017. [2] Mun, Jonghwan, Minsu Cho, and Bohyung Han. "Local-global video-text interactions for temporal grounding." CVPR. 2020. 6

6. Motivation  Observations  Proposal-based Methods: Achieving high recall rate with redundant candidate moments but suffering from the constant-scale proposals.  Proposal-free Methods: Obtaining better flexibility and efficiency on temporal localization but showing more weakness on recall rate. (a) Proposal-based method (c) Semantic Decoupling Network (Ours) (b) Proposal-free method If there exists a framework that inherits their merits and overcomes the defects? 7

7. Our Proposed Method  Semantic Decoupling  Assumption: the proposal-based method performs well because the semantics of video segments can be determined by corresponding available temporal scales.  By observing the proportion of valid segments in each untrimmed video, we also find the untrimmed video contains massive irrelevant content. “Person” “Door” “Person” “Door” “Walking” “Person walks into the door to fetch his cloth” (a) Semantic Completeness (b) Statistical results on proportion of valid content  We argue that the more temporal information will turn static objectives into action and, eventually, the complete events. The occurrence frequency also decreases according to the semantic level. [3] Xun Jiang, Xing Xu*, Jingran Zhang, Fumin Shen, Zuo Cao and Heng Tao Shen, "SDN: Semantic Decoupling Network for Temporal Language 8 Grounding“, TNNLS, 2022

8. Our Proposed Method  Contributions  We propose a novel framework, Semantic Decoupling Network (SDN), which inherits the benefits of proposal-based and proposal-free methods and overcomes their defects.  It consists of three key components: (1) Semantic Decoupling Module (SDM), (2) Context Modeling Block (CMB), (3) Semantic Cross-level Aggregating Module (SCAM). 9

9. Experiments  Datasets  ActivityNet-Caption        More than 20k videos from Youtube Open domain More than 15k vocabularies Long videos with 117s averaging Complex semantics Sufficient contexts Charades-STA       More than 6k videos from Youtube Indoors domain About 1300 vocabularies Short videos with 30s averaging Simple semantics Insufficient contexts 10

10. Experiments  Overall Comparisons  We evaluate our methods on three widely used datasets, i.e., Charades-STA, ActivityNet- Caption, and TACoS.  The experimental results demonstrate our method outperforms both proposal-based and proposal-free SOTA methods at the time. Comparisons on TACoS Comparisons on Charades-STA Comparisons on ActivityNet-Caption 11

11. Experiments  Visualization of Retrieval Results  We visualize several event retrieval cases on short videos and long videos both, and make a comparison with the SOTA methods at the time.  Our methods reveals obvious superiority on localizing events precisely (proposal-free) and understanding global contextual information (proposal-based). 4 test cases from the Charades-STA (subfigure a, b) and TACoS (subfigure c, d) datasets. 12

12. Rethinking  Video Sentence Grounding (VSG)  Almost all the previous methods study the task that aims to localize an event from an untrimmed video with a sentence query in natural language. Untrimmed Video (Multiple Events) … Sentence Query (Single Event) Two young girls are standing in the kitchen preparing to cook. … Single-Multi Localization Target Moment [CTRL Gao et al, 2017] [MCN Hendricks et al, 2017] [LGI Mun et al, 2020] … [0s, 15.67s] 13

13. Rethinking  Video Paragraph Grounding  A new task that targets to localize multiple moments from an untrimmed video with a given natural language paragraph query (multiple sentence queries). Untrimmed Video (Multiple Events) … … Paragraph Query (Multiple Events) After, the two continue to stir a the They then open box in are an of contents brownies…get Two young girls egg out in of the the kitchen fridge. standing preparing to cook. Target Moments [0s, 15.67s] Multi-Multi Localization (Not fully explored) [15.67s, 28.73s] [50.12s, 57.59s] 14

14. Rethinking  Multi-Multi Localization: Chance and Challenge  The paragraph queries bring more context information, but also increase the difficulty in multimodal representation and alignment 15

15. Our Proposed Method  Graph-based Transformer with Language Reconstruction (GTLR) A. Multimodal Graph Encoder that conducts message passing among fine-grained features B. Event-wise Decoder that explores contexts and achieve multi-multi localization C. Language Reconstructor that aligns events with their corresponding sentences C A od En Position Embedding C Concatenate Clip-level Features Sentence.3: Sentence.2: Word-level Features Sentence.1: A woman is wrapping a box. Sentence-level Features B [4] Xun Jiang, Xing Xu*, Jingran Zhang, Fumin Shen, Zuo Cao and Xunliang Cai, “GTLR: Graph-Based Transformer with Language Reconstruction for Video Paragraph Grounding.” ICME. 2022. (Best Student Paper Award) 16

16. Experiments  Datasets  ActivityNet-Caption        More than 20k videos from Youtube Open domain More than 15k vocabularies Long videos with 117s averaging Complex semantics Sufficient contexts Charades-STA       More than 6k videos from Youtube Indoors domain About 1300 vocabularies Short videos with 30s averaging Simple semantics Insufficient contexts 17

17. Experiments  Overall Comparison    Our method outperforms conventional VSG methods with a large margin. It also achieves greatly improvement compared with the previous VPG method. The GTLR performs better with more context information in paragraph queries. Overall Comparison on Two Benchmark Datasets ActivityNet-Caption Charades-STA Method Venue CBP ACL’20 54.3 35.76 17.8 36.85 - 36.80 18.87 - CPNet AAAI’21 - 40.56 21.63 40.65 - 40.32 22.47 37.36 BPNet AAAI’21 58.98 42.07 24.69 42.11 55.46 38.25 20.51 38.03 SV-VMR ICME’21 61.39 45.21 27.32 - - 38.09 19.98 - FVMR CVPR’21 53.12 41.48 29.12 - - 38.16 18.22 - DepNet AAAI’21 72.81 55.91 33.46 - 58.47 40.23 21.08 39.11 GTLR Ours 77.49 60.80 37.73 55.20 62.96 44.31 22.80 42.37 IoU=0.3 IoU=0.5 IoU=0.7 mIoU IoU=0.3 IoU=0.5 IoU=0.7 mIoU 18

18. Experiments  Qualitative Analysis    The GTLR achieves remarkable improvement on the localizing precision. Language Reconstruction makes it more explainable and intractable. As constraints for multimodal alignments, the reconstructed words show a semantic-clustering tendency. Visualization of two typical grounding examples by our GTLR and the counterpart DepNet [AAAI’21] 19

19. Visualization Demo  Cross-modal Video Content Retrieval   Our GTLR method effectively aligns two modalities and parallelly retrieval multiple events. This work is also presented for the ICME 2022 Best Student Paper Award. Query: Two young girls stand in a kitchen, with one holding a bag of something and talking and the other standing on a chair. Then, The two collaborate to … A case from the ActivityNet-Caption dataset ICME 2022 Best Student Paper 20

20. Discussion  Existing Challenges for VPG  Intra-modality contextual information is not explored sufficient in GTLR because of the fine-grained cross-modal interactions.  Video-Paragraph pairs require heavy annotation cost compared with Video- Sentence pairs. Pair 1 Pair 2 Pair 5 Video-Paragraph Annotations Video-Sentence Annotations 21

21. Our Proposed Method Semi-supervised Video-Paragraph Transformer (SVPTR)  We first build a base model, namely, Video-Paragraph TRansformer (VPTR).  Based on the fully-supervised base model VPTR, we introduce the mean-teacher framework thus achieve the Semi-supervised Learning VPG. [5] Xun Jiang, Xing Xu*, Jingran Zhang, Fumin Shen, Zuo Cao, Heng Tao Shen, “Semi-Supervised Video Paragraph Grounding With Contrastive Encoder.” CVPR. 2022. 22

22. Experiments  Overall Comparisons  We evaluate our method under fully-supervised and semi-supervised settings respectively. It shows competitive performance with less temporal annotated data.  Our proposed SVPTR method achieves remarkable improvement compared with both previous VSG and VPG methods. Semi-supervised Performance on the ActivityNet- Caption, TACoS, and Charades-CD-OOD Fully-supervised Performance on the TACoS Fully-supervised Performance on the ActivityNet-Caption 23

23. Experiments  Further Analysis and Visualization  The experimental results on different proportions of labeled training data show the effectiveness of our SVPTR method.  We also visualize the attention weights of decoders to furtherly illustrate the process of event-wise cross-modal alignments. Analysis on the annotated data Visualization of attention weights in event-wise decoder 24

24. The Latest Works  Exploration on Model Practice: Data and Computation  Existing methods follows a multimodal fusion pipeline to retrieve video content, which leads to heavy computational consumption.  Completely temporal annotations are subjective and hard to label thus bringing high data cost. We achieve more practical retrieval paradigm with a fair trade-off among data cost, model efficiency, and retrieval accuracy! 25

25. Our Proposed Method  Cheaper and Faster Moment Retrieval (CFMR)  We proposed Concept-based Multimodal Alignment module that avoids online cross-modal interactions widely used in previous methods.  We introduced point-level supervised learning to reduce annotation cost and Point- guided Contrastive Learning to leverage incomplete supervision signals. [6] Xun Jiang, Zailei Zhou, Xing Xu*, Yang Yang, Guoqing Wang, Heng Tao Shen, Faster Video Moment Retrieval with Point-Level Supervision. ACM MM. 2023. 26

26. Experiments  Retrieval Accuracy  Similarly, we evaluate our proposed CFMR method on the Charades-STA, ActivityNet-Captions, and TACoS datasets.  During training, the complete temporal annotations are inaccessible, but only point- level supervision provided. 27

27. Experiments  Retrieval Efficiency  In the inference, the semantic reconstructor is disabled and only two concept encoders are deployed, thus no cross-modal interactions or fusion is required.  As our method bypasses cross-modal interactions, the online computation, which can be only propelled after receiving users’ queries, has been reduced significantly. 28

28. Experiments  Further Analysis  We furtherly analyze our method on ablation studies, retrieval efficiency on long- term videos with different lengths, visualizations of multimodal representation, and performance on out-of-distribution datasets. Ablation Studies Efficiency Analysis Multimodal Representation OOD Performance 29

29. Experiments  Further Analysis  Finally, we also conduct qualitative analysis to make comprehensive comparisons on retrieval performance, efficiency, and annotation cost. 30

30. The Latest Works  Towards Applications: Retrieving Content from Video Collections  Exploring a Novel Fine-grained Video Retrieval Paradigm: a unified framework for retrieving Video-Text Retrieval and Video Content Retrieval.  Existing fusion-based methods are NOT compatible with video-text retrieval, as it is IMPOSSIBLE to conduct cross-modal interactions with each video in collections.  Following the text-based video retrieval seetings: NO additional temporal annotations. 31

31. Multi-Granularity Video Content Retrieval  Our Solution: Joint Search and Grounding (JSG)  Retrieving untrimmed videos that contains multiple events with partial text queries from video collections and synchronously retrieving the most related video events.  We design two branches to tackle this challenging problem, i.e., Glance Branch and Gaze Branch, which retrieve video files and events collaboratively. [7] Zhiguo Chen, Xun Jiang, Xing Xu*, Zuo Cao, Yijun Mo, Heng Tao Shen, Joint Searching and Grounding: Multi-Granularity Video Content Retrieval. ACM MM. 2023. 32

32. Experiments  Evaluate on Two Subtasks  We evaluate our JSG method on both video-level retrieval and event-level retrieval tasks, which comprehensively show the multi-granularity video retrieval performance.  Note that ALL temporal annotations are INACCESSIBLE. 33

33. Experiments  Further Analysis  We also conduct more ablation studies and observe the retrieval performance on two branches, i.e., Glance Branch and Gaze Branch, to take quantitative analysis on our proposed JSG method. 34

34. Experiments  Further Analysis  Here we demonstrate some retrieval cases from our experimental datasets. 35

35. Generative LLM-Driven Video Understanding  The prosperity of LLMs and the development of NLP research have demonstrated that data-centric AI is a certain way to well-performing models with high generalization.  LLMs-Driven pre-trained multimodal models will be a popular multimodal learning paradigm.  Video-Text Alignment may by an essential topic in “Video+LLM” studies. Mini-GPT[Zhu, et al., arXiv’23] BLIP-2 [Li, et al., arXiv’23] LLAVA [Liu, et al., arXiv’23] [8] Deyao Zhu, et al. "MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models." arXiv. 2023. [9] Junnan Li, et al. "BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models." arXiv. 2023. [10] Haotian Liu, et al. “Visual Instruction Tuning”. arXiv. 2023. 36

36. Generative LLM-Driven Video Understanding  Leveraging Generative LLMs is a booming research on Video-Language study, particularly for video understanding.  Video-ChatGPT [11] Maaz, Muhammad, et al. "Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Models." arXiv. 2023. 37

37. Generative LLM-Driven Video Understanding  Leveraging Generative LLMs is a booming research on Video-Language study, particularly for video understanding.  VideoChat (AskAnything) [12] KunChang Li, et al. " VideoChat: Chat-Centric Video Understanding." arXiv. 2023. 38

38. Generative LLM-Driven Video Understanding  Leveraging Generative LLMs is a booming research on Video-Language study, particularly for video understanding.  Video-LLAMA: Bridging video/audio models and generative LLMs with Q-Former. [13] Hang Zhang, et al. "Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understanding." EMNLP. 2023. 39

39. Generative LLM-Driven Video Understanding  Leveraging Generative LLMs is a booming research on Video-Language study, particularly for video understanding.  Video-LLAVA: Learning unified visual representations by a LanguageBind encoder before encoding with LLMs. [14] Bin Zhu, et al. "LanguageBind: Extending Video-Language Pretraining to N-modality by Language-based Semantic Alignment." arXiv. 2023. 40

40. Generative LLM-Driven Video Understanding  Leveraging Generative LLMs is a booming research on Video-Language study, particularly for video understanding.  TimeChat: LLM-Driven Video-Language models for Time-sensitive Tasks [15] Shuhuai Ren, et al. " TimeChat: A Time-sensitive Multimodal Large Language Model for Long Video Understanding." arXiv. 2023. 41

41. Generative LLM-Driven Video Understanding  Leveraging Generative LLMs is a booming research on Video-Language study, particularly for video understanding.  GPT-4V: What can we do if we combine video retrieval in LLMs? GPT-4 Turbo with Vision in Microsoft Azure Otter with Apple Vision Pro 42

42. Discussions for Future Work  The prosperity of LLMs and the development of NLP research have demonstrated that data-centric AI is a certain way to well-performing models with high generalization.  LLMs-Driven pre-trained multimodal models will be a popular multimodal learning paradigm.  Task-specific research will be highly competitive, and the task will be more and more complicated.  Potential future research interests may follows two ways: 1) embracing LLMs and 2) exploring model-agnostic theoretical evidence for multimodal video learning. VideoChat [Li, et al., arXiv’23] Video-LLAVA [Zhu, et al., arXiv’23] TimeChat [Ren, et al., arXiv’23] [12] KunChang Li, et al. " VideoChat: Chat-Centric Video Understanding." arXiv. 2023. [14] Bin Zhu, et al. "LanguageBind: Extending Video-Language Pretraining to N-modality by Language-based Semantic Alignment." arXiv. 2023. [15] Shuhuai Ren, et al. " TimeChat: A Time-sensitive Multimodal Large Language Model for Long Video Understanding." arXiv. 2023. 43

43. 敬请批评指正！ 44