Paper deep dive

Learning Question-Aware Keyframe Selection with Synthetic Supervision for Video Question Answering

Minchan Kwon, Hyounguk Shon, Junmo Kim

Year: 2026Venue: arXiv preprintArea: cs.CVType: PreprintEmbeddings: 23

Intelligence

Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 94%

Last extracted: 3/22/2026, 5:16:39 AM

Summary

The paper introduces a question-aware keyframe selection framework for VideoQA that addresses sparse supervision and frame redundancy. It utilizes synthetic keyframe labels derived from Large Multimodal Models (LMMs) and a Question-conditioned Coverage Regularization (QCCR) module to promote diverse, informative frame selection, achieving state-of-the-art performance on the NExT-QA benchmark.

Entities (5)

NExT-QA · dataset · 100%Question-conditioned Coverage Regularization · methodology · 98%Synthetic Keyframe Supervision · methodology · 98%InstructBLIP · model · 95%Qwen2.5-VL · model · 95%

Relation Signals (3)

Question-conditioned Coverage Regularization → reduces → Frame Redundancy

confidence 95% · QCCR ensures temporal diversity and reduces redundant choices.

NExT-QA → evaluates → VideoQA

confidence 90% · Experiments on NExT-QA show that our method significantly improves accuracy.

Synthetic Keyframe Supervision → improves → VideoQA

confidence 90% · Synthetic keyframe supervision provides high-level semantic guidance for VideoQA.

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Large multimodal models (LMMs) have recently demonstrated remarkable performance in video question answering (VideoQA), yet reasoning over video remains challenging due to high inference cost and diluted information. Keyframe selection offers efficiency and sharper reasoning but suffers from sparse supervision and redundant frame choices when relying only on image-text similarity. We present a question-aware keyframe selection framework with two components: pseudo keyframe labels derived from LMMs that provide informative supervision and a coverage regularization that promotes diverse, complementary evidence across time. Experiments on NExT-QA show that our method significantly improves accuracy, especially for temporal and causal question types, establishing keyframe selection as an effective and learnable module for VideoQA.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

22,233 characters extracted from source content.

Expand or collapse full text

LEARNING QUESTION-AWARE KEYFRAME SELECTION WITH SYNTHETIC SUPERVISION FOR VIDEO QUESTION ANSWERING Minchan Kwon † Hyounguk Shon † Junmo Kim Korea Advanced Institute of Science and Technology, South Korea ABSTRACT Large multimodal models (LMMs) have recently demon- strated remarkable performance in video question answering (VideoQA), yet reasoning over video remains challenging due to high inference cost and diluted information. Keyframe selection offers efficiency and sharper reasoning but suffers from sparse supervision and redundant frame choices when relying only on image–text similarity. We present a question- aware keyframe selection framework with two components: pseudo keyframe labels derived from LMMs that provide informative supervision and a coverage regularization that promotes diverse, complementary evidence across time. Ex- periments on NExT-QA show that our method significantly improves accuracy, especially for temporal and causal ques- tion types, establishing keyframe selection as an effective and learnable module for VideoQA. Index Terms— Video Question Answering, Keyframe Se- lection, Large Multimodal Model, Video Reasoning, Tempo- ral Reasoning 1. INTRODUCTION Large multimodal models (LMMs) that extend language models with visual understanding have recently achieved strong performance on Video Question Answering (VideoQA). Systems such as InstructBLIP and LLaVA further narrow the gap between open-ended language queries and long, visually rich content [1, 2, 3]. Despite these gains, reasoning on se- quences of images in VideoQA remains challenging. As the number of frames grows, inference cost scales substantially, and the model’s ability to localize question-relevant visual evidence within lengthy streams degrades [4]. These lim- itations are particularly pronounced in scene-/episode-level settings, where answers depend on sparse, widely separated moments that are easily drowned out by redundant context. An effective solution lies in keyframe selection, reducing a video to a compact, question-aware subset of frames that captures evidence while eliminating irrelevant content. Such selection promises a dual benefit. First, it improves efficiency by shortening the tokenized visual sequence the LMM must † Equal contribution. process. Second, it sharpens reasoning by concentrating at- tention on informative moments rather than forcing the model to sift through long stretches of distractors. However, learning a keyframe selector for VideoQA faces a fundamental obstacle: supervision is extremely sparse. There is no universally agreed ground-truth set of keyframes, and typical VideoQA datasets provide only question–answer pairs without temporal annotations. Furthermore, the dis- crete nature of keyframe selection (i.e., choosing the top-k frames) is inherently non-differentiable, making end-to-end training challenging. To circumvent this, previous works have resorted to noisy gradient approximation techniques [5, 6], while some utilize reinforcement learning to directly opti- mize for selection policies [4]. However, these methods often introduce significant algorithmic complexity and training in- stability. Consequently, many resort to framewise image–text similarity (e.g., CLIP [7]) to rank frames against the ques- tion [8, 5, 6], despite its own set of critical shortcomings: 1. Temporal/causal relations are ignored. Framewise matching treats frames independently and fails to cap- ture inter-frame dependencies central to temporal and causal reasoning; datasets such as NExT-QA, AGQA, and CLEVRER explicitly probe these abilities [9, 10, 11]. Questions such as “Why did the character pick up the phone?” or “What happens after the door opens?” require relations across frames that static image–text alignment cannot capture. 2. Redundancy under top-score selection. Selecting the top-k frames by similarity often yields temporally ad- jacent near-duplicates, leading to poor coverage of the diverse evidence a question may require (e.g., setup, ac- tion, consequence, before/after states, multiple partic- ipants). Diversity-/coverage-oriented selection objec- tives (e.g., AKS [12]) directly address this issue. 3. A supervision paradox. To pick the right frames, a selector needs a high-level understanding of the video–question interaction that we hope to enable downstream—yet such understanding is not directly supervised by standard VideoQA labels [9]. Our approach. We address these challenges with two com- plementary ideas: arXiv:2603.14953v1 [cs.CV] 16 Mar 2026 "What type of question is this?" Question Type Prompt "What does the child do after ending the call?" Question Image Encoder Text Encoder Backbone LLM Guassian Generator CLIP Similarity Question Type "Watching Camera" Predicted Answer "The boy watching camera." Ground Truth Video Keyframe Distribution VLM Ranking Synthetic Keyframe Question-Conditioned Coverage Regularization (QCCR) Predicted Keyframes LMM "What does the child do after ending the call?" Question Fig. 1: An overview of our proposed question-aware keyframe selection framework. The model processes the video with an image encoder and the question with a text encoder. The question embedding is then used in two parallel streams: (1) it is fed into a Gaussian Generator to create an initial keyframe distribution, and (2) it is used to generate synthetic keyframes via CLIP similarity and VLM ranking, which serve as weak supervision. To adapt the selection process, the LMM’s own backbone is re-purposed with a prompt to determine the question type. This information guides the Question-Conditioned Coverage Regularization (QCCR) module, which refines the initial distribution. Finally, the predicted keyframes are passed to the downstream LMM to generate the final answer. • Synthetic keyframe label generation. To overcome the scarcity of supervision, we prompt a strong LMM to produce coarse rationales (e.g., timestamps or frame indices) for its answers and transform these signals into pseudo keyframe labels. Although noisy, these labels encode higher-level video–question relations (including temporal and causal cues) that pure image– text matching misses. We further stabilize learning by aggregating multiple prompts/views and aligning selection quality with downstream answer correctness. • Question-conditioned coverage-regularized selec- tion. Starting from a similarity-based scoring back- bone (which can leverage CLIP or the vision encoder of an LMM), we introduce a question-conditioned cov- erage regularization term that explicitly discourages redundant choices and promotes diversity across se- mantics and time. Concretely, the selector is rewarded for picking frames that collectively “cover” more video frames, rather than saturating a single moment with near-duplicates. Together, these components yield a question-conditioned selector that (i) reduces inference cost by passing far fewer frames to the LMM and (i) preserves or improves answer quality by focusing on evidence-bearing moments, with par- ticular gains on temporal/causal categories. Our contributions are summarized as follows: 1. We design a question-aware keyframe selection module for video question answering, aiming to improve both efficiency and reasoning quality. 2. We develop a method to derive synthetic keyframe su- pervision from an LMM, injecting high-level video– question relations into the selector without requiring manual temporal annotations. 3. We introduce a question-conditioned coverage regular- ization that reduces redundancy and promotes comple- mentary evidence across time, conditioned on question semantics. 4. We evaluate our method on the NExT-QA bench- mark [9], showing notable gains on causal and temporal questions, and conduct ablations isolating the effects of coverage regularization and synthetic supervision. By coupling principled diversity with informative pseudo- labels, our approach transforms keyframe selection from a heuristic process into a learnable, question-aligned module, enabling LMMs to reason over long frame sequences at a fraction of the computational cost. 2. METHOD 2.1. Framework Overview Our goal is to learn a keyframe selector that identifies N keyframes from a video V = x 0 ,x 1 ,...,x T and a ques- tion Q. We adopt the Gaussian Generator framework from GCG [5] to design the keyframe selector. The Gaussian Gen- erator takes input video embeddings from the CLIP image encoder and question embeddings from the text encoder. It outputs K Gaussian masks g =g 1 ,...,g K . The keyframe probability distribution is defined as: p = Norm Σ K k=1 g k p∈ [0, 1], where Norm(·) normalizes the value to [0, 1]. The Gaus- sian masks are trained with pseudo timestamp labels w = [w 1 ,...,w K ] using MSE loss: L mse = 1 K K X k=1 (g k − w k ) 2 . We select the top-N keyframes from the Gaussian masks. These frames are passed through the Q-former [1] and then into the LMM. The final answer is trained with a VQA loss: L vqa =L VQA (A, ˆ A), where A is the ground truth answer and ˆ A is the model output. 2.2. Synthetic Keyframe Supervision (SKS) We propose a synthetic keyframe generation framework to provide temporal grounding. Following GCG, we first com- pute CLIP similarities and select the top-N frames as super- vision. However, CLIP similarity assigns similar scores to visually similar scenes, regardless of information content. It also ignores temporal order, making it weak for causal rea- soning. To address this, we leverage LMMs for keyframe supervision. LMMs capture temporal and causal relations through attention. We design prompts that guide the LMM to select frames relevant to answering the question. The system prompt given to the LMM is designed around four prioritized selection rules. 1. Informative first - Choose frames that explicitly show the entities/actions/clues referenced in the question or answer choices. 2. Non-redundancy - If multiple frames are nearly identi- cal, keep the earliest, clearer one. 3. Temporal logic - When sequence matters (before/after cues), preserve correct order. 4. Fallback to uniform - Only when none of the frames satisfy (1)-(3), select four frames that are as evenly spaced as possible across the video. The LMM is prompted to generate the indices of the selected frames. Finally, the model is trained with both CLIP-based supervision and synthetic keyframe supervision. 2.3. Question-conditioned Coverage Regularization Different question types require different keyframes. We propose Question-conditioned Coverage Regularization (QCCR) NExT-QA DescriptiveTemporalCausalAverage Co-Mem54.450.045.948.5 HCRN54.049.347.148.9 HGA 57.849.148.150.0 IGV59.651.748.651.3 HQGA59.452.349.051.8 B2A58.349.047.449.6 VCSR 62.351.553.054.1 VGT67.354.552.855.7 Raformer 67.857.758.259.6 TranSTR70.060.259.761.5 GCG69.074.467.870.4 SeViLA80.866.471.971.5 Ours77.871.571.673.3 Table 1: Performance comparison with state-of-the-art methods on the NEXT-QA benchmark. All methods are evaluated on accuracy across three question categories. Our method achieves the highest average accuracy, demonstrating its overall effectiveness. to adapt keyframe selection at test time. Coverage regular- ization [12] is used to enhance the post-hoc selection of keyframes given the scores associated with each video frame. At test time, the coverage regularization implicitly maximizes L cov = X t∈I s(Q,x t ) + λ(Q)· c(I), where s(Q,x t ) is the score of the t-th frame, I is the set of keyframe indices, and c(I) is the coverage regulariza- tion. The regularization strength λ controls how uniform the keyframe indices should be distributed across time. If the regularization is either too weak or too strong, the se- lected keyframes fails to capture the information required for answering the question. Therefore, coverage regularization needs to be applied adaptively depending on the semantics of the question. To this end, we propose to condition λ(Q) to the question and adapt to the type of the question. This requires that the keyframe selector must have the capability to under- stand the question prompt. Therefore, we re-use the LLM backbone with a query prompt containing the question to pre- dict its type. For each question type, we assign a predefined coverage regularization strength. 3. EXPERIMENT 3.1. Experiment Setting Dataset. We use the NExT-QA dataset [9]. NExT-QA con- tains 5.4k videos with an average length of 44s and 52k QA pairs. The dataset covers three types of questions: descrip- tive, temporal, and causal. Question : "What does the child do after ending the call?" CLIP Score 0.340.340.340.310.28 VLM Ranking 25431 Frames Fig. 2: A qualitative example illustrating the effectiveness of VLM Ranking over CLIP similarity. For the question “What does the child do after ending the call?”, frames with high CLIP scores are often re- dundant while the VLM Ranking correctly identifies the temporally relevant frames that occur after the call has ended. Baselines. We compare our method against representative VideoQA baselines. One group of methods focuses on re- lational reasoning, using hierarchical structures [13] or het- erogeneous graphs [14, 15]. Others improve performance through robust video-language alignment [16], causal reason- ing [17], or adaptive frame selection [18]. We also compare against SeViLA [19], a strong LMM-based method. Differ- ent from previous works, our model employs CLIP [7] for weak supervision and a lightweight Gaussian Generator to learn multiple masks. Experiment Details. We set T = 32 frames per video and select N = 4 keyframes. For pseudo-label generation, we use Qwen2.5-VL [20] as the LMM, which is a state-of-the- art open-source LMM. All hyperparameters for GCG training are kept consistent with the original settings. We adopt In- structBLIP [1] as the LMM backbone and EVA-CLIP [21] as the text–image encoder for the Gaussian Generator. 3.2. Main Results Table 1 reports the main results. Our method achieves the highest average score, demonstrating a robust and well- balanced performance across all question types. This demon- strates that our method performs robustly across question types, rather than achieving high scores due to being partic- ularly strong on specific questions. Compared to our direct baseline, GCG, there was a significant performance improve- ment in both descriptive and causal questions. This highlights that our proposed synthetic keyframes and QCCR are effec- tive at capturing complex semantic and causal relationships that simple similarity-based selection misses. Furthermore, while the state-of-the-art method SeViLA excels in the de- scriptive category, our model not only outperforms it on temporal questions but also secures a better overall average. Considering that SeViLA employs an LMM with billions of parameters during keyframe selection, our method, which utilises only a Gaussian generator in the form of a linear layer, also offers computational advantages. NExT-QA DescriptiveTemporalCausalAverage Ours77.871.571.673.3 w/o QCCR 75.967.970.971.5 w/o SKS 69.074.467.870.4 Table 2: Ablation study of our core components on the NEXT-QA dataset. The performance drop in both cases confirms that each com- ponent makes a crucial contribution to the final performance. 3.3. Qualitative Results Figure 2 displays the frames for specific questions from the NExT-QA dataset, alongside their corresponding CLIP scores and VLM Rankings. The CLIP score assigns higher scores to frames containing “child” and “call” elements mentioned in the question, whereas the VLM Ranking assigns the highest ranking to frames occurring after the call has ended, reflecting the causal and temporal nature of the question. Conversely, it assigned the lowest ranking to the unhelpful phone-answering scene. This demonstrates that relying on CLIP for ground truth keyframes presents difficulties in causal scenarios. Our method, which utilizes both VLM and CLIP grounding re- sults, can blend the strengths of both approaches. 3.4. Ablation Study To disentangle the contributions of each component, we conducted ablation experiments on the NExT-QA dataset (Ta- ble 2). Removing QCCR caused a clear drop, especially on temporal and causal questions. Without it, the model over- selected redundant frames, reducing evidence diversity. Elim- inating synthetic keyframe supervision led to the sharpest de- cline, with descriptive accuracy falling from 77.8 to 69.0. Pure similarity-based selection failed to capture semantics, focusing on visually dominant but uninformative frames. In- terestingly, the w/o SKS scores higher on temporal questions than our full method. We hypothesize this is because the sim- ple similarity-based selection aligns well with certain patterns in the temporal questions, but its poor performance elsewhere highlights a lack of generalizability. 4. CONCLUSION In this work, we addressed the key challenges of super- vision scarcity and frame redundancy in keyframe selection for VideoQA. We introduced two complementary solutions: Synthetic Keyframe Supervision (SKS) to provide high-level semantic guidance, and Question-conditioned Coverage Reg- ularization (QCCR) to ensure temporal diversity. We showed that with proper supervision and regularization, keyframe se- lection can be elevated from a simple retriever to a stronger learnable component, achieving state-of-the-art performance in NExT-QA. 5. REFERENCES [1] Wenliang Dai, Junnan Li, Dongxu Li, A. M. H. Tiong, Junqi Zhao, Weisheng Wang, Boyang Albert Li, Pascale Fung, and Steven C. H. Hoi, “Instructblip: Towards general-purpose vision-language models with instruc- tion tuning,” in Neural Information Processing Systems, 2023. [2] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee, “Visual instruction tuning,” in Neural Information Processing Systems, 2023. [3] Hang Zhang, Xin Li, and Lidong Bing,“Video- llama:An instruction-tuned audio-visual language model for video understanding,”arXiv preprint arXiv:2306.02858, 2023. [4] Sicheng Yu, CHENGKAI JIN, Huanyu Wang, Zheng- hao Chen, Sheng Jin, ZHONGRONG ZUO, XU XI- AOLEI, Zhenbang Sun, Bingni Zhang, Jiawei Wu, Hao Zhang, and Qianru Sun, “Frame-voyager: Learning to query frames for video large language models,” in The Thirteenth International Conference on Learning Rep- resentations, 2025. [5] Haibo Wang, Chenghang Lai, Yixuan Sun, and Weifeng Ge, “Weakly supervised gaussian contrastive ground- ing with large multimodal models for video question an- swering,” in Proceedings of the 32nd ACM International Conference on Multimedia, 2024, p. 5289–5298. [6] Jianxin Liang, Xiaojun Meng, Yueqian Wang, Chang Liu, Qun Liu, and Dongyan Zhao, “End-to-end video question answering with frame scoring mechanisms and adaptive sampling,” arXiv preprint arXiv:2407.15047, 2024. [7] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sas- try, Amanda Askell, Pamela Mishkin, Jack Clark, et al., “Learning transferable visual models from natural lan- guage supervision,” in International conference on ma- chine learning. PmLR, 2021, p. 8748–8763. [8] Hao Liang, Jiapeng Li, Tianyi Bai, Xijie Huang, Linzhuang Sun, Zhengren Wang, Conghui He, Bin Cui, Chong Chen, and Wentao Zhang, “Keyvideollm: To- wards large-scale video keyframe selection,”arXiv preprint arXiv:2407.03104, 2024. [9] Junbin Xiao, Xindi Shang, Angela Yao, and Tat-Seng Chua,“Next-qa: Next phase of question-answering to explaining temporal actions,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2021, p. 9777–9786. [10] Madeleine Grunde-McLaughlin, Ranjay Krishna, and Maneesh Agrawala, “Agqa: A benchmark for compo- sitional spatio-temporal reasoning,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pat- tern Recognition, 2021, p. 11287–11297. [11] Kexin Yi*, Chuang Gan*, Yunzhu Li, Pushmeet Kohli, Jiajun Wu, Antonio Torralba, and Joshua B. Tenenbaum, “Clevrer: Collision events for video representation and reasoning,” in International Conference on Learning Representations, 2020. [12] Xi Tang, Jihao Qiu, Lingxi Xie, Yunjie Tian, Jianbin Jiao, and Qixiang Ye, “Adaptive keyframe sampling for long video understanding,” in Proceedings of the Com- puter Vision and Pattern Recognition Conference, 2025, p. 29118–29128. [13] T. Le, Vuong Le, S. Venkatesh, and T. Tran, “Hierar- chical conditional relation networks for video question answering,” in Computer Vision and Pattern Recogni- tion, 2020. [14] Pin Jiang and Yahong Han, “Reasoning with heteroge- neous graph alignment for video question answering,” in AAAI Conference on Artificial Intelligence, 2020. [15] Junbin Xiao, Angela Yao, Zhiyuan Liu, Yicong Li, Wei Ji, and Tat seng Chua, “Video as conditional graph hier- archy for multi-granular question answering,” in AAAI Conference on Artificial Intelligence, 2021. [16] Yicong Li, Xiang Wang, Junbin Xiao, Wei Ji, and Tat seng Chua, “Invariant grounding for video question an- swering,” in Computer Vision and Pattern Recognition, 2022. [17] Jungin Park, Jiyoung Lee, and K. Sohn, “Bridge to answer: Structure-aware graph interaction network for video question answering,” in Computer Vision and Pat- tern Recognition, 2021. [18] Yicong Li, Junbin Xiao, Chun Feng, Xiang Wang, and Tat seng Chua, “Discovering spatio-temporal rationales for video question answering,” in IEEE International Conference on Computer Vision, 2023. [19] Antoine Yang, Antoine Miech, Josef Sivic, I. Laptev, and C. Schmid, “Zero-shot video question answering via frozen bidirectional language models,” in Neural Information Processing Systems, 2022. [20] Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wen- bin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang, Haiyang Xu, and Junyang Lin, “Qwen2.5-vl technical report,” arXiv preprint arXiv:2502.13923, 2025. [21] Quan Sun, Yuxin Fang, Ledell Wu, Xinlong Wang, and Yue Cao, “Eva-clip: Improved training techniques for clip at scale,” arXiv preprint arXiv:2303.15389, 2023.

Instant research discovery