Paper deep dive
Talking to Me or Someone Else? Rethinking Talk-to-Me Detection in Egocentric Videos
Feiyu Du, Xi He, Jia Li, Yapeng Tian, Weili Wu
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 93%
Last extracted: 9/20/2026, 4:20:10 AM
Summary
This paper rethinks Talk-to-Me (TTM) detection in egocentric videos by reformulating it from an offline clip-level binary classification task to an online, frame-level prediction problem. The authors introduce the Online TTM Dataset, derived from Ego4D, containing 406 video clips and ~900K frames annotated with four distinct social speaking states: Background (BG), Self-talking (SELF), Talking-to-Others (TTO), and Talking-to-Me (TTM). They propose a multimodal online model that integrates visual cues (facial appearance, head pose, mouth motion, wearer-centered geometry), acoustic features, and Whisper-derived speech semantics. The model achieves 75.5% frame-level F1, demonstrating that structured four-state modeling and semantic audio cues significantly improve TTM recognition over strong baselines.
Entities (9)
Relation Signals (9)
Online TTM Dataset → hasclasses → Talking-to-Me (TTM)
confidence 99% · labeled with frame-level social interaction categories (e.g., background, TTM, talking-to-others, self-talking)
Multimodal Model → achievesperformance → 75.5% frame-level F1
confidence 98% · achieves 75.5% frame-level F1 on TTM
Online TTM Dataset → hassize → 900K frames
confidence 98% · consisting of 406 egocentric video clips with approximately 900K annotated frames
Online TTM Dataset → derivedfrom → Ego4D
confidence 95% · by extending the Ego4D social interaction benchmark
Online TTM Dataset → hasclasses → Self-talking (SELF)
confidence 95% · labeled with frame-level social interaction categories (e.g., background, TTM, talking-to-others, self-talking)
Online TTM Dataset → hasclasses → Background (BG)
confidence 95% · labeled with frame-level social interaction categories (e.g., background, TTM, talking-to-others, self-talking)
Online TTM Dataset → hasclasses → Talking-to-Others (TTO)
confidence 95% · labeled with frame-level social interaction categories (e.g., background, TTM, talking-to-others, self-talking)
Multimodal Model → uses → Whisper
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Online understanding of who is talking to the camera wearer is a key capability for egocentric social interaction. However, existing talk-to-me (TTM) studies are commonly formulated as offline clip-level recognition, which is poorly aligned with online interaction and overlooks the diverse non-TTM speaking states that naturally arise in egocentric videos. In this paper, we revisit this problem by reformulating it as an online, frame-level prediction task. Instead of treating TTM as a binary problem against a single negative class, we model it in the presence of diverse and previously underexplored non-TTM states, such as talking-to-others, self-talking, and background conditions. To support this new formulation, we construct an Online TTM Dataset consisting of 406 egocentric video clips with approximately 900K annotated frames, each labeled with frame-level social interaction categories (e.g., background, TTM, talking-to-others, self-talking), by extending the Ego4D social interaction benchmark. In this benchmark, we evaluate five adapted baselines and develop a new model that integrates social cues across modalities. Experimental results show that our multimodal model, which jointly leverages audio, visual, and speech-semantic cues, achieves 75.5% frame-level F1 on TTM, outperforming strong baselines and enabling a systematic analysis of how different speaking states affect TTM recognition.
Tags
Links
- Source: https://arxiv.org/abs/2609.14118v1
- Canonical: https://arxiv.org/abs/2609.14118v1
Trouble viewing inline? Open PDF directly →
Full Text
60,839 characters extracted from source content.
Expand or collapse full text
Talking to Me or Someone Else? Rethinking Talk-to-Me Detection in Egocentric Videos Conference: Proceedings of the 34th ACM International Conference on Multimedia; November 10–14, 2026; Rio de Janeiro, BrazilProceedings of the 34th ACM International Conference on Multimedia (M ’26), November 10–14, 2026, Rio de Janeiro, BrazilDOI: 10.1145/3767308.3836441ISBN: 979-8-4007-2213-4/2026/11CCS: Information systems Multimedia information systems Feiyu Du Affiliation: The University of Texas at Dallas, Richardson, United States email: feiyu.du@utdallas.edu , Xi He Affiliation: The University of Texas at Dallas, Richardson, United States email: xi.he@utdallas.edu , Jia Li Affiliation: The University of Texas at Dallas, Richardson, United States email: jia.li@utdallas.edu , Yapeng Tian Note: Corresponding author. Affiliation: The University of Texas at Dallas, Richardson, United States email: yapeng.tian@utdallas.edu and Weili Wu Affiliation: The University of Texas at Dallas, Richardson, United States email: weiliwu@utdallas.edu © c Abstract. Online understanding of who is talking to the camera wearer is a key capability for egocentric social interaction. However, existing talk-to-me (TTM) studies are commonly formulated as offline clip-level recognition, which is poorly aligned with online interaction and overlooks the diverse non-TTM speaking states that naturally arise in egocentric videos. In this paper, we revisit this problem by reformulating it as an online, frame-level prediction task. Instead of treating TTM as a binary problem against a single negative class, we model it in the presence of diverse and previously underexplored non-TTM states, such as talking-to-others, self-talking, and background conditions. To support this new formulation, we construct an Online TTM Dataset consisting of 406 egocentric video clips with approximately 900K annotated frames, each labeled with frame-level social interaction categories (e.g., background, TTM, talking-to-others, self-talking), by extending the Ego4D social interaction benchmark. In this benchmark, we evaluate five adapted baselines and develop a new model that integrates social cues across modalities. Experimental results show that our multimodal model, which jointly leverages audio, visual, and speech-semantic cues, achieves 75.5% frame-level F1 on TTM, outperforming strong baselines and enabling a systematic analysis of how different speaking states affect TTM recognition. Keywords: Egocentric Video Understanding, Talk-to-me Recognition, Multimodal Learning †c-license: by Figure 1. We introduce the Online TTM Dataset, a multimodal benchmark for online who-is-talking-to-me perception in realistic first-person scenarios. It temporally grounds social speaking states in continuous egocentric video streams while integrating visual, audio, and speech information. Within such a continuous first-person stream, the current social speaking state may shift over short temporal windows from background to talking-to-others, talking-to-me, or self-talking.An overview figure showing a continuous egocentric video stream annotated with four social speaking states over time. The visual layout highlights how the state of interaction changes within a first-person scene, including background, talking-to-others, talking-to-me, and self-talking. The figure emphasizes temporal transitions between these states and illustrates that wearer-directed speech appears as only one part of a richer stream of multimodal social activity. 1. Introduction Understanding who is talking to the camera wearer is a fundamental capability for egocentric social perception systems (Jiang et al., 2022; Ryan et al., 2023; Clarke et al., 2023; Clarke et al., 2025; Plizzari et al., 2024). A wearable assistant, an augmented reality interface, or a human-centered robot should not only detect the presence of speech, but also determine whether a visible person is actually addressing the wearer at the current moment. This capability is central to a wide range of interaction-driven applications, such as deciding when a system should respond, identifying the relevant speaker in a multi-person scene, and distinguishing direct interaction from background social activity (Dhaussy et al., 2023; Xu et al., 2022). Despite its importance, the talk-to-me (TTM) problem in egocentric video (Kong et al., 2024) has so far been studied primarily under an offline clip-level formulation. In this setting, each video segment is assigned a single label representing its speaking state. While such a design is reasonable for offline recognition, it is not well aligned with online egocentric interaction. In natural social scenes, speaking states often evolve within short temporal windows: a person may first talk to someone else and then turn to the wearer; the wearer may begin speaking; and background music or environmental noise may coexist with socially relevant cues (Tao et al., 2021; Wang et al., 2024). Consequently, the prevailing clip-level formulation makes it difficult to capture these state transitions and to study how they influence TTM recognition at frame-level granularity (Alcázar et al., 2020; Alcázar et al., 2021; Min et al., 2022). A further limitation lies in how the problem is commonly formulated. Existing TTM settings are often reduced to a binary distinction between talking-to-me and not talking-to-me (Grauman et al., 2022). However, the negative side of this formulation is far from uniform. In egocentric social scenes, non-TTM may include background or non-interactive conditions, self-talking by the wearer, and talking-to-others. These cases differ substantially in both social meaning and recognition difficulty. Background is often relatively easy to reject, self-talking places the wearer in the role of speaker, while talking-to-others contains genuine speech and visible facial motion but with a different addressee. Merging all of them into a single negative class hides these differences and makes it difficult to understand what actually causes TTM errors (Zhang et al., 2021). Fig. 1 illustrates how the social speaking state changes over time in a continuous first-person video stream. Motivated by these issues, we revisit TTM from two aspects. First, rather than treating it as an offline clip-level recognition problem, we reformulate it as an online frame-level prediction task, where the model must decide at each moment whether a visible person is speaking to the wearer using only the current and past observations. Second, rather than grouping all non-TTM cases into a single class, we explicitly preserve their differences and study TTM in the presence of multiple socially meaningful speaking states. To support this setting, we construct the Online TTM Dataset, a curated benchmark with approximately 900K frames derived from the Ego4D Social Dataset. Specifically, we reconstruct frame-level supervision for online prediction and re-sample the data to mitigate the substantial class imbalance in the original distribution. During this process, we preserve temporal context around TTM events, continuity within segments, and informative state boundaries, so that the benchmark contains sufficient cross-state variation to support learning transitions between speaking states (Guermal et al., 2024). We further clean the data by removing missing-frame cases, annotation coverage issues, and samples with prolonged absence of valid face boxes, including consecutive black frames. On top of this benchmark, we adapt five representative baselines and develop an online multimodal temporal model for TTM recognition. Unlike offline formulations, the model operates in a streaming manner and predicts the current speaking state using only the current frame and recent history, without access to future frames (Köpüklü et al., 2021; Liao et al., 2023). To resolve socially confusable cases, the model explicitly integrates fine-grained facial cues, including head-pose-aware visual representations and mouth-motion dynamics (Braga et al., 2024; Jung et al., 2024). We further introduce wearer-centered geometric cues to explicitly encode the spatial importance of each face in multi-face scenes, including normalized position, scale, and aspect ratio. These cues provide a strong prior for identifying which face is more likely to be close to the camera, occupy the wearer’s visual focus, and therefore serve as the current interaction target. In addition, the model combines conventional acoustic features with Whisper-derived speech semantics, enabling it to jointly reason about speaking activity, interaction target, and semantic context. By augmenting audio with text-semantic information extracted from speech, we further examine whether richer semantic cues are more effective than acoustic features alone for distinguishing TTM from socially similar speaking states. Experiments on the Online TTM Dataset show that our proposed online multimodal model achieves 75.5% frame-level F1 on TTM and consistently outperforms strong baselines across frame-level F1 and segment-level mAP. Beyond overall performance, the proposed benchmark also enables a more comprehensive analysis of the problem. We evaluate the effect of enhancing acoustic features with Whisper-derived speech semantics, and conduct attribution analysis to examine how different speaking states influence TTM recognition. These results show that semantically enhanced audio representations are substantially more effective than conventional acoustic features alone, while four-state modeling provides a more informative formulation for understanding TTM errors and socially confusable cases. Our contributions are summarized as follows: 1) We revisit egocentric TTM from two perspectives: we reformulate it from offline clip-level recognition to online frame-level prediction, and replace the simplified TTM and non-TTM setting with a socially structured formulation that better reflects realistic egocentric interaction. 2) We build the Online TTM Dataset, a curated benchmark with approximately 900K frames derived from the Ego4D Social Dataset by reconstructing frame-level supervision, mitigating severe class imbalance through temporally coherent re-sampling, and cleaning noisy cases such as missing frames, annotation coverage issues, and prolonged absence of valid face boxes. 3) We adapt five representative baselines and develop an online multimodal temporal model that integrates fine-grained facial cues, wearer-centered geometry, acoustic features, and Whisper-derived speech semantics, achieving 75.5% frame-level F1 on TTM while enabling systematic analysis of the effect of different speaking states on recognition. (a) (b) Figure 2. Qualitative illustration of the original Ego4D social annotations from two different clips. (a) One clip is dominated by long background intervals, while TTM segments are sparse and fragmented. (b) Another clip contains frequent transitions among speaking states, with blank intervals indicating frames without valid social-state annotations. 2. Related Work 2.1. Egocentric social interaction Egocentric social interaction understanding studies how surrounding people attend to and communicate with the camera wearer. A key step in making this problem measurable at scale is Ego4D, whose Social benchmark explicitly defines wearer-centered tasks such as Looking-at-Me (LAM) and Talking-to-Me (TTM) (Ego4D, ). This benchmark shifts the focus from generic first-person scene understanding to a more interaction-centered question: whether a visible social partner is directing attention or speech toward the wearer. Building on this benchmark, subsequent work has mainly advanced TTM within the original wearer-directed prediction setting. EgoTask Translation (EgoT2) shows that heterogeneous Ego4D tasks can benefit one another through cross-task translation, indicating that wearer-centered social prediction can leverage broader egocentric task structure (Xue et al., 2023; Peirone et al., 2024). QuAVF then targets the Ego4D TTM challenge more directly through quality-aware audio-visual fusion, showing that reliability-aware integration of visual and audio cues improves benchmark performance (Lin et al., 2023). Recent work has further expanded the problem from local audio-visual evidence to richer interaction structure (Huang et al., 2023; Chen et al., 2024). In particular, long-term social context has been shown to be important for egocentric addressee detection (Kong et al., 2024), while AV-CONV moves beyond isolated wearer-directed decisions by modeling conversational relations among multiple participants from egocentric observations (Jia et al., 2024). Face-voice association and missing-modality robustness further broaden wearer-centered speech understanding beyond binary wearer-directed classification (Tao et al., 2024b; Qian et al., 2026; Saeed et al., 2024). CASTLE 2024 and its associated ACM Multimedia Grand Challenge support multimodal analysis of synchronized ego- and exo-centric recordings, but do not specifically address frame-level wearer-directed speaking-state recognition (Rossetto et al., 2025a; Rossetto et al., 2025b). 2.2. Active Speaker Detection Active Speaker Detection (ASD) asks whether a visible face is speaking at a given moment and has become a core problem in audio-visual scene understanding. AVA-ActiveSpeaker established a large-scale benchmark for this task and enabled systematic study of frame-level speaker activity from synchronized face tracks and audio (Roth et al., 2020). With this benchmark in place, subsequent work progressively strengthened the modeling toolkit. Earlier ASD and meeting-oriented systems also explored weak supervision and meeting-specific audiovisual fusion (Chakravarty et al., 2016; Madrigal et al., 2021; Pibre et al., 2023; Berghi and Jackson, 2023). TalkNet highlights the importance of long-range temporal reasoning for audio-visual active speaker detection (Tao et al., 2021). EASEE pushes ASD toward an end-to-end formulation by jointly optimizing representation learning and speaker prediction (Alcazar et al., 2022). ASD-Transformer shows that efficient self- and cross-modal transformer designs can be effective for ASD (Datta et al., 2022; Radman and Laaksonen, 2024; Wuerkaixi et al., 2022). Target-specific formulations and richer fusion strategies have also been explored for harder speaker-selection settings (Jiang et al., 2023; Xiong et al., 2022; Assuncao et al., 2021; Li et al., 2023; Yin et al., 2025). LoCoNet further combines long-term intra-speaker context with short-term inter-speaker competition, reflecting a more structured treatment of temporal and multi-person information in ASD (Wang et al., 2024). More recent work extends ASD from offline recognition toward deployment-oriented settings. Gurvich et al. study an online causal ASD system designed for low-power edge use (Gurvich et al., 2024; Liao et al., 2025), and Kundu et al. analyze efficient and streaming ASD under explicit context constraints (Kundu et al., 2025). This evolution (Braga et al., 2024; Tao et al., 2024a) makes ASD highly relevant to our work from a methodological perspective, since it provides mature designs for multimodal fusion, temporal reasoning, and causal inference. Standard ASD detects visible speakers, while egocentric TTM specifically identifies speech directed to the camera wearer (Yang et al., 2023; Kang et al., 2020; Pan et al., 2024). We adapt ASD methods to this distinct social setting by introducing a structured four-state prediction framework. Table 1. Comparison between the Ego4D social annotations and our Online TTM Dataset. Dataset Prediction unit Granularity Videos Total frames TTM ratio Online eval. Benchmark purpose Ego4D Social Segment label Clip-level 439 3,951,000 11.49% × Offline clip recognition Online TTM (ours) State label Frame-level 406 885,866 41.49% ✓ Online frame prediction 3. Problem Formulation and Dataset 3.1. Problem Definition We revisit talk-to-me (TTM) from the perspective of online egocentric social interaction and reformulate it as an online frame-level task. Given a first-person video stream, the goal is to infer, at each time step, whether a visible person is currently speaking to the camera wearer. Conventional TTM settings are typically organized around short video segments, where each segment is assigned a single speaking label, and interaction is treated as a coarse temporal unit. In contrast, we define TTM as a frame-level prediction task: at each time step, the model predicts the speaking state of the current moment rather than assigning a single label to an entire segment. This formulation provides a more direct characterization of online egocentric social interaction. Formally, let xtx_t denote the multimodal observation at time step t, including the visual scene and its associated audio context. Under the online setting, the model is allowed to access the current observation together with a finite history of past observations, but not future frames. Let x1:t=x1,x2,…,xtx_1:t=\x_1,x_2,…,x_t\ denote the available observation sequence up to time step t. The model predicts (1) y^t=f(x1:t), y_t=f(x_1:t), where y^t y_t is the estimated speaking state at the current moment. Rather than treating TTM as a binary event against an undifferentiated negative class, we define the task over four speaking states: (2) yt∈BG,SELF,TTO,TTM,y_t∈\BG,SELF,TTO,TTM\, where BG denotes background or non-interactive conditions, SELF denotes self-talking by the wearer, TTO denotes talking-to-others, and TTM denotes talking-to-me. Preserving this structure in the label space is more informative than collapsing all non-TTM cases into a single negative class. Although TTM remains the primary target, BG, SELF, and TTO provide socially distinct decision boundaries that help separate TTM from neighboring speaking states. 3.2. Challenges of Online Egocentric TTM Online egocentric TTM remains challenging even under the structured four-state formulation. First, wearer-directed speech is often temporally local and dynamically evolving. In realistic first-person social interaction, speaking states may change within a short interval: a person may first talk to someone else and then turn to the wearer, and the wearer may begin speaking during an ongoing interaction. As a result, a key difficulty is not only whether TTM occurs, but also when it occurs and how it is separated from neighboring states over time. Second, TTM must be distinguished from socially confusable speaking states rather than from a generic negative class. In particular, TTO and SELF often share strong visual and acoustic overlap with TTM. Talking-to-others still contains genuine speech and visible lip movements, while self-talking may also co-occur with salient audio activity and wearer-centered attention cues. These states therefore introduce ambiguity, making our task substantially more difficult than previous speech-versus-background detection. Third, the task is subject to an online causal constraint. At inference time, the model must make a decision using only the current observation and its preceding context, without access to future frames. This prevents the model from relying on future evidence to resolve ambiguous moments and makes the problem more consistent with real deployment settings in wearable assistants, augmented reality systems, and interactive egocentric perception. Figure 3. Overview of the proposed online multimodal model for egocentric social speaking recognition. From a continuous first-person video stream, we construct a person-centered visual stream and an aligned audio-text stream. The visual encoder extracts complementary cues from facial appearance, head-related representation, mouth motion, and wearer-centered geometry, while the audio-text branch captures frame-synchronous acoustic evidence and Whisper-derived speech semantics. These features are fused by cross-attention and then processed by a causal temporal Transformer with a causal mask to produce frame-level four-state predictions over BG, SELF, TTO, and TTM.A pipeline diagram of the proposed online multimodal model. A continuous egocentric video stream is split into a person-centered visual branch and an aligned audio-text branch. The visual side contains modules for facial appearance, head-related features, mouth motion, and wearer-centered geometry. The audio-text side provides acoustic evidence and text-semantic features. These representations are fused through cross-attention and then sent to a causal temporal Transformer, which outputs frame-level predictions for four classes: background, self-talking, talking-to-others, and talking-to-me. 3.3. Dataset Construction and Statistics The original Ego4D social annotations (Ego4D, ) already include four speaking states, namely BG, SELF, TTO, and TTM. However, the annotations are organized for clip-level recognition, where each segment is assigned a single social-state label. While such a formulation is suitable for offline recognition, it is not directly applicable to the online frame-level TTM task studied in this paper, where the model must determine at each time step whether someone is currently speaking to the wearer. To address this mismatch, we first reconstruct frame-level supervision from the original segment-level annotations, so that each frame is associated with a speaking state. As illustrated in Fig. 2, the original Ego4D social annotations contain long background intervals, sparse TTM segments, frequent alternation among speaking states, and substantial unlabeled gaps. Beyond the mismatch in temporal granularity, the original Ego4D distribution also exhibits a strong class imbalance: TTM accounts for only 11.49% of all usable frames, while the remaining data are dominated by background, talking-to-others, and self-talking cases. To address this issue, we retain all usable TTM frames and re-sample BG, SELF, and TTO in a task-aware manner. Importantly, this re-sampling is not designed merely to achieve numerical balance. Instead, we preserve non-TTM samples that are temporally close to TTM events, locally continuous within interaction windows, and sufficiently informative around state boundaries. The resulting benchmark therefore not only alleviates the original class bias, but also provides a more structured negative space for distinguishing TTM from neighboring speaking states. If the source data were sampled naively, many realistic state transitions would be lost, and the task would degenerate into isolated frame classification. To avoid this, we explicitly preserve a sufficient number of cross-state transitions during benchmark construction, so that the final data contain not only static speaking states, but also realistic transitions between them. In particular, around 40% of the sampled windows contain at least two speaking states, allowing the benchmark to cover transitions such as TTO-to-TTM and SELF-to-BG. This design enables models to learn how one social speaking state changes into another, making the benchmark more faithful to the way wearer-directed speech emerges in online interaction and more suitable for analyzing why TTM is often confused with certain non-TTM states. The original annotations also contain severe noise that can interfere with online modeling, including missing frames, incomplete annotation coverage, prolonged absence of valid face bounding boxes, and consecutive black-frame segments. If left untreated, such noise would introduce artificial discontinuities unrelated to the underlying interaction semantics. To improve data quality, we therefore apply a systematic cleaning procedure. For occasional face disappearance or local missing observations, we retain the natural perturbations that may occur in realistic scenes; for clearly corrupted sequences, we remove them. Specifically, we filter out samples with missing frames or abnormal annotation coverage, and discard sequences within the same labeled state when the proportion of black frames or no-face frames exceeds 40%. Through this process, the resulting data preserve the complexity of realistic first-person interaction while substantially reducing severe noise that would otherwise affect model performance and analysis. After frame-level reconstruction, task-aware re-sampling, and data cleaning, we obtain the Online TTM Dataset. The final benchmark contains 885,866 high-quality frame-level samples. Starting from 439 candidate videos, we retain 406 videos and split them into 334/21/51 videos for training, validation, and testing, respectively. The final label distribution comprises 367,529 TTM, 100,321 SELF, 170,321 TTO, and 247,695 BG frames, resulting in a substantially more balanced distribution than the original Ego4D annotations. More importantly, the Online TTM Dataset is not only a more suitable frame-level dataset in scale and balance, but also a benchmark that simultaneously supports online frame-level prediction, TTM-centric evaluation, and realistic socially complex interaction scenarios. A comparison between the original Ego4D social annotations and our Online TTM Dataset is summarized in Table 1. 4. Method 4.1. Overview We propose a multimodal model for online egocentric TTM recognition. Given a short first-person temporal window, the model predicts the current social speaking state at each time step under a causal setting, where only the current and past observations are accessible. At time step t, the model receives a person-centered face crop ItI_t, a frame-synchronous audio input xtaudiox_t^audio, wearer-centered geometric cues gtg_t, a pre-extracted head representation hth_t, and a text-semantic input xttextx_t^text. The model outputs a four-way prediction (3) y^t∈BG,SELF,TTO,TTM, y_t∈\BG,SELF,TTO,TTM\, corresponding to background, self-talking, talking-to-others, and talking-to-me, respectively. As shown in Fig. 3, the proposed framework consists of three components. First, a person-centered visual branch extracts complementary cues from target face tracks, including facial appearance, head-related information, mouth motion, and wearer-centered geometry. Second, an audio-text branch encodes frame-synchronous acoustic evidence together with Whisper-derived speech semantics. Third, the visual and audio-text representations are fused by cross-attention and then modeled by a causal temporal Transformer with a causal mask for online temporal reasoning, followed by frame-level four-state prediction. 4.2. Visual Encoder The visual encoder captures complementary person-centered cues for online TTM, including facial appearance, head-related information, mouth dynamics, and wearer-centered geometry. For each frame, we use a target face crop as the primary visual input. When the annotated interaction target is available, we use the corresponding face bounding box; otherwise, we fall back to the largest visible face in the frame. For SELF frames, we use a zero-filled face crop together with zero-valued person-centered features. Global facial feature extraction. We use a lightweight ViT-style encoder (Dosovitskiy et al., 2020) to extract a global facial representation vtvitv_t^vit from each face crop in the target face track. Each face crop is resized to 224×224224× 224, tokenized into non-overlapping 16×1616× 16 patches, and processed by a 4-layer Transformer encoder with 4 attention heads. The final CLS token is used as the global facial representation. This branch encodes facial appearance and contextual information, providing a stable visual basis for subsequent fine-grained branches. Head-focused representation. Global appearance alone is insufficient for determining whether the interaction is directed toward the wearer, since a visible person may not actually be facing the wearer while speaking (as shown in Sec. 5.3). We therefore design a head-related branch. Specifically, we use a pretrained 6DRepNet (Hempel et al., 2024) to extract a 2048D hidden-layer representation for each face and feed this representation into a lightweight MLP to obtain a head feature vtheadv_t^head. Compared with using only a few explicit pose angles, this design preserves richer information about head orientation and pose variation, such as whether the face is frontal, noticeably turned away, or undergoing motion such as turning or nodding. Mouth-motion encoder. To explicitly model speaking activity, we design a mouth-motion branch. This branch focuses on the lower half of the face as an approximate mouth region, first extracts a compact static mouth representation with a small CNN, then computes temporal differences between adjacent frames, and finally applies causal temporal convolution to capture short-term lip-motion patterns. The motivation is that, if a person is speaking to the wearer, the mouth should exhibit temporally evolving motion rather than remain visually static. Temporal differencing encourages the model to focus on changes in mouth configuration instead of static mouth appearance, while causal temporal convolution further strengthens the modeling of local articulation dynamics. Geometry encoder. Spatial position is also an important cue for identifying the interaction target in egocentric video, especially when multiple faces appear in the same frame. We therefore derive wearer-centered geometric cues from the selected face bounding box and represent them as gt=[cx,cy,a,r],g_t=[c_x,c_y,a,r], where cxc_x and cyc_y denote the normalized box center, a is the normalized box area, and r is the aspect ratio. We then project gtg_t into a learned embedding vtgeov_t^geo. These cues provide an explicit spatial prior: the person speaking to the wearer is often more central, visually larger, and more prominent in the first-person view, whereas small or peripheral faces are more likely to be side participants or background people. This wearer-centered geometry is particularly helpful for disambiguation in multi-person scenes. Finally, we concatenate all branches into a unified visual representation vt=[vtvit;vthead;vtmouth;vtgeo].v_t=[v_t^vit;v_t^head;v_t^mouth;v_t^geo]. This representation jointly encodes who the visible person is, whether the person is speaking, whether the head is oriented toward the wearer, and whether the face is spatially interaction-relevant, making it better aligned with the evidence required for our online egocentric TTM task. 4.3. Audio and Speech-Semantic Encoding Acoustic evidence alone is often insufficient for socially grounded TTM recognition, especially when TTM and TTO share similar speech patterns and background noise. To address this issue, we jointly model acoustic evidence and speech semantics. At each time step, we extract a frame-synchronous waveform chunk from the original audio stream and align an utterance-level semantic feature to the current video frame according to its temporal span. The waveform chunk is then encoded with a lightweight temporal CNN to produce an audio representation. This branch captures local speech-related acoustic patterns and provides direct evidence of speaking activity, complementing the visual cues from the person-centered stream. However, the social meaning of an utterance is often reflected more strongly in its semantic content than in acoustics alone. We therefore further incorporate Whisper (Radford et al., 2023) to extract speech-semantic representations derived from ASR transcripts, aligned with the temporal span of each utterance. These representations are projected into the shared multimodal space and serve as an additional semantic cue. The intuition is that, in conversational settings, spoken content may implicitly indicate whether an utterance is directed toward the wearer, participates in the current interaction, or corresponds to background speech. Compared with raw acoustic signals, such semantic information can provide stronger cues for socially grounded TTM detection. The audio-text branch can be summarized as (4) at=Eaud(xtaudio),s~t=xttext,a_t=E_aud(x_t^audio), s_t=x_t^text, where ata_t denotes the acoustic representation and s~t s_t denotes the pre-extracted text-semantic feature. In implementation, we first use Whisper to perform automatic speech recognition (ASR) and generate speech transcripts, and then apply a MiniLM text encoder (Wang et al., 2020) to extract a text-semantic feature from transcripts. Table 2. Comparison with baselines on the Online TTM Dataset. Frame-level F1 (%) Segment-level (%) Efficiency Method BG SELF TTO TTM TTM mAP Params (M) Model latency (ms/frame) Ego4D baseline (Ego4D, ) 49.9 59.2 2.3 61.3 62.6 18.066 8.2 Streaming ASD (Kundu et al., 2025) 66.5 69.4 25.3 66.3 72.8 3.451 4.0 CMeRT (Pang et al., 2025) 64.9 71.0 25.0 65.9 71.0 38.088 12.0 LoCoNet (Wang et al., 2024) 64.0 63.4 22.7 63.3 70.6 17.304 7.4 ProTAS (Shen and Elhamifar, 2024) 62.3 60.7 20.3 60.1 68.5 20.638 6.4 Ours 99.4 72.6 40.2 75.5 79.8 5.876 4.49 4.4. Online Multimodal Fusion and Causal Temporal Modeling Since social speaking states evolve over time, the model must jointly perform multimodal fusion and causal temporal modeling under an online constraint. We therefore first project the visual and audio features and fuse them via cross-attention in a shared hidden space. Specifically, visual features are used as queries, while audio features are used as keys and values, so that speech-related evidence can refine the person-centered visual stream. The same windowed causal mask is also applied in cross-attention, ensuring that cross-modal interaction at time step t depends only on the current and recent past observations. We also explored the reverse direction, but found it to be noisier in practice; therefore, we adopt visual-query, audio-value fusion in the final model. After cross-modal fusion, the projected text-semantic feature is added to the fused representation. This allows the model to integrate not only visual appearance and acoustic activity, but also higher-level semantic context associated with speech. The resulting multimodal sequence is then passed to a causal temporal Transformer, where each block consists of masked multi-head self-attention and a feed-forward network. To enforce online inference, we apply a windowed causal attention mask (5) Mij=0,i−τ+1≤j≤i,−∞,otherwise,M_ij= cases0,&i-τ+1≤ j≤ i,\\ -∞,&otherwise, cases where τ=32τ=32 is the past context length. This mask ensures that the token at time step i only attends to the current and recent past observations, without access to future frames. The overall online fusion-and-temporal reasoning process can be summarized as (6) pt=Softmax(W[Etemp(CrossAttn(v1:T,a1:T;M)+Wss~1:T;M)]t),p_t=Softmax\! (W [E_temp\! (CrossAttn(v_1:T,a_1:T;M)+W_s s_1:T;\,M ) ]_t ), where WsW_s denotes a learnable projection for the speech-semantic feature, EtempE_temp denotes the causal temporal Transformer, and M is the windowed causal attention mask. 4.5. Training Objective Under the four-state formulation, given the predicted class distribution ptp_t at time step t, we optimize the model using frame-level cross-entropy over all valid, non-padded frames: (7) ℒ=−∑t∈Ω∑c=14yt,clogpt,c,L=- _t∈ _c=1^4y_t,c p_t,c, where Ω denotes the set of valid time steps in a batch. In practice, padded positions are ignored in both the loss computation and the attention masks, and label smoothing is applied during training for more stable optimization. 5. Experiments 5.1. Experimental Setup All experiments are conducted on the proposed Online TTM Dataset. For fair comparison, all baselines are adapted to the same four-state label space and causal online setting, and are retrained under the same train/val/test split. At the frame level, we report F1 for each class, with particular emphasis on TTM F1 in the main results. At the segment level, we report TTM segment mAP. All models are implemented in PyTorch. Training is performed using AdamW with an initial learning rate of 3×10−43× 10^-4, a batch size of 512, and 60 training epochs. We further report model size in Params (M) and model-only inference latency in ms/frame as efficiency indicators. Figure 4. Results comparison of our method and Streaming ASD on challenging scenarios from the Online TTM test set. Green boxes denote TTM targets, orange boxes denote TTO targets, and red circles indicate false predictions. The first row shows the ground truth, the second row shows our predictions, and the third row shows the predictions of Streaming ASD.A qualitative comparison figure showing several challenging examples from the Online TTM test set. Each example is arranged in three rows: the first row shows the ground truth, the second row shows predictions from the proposed method, and the third row shows predictions from Streaming ASD. Faces identified as talking-to-me or talking-to-others are marked with colored boxes, and incorrect predictions are marked with circles. The figure shows that the proposed method more often places the correct target annotation on socially confusable faces than the baseline. 5.2. Experimental Results We compare our method against several representative baselines under the same online four-state protocol. The Ego4D baseline (Ego4D, ) serves as a task-specific reference inherited from the original benchmark. Streaming ASD (Kundu et al., 2025) serves as a causal audio-visual streaming baseline under strict online inference constraints. CMeRT (Pang et al., 2025) and LoCoNet (Wang et al., 2024) provide two representative temporal backbones for online context modeling. ProTAS (Shen and Elhamifar, 2024) is adapted as an additional progress-aware causal temporal baseline. Table 2 reports the main comparison results on the Online TTM Dataset. Our method achieves the best results on all four class-wise F1 metrics and the highest TTM segment mAP, including 75.5% frame-level F1 on TTM and 79.8% TTM segment mAP. Table 2 also compares model size and inference latency. Our method achieves the best overall recognition performance while remaining computationally efficient, with a compact model size of 5.876M parameters and a model-only inference latency of 4.49 ms/frame. Including face detection, 6DRepNet, Whisper ASR, and MiniLM, the complete streaming pipeline runs at 48.1 ms/frame (20.8 FPS), with Whisper accounting for most of the runtime. We further analyze the effects of the label formulation and temporal setting. Four-state training improves TTM F1 from 64.2% under binary training to 75.5%, indicating that explicitly modeling socially distinct negative states provides more informative decision boundaries. Offline inference achieves 87.2% TTM F1, compared with 75.5% under the causal online setting, quantifying the performance cost of operating without future observations. Compared with Streaming ASD, our method improves TTM F1 by 9.2% and TTM segment mAP by 7.0%. The gain on TTO is even larger, reaching 14.9%, which is particularly important because TTO is one of the most socially confusable competing states for TTM. These results suggest that our method better distinguishes wearer-directed speech from speech directed to others. Compared with the Ego4D baseline, the improvement is even larger across all socially meaningful speaking states, indicating that simply adapting a generic benchmark baseline to the online frame-level setting is insufficient. Figure 4 further presents qualitative comparisons with Streaming ASD on challenging examples from the Online TTM test set, where our method produces more accurate predictions in socially confusable cases. Table 3. Ablation study of the main components. Components Frame-level F1 (%) Segment-level (%) ViT Head Mouth Geom Text BG SELF TTO TTM TTM mAP ✓ 56.33 14.28 0.05 62.27 67.00 ✓ ✓ 64.80 63.21 0.15 63.23 71.07 ✓ ✓ ✓ 66.68 70.45 19.42 64.39 72.13 ✓ ✓ ✓ ✓ 70.67 70.88 26.07 65.00 72.21 ✓ ✓ ✓ ✓ ✓ 99.42 72.60 40.19 75.54 79.78 5.3. Ablation We next analyze the contribution of the main components. Starting from a ViT-only baseline, we gradually add the Head-focused representation (Head), the Mouth-motion encoder (Mouth), the Geometry encoder (Geom), and finally the Whisper-derived text-semantic branch (Text). The results are summarized in Table 3. First, adding Head on top of ViT leads to clear improvements on BG, SELF, and TTM, indicating that head-related cues provide useful information for distinguishing socially meaningful speaking states. This is consistent with the intuition that wearer-directed interaction is closely related to whether a visible person is oriented toward the wearer. Second, adding Mouth further improves the results, especially on SELF and TTO, suggesting that local mouth dynamics provide useful evidence for speaking activity, although their gain on TTM is relatively limited. Third, introducing Geom yields additional improvements, particularly on BG and TTO, supporting our hypothesis that wearer-centered spatial priors, such as face centrality and prominence in the egocentric view, are important for identifying socially relevant targets. The largest gain comes from the introduction of the text-semantic branch. When Whisper-derived text-semantic features are added, performance improves substantially across all classes, with especially large gains on BG, TTO, and TTM. In particular, the frame-level F1 of TTM increases from 65.00% to 75.54%, while the TTM segment mAP rises from 72.21% to 79.78%. The improvement on TTO is also substantial, increasing from 26.07% to 40.19%. These results suggest that although progressively enriched visual cues improve social speaking recognition, higher-level semantic information is critical for resolving cases where audio-visual evidence alone remains ambiguous. Additional robustness tests show graceful degradation under missing or corrupted inputs. Zeroing the audio or face input yields 72.8% and 69.4% TTM F1, respectively, while 30% and 50% face-box dropout result in 69.8% and 67.2% TTM F1. Together with the no-text result in Table 3, these results indicate that the model does not rely exclusively on any single modality. To examine whether the text-semantic branch mainly exploits transcript-level shortcuts, we further shuffle transcripts across speaking segments while preserving silent intervals. Under this control, BG F1 remains nearly unchanged, decreasing from 99.42% to 98.8%, whereas TTM F1 drops from 75.5% to 69.6%. This result suggests that the BG improvement is largely associated with speech-presence information, while accurate TTM recognition benefits from temporally aligned semantic content. Table 4. TTM-centered confusion attribution with and without text semantics. All values denote the number of frames. TTM False Negatives TTM False Positives Variant BG SELF TTO BG SELF TTO No Text 17,567 9,511 49 15,583 1,305 25,694 Full Model 111 5,106 14,187 0 5,297 21,493 5.4. Error Attribution Analysis The text-semantic branch produces the largest redistribution of TTM-related errors among all model components. We therefore focus on the TTM-centered confusion patterns of the full model and the variant without text semantics. Specifically, we analyze: (1) false negatives, where the ground-truth label is TTM but the model predicts a non-TTM state; and (2) false positives, where the model predicts TTM but the GT label belongs to another state. As shown in Table 4, removing the text-semantic branch causes substantial confusion between BG and TTM. Aggregating both directions, BG-related TTM confusion decreases from 33,150 frames without text semantics to only 111 frames in the full model. This shows that semantic information largely resolves speech-versus-background ambiguity. After adding text semantics, the largest remaining error is TTO predicted as TTM, with 21,493 frames, followed by TTM predicted as TTO, with 14,187 frames. The remaining errors therefore concentrate on socially similar speaking states rather than background conditions. This supports our task formulation: the main challenge is not merely detecting whether speech is present, but determining whether the speech is directed to the camera wearer. 6. Conclusion We revisit egocentric talk-to-me (TTM) recognition from the perspective of online social interaction. Instead of the conventional offline clip-level formulation, we reformulate TTM as an online frame-level prediction task under a causal setting, enabling moment-level reasoning about wearer-directed speech. To support this setting, we construct the Online TTM Dataset, a large-scale benchmark derived from Ego4D Social with approximately 900K frame-level annotations, preserving realistic transitions among background, self-talking, talking-to-others, and talking-to-me states. We further develop an online multimodal model that integrates visual cues, wearer-centered geometry, acoustic features, and ASR-derived speech semantics, achieving 75.5% frame-level F1 on TTM and outperforming strong baselines. Acknowledgements. This work was supported by the National Eye Institute of the National Institutes of Health (NIH) under Award Number R01EY037100. The content is solely the responsibility of the authors and does not necessarily represent the official views of NIH. References Alcázar et al. (2020) J. L. Alcázar, F. Caba, L. Mai, F. Perazzi, J. Lee, P. Arbeláez, and B. Ghanem Active speakers in context. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, p. 12465–12474. Cited by: §1. Alcázar et al. (2021) J. L. Alcázar, F. Caba, A. K. Thabet, and B. Ghanem Maas: multi-modal assignation for active speaker detection. In Proceedings of the IEEE/CVF international conference on computer vision, p. 265–274. Cited by: §1. Alcazar et al. (2022) J. L. Alcazar, M. Cordes, C. Zhao, and B. Ghanem End-to-end active speaker detection. In European Conference on Computer Vision, p. 126–143. Cited by: §2.2. Assuncao et al. (2021) G. Assuncao, N. Goncalves, and P. Menezes Bio-inspired modality fusion for active speaker detection. Applied Sciences 11 (8), p. 3397. Cited by: §2.2. Berghi and Jackson (2023) D. Berghi and P. J. Jackson Leveraging visual supervision for array-based active speaker detection and localization. IEEE/ACM transactions on audio, speech, and language processing 32, p. 984–995. Cited by: §2.2. Braga et al. (2024) O. Braga, W. Xia, K. Johnson, A. Chuang, Y. Ye, O. Siohan, and T. A. Nguyen Large scale self-supervised pretraining for active speaker detection. In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), p. 10036–10040. Cited by: §1, §2.2. Chakravarty et al. (2016) P. Chakravarty, J. Zegers, T. Tuytelaars, and H. Van hamme Active speaker detection with audio-visual co-training. In Proceedings of the 18th ACM international conference on multimodal interaction, p. 312–316. Cited by: §2.2. Chen et al. (2024) C. Chen, K. Ashutosh, R. Girdhar, D. Harwath, and K. Grauman Soundingactions: learning how actions sound from narrated egocentric videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 27252–27262. Cited by: §2.1. Clarke et al. (2023) J. Clarke, Y. Gotoh, and S. Goetze Improving audiovisual active speaker detection in egocentric recordings with the data-efficient image transformer. In 2023 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), p. 1–8. Cited by: §1. Clarke et al. (2025) J. Clarke, Y. Gotoh, and S. Goetze Speaker embedding informed audiovisual active speaker detection for egocentric recordings. In ICASSP 2025-2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), p. 1–5. Cited by: §1. Datta et al. (2022) G. Datta, T. Etchart, V. Yadav, V. Hedau, P. Natarajan, and S. Chang Asd-transformer: efficient active speaker detection using self and multimodal transformers. In ICASSP 2022-2022 IEEE international conference on acoustics, speech and signal processing (ICASSP), p. 4568–4572. Cited by: §2.2. Dhaussy et al. (2023) T. Dhaussy, B. Jabaian, F. Lefèvre, and R. Horaud Audio-visual speaker diarization in the framework of multi-user human-robot interaction. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), p. 1–5. Cited by: §1. Dosovitskiy et al. (2020) A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, et al. An image is worth 16x16 words: transformers for image recognition at scale. Cited by: §4.2. [14] Ego4D Social interactions benchmark. Note: https://ego4d-data.org/docs/benchmarks/social/Official benchmark documentation Cited by: §2.1, §3.3, Table 2, §5.2. Grauman et al. (2022) K. Grauman, A. Westbury, E. Byrne, Z. Chavis, A. Furnari, R. Girdhar, J. Hamburger, H. Jiang, M. Liu, X. Liu, et al. Ego4d: around the world in 3,000 hours of egocentric video. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, p. 18995–19012. Cited by: §1. Guermal et al. (2024) M. Guermal, A. Ali, R. Dai, and F. Bremond Joadaa: joint online action detection and action anticipation. In Proceedings of the IEEE/CVF winter conference on applications of computer vision, p. 6889–6898. Cited by: §1. Gurvich et al. (2024) I. Gurvich, I. Leichter, D. R. Palle, Y. Asher, A. Vinnikov, I. Abramovski, V. Gopal, R. Cutler, and E. Krupka A real-time active speaker detection system integrating an audio-visual signal with a spatial querying mechanism. In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), p. 8781–8785. Cited by: §2.2. Hempel et al. (2024) T. Hempel, A. A. Abdelrahman, and A. Al-Hamadi Toward robust and unconstrained full range of rotation head pose estimation. IEEE Transactions on Image Processing 33, p. 2377–2387. Cited by: §4.2. Huang et al. (2023) C. Huang, Y. Tian, A. Kumar, and C. Xu Egocentric audio-visual object localization. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, p. 22910–22921. Cited by: §2.1. Jia et al. (2024) W. Jia, M. Liu, H. Jiang, I. Ananthabhotla, J. M. Rehg, V. K. Ithapu, and R. Gao The audio-visual conversational graph: from an egocentric-exocentric perspective. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 26396–26405. Cited by: §2.1. Jiang et al. (2022) H. Jiang, C. Murdock, and V. K. Ithapu Egocentric deep multi-channel audio-visual active speaker localization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 10544–10552. Cited by: §1. Jiang et al. (2023) Y. Jiang, R. Tao, Z. Pan, and H. Li Target active speaker detection with audio-visual cues. Cited by: §2.2. Jung et al. (2024) C. Jung, S. Lee, K. Nam, K. Rho, Y. J. Kim, Y. Jang, and J. S. Chung Talknce: improving active speaker detection with talk-aware contrastive learning. In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), p. 8391–8395. Cited by: §1. Kang et al. (2020) W. Kang, B. C. Roy, and W. Chow Multimodal speaker diarization of real-world meetings using d-vectors with spatial features. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), p. 6509–6513. Cited by: §2.2. Kong et al. (2024) D. Kong, F. Khan, X. Zhang, P. Singhal, and Y. N. Wu Long-term social interaction context: the key to egocentric addressee detection. In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), p. 8250–8254. Cited by: §1, §2.1. Köpüklü et al. (2021) O. Köpüklü, M. Taseska, and G. Rigoll How to design a three-stage architecture for audio-visual active speaker detection in the wild. In Proceedings of the IEEE/CVF international conference on computer vision, p. 1193–1203. Cited by: §1. Kundu et al. (2025) A. Kundu, Y. Jin, M. Sekhavat, M. Horton, D. Tormoen, and D. Naik An efficient and streaming audio visual active speaker detection system. In ICASSP 2025-2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), p. 1–5. Cited by: §2.2, Table 2, §5.2. Li et al. (2023) Z. Li, B. Zhao, and Y. Yuan Bio-inspired audiovisual multi-representation integration via self-supervised learning. In Proceedings of the 31st ACM International Conference on Multimedia, p. 3755–3764. Cited by: §2.2. Liao et al. (2025) J. Liao, H. Duan, K. Feng, W. Zhao, Y. Yang, L. Chen, and Y. Chen Lr-asd: lightweight and robust network for active speaker detection. International Journal of Computer Vision 133 (7), p. 4749–4769. Cited by: §2.2. Liao et al. (2023) J. Liao, H. Duan, K. Feng, W. Zhao, Y. Yang, and L. Chen A light weight model for active speaker detection. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, p. 22932–22941. Cited by: §1. Lin et al. (2023) H. Lin, C. Wang, M. Chen, S. Fu, and Y. F. Wang Quavf: quality-aware audio-visual fusion for ego4d talking to me challenge. arXiv preprint arXiv:2306.17404. Cited by: §2.1. Madrigal et al. (2021) F. Madrigal, F. Lerasle, L. Pibre, and I. Ferrané Audio-video detection of the active speaker in meetings. In 2020 25th International Conference on Pattern Recognition (ICPR), p. 2536–2543. Cited by: §2.2. Min et al. (2022) K. Min, S. Roy, S. Tripathi, T. Guha, and S. Majumdar Learning long-term spatial-temporal graphs for active speaker detection. In European conference on computer vision, p. 371–387. Cited by: §1. Pan et al. (2024) Z. Pan, G. Wichern, F. G. Germain, A. Subramanian, and J. Le Roux Late audio-visual fusion for in-the-wild speaker diarization. In 2024 IEEE International Conference on Acoustics, Speech, and Signal Processing Workshops (ICASSPW), p. 174–178. Cited by: §2.2. Pang et al. (2025) Z. Pang, F. Sener, and A. Yao Context-enhanced memory-refined transformer for online action detection. In Proceedings of the Computer Vision and Pattern Recognition Conference, p. 8700–8710. Cited by: Table 2, §5.2. Peirone et al. (2024) S. A. Peirone, F. Pistilli, A. Alliegro, and G. Averta A backpack full of skills: egocentric video understanding with diverse task perspectives. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 18275–18285. Cited by: §2.1. Pibre et al. (2023) L. Pibre, F. Madrigal, C. Equoy, F. Lerasle, T. Pellegrini, J. Pinquier, and I. Ferrané Audio-video fusion strategies for active speaker detection in meetings. Multimedia Tools and Applications 82 (9), p. 13667–13688. Cited by: §2.2. Plizzari et al. (2024) C. Plizzari, G. Goletto, A. Furnari, S. Bansal, F. Ragusa, G. M. Farinella, D. Damen, and T. Tommasi An outlook into the future of egocentric vision: c. plizzari et al.. International Journal of Computer Vision 132 (11), p. 4880–4936. Cited by: §1. Qian et al. (2026) X. Qian, X. Zhu, A. Brutti, and D. Liang EgoAdapt: enhancing robustness in egocentric interactive speaker detection under missing modalities. arXiv preprint arXiv:2603.18082. Cited by: §2.1. Radford et al. (2023) A. Radford, J. W. Kim, T. Xu, G. Brockman, C. McLeavey, and I. Sutskever Robust speech recognition via large-scale weak supervision. In International conference on machine learning, p. 28492–28518. Cited by: §4.3. Radman and Laaksonen (2024) A. Radman and J. Laaksonen AS-net: active speaker detection using deep audio-visual attention. Multimedia Tools and Applications 83 (28), p. 72027–72042. Cited by: §2.2. Rossetto et al. (2025a) L. Rossetto, W. Bailer, D. Dang-Nguyen, G. Healy, B. Þ. Jónsson, O. Kongmeesub, H. Le, S. Rudinac, K. Schöffmann, F. Spiess, et al. The castle 2024 dataset: advancing the art of multimodal understanding. In Proceedings of the 33rd ACM International Conference on Multimedia, p. 12629–12635. Cited by: §2.1. Rossetto et al. (2025b) L. Rossetto, W. Bailer, C. Gurrin, D. T. Dang Nguyen, K. Schoeffmann, and A. Tran Overview of the first castle grand challenge at acm multimedia 2025. In Proceedings of the 33rd ACM International Conference on Multimedia, p. 14271–14272. Cited by: §2.1. Roth et al. (2020) J. Roth, S. Chaudhuri, O. Klejch, R. Marvin, A. Gallagher, L. Kaver, S. Ramaswamy, A. Stopczynski, C. Schmid, Z. Xi, et al. Ava active speaker: an audio-visual dataset for active speaker detection. In ICASSP 2020-2020 IEEE international conference on acoustics, speech and signal processing (ICASSP), p. 4492–4496. Cited by: §2.2. Ryan et al. (2023) F. Ryan, H. Jiang, A. Shukla, J. M. Rehg, and V. K. Ithapu Egocentric auditory attention localization in conversations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 14663–14674. Cited by: §1. Saeed et al. (2024) M. S. Saeed, S. Nawaz, M. Moscati, R. K. Das, M. S. Tahir, M. Z. Zaheer, M. I. Liaqat, M. H. Khan, K. Nandakumar, M. H. Yousaf, et al. A synopsis of fame 2024 challenge: associating faces with voices in multilingual environments. In Proceedings of the 32nd ACM International Conference on Multimedia, p. 11333–11334. Cited by: §2.1. Shen and Elhamifar (2024) Y. Shen and E. Elhamifar Progress-aware online action segmentation for egocentric procedural task videos. In 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), p. 18186–18197. Cited by: Table 2, §5.2. Tao et al. (2021) R. Tao, Z. Pan, R. K. Das, X. Qian, M. Z. Shou, and H. Li Is someone speaking? exploring long-term temporal features for audio-visual active speaker detection. In Proceedings of the 29th ACM international conference on multimedia, p. 3927–3935. Cited by: §1, §2.2. Tao et al. (2024a) R. Tao, X. Qian, R. K. Das, X. Gao, J. Wang, and H. Li Enhancing real-world active speaker detection with multi-modal extraction pre-training. IEEE Transactions on Multimedia 27, p. 2362–2373. Cited by: §2.2. Tao et al. (2024b) R. Tao, Z. Shi, Y. Jiang, D. Truong, E. Chng, M. Alioto, and H. Li Multi-stage face-voice association learning with keynote speaker diarization. In Proceedings of the 32nd ACM International Conference on Multimedia, p. 11342–11347. Cited by: §2.1. Wang et al. (2020) W. Wang, F. Wei, L. Dong, H. Bao, N. Yang, and M. Zhou Minilm: deep self-attention distillation for task-agnostic compression of pre-trained transformers. Advances in neural information processing systems 33, p. 5776–5788. Cited by: §4.3. Wang et al. (2024) X. Wang, F. Cheng, and G. Bertasius Loconet: long-short context network for active speaker detection. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, p. 18462–18472. Cited by: §1, §2.2, Table 2, §5.2. Wuerkaixi et al. (2022) A. Wuerkaixi, Y. Zhang, Z. Duan, and C. Zhang Rethinking audio-visual synchronization for active speaker detection. Cited by: §2.2. Xiong et al. (2022) J. Xiong, Y. Zhou, P. Zhang, L. Xie, W. Huang, and Y. Zha Look&listen: multi-modal correlation learning for active speaker detection and speech enhancement. IEEE Transactions on Multimedia 25, p. 5800–5812. Cited by: §2.2. Xu et al. (2022) E. Z. Xu, Z. Song, S. Tsutsui, C. Feng, M. Ye, and M. Z. Shou Ava-avd: audio-visual speaker diarization in the wild. In Proceedings of the 30th ACM International Conference on Multimedia, p. 3838–3847. Cited by: §1. Xue et al. (2023) Z. Xue, Y. Song, K. Grauman, and L. Torresani Egocentric video task translation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 2310–2320. Cited by: §2.1. Yang et al. (2023) C. Yang, M. Chen, Y. Wang, and Y. Wang Uncertainty-guided end-to-end audio-visual speaker diarization for far-field recordings. In Proceedings of the 31st ACM international conference on multimedia, p. 4031–4041. Cited by: §2.2. Yin et al. (2025) Y. Yin, X. Yang, L. Liang, X. Li, and Y. Zou Audio-faces intra-frame alignment with graph attention networks for active speaker detection. In ICASSP 2025-2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), p. 1–5. Cited by: §2.2. Zhang et al. (2021) Y. Zhang, S. Liang, S. Yang, X. Liu, Z. Wu, S. Shan, and X. Chen Unicon: unified context network for robust active speaker detection. In Proceedings of the 29th ACM international conference on multimedia, p. 3964–3972. Cited by: §1.