Paper deep dive

Noise Adaptive Streaming Audio-Visual Speech Token Enhancement for Robust Full-Duplex Spoken Dialogue Models

Bella Godiva, Yeonju Kim, Yong Man Ro

Year: 2026Venue: arXiv preprintArea: cs.SDType: PreprintEmbeddings: 59

Intelligence

Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 93%

Last extracted: 9/14/2026, 5:09:14 AM

Summary

The paper introduces AV-STE, a modular streaming audio-visual front-end designed to enhance robustness in full-duplex spoken dialogue models under noisy conditions. Unlike previous methods that require costly retraining of large speech models, AV-STE restores corrupted semantic speech tokens using visual lip cues and noisy audio before they reach a frozen speech LLM (Moshi). It employs a streaming audio-visual encoder and a noise-adaptive fusion module that uses soft token cross-attention and entropy-based modulation to balance visual and auditory inputs. Experiments on LRS3 and Seamless Interaction datasets demonstrate significant improvements in response coherence and semantic token accuracy under various noise and interference conditions.

Entities (9)

AV-STE · modelmethod · 98%LRS3 · dataset · 95%Moshi · model · 95%Seamless Interaction · dataset · 93%Mimi · modelcomponent · 92%Noise-Adaptive Fusion · module · 92%Streaming AV Encoder · module · 92%AudioSet · dataset · 90%AV-HuBERT · modelcomponent · 90%

Relation Signals (7)

AV-STE → evaluatedon → LRS3

confidence 95% · Experiments on LRS3 and Seamless Interaction show that AV-STE substantially improves semantic-token recovery...

Noise-Adaptive Fusion → partof → AV-STE

confidence 95% · The proposed framework consists of two main components: ... (2) a Noise-Adaptive Fusion module...

Streaming AV Encoder → partof → AV-STE

confidence 95% · The proposed framework consists of two main components: (1) a Streaming Audio-Visual Encoder...

AV-STE → uses → Moshi

confidence 95% · When integrated with frozen Moshi, AV-STE improves average GPT-4o-judged response coherence...

AV-STE → evaluatedon → Seamless Interaction

confidence 93% · Gains also transfer to out-of-domain Seamless Interaction.

AV-STE → uses → Mimi

confidence 92% · We use Moshi (Défossez et al., 2024) as the frozen full-duplex spoken dialogue model and its Mimi tokenizer as the speech-token representation.

Streaming AV Encoder → initializedfrom → AV-HuBERT

confidence 90% · We initialize the Streaming AV Encoder from AV-HuBERT (Shi et al., 2022)...

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Full-duplex spoken dialogue systems enable simultaneous listening and speaking, but their audio-only perception often fails under background noise and overlapping speech, leading to incoherent responses. Recent audio-visual dialogue approaches show that incorporating visual cues such as lip movements improve robustness under audio corruption. However, existing approaches often adapt the large speech dialogue model itself to process visual input, requiring costly multimodal training. We propose AV-STE, a modular streaming audio-visual front-end that restores corrupted semantic speech tokens from noisy audio and lip video before they reach the speech LLM. The downstream dialogue model remains entirely frozen, preserving its pretrained conversational capabilities. When integrated with frozen Moshi, AV-STE improves average GPT-4o-judged response coherence from 1.42 to 1.91 under same-dataset speaker interference while largely preserving turn-taking behavior. Gains also transfer to out-of-domain Seamless Interaction.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

57,002 characters extracted from source content.

Expand or collapse full text

Noise Adaptive Streaming Audio-Visual Speech Token Enhancement for Robust Full-Duplex Spoken Dialogue Models Bella Godiva Yeonju Kim Yong Man Ro †thanks: Corresponding author. Affiliation: Integrated Vision and Language Lab, KAIST, South Korea Affiliation: bellagodiva, yeonju7.kim, ymro@kaist.ac.kr Abstract Full-duplex spoken dialogue systems enable simultaneous listening and speaking, but their audio-only perception often fails under background noise and overlapping speech, leading to incoherent responses. Recent audio-visual dialogue approaches show that incorporating visual cues such as lip movements improve robustness under audio corruption. However, existing approaches often adapt the large speech dialogue model itself to process visual input, requiring costly multimodal training. We propose AV-STE, a modular streaming audio-visual front-end that restores corrupted semantic speech tokens from noisy audio and lip video before they reach the speech LLM. The downstream dialogue model remains entirely frozen, preserving its pretrained conversational capabilities. When integrated with frozen Moshi, AV-STE improves average GPT-4o-judged response coherence from 1.42 to 1.91 under same-dataset speaker interference while largely preserving turn-taking behavior. Gains also transfer to out-of-domain Seamless Interaction. Code and models are available: https://github.com/bellagodiva/av-ste.git 1 Introduction Full-duplex spoken dialogue models enable real-time interaction by listening and speaking simultaneously, supporting natural behaviors such as interruptions and backchannels. However, their robustness to acoustic interference remains underexplored. In real-world environments, background noise and competing speakers can corrupt the user’s speech representation, leading to degraded understanding and less coherent responses. Figure 1: In noisy environments, background sounds such as music, crowd noise, or interfering speech can degrade speech perception in full-duplex spoken dialogue models, leading to irrelevant responses. AV-STE uses visual lip cue to enhance corrupted speech token and improve response coherence under noisy conditions, without retraining the underlying speech LLM. Since current full-duplex dialogue models depend primarily on audio input, their performance can degrade substantially under noisy conditions. In human communication, visual cues such as lip movements provide complementary information when speech is ambiguous or corrupted. Prior work in audio-visual speech recognition (AVSR) (Ma et al., 2023; Yeo et al., 2025) has demonstrated the benefit of visual information for robust speech understanding. However, AVSR systems primarily predict text, whereas native speech-to-speech dialogue models operate directly on speech-token representations and must process user input continuously with low latency. Using AVSR would introduce an additional speech-to-text and text-to-speech-token conversion stage. We therefore focus on directly enhancing corrupted speech tokens using visual information. AV-Dialog (Chen et al., 2025b) further demonstrates that visual information can improve spoken dialogue robustness by grounding the dialogue model in audio-visual input through large-scale training on paired audio-visual conversational data. However, such training is computationally expensive and data-intensive. Moreover, adapting a pretrained dialogue model to a new modality requires careful training to retain its existing language and conversational capabilities. This motivates a practical question: can visual information improve dialogue robustness without retraining the speech LLM itself? To this end, we propose AV-STE, a streaming audio-visual semantic token enhancement framework for robust full-duplex dialogue. Rather than adapting the dialogue model to process visual input, AV-STE uses a streaming audio-visual encoder and noise-adaptive fusion to restore corrupted semantic speech tokens from noisy audio and lip video before they reach the dialogue model. The enhanced tokens are streamed directly into the pretrained speech LLM, which remains entirely frozen, avoiding costly audio-visual retraining and enabling plug-and-play integration with existing speech-to-speech dialogue systems. Experiments on LRS3 and Seamless Interaction show that AV-STE substantially improves semantic-token recovery under non-speech noise and speaker interference, including same-dataset multi-speaker interference and out-of-domain conditions. When integrated with frozen Moshi, AV-STE consistently improves response coherence across noisy SNR levels while preserving turn-taking behavior. These results demonstrate that visual speech cues can improve the robustness of full-duplex spoken dialogue by recovering corrupted semantic speech tokens without retraining the underlying speech LLM. 2 Related Work 2.1 Full-Duplex Spoken Dialogue Models Recent speech LLMs can be broadly categorized as half- or full-duplex depending on whether they can listen and speak simultaneously. Many speech LLMs operate in a half-duplex manner (Zeng et al., 2024; Xu et al., 2025; Ding et al., 2025), alternating between listening and speaking. While effective for turn-based interaction, this limits natural conversational behaviors such as interruption and backchanneling. Full-duplex speech LLMs (Défossez et al., 2024; Veluri et al., 2024; Zhang et al., 2025; Wang et al., 2024; Chen et al., 2025a; Yu et al., 2025) instead process streaming user speech while generating responses, enabling more natural turn-taking. Some systems use modular listening and speaking components, such as Freeze-Omni (Wang et al., 2024) and MinMo (Chen et al., 2025a), whereas models such as Moshi (Défossez et al., 2024) operate directly over speech-token streams within a unified architecture. Our work builds on this latter setting and improves robustness to acoustic corruption while keeping the pretrained speech LLM frozen. 2.2 Audio-Visual Spoken Dialogue Models Recent work has also incorporated visual information directly into spoken dialogue systems. MoshiVis (Royer et al., 2025) extends Moshi with visual input for understanding and discussing image content. In contrast, AV-STE uses continuous lip video as a complementary signal for recovering speech information under acoustic corruption. AV-Dialog (Chen et al., 2025b) shares a similar motivation of incorporating user face video into dialogue systems. However, it incorporates audio-visual grounding by training the dialogue model on paired audio-visual conversational data. AV-STE instead introduces visual information at the speech-token level, restoring corrupted semantic tokens before they reach the dialogue model. This allows the underlying speech LLM to remain frozen and enables AV-STE to serve as a plug-and-play robustness module for pretrained speech-to-speech dialogue systems. Figure 2: Overview of AV-STE. (a) Given noisy speech and the corresponding lip video, AV-STE first extracts visual speech features using a Streaming Audio-Visual Encoder. The Noise-Adaptive Fusion module then combines these features with speech-tokenizer outputs through soft token cross-attention and noise-adaptive modulation to predict clean semantic tokens. (b) The enhanced semantic tokens are streamed into a frozen full-duplex speech LLM together with the acoustic tokens from the speech tokenizer, enabling more robust response generation under noisy conditions. (c) A bounded-lookahead mask is used to preserve low-latency streaming inference. 2.3 Speech Token Representations Recent full-duplex speech dialogue systems operate directly on discrete speech token representations rather than text. Discrete speech representations generally capture different levels of speech information. Semantic tokens derived from self-supervised speech models such as HuBERT (Hsu et al., 2021) emphasize linguistic content, while neural audio codecs such as EnCodec (Défossez et al., 2022) and SoundStream (Zeghidour et al., 2021) encode acoustic information needed for waveform reconstruction. More recent tokenizers, including SpeechTokenizer (Zhang et al., 2024) and Mimi (Défossez et al., 2024), combine semantic and acoustic information within a multi-codebook representation. In Mimi, the first codebook is semantically distilled, while the remaining codebooks primarily encode acoustic information. AV-STE specifically enhances this semantic-token stream while retaining the acoustic-token streams unchanged, allowing the enhanced representation to be consumed directly by Moshi. Although we evaluate AV-STE with Mimi and Moshi, the token-level formulation could be adapted to other speech models that use semantic-token representations. 3 Audio-Visual Speech Token Enhancer (AV-STE) 3.1 Overview We propose AV-STE, a lightweight audio-visual speech token enhancement module that improves the robustness of frozen full-duplex speech dialogue models under noisy and interfering-speaker conditions. In this work, we focus on semantic speech tokens because they primarily encode linguistic content, which is most critical for speech understanding and coherent response generation. This choice is also motivated by the nature of visual speech, where lip movements provide strong phonetic cues but limited information about fine-grained acoustic attributes such as timbre, prosody, and background noise. We therefore focus on enhancing the semantically distilled token stream while retaining the original acoustic-token streams unchanged. Given noisy semantic speech tokens noisyz^noisy and synchronized facial video v of the target speaker, AV-STE predicts clean semantic speech tokens y before they are fed into a frozen speech LLM: ∼pθ(∣noisy,).y p_θ (y ^noisy,v ). (1) The proposed framework consists of two main components: (1) a Streaming Audio-Visual Encoder, which extracts visual speech features from noisy audio and synchronized lip video, and (2) a Noise-Adaptive Fusion module, which uses cross-attention to combine visual speech features with noisy tokenizer representations and adaptively adjusts their contribution according to tokenizer uncertainty. The resulting enhanced semantic tokens are then passed directly to the frozen speech LLM, enabling robust audio-visual input processing without modifying the underlying dialogue model. The full architecture of AV-STE is illustrated in Figure 2. 3.2 Streaming AV Encoder We initialize the Streaming AV Encoder from AV-HuBERT (Shi et al., 2022), a self-supervised audio-visual speech model that achieves strong performance on audio-visual speech recognition. AV-HuBERT consists of modality-specific audio and visual encoders followed by a shared Transformer encoder, which extracts synchronized audio-visual features from input speech waveforms and facial video frames at 25 Hz. Since the original AV-HuBERT is bidirectional and designed for offline inference, we modify the backbone to support low-latency streaming inference. First, we replace all temporal convolutions in the video frontend with causal convolutions using left-only temporal padding, ensuring that each visual feature depends only on current and past frames. Second, we replace bidirectional self-attention in every Transformer layer with causal attention using a bounded-lookahead mask (Moritz et al., 2020; Shi et al., 2020), as illustrated in Figure 2(c). Specifically, we add the following mask to the pre-softmax attention logits: Mij=0,j≤i+L,−∞,j>i+L,M_ij= cases0,&j≤ i+L,\\ -∞,&j>i+L, cases (2) where L denotes the allowed future lookahead in frames. When L=0L=0, the model is strictly causal; when L>0L>0, the model accesses a bounded number of future frames while preserving streaming inference through fixed-latency future context. A small lookahead improves temporal audio-visual alignment while maintaining low streaming latency. We use L=4L=4 (160 ms at 25 Hz) in our main experiments and further ablate different lookahead sizes in Figure 4. To match the target speech token rate used by the downstream speech LLM, we additionally insert a strided causal convolution after the transformer encoder to downsample the feature sequence while preserving the streaming constraint. 3.3 Noise-Adaptive Fusion The streaming AV encoder produces visual speech features tavh^av_t that are robust to background noise and interfering speakers. However, visual speech cues can be phoneme-ambiguous, since multiple phonemes may correspond to identical mouth shape (Yeo et al., 2024), making AV encoder predictions less reliable than the original speech tokens under clean and low-noise conditions. In contrast, the speech tokenizer provides fine-grained speech tokens learned from large-scale speech data, but its predictions may degrade under severe noise. To exploit their complementary strengths, we introduce a noise-adaptive fusion module that incorporates tokenizer predictions into the AV representation through cross-attention and adaptively modulates their contribution based on tokenizer uncertainty. Soft Token Cross Attention. Directly attending to hard token embeddings can be unreliable under noisy or interfering-speaker conditions, where the tokenizer distribution is often ambiguous. Hard arg⁡max selection discards this uncertainty and may provide misleading token-level evidence to the cross-attention module. We therefore attend to the soft tokenizer distribution instead of discrete token embeddings. Specifically, we compute a distribution over tokenizer logits and use it to form a soft token representation: t=softmax⁡(tτ),t=tproj,p_t=softmax\! ( l_tτ ), _t=p_t\,E\,W_proj, (3) where t∈ℝVl_t ^V denotes the tokenizer logits, τ is a temperature hyperparameter, ∈ℝV×deE ^V× d_e is a token embedding table learned jointly with the fusion module, and proj∈ℝde×dW_proj ^d_e× d projects the soft embedding to the model dimension. We then fuse te_t with the audio-visual features through cross-attention (Vaswani et al., 2017): ^t=CrossAttn(tav,1:T,1:T), c_t=CrossAttn\! (h^av_t,\;e_1:T,\;e_1:T ), (4) t=LayerNorm⁡(tav+^t).c_t=LayerNorm\! (h^av_t+ c_t ). (5) We use AV features tavh^av_t as queries and soft token embeddings te_t as keys/values, allowing noise-robust visual cues to guide retrieval from the corrupted speech-token representation. The reverse direction performs worse under noise (Appendix A). To support streaming inference, we apply a bounded-lookahead mask (Figure 2c) that restricts each query position t to attend only to past key-value positions t′≤t+Lt ≤ t+L, consistent with the backbone lookahead L. Noise-Adaptive Modulation. Although cross-attention incorporates the speech-token representation into the AV representation, the speech-token representation should not contribute equally under all noise conditions. In clean or low-noise speech, the tokenizer predictions are usually reliable. However, in severe noise or interfering speech, the tokenizer distribution may contain incorrect acoustic information. We therefore introduce a frame-level modulation λt∈[0,1] _t∈[0,1] that adaptively controls how much weight the final representation places on the speech-token representation te_t versus the cross-attended AV representation tc_t depending on the noise level. We use the normalized entropy of the tokenizer distribution as a reliability prior, since entropy measures uncertainty over the full token distribution (Laptev and Ginsburg, 2023): H^t H_t =−∑v=1Vpt,vlogpt,vlog⁡V, = - _v=1^Vp_t,v p_t,v V, (6) λ^tent λ^ent_t =1−H^t. =1- H_t. where H^t∈[0,1] H_t∈[0,1] denotes the normalized entropy. A lower entropy indicates a sharper tokenizer distribution and thus higher estimated reliability, while a higher entropy indicates greater uncertainty. Because entropy alone may not capture all reliability cues, we refine this prior with a learnable modulation gate conditioned on both the speech-token and AV representations: λt=σ(wλ^tent+MLP([t∥tav])), _t=σ\! (w λ^ent_t+MLP\! ([e_t\,\|\,h^av_t] ) ), (7) where w is a learnable scalar initialized to 11. The final layer of the MLP is zero-initialized, so the gate initially follows the entropy-based prior and gradually learns residual corrections during training. The final fused representation is computed as ~t=λtt+(1−λt)t. h_t= _te_t+(1- _t)c_t. (8) Rather than following a fixed rule, λt _t is learned jointly with the rest of the model, allowing it to adaptively weight the speech-token and AV representations based on the noise condition. Finally, a linear projection maps the fused representation ~t h_t to logits over the semantic token vocabulary, and the full AV-STE model is trained end-to-end using frame-wise cross-entropy against clean target speech tokens. 4 Experimental Setup 4.1 Datasets and Preprocessing LRS3. (Afouras et al., 2018) contains approximately 433 hours of audio-visual speech, with video at 25 fps and audio at 16 kHz. We use LRS3 as the target-speech dataset for training and in-domain evaluation. Following the official LRS3 preprocessing pipeline, we crop the mouth region from each face video and convert it into 96×96 grayscale frames. AudioSet. (Gemmeke et al., 2017) is a large-scale audio event dataset containing 2,084,320 human-labeled 10-second clips from YouTube videos. It covers a broad range of sounds, including human and animal sounds, musical instruments, music genres, and everyday environmental noise. We sample environmental sounds as non-speech noise and conversational speech as cross-dataset speaker interference. Seamless Interaction. (Agrawal et al., 2025) is a large-scale dyadic interaction video corpus, used here exclusively for out-of-domain evaluation with no fine-tuning. Videos are recorded at 1080×1920 and 30 fps with 48 kHz audio; we resample audio to 16 kHz and crop a 96×96 grayscale mouth region at 25 fps to match the LRS3 preprocessing format. No Seamless Interaction data is used in AV-STE training. Table 1: Semantic-token accuracy (%) across in-domain, same-dataset speaker-interference, and out-of-domain settings at five SNR levels. AV-STE is first trained on LRS3 with AudioSet non-speech noise and speaker interference, then further trained with LRS3 same-dataset speaker interference before evaluation on the same-dataset speaker interference and out-of-domain settings. Target Model Noise / Interference Clean 10 dB 5 dB 0 dB -5 dB -10 dB Avg. In-domain LRS3 LRS3 Mimi Non-speech (AudioSet) 100.00 59.11 41.30 22.62 10.33 4.81 27.63 LRS3 Mimi + AV-STE Non-speech (AudioSet) 76.78 73.48 71.87 69.48 65.34 59.92 68.02 LRS3 Mimi Speaker (AudioSet) 100.00 54.81 37.61 22.99 12.25 7.19 26.97 LRS3 Mimi + AV-STE Speaker (AudioSet) 76.78 73.17 71.76 69.20 65.82 60.52 68.09 Same-dataset LRS3 speaker interference LRS3 Mimi Speaker (LRS3) 100.00 38.43 15.83 5.26 2.22 1.31 12.61 LRS3 Mimi + AV-STE Speaker (LRS3) 76.78 72.46 70.27 66.94 62.17 56.18 65.60 Out-of-domain Seamless Interaction Seamless Mimi Non-speech (AudioSet) 100.00 36.80 22.49 12.35 5.93 3.77 16.27 Seamless Mimi + AV-STE Non-speech (AudioSet) 43.79 41.89 40.12 38.11 34.42 29.28 36.76 Seamless Mimi Speaker (Seamless) 100.00 35.21 21.13 12.14 6.92 3.62 15.80 Seamless Mimi + AV-STE Speaker (Seamless) 43.79 38.66 37.29 35.97 33.42 29.48 34.96 4.2 Training and Evaluation Data Synthesis Initial training. For initial AV-STE training, each LRS3 utterance is sampled as clean speech (20%), mixed with AudioSet non-speech noise (40%), or mixed with AudioSet conversational speech (40%). For noisy samples, the SNR is uniformly sampled from -10 to 10dB. Same-dataset speaker interference. To reduce shortcuts from dataset-specific characteristics, following Chen et al. (2025b), we further train the initial checkpoint by replacing AudioSet conversational speech with interference from 1–4 randomly selected LRS3 speakers. Interfering speakers are randomly sampled and mixed with the target utterance on the fly during training. We retain the same 20/40/40 sampling ratio and SNR range. This resulting checkpoint is used for all subsequent same-dataset and out-of-domain evaluations. Out-of-domain evaluation. For Seamless Interaction, we use AudioSet for non-speech noise and 1-4 randomly selected Seamless speakers for speaker interference. No Seamless Interaction data is used for training. Single-turn dialogue evaluation. Following Full-Duplex-Bench (Lin et al., 2025), we evaluate individual utterances rather than multi-turn dialogues. We select 125 utterances each from LRS3 and Seamless Interaction, each 3-10s long. The Seamless dialog set is also used for semantic-token accuracy evaluation. 4.3 Training Configuration We optimize AV-STE with Adam (β1=0.9 _1=0.9, β2=0.98 _2=0.98) using a tri-stage learning-rate schedule with 4,000 warmup steps followed by 35,000 decay steps to 5%5\% of the peak learning rate of 10−410^-4. Gradients are clipped to a maximum norm of 5.0. Training runs for up to 6 epochs (approximately 73 GPU hours) with FP16 mixed precision on a single A6000 GPU. We select the checkpoint with the lowest validation loss. 4.4 Model Configuration We use Moshi (Défossez et al., 2024) as the frozen full-duplex spoken dialogue model and its Mimi tokenizer as the speech-token representation. Mimi uses a hybrid semantic-acoustic representation: its first codebook is semantically distilled, while the remaining codebooks primarily encode acoustic information. AV-STE enhances only the first semantic-token stream; the acoustic-token streams remain unchanged and are passed with the enhanced tokens to Moshi. We use AV-HuBERT Large (Shi et al., 2022) as the audio-visual encoder. Our main streaming configuration uses a 4-frame lookahead (L=4L=4), corresponding to 160 ms at 25 Hz. The soft-token cross-attention module uses 4 heads, 256-dimensional token embeddings, dropout of 0.1, and temperature τ=1.0τ=1.0. A linear projection maps the 1024-dimensional AV-HuBERT features to Mimi’s 2,048-token semantic vocabulary. The final AV-STE model contains 335M parameters. 5 Experimental Results and Analysis 5.1 Semantic Speech Token Enhancement Evaluation In all downstream dialogue experiments, Moshi is kept frozen. AV-STE replaces only the semantic-token stream from Mimi’s semantically distilled first codebook, while the remaining acoustic-token streams are retained unchanged. Since Moshi directly consumes these enhanced semantic tokens, we use semantic-token accuracy against clean Mimi tokens as our primary token-level metric. We additionally report reconstructed-speech WER as a secondary diagnostic metric. Evaluation protocol. Given noisy audio and the corresponding lip video, AV-STE predicts the enhanced semantic-token sequence. Semantic-token accuracy is computed against Mimi tokens extracted from the corresponding clean speech. For reconstructed-speech WER, the enhanced semantic tokens are combined with the acoustic codebooks from the noisy input and decoded into a waveform using the Mimi decoder. We transcribe the reconstructed waveform with OpenAI Whisper and compute WER against the clean ground-truth transcription. Table 2: Ablation of input modalities and AV-STE components on LRS3. Results are average semantic-token accuracy (%) over five SNR levels under AudioSet non-speech noise and speaker interference. S-AVH denotes our streaming AV-HuBERT implementation, NAM is our Noise Adaptive Modulation, and CA is cross-attention. Full per-SNR results are in Appendix B. Model / Variant Clean Non-Speech Speaker Mimi 100.00 27.63 26.97 S-AVH (A) 75.65 59.25 58.15 S-AVH (V) 32.41 35.13 35.27 S-AVH (A+V) 76.47 67.48 67.77 + Hard-CA 75.12 66.37 66.60 + Soft-CA 76.11 67.64 67.86 + Soft-CA+NAM (AV-STE) 76.78 68.02 68.09 Semantic-token recovery. Table 1 shows that AV-STE substantially improves semantic-token recovery under both non-speech noise and speaker interference. On LRS3, average accuracy increases from 27.63% to 68.02% under non-speech noise and from 26.97% to 68.09% under AudioSet speaker interference. The gains persist across all evaluated SNRs and become especially pronounced under severe corruption; at −10-10 dB non-speech noise, accuracy increases from 4.81% to 59.92%. This shows that visual cues provide complementary information when the acoustic representation is corrupted. Same-dataset speaker interference. Speaker interference is substantially more challenging when the target and interferers are drawn from the same dataset. Compared with AudioSet speaker interference, Mimi performs worse at every SNR. This may reflect both the greater similarity between target and interfering speech and the 1-4 competing speakers used in this setting. After further training on LRS3 speaker interference, AV-STE performs robustly under same-dataset multi-speaker interference, improving Mimi’s average accuracy from 12.61% to 65.60%. The contrast is especially pronounced at −10-10 dB, where Mimi drops to 1.31% accuracy while AV-STE maintains 56.18%. Out-of-domain generalization. On the 125-clip Seamless dialogue evaluation set described in Section 4, AV-STE improves average noisy accuracy from 16.27% to 36.76% under non-speech noise and from 15.80% to 34.96% under same-dataset speaker interference. However, clean accuracy decreases from 76.78% on LRS3 to 43.79% on Seamless, indicating sensitivity to domain shift despite consistent gains under noisy conditions. Ablation and modality. Table 2 shows that audio-visual S-AVH outperforms audio-only S-AVH under both noise types, confirming the benefit of visual cues. Among the fusion variants, Soft-CA+NAM achieves the highest average accuracy and is therefore used as the final AV-STE configuration. Figure 3 further shows that the benefit of audio-visual fusion increases as SNR decreases. Figure 3: Semantic token accuracy (%) across SNR levels, averaged over non-speech noise and speaker interference. The advantage of audio-visual fusion increases as SNR decreases. (Appendix B for details). Figure 4: Semantic token accuracy at 0 dB across lookahead frames, averaged over audio set’s speaker interference and non-speech noise. Accuracy peaks at 4 frames (160 ms), offering the best accuracy–latency trade-off. Effect of lookahead frames. Increasing lookahead provides more future context but also adds streaming latency. As shown in Figure 4, accuracy improves up to L=4L=4 and shows diminishing returns beyond this point. We therefore use L=4L=4 (160 ms) as the best accuracy–latency trade-off. Reconstructed WER. Appendix C reports WER as a secondary diagnostic, since Moshi directly consumes enhanced semantic tokens rather than reconstructed speech. Table 3: Comparison of turn-taking and response semantic coherence under clean and noisy conditions. TOR denotes take-over rate, latency measures response delay in seconds, and GPT denotes GPT-based response coherence. GPT scores are underlined to highlight coherence comparisons, and the better GPT score in each condition is bolded. Noise Type Non-Speech (AudioSet) Speaker Interference (AudioSet) Models Moshi Moshi+AV-STE Moshi Moshi+AV-STE Metrics TOR ↑ Latency (s) ↓ GPT ↑ TOR ↑ Latency (s) ↓ GPT ↑ TOR ↑ Latency (s) ↓ GPT ↑ TOR ↑ Latency (s) ↓ GPT ↑ Clean 0.520 0.720 2.34 0.608 0.534 2.11 0.520 0.720 2.34 0.608 0.534 2.11 10 dB 0.560 1.056 1.87 0.488 0.553 2.15 0.488 0.730 2.00 0.496 0.756 2.03 5 dB 0.496 0.777 2.00 0.520 0.959 2.12 0.528 0.872 1.98 0.568 0.614 2.21 0 dB 0.512 0.926 1.83 0.496 1.053 1.97 0.624 0.793 1.62 0.528 0.795 1.85 -5 dB 0.624 0.660 1.33 0.552 1.125 1.86 0.600 0.778 1.39 0.624 0.873 1.90 -10 dB 0.544 0.991 1.25 0.520 1.008 2.00 0.656 0.652 1.22 0.568 0.660 1.94 Table 4: Qualitative example of AV-STE recovering response coherence under −10-10 dB non-speech noise. User: “My mother was on the line to tell me that doctors had found a lump in his stomach, terminal cancer .. ” Moshi: “Oh, I’m so glad you asked. I’m a big fan of the London Underground. I’ve been using it for years.” Moshi + AV-STE: “That’s not good ..” 5.2 Full-Duplex Turn-Taking and Semantic Coherence Evaluation We next examine whether improved semantic-token recovery translates into more robust full-duplex dialogue generation. Specifically, we evaluate whether AV-STE improves response coherence under acoustic corruption without disrupting the turn-taking behavior of the underlying speech LLM. Evaluation protocol. AV-STE takes noisy audio, the corresponding lip video, and noisy semantic-token logits as input and predicts enhanced semantic tokens. These are combined with the unchanged acoustic-token streams from the noisy input and streamed into frozen Moshi (Défossez et al., 2024) for response generation, which serves as our primary backbone and baseline. For same-dataset speaker interference, we additionally compare with PersonaPlex (Roy et al., 2026) and PASE (Rong et al., 2025) + Moshi. PASE is a recent generative speech-enhancement model that leverages WavLM phonological representations to recover clean speech from noisy input. PASE enhances the complete noisy utterance before it is passed to Moshi, providing a non-streaming audio-only enhancement baseline against AV-STE’s streaming audio-visual token enhancement. Metrics. Following Lin et al. (2025), we evaluate the generated responses using three metrics (see Appendix D for details): • Takeover Rate (TOR): The proportion of turns in which the model produces non-silent, non-backchannel speech. A higher TOR indicates more successful turn-taking behavior. • Latency: The average delay between the end of the user’s utterance and the start of the model’s response, measured in seconds. Lower latency indicates smoother turn-taking. • GPT: A GPT-based semantic coherence score that measures the relevance of the model’s response to the user’s utterance on a scale from 1 to 5. Higher scores indicate better response coherence. In-domain dialogue robustness. AV-STE substantially improves response coherence as acoustic corruption becomes more severe, while largely preserving Moshi’s turn-taking behavior. Table 3 shows consistent coherence gains across all noisy SNR levels under both non-speech noise and speaker interference. At -10 dB, for example, GPT coherence increases from 1.25 to 2.00 under non-speech noise and from 1.22 to 1.94 under speaker interference. In contrast, TOR and latency remain close to those of Moshi, indicating that restoring the semantic-token stream improves response relevance without altering the full-duplex interaction behavior. Table 5: GPT-4o response coherence (↑ ) under same-dataset LRS3 speaker interference and out-of-domain Seamless Interaction. Avg. is computed over the five noisy SNR levels. †PASE is an offline, non-causal generative speech enhancement (SE) model, making it incompatible with streaming full-duplex interaction and is included only as a non-streaming reference. Target Noise / Interference Model Clean 10 dB 5 dB 0 dB -5 dB -10 dB Avg. LRS3 Non-speech (AudioSet) PersonaPlex 1.64 1.51 1.46 1.33 1.12 1.11 1.31 Moshi 2.34 1.87 2.00 1.83 1.33 1.25 1.66 SE (PASE) † + Moshi 2.23 2.03 2.34 2.21 2.34 1.61 2.11 AV-STE + Moshi 2.11 2.15 2.12 1.97 1.86 2.00 2.02 Speaker (LRS3) PersonaPlex 1.64 1.45 1.13 1.02 1.03 1.07 1.17 Moshi 2.34 1.94 1.61 1.25 1.14 1.15 1.42 SE (PASE) † + Moshi 2.23 2.15 2.00 1.77 1.29 1.12 1.67 AV-STE + Moshi 2.11 2.00 2.12 1.85 1.86 1.70 1.91 Seamless Non-speech (AudioSet) Moshi 2.38 2.13 1.68 1.50 1.27 1.30 1.58 AV-STE + Moshi 2.70 2.27 2.36 2.25 1.78 1.84 2.10 Speaker (Seamless) Moshi 2.38 1.98 1.64 1.44 1.24 1.18 1.50 AV-STE + Moshi 2.70 2.36 2.34 1.88 1.91 1.64 2.03 Same-dataset speaker interference. Visual grounding is particularly beneficial when competing speech becomes difficult to distinguish acoustically. Under same-dataset LRS3 interference, AV-STE+Moshi achieves the highest average coherence score of 1.91, compared with 1.42 for Moshi, 1.17 for PersonaPlex, and 1.67 for SE (PASE)+Moshi (Table 5). Although PASE performs best at 10 dB, AV-STE performs best from 5 to -10 dB, suggesting that visual cues provide complementary target-speech information as acoustic interference becomes more severe. Out-of-domain generalization. For Seamless Interaction, we evaluate clean speech, AudioSet non-speech noise, and same-dataset speaker interference by mixing each target with 1–4 other Seamless speakers. No Seamless Interaction data is used for training. In Seamless Interaction, AV-STE improves average noisy coherence from 1.58 to 2.10 under non-speech noise and from 1.50 to 2.03 under same-dataset speaker interference. Thus, despite the token-level domain sensitivity observed on Seamless Interaction, visual semantic enhancement continues to improve downstream response coherence, suggesting that the recovered representation remains useful to the frozen speech LLM under domain shift. Qualitative analysis. Table 4 shows a qualitative example where AV-STE recovers response coherence at -10 dB SNR. See Appendix E for more qualitative samples. 6 Conclusion We presented AV-STE, a streaming audio-visual module that improves the robustness of full-duplex spoken dialogue without retraining the underlying speech LLM. AV-STE uses visual speech cues to recover corrupted semantic tokens before they are consumed by frozen Moshi, while leaving the remaining acoustic token streams unchanged. Across non-speech noise, speaker interference, and same-dataset multi-speaker interference, AV-STE substantially improves semantic-token recovery, with particularly large gains under severe acoustic corruption. The improvements also transfer to the out-of-domain Seamless Interaction dataset. Downstream, the enhanced tokens improve spoken-response coherence across noisy conditions without degrading turn-taking behavior. These results demonstrate the value of visual speech information as a complementary signal for robust full-duplex spoken dialogue systems. 7 Limitations AV-STE currently assumes reliable visual input, and its robustness to visual corruption, occlusion, or missing video remains unexplored. Although AV-STE improves noisy semantic-token recovery on the out-of-domain Seamless Interaction dataset, its lower clean-token accuracy indicates sensitivity to domain shift. Finally, AV-STE introduces a small representation mismatch on clean speech, as its predicted semantic tokens do not exactly reproduce native Mimi tokens. Future work can improve cross-domain robustness using more diverse audio-visual conversational data and jointly consider acoustic and visual corruptions. Acknowledgments This work was supported by the National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIT) (No. RS-2022-NR070162) and Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government(MSIT) (No. RS-2020-I200004, Development of Previsional Intelligence based on Long-Term Visual Memory Network) References Afouras et al. (2018) T. Afouras, J. S. Chung, and A. Zisserman LRS3-TED: a large-scale dataset for visual speech recognition. CoRR abs/1809.00496. External Links: Link, 1809.00496 Cited by: §4.1. Agrawal et al. (2025) V. Agrawal, A. Akinyemi, K. Alvero, M. Behrooz, J. Buffalini, F. M. Carlucci, J. Chen, J. Chen, Z. Chen, S. Cheng, P. Chowdary, J. Chuang, A. D’Avirro, J. Daly, N. Dong, M. Duppenthaler, C. Gao, J. Girard, M. Gleize, S. Gomez, H. Gong, S. Govindarajan, B. Han, S. He, D. Hernandez, Y. Hristov, R. Huang, H. Inaguma, S. Jain, R. Janardhan, Q. Jia, C. Klaiber, D. Kovachev, M. Kumar, H. Li, Y. Li, P. Litvin, W. Liu, G. Ma, J. Ma, M. Ma, X. Ma, L. Mantovani, S. Miglani, S. Mohan, L. Morency, E. Ng, K. Ng, T. A. Nguyen, A. Oberai, B. Peloquin, J. Pino, J. Popovic, O. Poursaeed, F. Prada, A. Rakotoarison, R. Ranjan, A. Richard, C. Ropers, S. Saleem, V. Sharma, A. Shcherbyna, J. Shen, J. Shen, A. Stathopoulos, A. Sun, P. Tomasello, T. Tran, A. Turkatenko, B. Wan, C. Wang, J. Wang, M. Williamson, C. Wood, T. Xiang, Y. Yang, J. Yao, C. Zhang, J. Zhang, X. Zhang, J. Zheng, P. Zhyzheria, J. Zikes, and M. Zollhoefer Seamless interaction: dyadic audiovisual motion modeling and large-scale dataset. External Links: 2506.22554, Link Cited by: §4.1. Chen et al. (2025a) Q. Chen, Y. Chen, Y. Chen, M. Chen, Y. Chen, C. Deng, Z. Du, R. Gao, C. Gao, Z. Gao, et al. Minmo: a multimodal large language model for seamless voice interaction. arXiv preprint arXiv:2501.06282. External Links: Link Cited by: §2.1. Chen et al. (2025b) T. Chen, B. Veluri, H. Gong, and S. Gollakota AV-dialog: spoken dialogue models with audio-visual input. External Links: 2511.11124, Link Cited by: §1, §2.2, §4.2. Défossez et al. (2022) A. Défossez, J. Copet, G. Synnaeve, and Y. Adi High fidelity neural audio compression. External Links: 2210.13438, Link Cited by: §2.3. Défossez et al. (2024) A. Défossez, L. Mazaré, M. Orsini, A. Royer, P. Pérez, H. Jégou, E. Grave, and N. Zeghidour Moshi: a speech-text foundation model for real-time dialogue. External Links: 2410.00037, Link Cited by: §2.1, §2.3, §4.4, §5.2. Ding et al. (2025) D. Ding, Z. Ju, Y. Leng, S. Liu, T. Liu, Z. Shang, K. Shen, W. Song, X. Tan, H. Tang, et al. Kimi-audio technical report. arXiv preprint arXiv:2504.18425. External Links: Link Cited by: §2.1. Gemmeke et al. (2017) J. F. Gemmeke, D. P. W. Ellis, D. Freedman, A. Jansen, W. Lawrence, R. C. Moore, M. Plakal, and M. Ritter Audio set: an ontology and human-labeled dataset for audio events. In 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), p. 776–780. Cited by: §4.1. Hsu et al. (2021) W. Hsu, B. Bolte, Y. H. Tsai, K. Lakhotia, R. Salakhutdinov, and A. Mohamed HuBERT: self-supervised speech representation learning by masked prediction of hidden units. CoRR abs/2106.07447. External Links: Link, 2106.07447 Cited by: §2.3. Laptev and Ginsburg (2023) A. Laptev and B. Ginsburg Fast entropy-based methods of word-level confidence estimation for end-to-end automatic speech recognition. In 2022 IEEE spoken language technology workshop (SLT), p. 152–159. Cited by: §3.3. Lin et al. (2025) G. Lin, J. Lian, T. Li, Q. Wang, G. Anumanchipalli, A. H. Liu, and H. Lee Full-duplex-bench: a benchmark to evaluate full-duplex spoken dialogue models on turn-taking capabilities. External Links: 2503.04721, Link Cited by: Appendix D, §4.2, §5.2. Ma et al. (2023) P. Ma, A. Haliassos, A. Fernandez-Lopez, H. Chen, S. Petridis, and M. Pantic Auto-avsr: audio-visual speech recognition with automatic labels. In ICASSP 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), p. 1–5. External Links: Link, Document Cited by: §1. Moritz et al. (2020) N. Moritz, T. Hori, and J. L. Roux Streaming automatic speech recognition with the transformer model. CoRR abs/2001.02674. External Links: Link, 2001.02674 Cited by: §3.2. OpenAI et al. (2024) OpenAI, :, A. Hurst, A. Lerer, A. P. Goucher, A. Perelman, A. Ramesh, A. Clark, A. Ostrow, A. Welihinda, A. Hayes, A. Radford, A. Mądry, A. Baker-Whitcomb, A. Beutel, A. Borzunov, A. Carney, A. Chow, A. Kirillov, A. Nichol, A. Paino, A. Renzin, A. T. Passos, A. Kirillov, A. Christakis, A. Conneau, A. Kamali, A. Jabri, A. Moyer, A. Tam, A. Crookes, A. Tootoochian, A. Tootoonchian, A. Kumar, A. Vallone, A. Karpathy, A. Braunstein, A. Cann, A. Codispoti, A. Galu, A. Kondrich, A. Tulloch, A. Mishchenko, A. Baek, A. Jiang, A. Pelisse, A. Woodford, A. Gosalia, A. Dhar, A. Pantuliano, A. Nayak, A. Oliver, B. Zoph, B. Ghorbani, B. Leimberger, B. Rossen, B. Sokolowsky, B. Wang, B. Zweig, B. Hoover, B. Samic, B. McGrew, B. Spero, B. Giertler, B. Cheng, B. Lightcap, B. Walkin, B. Quinn, B. Guarraci, B. Hsu, B. Kellogg, B. Eastman, C. Lugaresi, C. Wainwright, C. Bassin, C. Hudson, C. Chu, C. Nelson, C. Li, C. J. Shern, C. Conger, C. Barette, C. Voss, C. Ding, C. Lu, C. Zhang, C. Beaumont, C. Hallacy, C. Koch, C. Gibson, C. Kim, C. Choi, C. McLeavey, C. Hesse, C. Fischer, C. Winter, C. Czarnecki, C. Jarvis, C. Wei, C. Koumouzelis, D. Sherburn, D. Kappler, D. Levin, D. Levy, D. Carr, D. Farhi, D. Mely, D. Robinson, D. Sasaki, D. Jin, D. Valladares, D. Tsipras, D. Li, D. P. Nguyen, D. Findlay, E. Oiwoh, E. Wong, E. Asdar, E. Proehl, E. Yang, E. Antonow, E. Kramer, E. Peterson, E. Sigler, E. Wallace, E. Brevdo, E. Mays, F. Khorasani, F. P. Such, F. Raso, F. Zhang, F. von Lohmann, F. Sulit, G. Goh, G. Oden, G. Salmon, G. Starace, G. Brockman, H. Salman, H. Bao, H. Hu, H. Wong, H. Wang, H. Schmidt, H. Whitney, H. Jun, H. Kirchner, H. P. de Oliveira Pinto, H. Ren, H. Chang, H. W. Chung, I. Kivlichan, I. O’Connell, I. O’Connell, I. Osband, I. Silber, I. Sohl, I. Okuyucu, I. Lan, I. Kostrikov, I. Sutskever, I. Kanitscheider, I. Gulrajani, J. Coxon, J. Menick, J. Pachocki, J. Aung, J. Betker, J. Crooks, J. Lennon, J. Kiros, J. Leike, J. Park, J. Kwon, J. Phang, J. Teplitz, J. Wei, J. Wolfe, J. Chen, J. Harris, J. Varavva, J. G. Lee, J. Shieh, J. Lin, J. Yu, J. Weng, J. Tang, J. Yu, J. Jang, J. Q. Candela, J. Beutler, J. Landers, J. Parish, J. Heidecke, J. Schulman, J. Lachman, J. McKay, J. Uesato, J. Ward, J. W. Kim, J. Huizinga, J. Sitkin, J. Kraaijeveld, J. Gross, J. Kaplan, J. Snyder, J. Achiam, J. Jiao, J. Lee, J. Zhuang, J. Harriman, K. Fricke, K. Hayashi, K. Singhal, K. Shi, K. Karthik, K. Wood, K. Rimbach, K. Hsu, K. Nguyen, K. Gu-Lemberg, K. Button, K. Liu, K. Howe, K. Muthukumar, K. Luther, L. Ahmad, L. Kai, L. Itow, L. Workman, L. Pathak, L. Chen, L. Jing, L. Guy, L. Fedus, L. Zhou, L. Mamitsuka, L. Weng, L. McCallum, L. Held, L. Ouyang, L. Feuvrier, L. Zhang, L. Kondraciuk, L. Kaiser, L. Hewitt, L. Metz, L. Doshi, M. Aflak, M. Simens, M. Boyd, M. Thompson, M. Dukhan, M. Chen, M. Gray, M. Hudnall, M. Zhang, M. Aljubeh, M. Litwin, M. Zeng, M. Johnson, M. Shetty, M. Gupta, M. Shah, M. Yatbaz, M. J. Yang, M. Zhong, M. Glaese, M. Chen, M. Janner, M. Lampe, M. Petrov, M. Wu, M. Wang, M. Fradin, M. Pokrass, M. Castro, M. O. T. de Castro, M. Pavlov, M. Brundage, M. Wang, M. Khan, M. Murati, M. Bavarian, M. Lin, M. Yesildal, N. Soto, N. Gimelshein, N. Cone, N. Staudacher, N. Summers, N. LaFontaine, N. Chowdhury, N. Ryder, N. Stathas, N. Turley, N. Tezak, N. Felix, N. Kudige, N. Keskar, N. Deutsch, N. Bundick, N. Puckett, O. Nachum, O. Okelola, O. Boiko, O. Murk, O. Jaffe, O. Watkins, O. Godement, O. Campbell-Moore, P. Chao, P. McMillan, P. Belov, P. Su, P. Bak, P. Bakkum, P. Deng, P. Dolan, P. Hoeschele, P. Welinder, P. Tillet, P. Pronin, P. Tillet, P. Dhariwal, Q. Yuan, R. Dias, R. Lim, R. Arora, R. Troll, R. Lin, R. G. Lopes, R. Puri, R. Miyara, R. Leike, R. Gaubert, R. Zamani, R. Wang, R. Donnelly, R. Honsby, R. Smith, R. Sahai, R. Ramchandani, R. Huet, R. Carmichael, R. Zellers, R. Chen, R. Chen, R. Nigmatullin, R. Cheu, S. Jain, S. Altman, S. Schoenholz, S. Toizer, S. Miserendino, S. Agarwal, S. Culver, S. Ethersmith, S. Gray, S. Grove, S. Metzger, S. Hermani, S. Jain, S. Zhao, S. Wu, S. Jomoto, S. Wu, Shuaiqi, Xia, S. Phene, S. Papay, S. Narayanan, S. Coffey, S. Lee, S. Hall, S. Balaji, T. Broda, T. Stramer, T. Xu, T. Gogineni, T. Christianson, T. Sanders, T. Patwardhan, T. Cunninghman, T. Degry, T. Dimson, T. Raoux, T. Shadwell, T. Zheng, T. Underwood, T. Markov, T. Sherbakov, T. Rubin, T. Stasi, T. Kaftan, T. Heywood, T. Peterson, T. Walters, T. Eloundou, V. Qi, V. Moeller, V. Monaco, V. Kuo, V. Fomenko, W. Chang, W. Zheng, W. Zhou, W. Manassra, W. Sheu, W. Zaremba, Y. Patil, Y. Qian, Y. Kim, Y. Cheng, Y. Zhang, Y. He, Y. Zhang, Y. Jin, Y. Dai, and Y. Malkov GPT-4o system card. External Links: 2410.21276, Link Cited by: Appendix D. Rong et al. (2025) X. Rong, Q. Hu, M. Yesilbursa, K. Wojcicki, and J. Lu PASE: leveraging the phonological prior of wavlm for low-hallucination generative speech enhancement. External Links: 2511.13300, Link Cited by: §5.2. Roy et al. (2026) R. Roy, J. Raiman, S. Lee, T. Ene, R. Kirby, S. Kim, J. Kim, and B. Catanzaro PersonaPlex: voice and role control for full duplex conversational speech models. arXiv preprint arXiv:2602.06053. External Links: Link Cited by: §5.2. Royer et al. (2025) A. Royer, M. Böhle, G. de Marmiesse, L. Mazaré, N. Zeghidour, A. Défossez, and P. Pérez Vision-speech models: teaching speech models to converse about images. External Links: 2503.15633, Link Cited by: §2.2. Sekoyan et al. (2025) M. Sekoyan, N. R. Koluguri, N. Tadevosyan, P. Zelasko, T. Bartley, N. Karpov, J. Balam, and B. Ginsburg Canary-1b-v2 & parakeet-tdt-0.6b-v3: efficient and high-performance models for multilingual asr and ast. External Links: 2509.14128, Link Cited by: Appendix D. Shi et al. (2022) B. Shi, W. Hsu, K. Lakhotia, and A. Mohamed Learning audio-visual speech representation by masked multimodal cluster prediction. External Links: 2201.02184, Link Cited by: §3.2, §4.4. Shi et al. (2020) Y. Shi, Y. Wang, C. Wu, C. Yeh, J. Chan, F. Zhang, D. Le, and M. L. Seltzer Emformer: efficient memory transformer based acoustic model for low latency streaming speech recognition. CoRR abs/2010.10759. External Links: Link, 2010.10759 Cited by: §3.2. Vaswani et al. (2017) A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin Attention is all you need. Advances in neural information processing systems 30. External Links: Link Cited by: §3.3. Veluri et al. (2024) B. Veluri, B. N. Peloquin, B. Yu, H. Gong, and S. Gollakota Beyond turn-based interfaces: synchronous llms as full-duplex dialogue agents. External Links: 2409.15594, Link Cited by: §2.1. Wang et al. (2024) X. Wang, Y. Li, C. Fu, Y. Shen, L. Xie, K. Li, X. Sun, and L. Ma Freeze-omni: a smart and low latency speech-to-speech dialogue model with frozen llm. arXiv preprint arXiv:2411.00774. External Links: Link Cited by: §2.1. Xu et al. (2025) J. Xu, Z. Guo, J. He, H. Hu, T. He, S. Bai, K. Chen, J. Wang, Y. Fan, K. Dang, et al. Qwen2. 5-omni technical report. arXiv preprint arXiv:2503.20215. External Links: Link Cited by: §2.1. Yeo et al. (2024) J. H. Yeo, S. Han, M. Kim, and Y. M. Ro Where visual speech meets language: vsp-llm framework for efficient and context-aware visual speech processing. External Links: 2402.15151, Link Cited by: §3.3. Yeo et al. (2025) J. H. Yeo, H. Rha, S. J. Park, and Y. M. Ro MMS-llama: efficient llm-based audio-visual speech recognition with minimal multimodal speech tokens. External Links: 2503.11315, Link Cited by: §1. Yu et al. (2025) W. Yu, S. Wang, X. Yang, X. Chen, X. Tian, J. Zhang, G. Sun, L. Lu, Y. Wang, and C. Zhang SALMONN-omni: a standalone speech llm without codec injection for full-duplex conversation. External Links: 2505.17060, Link Cited by: §2.1. Zeghidour et al. (2021) N. Zeghidour, A. Luebs, A. Omran, J. Skoglund, and M. Tagliasacchi SoundStream: an end-to-end neural audio codec. External Links: 2107.03312, Link Cited by: §2.3. Zeng et al. (2024) A. Zeng, Z. Du, M. Liu, K. Wang, S. Jiang, L. Zhao, Y. Dong, and J. Tang Glm-4-voice: towards intelligent and human-like end-to-end spoken chatbot. arXiv preprint arXiv:2412.02612. External Links: Link Cited by: §2.1. Zhang et al. (2025) Q. Zhang, L. Cheng, C. Deng, Q. Chen, W. Wang, S. Zheng, J. Liu, H. Yu, C. Tan, Z. Du, and S. Zhang OmniFlatten: an end-to-end GPT model for seamless voice conversation. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), W. Che, J. Nabende, E. Shutova, and M. T. Pilehvar (Eds.), Vienna, Austria, p. 14570–14580. External Links: Link, Document, ISBN 979-8-89176-251-0 Cited by: §2.1. Zhang et al. (2024) X. Zhang, D. Zhang, S. Li, Y. Zhou, and X. Qiu SpeechTokenizer: unified speech tokenizer for speech large language models. External Links: 2308.16692, Link Cited by: §2.3. Zheng et al. (2023) L. Zheng, W. Chiang, Y. Sheng, S. Zhuang, Z. Wu, Y. Zhuang, Z. Lin, Z. Li, D. Li, E. P. Xing, H. Zhang, J. E. Gonzalez, and I. Stoica Judging llm-as-a-judge with mt-bench and chatbot arena. External Links: 2306.05685, Link Cited by: Appendix D. Appendix A Cross-Attention Direction Ablation We use AV features as queries and soft speech-token embeddings as keys/values, allowing noise-robust AV cues to guide retrieval from corrupted token representations. Reversing the roles degrades WER under both noise conditions, supporting our design choice (Table 6). Table 6: WER (%) for cross-attention direction. Lower is better. Clean Non-Speech Speaker AV query (ours) 7.7 65.72 58.74 Token query 7.5 66.33 59.72 Table 7: Semantic-token accuracy (%) across noise types and SNR levels. Non-speech noise and speech interference are sampled from AudioSet. All models use a 4-frame lookahead. Mimi clean accuracy is 100% by definition because its clean tokens are used as the reference targets. Clean Non-Speech Noise (AudioSet) Speech Interference (AudioSet) Method Clean 10 dB 5 dB 0 dB −-5 dB −-10 dB Avg. 10 dB 5 dB 0 dB −-5 dB −-10 dB Avg. Mimi 100.00† 59.11 41.30 22.62 10.33 4.81 27.63 54.81 37.61 22.99 12.25 7.19 26.97 S-AV-HuBERT (A) 75.65 71.53 69.19 64.59 53.56 37.38 59.25 70.64 67.76 62.27 51.77 38.31 58.15 S-AV-HuBERT (V) 32.41 32.69 33.72 34.91 36.45 37.87 35.13 33.26 34.15 35.54 36.22 37.16 35.27 S-AV-HuBERT (A+V) 76.47 72.92 71.43 68.89 64.78 59.38 67.48 72.83 71.50 68.98 65.33 60.23 67.77 AV-STE (Hard-CA) 75.12 71.58 70.15 67.75 63.78 58.59 66.37 71.54 70.22 67.94 64.17 59.13 66.60 AV-STE (Soft-CA) 76.11 73.18 71.43 69.25 64.96 59.38 67.64 73.10 71.41 69.03 65.51 60.26 67.86 AV-STE (Soft-CA+NAM) 76.78 73.48 71.87 69.48 65.34 59.92 68.02 73.17 71.76 69.20 65.82 60.52 68.09 †Mimi clean accuracy is 100% by definition; therefore, it is not directly comparable to predicted-token accuracy. Table 8: Detailed WER (%) results across noise types and SNR levels. Lower is better. Condition Mimi AV-STE (Soft-CA+NAM) AV-STE (Soft-CA) AV-STE (Hard-CA) S-AV-HuBERT (A+V) S-AV-HuBERT (A) S-AV-HuBERT (V) Clean 7.6 7.8 7.7 7.7 7.9 7.2 10.8 Non-speech noise (AudioSet) 10 dB 21.1 16.0 17.4 16.7 16.6 17.4 22.1 5 dB 37.3 26.0 26.6 25.1 25.6 27.1 35.8 0 dB 66.6 49.2 46.4 49.1 46.8 50.5 56.5 -5 dB 107.6 84.0 89.4 88.5 87.6 90.8 84.8 -10 dB 186.2 139.2 148.8 127.2 155.2 136.4 141.8 Speaker interference (AudioSet) 10 dB 26.4 17.3 17.5 17.8 17.3 18.0 25.0 5 dB 44.7 30.2 32.5 29.7 34.7 32.9 38.2 0 dB 75.7 55.0 53.7 56.7 54.1 57.1 63.2 -5 dB 100.7 88.4 86.0 87.9 82.7 85.5 89.7 -10 dB 126.8 100.2 104.0 116.3 105.8 113.5 124.8 Appendix B Detailed Semantic Token Accuracy Results Table 7 provides the full per-SNR semantic token accuracy results for each noise type. The results support the trend in Figure 3: AV-STE improves over the audio-only Mimi baseline across noise conditions, with the largest gains at low SNRs. Appendix C Detailed WER Results Table 8 provides the full per-SNR WER results for each noise type. AV-STE improves over the audio-only Mimi baseline across noise conditions, with the largest gains at low SNRs. Appendix D Metrics Implementation Take Over Rate (TOR). For each response audio, we obtain word-level timestamps via automatic speech recognition using NeMo Parakeet-TDT-0.6B Sekoyan et al. (2025) and merge consecutive word chunks separated by less than 0.8 s into utterances. Following Full-Duplex-Bench Lin et al. (2025), an utterance is deemed take over and not mere backchannel if it has duration ≥1.0≥ 1.0 s or word count ≥3≥ 3. However, unlike Full-Duplex-Bench, we count a takeover as successful only if the model response starts within a valid turn-taking window, from 0.5 s before the end of the user’s utterance onward. This prevents premature interruptions from being counted as successful takeovers. Latency. We compute the mean signed latency δ over all successful takeovers. Because the model may begin responding slightly before the user’s utterance ends (up to 0.5 s of natural overlap), negative latency values are clipped to zero, so the metric reflects only the delay after the user’s turn ends rather than rewarding early interruptions. GPT-based coherence evaluation. For response semantic coherence, we use GPT-4o (OpenAI et al., 2024) as an automatic judge, following prior work on LLM-based evaluation (Zheng et al., 2023). For each successful takeover, the judge receives the user’s utterance transcript and the generated response, and scores how relevant the generated response is to the user’s utterance on an integer scale from 1 (irrelevant) to 5 (fully relevant). The final GPT score is the mean over all evaluated samples. We use the following prompt for GPT-based relevance scoring: You are an expert evaluator for dialogue systems. Your task is to judge how relevant the assistant response is to the user’s question or prior utterance. Evaluate ONLY relevance. Do not evaluate fluency, politeness, grammar, verbosity, factuality, or timing unless they directly affect relevance. Scoring rubric: • 5: Fully relevant; directly addresses the question, request, or topic. • 4: Mostly relevant; clearly related but misses a small part. • 3: Partially relevant; related but incomplete, vague, or misaligned. • 2: Weakly relevant; only small overlap with the topic. • 1: Irrelevant or nearly irrelevant. [USER_UTTERANCE] user_text [ASSISTANT_RESPONSE] response_text Return valid JSON with exactly these keys: "score": <integer 1--5>, "reason": "<brief explanation>" Appendix E Qualitative Evaluation Table 9 shows representative examples where AV-STE improves response coherence over the Moshi baseline. Under noisy conditions, Moshi often produces generic or off-topic responses, while AV-STE generates responses that better match the user’s utterance. Table 9: Qualitative examples where AV-STE improves response coherence over the Moshi baseline. GPT scores are shown in parentheses. Noise User Utterance Moshi Response (Moshi+AV-STE) Response Speech −10-10 dB “I think what that means is that people just couldn’t see what was in front of them.” “Sure, I saw them going up the flight of stairs.” (GPT: 1) “Yeah, it’s like they were blind to the reality that was right in front of them.” (GPT: 5) Speech −5-5 dB “We have the tools but we lack the collective will and momentum.” “Yeah, I think they are. I’ve seen them before.” (GPT: 1) “I agree. We need to work together.” (GPT: 4) Speech 00 dB “So we need a system that helps all our teachers be as good as the best.” “I’m sorry, but I don’t.” (GPT: 1) “Yeah, I agree. We need a system that allows teachers to share their expertise in best practices.” (GPT: 5) Speech 55 dB “What is the exact relationship between levels of greenhouse gases and planetary..” “The exact relationship between six and five is subtlely of one.” (GPT: 1) “The relationship between levels of greenhouse gases and planetary heating is..” (GPT: 3) Speech 1010 dB “They all look the same under a microscope, I mean to a first approximation.” “Well, it’s a good thing that you’re being careful, but if you’re really worried,” (GPT: 1) “Yeah, yeah, you’re right. I mean, they all look alike, don’t they?” (GPT: 5) Non-Speech −10-10 dB “We were making what was invisible visible.” “The story takes place in the nineteen sixties.” (GPT: 1) “Invisible means not able to be seen or detected, while visible means able to be.” (GPT: 2) Non-Speech −5-5 dB “We can create a decentralized database that has the same efficiency of a monopoly.” “Well, the key is to use a mild cleaner and gently rub the stains with a soft cloth.” (GPT: 1) “Okay, what do you mean by efficiency?” (GPT: 5) Non-Speech 00 dB “They are too busy putting out daily fires to think ahead strategically.” “Stay calm, let’s get everyone out safely.” (GPT: 1) “That’s true. They have a lot on.” (GPT: 3) Non-Speech 55 dB “You don’t have to bring any quarters because the washer and dryer are free.” “But I have to do the dishes, it’s my job.” (GPT: 1) “Okay, great. So I don’t have to bring any quarters.” (GPT: 5) Non-Speech 1010 dB “Without it we literally can’t interface with others.” “Yeah, that’s true.” (GPT: 2) “Yeah, it’s a crucial tool for communication and interaction.” (GPT: 5)

Instant research discovery