Paper deep dive
OmniEcho: Spatial Audio Understanding for Embodied Agents
Ruixun Liu, Yuxuan Wang, Jiacheng Xie, Yuhuan You, Donghua Cai, Junming Lin, Xiong-Hui Chen, Zhifang Guo, Yunfei Chu, Qize Yang, Xize Cheng, Jin Xu, Yiwu Zhong
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 93%
Last extracted: 9/27/2026, 2:40:27 AM
Summary
The paper introduces OmniEcho, a spatially aware omni-modal model for embodied agents, and OmniEchoBench, a unified benchmark for spatial audio-visual perception and navigation. OmniEchoBench comprises six tasks across 197 real-world scenes with First-Order Ambisonics (FOA) audio. OmniEcho integrates an FOA spatial encoder with a pretrained semantic audio pathway to enable spatial audio-visual question answering and sound-guided navigation, achieving state-of-the-art performance on perception tasks and competitive results on navigation.
Entities (8)
Relation Signals (7)
OmniEchoBench → evaluatestask → Sound-Guided Navigation
confidence 95% · OmniEchoBench, a unified benchmark for spatial audio-visual perception and audio-vision-language navigation.
OmniEchoBench → evaluatestask → Spatial Audio-Visual Perception
confidence 95% · OmniEchoBench, a unified benchmark for spatial audio-visual perception and audio-vision-language navigation.
OmniEchoBench → usesaudioformat → First-Order Ambisonics
confidence 95% · with first-order ambisonics (FOA) audio collected from 30 real-world environments.
OmniEcho → usescomponent → FOA spatial encoder
confidence 95% · It introduces an FOA spatial encoder alongside a pretrained semantic audio pathway.
OmniEcho → builton → Qwen3-Omni
confidence 92% · we propose OmniEcho, a spatially aware omni-modal model built on Qwen3-Omni
OmniEcho → achievesperformanceon → Spatial Audio-Visual Perception
confidence 90% · OmniEcho achieves state-of-the-art performance on spatial audio-visual perception.
OmniEcho → achievesperformanceon → Sound-Guided Navigation
confidence 85% · For our sound-guided navigation, OmniEcho reaches a performance level close to that of traditional vision-language navigation.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Humans can effortlessly localize the direction of a sound source and integrate it with visual cues for reasoning, yet this remains challenging for embodied agents. In particular, it is still unclear how to effectively evaluate and model spatial audio understanding in embodied settings. To address this gap, we introduce \textbf{OmniEchoBench}, a unified benchmark for spatial audio-visual perception and audio-vision-language navigation. OmniEchoBench comprises six tasks over 197 real-world spatial audio-visual scenes, 2,972 question-answer pairs, and 900 navigation samples with first-order ambisonics (FOA) audio collected from 30 real-world environments. To enable scalable training supervision, we develop a controllable rendering pipeline for spatial audio. It preserves geometric consistency among sound sources, visual observations, and agent trajectories. Building on this, we propose \textbf{OmniEcho}, a spatially aware omni-modal model. It introduces an FOA spatial encoder alongside a pretrained semantic audio pathway. Extensive experiments show that OmniEcho achieves state-of-the-art performance on spatial audio-visual perception. For our sound-guided navigation, OmniEcho reaches a performance level close to that of traditional vision-language navigation. These results demonstrate that spatial audio can serve as a valuable signal for embodied scene reasoning and navigation, while also highlighting fine-grained spatial localization and distance estimation as important open challenges.
Tags
Links
- Source: https://arxiv.org/abs/2609.23407v1
- Canonical: https://arxiv.org/abs/2609.23407v1
Trouble viewing inline? Open PDF directly →
Full Text
119,135 characters extracted from source content.
Expand or collapse full text
Under review OMNIECHO: SPATIAL AUDIO UNDERSTANDING FOR EMBODIED AGENTS Ruixun Liu 1,2* Yuxuan Wang 3 Jiacheng Xie 1 Yuhuan You 1* Donghua Cai 4* Junming Lin 4* Xiong-Hui Chen 3 Zhifang Guo 3 Yunfei Chu 3 Qize Yang 3 Xize Cheng 3 Jin Xu 3 Yiwu Zhong 1,2† 1 School of Intelligence Science and Technology, Peking University, 2 State Key Laboratory of General Artificial Intelligence, Peking University, 3 Alibaba Token Hub, Alibaba Group, 4 Tsinghua University ABSTRACT Humans can effortlessly localize the direction of a sound source and integrate it with visual cues for reasoning, yet this remains challenging for embodied agents. In particular, it is still unclear how to effectively evaluate and model spatial au- dio understanding in embodied settings. To address this gap, we introduce Om- niEchoBench, a unified benchmark for spatial audio-visual perception and audio- vision-language navigation. OmniEchoBench comprises six tasks over 197 real- world spatial audio-visual scenes, 2,972 question-answer pairs, and 900 naviga- tion samples with first-order ambisonics (FOA) audio collected from 30 real-world environments. To enable scalable training supervision, we develop a controllable rendering pipeline for spatial audio. It preserves geometric consistency among sound sources, visual observations, and agent trajectories. Building on this, we propose OmniEcho, a spatially aware omni-modal model. It introduces an FOA spatial encoder alongside a pretrained semantic audio pathway. Extensive ex- periments show that OmniEcho achieves state-of-the-art performance on spatial audio-visual perception. For our sound-guided navigation, OmniEcho reaches a performance level close to that of traditional vision-language navigation. These results demonstrate that spatial audio can serve as a valuable signal for embod- ied scene reasoning and navigation, while also highlighting fine-grained spatial localization and distance estimation as important open challenges. 1INTRODUCTION Embodied agents rely on rich multimodal signals to perceive, reason, and act. Recent advances in omni-modal understanding have substantially improved the ability to jointly process vision, lan- guage, and audio (Deshmukh et al., 2026; Liu et al., 2026; Xu et al., 2025b; Liu et al., 2025; Ye et al., 2026). Meanwhile, progress in vision-language navigation (VLN) has enabled agents to rea- son about their surroundings and plan actions from language instructions (Wei et al., 2026; Wu et al., 2025; Wang et al., 2026; Wei et al., 2025). However, spatial audio, a common and important infor- mation source to humans, remains largely underexplored. Humans can effortlessly follow spatial auditory cues to localize target objects even if they are occluded, and to search for target objects beyond the current field of view. For embodied agents, such an ability is equally important. Sounds can provide information about events outside the camera’s view, reveal the direction of targets, and guide agents toward their locations (Yang et al., 2024; Chen et al., 2020a; Gan et al., 2020). In this work, we address spatial audio understanding for omni-modal embodied agents. The agent receives first-order ambisonics (FOA) spatial audio together with visual observations and, when ap- plicable, language instructions. It then reasons about the surrounding environment to answer spatial questions or navigate toward a target. There arise three critical challenges in this context: (1) Col- lecting real-world spatial-audio data is particularly costly. The received signal jointly depends on source and listener poses, scene geometry, reflections, and reverberation, which must be faithfully * Work done during internship at Alibaba Group. † Corresponding Author. 1 arXiv:2609.23407v1 [cs.SD] 20 Sep 2026 Under review D B A C OmniEcho Perception & Navigation MLLM Perception or Navigation Instruction RGB Obs. Audio Spatial Audio Spatial Audio Perception OmniEcho Bench Speech Command(395) Source Direction (1069) 3D Localization (521) Source Motion (473) Camera Rotation (309) Cognitive Map (600) Robot, come here Come where? Robot, come here (a) Traditional Omni model Sound source Self (robot) (b) Our Spatial Omni model Instruction RGB Obs. Audio Non-Speech Audio (505) Spatial Audio Navigation Figure 1: Overview of OmniEcho and OmniEchoBench. Left: OmniEcho extends a traditional omni model with spatial sound for perception and navigation. Right: OmniEchoBench benchmarks spatial perception and spatial audio navigation through six tasks. captured while source trajectories and listener motion are accurately annotated. (2) Simulation can provide large-scale training data. However, effective training data should not only include rendered audio, but also maintain physical consistency with visual observations and moving agents. (3) Exist- ing pre-trained models already excel in processing omni-modal signals. How to effectively integrate spatial audio into these pre-trained models remains an open question. To address these challenges, we first introduce OmniEchoBench, a unified benchmark for spatial audio-visual perception and audio-vision-language navigation, collected from real-world scenes. We also develop a scalable spatial-audio data construction pipeline. For perception, OmniEchoBench covers audio-visual scenarios involving visible, occluded, and dynamically changing sound sources, requiring agents to reason about source direction, 3D location, source motion, camera rotation, and cognitive maps (Tolman, 1948). For navigation, we collect spatial audio from dense positions in 30 real-world scenes and construct sound-guided navigation trajectories, together with a Habitat-based simulation platform for closed-loop evaluation (Savva et al., 2019). Moreover, to provide scalable supervision for model development, we synthesize large-scale training data through controllable spatial audio rendering. For spatial perception, we construct audio-visual scenes with manually specified sound-source configurations and trajectories, and render them into FOA audio to generate spatial question-answering data. For navigation, we augment existing VLN trajectories (Anderson et al., 2018; Ku et al., 2020; Wang et al., 2025; 2023) by placing sound sources at target locations and simulating acoustic effects such as distance attenuation and room reverberation. Building on this, we propose OmniEcho, a spatially aware omni-modal foundation model built on Qwen3-Omni (Xu et al., 2025b). We design an FOA encoder and introduce a three-stage training procedure to align semantic and spatial information. By preserving the model’s pretrained semantic audio pathway while introducing a spatial pathway, OmniEcho is able to jointly understand auditory semantics, spatial information, and environmental observations. We evaluate both spatial ques- tion answering and sound-guided navigation. Experiments show that existing omni-modal models struggle on our benchmark, whereas OmniEcho achieves state-of-the-art performance on spatial perception tasks. Although our sound-guided navigation task is more challenging than traditional text-guided VLN, OmniEcho achieves performance close to that of VLN methods. These results highlight the effectiveness of spatial audio guidance and the applicability of our task formulation. Our main contributions are summarized as follows: • To our knowledge, we are the first to address spatial audio understanding for omni-modal embodied agents, capable of spatial audio-visual QA and sound-guided navigation. • We introduce OmniEchoBench, a benchmark consisting of spatial audio perception and navigation, with human annotations in real-world scenes. We further construct large-scale training data through spatial audio simulation, providing scalable training supervision. • We present OmniEcho, a foundation model that can understand spatial audio and sup- port both spatial audio perception and navigation. Extensive experiments demonstrate its strong performance on OmniEchoBench, while revealing fundamental limitations of exist- ing omni-modal models in spatial audio reasoning. 2 Under review 2RELATED WORK Traditional Vision-Language Navigation.Vision-and-language navigation (VLN) requires an embodied agent to follow natural-language instructions by grounding route descriptions, landmarks, spatial relations, and action choices in visual observations and navigation history. Since Room-to- Room (R2R) introduced instruction-guided navigation in real indoor environments using the Mat- terport3D Simulator (Anderson et al., 2018), VLN has evolved from short-horizon route following toward more realistic and diagnostic navigation settings. Recent work has built fine-grained bench- marks for instruction understanding (Wang et al., 2024c) and studied online robustness through test- time adaptation (Gao et al., 2023) and obstructed environments (Hong et al., 2024). Other studies have improved reasoning or memory through causal learning, adaptive history filtering, language- based perceptual representations, explicit LLM reasoning, and bidirectional cross-modal model- ing (Wang et al., 2024a; He et al., 2024; Pan et al., 2024; Zhou et al., 2024; Wu et al., 2025). Despite these advances, representative VLN models remain fundamentally vision-text driven: their decisions mainly depend on RGB/RGB-D observations, textual instructions, object or landmark descriptions, navigation histories, or language-converted scene representations. They do not treat spatial acoustic cues, such as sound-source direction, distance, and spatial layout, as native evidence for navigation. We instead introduce a navigation model with spatial audio understanding capability. Audio-Visual Navigation.Audio-visual navigation studies embodied agents that use auditory and visual observations to reach sound-emitting targets in 3D environments. Look, Listen, and Act first formalized audio-visual embodied navigation as shortest-path planning toward a sound source from egocentric audio-visual inputs (Gan et al., 2020). SoundSpaces and SoundSpaces 2.0 then established widely used benchmarks by adding geometry-based acoustic rendering to Habitat over Matterport3D and Replica. However, these benchmarks rely on rendered acoustics and do not cover spoken-instruction navigation (Chen et al., 2020a; 2022). Subsequent work improved waypoint selection, acoustic memory, semantic sound handling, and robustness to moving or noisy sources (Chen et al., 2020b; 2021; Younes et al., 2023; Chen et al., 2023; Shi et al., 2025). Other studies explored language-mediated interaction or planning for audio-visual navigation, including AVLEN, CAVEN, AVLMaps, and RILA (Paul et al., 2022; Liu et al., 2024; Huang et al., 2023; Yang et al., 2024). However, these studies do not evaluate navigation in scenarios with real-world recorded spatial audio. In contrast, we introduce a unified benchmark for simultaneously evaluating spatial audio perception and navigation, and we provide a navigation simulator built upon real-world collected FOA audio, thereby offering more realistic and reliable acoustic evidence for navigation. Spatial Audio Understanding. Recent work has improved spatial acoustic modeling from three complementary angles. One line develops encoders that use binaural phase differences, intensity cues, GCC-PHAT features, microphone geometry, or task-disentangled branches, improving local- ization, distance estimation, and spatial reasoning (Zheng et al., 2024; Wilkinghoff & Tan, 2026; Dementyev et al., 2026). A second line aligns spatial audio with language, either through con- trastive learning over open-vocabulary text or through structured embeddings that separate semantic and spatial factors for retrieval, understanding, and editing (Devnani et al., 2024; Hu et al., 2025). A third line connects spatial audio representations to audio-language models for natural-language description and reasoning about direction, distance, reverberation, and multi-source relations (Sak- shi et al., 2025; Jiang et al., 2026; Biswas et al., 2026; You et al., 2026; Liu et al., 2026). Overall, these works demonstrate that spatial audio can be encoded, aligned with language, and used for language-based spatial reasoning. Recent work has begun to equip MLLMs with spatial audio un- derstanding. The closest related work is Spatial-Omni (Zhu et al., 2026), which explores spatial audio understanding. However, it does not incorporate visual frames during training and does not address sound-guided navigation. In contrast, our benchmark uses real-world FOA recordings and evaluates both spatial audio-visual understanding and sound-guided navigation. 3BENCHMARK AND DATASETS Overview. This section presents the benchmark and dataset constructed in this work, including OmniEchoBench-QA for evaluating spatial-audio understanding and OmniEchoBench-Nav for as- sessing sound-guided navigation. It further describes the training data synthesis pipeline, which is built upon a unified FOA spatial-audio rendering framework to generate spatial audiovisual question- answering data, audio-only data, and sound-guided navigation data. 3 Under review Approx Body Degrees Body At Time World Heading Degrees Change Consistency OmniEchoBench-QA 2972 QA Pairs 197 Video Scenes At t=7.5 s, in the CAMERA-BODY frame(relative to where the camera iscurrently pointing), in which approximate direction is the source of a buzzing vibration sound? At t=9.0 s, give the position of the source of a buzzing vibration sound in the CAMERA-BODY frame (relative to where the camera is currently pointing) as an approximate direction plus a distance in metres At t=15.0 s, by how many degrees and in which direction must the camera rotate so that the source of a buzzing vibration sound becomes centred in its view? In the WORLD frame (fixed to the camera‘s initial heading), what is the overall direction of travel of the source of thetoy carfrom t=31.5 to t=50.0 s? Source Direction 3D Localization Camera Rotation Source Motion behind and to the right to the right, about 1.1 m 45 degrees to the right back Cognitive Map Look at the top-down view centered on you (up is your front) and the 4-channel FOA spatial audio recorded at your position. The sound source is among the marked locations A, B, C, and D, determine where the sound most likely comes from. A 8 Sub-tasks 600 Bird's-eye View Images Figure 2: Overview of OmniEchoBench-QA, which comprises 2,972 QA pairs across 197 scenes and 600 bird’s-eye view images, assessing a model’s understanding of spatial audio. Sound SourceRecording Points (FOA) Robot Instruction: find the sound source of the music. History obs. Current obs. I am currently standing at the doorway of a room, looking into the living-room through the dim doorway. On the left side of the living-room is a table, and at its far end lies a corridor. I hear the music coming from slightly forward-left, at an approximate distance of 4.6meters, indicating that the sound source is deep inside the living-room, beyond the doorway. Since the path through the doorway is unobstructed and the sound is slightly to the left, my plan is to walk forward through the doorway and then adjust my heading slightly leftward to approach the source of the music. Meta-action: move forward and move forward-left [0.2500, 1.0, 0.0, 0.0], [0.5000, 1.0, 0.0, 0.0], [0.7500, 2.0, 0.0, 0.0], [1.0000, 2.0, 0.0, 0.0], [1.2500, 3.0, 0.0, 0.0], [1.2500, 3.0, 0.26175], [1.4914, 3.0, 0.26175], [1.4914, 3.0, 0.0647, 0.26175], [1.4914, 3.0, 0.0647, 0.0] FOA Audio: 12900 recording points 900 navigation tasks 30 scenarios OmniEchoBench-Nav Figure 3: Overview of OmniEchoBench-Nav. Collected across 30 real-world scenes, the benchmark contains 900 navigation tasks, where agents use historical frames, the current view, FOA audio, and an instruction to navigate toward the target. 3.1OMNIECHOBENCH-QA Design principle. As shown in Figure 2, we introduce OmniEchoBench-QA to evaluate whether an omni-modal model can perceive and reason about sound-source geometry from egocentric observa- tions. This benchmark emphasizes sound sources that lie outside the current visual field or dynami- cally enter and leave the field of view over time. Unlike synthetic setups, all data are collected from real human performances. We first author a set of scripts (screenplays) that specify the acoustic events, the actors’ motion, and their spatio-temporal relation to the camera (see the accompanying screenplay sheet), deliberately emphasizing off-screen and boundary-crossing sound sources so that the visual stream alone is insufficient for answering the questions. Human actors then perform and record each script, capturing, in a single synchronized take, a first-person main-view video, a 360 ◦ panoramic video, and a four-channel first-order Ambisonics (FOA, ACN/SN3D) audio track. The panoramic video is used only during annotation; at test time, the model is given a single-view video (or a still image) together with the FOA audio, matching a realistic embodied-perception setting. Annotation and question construction. Using the panoramic recording as the ground-truth refer- ence, annotators label the real-time position of every active sound source at 2 Hz, yielding a dense spatio-temporal trajectory for each source relative to the recording device. Then we apply rule-based sampling to instantiate questions in several typed categories, automatically deriving the ground-truth answers. The benchmark comprises two complementary subsets and 2,972 QA pairs in total. The first subset (2,372 QA) probes fine-grained temporal spatial reasoning over the recorded videos: source direction (1,069), 3D localization (521), source motion (473), and camera rotation (309). The second subset (600 QA) targets audio-driven source localization from a bird’s-eye view. In this setting, the model is presented with an ego-centered top-down map (with “up” aligned to its forward direction) together with the co-located FOA clip collected in navigation scenes. The model must then select the most plausible sound-source location from four candidates marked A,B,C,D. This setup more directly isolates the model’s ability to integrate spatial hearing with a cognitive map. Additional information can be found in the Appendix A.2.1. 4 Under review Seedance and Verifier VLN- Simulator RGB obs. Video Trajectory Qwen TTS/ Seedance Sampling [label, x, y, z] [label, x, y, z] [label, x, y, z] LLM QA Task/Scene Description (audio only & audio-vison) (b) FOA Audio only 318k 27k (a) Video w/ FOA 118k (c) Navigation sample w/ FOA FOA Rendering FOA (4 channel) W Y Z X Task/Scene Description (navigation) Figure 4: Overview of our training data generation pipeline. From simulated scenes, the system generates QA pairs, videos and sound event annotations and renders mono audio into FOA data. 3.2OMNIECHOBENCH-NAV Design and Annotation. As shown in Figure 3, we introduce OmniEchoBench-Nav, a real-world indoor navigation benchmark that jointly supports spatial-audio and language-instruction naviga- tion over the same set of scenes and targets. The benchmark comprises 30 real-scanned indoor environments, distributed over single-room (10) and multi-room cross-room layouts (20, spanning two to four connected rooms), each provided as a textured mesh (.glb) together with a top-down floor render. Each scene contains 30 navigation samples, giving 900 samples in total. In each sam- ple, a single sound source is placed at a known 3D position. Its signal is captured at a dense grid of receiver positions with known coordinates, producing 12,900 FOA recordings (4-channel, 48 kHz) that together form a spatially dense acoustic field of the environment. Sources cover a broad range of everyday audio, split into non-speech environmental events (e.g., mechanical, alarm, doorbell, ani- mal, footsteps; 505 samples) and spoken commands (e.g., help requests, calls; 395 samples). Each source is annotated with its source device, room, semantic description, and a natural-language de- scription of the navigation target. At test time, the agent is provided with the spatial audio recorded at the receiver position nearest to its current location, with directional remapping performed accord- ing to its current orientation. Based on this perceived spatial audio, the embodied agent must infer the location of the target sound source and navigate toward it. The Appendix A.2.2 describes the real-world scenes and acoustic-field collection process in greater detail. 3.3SPATIAL AUDIOVISUAL SIMULATION FOR DATA SYNTHESIS We build all spatial perception and navigation data on top of a unified spatial-audio rendering frame- work. Each sounding source is represented by a time-varying 3D trajectory, while the listener is co- located with the camera. We adopt a right-handed listener frame with the camera at the origin facing +y; azimuth θ ∈ [−180 ◦ , 180 ◦ ] is defined with 0 ◦ straight ahead, +90 ◦ to the left, and −90 ◦ to the right. Given the source trajectories and listener pose, we encode each monophonic source stem into FOA (4 channels, ACN/SN3D) using real spherical harmonics (Nachbar et al., 2011; Zotter & Frank, 2019): a(t) = X i g i r i (t) Y θ i (t),φ i (t) s i (t),(1) where s i (t), r i (t), and g i denote the source stem, source–listener distance, and source gain, respec- tively. Source directions are transformed into the current listener frame to maintain spatial alignment with the visual observations, with optional distance attenuation and room effects. This unified for- mulation is shared by both spatial audiovisual perception and sound-guided navigation. Figure 4 illustrates two training-data pipelines. For spatial audiovisual QA, we generate dynamic scenes with Seedance (Seedance et al., 2026) and combine their visual content with trajectory- grounded sound events, from which FOA audio and QA pairs are synthesized. We further synthesize audio-only training data to support FOA encoder pretraining and QA. For navigation, we augment existing VLN trajectories with destination-conditioned sound events and render the corresponding 5 Under review FOA waveform W X Y Z Text Input or “Is the source of a motor sound visible?” “I’m stuck. Come to help me.” Visual Tokens Text Tokens Answer “No.” Predicted waypoints (푥 푘 ,푦 푘 ,휃 푘 ) 푘=1 8 PerceptionNavigation Qwen3-Omni-30B-A3B Visual Tower Text Encoder Single channel Down-mix to Mono Native Audio Tower Audio Tokens Fill Reserved Placeholders <foa_pad><foa_pad> Projector MLP 2× FOA Encoder FOA Tower FOA Tokens ... Conv n× Transformer 5× ... Projected Embeddings ... 384-d Spatial Tokens Temporal Resampling Image Video Figure 5: Overview of our unified spatial omni model built on Qwen3-Omni-Thinker. The pretrained FOA encoder extracts spatial tokens; these tokens are projected into the language embedding space by a trainable projector, and inserted after the corresponding semantic audio tokens. FOA observations along sampled trajectories. The datasets provide synchronized visual observa- tions, spatial audio, and geometry-grounded supervision for both perception and navigation. Further details of the data generation and annotation procedures are provided in the Appendix A.1. 4METHOD 4.1TASK FORMULATION We unify spatial audio-visual perception and navigation within a single embodied-agent framework. For perception tasks, the model takes FOA audio A, a video sequence V or a single image I , and a question q, and predicts an answer a. For navigation tasks, we follow the setting of InternNav (Wei et al., 2026). At each decision step, the model receives a language instruction L, T frames captured by a forward-facing camera, and the FOA spatial audio A t within the current temporal window, and predicts a trajectory over the next K = 8 time steps. We uniformly sample T − 1 keyframes from past observations and combine them with the current frame as visual input. This formulation enables unified spatial audio-visual perception and navigation within a single framework. 4.2SPATIAL AUDIO ENCODING Stage 1: Semantic alignment of the FOA encoder. We first pretrain a lightweight FOA encoder f θ that maps an FOA clip into a temporal sequence of d-dimensional tokens (d = 384). Each clip is converted into a 5-channel input map X∈R 5×128×T comprising the log-mel spectrogram of the omnidirectional (W ) channel together with the three active-intensity components (i x ,i y ,i z ) and the diffuseness δ, all in the ACN/SN3D convention. The encoder is trained to align its representations with a frozen CLIP text encoder (Radford et al., 2021) using a SigLIP objective (Zhai et al., 2023), thereby establishing an open-vocabulary sound-semantic space. Training uses a 100k-clip corpus of synthetic FOA scenes containing 1–4 sound sources with diverse spatial configurations. Stage 2: Query-conditioned spatial localization. Starting from the semantically aligned encoder, we further fine-tune it for query-conditioned sound-source localization. Given a text query specify- ing a target source, a cross-attention localization head attends to the encoder tokens and predicts the source’s azimuth, elevation, and distance. The model is optimized with a localization loss together with the Stage-1 semantic objective to preserve the learned sound semantics. Stage 3: Integration into the Omni backbone. As shown in Figure 5, we graft the frozen FOA encoder f θ ⋆ into the Qwen3-Omni model (Xu et al., 2025b). Each FOA clip is processed along two complementary paths: (i) its W channel is down-mixed to mono and fed to the original, frozen audio tower to preserve the model’s native semantic audio tokens; and (i) the same clip is passed 6 Under review Table 1: Results on our OmniEchoBench-QA. Our OmniEcho performs strongly in both audio- only and audio-visual settings, outperforming most evaluated models in overall accuracy and across the individual spatial reasoning tasks. ’A-single’ denotes single-channel audio, ’A-foa’ denotes four-channel FOA audio, and ’V’ denotes visual input. MethodBase ModelSetting Camera Rotation Source Direction 3D Localization Source Motion Cognitive Map Overall Random--33.612.5-23.825.0- Audio only Qwen3-Omni-30B-A3B (Xu et al., 2025b)Qwen3-Omni-30B-A3BA-single22.718.48.322.0-17.5 Qwen2.5-Omni-7B (Xu et al., 2025a)Qwen2.5-Omni-7BA-single22.78.34.614.610.6 GLM4-Voice-9B (Zeng et al., 2024)GLM4-9BA-single20.49.55.420.3-12.2 SO-7B (Zhu et al., 2026)Qwen2.5-Omni-7BA-foa21.47.10.020.7-10.1 OmniEcho (Ours)Qwen3-Omni-30B-A3BA-foa29.122.114.024.9-21.8 Audio & Vision Qwen3-Omni-30B-A3B (Xu et al., 2025b)Qwen3-Omni-30B-A3BA-single + V32.716.37.720.922.318.5 Qwen2.5-Omni-7B (Xu et al., 2025a)Qwen2.5-Omni-7BA-single + V24.98.05.414.020.012.6 SO-7B (Zhu et al., 2026)Qwen2.5-Omni-7BA-foa + V21.45.20.017.122.811.4 OmniEcho (Ours)Qwen3-Omni-30B-A3BA-foa + V41.722.614.224.747.528.5 Table 2: Results on our OmniEchoBench-Nav. Our benchmark is designed to evaluate sound- guided navigation, and meanwhile, additionally provides text instruction for traditional text-guided VLN methods. Even if audio guidance is more challenging than text instruction, our method achieves close performance as previous VLN methods. ModelBase ModelGuidance Val-Unseen SR↑ SPL↑ NE↓OS↑TL Seq2Seq (Krantz et al., 2020)End-to-endText Instruction11.39.44.1453.00 5.51 CMA (Krantz et al., 2020)End-to-endText Instruction9.88.54.3448.56 5.00 InternVLA-N1 (Intern Robotics, 2025)Qwen2.5-VL-7BText Instruction17.816.72.8763.89 4.67 Soundspaces (Chen et al., 2020a)End-to-endBinaural audio5.43.95.3225.67 4.16 OmniEchoQwen3-Omni-30B-A3BFOA audio16.211.54.0952.89 7.02 through f θ ⋆ to obtain spatial tokens, which are temporally resampled to the 7 Hz audio-token grid and mapped into the language embedding space by a trainable projector. The projected spatial tokens occupy reserved placeholder embedding rows (<foapad>) and are inserted immediately after the corresponding audio tokens in temporal order. Only the LLM parameters Ψ and the projector φ are updated, while the FOA encoder, audio tower, and visual tower remain frozen. Additional encoder architecture and optimization details for all three training stages are included in the Appendix A.3. 5EXPERIMENTS 5.1SETUP Implementation Details. We build our model upon the pretrained Qwen3-Omni-30B-A3B (Xu et al., 2025b). Specifically, the model is trained on spatial audio-visual QA together with navigation- oriented supervision, resulting in 363,193 training examples in total. The navigation subset con- sists of 139,142 examples from ScaleVLN (Wang et al., 2023), 92,418 from RxR (Ku et al., 2020), 31,577 from R2R (Anderson et al., 2018), and 28,969 from VLN-PE (Wang et al., 2025). Stage 3 training is conducted on 32 NVIDIA A100 GPUs and requires approximately four days. Benchmarks. For spatial audio-visual perception, we report results on OmniEchoBench-QA. For navigation, we evaluate on OmniEchoBench-Nav as well as the VLN-CE R2R val unseen split (Anderson et al., 2018; Krantz et al., 2020). More details are documented in the Appendix A.4. 5.2MAIN RESULTS 5.2.1SPATIAL AUDIO-VISUAL QUESTION ANSWERING RESULTS Table 1 reports the results on OmniEchoBench-QA. Under both the audio-only and audio-visual settings, OmniEcho outperforms evaluated models overall and across individual tasks. In the audio- only setting, OmniEcho surpasses the Qwen3-Omni-30B-A3B (Xu et al., 2025b) backbone which 7 Under review Table 3: Performance on VLN-CE R2R with simulated spatial audio. Although sound-guided navigation is more challenging than text-guided VLN, our method achieves comparable performance to previous baselines, demonstrating the potential of our model and dataset for future research. ModelBase ModelGuidance Val-Unseen SR↑SPL↑NE↓OS↑TL Seq2Seq (Krantz et al., 2020)End-to-endText Instruction21.0-7.8128.08.39 CMA (Krantz et al., 2020)End-to-endText Instruction22.020.08.1728.07.42 VLN-R1 (Qi et al., 2025)Qwen2-VL-7B (Wang et al., 2024b)Text Instruction30.221.87.041.210.0 InternVLA-N1 (Intern Robotics, 2025)Qwen2.5-VL-7B (Bai et al., 2025)Text Instruction50.643.34.7356.7- NAViLA-SAGE (Miao et al., 2026)navila-siglip-llama3-8b-v1.5-pretrainText Instruction38.036.0-51.0- OmniEchoQwen3-Omni-30B-A3B (Xu et al., 2025b)Audio Instruction22.216.37.3124.55.54 has only single-channel audio encoder, demonstrating the effectiveness of our FOA encoder. Om- niEcho significantly outperforms SO-7B (Zhu et al., 2026) on our real-world collected benchmark. Although SO-7B claims to employ spatial-audio training, it lacks visual supervision during training and still suffers from a sim-to-real gap. These findings suggest that existing methods remain limited in their robustness for spatial-audio understanding in real-world scenarios. Comparing OmniEcho across the two input settings, removing visual input leaves its performance on Source Direction, 3D Localization, and Source Motion largely unchanged, indicating that the model can perform these forms of spatial reasoning primarily from FOA audio. Camera Rotation, however, degrades substantially without vision, suggesting that this task depends more strongly on visual evidence of viewpoint changes. This pattern is consistent with the FOA-encoder abla- tion, where explicit FOA features benefit source-centric spatial tasks but provide limited gains for camera-rotation reasoning. Together, these results suggest that FOA primarily captures sound-source geometry, while visual observations remain important for camera-centric reasoning. We acknowledge that training with synthetically generated simulation data introduces bias. We align the distribution of simulated sound-source states with that of real-world states. Empirically, the simulated data has been shown to endow the model with a fundamental capability for spatial audio perception. In addition, the experiments in Appendix A.5.1 suggest that, in the future, the training distribution could potentially be corrected using a small amount of real spatial-audio data. 5.2.2SPATIAL AUDIO NAVIGATION RESULTS Baseline Selection. SoundSpaces-based models, AVLEN, and RILA (Chen et al., 2020a; Paul et al., 2022; Yang et al., 2024) are originally designed for simulated acoustic observations and do not sup- port FOA input. To provide an informative comparison, we include the open-source SoundSpaces model and convert the FOA audio into binaural audio. Nevertheless, some navigation instructions in OmniEchoBench-Nav are conveyed through the spatial audio itself, requiring the agent not only to pursue a sound-emitting target but also to perform joint semantic understanding and spatial ground- ing. Therefore, the task formulations of these methods cannot be fully aligned with ours. Results on OmniEchoBench. Table 2 reports the results on the OmniEchoBench-Nav. Following text-guided models, we adopt the agent-stopping setting, in which each model autonomously decides when to terminate. OmniEcho achieves 16.2% SR and 11.5% SPL, substantially outperforming the conventional text-guided baselines Seq2Seq and CMA. Compared with InternVLA-N1 (Intern Robotics, 2025), OmniEcho is lower by 1.6 points in SR and 5.2 points in SPL. These results indicate that, without route-level textual guidance or goal images, sound-guided navigation in real-world en- vironments can still approach the performance of some traditional text-guided baselines. In contrast, SoundSpaces with binaural audio performs markedly worse in real spatial-audio settings and strug- gles to determine when to stop autonomously. For OmniEcho and other spatial-audio navigation models, efficient navigation and reliable stopping remain significant challenges. Results on VLN-CE R2R. Table 3 further reports results on the simulated VLN-CE R2R Val- Unseen split (Krantz et al., 2020). We again use agent stopping for the cross-model comparison. Our OmniEcho achieves an SR of 22.2%, slightly exceeding Seq2Seq and CMA (Krantz et al., 2020), although its SPL of 16.3% is below CMA (20.0%). Its NE of 7.31 is better than those of Seq2Seq and CMA, and its trajectory length of 5.54 is the shortest among methods with reported TL. Nevertheless, OmniEcho remains behind VLN-R1 (Qi et al., 2025), NAViLA-SAGE (Miao et al., 2026), and InternVLA-N1 (Intern Robotics, 2025) in SR and SPL. Overall, spatial-audio guidance is 8 Under review Table 4: Ablation of the native audio encoder, FOA encoder, and Stage 3 training strategy on OmniEchoBench-QA. The best overall performance is obtained by combining both encoders and freezing the FOA encoder during Stage 3. Native audio encoder FOA encoder FOA encoder (stage 3) Camera Rotation Source textbfDirection 3D Localization Source Motion Cognitive Map Overall ✓training37.521.613.219.545.226.2 ✓✗frozen43.716.214.216.321.819.8 ✗✓frozen40.118.913.621.047.226.2 ✓frozen41.722.614.224.747.528.5 ↓ Lower is better↓ Lower is better↓ Lower is better Figure 6: Ablation on input representation and Stage-1 pretraining. feat5+stage1 achieves the best azimuth, elevation, and distance errors, outperforming raw4+stage1 and feat5 (w/o stage 1 pretraining), showing that feature-based input is more effective than raw FOA waveform input and that Stage-1 pretraining further improves convergence and final accuracy. competitive with earlier text-guided VLN baselines in success rate, but a clear gap remains relative to stronger text-guided systems, especially in path efficiency. We provide additional analyses of navigation failure modes and the effect of ground-truth sound direction in Appendix A.5.2 and A.5.3. 5.3ABLATION STUDY Ablation on native audio encoder and FOA encoder. Table 4 ablates three design choices in Qwen3-Omni-30B-A3B (Xu et al., 2025b): introducing the FOA encoder, retaining the native au- dio encoder, and freezing the FOA encoder during Stage 3. The full configuration achieves the best Overall score of 28.5. Compared with the native-audio-only baseline, adding the frozen FOA encoder improves the Overall score and most subtasks, confirming that the FOA encoder provides essential spatial cues beyond those captured by the native audio pathway. The slight decrease on Camera Rotation suggests that additional visual observations reduce the model’s reliance on spatial audio cues. Removing the native audio encoder while retaining the frozen FOA encoder reduces the Overall score from 28.5 to 26.2, indicating that the native semantic audio tokens remain comple- mentary to the spatial representation. Finally, updating rather than freezing the FOA encoder during Stage 3 degrades every subtask, suggesting that freezing better preserves the spatial representations learned in the first two stages and avoids interference during multimodal instruction tuning. Ablation on input representation and Stage-1 pretraining. As shown in Figure 6, feat5+stage1 achieves the best performance, with 17.9° azimuth error / 7.70° elevation error / 1.13 distance er- ror, substantially outperforming raw4+stage1 (30.5° / 13.8° / 1.65). This suggests that, compared with directly modeling the raw FOA waveform, the pre-extracted 5-channel acoustic features pro- vide stronger localization cues. Moreover, comparing feat5 with feat5+stage1 shows that removing Stage-1 semantic alignment pretraining degrades performance, indicating that such initialization im- proves optimization and helps the model converge to a better solution. Further tests show that simply adding an FOA encoder, without Stage 1 and Stage 2 training, does not bring clear gains over the variant without the FOA encoder. This result highlights the necessity of dedicated pretraining to avoid interference during multimodal instruction tuning that jointly mixes planning and QA tasks. 6CONCLUSION We introduce OmniEchoBench, a unified real-world benchmark for spatial audio–visual perception and sound-guided navigation, together with a controllable spatial audio rendering pipeline for data generation. We further propose OmniEcho, an omni-modal model that integrates an FOA encoder with a pretrained semantic audio pathway to jointly capture auditory semantics and spatial infor- mation. Experiments demonstrate state-of-the-art performance in spatial perception and navigation 9 Under review performance close to that of traditional text-guided VLN methods. These results highlight the value of spatial audio for embodied scene reasoning and navigation, while identifying fine-grained spatial localization and distance estimation as key remaining challenges. We hope this work inspires future research on embodied agents that jointly reason about what they perceive and where it originates. AI USE STATEMENT In this work, we used generative AI tools for generating synthetic datasets. Specifically, generative AI assisted in constructing the simulated spatial audio–visual and sound-guided navigation training data, including the generation of synthetic scene descriptions, audio-visual media, sound events, and associated annotations. The corresponding data-generation and validation procedures are reported in the main text and Appendix A.1. We did not use generative AI tools for the other tasks with required disclosure, including developing the core scientific contributions, designing the research methodology or experiments, conducting experimental analysis, selecting or verifying references, or making scientific claims; the remaining required-disclosure tasks are not applicable to this work. Additionally, we used generative AI tools to aid or polish writing, including language editing, gram- mar correction, readability improvements, and suggestions of alternative wording. We also used generative AI tools for research ideation or execution, specifically to assist with writing code. We have reviewed all AI-assisted work. All AI-assisted text was carefully edited and approved by the authors; AI-assisted code was inspected and tested for correctness; synthetic data were checked us- ing the validation procedures described in the paper; and all technical content, claims, citations, and reported results were manually verified. We take responsibility for the final content of this work, in- cluding text, claims, code, data, annotations, and other artifacts produced with the aid of generative AI. REFERENCES Jont B Allen and David A Berkley. Image method for efficiently simulating small-room acoustics. The Journal of the Acoustical Society of America, 65(4):943–950, 1979. Peter Anderson, Qi Wu, Damien Teney, Jake Bruce, Mark Johnson, Niko S ̈ underhauf, Ian Reid, Stephen Gould, and Anton Van Den Hengel.Vision-and-language navigation: Interpreting visually-grounded navigation instructions in real environments. In 2018 IEEE/CVF conference on computer vision and pattern recognition, p. 3674–3683. IEEE, 2018. Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang, Haiyang Xu, and Junyang Lin. Qwen2.5-vl technical report, 2025. URL https://arxiv.org/abs/2502.13923. Subrata Biswas, Mohammad Khan, and Bashima Islam. Owl: Geometry-aware spatial reasoning for audio large language models. In International Conference on Learning Representations, volume 2026, p. 20685–20710, 2026. Changan Chen, Unnat Jain, Carl Schissler, Sebastia Vicenc Amengual Gari, Ziad Al-Halah, Vamsi Krishna Ithapu, Philip Robinson, and Kristen Grauman. Soundspaces: Audio-visual nav- igation in 3d environments. In European conference on computer vision, p. 17–36. Springer, 2020a. Changan Chen, Sagnik Majumder, Ziad Al-Halah, Ruohan Gao, Santhosh Kumar Ramakrishnan, and Kristen Grauman. Learning to set waypoints for audio-visual navigation. arXiv preprint arXiv:2008.09622, 2020b. Changan Chen, Ziad Al-Halah, and Kristen Grauman. Semantic audio-visual navigation. In 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), p. 15511–15520. IEEE, 2021. 10 Under review Changan Chen, Carl Schissler, Sanchit Garg, Philip Kobernik, Alexander Clegg, Paul Calamia, Dhruv Batra, Philip Robinson, and Kristen Grauman. Soundspaces 2.0: A simulation platform for visual-acoustic learning. Advances in Neural Information Processing Systems, 35:8896–8911, 2022. Jinyu Chen, Wenguan Wang, Si Liu, Hongsheng Li, and Yi Yang. Omnidirectional information gathering for knowledge transfer-based audio-visual navigation. In 2023 IEEE/CVF International Conference on Computer Vision (ICCV), p. 10959–10969, 2023. doi: 10.1109/ICCV51070. 2023.01009. Artem Dementyev, Wazeer Zulfikar, Sinan Hersek, Pascal Getreuer, Anurag Kumar, and Vivek Ku- mar. Phasecoder: Microphone geometry-agnostic spatial audio understanding for multimodal llms. arXiv preprint arXiv:2601.21124, 2026. Amala Sanjay Deshmukh, Kateryna Chumachenko, Tuomas Rintamaki, Matthieu Le, Tyler Poon, Danial Mohseni Taheri, Ilia Karmanov, Guilin Liu, Jarno Seppanen, Arushi Goel, et al. Nemotron 3 nano omni: efficient and open multimodal intelligence. arXiv preprint arXiv:2604.24954, 2026. Bhavika Devnani, Skyler Seto, Zakaria Aldeneh, Alessandro Toso, Elena Menyaylenko, Barry-John Theobald, Jonathan Sheaffer, and Miguel Sarabia. Learning spatially-aware language and audio embeddings. Advances in Neural Information Processing Systems, 37:33505–33537, 2024. Chuang Gan, Yiwei Zhang, Jiajun Wu, Boqing Gong, and Joshua B Tenenbaum. Look, listen, and act: Towards audio-visual embodied navigation. In 2020 IEEE International Conference on Robotics and Automation (ICRA), p. 9701–9707. IEEE, 2020. Junyu Gao, Xuan Yao, and Changsheng Xu. Fast-slow test-time adaptation for online vision-and- language navigation. arXiv preprint arXiv:2311.13209, 2023. Keji He, Ya Jing, Yan Huang, Zhihe Lu, Dong An, and Liang Wang. Memory-adaptive vision-and- language navigation. Pattern Recognition, 153:110511, 2024. Haodong Hong, Sen Wang, Zi Huang, Qi Wu, and Jiajun Liu. Navigating beyond instructions: Vision-and-language navigation in obstructed environments. In Proceedings of the 32nd ACM international conference on multimedia, p. 7639–7648, 2024. Hangrui Hu, Xinfa Zhu, Ting He, Dake Guo, Bin Zhang, Xiong Wang, Zhifang Guo, Ziyue Jiang, Hongkun Hao, Zishan Guo, et al. Qwen3-tts technical report. arXiv preprint arXiv:2601.15621, 2026. Jinbo Hu, Yin Cao, Ming Wu, Zhenbo Luo, and Jun Yang. Salm: Spatial audio language model with structured embeddings for understanding and editing. arXiv preprint arXiv:2507.16724, 2025. Chenguang Huang, Oier Mees, Andy Zeng, and Wolfram Burgard. Audio visual language maps for robot navigation. In International Symposium on Experimental Robotics, p. 105–117. Springer, 2023. Intern Robotics. InternVLA-N1: An open dual-system vision-language navigation foundation model with learned latent plans. Technical report, Intern Robotics and Shanghai AI Labora- tory, 2025. URL https://internrobotics.github.io/internvla-n1.github. io/static/pdfs/InternVLA_N1.pdf. Xilin Jiang, Hannes Gamper, and Sebastian Braun. Sci-phi: A large language model spatial audio descriptor. IEEE Open Journal of Signal Processing, 2026. Jacob Krantz, Erik Wijmans, Arjun Majumdar, Dhruv Batra, and Stefan Lee. Beyond the nav- graph: Vision-and-language navigation in continuous environments. In European Conference on Computer Vision, p. 104–120. Springer, 2020. Alexander Ku, Peter Anderson, Roma Patel, Eugene Ie, and Jason Baldridge. Room-across-room: Multilingual vision-and-language navigation with dense spatiotemporal grounding. In Proceed- ings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), p. 4392–4412, 2020. 11 Under review Xiulong Liu, Sudipta Paul, Moitreya Chatterjee, and Anoop Cherian. Caven: An embodied conver- sational agent for efficient audio-visual navigation in noisy environments. In Proceedings of the AAAI conference on artificial intelligence, volume 38, p. 3765–3773, 2024. Zhan Liu, Changli Tang, Yuxin Wang, Zhiyuan Zhu, Youjun Chen, Yiwen Shao, Tianzi Wang, Lei Ke, Zengrui Jin, and Chao Zhang. Jaeger: Joint 3d audio-visual grounding and reasoning in simulated physical environments. arXiv preprint arXiv:2602.18527, 2026. Zuyan Liu, Yuhao Dong, Jiahui Wang, Ziwei Liu, Winston Hu, Jiwen Lu, and Yongming Rao. Ola: Pushing the frontiers of omni-modal language model. arXiv preprint arXiv:2502.04328, 2025. Bingchen Miao, Rong Wei, Zhiqi Ge, Shiqi Gao, Jingzhe Zhu, Renhan Wang, Siliang Tang, Jun Xiao, Rui Tang, Juncheng Li, et al. Towards physically executable 3d gaussian for embodied navigation. In International Conference on Learning Representations, volume 2026, p. 22945– 22965, 2026. Christian Nachbar, Franz Zotter, Etienne Deleflie, and Alois Sontacchi. Ambix-a suggested am- bisonics format. In Ambisonics Symposium, volume 2011, 2011. Bowen Pan, Rameswar Panda, SouYoung Jin, Rogerio Feris, Aude Oliva, Phillip Isola, and Yoon Kim. Langnav: Language as a perceptual representation for navigation. In Findings of the Asso- ciation for Computational Linguistics: NAACL 2024, p. 950–974, 2024. Sudipta Paul, Amit Roy-Chowdhury, and Anoop Cherian. Avlen: Audio-visual-language embodied navigation in 3d environments. Advances in Neural Information Processing Systems, 35:6236– 6249, 2022. Zhangyang Qi, Zhixiong Zhang, Yizhou Yu, Jiaqi Wang, and Hengshuang Zhao. Vln-r1: Vision- language navigation via reinforcement fine-tuning. arXiv preprint arXiv:2506.17221, 2025. Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, p. 8748–8763. PmLR, 2021. S Sakshi, Vaibhavi Lokegaonkar, Neil Zhang, Ramani Duraiswami, Sreyan Ghosh, Dinesh Manocha, and Lie Lu. Spur: A plug-and-play framework for integrating spatial audio understand- ing and reasoning into large audio-language models. arXiv preprint arXiv:2511.06606, 2025. Manolis Savva, Abhishek Kadian, Oleksandr Maksymets, Yili Zhao, Erik Wijmans, Bhavana Jain, Julian Straub, Jia Liu, Vladlen Koltun, Jitendra Malik, et al. Habitat: A platform for embodied ai research. In 2019 IEEE/CVF International Conference on Computer Vision (ICCV), p. 9338– 9346. IEEE, 2019. Team Seedance, De Chen, Liyang Chen, Xin Chen, Ying Chen, Zhuo Chen, Zhuowei Chen, Feng Cheng, Tianheng Cheng, Yufeng Cheng, et al. Seedance 2.0: Advancing video generation for world complexity. arXiv preprint arXiv:2604.14148, 2026. Zhanbo Shi, Lin Zhang, Linfei Li, and Ying Shen. Towards audio-visual navigation in noisy environ- ments: A large-scale benchmark dataset and an architecture considering multiple sound-sources. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, p. 14673–14680, 2025. Edward C Tolman. Cognitive maps in rats and men. Psychological review, 55(4):189, 1948. Liuyi Wang, Zongtao He, Ronghao Dang, Mengjiao Shen, Chengju Liu, and Qijun Chen. Vision- and-language navigation via causal learning. In 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), p. 13139–13150. IEEE, 2024a. Liuyi Wang, Xinyuan Xia, Hui Zhao, Hanqing Wang, Tai Wang, Yilun Chen, Chengju Liu, Qijun Chen, and Jiangmiao Pang. Rethinking the embodied gap in vision-and-language navigation: A holistic study of physical and visual disparities. In 2025 IEEE/CVF International Conference on Computer Vision (ICCV), p. 9455–9465. IEEE, 2025. 12 Under review Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Junyang Lin. Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution, 2024b. URL https://arxiv.org/abs/2409. 12191. Shuo Wang, Yucheng Wang, Guoxin Lian, Yongcai Wang, Maiyue Chen, Kaihui Wang, Bo Zhang, Zhizhong Su, Yutian Zhou, Wanting Li, et al. Progress-think: Semantic progress reasoning for vision-language navigation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 4076–4086, 2026. Zehao Wang, Minye Wu, Yixin Cao, Yubo Ma, Meiqi Chen, and Tinne Tuytelaars. Navigating the nuances: A fine-grained evaluation of vision-language navigation. In Findings of the Association for Computational Linguistics: EMNLP 2024, p. 4681–4704, 2024c. Zun Wang, Jialu Li, Yicong Hong, Yi Wang, Qi Wu, Mohit Bansal, Stephen Gould, Hao Tan, and Yu Qiao. Scaling data generation in vision-and-language navigation. In 2023 IEEE/CVF International Conference on Computer Vision (ICCV), p. 11975–11986. IEEE, 2023. Meng Wei, Chenyang Wan, Xiqian Yu, Tai Wang, Yuqiang Yang, Xiaohan Mao, Chenming Zhu, Wenzhe Cai, Hanqing Wang, Yilun Chen, et al. Streamvln: Streaming vision-and-language navi- gation via slowfast context modeling. arXiv preprint arXiv:2507.05240, 2025. Meng Wei, Chenyang Wan, Peng Peng, Xiqian Yu, Yuqiang Yang, Delin Feng, Wenzhe Cai, Chen- ming Zhu, Tai Wang, Jiangmiao Pang, et al. Ground slow, move fast: A dual-system foundation model for generalizable vision-language navigation. In International Conference on Learning Representations, volume 2026, p. 12380–12396, 2026. Kevin Wilkinghoff and Zheng-Hua Tan. Dspast: Disentangled representations for spatial audio reasoning with large language models. In ICASSP 2026-2026 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), p. 14747–14751. IEEE, 2026. Jie Wu, Chunlei Wu, Xiuxuan Shen, Fengjiang Wu, and Leiquan Wang. Recursive bidirectional cross-modal reasoning network for vision-and-language navigation. Expert Systems with Appli- cations, 270:126442, 2025. Jin Xu, Zhifang Guo, Jinzheng He, Hangrui Hu, Ting He, Shuai Bai, Keqin Chen, Jialin Wang, Yang Fan, Kai Dang, Bin Zhang, Xiong Wang, Yunfei Chu, and Junyang Lin. Qwen2.5-omni technical report, 2025a. URL https://arxiv.org/abs/2503.20215. Jin Xu, Zhifang Guo, Hangrui Hu, Yunfei Chu, Xiong Wang, Jinzheng He, Yuxuan Wang, Xian Shi, Ting He, Xinfa Zhu, et al. Qwen3-omni technical report. arXiv preprint arXiv:2509.17765, 2025b. Zeyuan Yang, Jiageng Liu, Peihao Chen, Anoop Cherian, Tim K Marks, Jonathan Le Roux, and Chuang Gan. Rila: Reflective and imaginative language agent for zero-shot semantic audio- visual navigation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 16251–16261, 2024. Hanrong Ye, Chao-Han Huck Yang, Arushi Goel, Wei Huang, Zhen Wan, Jinchuan Tian, An-Chieh Cheng, Ligeng Zhu, Yuanhang Su, Yuming Lou, et al. Omnivinci: Enhancing architecture and data for omni-modal understanding llm. In International Conference on Learning Representa- tions, volume 2026, p. 56101–56138, 2026. Yuhuan You, Lai Wei, Xihong Wu, and Tianshu Qu. The world is not mono: Enabling spatial understanding in large audio-language models. arXiv preprint arXiv:2601.02954, 2026. Abdelrahman Younes, Daniel Honerkamp, Tim Welschehold, and Abhinav Valada. Catch me if you hear me: Audio-visual navigation in complex unmapped environments with moving sounds. IEEE Robotics and Automation Letters, 8(2):928–935, 2023. Aohan Zeng, Zhengxiao Du, Mingdao Liu, Kedong Wang, Shengmin Jiang, Lei Zhao, Yuxiao Dong, and Jie Tang. Glm-4-voice: Towards intelligent and human-like end-to-end spoken chatbot, 2024. URL https://arxiv.org/abs/2412.02612. 13 Under review Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. In 2023 IEEE/CVF International Conference on Computer Vision (ICCV), p. 11941–11952. IEEE, 2023. Zhisheng Zheng, Puyuan Peng, Ziyang Ma, Xie Chen, Eunsol Choi, and David Harwath. Bat: Learn- ing to reason about spatial sounds with large language models. arXiv preprint arXiv:2402.01591, 2024. Gengze Zhou, Yicong Hong, and Qi Wu. Navgpt: Explicit reasoning in vision-and-language naviga- tion with large language models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, p. 7641–7649, 2024. Zhiyuan Zhu, Yixuan Chen, Yiwen Shao, Wenxiang Guo, Changhao Pan, Yu Zhang, Yuxiang Wang, Wei Liu, Houhua Zhang, Chengkuan Zeng, et al. Spatial-omni: Spatial audio understanding integration in multimodal llms via foa encoding. arXiv preprint arXiv:2606.10738, 2026. Franz Zotter and Matthias Frank. Ambisonics: A practical 3D audio theory for recording, studio production, sound reinforcement, and virtual reality. Springer Nature, 2019. 14 Under review AAPPENDIX A.1DATA GENERATION DETAILS A.1.1SPATIAL AUDIOVISUAL SIMULATION FOR DATA SYNTHESIS. 051015202530 Share of specified-source questions (%) Impact / handling Other / misc Alarm / alert / electronic Human / voice Machinery / motor Music / media Footsteps / locomotion Ambient / material Animal Rolling / vehicle / cart Water / liquid Tools / craft 16 20 8 7 8 5 5 7 5 4 8 7 27 16 27 8 4 4 5 3 3 4 (a) Sound-source type distribution trainset OmniEchoBench-QA 02468 Source distance (m) 0.00 0.05 0.10 0.15 0.20 0.25 0.30 Density KS = 0.20 median 2.8 = 2.8 m (b) Sound-source distance distribution trainset (μ=3.51 m, n=15,914) OmniEchoBench-QA (μ=3.01 m) moving source static snapshot detect / count / ID 0 20 40 60 80 Share of questions (%) 5 60 35 20 80 0 (c) Sound-source motion / task type trainset OmniEchoBench-QA Figure 7: Distributional alignment between our synthetic training corpus (trainset, 145 103 sam- ples; blue) and the real-recorded OmniEchoBench-QA benchmark (orange), across three axes. (a) Sound-source type. Normalised over specified-source questions, the two corpora now share a comparable semantic mix (JSD = 0.14 bits); the residual gap is the benchmark’s heavier empha- sis on alarm/electronic tones, against the trainset’s broader coverage of machinery, tools, and water sounds. (b) Source distance. The two distance distributions now largely overlap, with an identical median of 2.8 m and a Kolmogorov–Smirnov statistic of D = 0.20. (c) Motion / task type. We further augment the training set with non-spatial detect/count/identify tasks, aiming to improve the model’s overall capability in spatial-audio perception. We first construct spatial audiovisual QA data from generated videos. A large language model (LLM) acts as a director: given previously generated scene descriptions to encourage diversity, it produces a structured JSON scene containing N foreground sources and one ambient source. For each source, the scene specifies a semantic label, target loudness (LUFS), onset and offset times, and a full trajectory(t k ,θ k ,φ k ,r k ) k . The same call also emits a full-scene video prompt describing all sounding entities and camera behavior, together with one isolated close-up prompt per source that depicts only that sounding body in an otherwise silent environment. The director is driven by the system prompt shown in the box below (translated from the original and abridged; diversity, visibility-dynamics, and fine-grained grid/angle rules are elided with [...]). A set of auxiliary LLM agents subsequently refines the proposed scene—adjusting per-source onset/offset timing and target loudness, and resolving textual–geometric direction/distance contradictions—before any me- dia are generated. Scene director — system prompt (translated, abridged) You are a Spatial Audio-Video Scene Director. Freely design one creative spatial-audio scene and write a matching video-generation script (prompt). The scene contains N_FOREGROUND foreground sources (e.g. a dripping tap, a cat, footsteps), role="foreground", and exactly 1 ambient/background source (e.g. rain, wind, traffic), 15 Under review role="ambient". COORDINATES AND SPATIAL LAYOUT - Right-handed frame; the listener faces +y. azimuth in [-180,180], 0 = straight ahead, +90 = left, -90 = right (+left / -right convention). elevation in [-90,90], + = up. distance in [0.5m, 15m]. - The listener sits at the origin, position=[0,0,0]. - Each source is described by a trajectory (>=2 timestamped points); speed between adjacent points < 10 m/s. Initial azimuths of different sources should be >= 30 deg apart. CAMERA / LISTENER ORIENTATION - The camera is bound to the listener; horizontal FOV ̃= 90 deg (half-FOV 45 deg). - listener.orientation.yaw is the camera azimuth and may vary over time via a trajectory. - Visibility: camera_relative_azimuth(t) = object_azimuth(t) - listener_yaw(t), normalized to [-180,180]; the source is visible when |...| < 45 deg. Each source outputs a visibility_window video_enter_t, video_exit_t. PHYSICAL-CONSISTENCY RULE FOR CAMERA ROTATION (critical) - yaw increasing to positive => "camera rotates left"; decreasing to negative => "rotates right". - When the camera turns toward X, the centered object must slide out toward not-X while a new object on the X side enters from the X edge to the center; violating this makes the video model produce reversed or frozen motion. [... do/don’t examples omitted ...] OUTPUT FORMAT (strict) -- output ONE JSON object only, no markdown, no explanation: "scene": "scene_id", "duration", "environment": "type":"indoor|outdoor", "room":"dimensions":[w,l,h],"rt60":0.3-0.8, "description", "listener": "position":[0,0,0], "orientation":"yaw","pitch","roll","trajectory":["t","yaw", ...], "objects": [ "id","type","label","role","loudness_lufs","snr_db","onset_t","offset_t", "trajectory":["t","azimuth","elevation","distance", ...], "visibility_window":"video_enter_t","video_exit_t", "wav_path" ] , "sub_prompts": [ "close-up video prompt per source (silent env, only that source sounding )", ...], "video_prompt": "full-scene prompt (3 parts: scene description; camera motion; METADATA appendix)" LOUDNESS: near (<2m) foreground -14..-18 LUFS; mid (2-5m) -20..-26; far/ambient -30..-40; ̃6 dB per distance doubling. snr_db is relative to the quietest (ambient=0 dB) source. TIMING: at least one foreground onset_t=0; continuous/ambient sources span the whole clip. SUB-PROMPTS: one close-up per source showing only that sounding body in an otherwise silent but reverberant environment; on-screen direction must match the source azimuth sign; static sources must be told to stay still. DIVERSITY: vary environment (indoor/outdoor, city/nature/home/industrial, day/night, quiet/ noisy) and source types across scenes; do not repeat similar scenes. [... extensive diversity, dynamic-visibility, and grid/angle-annotation rules omitted ...] Before media generation, we validate the proposed scene for source counts, timing, spatial ranges, velocities, and visibility intervals. Invalid scenes are regenerated. We further compare directional and distance language in the prompts against the numerical trajectories, and a second LLM revises inconsistent descriptions. This checking–revision loop is repeated for up to three rounds. Additional planning adjusts source activity schedules and target loudness so that ambient sounds span the scene, transient events occur at plausible times, and foreground sounds remain audible over the background. We then submit the prompts to Seedance (Seedance et al., 2026). The full-scene prompt yields the rendered video V , while each close-up prompt yields an isolated clip whose audio track is extracted as that source’s monophonic stem s i . Stems are trimmed or loop-extended to the intended duration, loudness-normalized to their target LUFS, and gated or faded before rendering. Because the director’s trajectories are only approximately consistent with the generated pixels, we recover source geometry directly from the rendered video. Under a static-camera assumption veri- fied by dense optical flow, objects are localized by open-vocabulary text-prompted segmentation and tracked across frames, and each sounding source is matched to a detected visual instance by label. 16 Under review For a detected center pixel (c x ,c y ) in a frame of size W × H , we recover the viewing direction under a pinhole camera model. Let f = W/2 tan(FOV/2) denote the horizontal focal length and f v the vertical focal length. We first compute normalized image-plane coordinates u = W/2− c x f , v = H/2− c y f v ,(2) and then recover azimuth and elevation as θ = atan2(u, 1), φ = atan2 v, p 1 + u 2 .(3) Monocular depth on key frames provides relative range estimates, which are anchored to the direc- tor’s scene scale to obtain metric distance. The resulting trajectories are smoothed and resampled at a fixed rate. Sources never observed on screen retain their scripted trajectory but are forced outside the FOV; dynamically exiting sources are linearly extrapolated; and disagreements between detected instance counts and the script trigger instance splitting or masking. This yields a corrected scene in which every source has a per-timestep (θ,φ,r) trajectory together with an explicit visibility state. Starting from the corrected scene, we augment the sample with additional off-screen events and camera motion. On-screen sources are kept unchanged, while 2–4 off-screen sources are appended at azimuths guaranteed to remain outside the possibly rotating FOV. We also account for sources entering or leaving the image and simulate their off-screen continuation. Each appended source follows a world-frame trajectory chosen from a static location, a straight walk, or a polyline with a single turn of at least 90 ◦ , and is voiced by a borrowed stem from a cross-sample library after loudness normalization. A camera yaw trajectory ψ(t) is written into the scene so that the rendered audio rotates consistently with the cropped or panned image. Given the final augmented scene, all stems are rendered into FOA using Eq. equation 1, with op- tional indoor reflections, and the resulting multichannel audio is muxed into the cropped or panned video without re-encoding the visual stream. We then generate QA pairs directly from the final trajectories rather than from an LLM response. At each queried time, we compute a source’s world- frame direction, its direction relative to the current camera, its distance, and whether it lies inside the camera frustum; over intervals, we compute changes from the same trajectories. From these quanti- ties we instantiate four categories of questions—direction (coarse 8-way and fine-grained degrees), motion (world heading and change), 3D localization, and camera rotation—covering eight subtypes across camera, camera-body, and world coordinate frames. Each video contributes 15–20 questions, with source and time selection balanced to cover both binary outcomes and the full set of direction labels. This process yields 26,585 training questions from 1,484 synthesized videos and the data distribution is shown in Figure 7. Audio-only Spatial Data We also synthesize spatial-audio questions without visual input. Each scene contains one to three independently generated sounds. We assign every source a time inter- val and either a static, constant-speed, accelerating, or decelerating 3D trajectory, then render the resulting mixture as FOA. Because the trajectories are known, we can directly ask about direction, elevation, distance, source count, motion, speed change, approach or recession, closest approach, and relations between sources. An LLM rewrites the questions and expresses the trajectory-based reasoning in natural language, while the answers remain fixed by the rendered scene. We obtain 18,518 questions from 2,000 moving-source scenes. The same single-source recordings are reused to construct 100,000 simpler static scenes for spatial- audio pretraining. Each clip contains one to four sound categories placed at independently sam- pled azimuths, elevations, and distances. These clips support the semantic-alignment and query- conditioned localization stages of the FOA encoder described in Section 4. A.1.2SOUND-GUIDED NAVIGATION DATA SYNTHESIS We convert four existing VLN corpora—R2R, RxR, VLN-PE, and ScaleVLN—into sound-guided navigation supervision through a six-stage pipeline: (i) destination sound-event proposal, (i) audio synthesis, (i) FOA rendering and sample construction, (iv) acoustic augmentation, (v) omni-modal SFT formatting, and (vi) chain-of-thought (CoT) annotation. 17 Under review Destination sound-event proposal. For each episode, a vision–language model (Qwen3.7-flash) observes the last five first-person frames captured at the destination (front camera, 1.25 m height) and proposes a plausible sound event, under a hard constraint that neither the sound content nor the derived task may reveal the destination location, room name, or nameable furniture, so that the agent must localize the target acoustically. Proposals are balanced roughly 50/50 between two categories: (1) a short spoken utterance (a person calling the robot, with diverse intents and an accompanying voice-style description of gender, age, timbre, emotion, and pace), and (2) a non-speech ambient event (e.g., knocking, running water, dragging furniture, an animal, a phone, or an appliance). The model returns a structured record containing the sound content, a natural-language task description, the voice/acoustic style, and its scene reasoning. The system prompt is reproduced verbatim below. Destination sound-event proposal — system prompt You are a scenario designer for an indoor "sound-source navigation" task. A robot starts somewhere in a house and must reach a specific target object/location ONLY by following a sound emitted from that target (plus a short text instruction). You are shown the LAST few first- person camera frames of a navigation episode; these frames show the destination area (the target). Your job: invent a plausible sound event that would emit from this destination, so the robot can localize it acoustically. Design two categories of scripts; pick ONE per episode (respect the requested category when given, and keep the two categories roughly balanced across the dataset): Category 1 - "spoken utterance that calls the robot over": the sound itself is a short spoken English utterance from a person at the destination. This is NOT limited to emergencies or calls for help - ANY natural reason a person would summon the robot is valid, and you should keep them DIVERSE across episodes, e.g.: asking for help ("Robot, I fell down, please come and find me."), requesting an item be fetched or carried, inviting it to come look at something, assigning a task, a casual/curious call, a playful summon, an impatient shout, etc. Vary the speaker, wording, tone and intent widely. The text_instruction can briefly restate the intent (e.g. "Go to the person who is calling you."). For Category 1 you MUST also fill "voice_style": one English sentence describing the SPEAKER’S VOICE for downstream TTS - gender, approximate age, timbre, emotion/tone, speaking pace , and optionally accent. Make these rich and varied across episodes. Category 2 - "ambient sound source + text command": the sound is a common non-speech sound believable in this room (knocking, running water / flushing, dragging furniture, a pet - dog barking / cat meowing, human chatter, music, TV, kettle whistle, phone ringing, etc.). The text_instruction is a short English command telling the robot what to home in on, e.g. "Find where the knocking sound is coming from." For Category 2, set "voice_style" to a short English description of the sound’s acoustic character or "" if not applicable. Use the visible scene ONLY to make the sound realistic (bathroom -> water/flush; kitchen -> clattering dishes / kettle; living room -> TV / music; bedroom -> alarm clock / phone). HARD CONSTRAINTS: - NEVER reveal the destination’s location, room name, or nameable furniture in either sound_content or text_instruction. The robot must find it by sound, not by being told where it is. (Do NOT say things like "in the bathroom" / "near the toilet".) - sound_content, text_instruction and voice_style must be in ENGLISH. - Keep each field to one short natural sentence. Return ONLY a JSON object, no markdown fences, with exactly these keys: "category": 1 or 2, "sound_content": "<the audio content: cat.1 = the spoken line; cat.2 = a short sound description>", "text_instruction": "<short English instruction given to the robot>", 18 Under review "voice_style": "<cat.1 = the speaker’s voice; cat.2 = the sound’s acoustic character or empty>", "scene_reasoning": "<one short sentence: why this sound fits what you see>" Table 5: Audio synthesis models used for sound-guided navigation data. Sound categorySynthesizerKey settings Spoken utteranceCustom-voice TTS (Qwen3-TTS) (Hu et al., 2026)7 speakers (4 F / 3 M), style-conditioned,≤ 2048 tokens Non-speech eventSeedance (Seedance et al., 2026)clip 10 s Table 6: Rendering and augmentation configuration for sound-guided navigation data. ComponentSetting Source placementepisode endpoint (center of final destination view) Distance attenuationinverse distance∝ 1/r, floor r min = 0.3 m, no peak normalization FOA formatfirst-order ambisonics, ACN/SN3D, 4 channels, order [W,Y,Z,X] Listener / camera height1.25 m History framesup to 8 (uniformly sampled) WaypointsK = 8, [x fwd , y left , ψ] relative to current pose Room reverberationR2R / RxR: image-source early reflections + cross-room attenuation (MP3D) ScaleVLN / VLN-PE: direct-path propagation only Directional noise (optional)1–3 plane-wave interferers (pink/white); el.±45 ◦ ;≥ 30 ◦ from source Noise SNRsampled from20, 15, 10, 5, 0 dB w.r.t. the W channel Audio synthesis. Spoken utterances are synthesized with a custom-voice text-to-speech model con- ditioned on the voice-style description, and non-speech events with Seedance, as summarized in Table 5. FOA rendering and sample construction. The proposed source is placed at the episode endpoint (the center of the final destination view). At every decision frame, we transform the endpoint into the listener’s current camera frame to obtain its forward/left/up direction components and distance r; the mono stem is inverse-distance attenuated (∝ 1/r, floored at r min = 0.3 m, with no per-clip peak normalization so that loudness preserves a distance cue) and encoded into 4-channel FOA via Eq. equation 1. Trajectory targets are the next K = 8 waypoints [x fwd , y left , ψ] (forward metres, left metres, yaw radians) expressed relative to the current pose. Key constants are listed in Table 6. Acoustic augmentation. For corpora with room geometry (R2R and RxR on Matterport3D), the shared renderer adds first-order image-source early reflections and attenuates direct sound across rooms (Allen & Berkley, 1979), and RxR further adds low-level directional background noise; geometry-free corpora (ScaleVLN on HM3D, VLN-PE) use direct-path propagation with distance attenuation only. On top of this, we provide an optional directional-noise augmentation that in- jects 1–3 plane-wave interferers (pink or white noise) at random azimuths (elevation within±45 ◦ , at least 30 ◦ from the true source direction), mixed at a signal-to-noise ratio sampled per clip from 20, 15, 10, 5, 0 dB referenced to the omnidirectional (W ) channel (Table 6). Omni-modal SFT format and CoT. Each SFT example interleaves up to eight uniformly sam- pled history frames, the current frame, and the co-located 4-channel FOA clip, followed by the task text; the target is the K = 8 waypoint list. A vision–language model then writes a first-person CoT rationale conditioned on the ground-truth source bearing (bucketed into eight directions with 22.5 ◦ /67.5 ◦ /112.5 ◦ /157.5 ◦ edges), distance (near < 1.5 m, far > 4 m), and two trajectory-derived meta-actions, without altering the trajectory targets. We additionally construct in-place turning ex- amples (sources to the side or behind the agent). Active-stop supervision converts stationary end- of-episode waypoints into a fixed eight-slot trajectory carrying a majority of <stop> tokens (≥ 5 of 8). The eight-way direction bucket, the near/far distance bucket, and the two trajectory-derived meta-actions are computed deterministically from geometry and injected into the CoT prompt as ground truth; the prompt template is shown below. 19 Under review Navigation chain-of-thought — system and task prompt [SYSTEM] You are writing first-person chain-of-thought (CoT) reasoning traces AS a sound-source navigation robot. You (the robot) hear a spatial-audio cue and must move toward the sound source while avoiding obstacles. You are given YOUR OWN recent egocentric RGB frames (oldest to newest, the last one is your current view), the task the sound is asking you to do, the GEOMETRICALLY COMPUTED direction and distance of the sound source relative to you, the ground-truth meta-action sequence, and the ground-truth future trajectory. Write a single coherent, physically plausible reasoning trace, STRICTLY IN THE FIRST PERSON (use ’I’, ’me’, ’my’, ’ahead of me’; NEVER say ’the robot’). The computed direction/distance and the meta-action sequence are the SOURCE OF TRUTH: never contradict them. Describe the visual scene ONLY from what you actually see in the frames. [TASK] Write a first-person chain-of-thought reasoning trace for my next navigation step. === Task the sound is asking me to do === instruction === Computed sound-source cue (ground truth -- do NOT contradict) === - Direction of the sound source relative to me: direction (azimuth az:+.0f deg, where + is to my left, 0 is straight ahead) - Distance: dist:.1f m (dist_bucket) - Elevation: el:+.0f deg (roughly at my ear level unless large) === Ground-truth motion (derived from the future trajectory -- do NOT contradict) === - My two main meta-actions (first half of my path, then second half): meta_seq - Net motion over this horizon: net_fwd:+.2f m forward, net_left:+.2f m to my left; my heading changes by net_yaw:+.0f deg. - Overall travel direction of my path: path_dir - deviation_note === What to write === Produce ONE first-person reasoning trace (a few sentences, natural English, no bullet headers, no markdown) that flows through these stages IN ORDER: 1. SCENE: briefly describe what I see in front of me / around me in my current view. 2. INSTRUCTION: restate, in one clause, what the sound is asking me to do. 3. SOUND DIRECTION: state where I hear the sound source (direction, dist_bucket) and tie it to what I see. 4. PLAN: give a short high-level plan that reconciles what I see (obstacles/openings) with where I hear the sound. If my path direction differs from the straight line to the source, explain it as going around an obstacle or leaving my current room; if the source is far, consider it may be in another room. 5. DECISION: conclude in ONE short first-person sentence that commits to my two sequential moves -- what I do over the first half of my path, then the second half -- consistent with: meta_seq. Do NOT write a "Meta actions:" label, do NOT enumerate sub-moves, and do NOT output coordinates. Write everything in the FIRST PERSON. Do NOT refer to "the robot" in the third person. Do NOT output the numeric trajectory or any coordinates. Stop at the meta-action level. Output STRICT JSON and nothing else, exactly: "cot": "<the first-person reasoning trace as a single string>" A.2BENCHMARK DETAILS A.2.1OMNIECHOBENCH-QA: REAL-WORLD CAPTURE AND ANNOTATION Capture rig and formats. Each script is performed and recorded live, producing three synchronized streams in a single take: a first-person main-view video (1920 px wide, 30 fps), a 360 ◦ panoramic 20 Under review ahead ahead- right right behind- right behind behind- left left ahead- left n = 1174 questions · 8-way bearing Sound-source direction (camera-body frame) 173 220 107 132 94 129 151 168 012345678 Distance to sound source (m) 0 20 40 60 80 Number of questions median 2.8 m n = 521 · range 0.2–7.1 m · 0.5 m bins Sound-source distance 2 13 74 51 84 45 59 39 48 31 40 25 6 22 SVABench-QA — spatial distribution of sound sources Figure 8: Direction and Distance distribution of OmniEchoBench-QA. Direction — coarse At t=6.5 s, in the CAMERA-BODY frame (relative to where the camera is currently pointing), in which approximate direction is the source of a rustling plastic sound? GT directly to the left Direction — degrees At t=29.0 s, by how many degrees and in which direction must the camera rotate (left or right, about the vertical axis) so that the source of music becomes centred in its view (body azimuth 0)? GT 101 degrees to the right 3D localization At t=6.5 s, give the position of the source of a rustling plastic sound in the CAMERA-BODY frame (relative to where the camera is currently pointing) as an approximate direction plus a distance in metres (e.g. 'ahead and to the right, about 2 m'). GT directly to the left, about 1.9 m Source motion — heading In the WORLD frame (fixed to the camera's initial heading; X = front, Y = left), what is the overall direction of travel of the source of a device alert tone from t=0.5 to t=7.0 s? GT directly to the right Source motion — change In the WORLD frame, does the direction of motion of the source of a metallic clinking sound change significantly (by more than ~45 deg) at any point during its trajectory? GT yes Camera rotation — degrees Between t=0.0 and t=14.0 s, by how many degrees and in which direction (left or right) did the camera rotate about its vertical axis? GT 94 degrees to the right Camera–source consistency Between t=0.5 and t=30.0 s, does the camera turn in the same overall direction (left or right) that the source of a rattling sound of a shaken bottle travels around the camera? GT yes Top-down localization (A/B/C/D) Four candidate locations are marked A, B, C and D in the top-down view. The sound source is: "a mechanical appliance sound". GT B Figure 9: Representative examples from OmniEchoBench-QA, covering eight spatial reasoning question types. video, and a four-channel FOA audio track recorded at 48 kHz. The FOA audio is stored with the omnidirectional channel first, and its horizontal axes are aligned so that the front direction corre- sponds to the +X axis and the listener’s left to the +Y axis. The 197 scripts are distributed across small indoor (50), medium indoor (50), large indoor (77), and open outdoor (20) settings. Annotation. Using the panoramic recording as ground-truth reference, annotators label the real- time position of every active sound source at 2 Hz (every 0.5 s), together with per-timestep flags for whether the source is visible in the main view, visible in the panorama, and audible. The panorama supplies the ”present but off-screen” evidence that the main view cannot, and is used only during annotation. Two capture regimes are used: a camera-fixed regime (the original FOA is kept) and a camera-moving regime, in which the camera yaw is recovered by optical flow and calibrated against annotated rotation anchors, and the FOA is rotated into the current camera frame so that audio and 21 Under review Table 7: OmniEchoBench-QA temporal-reasoning categories, subtypes, coordinate frames, and counts (from spatialqatest). The four geometric categories (2,372 QA) are reported in the main paper; Visibility is additionally annotated. CategorySubtype(s)FrameCount Source Directionapprox. 8-way / degreescamera-body1,069 3D Localizationbearing + distancecamera-body521 Source Motionworld heading / changeworld473 Camera Rotationdegrees / consistencycamera / mixed309 image remain aligned. At test time, the model receives the main-view video (or a single frame) muxed with the processed four-channel FOA. Question construction. All ground-truth answers are computed deterministically from the anno- tated trajectories rather than from a language model: at a queried time we compute the source’s world and camera-body bearing, its distance, and its visibility; over an interval we compute changes from the same trajectories. Answers are short phrases drawn from closed buckets (eight-way di- rection words, N degrees to the left/right, yes/no, or a direction plus a metric distance), and per- (category, subtype) answer-bucket balancing is applied. The resulting temporal-reasoning cate- gories and counts are listed in Table 7. The main paper reports the four geometric categories (Direction, 3D Localization, Motion, Camera Rotation; 2,372 QA); a Visibility category is addi- tionally annotated and available. The distribution of direction and distance is shown in Figure 8. The Cognitive Map subset (600 QA) is built on the navigation scenes below. Example QAs from OmniEchoBench-QA are shown in Figure 9. Coordinate conventions. The world frame is fixed to the camera’s initial heading, with +X ahead and +Y to the left; the camera-body frame at time t rotates with the recovered camera yaw θ(t), and body azimuth equals world azimuth minus θ(t). Azimuth is measured counter-clockwise (0 ◦ ahead, 90 ◦ left, 180 ◦ behind, 270 ◦ right). A.2.2OMNIECHOBENCH-NAV: REAL-WORLD SCENE AND ACOUSTIC-FIELD COLLECTION Scenes. The benchmark contains 30 real-scanned indoor environments, split into 10 single-room and 20 multi-room (cross-room) layouts spanning two to four connected rooms. Each scene is provided as a textured mesh (.glb) together with a top-down render and a pixel-to-mesh affine calibration; walkability, collision checks, and shortest-path planning are provided by an occupancy- grid navmesh derived from the mesh. Acoustic field. Each scene contains 30 navigation samples (900 in total). In every sample a single mono source is placed at a known 3D position, and its signal is captured at a dense grid of 12–17 receiver (”microphone”) positions on an approximately 1 m grid at ∼ 1 m height, each stored as a world-aligned four-channel FOA recording (ACN/SN3D, order [W,Y,Z,X], 48 kHz), forming a spatially dense acoustic field of the environment. Every receiver’s coordinates are stored explicitly. Sources span non-speech environmental events (505 samples) and spoken commands (395 samples), across a set of playback devices; each sample is annotated with the source device, source room, semantic description, spoken-command text (when applicable), speaker description, and a natural- language navigation-target description. Table 8 summarizes the composition. Test-time perception. At each step, the agent is served the FOA recording from the receiver nearest to its current position. Although the recordings are stored in a world-aligned coordinate frame, their horizontal directional components are rotated into the agent’s current egocentric frame according to the camera yaw before being passed to the model. The visual observation is rendered from the same pose and orientation (+x forward, 90 ◦ horizontal FOV, 15 ◦ downward pitch), ensuring that the FOA forward direction remains aligned with the camera view as the agent turns. The Cognitive Map QA subset (600 four-option A/B/C/D questions) is built on these scenes: it pairs the egocentrically aligned nearest-receiver FOA with a top-down map on which the true source and three distractor floor points (each≥ 40 ◦ apart in bearing) are marked, and asks which marked location is the source. In the ego-up variant the map is rotated so image-up is the robot’s forward direction and the robot is centered. 22 Under review Table 8: OmniEchoBench-Nav composition. 30 scenes × 30 samples = 900 navigation samples; each sample provides a dense FOA receiver grid. PropertyValue Scenes30 (10 single-room, 20 multi-room / 2–4 rooms) Navigation samples900 (30 per scene) Receivers per sample12–17 (dense∼ 1 m grid,∼ 1 m height) Source typesnon-speech 505, spoken command 395 FOA format4-channel, ACN/SN3D [W,Y,Z,X], 48 kHz, world-aligned Scene assetstextured .glb mesh + top-down render + occupancy navmesh Cognitive-Map QA600 four-option (A/B/C/D) questions over all 30 scenes Table 9: FOA encoder architecture (∼ 11.9 M parameters). The encoder is shared across all three training stages and frozen at Stage 3. ModuleConfiguration Input feature map5× 128× T : log-mel of W + DirAC intensity (i x ,i y ,i z ) + diffuseness STFT front-end16 kHz, FFT 400, hop 160, 128 mel bins Conv. frontend3× Conv2d (stride 2, GELU), 8× time downsampling (∼ 12.5 tokens/s) Transformer5 layers, d model = 384, 6 heads, FFN 1536, pre-norm, sinusoidal PE Semantic head (Stage 1)learnable-query attention pooling→ 512-d clip embedding Text encoder (frozen)CLIP ViT-B/16 (Radford et al., 2021), 512-d joint space, C = 644 sound labels Localization head (Stage 2)query-conditioned cross-attention→ (az, el, dist) A.2.3DATA PRIVACY AND DE-IDENTIFICATION Because OmniEchoBench contains audiovisual recordings captured in real-world environments, we process the data to minimize the disclosure of personally identifiable information before any sample is included in the benchmark or prepared for release. In all visual streams, every visible face and human figure, including incidental bystanders, is mosaic-blurred. Only these de-identified visual streams and privacy-preserving audio recordings are included in the benchmark and prepared for release. The released data are intended for research on spatial audio–visual perception and naviga- tion, not for identity recognition, person re-identification, speaker identification, or other biometric analysis. A.3TRAINING DETAILS OmniEcho is trained in three stages: two stages pretrain the standalone FOA encoder (semantic alignment, then query-conditioned localization), and a third stage integrates the frozen encoder into the Qwen3-Omni backbone via a trainable projector. The FOA encoder is a compact (∼ 11.9 M- parameter) convolution–Transformer network; its architecture is summarized in Table 9. Stage 1: Semantic alignment of the FOA encoder. We first pretrain a lightweight FOA audio encoder f θ that maps an FOA audio clip to a temporal sequence of d-dimensional tokens (d = 384). Each clip is converted into a 5-channel input map X ∈R 5×128×T comprising the log-mel spec- trogram of the omnidirectional (W ) channel together with the three active-intensity components (i x ,i y ,i z ) and the diffuseness δ, all in the ACN/SN3D convention. A convolutional frontend down- samples time by 8× (yielding ∼12.5 tokens/s), and a 5-layer Transformer produces token embed- dings. An attention-pooling semantic head aggregates these tokens into a clip embedding a∈R 512 . To inherit an open-vocabulary sound-semantic space, we align the clip embedding with frozen CLIP text embeddings using the balanced sigmoid objective defined below. This balancing is important because positive audio–label pairs constitute only∼0.3% of all clip–label pairs in our training setup. Stage 1 is trained on a 100k-clip corpus of synthetic static FOA scenes (1–4 sources, 2–8 s), rendered with an order-1 spherical-harmonic encoder from mono stems at known azimuth, elevation, and distance. 23 Under review Balanced semantic alignment objective. Let a i ∈R 512 denote the audio embedding of clip i, and let t j ∈R 512 be the frozen CLIP text embedding of label j, where j ∈1,...,C and C = 644. Before similarity computation, both embeddings are ℓ 2 -normalized: ̃ a i = a i ∥a i ∥ 2 , ̃ t j = t j ∥t j ∥ 2 .(4) We then form sigmoid logits as s ij = α ̃ a ⊤ i ̃ t j + b, α = exp(γ),(5) where α is a learnable logit scale and b is a learnable bias. For clip i, letP i ⊆ 1,...,C be the set of present labels and let N i = 1,...,C i be the set of absent labels. We use a per-clip class-balanced sigmoid loss, L (i) sem = 1 2 1 |P i | X j∈P i BCE(s ij , 1) + 1 |N i | X j∈N i BCE(s ij , 0) ,(6) and average it over the mini-batch: L sem = 1 B B X i=1 L (i) sem .(7) Here BCE(s,y) =− y logσ(s) + (1− y) log(1− σ(s)) .(8) The sameL sem is used in both Stage 1 and Stage 2. Stage 2: Query-conditioned spatial localization. Initialized from Stage 1, the encoder is jointly fine-tuned for sound-source localization. Given a text query embedding q describing one source present in the clip, a localization head attends over the encoder tokens via query-conditioned cross- attention and regresses the source direction and range. Rather than predicting azimuth a and ele- vation e directly, the head emits their (sin, cos) encodings together with a log-distance, i.e., a raw 5-vector (ˆs a , ˆc a , ˆs e , ˆc e , ˆ ℓ). The two angular pairs are ℓ 2 -normalized before the loss is applied, which avoids the±180 ◦ wrap-around discontinuity. Stage 2 minimizes a localization regression loss while retaining the same semantic objectiveL sem as a regularizer to prevent semantic drift: L stage2 =L ang + λ d L dist + λ s L sem , λ d = λ s = 0.5.(9) The angular term is the mean-squared error between the normalized predicted (sin, cos) pairs and the ground-truth pose (a,e): L ang = b u a − (sina, cosa) 2 2 + b u e − (sine, cose) 2 2 ,(10) with b u a = (ˆs a , ˆc a ) ∥(ˆs a , ˆc a )∥ 2 , b u e = (ˆs e , ˆc e ) ∥(ˆs e , ˆc e )∥ 2 .(11) The range is supervised in the log domain with an ℓ 1 term against the ground-truth distance r: L dist = ˆ ℓ− logr .(12) Both encoder stages are trained on the same 100k-clip synthetic FOA corpus; the optimization set- tings are listed in Table 10. Figure 10 shows the encoder pretraining dynamics. In Stage 1, the semantic loss drops to ∼ 0.23 while open-vocabulary retrieval Top-1 rises to ∼ 37% over 644 labels; in Stage 2, the mean az- imuth error falls to ∼ 16 ◦ , elevation error to ∼ 7 ◦ , and distance error to ∼ 1.1 m, confirming that the compact encoder becomes both semantically grounded and spatially discriminative before it is frozen. 24 Under review Table 10: FOA encoder pretraining hyperparameters (Stages 1 and 2). SettingStage 1 (semantic)Stage 2 (localization) Training corpus100k synthetic FOA clips, 1–4 sources, 2–8 s ObjectiveSigLIP (balanced) (Zhai et al., 2023) L ang + 0.5L dist + 0.5L sem Initializationfrom scratchwarm-start from Stage 1 OptimizerAdamW (β = 0.9, 0.95), weight decay 0.05, grad clip 1.0 Peak / min LR3e-4 / 1e-51e-4 / 1e-6 Schedule / warmupcosine, warmup fraction 0.05 Batch size (per GPU)3232 Epochs / GPUs10 / 210 / 2 PrecisionAMP 0200040006000800010000 Training step 0.25 0.50 0.75 1.00 1.25 1.50 SigLIP loss (a) Stage 1: semantic alignment SigLIP loss Top-1 (%) 050001000015000 Training step 0 20 40 60 80 Angular error (deg) (b) Stage 2: bearing error Azimuth Elevation 050001000015000 Training step 0.5 1.0 1.5 2.0 2.5 Distance error (m) (c) Stage 2: distance & total loss Distance err (m) Total loss 0 10 20 30 40 Retrieval Top-1 (%) 0.2 0.4 0.6 0.8 1.0 1.2 Total loss Figure 10: FOA encoder pretraining curves (smoothed). (a) Stage 1 semantic alignment: SigLIP loss and open-vocabulary retrieval Top-1 accuracy. (b) Stage 2 query-conditioned localization: mean azimuth and elevation error. (c) Stage 2 distance error and total loss. Stage 3: Integration into the Omni backbone. We graft the frozen FOA encoder into Qwen3- Omni-30B-A3B (a Mixture-of-Experts model with ∼ 3B active parameters). Each FOA clip is processed along two paths: its W channel is down-mixed to mono and fed to the original frozen audio tower to preserve the native semantic audio tokens, while the full four-channel clip is passed through the frozen encoder to produce spatial tokens. A trainable projector (LayerNorm, then 384 → 2048 → 2048 with GELU) maps the encoder tokens into the language embedding space; the spatial tokens are time-resampled by linear interpolation to match the co-located audio span’s token count, and are written into reserved <foa pad> placeholder rows bracketed by learn- able <foabos>/<foaeos> tokens with no vocabulary expansion. Only the projector and the language-model parameters are updated; the FOA encoder, audio tower, and visual tower remain frozen. The final training composition is summarized in Table 12. A.4EVALUATION DETAILS A.4.1SPATIAL AUDIO-VISUAL QA EVALUATION Because model outputs are short free-form phrases, we evaluate each prediction with a two-stage protocol that combines lightweight LLM-based extraction with deterministic rule-based judging. In the first stage, the model output is first stripped of any <think> reasoning, and an extractor LLM normalizes the remaining free text into a small set of structured fields: yesno (yes/no), dirset (a subset ofleft, right, front, back), degrees (float), distancem (float, in meters), letter (A/B/C/D), and a free-text answer. Extraction is performed with a text LLM, for which we use qwen3.7-flash. The extractor is queried with a fixed system prompt together with a user message containing only the question and the model answer. The LLM is used solely for answer normalization, not for judging correctness; final correctness is determined entirely in the subsequent rule-based stage, which keeps scoring fully reproducible. In the second stage, correctness is decided by fixed rules through comparison with the gold reference whenever a task-specific rule applies, as shown in Table 13. Multiple-choice questions (Cognitive Map) require exact letter match. Directional questions use combinations of the four axis tokens 25 Under review Table 11: Stage 3 integration hyperparameters. SettingValue Base modelQwen3-Omni-30B-A3B (Xu et al., 2025b) (MoE,∼ 3B active) Trainable / frozenFOA projector + LLM / FOA encoder, audio tower, visual tower Hardware32× A100 (4 nodes× 8),∼ 4 days ParallelismFSDP2, Ulysses seq.-parallel = 4, expert-parallel = 8 Precisionbf16 mixed precision, gradient checkpointing Global / micro batch size128 / 4 Max sequence length32,768 OptimizerAdamW, weight decay 0.1, max grad norm 1.0 Peak / min LR7e-6 / 7e-7 Schedule / warmupcosine, warmup ratio 0.033 Epochs2 Table 12: Training data composition. The navigation examples are sampled from much larger corpus pools with probability∼ 0.3 after action rebalancing. SourceFull poolUsed (final mix) Spatial audio-visual QA—26,585 Spatial audio QA-18,518 ScaleVLN (Wang et al., 2023)403,215139,142 RxR (Ku et al., 2020)324,34292,418 R2R (Anderson et al., 2018)112,44831,577 VLN-PE (Wang et al., 2025)78,00628,969 In-place rotation data (from ScaleVLN)—25,984 Total—363,193 front/back/left/right, giving eight-way resolution; a prediction is correct only if its direction set exactly matches the ground-truth set, with the opposite direction absent. Angular questions require the reported magnitude to fall within a tolerance of±45 ◦ and the turn direction to be correct. 3D localization questions require the reported distance to fall within ±2 m and the bearing to be correct. Motion questions are scored using the corresponding direction or degree rule depending on subtype. Overall accuracy is computed as the plain average of correctness over all evaluated samples (micro-average), i.e., the total number of correct predictions divided by the total number of samples. Extractor system prompt You normalize a model’s free-text answer for a spatial-audio video QA benchmark. You are given the QUESTION and the MODEL_ANSWER (which may contain reasoning; the real answer is usually its last line). Extract, FROM THE MODEL_ANSWER ONLY, [yesno / dir_set / degrees / distance_m / letter / answer]. Map ahead/forward$→$front, behind/rear/backward$\ rightarrow$back. For multiple choice, pick the model’s FINAL choice, not options it merely discusses. Return ONLY a JSON object. Table 13: QA scoring rules and tolerances by category. CategoryCorrectness rule Cognitive Map (A/B/C/D)exact letter match Visibilityyes/no matches ground-truth bucket Source Direction (8-way)axis set equals GT set, opposite absent Direction / Camera Rotation (degrees) |∆θ|≤ 45 ◦ and correct turn direction 3D Localization|∆dist|≤ 2 m and correct bearing Source Motiondirection / degrees rule per subtype 26 Under review A.4.2SOUND-GUIDED NAVIGATION EVALUATION Closed-loop setup. Navigation is evaluated in the Habitat simulator using the same episode set (identical start and goal) as the text-instruction baselines. In the sound-guided setting, the agent receives FOA observations together with a short task instruction that identifies the sound event to follow but does not reveal the destination or provide route-level spatial guidance. At each decision step, the agent’s live pose selects the nearest receiver in the acoustic field, and the recording’s hor- izontal directional components are remapped into the agent’s egocentric frame before being passed (truncated to 10 s) to the model server. The model is queried once per decision step and returns K = 8 relative waypoints [x fwd ,y left ,ψ]; these are converted to discrete actions and executed be- fore the next replan. Evaluation is parallelized by sharding scenes across GPUs, each running an independent model server and worker, and per-shard metrics are merged by an episode-weighted average. Action space and caps. The discrete action space is stop, move forward, turnleft, turnright with a forward step of 0.25 m and a turn angle of 15 ◦ . A waypoint segment is converted to round(∆pos/0.25) forward steps when its displacement dominates, otherwise to round(|∆ψ|/15 ◦ ) turn actions. Episodes are capped at 50 model decisions (500 executed steps), and a stuck detector forces termination if net displacement over a 16-step window stays below 0.25 m. Stopping condition. All reported navigation experiments use agent self-stop. The model must terminate on its own: a stop is triggered when it emits an empty trajectory or when the maximum waypoint displacement is below 0.3 m with negligible yaw (pure in-place turns are executed, not treated as arrival). Metrics. We report the metrics with a 1 m success radius in OmniEchoBench-Nav and 3 m success radius in VLN-CE R2R val-unseen, defined in Table 14. NE and TL are averaged with nanmean to exclude errored episodes. Table 14: Navigation metrics The success radius r s is 1 m for OmniEchoBench-Nav and 3 m for VLN-CE R2R val-unseen, the same as in prior work. MetricDefinition SR↑success rate: agent stops within r s of the source SPL↑success weighted by (shortest path)/(max(path, shortest path)) NE↓navigation error: final distance to goal (m) OS↑oracle success: ever within r s along the trajectory TLtrajectory length: total path length (m) A.5FURTHER EXPERIMENTS A.5.1SIM2REAL ANALYSIS Table 15: Sim-to-real transfer results. OmniEchoBench is split into two subsets, with 1000 sam- ples used for training and the others for testing. Fine-tuning on the real-data subset improves perfor- mance on the held-out subset across the evaluated task categories, suggesting that real-data adapta- tion can help address limitations of training solely on synthetic data. Training with real-world data Camera Rotation Source textbfDirection 3D Localization Source textbfMotion Cognitive Map Overall ✗41.722.614.224.747.528.5 ✓44.331.021.735.648.034.9 Table 15 presents the sim-to-real transfer results on OmniEchoBench, where the benchmark is split into training and test subsets. Fine-tuning with real-world data yields a clear improvement in overall performance, raising the average score from 28.5 to 34.9. Performance improves across all task categories, including Camera Rotation (+2.6), Direction (+8.4), 3D Localization (+7.5), Motion (+10.9), and Cognitive Map (+0.5). These results suggest that even a limited amount of real-world data can substantially enhance the model’s spatial-audio reasoning and generalization ability in real 27 Under review environments. More importantly, these results are consistent with a noticeable sim-to-real gap in OmniEchoBench for certain question types, while yielding smaller adaptation gains on other tasks (e.g., Camera Rotation and Cognitive Map). A.5.2VLN FAILURE MODES Table 16: OmniEcho on OmniEchoBench-Nav. AspectMetricValue Trajectory Success Rate (SR)16.22% SPL11.54% Oracle Success (OS, once≤1 m)52.89% Navigation Error (NE)4.09 m Trajectory Length (TL)7.02 m Direction (coarse) Step-wise closer rate60.2% Net-approach episode rate64.7% Mean net approach+2.07 m Distance MAE|pred− true|2.20 m Median AE1.61 m Bias (pred−true)+0.70 m Pearson r0.27 Table 17: Failure-mode decomposition (900 episodes). OutcomeRateNote Success (≤1 m & stopped)16.22%— Reached but no successful stop36.67%OS−SR; active-stop failure Never reached (>1 m all steps)47.11% 100−OS; navigation failure Failure-mode analysis. On OmniEchoBench-Nav, the agent’s coarse directional understanding is largely functional: it moves closer to the source on 60.2% of decision steps and net-approaches the target in 64.7% of episodes, reaching within 1 m at some point in 52.9% of episodes (OS). Yet the final success rate is only 16.2%, and the failure decomposition in Table 17 localizes the gap almost entirely downstream of direction: 36.7% of episodes reach the source but fail to stop there, and a further 47.1% never arrive. Rather than a direction-sensing deficiency, the dominant bottleneck is the inability to terminate the trajectory at the right moment. Because the stopping decision is made from a single instantaneous observation, the agent lacks the temporal context to recognize that the sound has stopped growing louder or that distance has ceased to decrease. A promising direction for future work is therefore to equip the agent with a lightweight memory of recent audio and pose observations, allowing it to detect the arrival condition from the trajectory of perceived distance rather than from a single frame, and we leave a fuller study of such memory-conditioned termination to future research. A.5.3VLN WITH GT DIRECTION Table 18: Performance of OmniEcho with GT direction on our OmniEchoBench-Nav. OmniEcho † receives the GT egocentric source direction at each decision step instead of inferring direction from FOA audio. Although GT direction improves SR and SPL, deciding when to stop autonomously remains the primary challenge. ModelBase ModelGuidance Val-Unseen SR↑ SPL↑ NE↓OS↑TL OmniEchoQwen3-Omni-30B-A3BFOA audio16.211.54.0952.89 7.02 OmniEcho † Qwen3-Omni-30B-A3B GT direction23.722.14.1434.33 3.47 28 Under review To isolate the effect of sound-direction estimation from downstream navigation, we evaluate an oracle variant of OmniEcho that receives the ground-truth egocentric direction of the target sound source at each decision step instead of inferring this cue from FOA audio. All other evaluation settings, including agent self-stopping, remain unchanged. As shown in Table 18, ground-truth direction increases SR from 16.2% to 23.7% and SPL from 11.5% to 22.1%, while reducing TL from 7.02 m to 3.47 m. Thus, eliminating directional ambiguity substantially improves path efficiency and the rate of successful termination. Nevertheless, the oracle does not improve final navigation error (4.14 m versus 4.09 m) and reduces OS from 52.89% to 34.33%. Perfect bearing alone therefore does not guarantee that the agent will enter the 1 m success region. Together with the failure-mode analysis above, these results show that accurate direction estimation improves navigation efficiency, but the primary challenge remains deciding when to stop autonomously after reaching the target. A.6VISUALIZATION EXAMPLES Figures 11–14 visualize representative predictions on OmniEchoBench-QA. The successful exam- ples cover sound-source motion, 3D localization, and camera rotation, illustrating the model’s ability to recover spatial states from FOA audio and visual observations. Figure 15 shows two failure ex- amples. In the 3D localization example, OmniEcho captures the coarse direction but predicts the source as directly to the right rather than ahead-right and underestimates its distance. In the camera- rotation example, it identifies the correct rotation direction but substantially overestimates the angle. Together, these visualizations highlight the remaining gap between coarse spatial understanding and fine-grained geometric estimation. Figures 16–19 visualize successful closed-loop episodes from four OmniEchoBench-Nav scenes. Each example pairs the global trajectory with intermediate egocentric observations, estimated sound directions, and planned actions, showing how spatial-audio cues are converted into scene- conditioned movement. Figure 20 provides a further visualization on a physical robot. Across successive steps, the robot updates the estimated direction and distance of an initially out-of-view alarm, plans around intervening obstacles using spatial audio and visual observations, and progres- sively approaches the source. 360° panorama (equirectangular) · t = 10.5 s t = 0.2 st = 5.4 st = 10.5 st = 15.6 st = 20.7 s Q: In the WORLD frame (fixed to the camera's initial heading; X = front, Y = left), what is the overall direction of travel of the source of a toy-car motor sound from t=0.5 to t=13.5 s? GT directly to the right OmniEcho (Ours) ✓ directly to the right Qwen3-Omni-30B ✗ directly ahead Qwen2.5-Omni-7B ✗ directly to the left SO-7B ✗ ahead Sound-source motion Figure 11: A successful sound-source motion example from OmniEchoBench-QA. 29 Under review 360° panorama (equirectangular) · t = 18.7 s t = 0.2 st = 9.5 st = 18.7 st = 27.9 st = 37.1 s Q: In the WORLD frame (fixed to the camera's initial heading; X = front, Y = left), what is the overall direction of travel of the source of a rustling plastic sound from t=12.0 to t=21.0 s? GT directly to the left OmniEcho (Ours) ✓ directly to the left Qwen3-Omni-30B ✗ directly behind Qwen2.5-Omni-7B ✗ directly to the right SO-7B ✗ behind Sound-source motion Figure 12: A successful sound-source motion example from OmniEchoBench-QA. 360° panorama (equirectangular) · t = 1.5 s t = 1.5 s (query)t = 8.3 st = 16.4 st = 24.5 st = 32.6 s Q: At t=1.5 s, give the position of the source of a device alert tone in the CAMERA-BODY frame (relative to where the camera is currently pointing) as an approximate direction plus a distance in metres (e.g. 'ahead and to the right, about 2 m'). GT behind and to the left, about 1.6 m OmniEcho (Ours) ✓ behind and to the left, about 1.6 m Qwen3-Omni-30B ✗ ahead, about 3 m Qwen2.5-Omni-7B ✗ behind, about 1 m SO-7B ✗ below 3D localization (direction + distance) Figure 13: A successful 3D localization example from OmniEchoBench-QA. 30 Under review 360° panorama (equirectangular) · t = 10.3 s t = 0.2 st = 5.3 st = 10.3 st = 15.3 st = 20.3 s Q: Between t=0.5 and t=20.5 s, does the camera turn in the same overall direction (left or right) that the source of a whistling sound travels around the camera? GT yes OmniEcho (Ours) ✓ yes Qwen3-Omni-30B ✗ No Qwen2.5-Omni-7B ✗ No SO-7B ✗ no Camera rotation Figure 14: A successful camera-rotation example from OmniEchoBench-QA. 3D localization At t=7.0 s, give the position of the source of a whirring motor sound in the CAMERA-BODY frame (relative to where the camera is currently pointing) as an approximate direction plus a distance in metres (e.g. 'ahead and to the right, about 2 m'). GT ahead and to the right, about 3.1 m OmniEcho ✗ directly to the right, about 2.5 m Camera rotation — degrees Between t=12.0 and t=20.0 s, by how many degrees and in which direction (left or right) did the camera rotate about its vertical axis? GT 47 degrees to the left OmniEcho ✗ 135 degrees to the left Figure 15: Selected failure cases on OmniEchoBench-QA. OmniEcho preserves part of the correct coarse orientation but misses finer geometric details: it predicts an ahead-right source as directly to the right and estimates a 135 ◦ leftward camera rotation instead of 47 ◦ . 31 Under review 1 2 Start Sound source Step 1 / 10 — agent reasoning I am standing in a narrow hallway with white walls and a grey floor, looking toward an opening that leads into a larger room with columns and curtains. I need to find the source of a falling object sound. I hear the sound coming from the front, slightly to my right, and it is quite far away at 10.2 meters, likely originating from the room beyond the hallway. To reach it, I need to proceed down the corridor, adjusting my path slightly to the right to align with the opening and the sound's direction. So I will first move forward-right to navigate the hallway and then move straight forward toward the sound source. Plan: move forward-right then move forward 1 heard: front-right Step 5 / 10 — agent reasoning I am standing in a hallway with a wall and a door on my left, a central pillar ahead, and a bedroom with a bed and curtains to my right. I need to find the source of a falling object sound. The sound is coming from the front, slightly to my left at a medium distance of 3.2 meters, likely from the area near the wooden door or the room beyond the central pillar. To reach it, I need to navigate past the central pillar and head toward the left side of the opening ahead. So I will first move forward-left to clear the central pillar and then move forward toward the sound source. Plan: move forward-left then move forward 2 heard: front-left Audio-guided navigation instruction: "Move to the source of the falling-object sound." Figure 16: Selected successful navigation trajectories on OmniEchoBench-Nav. 1 2 Start Sound source Step 3 / 11 — agent reasoning I am standing in a room with wooden floors, facing a wall partition that separates a bedroom with blue curtains and a bed on the left from a room with window blinds on the right. The sound is asking me to go to the person who is calling me. I hear the sound coming from my front-left at a medium distance of 2.4 meters, which aligns with the direction of the bedroom opening. To reach the source, I need to navigate around the central wall partition towards the left side. So I will first move forward-left to clear the partition and then move forward-left to approach the person. Plan: move forward-left then move forward-left 1 heard: front-left Step 4 / 11 — agent reasoning I am standing in a bedroom with a bed covered in white linens directly ahead of me, blue curtains to the right, and a white door frame on my left. The sound is asking me to go to the person who is calling me. I hear the sound coming from the front, slightly to my right at a medium distance of 2.2 meters and roughly at ear level. Since the bed is directly in my path, I need to navigate around it to reach the person. I will first move forward-right to clear the bed and head toward the open space, and then move forward-left to align myself with the sound source. So I will first move forward-right and then move forward- left to reach the person. Plan: move forward-right then move forward-left 2 heard: front-right Audio-guided navigation instruction: "Move to the source of the calling voice." Figure 17: Selected successful navigation trajectories on OmniEchoBench-Nav. 32 Under review 1 2 Start Sound source Step 1 / 11 — agent reasoning I am in a room with a bed directly ahead of me, a TV on the left wall, and a doorway leading to a hallway with an air conditioner in the background. I need to find where the animal sound is coming from. I hear the sound coming from the front, slightly to my left, and it is far away at 10.2 meters, likely originating from the hallway or the area beyond the doorway. Since the bed is blocking my direct path, I need to navigate around it to reach the open space. So I will first move forward-left to clear the bed and then move forward-right to align with the path toward the sound source. Plan: move forward-left then move forward-right 1 heard: front-left Step 4 / 11 — agent reasoning I am standing in a narrow room with a bed on my right and a bench on my left, facing a long hallway that leads to a closed wooden door. I need to find where the animal sound is coming from. I hear the sound coming from the front, slightly to my right, and it is quite far away at 10.2 meters, likely originating from down the hallway or near the door at the end. To reach the source, I need to navigate down this corridor, adjusting my path slightly to the right to align with the hallway and avoid the furniture on my left. So I will first move forward-right to enter the hallway properly and then move straight forward toward the sound. Plan: move forward-right then move forward 2 heard: front-right Audio-guided navigation instruction: "Move to the source of the animal sound." Figure 18: Selected successful navigation trajectories on OmniEchoBench-Nav. 1 2 Start Sound source Step 5 / 12 — agent reasoning I am currently facing a beige wall with a handle on my right, while an open doorway to my left reveals a hallway with white walls and wooden floors. I need to find the source of a dragging sound. I hear this sound coming from my front-left at a medium distance of 2.2 meters, which aligns with the open hallway visible through the doorway. Since the wall is directly in front of me, I need to navigate through the opening on my left to reach the source. So I will first move forward-left to enter the hallway and then move forward to get closer to the sound. Plan: move forward-left then move forward 1 heard: front-left Step 7 / 12 — agent reasoning I am standing in a room with a refrigerator on my left, a glass door straight ahead, and an open doorway leading to a kitchen on my right. I need to find the source of a dragging sound. The sound is coming from the front, slightly to my right at a medium distance of 2.2 meters, which aligns with the direction of the kitchen opening. To reach the source, I need to navigate past the refrigerator and head toward the kitchen area. So I will first move forward-right to clear the refrigerator and approach the kitchen entrance, and then move forward into the kitchen to locate the sound. Plan: move forward-right then move forward 2 heard: front-right Audio-guided navigation instruction: "Move to the source of the dragging sound." Figure 19: Selected successful navigation trajectories on OmniEchoBench-Nav. 33 Under review Step 1 / 13 1 ♪ alarm: left · ~4.3 m I am standing in a cluttered room with wooden flooring, facing a back wall lined with desks, computer towers, and a tripod, with a red fire extinguisher and a box to my right. I hear the sound coming from my left, about 4.3 meters away at roughly ear level, likely from the open area beyond the equipment on the left side. Since the direct path to the left is blocked by the fire extinguisher and the room's layout, I need to move into the open central space to navigate around these obstacles. So I will first move forward-left to clear the immediate clutter and then move forward to approach the sound source. Meta action: move forward-left then move forward forward LR alarm · ~4.3 m Step 4 / 13 2 ♪ alarm: left · ~1.5 m I am currently positioned very low to the ground, looking out from under a white table with a large cardboard box to my left and a black equipment case ahead, with a robot and speakers visible in the background. I hear the sound coming from my left, about 1.5 meters away at roughly ear level, likely from behind the cardboard box or the table leg. Since the direct path to the left is blocked by the table and the box, I cannot move straight toward the sound; instead, I need to navigate around these obstacles to get a better angle. So I will first move forward-right to clear the table leg and the box, and then move forward to approach the sound source. Meta action: move forward-right then move forward forward LR alarm · ~1.5 m Step 11 / 13 3 ♪ alarm: left · ~1.3 m I am currently in a cluttered indoor space, facing a white table leg and a white vertical structure on my left, with a large dark object emitting a green glow on my right. I hear the sound coming from my left, about 1.3 meters away at roughly ear level, likely from behind the white structure or in the adjacent area. Since the direct path to the left is blocked by the table and the dark object, I cannot move straight toward the sound; instead, I need to navigate around these obstacles to get a better angle. So I will first move forward-left to clear the immediate clutter and then move forward to approach the alarm. Meta action: move forward-left then move forward forward LR alarm · ~1.3 m Step 13 / 13 4 ALARM SOURCE (GT) ≈1.5 m · front-right ♪ alarm: front-right · ~1.5 m I am currently in a cluttered room with a workbench and cardboard boxes directly ahead, blocking my path, with a doorway visible in the background. I hear the sound coming from my front- right, about 1.5 meters away at ear level, likely from the workbench. Since the workbench and boxes block a direct path to the right, I cannot go straight there; instead, I need to navigate around the central obstacles to get a better angle. So I will first move forward to clear the workbench and then move forward-right to approach the sound source. Meta action: move forward then move forward-right forward LR alarm · ~1.5 m Real-world deployment: sound-guided navigation to an out-of-view alarm Goal: "Find where the alarm is coming from." Figure 20: Qualitative example of OmniEcho deployed on a physical robot for sound-guided nav- igation. The robot progressively localizes and approaches an initially out-of-view alarm by jointly reasoning over egocentric spatial audio and visual observations while navigating around obstacles. 34