Paper deep dive
Audio-Visual Segmentation via Depth-Guided Collaborative Modeling
Zhaojin Fu, Yuyang Hong, Qi Yang, Zili Wang, Kun Ding, Shiming Xiang, Bin Fan
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 91%
Last extracted: 8/23/2026, 1:48:26 AM
Summary
The paper introduces DGCM-AVS, a tri-modal framework for Audio-Visual Segmentation (AVS) that incorporates estimated depth as a spatial structural cue to improve cross-modal alignment and segmentation robustness. The method features a Depth-Aware Dynamic Modulator (DADM) for separating adjacent objects and preserving intra-object consistency, and a Depth-Guided Progressive Fusion (DGPF) module to align audio and visual features using depth as an intermediate bridge. DGCM-AVS achieves state-of-the-art performance on the AVSS dataset.
Entities (11)
Relation Signals (11)
DGCM-AVS → usesmodule → Depth-Aware Dynamic Modulator
confidence 95% · we design a Depth-Aware Dynamic Modulator to improve the separation of adjacent objects
DGCM-AVS → usesmodule → Depth-Guided Progressive Fusion
confidence 95% · we propose Depth-Guided Progressive Fusion, which uses depth as an intermediate bridge
DGCM-AVS → achievesperformanceon → AVSS
confidence 92% · DGCM-AVS achieves relative improvements of 10.2 percent in M_J and 8.7 percent in M_F on the AVSS dataset.
Depth-Guided Progressive Fusion → aligns → Audio-Visual Segmentation
confidence 90% · DGPF... progressively align audio cues with visual features.
Depth-Aware Dynamic Modulator → improves → Audio-Visual Segmentation
confidence 90% · DADM improves inter-object discrimination while preserving intra-object consistency.
DGCM-AVS → usesaudioencoder → VGGish
confidence 90% · A pretrained VGGish faT(·) [15] is used to extract frame-level audio features
DGCM-AVS → usesdepthestimator → Depth Anything V2
confidence 90% · we use Depth Anything V2 [39] to estimate depth information
DGCM-AVS → achievesperformanceon → AVSBench-Semantic
confidence 88% · achieves state-of-the-art performance on both the AVSBench-Object and AVSBench-Semantic.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Audio-Visual Segmentation (AVS) is a fundamental task in multimodal perception that performs pixel-level segmentation of sounding objects in videos by leveraging both visual and audio cues. It has broad applications in video understanding, human-computer interaction, and autonomous driving. However, most existing AVS methods do not explicitly model geometric cues such as relative distance and occlusion, thereby limiting the robustness of cross-modal alignment. In human perception, spatial structure is naturally integrated with audio-visual evidence to accurately localize sounding objects. Motivated by this, we incorporate estimated depth as a spatial structural cue for AVS and propose DGCM-AVS, a tri-modal framework that jointly models audio, visual, and depth information. Specifically, we design a Depth-Aware Dynamic Modulator to improve the separation of adjacent objects while preserving intra-object feature consistency. Furthermore, we propose Depth-Guided Progressive Fusion, which uses depth as an intermediate bridge to progressively align audio cues with visual features. Compared to state-of-the-art methods, DGCM-AVS achieves relative improvements of 10.2 percent in M_J and 8.7 percent in M_F on the AVSS dataset. We believe our study highlights depth as a promising yet underexplored modality for AVS and may encourage further research in this direction.
Tags
Links
- Source: https://arxiv.org/abs/2608.16285v1
- Canonical: https://arxiv.org/abs/2608.16285v1
Trouble viewing inline? Open PDF directly →
Full Text
67,997 characters extracted from source content.
Expand or collapse full text
Audio-Visual Segmentation via Depth-Guided Collaborative Modeling Zhaojin Fu Yuyang Hong Qi Yang Zili Wang Kun Ding Shiming Xiang Bin Fan Thanks: This work was supported by the National Natural Science Foundation of China (Grant Nos. U24A20218 and 62306310). Thanks: Zhaojin Fu and Bin Fan are with the School of Intelligent Science and Technology, University of Science and Technology Beijing, Beijing 100083, China (e-mail: d202410395@xs.ustb.edu.cn; bin.fan@ieee.org). Thanks: Yuyang Hong, Qi Yang, Zili Wang, Kun Ding, and Shiming Xiang are with the Institute of Automation, Chinese Academy of Sciences, Beijing 100190, China. (e-mail: hongyuyang2023@ia.ac.cn; yangqi2021@ia.ac.cn; ziliwang2022@ia.ac.cn; kun.ding@ia.ac.cn; smxiang@nlpr.ia.ac.cn) Thanks: Corresponding author: Bin Fan (e-mail: bin.fan@ieee.org). Abstract Audio-Visual Segmentation (AVS) is a fundamental task in multimodal perception that performs pixel-level segmentation of sounding objects in videos by leveraging both visual and audio cues. It has broad applications in video understanding, human–computer interaction, and autonomous driving. However, most existing AVS methods do not explicitly model geometric cues such as relative distance and occlusion, thereby limiting the robustness of cross-modal alignment. In human perception, spatial structure is naturally integrated with audio-visual evidence to accurately localize sounding objects. Motivated by this, we incorporate estimated depth as a spatial structural cue for AVS and propose DGCM-AVS, a tri-modal framework that jointly models audio, visual, and depth information. Specifically, we design a Depth-Aware Dynamic Modulator to improve the separation of adjacent objects while preserving intra-object feature consistency. Furthermore, we propose Depth-Guided Progressive Fusion, which uses depth as an intermediate bridge to progressively align audio cues with visual features. Compared to state-of-the-art methods, DGCM-AVS achieves relative improvements of 10.2% in ℳM_J and 8.7% in ℳℱM_F on the AVSS dataset. We believe our study highlights depth as a promising yet underexplored modality for AVS and may encourage further research in this direction. Index Terms: Audio-Visual Segmentation, Depth, Multi-modal Vision, Collaboration. I Introduction With auditory cues, humans can focus visual attention on sounding objects. This cross-modal perceptual mechanism is crucial for understanding complex scenes. As applications such as human-computer interaction and autonomous driving continue to expand, it becomes increasingly important to endow machines with similar cross-modal perceptual capabilities. Accordingly, Audio-Visual Segmentation (AVS) [24, 17, 22] has attracted increasing attention in recent years. In its general form, AVS leverages both audio and visual cues to localize sounding objects and segment them at the pixel level. Depending on the setting, AVS can be formulated as binary foreground-background segmentation, or as a semantic task that additionally predicts category labels for the segmented sounding objects. Fig. 1: (a) Representative AVS methods that align audio and visual features. (b) Illustrates that humans exploit depth cues together with audio evidence to disambiguate sounding objects. (c) Presents our core idea, which aims to perform the AVS task in a human-like way. Existing AVS methods can be broadly viewed as improving either cross-modal semantic alignment or mask boundary quality, although many designs combine these ingredients in different forms. One representative line focuses on semantic alignment [17, 41, 8], as shown in Fig. 1(a), where audio is used to modulate semantic visual features through cross-modal interaction. However, semantic cues alone are often insufficient for precise localization and boundary delineation, especially when multiple candidates have similar appearances or the sounding object is partially occluded. Another line leverages strong visual priors from large-scale vision models to refine segmentation masks [37, 40]. With visual prompting, these methods can produce sharper boundaries and improved region separation. Nonetheless, excessive prompt-induced partitioning can over-segment spatially adjacent regions, undermining object integrity and causing fragmented masks or semantic inconsistency. Despite their progress, the lack of explicit spatial-structure constraints can makes cross-modal correspondence less robust. As a result, segmentation can become unstable in complex scenes, often exhibiting attention drift to salient but silent regions and boundary leakage. Inspired by human spatial perception, we suggest that explicitly introducing scene spatial structure into AVS can constrain cross-modal correspondence, thereby improving the stability and robustness of segmentation. In particular, when multiple candidate objects have similar appearances or are partially occluded, spatial cues such as depth ordering and occlusion relationships can help determine which visible entity is more likely to correspond to the audio signal. This reduces attention drift and boundary leakage, as illustrated in Fig. 1(b). Recently, researchers have increasingly recognized that spatial information is crucial for human-like multimodal perception and have begun exploring its integration into sound source localization [1] and multimodal learning [43]. However, these efforts do not explicitly model the pixel-level localization and fine boundary delineation required by AVS, and therefore cannot directly satisfy the requirements for accurate segmentation. Therefore, a more systematic approach is needed to endow AVS models with spatial awareness, enabling more robust cross-modal alignment. We propose the Depth-Guided Collaborative Modeling for Audio-Visual Segmentation (DGCM-AVS) to foster more effective collaboration among the audio, visual, and depth modalities. Since depth provides a direct cue to scene spatial structure, we incorporate it into our model design, as shown in Fig. 1(c), to equip the model with two capabilities: (1) improve the discrimination of objects at different distances and positions while preserving structural integrity. (2) using depth as a bridge to align audio and visual features more effectively. For capability (1), we propose the Depth-Aware Dynamic Modulator (DADM), which adopts a scale-based approximation of frequency separation. Specifically, it uses neighborhood aggregation to form a stable, region-level low-frequency visual representation, and leverages a depth residual to highlight boundary-sensitive high-frequency structural cues. By fusing these complementary cues, DADM improves inter-object discrimination while preserving intra-object consistency. For capability (2), we propose Depth-Guided Progressive Fusion module (DGPF), which uses depth information as a bridge to align audio and visual features in two stages. In the first stage, Audio Depth Symbiotic, audio features are projected into the geometric space to establish the correspondence between sound and scene structure. In the second stage, Target Search, audio and visual semantics are aligned in the spatial domain to produce accurate segmentation masks. In this paper, our main contributions are as follows: • To the best of our knowledge, DGCM-AVS is the first to explore the potential of using depth information in AVS. It introduces a collaborative modeling framework across audio, visual, and depth information. This method offers new insights for advancing the AVS field and achieves state-of-the-art performance on both the AVSBench-Object and AVSBench-Semantic. • We propose the Depth-Aware Dynamic Modulator module to embed depth information into image features. This design enhances intra-object consistency and strengthens inter-object discrimination. • We propose the Depth-Guided Progressive Fusion module for gradual interaction across audio, visual, and depth cues. Depth information serves as a bridge to improve alignment between audio and visual features. I Related Work I-A Audio-Visual Segmentation AVS aims to segment sounding objects at the pixel level by jointly leveraging audio and visual cues in complex scenes. AVSBench [41] introduced two benchmark datasets, AVSBench-Object and AVSBench-Semantic, and proposed a baseline that embeds audio features into visual representations for sounding-object localization. Specifically, AVSBench-Object focuses on binary foreground–background segmentation. AVSBench-Semantic defines a more challenging setting where models are required to segment sounding objects and predict their semantic categories. Following this, several studies [20, 2, 9, 19, 28, 24, 30] approached the task from the perspective of semantic alignment, using audio cues as queries to retrieve visual targets and enhance cross-modal representations. ECMVAE [30] formulates audio-visual segmentation as a conditional multimodal variational autoencoding problem. It disentangles modality-shared and modality-specific latent codes, and optimizes them with orthogonality constraints and cross-modal mutual-information regularization to better capture sounding objects. While effective in establishing semantic-level correspondence, semantic-alignment-based methods generally lack explicit spatial constraints and thus struggle with precise localization in complex scenes. Another line of work [37, 40] explored boundary enhancement with large vision models, for example using SAM [18] to provide visual prompts. These approaches improved boundary sharpness but often over-emphasized local edges, which compromised object integrity. Other studies [10, 36] introduced textual descriptions to support audio-visual alignment. Although textual guidance enriches cross-modal connections, it can be unreliable for target localization under distractors, occlusions, or cluttered scenes. In summary, existing methods have explored semantic alignment, boundary enhancement, and semantic description, but none explicitly model the spatial structure of scenes. To address this limitation, we propose DGCM-AVS, which introduces depth as explicit geometric cue to strengthen cross-modal correspondence and improve the consistency and robustness of AVS predictions. I-B Depth estimation Depth estimation predicts a pixel-wise depth map from a monocular or stereo image to describe the 3D geometry of a scene. Recently, some methods learn depth with explicit geometric constraints and aim for metric-consistent structure [3, 16, 31]. Other methods [4, 38, 39] emphasize large-scale pretraining to learn transferable depth representations and achieve strong generalization. Although their designs differ, they often provide reliable cues about relative layout, depth gradients, and depth discontinuities in practice. In computer vision [33, 32, 6, 12, 11, 26, 27], depth maps are commonly adopted as transferable geometric priors, and they are particularly useful under appearance degradation such as low illumination, blur, and occlusion, where RGB features are more affected by texture and lighting changes. In deblurring, DAVIDE [33] injects depth features into the network across multiple stages and uses depth-aware fusion. The gains are larger when the temporal context is short, suggesting that depth helps stabilize structure recovery when appearance evidence is weak. In semantic segmentation, Texture-Diffusion [32] shows that depth provides reliable layout and boundary discontinuity cues. It uses cross-modal diffusion and structural consistency constraints to reduce information loss caused by modality gaps, which improves robustness. For low-light segmentation, DPSAM [26] introduces depth into feature refinement via cross-modal attention. This reduces failures caused by degraded RGB features and texture confusion, and it improves prediction stability and boundary quality. For occluded overlapped text segmentation, MOTS [27] uses depth-guided decoding and cross-attention to model occlusion hierarchies, which improves boundary localization in overlapped scenes. Overall, these studies show that depth information often provides a more stable global layout and scale structure, which supports region-level localization. It also offers clearer boundary discontinuities and occlusion breaks, which improves boundary adherence and detail recovery. Since these geometric cues are usually more robust than RGB texture and illumination, depth can provide structural constraints for cross-modal representation learning and can improve cross-modal alignment to some extent [1, 43]. These insights also match key challenges in audio-visual segmentation. Audio cues are often coarse and region-level, and occlusion can more easily cause candidate confusion and boundary drift. Based on this, we introduce estimated depth into DGCM-AVS as an auxiliary geometric prior. We focus on boundary and occlusion discontinuities and global layout cues. During feature fusion, these cues provide more stable structural constraints for audio-guided localization and further refine segmentation boundaries, which improves audio-visual alignment and final segmentation performance in occluded scenes. Fig. 2: Overview of the proposed DGCM-AVS. (a) DADM extracts different frequency components from image and depth features to improve intra-object consistency and enhance inter-object distinction. (b) DGPF uses depth information as a bridge to progressively complete the Audio Depth Symbiotic stage (A-D Symbiotic) and Target Search stages, achieving cross-modal feature alignment. (c) Feature Calibrator further strengthens the target-related feature responses. Algorithm 1 DGCM-AVS for AVSS Input: Train dataset D, mask label ℳM, class label C Output: ℳoutM_out, outC_out 1: The total loss ℒ=λbceℒbce+λdiceℒdice+λceℒceL= _bceL_bce+ _diceL_dice+ _ceL_ce, with λbce=5 _bce=5, λdice=5 _dice=5, λce=2 _ce=2 2: for xv,xmel∈x_v,x_mel do 3: ℱv=fvT(xv)F_v=f_v^T(x_v), ℱv=ℱvii=14F_v=\F_v^i\_i=1^4 4: Depth information: xd=Depth Generator(xv)x_d=Depth Generator(x_v) 5: ℱd=fdT(xd)F_d=f_d^T(x_d), ℱd=ℱdii=14F_d=\F_d^i\_i=1^4 6: ℱa=faT(xmel)F_a=f_a^T(x_mel) 7: Depth-Aware Dynamic Modulator fDADM(⋅)f_DADM(·): 8: for i in 44 do 9: ℱoi=fDADMi(ℱvi,ℱdi)F_o^i=f_DADM^i(F_v^i,~F_d^i) 10: end for 11: Multi-scale Deformabl Attention fMSDA(⋅)f_MSDA(·): 12: ℱo=fMSDA(ℱo)F_o=f_MSDA(F_o) 13: Depth-Guided Progressive Fusion fDGPF(⋅)f_DGPF(·): 14: ℱv1′,ℱa′=fDGPF(ℱa,ℱd,ℱo)F_v^1 ,F_a^ =f_DGPF(F_a,~F_d,~F_o) 15: Transformer Decoder fDeocder(⋅)f_Deocder(·): 16: ℳout,out=fDeocder(ℱa′,ℱv1′,ℱo)M_out,~C_out=f_Deocder(~F_a^ ,~F_v^1 ,~F_o) 17: Loss=ℒ((out,),(ℳout,ℳ))Loss=L((C_out,~C),~(M_out,~M)) 18: end for 19: return LossLoss I Method I-A Overall Architecture The DGCM-AVS framework is shown in Fig. 2. The Perceptual Element Encoding first builds perceptual representations from video, depth, and audio. The Depth-Aware Dynamic Modulator injects depth-derived structural cues into visual features via a dynamic low-/high-frequency fusion scheme. This enhances the model’s ability to discriminate adjacent objects, improving inter-object separability. Subsequently, to capture object details across different scales, we apply Multi-Scale Deformable Attention, inspired by Deformable DETR [44], enabling effective multi-scale feature interaction. The Depth-Guided Progressive Fusion is built upon depth information and adopts a two-stage strategy to progressively integrate audio and visual features for audio-visual alignment. This design is inspired by how humans identify sounding objects in complex environments. Finally, we adopt the Transformer decoder [5] to reconstruct the final segmentation masks. To further explain the proposed DGCM-AVS, we present its algorithmic process in Algorithm 1. The algorithm shows the complete pipeline from multi-modal feature extraction to semantic segmentation. I-B Perceptual Element Encoding Given an input video xv∈ℝT×3×H×Wx_v ^T× 3× H× W, we first extract visual features using a visual encoder fvT(⋅)f_v^T(·), producing multi-scale features ℱv=ℱvii=14F_v=\F_v^i\_i=1^4, where ℱvi∈ℝT×Ci×H2i+1×W2i+1F_v^i ^T× C_i× H2^i+1× W2^i+1. Since current AVS benchmarks do not include depth information, we use Depth Anything V2 [39] to estimate depth information xdx_d, which is converted to RGB format to match the visual input. A separate depth encoder fdT(⋅)f_d^T(·) is used to extract multi-scale depth features ℱd=ℱdii=14F_d=\F_d^i\_i=1^4, with ℱdi∈ℝT×Ci×H2i+1×W2i+1F_d^i ^T× C_i× H2^i+1× W2^i+1. For both the visual encoder fvT(⋅)f_v^T(·) and the depth encoder fdT(⋅)f_d^T(·), we experiment with two backbone networks: ResNet-50 [14] and PVT-v2 [35]. For audio, the raw waveform is first resampled to a 16 kHz mono signal. We then compute the Mel-spectrogram xmel∈ℝT×96×64x_mel ^T× 96× 64 using Short-Time Fourier Transform. A pretrained VGGish faT(⋅)f_a^T(·) [15] is used to extract frame-level audio features ℱa∈ℝT×DF_a ^T× D, where D=128D=128 and T is the number of video frames. I-C Depth-Aware Dynamic Modulator Visual features contain both low- and high-frequency components. The low-frequency part mainly describes global structure and semantic layout, providing stable and robust cues for coarse localization. In contrast, the high-frequency part contains richer fine textures, but it is more sensitive to noise and occlusion, and it often struggles to distinguish visually similar objects, which may confuse the model. Depth provides complementary benefits. In the spatial domain, depth corresponds more directly to geometry and discontinuities, so it can produce cleaner local changes around boundaries and occlusions. For AVS, audio often provides region-level cues related to the sounding object. Therefore, we need stable semantics and layout to support audio-visual alignment, and we also need reliable boundary cues to refine the mask. As shown in Fig. 3, we visualize the RGB frame and its depth map in the frequency domain. Both spectra show strong energy near the center, which suggests that the two modalities share global structural information. However, their behaviors differ in the high-frequency region. The RGB spectrum is more scattered and is easily affected by textures and lighting changes. These appearance-driven components can introduce sound-irrelevant distractions in AVS and reduce the stability of audio-visual alignment. In contrast, the depth spectrum is more concentrated along contours and shows clearer structure with less interference. The reconstructed high-frequency edges are also more continuous and align better with object boundaries. Based on these observations, we use low-frequency visual cues as a stable semantic basis for audio-guided coarse localization. We use high-frequency depth cues to tighten and refine the boundaries. Motivated by the above observations, we propose the Depth-Aware Dynamic Modulator (DADM) in Fig. 2 (a). To provide an explicit and reproducible definition of high- and low-frequency components, we adopt a scale-based approximation for frequency separation in the spatial domain. Neighborhood aggregation is treated as a low-pass operator, since aggregating features over a larger receptive field performs stronger local averaging, suppresses rapid spatial variations, and yields a low-frequency representation. Under this formulation, the low-frequency component is the aggregated feature, whereas the high-frequency component is the residual between the original feature and the aggregated feature, HighFreq(F)=F−LowFreq(F)HighFreq(F)=F-LowFreq(F). This residual emphasizes fast-changing structural cues in depth, including boundary discontinuities and occlusion-induced breaks. Following this design, the low-frequency branch uses a larger window kl=5k_l=5 to integrate wider context and produce a stable semantic basis that aligns with region-level audio guidance and preserves global layout consistency. The high-frequency branch uses a smaller window kh=3k_h=3 to retain finer local variations, which enables the residual signal to capture depth discontinuities at object boundaries more effectively. Fig. 3: Frequency-domain comparison of RGB and depth. The figure shows log power spectra, zoomed views of low/high-frequency regions, and edge responses reconstructed from the mid-to-high frequency band. Formally, the visual features ℱvF_v and depth features ℱdF_d are first processed with channel attention to generate importance weights AvA_v and AdA_d, which emphasize feature channels related to sounding objects. Then, dynamic perception kernels are constructed along high- and low-frequency paths. For the high-frequency path, we first obtain compressed features ℱ~d=Conv1×1(ℱd)∈ℝT×c×h×w F_d=Conv_1× 1 (F_d ) ^T× c× h× w, and we then process it within a kh=3k_h=3 window to focus on local depth variations. A kernel normalization operation generates a local perception kernel KHK_H: KH=(ϕ(HConv(ℱ~d)))∈ℝT×1×kh2×h×w,K_H=T (φ (HConv ( F_d ) ) ) ^T× 1× k_h^2× h× w, (1) The perception kernel is constructed based on the input features, allowing it to adapt dynamically to different inputs and enabling input-dependent feature perception. Here, (⋅)T (· ) represents the kernel normalization operation ∑p=1kh2KH(p)=1 _p=1^k_h^2K_H (p )=1, ϕ(⋅)φ (· ) represents a feature reshaping function into format T×1×kh2×h×wT× 1× k_h^2× h× w, and HConv(⋅)HConv (· ) denotes a convolutional operator with kernel size 3×33× 3. Moreover, the feature ℱ~d F_d is padded using reflection padding to obtain ℱ~d′∈ℝT×c×(h+2p)×(w+2p) F_d ^T× c× (h+2p )× (w+2p ), p=⌊kh/2⌋p= k_h/2 , k∈1,…,kh2k∈ \1,...,k_h^2 \ represents the linear index of a pixel within the kh×khk_h× k_h neighborhood. For each spatial location (i,j)(i,j), a kh×khk_h× k_h receptive window is centered to extract neighboring features. These are then unfolded and reshaped to obtain the aggregated neighborhood feature UdU_d: Ud=Unfold(ℱ~d′)∈ℝT×c×kh2×h×w,U_d=Unfold( F_d ) ^T× c× k_h^2× h× w, (2) This design allows the model to capture relative spatial relationships among neighboring pixels while maintaining spatial sensitivity during subsequent high-frequency kernel computations. Similarly, in the low-frequency perception branch, a kl=5k_l=5 receptive window is applied to compute the low-frequency kernel KL∈ℝT×1×kl2×h×wK_L ^T× 1×k_l^2× h× w and the corresponding aggregated neighborhood feature Uv∈ℝT×c×kl2×h×wU_v ^T× c×k_l^2× h× w. Finally, UdU_d and UvU_v are convolved with their respective perception kernels KHK_H and KLK_L. Channel attention and residual fusion are applied to jointly leverage region-level semantics and boundary-sensitive depth cues. This process can be formulated as: ℱd′=∑k=1kh2(Ud[:,:,k,:,:]⊙KH[:,0,k,:,:]),F_d = _k=1^k_h^2 (U_d [:,:,k,:,: ] K_H [:,0,k,:,: ] ), (3) ℱv′=∑k=1kl2(Uv[:,:,k,:,:]⊙KL[:,0,k,:,:]),F_v = _k=1^k_l^2 (U_v [:,:,k,:,: ] K_L [:,0,k,:,: ] ), (4) ℱdH=(ℱd−ℱd′)⊙Ad+ℱd,ℱvL=ℱv′⊙Av+ℱv,F_d^H= (F_d-F_d ) A_d+F_d,F_v^L=F_v A_v+F_v, (5) ℱo=Conv3×3(ℱdH+ℱvL),F_o=Conv_3× 3 (F_d^H+F_v^L ), (6) where ⊙ represents element-wise multiplication, ℱo∈ℱoii=14F_o∈ \F_o^i \_i=1^4, ℱoi∈ℝT×Ci×H2i+1×W2i+1F_o^i ^T× C_i× H2^i+1× W2^i+1. ℱd′F_d can be interpreted as a locally aggregated approximation of the depth feature that suppresses rapid spatial variations. The high-frequency depth cue is explicitly represented by the residual ℱd−ℱd′F_d-F_d , which exhibits strong responses at depth discontinuities and object boundaries. This residual is then reweighted by AdA_d and added back to ℱdF_d to enhance boundary sensitivity. Meanwhile, ℱv′F_v provides a smoother, low-frequency visual representation that is less affected by texture noise, supporting stable AVS. As shown in Fig. 4, we visualize the two branches of DADM and their fused feature. The low-frequency branch output ℱv′F_v mainly shows region-level responses. It can stably cover the main area of the sounding object and reflects strong semantic consistency. However, its responses are often smooth, which may lead to blurred boundaries or slight over-expansion. In contrast, the high-frequency branch ℱd−ℱd′F_d-F_d , constructed from depth residuals, responds more strongly to depth discontinuities. It highlights object contours and local structural changes, providing cleaner geometric cues for boundary refinement. The fused feature ℱoF_o effectively combines these complementary cues. It preserves the integrity of the object region while enhancing contour clarity and suppressing background interference. Accordingly, the predictions on the MS3 and AVSS benchmarks match the GT more closely, especially along boundaries and in local details, which further verifies the effectiveness of DADM. Fig. 4: Visualization of Intermediate Features in DADM. We visualize the intermediate features from the low-frequency branch ℱv′F_v , the high-frequency branch ℱd−ℱd′F_d-F_d , and the final fused feature ℱoF_o. We also show the model predictions on the MS3 and AVSS benchmarks, together with the corresponding GT. I-D Depth-Guided Progressive Fusion Motivated by the observation that depth provides explicit spatial cues, we incorporate it into the model to enhance audio-visual alignment. To this end, we propose the Depth-Guided Progressive Fusion (DGPF) (Fig. 2(b)), which leverages depth as a bridge to progressively establish semantic correspondence between audio and sounding objects. It consists of two stages: in Audio Depth Symbiotic, audio features are projected into the spatial domain to highlight candidate sounding regions; in Target Search, depth guidance together with audio-visual semantics enables precise localization and segmentation. A Depth Bridge is designed for both stages to enable effective fusion, as in Fig. 5. Unlike conventional fusion strategies, DGPF explicitly exploits depth as a progressive bridge, ensuring geometric alignment and semantic consistency. In the Audio Depth Symbiotic stage, the depth feature ℱd1F_d^1 is first encoded with sine-based positional encoding to generate the query vector Qd=ℱd1pWQQ_d=F_d^1pW_Q. In parallel, the value vector Vd=ℱd1WVdV_d=F_d^1W_V^d is obtained directly from ℱd1F_d^1 through a linear transformation. Then, the Feature Calibrator performs multi-scale processing on VdV_d, applies query-guided weighting, and incorporates the result into QdQ_d, yielding the updated query Qd′Q_d . The audio value vector is computed as Va=ℱaWVaV_a=F_aW_V^a, and the key is obtained by adding learnable positional encoding to the audio features, resulting in Ka=ℱapWKK_a=F_a^pW_K. By sharing Qd′Q_d and KaK_a, the model projects audio features into the spatial domain and highlights candidate sounding regions more effectively. This design follows a common practice in Transformer [34] and DETR [44]. Injecting positional information into the query Q and the key K is sufficient to make the attention weights spatially aware, while the value V mainly preserves the semantic content to be aggregated. This process can be formulated as: ℱd1′=Softmax(Qd′KaTd)Va+ℱd1,F_d^1 =Softmax ( Q_d K_a^T d )V_a+F_d^1, (7) ℱad=Softmax((Qd′KaTd)T)Vd+ℱa.F_ad=Softmax ( ( Q_d K_a^T d )^T )V_d+F_a. (8) Fig. 5: Depth Bridge for DGPF. Serving as a structural cue, depth embeds features in A–D symbiosis and facilitates audio–visual alignment during Target Search. We propose the Feature Calibrator (Fig. 2(c)), which adopts a dual-branch design. One branch applies multi-scale depthwise separable convolutions k(⋅)C_k(·) to capture boundary textures, while the other uses adaptive pooling s(⋅)P_s(·) followed by bilinear interpolation Is(⋅)I_s(·) to model global spatial layout. The query QdQ_d modulates the texture features to suppress irrelevant noise, and the spatial layout features provide a stable structure for alignment, thereby enhancing intra-object consistency. The updated query is computed as ΔQ=Qd⊙Concat(k(Vd))k=3,5,7+Concat(Is(s(Vd)))s=1,2,3,6 Q=Q_d (C_k(V_d) )_k=\3,5,7\+Concat (I_s(P_s(V_d)) )_s=\1,2,3,6\, and the final query becomes Qd′=Qd+ΔQ_d =Q_d+ Q. The Feature Calibrator enhances the discriminability of target-related features beyond conventional query-modulation strategies. Next, we perform modality-level aggregation: ℱv1=ℱd1′+ℱo1F_v^1=F_d^1 +F_o^1, ℱa′=ℱad+ℱaF_a =F_ad+F_a. Here, the depth feature ℱd1′F_d^1 already contains audio information, while the visual feature ℱo1F_o^1, fused by DADM, is rich in semantic content and encodes scene geometry. Similarly, the updated audio feature ℱa′F_a also contains depth information. Finally, in the Target Search stage, we apply Depth Bridge to ℱv1F_v^1 and ℱa′F_a , using depth information as a bridge to improve alignment between the audio and visual modalities. Unlike the first stage, which focuses on candidate region enhancement, Target Search refines the correspondence by jointly leveraging audio, visual, and depth semantics for precise localization and segmentation. The computation follows Equations (7) and (8), resulting in the updated features ℱv1′F_v^1 and ℱa′F_a^ . TABLE I: Quantitative results on the Single Sound Source Segmentation (S4) and Multiple Sound Source Segmentation (MS3), and AVSBench-Semantic (AVSS). Bold indicates the best result, and underline indicates the second best. S4 MS3 AVSS Method Backbone ℳM_J ℳℱM_F ℳM_J ℳℱM_F ℳM_J ℳℱM_F AVSBench [41] ResNet-50 72.8 84.8 47.9 57.8 20.2 25.2 AVS-BiGen [13] ResNet-50 74.1 85.4 45.0 56.8 - - BAVS [23] ResNet-50 78.0 85.3 50.2 62.4 24.7 29.6 AVSegFormer [8] ResNet-50 76.4 86.7 53.8 65.6 26.6 31.5 SelM [19] ResNet-50 76.6 86.2 54.5 65.6 31.9 37.2 UFE [25] ResNet-50 79.0 87.5 55.9 64.5 - - QDFormer [21] ResNet-50 77.6 86.0 59.6 63.5 - - COMBO [40] ResNet-50 81.7 90.1 54.5 66.6 33.3 37.3 DiffusionAVS [29] ResNet-50 75.8 86.9 49.8 62.1 - - DGCM-AVS (Ours) ResNet-50 83.3 91.3 61.2 74.4 39.6 43.5 AVSBench [41] PVT-v2 78.7 87.9 54.0 64.5 29.8 35.3 AVS-BiGen [13] PVT-v2 81.7 90.4 55.1 66.8 - - BAVS [23] PVT-v2 82.0 88.6 58.6 65.5 32.6 36.4 AVSegFormer [8] PVT-v2 83.1 90.5 61.3 73.0 37.3 42.8 SelM [19] PVT-v2 83.5 91.2 60.3 71.3 41.3 46.9 UFE [25] PVT-v2 83.2 90.4 62.0 70.9 - - QDFormer [21] Swin-Tiny 79.5 88.2 61.9 66.1 - - COMBO [40] PVT-v2 84.7 91.9 59.2 71.2 42.1 46.1 DiffusionAVS [29] PVT-v2 81.5 90.3 59.6 71.2 38.1 43.0 DGCM-AVS (Ours) PVT-v2 85.2 92.4 63.2 75.5 46.4 51.0 Fig. 6: Qualitative comparisons on S4. (a) audio, (b) frames, (c) AVSBench [41], (d) AVSegFormer [8], (e) COMBO [40], (f) DGCM-AVS (Ours), (g) GT. The green denotes missed regions and red indicates over-segmented areas relative to GT. IV Experiments IV-A Dataset AVSBench-Object [42] consists of two subsets: Single Sound Source Segmentation (S4) and Multiple Sound Source Segmentation (MS3). The S4 subset includes 4,932 five-second video clips, each containing a single sounding object. Only the first frame of each five-frame sequence is annotated, yielding a semi-supervised setting. The dataset uses a 70/15/15 train/val/test split. The MS3 subset contains 424 fully annotated videos with multiple simultaneous sound sources, reflecting more complex auditory scenes. AVSBench-Semantic (AVSS) [41] extends AVSBench-Object to the audio-visual semantic segmentation task. It follows the same train/val/test split as S4. The dataset includes all AVSBench-Object samples and adds semantic labels to them. It also adds new 10-second videos with frame-level semantic masks, bringing the total to 11,356 videos. This extension increases task complexity and makes semantic modeling and generalization more challenging. IV-B Implementation Details Training settings. All experiments are conducted on NVIDIA RTX 4090 GPU with input resolution fixed at 224×224224× 224. We use Adam with an initial learning rate of 10−410^-4, weight decay of 0.05, and a batch size of 6. The model is trained for 90K iterations on S4 and AVSS, and 20K on MS3. Metrics. Following the standard protocol [41, 8], we use the Jaccard index [7] (ℳM_J) and the F-score (ℳℱM_F) as evaluation metrics. The Jaccard index is defined as ℳ=|P∩G||P∪G|M_J= |P∩ G | |P∪ G |, where P and G denote the predicted and ground-truth masks, and |⋅| |· | denotes the number of pixels. The F-score is defined as ℳℱ=(1+β2)×Precision×Recallβ2×Precision+RecallM_F= (1+β^2 )× Precision× Recallβ^2× Precision+Recall, where β2β^2 is set to 0.3. Precision measures how accurate the predicted foreground pixels are, and Recall measures how completely the true foreground pixels are detected. ℳM_J measures the region-level intersection-over-union between the predicted and ground-truth masks, and reflects how well the overall shape and coverage of the foreground region are matched. ℳℱM_F depends on both Precision and Recall, and focuses more on the quality of the segmentation boundaries, being sensitive to false positives and false negatives along object edges. These two metrics evaluate the audio-visual segmentation results from the complementary perspectives of boundary accuracy and region similarity, and thus provide a more comprehensive assessment of the model performance. Loss. The total loss includes classification and segmentation terms, defined as ℒ=λbceℒbce+λdiceℒdice+λceℒceL= _bceL_bce+ _diceL_dice+ _ceL_ce, where ℒceL_ce is the cross-entropy loss for class prediction, and ℒbceL_bce and ℒdiceL_dice are the binary cross-entropy loss and Dice loss used for mask supervision. Following prior works [24, 40, 36], the weights are set to λbce=5 _bce=5, λdice=5 _dice=5, and λce=2 _ce=2. IV-C Main Results Quantitative Analysis. Tab. I summarizes the quantitative performance of DGCM-AVS on the S4, MS3, and AVSS benchmarks, compared with representative methods. On the S4 benchmark, DGCM-AVS-R50 outperforms prior methods by 1.6 ℳM_J and 1.2 ℳℱM_F, while DGCM-AVS-PVT achieves gains of 0.5 ℳM_J and 0.5 ℳℱM_F. On MS3, DGCM-AVS-R50 achieves substantial improvements of 1.6 ℳM_J and 7.8 ℳℱM_F, and DGCM-AVS-PVT improves by 1.2 ℳM_J and 2.5 ℳℱM_F. On AVSS, our model also attains the best performance: DGCM-AVS-R50 improves by 6.3 ℳM_J and 6.2 ℳℱM_F, and DGCM-AVS-PVT achieves 4.3 ℳM_J and 4.1 ℳℱM_F. Overall, DGCM-AVS consistently surpasses prior methods across all benchmarks, with particularly notable improvements on the more complex MS3 and AVSS datasets. While the gain on S4 is relatively small, this can be attributed to its simple scenes and limited interference, where most methods already perform strongly. In contrast, the significant improvements on MS3 and AVSS highlight that depth cues are especially beneficial in challenging scenarios involving multiple sound sources, occlusion, or cluttered backgrounds. These results support that depth can act as an effective bridge for cross-modal alignment and provide complementary structural information that is difficult to obtain from audio-visual cues alone. Qualitative Analysis on S4. Fig. 6 presents additional qualitative comparisons on the S4. In the first set of examples, the red tail of the helicopter closely resembles the background factory in color and texture, making boundary segmentation highly challenging. Methods such as AVSBench and AVSegFormer, which do not incorporate explicit spatial cues, struggle to separate the object from the background, while COMBO provides only limited improvement due to imperfect visual prompts. In contrast, DGCM-AVS leverages depth cues via the DADM to produce sharper boundaries and maintain intra-object consistency. Furthermore, DGPF treats depth as a bridging modality and employs a two-stage strategy to facilitate more explicit alignment between audio cues and visual targets. In the second set of examples, the sounding object is partially occluded by tree branches, and its tail shares textures with the surroundings. All baseline methods degrade noticeably under occlusion, whereas DGCM-AVS demonstrates more stable segmentation, even though minor edge inaccuracies remain. In the third set of examples, AVSBench and AVSegFormer show obvious target recognition errors, while DGCM-AVS effectively leverages geometric cues from depth information to preserve the structural integrity of the object and achieve higher segmentation accuracy than COMBO. On the S4, although the scenes often involve challenges such as color similarity, texture confusion, or partial occlusion, DGCM-AVS effectively integrates audio-visual features with spatial structural cues to preserve object integrity and sharpen boundaries, thereby achieving higher segmentation accuracy than existing methods. Fig. 7: Qualitative comparisons on MS3. (a) audio, (b) frames, (c) AVSBench [41], (d) AVSegFormer [8], (e) COMBO [40], (f) DGCM-AVS (Ours), (g) GT. The green denotes missed regions and red indicates over-segmented areas relative to GT. Qualitative Analysis on MS3. Fig. 7 shows more comparison results on MS3. In the first set of examples, AVSBench and AVSegFormer struggle to capture the boundaries of the instrument accurately. This is mainly because the sounding objects are surrounded by high-frequency visual patterns, which interfere with the true contours and lead to incorrect predictions. COMBO shows instability, mistakenly identifying the person in the scene as a sound source in some frames. The second set of examples includes more semantically similar distractors in the background. Under this challenging condition, our method produces results that are most aligned with the ground truth and successfully identifies all sounding objects. In contrast, AVSBench and AVSegFormer are clearly affected by background noise and mistakenly include silent regions in their predictions. COMBO fails to segment the “man” object, mainly because it relies on prompt information from SAM. Such prompts may break the integrity of the target object, leading to incomplete predictions. In the third set of examples, the video contains different sounding objects at different times, resulting in temporal discontinuity of the targets. In this case, our method shows more stable performance than the baselines and accurately segments each sounding object. These results on MS3 suggest that depth, as a carrier of spatial structure, enhances the model’s understanding of complex scenes and helps distinguish object boundaries more effectively. Moreover, aligning audio and visual features through depth promotes more effective cross-modal interaction, guiding the model toward potential sounding regions and improving segmentation quality. Fig. 8: Qualitative comparisons on AVSS. We provide a full visualization of the semantic segmentation performed on 10-frame sequences. The letter labels represent, in order: (a) audio, (b) frames, (c) AVSBench [41], (d) AVSegFormer [8], (e) COMBO [40], (f) DGCM-AVS (Ours), (g) GT. Qualitative Analysis on AVSS. The AVSS dataset features higher sample complexity and longer video sequences. To illustrate the model’s performance under such challenging conditions, we present qualitative results on a representative 10-frame sample, as shown in Fig. 8. In terms of segmentation completeness, AVSBench and AVSegFormer produce coarse boundary delineation for sounding objects, leading to reduced structural integrity. For target identification, COMBO exhibits missing sounding objects and misclassification errors, and its predictions show temporal inconsistency across frames. In contrast, our method achieves the best semantic segmentation results. This advantage comes from two main aspects. First, depth information provides a clearer structure of the scene. Our proposed DADM module fully leverages this geometric information to help maintain feature consistency within objects. Second, DGPF maps audio and visual features into a feature space guided by depth, which enhances the alignment between modalities. These results validate the effectiveness and feasibility of using depth information in AVS. TABLE I: Efficiency and Performance Comparison on S4. Bold indicates the best result. Method Audio Backbone Backbone ℳM_J (S4) ℳℱM_F (S4) Params FLOPs Inference Time AVSBench [41] VGGish ResNet-50 72.8 84.8 90.7M 166.6G 17.7ms AVSegFormer [8] VGGish ResNet-50 76.4 86.7 150.9M 210.1G 176.5ms DGCM-AVS (Ours) VGGish ResNet-50 83.3 91.3 224.3M 183.4G 60.7ms Efficiency and Performance Comparison. Tab. I compares AVSBench, AVSegFormer, and our DGCM-AVS on the S4 benchmark. For a fair comparison, all three methods use VGGish as the audio backbone and ResNet-50 as the visual backbone. In terms of performance, DGCM-AVS achieves the best results, with ℳM_J of 83.3 and ℳℱM_F of 91.3, significantly outperforming the two representative methods. This gain mainly stems from our proposed depth-guided collaborative modeling mechanism, which enables the model to more stably constrain cross-modal correspondence under candidate confusion, occlusions, and cluttered backgrounds, thereby reducing attention drift and boundary leakage and improving both localization and boundary quality. In contrast, AVSBench adopts a lightweight convolution-based modeling paradigm and thus runs faster, but its cross-modal interaction and structural constraints are relatively limited in complex scenes, leading to inferior accuracy. AVSegFormer benefits from stronger global modeling and achieves improved accuracy, but it also introduces higher computational overhead, resulting in the slowest inference. From the perspective of computational complexity, DGCM-AVS requires 183.4G FLOPs, which is notably lower than AVSegFormer (210.1G), indicating that our performance improvement is not simply obtained by substantially increasing computation. Regarding inference time, DGCM-AVS runs at 60.7ms, falling between AVSBench and AVSegFormer, and is considerably faster than AVSegFormer. Although DGCM-AVS incorporates depth information and thus has a larger parameter scale, it still follows a single forward-pass pipeline without additional iterative refinement, maintaining a reasonable inference cost while achieving higher accuracy. In practical deployments, parameter storage is typically cheaper than sustained computation, and parameters mainly affect the static memory footprint rather than per-inference computation. Considering the advantages in both accuracy and efficiency, DGCM-AVS demonstrates a clear competitive edge. Failure Case Analysis. Although DGCM-AVS achieves strong performance on the three benchmarks, it still shows limitations in a few complex audio-visual scenes. Fig. 9 presents three representative challenging cases from S4, MS3, and AVSS. In the first case, only the ambulance is sounding, but in the second frame the ambulance almost moves out of view and a police motorbike appears. Because the two sirens sound very similar, the model wrongly localizes the motorbike as the sounding object. In the second case, the baby makes sound only in the first frame and is correctly segmented there, but in the fully silent second frame the model still follows the previous prediction and labels the baby as sounding. The third case shows a similar error when the ukulele becomes silent but is still predicted as a sounding region. These results indicate that DGCM-AVS, while stable in capturing object contours with the help of depth, is not robust enough when the scene contains multiple similar sound sources, sources moving out of view, or rapid changes in the sounding state. We believe there are two main reasons. First, such complex patterns are very rare in current benchmarks, so the training data under-represents these cases. Second, the continuous and aligned audio–video inputs cause strong temporal dependence, or even entanglement, in the audio features, which makes the model biased toward historical objects on silent frames or frames with source switching. Fig. 9: Failure cases of DGCM-AVS on S4, MS3, and AVSS (left to right). For each example, we show (a) audio, (b) frames, (c) DGCM-AVS (ours), and (d) GT. IV-D Ablation Study Fig. 10: Visualization of the ablation study to show the efficacy of each proposed component. Validation of DADM and Backbone Weights. As shown in Tab. I, shared encoder refers to using the same encoder for both visual and depth information. DGCM-AVS uses independent encoders for each modality. w/o depth indicates that depth information is excluded from the model, and both branches in DADM operate solely on visual features. w/o DADM removes the entire module, and depth and visual features are fused by direct addition. The results show that using independent encoders helps the model better capture modality-specific structures, leading to improved performance. Introducing depth information helps the model better localize and segment sounding objects by providing additional spatial cues. Adding the DADM enables complementary fusion of low-frequency visual features and high-frequency depth cues, improving edge sensitivity and robustness to noise, which leads to more accurate segmentation. TABLE I: Validation of DADM and Weights on ResNet-50. Method S4 AVSS ℳM_J ℳℱM_F ℳM_J ℳℱM_F shared encoder 82.9 91.0 39.4 43.3 w/o depth 81.1 90.0 37.0 40.9 w/o DADM 83.2 90.9 38.7 42.4 DGCM-AVS 83.3 91.3 39.6 43.5 TABLE IV: Validation the khk_h and klk_l window size of DADM. Method S4 AVSS ℳM_J ℳℱM_F ℳM_J ℳℱM_F DADMkl=7,kh=5DADM_k_l=7,k_h=5 83.1 91.1 39.2 43.1 DADMkl=7,kh=3DADM_k_l=7,k_h=3 82.7 90.8 38.9 42.6 DADMkl=5,kh=3DADM_k_l=5,k_h=3 83.3 91.3 39.6 43.5 Validation the Window Size of DADM. As shown in Tab. IV, we evaluate different window sizes (kl,kh)(k_l,k_h) used in the low-frequency aggregation branch and the high-frequency residual branch of DADM. The best performance is achieved with (kl=5,kh=3)(k_l=5,k_h=3). Specifically, kl=5k_l=5 provides sufficient context to produce a smoother and more stable aggregated representation, which helps preserve region-level semantic consistency and supports robust alignment with audio cues. In contrast, a smaller local window kh=3k_h=3 better captures rapid spatial changes around object boundaries, making the explicitly modeled residual cue ℱd−ℱd′F_d-F_d more concentrated and thus improving boundary sensitivity. Larger window settings kl=7k_l=7 lead to over-smoothing and neighborhood mixing, which blur fine boundary details and increase computational cost. Overall, we adopt (kl=5,kh=3)(k_l=5,k_h=3) as it achieves the best performance while preserving stable region responses and clear boundaries. TABLE V: Validation of DGPF structures on ResNet-50. Method S4 AVSS ℳM_J ℳℱM_F ℳM_J ℳℱM_F only AV 82.9 90.9 38.7 42.4 w/o audio 82.4 91.1 38.0 41.8 w/o FC 82.9 91.2 39.2 43.0 DGCM-AVS 83.3 91.3 39.6 43.5 Validation of DGPF. As shown in Tab. V, only AV refers to directly performing the Target Search stage for audio-visual alignment, without involving depth as an intermediate bridge. w/o audio removes the audio modality completely, keeping only the direct fusion of visual and depth information. w/o FC removes the Feature Calibrator, which reduces the model’s focus on potential target regions. Compared to these variants, the full DGPF model demonstrates the effectiveness of each component and the benefit of their integration. The Feature Calibrator enhances the model’s attention to potential targets within visual and depth features, resulting in more pronounced improvements on the AVSS dataset. Audio features further serve as guidance, enabling the model to locate sounding objects based on auditory cues. Built upon these cues, DGPF adopts a two-stage strategy in which depth acts as a bridge between modalities, progressively aligning audio and visual features to support accurate segmentation. TABLE VI: The impact of different depth information. Method S4 AVSS ℳM_J ℳℱM_F ℳM_J ℳℱM_F + DP 82.0 90.5 37.7 41.7 + DA 83.2 90.9 37.6 41.7 + DA-V2 83.3 91.3 39.6 43.5 Visualization for Ablation Studies. Fig. 10 illustrates the visual impact of different ablation settings. It can be observed that removing any key component leads to distinct degradations. Without audio guidance, (a) the model loses the constraint of sounding cues, and the attention is more easily attracted by visually salient regions and diffuses into the background, resulting in evident false positives. Without depth information, (b) the lack of geometric structure and boundary cues weakens the model’s ability to characterize object shape, making predictions prone to adhesion to adjacent regions and exhibiting both false positives and false negatives. Furthermore, removing DADM (Fig. 10(c)) noticeably reduces the model’s perception of scene structural information, while intra-object features become less continuous and eventually cause false detections, indicating that DADM helps maintain intra-object consistency and improves boundary sensitivity between adjacent objects. Removing FC (Fig. 10(d)) makes the fusion and calibration process less stable, with weaker and less complete attention responses, which tends to introduce noisy regions and miss thin structures. When only audio-visual fusion is retained (Fig. 10(e)), although the attention can still fall near the sounding objects, the absence of structural constraints makes the focus more likely to drift, leading to coarse boundaries and unstable object recognition. In contrast, the full DGCM-AVS model (Fig. 10(f)) can more stably focus on the sounding objects, producing predictions with region coverage more consistent with the GT and clearer objects contours. These results suggest that audio provides a semantic anchor for sounding objects, depth offers geometric and boundary constraints, and DADM and DGPF work complementarily to achieve more robust and accurate audio-guided segmentation. Fig. 11: The impact of depth information. (a) frames, (b) DP results, (c) AVS results based on DP , (d) DA-V2 results, (e) AVS results based on DA-V2. Validation of the Impact of Different Depth Information. As shown in Tab. VI, we compare three recent models used as depth generators to study how different types of depth information can help align audio and visual modalities. Due to the high resource demands of the three depth generators with their best pretrained weights, we adopt a two-stage pipeline: depth information is first extracted, followed by the AVS task. Depth Pro (DP) [3] is based on geometry-aware modeling, while Depth Anything (DA) [38] and Depth Anything v2 (DA-V2) [39] rely on vision-centric pretraining. Among them, DA-V2 achieves clearly better results than the others. As in Fig. 11, we compare the robustness of DP and DA-V2 in generating depth information under different scenes. DA-V2 produces more stable and accurate depth information than DP, especially in complex environments. The results show that although high-quality depth information can further improve performance, it still requires the proposed DGCM-AVS framework to effectively exploit such spatial cues in order to realize their full potential. Fortunately, with the recent advances in depth estimation, it is now possible to obtain more reliable depth, which provides strong support for further research in audio-visual segmentation. V Conclusion In this work, we propose DGCM-AVS, a depth-guided framework for audio-visual segmentation. To the best of our knowledge, this is the first AVS framework that explicitly uses depth as a bridging cue to guide alignment between audio and visual modalities. The Depth-Aware Dynamic Modulator effectively leverages boundary-sensitive high-frequency cues from depth maps and semantically informative low-frequency cues from visual features. It improves intra-object feature consistency while enhancing inter-object discrimination. The Depth-Guided Progressive Fusion adopts a two-stage design that uses depth as a bridge to progressively align audio cues with visual features. DGCM-AVS achieves promising results on both AVSBench-Object and AVSBench-Semantic benchmarks. These results suggest that incorporating depth cues is a promising direction for improving robustness in AVS. VI Future Work Future work could address the observed failure modes from both data and methodological perspectives. On the data side, it would be beneficial to incorporate more training videos with complex audio-visual scenes to improve generalization to long-tail cases. On the methodological side, it would be useful to explore explicit temporal decoupling and contrastive learning strategies that encourage models to focus on sound cues synchronized with the current frame and suppress residual signals from previous frames. In addition, it would be interesting to study AVS in embodied settings, where segmentation can be tightly coupled with perception–action loops to support more reliable scene understanding and interaction in real-world environments. References [1] D. Berghi and P. J. Jackson (2024) Leveraging reverberation and visual depth cues for sound event localization and detection with distance estimation. arXiv preprint arXiv:2410.22271. Cited by: §I, §I-B. [2] S. Bhosale, H. Yang, D. Kanojia, J. Deng, and X. Zhu (2025) Unsupervised audio-visual segmentation with modality alignment. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), Vol. 39. Cited by: §I-A. [3] A. Bochkovskiy, A. Delaunoy, H. Germain, M. Santos, Y. Zhou, S. Richter, and V. Koltun (2025) Depth pro: sharp monocular metric depth in less than a second. In International Conference on Learning Representations (ICLR), Vol. 2025, p. 75602–75637. Cited by: §I-B, §IV-D. [4] S. Chen, H. Guo, S. Zhu, F. Zhang, Z. Huang, J. Feng, and B. Kang (2025) Video depth anything: consistent depth estimation for super-long videos. In Proceedings of the Computer Vision and Pattern Recognition Conference (CVPR), p. 22831–22840. Cited by: §I-B. [5] B. Cheng, I. Misra, A. G. Schwing, A. Kirillov, and R. Girdhar (2022) Masked-attention mask transformer for universal image segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), p. 1290–1299. Cited by: §I-A. [6] X. Cheng, Z. Fu, and J. Yang (2020) Zero-shot image super-resolution with depth guided internal degradation learning. In European Conference on Computer Vision (ECCV), p. 265–280. Cited by: §I-B. [7] M. Everingham, L. Van Gool, C. K. Williams, J. Winn, and A. Zisserman (2010) The pascal visual object classes (voc) challenge. International journal of computer vision (IJCV) 88 (2), p. 303–338. Cited by: §IV-B. [8] S. Gao, Z. Chen, G. Chen, W. Wang, and T. Lu (2024) Avsegformer: audio-visual segmentation with transformer. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), Vol. 38. Cited by: §I, Fig. 6, TABLE I, TABLE I, Fig. 7, Fig. 8, §IV-B, TABLE I. [9] S. Gong, Y. Zhuge, L. Zhang, Y. Wang, P. Zhang, L. Wang, and H. Lu (2025) Avs-mamba: exploring temporal and multi-modal mamba for audio-visual segmentation. IEEE Transactions on Multimedia (IEEE TMM). Cited by: §I-A. [10] R. Guo, L. Qu, D. Niu, Y. Qi, W. Yue, J. Shi, B. Xing, and X. Ying (2024) Open-vocabulary audio-visual semantic segmentation. In Proceedings of the 32nd ACM International Conference on Multimedia (ACM M), p. 7533–7541. Cited by: §I-A. [11] T. Guo, B. Lu, F. Wang, and Z. Lu (2025) Depth-aware super-resolution via distance-adaptive variational formulation. Journal of Electronic Imaging 34 (5), p. 053018–053018. Cited by: §I-B. [12] Y. Guo, M. Xu, L. Jiang, X. Deng, Y. Zhang, and Y. Liu (2025) Compressed image super-resolution based on invertible degradation and restoration. Pattern Recognition, p. 112532. Cited by: §I-B. [13] D. Hao, Y. Mao, B. He, X. Han, Y. Dai, and Y. Zhong (2024) Improving audio-visual segmentation with bidirectional generation. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), Vol. 38, p. 2067–2075. Cited by: TABLE I, TABLE I. [14] K. He, X. Zhang, S. Ren, and J. Sun (2016) Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), p. 770–778. Cited by: §I-B. [15] S. Hershey, S. Chaudhuri, D. P. Ellis, J. F. Gemmeke, A. Jansen, R. C. Moore, M. Plakal, D. Platt, R. A. Saurous, B. Seybold, et al. (2017) CNN architectures for large-scale audio classification. In 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), p. 131–135. Cited by: §I-B. [16] M. Hu, W. Yin, C. Zhang, Z. Cai, X. Long, H. Chen, K. Wang, G. Yu, C. Shen, and S. Shen (2024) Metric3d v2: a versatile monocular geometric foundation model for zero-shot metric depth and surface normal estimation. IEEE Transactions on Pattern Analysis and Machine Intelligence (IEEE T-PAMI). Cited by: §I-B. [17] S. Huang, R. Ling, T. Hui, H. Li, X. Zhou, S. Zhang, S. Liu, R. Hong, and M. Wang (2025) Revisiting audio-visual segmentation with vision-centric transformer. In Proceedings of the Computer Vision and Pattern Recognition Conference (CVPR), p. 8352–8361. Cited by: §I, §I. [18] A. Kirillov, E. Mintun, N. Ravi, H. Mao, C. Rolland, L. Gustafson, T. Xiao, S. Whitehead, A. C. Berg, W. Lo, et al. (2023) Segment anything. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), p. 4015–4026. Cited by: §I-A. [19] J. Li, S. Yu, Y. Wang, L. Wang, and H. Lu (2024) SelM: selective mechanism based audio-visual segmentation. In Proceedings of the 32nd ACM International Conference on Multimedia (ACM M), Cited by: §I-A, TABLE I, TABLE I. [20] K. Li, Z. Yang, L. Chen, Y. Yang, and J. Xiao (2023) Catr: combinatorial-dependence audio-queried transformer for audio-visual video segmentation. In Proceedings of the 31st ACM International Conference on Multimedia (ACM M), p. 1485–1494. Cited by: §I-A. [21] X. Li, J. Wang, X. Xu, X. Peng, R. Singh, Y. Lu, and B. Raj (2024) Qdformer: towards robust audiovisual segmentation in complex environments with quantization-based semantic decomposition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), p. 3402–3413. Cited by: TABLE I, TABLE I. [22] C. Liu, P. Li, L. Yang, D. Wang, L. Li, and X. Yu (2025) Robust audio-visual segmentation via audio-guided visual convergent alignment. In Proceedings of the Computer Vision and Pattern Recognition Conference (CVPR), p. 28922–28931. Cited by: §I. [23] C. Liu, P. Li, H. Zhang, L. Li, Z. Huang, D. Wang, and X. Yu (2024) Bavs: bootstrapping audio-visual segmentation by integrating foundation knowledge. IEEE Transactions on Multimedia (IEEE TMM). Cited by: TABLE I, TABLE I. [24] C. Liu, L. Yang, P. Li, D. Wang, L. Li, and X. Yu (2025) Dynamic derivation and elimination: audio visual segmentation with enhanced audio semantics. In Proceedings of the Computer Vision and Pattern Recognition Conference (CVPR), p. 3131–3141. Cited by: §I, §I-A, §IV-B. [25] J. Liu, Y. Liu, F. Zhang, C. Ju, Y. Zhang, and Y. Wang (2024) Audio-visual segmentation via unlabeled frame exploitation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), p. 26328–26339. Cited by: TABLE I, TABLE I. [26] P. Liu, J. Deng, L. Duan, W. Li, and F. Lv (2025) Segmenting anything in the dark via depth perception. IEEE Transactions on Multimedia (IEEE TMM). Cited by: §I-B. [27] Y. Liu, X. Xie, Y. Liu, and X. Bai (2025) Multi-scenario overlapping text segmentation with depth awareness. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), p. 17454–17463. Cited by: §I-B. [28] J. Ma, P. Sun, Y. Wang, and D. Hu (2024) Stepping stones: a progressive training strategy for audio-visual semantic segmentation. In European Conference on Computer Vision (ECCV), p. 311–327. Cited by: §I-A. [29] Y. Mao, J. Zhang, M. Xiang, Y. Lv, D. Li, Y. Zhong, and Y. Dai (2025) Contrastive conditional latent diffusion for audio-visual segmentation. IEEE Transactions on Image Processing (IEEE TIP). Cited by: TABLE I, TABLE I. [30] Y. Mao, J. Zhang, M. Xiang, Y. Zhong, and Y. Dai (2023) Multimodal variational auto-encoder based audio-visual segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), p. 954–965. Cited by: §I-A. [31] L. Piccinelli, Y. Yang, C. Sakaridis, M. Segu, S. Li, L. Van Gool, and F. Yu (2024) UniDepth: universal monocular metric depth estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), p. 10106–10116. Cited by: §I-B. [32] W. Sun, Y. Li, Q. Ye, J. Jiao, and Y. Zhou (2024) Depth-guided texture diffusion for image semantic segmentation. IEEE Transactions on Circuits and Systems for Video Technology (IEEE TCSVT). Cited by: §I-B. [33] G. F. Torres, J. Kalliola, S. Tripathy, E. Acar, and J. Kämäräinen (2024) DAVIDE: depth-aware video deblurring. In European Conference on Computer Vision (ECCV), p. 161–179. Cited by: §I-B. [34] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin (2017) Attention is all you need. Advances in Neural Information Processing Systems (NeurIPS) 30. Cited by: §I-D. [35] W. Wang, E. Xie, X. Li, D. Fan, K. Song, D. Liang, T. Lu, P. Luo, and L. Shao (2021) Pyramid vision transformer: a versatile backbone for dense prediction without convolutions. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), Cited by: §I-B. [36] Y. Wang, P. Sun, D. Zhou, G. Li, H. Zhang, and D. Hu (2024) Ref-avs: refer and segment objects in audio-visual scenes. In European Conference on Computer Vision (ECCV), p. 196–213. Cited by: §I-A, §IV-B. [37] Y. Wang, H. Xu, Y. Liu, J. Li, and Y. Tang (2025) SAM2-love: segment anything model 2 in language-aided audio-visual scenes. In Proceedings of the Computer Vision and Pattern Recognition Conference (CVPR), p. 28932–28941. Cited by: §I, §I-A. [38] L. Yang, B. Kang, Z. Huang, X. Xu, J. Feng, and H. Zhao (2024) Depth anything: unleashing the power of large-scale unlabeled data. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), p. 10371–10381. Cited by: §I-B, §IV-D. [39] L. Yang, B. Kang, Z. Huang, Z. Zhao, X. Xu, J. Feng, and H. Zhao (2024) Depth anything v2. Advances in Neural Information Processing Systems (NeurIPS) 37, p. 21875–21911. Cited by: §I-B, §I-B, §IV-D. [40] Q. Yang, X. Nie, T. Li, P. Gao, Y. Guo, C. Zhen, P. Yan, and S. Xiang (2024) Cooperation does matter: exploring multi-order bilateral relations for audio-visual segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Cited by: §I, §I-A, Fig. 6, TABLE I, TABLE I, Fig. 7, Fig. 8, §IV-B. [41] J. Zhou, X. Shen, J. Wang, J. Zhang, W. Sun, J. Zhang, S. Birchfield, D. Guo, L. Kong, M. Wang, et al. (2024) Audio-visual segmentation with semantics. International Journal of Computer Vision (IJCV). Cited by: §I, §I-A, Fig. 6, TABLE I, TABLE I, Fig. 7, Fig. 8, §IV-A, §IV-B, TABLE I. [42] J. Zhou, J. Wang, J. Zhang, W. Sun, J. Zhang, S. Birchfield, D. Guo, L. Kong, M. Wang, and Y. Zhong (2022) Audio–visual segmentation. In European Conference on Computer Vision (ECCV), Cited by: §IV-A. [43] B. Zhu, B. Lin, M. Ning, Y. Yan, J. Cui, W. HongFa, Y. Pang, W. Jiang, J. Zhang, Z. Li, et al. (2024) Languagebind: extending video-language pretraining to n-modality by language-based semantic alignment. In International Conference on Learning Representations (ICLR), Vol. 2024, p. 9588–9608. Cited by: §I, §I-B. [44] X. Zhu, W. Su, L. Lu, B. Li, X. Wang, and J. Dai (2020) Deformable detr: deformable transformers for end-to-end object detection. arXiv preprint arXiv:2010.04159. Cited by: §I-A, §I-D.