Paper deep dive
Less is More: Encoder-only Audio-Visual Segmentation
Ilpo Viertola, Vladimir Iashin, Sophie Tötterström, Esa Rahtu
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 92%
Last extracted: 9/25/2026, 4:55:04 AM
Summary
The paper introduces EASE (Encoder-only Audio-Visual Segmentation), a simplified and efficient model for Audio-Visual Semantic Segmentation (AVSS). By utilizing a plain Vision Transformer (ViT) backbone and removing redundant components found in previous Transformer-based approaches, EASE achieves state-of-the-art performance while being up to 3x faster (365 FPS) and training in under 11 GPU-hours. Key innovations include a Gumbel-Softmax-based audio feature enhancement module using learned semantic clustering and a mid-backbone audio query injection strategy with bidirectional audio-visual fusion.
Entities (9)
Relation Signals (6)
EASE → achievesperformance → 365 FPS
confidence 95% · EASE runs at up to 365 FPS, 3x faster than prior State-of-the-Art (SotA) AVS models
EASE → evaluatedon → AVSBench
confidence 95% · We utilize the widely adopted audio-visual segmentation dataset AVSBench
EASE → usesbackbone → ViT
confidence 95% · EASE is a simplified approach for AVS built on top of a plain ViT architecture
EASE → outperforms → Prior SotA AVS models
confidence 90% · 3x faster than prior State-of-the-Art (SotA) AVS models at comparable accuracy
EASE → usesaudioencoder → VGGish
confidence 90% · Audio is encoded with a pretrained and frozen VGGish encoder
EASE → usesfusiontechnique → Gumbel-Softmax Clustering
confidence 90% · introduces a Gumbel-Softmax-based audio feature enhancement module... learned soft clustering
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Audio-Visual Semantic Segmentation (AVSS) aims to identify, segment, and classify sound-emitting objects in video frames. Previous Transformer-based AVSS approaches largely inherit design principles from image segmentation models. Recent studies show that these image segmentation models contain redundant components that contribute little to the segmentation performance. Following this insight, we propose Encoder-only Audio-Visual Segmentation (EASE). EASE runs at up to 365 FPS, 3x faster than prior State-of-the-Art (SotA) AVS models at comparable accuracy, and trains in under 11 GPU-hours. Furthermore, we achieve SotA AVSS performance across different backbones and input resolutions. Our results demonstrate that AVSS can be both simpler and faster, providing a scalable foundation for future research and real-time applications. Code, model weights, and samples are available at this https URL
Tags
Links
- Source: https://arxiv.org/abs/2609.29121v1
- Canonical: https://arxiv.org/abs/2609.29121v1
Trouble viewing inline? Open PDF directly →
Full Text
31,845 characters extracted from source content.
Expand or collapse full text
Less is More: ENCODER-ONLY AUDIO-VISUAL SEGMENTATION Ilpo Viertola †thanks: We acknowledge the Finnish IT Center for Science (CSC) for computational resources. Vladimir Iashin Sophie Tötterström Esa Rahtu Abstract AVSS (AVSS) aims to identify, segment, and classify sound-emitting objects in video frames. Previous Transformer-based AVSS approaches largely inherit design principles from image segmentation models. Recent studies show that these image segmentation models contain redundant components that contribute little to the segmentation performance. Following this insight, we propose EASE (EASE). EASE runs at up to 365 FPS, 3× faster than prior SotA (SotA) AVS models at comparable accuracy, and trains in under 11 GPU-hours. Furthermore, we achieve SotA AVSS performance across different backbones and input resolutions. Our results demonstrate that AVSS can be both simpler and faster, providing a scalable foundation for future research and real-time applications. Code, model weights, and samples are available at ease-avs.notion.site. Index Terms: Semantic Segmentation, Audio-Visual Learning †address: 1Tampere University 1 Introduction AVS (AVS) aims to localize and segment sounding objects in a video. Traditionally, the AVS task is divided into three sub-tasks: S4 (S4), MS3 (MS3), and AVSS (AVSS) [39, 38]. For the first two, the goal is to produce a pixel-level binary mask indicating pixels emitting the sounds. For the semantic segmentation, the goal is also to categorize the masked objects. Since AVSS is larger in size, a more complex task, and includes all samples from S4 and MS3, we focus on it in this work. (a) Common Transformer-based AVS approach. (b) Our EASE (EASE) Figure 1: Comparison between our simplified approach and common Transformer-based AVS methods. Our approach eliminates unnecessary architectural complexity, resulting in a more efficient model while preserving high segmentation performance. Many AVS studies draw inspiration from image segmentation research [13, 36, 37, 32, 26, 22, 20]. Recent works [18, 5, 12] have explored redundant components in current SotA (SotA) image segmentation models, while the plain ViT (ViT) [10] has demonstrated strong generalization across a wide range of computer vision tasks [29, 33, 9]. Additional task-specific components implemented on top of a ViT backbone in image segmentation models often increase model size and computational cost without substantially improving segmentation quality, motivating efforts towards more efficient and modular architectures [18, 5, 12]. A common Transformer-based [34] AVS approach is illustrated in Fig. 1(a). We adapt the core design principles of [18, 5, 12] to AVS by removing task-specific building blocks in favor of a plain ViT architecture, and introduce EASE (EASE). Our approach is shown in Fig. 1(b). It enables integrating VFM (VFM)s [33, 29] to leverage their strong visual representations while reducing complexity. We experiment with two visual backbones: the commonly used AVS backbone PVTv2 (PVTv2) [35], and the widely adopted VFM DINOv3 [33]. Thanks to extensive pretraining and high VFM capacity, we achieve SotA AVSS performance with 3×3× higher throughput. The scalable vanilla ViT [10] architecture further enables optimized attention implementations such as FlashAttention [8] for faster training and inference. Our contributions can be summarized as follows: i) a simple, strong, and efficient baseline for AVS that enables adopting VFMs for AVS, i) a model-agnostic cross-modal fusion technique with audio feature enhancement, and i) extensive experiments on all AVS sub-tasks to demonstrate the balance between segmentation performance and efficiency of our method. 2 Background Audio-visual segmentation (AVS) refers to the task of predicting pixel-wise masks for the sound-producing objects in a video. It was first proposed by Zhou et al. [39, 38], and it is closely related to SSL (SSL) [1, 2, 31]. AVS consists of three sub-tasks, each with a corresponding dataset: S4 (S4), MS3 (MS3), and AVSS (AVSS). Since the first two concentrate on binary predictions (sound-producing vs. silent pixel), we mainly focus on the AVSS task, where each pixel is also assigned a class label. S4 and MS3 data are subsets of AVSS. Together with the datasets, Zhou et al. [39, 38] introduced a CNN (CNN) based temporal pixel-wise audio-visual interaction (TPAVI) network. Later AVS methods utilize cross-attention [34] and use audio as a reference query [13, 36, 32, 26, 17, 6, 21, 24] to fuse cross-modal information. To emphasize audio, recent works have introduced bidirectional attention, where audio is used to compute the key and value vectors [37, 4]. Many of the aforementioned approaches draw inspiration from Mask Transformer [3]. Adaptation of the Mask Transformer-based approach for AVS relies on semantic alignment of visual and auditory features [37, 26, 21, 4, 23]. Alternatively, Gong et al. [14] explore selective state-space models for AVS. Mao et al. [28, 27] explore modeling a shared modality space using a conditional variational autoencoder [19] and a diffusion model [16] to learn an effective multimodal latent space for segmentation. Many AVS models are built on top of image segmentation models. As a result, they inherit the same architectural components: hierarchical backbones, multi-scale feature pyramids, and localized attention. These components add task-specific complexity to the plain ViT architecture utilized by modern VFMs. Because AVS models deviate from plain ViT design, they cannot benefit from VFM pretraining without significant re-engineering. Kerssies et al. [18] demonstrate this incompatibility in the image segmentation domain. This motivates us to build a simpler, efficient, and scalable alternative to specialized architectures in the audio-visual segmentation domain. 3 Method Figure 2: Overview of EASE. Audio and visual frames are encoded, and the encoded audio is enhanced by soft clustering it to semantically representative auditory centers. The query generator is used to generate sparse audio queries before concatenating them with the visual features. The concatenated sequence is processed with the remaining L2L_2 encoder blocks and fused at each block with the enhanced audio features. The connection between enhanced audio features and AV Fusion is omitted for clarity. Finally, a mask module is employed to produce the final mask predictions from the visual tokens and the class predictions from the sparse audio query tokens. EASE (EASE), presented in Fig. 2, is a simplified approach for AVS built on top of a plain ViT architecture making it fully compatible with modern VFMs. It also introduces new components: a Gumbel-Softmax-based audio feature enhancement module, a mid-backbone audio query injection strategy, and a bidirectional audio-visual fusion module with learned soft clustering. Our model achieves state-of-the-art performance while being up to 3×3× faster than existing methods. Audio Feature Extraction. We follow the standard pipeline established in [39, 38]. Audio is encoded with a pretrained and frozen VGGish [15] encoder, producing frame-level features a∈ℝT×128 f_a∈R^T× 128, where T is the number of frames. This yields a single feature vector per frame, which lacks discriminative semantic structure across different sound sources and intra-class variations. To address this, we introduce a clustering step that groups audio features around K learned semantic centers a∈ℝT×K×128 c_a∈R^T× K× 128. Each frame is assigned to the centers based on feature similarity, producing audio representations that better capture semantic variation within and across sound categories. We refer to this process as audio feature enhancement. In our experiments, setting K to the number of semantic classes in AVSBench [39, 38] yielded the best results. The cluster assignments are learned end-to-end via the Gumbel-Softmax operation, which enables discrete-like cluster selection while remaining fully differentiable during training. Standard softmax produces soft, distributed assignments across all K centers, whereas Gumbel-Softmax encourages sparser, more committed assignments that correspond to distinct semantic categories. Following [25], we use a c_a to compute a discriminative semantic enhancement for a f_a, which reduces semantic bias arising from intra-class variation. The resulting enhanced audio feature is denoted ^a∈ℝT×K×128 f_a∈R^T× K× 128. Our approach differs from DDESeg [25], which precomputes the audio feature bank prior to training. By learning the cluster centers and assignments end-to-end, our method eliminates dataset-level precomputation and adaptively captures the semantic structure of the audio, improving the discriminative power of the features for AVS. Mask Prediction. The mask prediction design adopts the plain ViT segmentation approach of Kerssies et al. [18] as its structural foundation. Our contribution is the mid-backbone audio query injection strategy, which adapts their design to the multimodal AVS setting. EASE employs a ViT-based [10] visual backbone to encode input video frames. Rather than processing visual and audio tokens jointly from the start, we split the backbone into two stages at block L1L_1. The first stage processes visual tokens independently, producing rich visual representations m∈ℝT×Nm×D f_m∈R^T× N_m× D, where NmN_m is the number of patch tokens, and D is the shared feature dimension. After L1L_1 blocks, we generate sparse audio queries from the enhanced audio feature ^a f_a using the query generator (Query Gen in Fig. 2). This is done by cross-attending ^a f_a with a learnable query vector, using ^a f_a as keys and values. The audio representation is then projected to a shared feature dimension D of the visual backbone, yielding a set of sparse queries ∈ℝT×Nq×D q∈R^T× N_q× D, where NqN_q is the number of query tokens. Since m f_m already carries positional information from the L1L_1 stage, learnable positional encodings are added only to q. The queries are then concatenated with the visual token sequence along the sequence dimension to form the joint cross-modal sequence (0)=[;m]∈ℝT×(Nq+Nm)×D s^(0)= [\, q\;;\; f_m\, ]∈R^T×(N_q+N_m)× D. At each block l∈L1+1,…,L2l∈\L_1+1,…,L_2\, we first split (l−1) s^(l-1) back into its constituent sequences (l−1) q^(l-1) and m(l−1) f_m^(l-1), apply the audio-visual fusion module to the visual tokens using ^a f_a, and re-form the joint sequence before passing it through the ViT block. All AVFusionAVFusion modules across L2L_2 blocks share weights, keeping the added parameter count negligible. Mask predictions are produced directly from (L2) s^(L_2). Following [18, 7], a small mask module predicts the final segmentation masks from mL2 f_m^L_2 and the class labels from L2 q^L_2. During training, we also apply this module after each L1+1,…,L2−1\L_1+1,…,L_2-1\ block to produce intermediate masks that constrain the self-attention of (l) q^(l), providing localization supervision. Since masked attention is expensive, we follow [18] and anneal it during training. Audio-Visual Fusion Module. The AVFusionAV~Fusion module takes visual mask features m(l−1) f_m^(l-1) and enhanced audio features ^a f_a as input and returns enhanced visual tokens ~m(l) f_m^(l). It operates only on token sequences of dimension D, so it imposes no constraints on the visual backbone and applies to any ViT-based model without modification. Prior to fusion, ^a f_a is projected to the shared dimension D. Stage 1: Visual-Guided Audio Suppression We cluster the visual tokens using Gumbel-Softmax clustering, with KvK_v learnable visual cluster centers, producing visual clusters m∈ℝT×Kv×D c_m ^T× K_v× D. In our experiments, Kv=5K_v=5 yielded the best results. We then attend ^a f_a over m c_m via cross-attention to assess how well each audio cluster is grounded in the visual scene, and map the result to a score ∈[0,1]T×Kv s∈[0,1]^T× K_v through a linear layer and sigmoid operation. Scaling ^a f_a with s yields visually-grounded audio features ~a f_a mitigating the interference of off-screen sounds. Stage 2: Audio-Conditioned Visual Feature Update We condition the visual tokens on ~a f_a via a second cross-attention, with m(l−1) f_m^(l-1) as queries and ~a f_a as keys and values, producing the updated visual tokens ~m(l) f_m^(l). These are re-concatenated with (l) q^(l) and processed through the next ViT block. 4 Experiments 4.1 Setup Datasets. We utilize the widely adopted audio-visual segmentation dataset AVSBench [39, 38]. The complete dataset includes three subsets: S4, MS3, and AVSS. Videos are 10 seconds long, with one frame per second extracted for segmentation. Input data is uniformly reshaped to 224×224224× 224 (2242224^2) or 384×384384× 384 (3842384^2). Following prior works, masks are predicted in 224×224224× 224 spatial size. Implementation. We utilize pretrained PVTv2 [35] and DINOv3 [33] as visual backbone networks. While PVTv2 is widely adopted in AVS research, our simplified model design enables the effective use of VFMs like DINOv3 with a modest computation budget. For PVTv2, we use Stage 3 visual tokens and disable spatial reduction for L2L_2 blocks to improve efficiency. In our experiments, L2=6L_2=6 and L2=4L_2=4 yielded the best results for PVTv2 and DINOv3 respectively. For DINOv3 training, we follow [18], including the loss computation. For PVTv2, we adopt the same training strategy but disable the layer-wise learning rate decay to enable full model fine-tuning. For training and testing, we utilize two NVIDIA A100 GPUs. We train S4 for 46 and 22 epochs, MS3 for 80 and 40 epochs, and AVSS for 36 and 22 epochs for PVTv2 and DINOv3 models, respectively. In GPU hours, training takes up to 7 and 5 hours for S4, up to 1.5 hours for both for MS3, and up to 15 and 11 hours for AVSS using the PVTv2 and DINOv3. All methods, except DDESeg [25], use the frozen VGGish [15] audio encoder (72M parameters) as their audio backbone. DDESeg [25] replaces VGGish [15] with HTSAT, a substantially smaller and more modern audio encoder (29M parameters). We report parameter counts, including the audio encoder, to ensure transparency in Table 1. Evaluation Metrics. For quantitative evaluation, we follow common practice and use the mean Jaccard index (J) [11] and F-score with β2=0.3β^2=0.3 (ℱF). To evaluate the model efficiency, we utilize inference speed frames per second (FPS) and the average number of floating-point operations (FLOPs) calculated with a batch size of 1 on a single NVIDIA RTX 4090 GPU. FLOPs are obtained using the DeepSpeed [30] profiler and reported as GFLOPs (FLOPs ×109× 10^9). Note that reported GLOPs are not operations per second but rather the number of floating-point operations. We evaluate all models using their official codebases while employing a unified dataloading pipeline to eliminate differences in dataloading efficiency. 4.2 Main Results Table 1: Quantitative comparison on AVSS [39, 38] with efficiency evaluations. We compare performance on AVSS and throughput between EASE and SotA approaches with open-source code. †Retrained on the standard data split and benchmark metrics. ‡DDESeg uses HTSAT (29M) as its audio encoder, while all other methods use VGGish (72M) [15]. Parameter counts are reported inclusive of audio encoders. AVSS Method Backb. Params Inp. ↑J ℱ↑F FPS↑ GFLOPs↓ AVSegFormer [13] PVTv2 186M 2242224^2 36.7 42.0 27 99 AVSegFormer [13] PVTv2 186M 5122512^2 37.3 42.8 23 504 AAVS [26] Swin-B 187M 3842384^2 48.5 53.2 69 151 Selm [20] Swin-B 186M 4482448^2 41.3 46.9 71 258 COMBO [37] PVTv2 499M 2242224^2 42.1 46.1 73 346 DDESeg†‡ [25] Transf. 133M 2242224^2 43.3 48.7 27 179 EASE PVTv2 147M 2242224^2 42.3 47.6 159 40 EASE PVTv2 147M 3842384^2 46.1 50.6 141 105 EASE ViT-B 187M 2242224^2 45.8 50.7 365 96 EASE ViT-B 187M 3842384^2 49.8 54.1 181 291 EASE ViT-L 424M 2242224^2 52.2 57.2 225 227 EASE ViT-L 424M 3842384^2 56.5 61.0 86 721 Table 2: Quantitative comparison on S4, MS3 and AVSS categorized by backbone model and input size [39, 38]. *MS3 model was pretrained on S4 data. †Retrained using standard data split and benchmark metrics. ‡DDESeg uses HTSAT as its audio encoder, while all other methods use VGGish [15]. AVSS S4 MS3 Method Backb. Inp. ↑J ℱ↑F ↑J ℱ↑F ↑J ℱ↑F TPAVI [39] PVTv2 2242224^2 29.8 35.2 78.7 87.9 54.0 64.5 CATR [21] PVTv2 2242224^2 32.8 38.5 84.4 91.3 62.7 74.5 ECMVAE [28] PVTv2 2242224^2 - - 81.7 90.1 57.8 70.8 AQFormer* [17] PVTv2 2242224^2 - - 81.6 89.4 62.2 72.7 AVSegFormer [13] PVTv2 2242224^2 36.7 42.0 82.1 89.9 58.4 69.3 COMBO [37] PVTv2 2242224^2 42.1 46.1 84.7 91.9 59.2 71.2 CAVP [6] PVTv2 2242224^2 30.4 35.3 78.8 88.9 55.8 67.1 EASE* PVTv2 2242224^2 42.3 47.6 80.4 89.7 60.8 68.7 AVSegFormer [13] PVTv2 5122512^2 37.3 42.8 83.1 90.5 61.3 73.0 AVS-Mamba [14] PVTv2 4482448^2 39.7 45.1 85.0 92.6 68.6 78.6 Selm [20] PVTv2 4482448^2 41.3 46.9 83.5 91.2 60.3 71.3 EASE* PVTv2 3842384^2 46.1 50.6 83.0 90.9 61.9 69.8 AVSC [23] Swin-B 2242224^2 - - 81.3 88.6 59.5 65.7 BAVS [24] Swin-B 2242224^2 33.6 37.5 82.7 89.8 59.6 65.9 QDFormer [22] Swin-T 2242224^2 - - 79.5 88.2 61.9 66.1 DDESeg†‡ [25] Trans. 2244224^4 43.3 48.7 89.4 93.2 67.8 75.1 EASE ViT-B 2242224^2 45.8 50.7 83.5 92.0 61.4 70.2 EASE ViT-L 2242224^2 52.2 57.2 85.5 92.9 65.0 71.1 AAVS [26] Swin-B 3842384^2 48.5 53.2 83.2 91.3 67.3 77.6 EASE ViT-B 3842384^2 49.8 54.1 86.2 93.3 62.9 73.6 EASE ViT-L 3842384^2 56.5 61.0 87.7 94.1 70.2 76.5 AVSS Quantitative Comparisons. Table 1 shows quantitative comparisons in AVSS [39, 38] benchmark combined with efficiency statistics. AVS-Mamba [14] or BAVS [24] are excluded from the efficiency evaluation as their source code is not publicly available. Their AVSS segmentation scores are reported in Table 2, where EASE with ViT-L achieves superior performance. For the DDESeg [25] model, weights are not publicly available, so we train and evaluate the model following the AVSS [39, 38] standard. For COMBO [37], we use precomputed maskige representations while evaluating the efficiency. Using PVTv2 [35] together with a lightweight audio-visual fusion module, our model outperforms methods with considerably more complex architectures. As noted in Section 4.1, direct parameter comparisons are complicated by inconsistent audio backbones. The smaller parameter count of DDESeg [25] stems from its smaller audio backbone rather than its visual or fusion design. Increasing the input frame size further improves performance while EASE maintains high computational efficiency. However, PVTv2 is not a plain ViT. It introduces a hierarchical pyramid structure with overlapping patch embeddings and spatial-reduction attention, which enables multi-scale feature extraction rather than a single-scale global representation. Since our approach is not dependent on a specialized architecture, we can utilize a VFM and optimization techniques designed for plain attention. By utilizing a pretrained ViT [33], we achieve SotA performance with superior model throughput. The benefit of large-scale pretraining of VFMs is clear in the AVSS task, where the class variation is high. Despite utilizing more parameters with ViT [10] based approaches, EASE achieves superior inference throughput. This highlights that the efficiency gains of EASE are architectural in nature, originating from the plain ViT design and its compatibility with optimized attention implementations such as FlashAttention [8]. Although our largest configuration incurs a higher GFLOPs count than the compared baselines, its throughput remains competitive, thanks to modern attention acceleration techniques. In applications such as video understanding, robotics, immersive media, and assistive technologies, model throughput is crucial: higher FPS enables real-time operation and smoother integration into larger processing pipelines. A more comprehensive comparison with existing methods on the AVSS task without efficiency metrics is in Table 2. S4 and MS3 Quantitative Comparisons. With the ViT-L backbone, EASE achieves the best MS3 Jaccard index and the best S4 F-score among all compared specialized methods [39, 13, 37, 26, 22, 20, 17, 6, 21, 24, 23, 14, 28, 25], despite using no task-specific components, as shown in Table 2. The results on ViT-B are also highly comparable. On the remaining metrics, specialized models retain an advantage in the low-data S4 and MS3 regimes. We attribute this to the small dataset sizes: the built-in inductive biases of specialized modules, such as multiscale feature extraction, audio source separation, and localized processing, help capture fine-grained spatial detail when training data is very limited. Such specialization may also increase the risk of overfitting and, as shown in Table 1, reduce overall efficiency. 4.3 Ablations Table 3: Effect of pretraining on MS3. We pretrain EASE with PVTv2 [35] on S4 data and fine-tune it using MS3. MS3 Method Backb. Input Pretrained ↑J ℱ↑F EASE PVTv2 2242224^2 55.3 67.4 EASE PVTv2 2242224^2 60.8 68.7 EASE PVTv2 3842384^2 59.4 70.1 EASE PVTv2 3842384^2 61.9 69.8 Effect of Pretraining on MS3. Since our approach does not rely on architectural inductive biases and PVTv2 [35] is less extensively pretrained than DINOv3 [33], we first pretrain the PVTv2-based model on the S4 task. This stage enables the model to learn basic audio-visual segmentation concepts before fine-tuning on the limited MS3 dataset. By pretraining the model with a 224×224224× 224 input size, we gain approximately 9% and 2% performance increases in the Jaccard index and the F-score, respectively. For the 384×384384× 384 input model, pretraining increases the Jaccard index by roughly 4%, but reduces the F-score by approximately 0.4%. Table 4: Ablation on different audio enhancers. EASE performance using DINOv3-B [33] 2242224^2 model on AVSBench-Semantic [38] benchmark. AVSS Audio Enhancer Type ↑J ℱ↑F Linear 44.3 49.3 Precomputed Clusters 43.9 48.8 Learned Clusters 45.8 50.7 Audio Enhancer. To assess the impact of audio feature enhancement, we evaluate three configurations: our learned clustering, the pre-calculated audio feature bank enhancement from DDESeg [25], and a single linear projection layer as a baseline. The learned clustering-based audio enhancement significantly outperforms the alternatives. The pre-calculated audio feature bank from DDESeg [25] does not improve over the single linear projection baseline, and in some cases reduces performance, suggesting its static nature cannot capture the context-dependent audio-visual interactions in segmentation. Table 5: Ablation on different Audio-Visual Fusion Modules. EASE performance using DINOv3-B [33] 2242224^2 model on AVSBench-Semantic [38] benchmarks. AVSS Fusion Module Type ↑J ℱ↑F Bi-way Attention 43.5 48.7 AVFusionAVFusion w/o Clustering 44.4 49.5 AVFusionAVFusion 45.8 50.7 Audio-Visual Fusion. We further ablate the design of the audio-visual fusion module. Table 5 compares three fusion configurations: a simple cross-attention baseline, the bi-directional attention module from CAVP [6], and the clustering-based bi-directional fusion used in EASE, which is inspired by DDESeg [25]. The clustering-based fusion module significantly outperforms the alternatives, effectively capturing the complex relationships between the modalities. 5 Conclusion We presented EASE, a simplified encoder-only architecture that removes task-specific complexity from AVSS models in favor of a plain ViT backbone. By combining learned audio enhancement with a lightweight fusion module, EASE achieves SotA accuracy at 3× the speed of prior methods. EASE shows AVS can exploit modern VFMs and offers a simpler, scalable foundation for future research. References [1] R. Arandjelovic and A. Zisserman (2017) Look, listen and learn. In ICCV, Cited by: §2. [2] R. Arandjelovic and A. Zisserman (2018) Objects that sound. In ECCV, Cited by: §2. [3] N. Carion, F. Massa, G. Synnaeve, N. Usunier, A. Kirillov, and S. Zagoruyko (2020) End-to-end object detection with transformers. In ECCV, Cited by: §2. [4] T. Chen, Z. Tan, T. Gong, Q. Chu, Y. Wu, B. Liu, et al. (2024) Bootstrapping audio-visual video segmentation by strengthening audio cues. TCSVT. Cited by: §2, §2. [5] W. Chen, X. Du, F. Yang, L. Beyer, X. Zhai, T.-Y. Lin, et al. (2022) A simple single-scale vision transformer for object localization and instance segmentation. In ECCV, Cited by: §1, §1. [6] Y. Chen, Y. Liu, H. Wang, F. Liu, C. Wang, H. Frazer, et al. (2024) Unraveling instance associations: a closer look for audio-visual segmentation. In CVPR, Cited by: §2, §4.2, §4.3, Table 2. [7] B. Cheng, I. Misra, A. G. Schwing, A. Kirillov, and R. Girdhar (2022) Masked-attention mask transformer for universal image segmentation. In CVPR, Cited by: §3. [8] T. Dao (2024) FlashAttention-2: faster attention with better parallelism and work partitioning. In ICLR, Cited by: §1, §4.2. [9] M. Dehghani, J. Djolonga, B. Mustafa, P. Padlewski, J. Heek, J. Gilmer, et al. (2023) Scaling vision transformers to 22 billion parameters. In ICML, Cited by: §1. [10] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, et al. (2021) An image is worth 16x16 words: transformers for image recognition at scale. ICLR. Cited by: §1, §1, §3, §4.2. [11] M. Everingham, L. Van Gool, C. K. I. Williams, J. Winn, and A. Zisserman (2010) The pascal visual object classes (voc) challenge. IJCV. Cited by: §4.1. [12] Y. Fang, B. Liao, X. Wang, J. Fang, J. Qi, R. Wu, et al. (2021) You only look at one sequence: rethinking transformer in vision through object detection. In NeurIPS, Cited by: §1, §1. [13] S. Gao, Z. Chen, G. Chen, W. Wang, and T. Lu (2024) Avsegformer: audio-visual segmentation with transformer. In AAAI, Cited by: §1, §2, §4.2, Table 1, Table 1, Table 2, Table 2. [14] S. Gong, Y. Zhuge, L. Zhang, Y. Wang, P. Zhang, L. Wang, et al. (2025) Avs-mamba: exploring temporal and multi-modal mamba for audio-visual segmentation. TMM. Cited by: §2, §4.2, §4.2, Table 2. [15] S. Hershey, S. Chaudhuri, D. P. W. Ellis, J. F. Gemmeke, A. Jansen, R. C. Moore, et al. (2017) CNN architectures for large-scale audio classification. In ICASSP, Cited by: §3, §4.1, Table 1, Table 1, Table 2, Table 2. [16] J. Ho, A. Jain, and P. Abbeel (2020) Denoising diffusion probabilistic models. In NeurIPS, Cited by: §2. [17] S. Huang, H. Li, Y. Wang, H. Zhu, J. Dai, J. Han, et al. (2023) Discovering sounding objects by audio queries for audio visual segmentation. arXiv preprint arXiv:2309.09501. Cited by: §2, §4.2, Table 2. [18] T. Kerssies, N. Cavagnero, A. Hermans, N. Norouzi, G. Averta, B. Leibe, et al. (2025) Your vit is secretly an image segmentation model. In CVPR, Cited by: §1, §1, §2, §3, §3, §4.1. [19] D. P. Kingma and M. Welling (2013) Auto-encoding variational bayes. ICLR. Cited by: §2. [20] J. Li, S. Yu, Y. Wang, L. Wang, and H. Lu (2024) Selm: selective mechanism based audio-visual segmentation. In ACM M, Cited by: §1, §4.2, Table 1, Table 2. [21] K. Li, Z. Yang, L. Chen, Y. Yang, and J. Xiao (2023) Catr: combinatorial-dependence audio-queried transformer for audio-visual video segmentation. In ACM M, Cited by: §2, §2, §4.2, Table 2. [22] X. Li, J. Wang, X. Xu, X. Peng, R. Singh, Y. Lu, et al. (2024) Qdformer: towards robust audiovisual segmentation in complex environments with quantization-based semantic decomposition. In CVPR, Cited by: §1, §4.2, Table 2. [23] C. Liu, P. P. Li, X. Qi, H. Zhang, L. Li, D. Wang, et al. (2023) Audio-visual segmentation by exploring cross-modal mutual semantics. In ACM M, Cited by: §2, §4.2, Table 2. [24] C. Liu, P. Li, H. Zhang, L. Li, Z. Huang, D. Wang, et al. (2024) BAVS: bootstrapping audio-visual segmentation by integrating foundation knowledge. TMM. Cited by: §2, §4.2, §4.2, Table 2. [25] C. Liu, L. Yang, P. Li, D. Wang, L. Li, and X. Yu (2025) Dynamic derivation and elimination: audio visual segmentation with enhanced audio semantics. In CVPR, Cited by: §3, §3, §4.1, §4.2, §4.2, §4.2, §4.3, §4.3, Table 1, Table 2. [26] J. Ma, P. Sun, Y. Wang, and D. Hu (2024) Stepping stones: a progressive training strategy for audio-visual semantic segmentation. In ECCV, Cited by: §1, §2, §2, §4.2, Table 1, Table 2. [27] Y. Mao, J. Zhang, M. Xiang, Y. Lv, D. Li, Y. Zhong, et al. (2025) Contrastive conditional latent diffusion for audio-visual segmentation. TIP. Cited by: §2. [28] Y. Mao, J. Zhang, M. Xiang, Y. Zhong, and Y. Dai (2023) Multimodal variational auto-encoder based audio-visual segmentation. In ICCV, Cited by: §2, §4.2, Table 2. [29] M. Oquab, T. Darcet, T. Moutakanni, H. V. Vo, M. Szafraniec, V. Khalidov, et al. (2024) DINOv2: learning robust visual features without supervision. TMLR. Cited by: §1, §1. [30] J. Rasley, S. Rajbhandari, O. Ruwase, and Y. He (2020) Deepspeed: system optimizations enable training deep learning models with over 100 billion parameters. In ACM SIGKDD, Cited by: §4.1. [31] A. Senocak, T.-H. Oh, J. Kim, M.-H. Yang, and I. S. Kweon (2018) Learning to localize sound source in visual scenes. In CVPR, Cited by: §2. [32] Z. Shi, Q. Wu, F. Meng, L. Xu, and H. Li (2024) Cross-modal cognitive consensus guided audio-visual segmentation. TMM. Cited by: §1, §2. [33] O. Siméoni, H. V. Vo, M. Seitzer, F. Baldassarre, M. Oquab, C. Jose, et al. (2025) DINOv3. Cited by: §1, §1, §4.1, §4.2, §4.3, Table 4, Table 4, Table 5, Table 5. [34] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, et al. (2017) Attention is all you need. In NeurIPS, Cited by: §1, §2. [35] W. Wang, E. Xie, X. Li, D.-P. Fan, K. Song, D. Liang, et al. (2022) Pvt v2: improved baselines with pyramid vision transformer. CVM. Cited by: §1, §4.1, §4.2, §4.3, Table 3, Table 3. [36] Z. Wang, Q. Yang, L. Shi, J. Yu, Q. Liang, F. Li, et al. (2024) AVESFormer: efficient transformer design for real-time audio-visual segmentation. arXiv preprint arXiv:2408.01708. Cited by: §1, §2. [37] Q. Yang, X. Nie, T. Li, P. Gao, Y. Guo, C. Zhen, et al. (2024) Cooperation does matter: exploring multi-order bilateral relations for audio-visual segmentation. In CVPR, Cited by: §1, §2, §2, §4.2, §4.2, Table 1, Table 2. [38] J. Zhou, X. Shen, J. Wang, J. Zhang, W. Sun, J. Zhang, et al. (2024) Audio-visual segmentation with semantics. IJCV. Cited by: §1, §2, §2, §3, §3, §4.1, §4.2, Table 1, Table 1, Table 2, Table 2, Table 4, Table 4, Table 5, Table 5. [39] J. Zhou, J. Wang, J. Zhang, W. Sun, J. Zhang, S. Birchfield, et al. (2022) Audio–visual segmentation. In ECCV, Cited by: §1, §2, §2, §3, §3, §4.1, §4.2, §4.2, Table 1, Table 1, Table 2, Table 2, Table 2.