Paper deep dive
Semi-Supervised Sound Event Detection with Conditional Mixup and Embedding-Level Contrastive Loss
Nian Shao, Xian Li, Xiaofei Li
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 98%
Last extracted: 7/5/2026, 3:04:27 AM
Summary
This paper proposes ATST-SEDv2, an improved semi-supervised sound event detection (SED) framework that enhances the previous ATST-SED model. The core innovation is 'conditional mixup', which unifies 'composition mixup' (used for pseudo-label learning) and 'perturbation mixup' (used for contrastive learning) into a single framework based on an interpolation coefficient threshold. The method also introduces embedding-level self-supervised contrastive losses to better leverage unlabeled data. Experimental results on the DESED dataset show that the proposed method achieves state-of-the-art performance, improving both PSDS1 and PSDS2 scores for both CRNN and ATST-SED architectures.
Entities (9)
Relation Signals (6)
Conditional Mixup → combines → Composition Mixup
confidence 100% · we propose conditional mixup, which combines composition mixup and perturbation mixup
Conditional Mixup → combines → Perturbation Mixup
confidence 100% · we propose conditional mixup, which combines composition mixup and perturbation mixup
ATST-SEDv2 → evaluatedon → DESED
confidence 100% · The resulting model achieves 0.645 PSDS1 and 0.822 PSDS2 on the DESED validation set
ATST-SEDv2 → improves → ATST-SED
confidence 100% · In this work, we further improve the framework by adopting an embedding-level self-supervised contrastive loss inspired by ATST-Frame pretraining.
Embedding-level Contrastive Loss → inspiredby → ATST-Frame
confidence 100% · adopting an embedding-level self-supervised contrastive loss inspired by ATST-Frame pretraining.
ATST-SEDv2 → uses → Conditional Mixup
confidence 100% · To resolve this mismatch, we propose conditional mixup, which combines composition mixup and perturbation mixup in one semi-supervised framework
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Sound event detection (SED) is a core module for acoustic environmental analysis, yet its performance is often limited by scarce labeled data. Recent systems leverage large pretrained audio foundation models, but effective fine-tuning remains challenging because labeled data are limited while unlabeled data are abundant. A previous work, ATST-SED, addressed this problem with a pseudo-label based semi-supervised fine-tuning framework. In this work, we further improve the framework by adopting an embedding-level self-supervised contrastive loss inspired by ATST-Frame pretraining. This contrastive objective better exploits unlabeled data during fine-tuning. One challenge is that mixup serves different roles in the two objectives: pseudo-label learning uses composition mixup, while contrastive learning treats mixup as a perturbation. To resolve this mismatch, we propose conditional mixup, which combines composition mixup and perturbation mixup in one semi-supervised framework and defines the corresponding embedding-level contrastive losses. The resulting model achieves 0.645 PSDS1 and 0.822 PSDS2 on the DESED validation set, establishing a new state of the art.
Tags
Links
- Source: https://arxiv.org/abs/2606.29901v1
- Canonical: https://arxiv.org/abs/2606.29901v1
Trouble viewing inline? Open PDF directly →
Full Text
35,021 characters extracted from source content.
Expand or collapse full text
Semi-Supervised Sound Event Detection with Conditional Mixup and Embedding-Level Contrastive Loss Nian Shao1,2, Xian Li2,3, Xiaofei Li2,3∗ 1Zhejiang University, Hangzhou, China.2School of Engineering, Westlake University, Hangzhou, China3Westlake University & Westlake Institute for Advanced Study, Hangzhou, China.∗Corresponding author. Email: lixiaofei@westlake.edu.cn Abstract Sound event detection (SED) is a core module for acoustic environmental analysis, yet its performance is often limited by scarce labeled data. Recent systems leverage large pretrained audio foundation models, but effective fine-tuning remains challenging because labeled data are limited while unlabeled data are abundant. A previous work, ATST-SED, addressed this problem with a pseudo-label based semi-supervised fine-tuning framework. In this work, we further improve the framework by adopting an embedding-level self-supervised contrastive loss inspired by ATST-Frame pretraining. This contrastive objective better exploits unlabeled data during fine-tuning. One challenge is that mixup serves different roles in the two objectives: pseudo-label learning uses composition mixup, while contrastive learning treats mixup as a perturbation. To resolve this mismatch, we propose conditional mixup, which combines composition mixup and perturbation mixup in one semi-supervised framework and defines the corresponding embedding-level contrastive losses. The resulting model achieves 0.645 PSDS1 and 0.822 PSDS2 on the DESED validation set, establishing a new state of the art. Keywords— sound event detection, semi-supervised learning, contrastive learning, mixup, ATST. I Introduction Sound conveys rich information about the surrounding environment, but recognizing sound events remains challenging for machines [26]. Sound event detection (SED) aims to detect sound events and their temporal boundaries in polyphonic audio scenes. Since frame-level annotation is expensive, SED suffers from severe label scarcity. In practice, SED systems are trained with multiple annotation types. These include weakly labeled data, which provide clip-level event tags without temporal boundaries [24]; strongly labeled synthetic and real data, which provide time-aligned event annotations [26, 7]; and unlabeled data [20]. This setup makes semi-supervised learning a natural choice for modern SED benchmarks. Prior work has improved semi-supervised SED with data augmentation, pseudo-label based learning, and stronger architectures for frame-level modeling [12, 10, 29, 18]. Recent SED systems also benefit from pretrained self-supervised audio models, including BEATs, AST, and ATST-Frame [3, 15, 16, 8]. Trained on large-scale audio corpora, these models provide stronger representations than task-specific training alone. In particular, ATST-Frame is trained with a frame-level self-supervised contrastive objective and learns frame-level audio representations that transfer well to temporally localized downstream tasks such as SED [16]. In prior ATST-SED work [23], ATST-Frame was introduced into a CRNN-based detector, and a two-stage pseudo-label based semi-supervised framework was proposed for fine-tuning ATST-Frame for SED, providing a strong baseline. However, the semi-supervised objectives commonly used in SED are dominated by consistency regularization [12, 21, 9]. These objectives are effective for lightweight SED models, but can be much less effective for SED systems built on large pretrained self-supervised encoders [22, 17, 13]. A likely reason is that consistency regularization is too weak compared with the much stronger pretraining tasks. By contrast, self-supervised contrastive learning is a more challenging representation-learning objective and has been shown to improve the linear separability and transferability of learned representations during pretraining [16, 4]. Among existing self-supervised contrastive frameworks, ATST-Frame is particularly suitable for SED because it is trained with a frame-level objective that matches the frame-level nature of SED. We therefore introduce its embedding-level self-supervised contrastive objective into semi-supervised fine-tuning, so that unlabeled data can be better leveraged than pseudo-label consistency alone. Pseudo-label learning and contrastive learning, however, benefit from different uses of mixup. In pseudo-label learning, mixup follows a composition view: the mixed sample should preserve the content of both source samples. In contrastive learning, mixup follows a perturbation view: the newly added sample should only perturb the input sample [19, 14, 16]. Both strategies are effective in their own settings. To exploit both of them within one framework, we propose conditional mixup, which combines composition mixup and perturbation mixup according to the interpolation coefficient. Based on the two mixup cases, we further define the corresponding pseudo-label losses and embedding-level contrastive losses, so that decision-level pseudo-label supervision and embedding-level contrastive supervision can complement each other during fine-tuning. On the DESED validation set, ATST-SEDv2 achieves 0.645 PSDS1 and 0.822 PSDS2 with cSEBB post-processing, establishing a new state of the art. The main contributions are as follows. First, we propose conditional mixup to combine composition mixup and perturbation mixup in one framework. Second, we design the corresponding pseudo-label and embedding-level contrastive losses, which provide an effective semi-supervised objective for both lightweight and pretraining-based SED models. Third, we achieve new state-of-the-art performance on the DESED validation set, reaching 0.645 PSDS1 and 0.822 PSDS2. I Method Polyphonic SED is a multi-class detection task in which a model detects the onset and offset of multiple sound events in an audio clip. This work focuses on the DESED dataset [26], which provides strongly labeled, weakly labeled, and unlabeled training data. To make use of both supervised and unsupervised data, the SED model is trained in a semi-supervised manner. In this work, we aim to strengthen semi-supervised fine-tuning for ATST-SED by adopting an embedding-level self-supervised contrastive objective inspired by ATST-Frame pretraining [16]. In this section, we first review the pseudo-label based semi-supervised fine-tuning baseline. We then describe the mixup strategies used in the baseline system and in contrastive learning, and introduce conditional mixup to unify them. Next, we describe the corresponding embedding-level self-supervised losses and summarize the overall training objective. I-A Pseudo-Label Based Two-Stage Fine-Tuning Baseline Figure 1: Model architecture of the ATST-SED model, which consists of an ATST-SED encoder and a SED classifier. We use ATST-SED as the baseline model [23]. As shown in Fig. 1, the model consists of an ATST-SED encoder and a nonlinear SED classifier [10]. The encoder consists of a convolutional recurrent neural network (CRNN) [10] and a pretrained ATST-Frame model [16]. The pretrained ATST-Frame model serves as an additional feature extractor alongside the 7-layer convolutional neural network (CNN) feature extractor. The features from the two branches are temporally aligned and fused, after which a recurrent neural network (RNN) performs contextual modeling before classification. The baseline fine-tuning framework is divided into two stages. Both stages are trained in a semi-supervised manner. In stage 1, the ATST-Frame model is kept frozen, and only the CRNN and SED classifier are optimized. The supervised binary cross-entropy (BCE) loss (ℒBCE L_BCE) is applied to the labeled data. For unlabeled data, the Mean Teacher (MT) loss (ℒMT L_MT) [25] is used. Specifically, an exponential moving average (EMA) teacher generates pseudo labels for the unlabeled data, and the MT loss minimizes the mean-square error between the student predictions and these pseudo labels. The total loss of stage 1 is ℒstage1=ℒBCE+rMTℒMT, L_stage1= L_BCE+r_MT L_MT, (1) where rMTr_MT is the ramp-up weight of ℒMT L_MT. In stage 2, all parameters are fine-tuned together. Since the amount of labeled data in DESED is still limited for tuning a large pretrained encoder, the baseline further applies mixup and frequency warping (FW) [16] to 50% of the training data to increase data diversity, and assigns larger weights to the unsupervised losses to make fuller use of the unlabeled data. Besides BCE and MT, interpolation consistency training (ICT) loss (ℒICT L_ICT) [27] is used together with mixup for semi-supervised learning. When mixup is applied, the ICT loss replaces the MT loss for the mixed samples. The total loss of stage 2 is ℒstage2=ℒBCE+rMTℒMT+rICTℒICT, L_stage2= L_BCE+r_MT L_MT+r_ICT L_ICT, (2) where rICTr_ICT is the ramp-up weight of ℒICT L_ICT. In this work, we keep the stage-1 training unchanged and only redesign the stage-2 fine-tuning strategy. Figure 2: Flowchart of the proposed pseudo-label and contrastive losses, when using perturbation mixup for data augmentation. I-B Conditional mixup The mixup [28] used in the baseline model combines two samples and treats the sound events from both audio clips as co-occurring in the mixed sample. Formally, for an input sample (i,i)(x_i,y_i) and a randomly selected auxiliary sample (j,j)(x_j,y_j), where ∈ℝT×Dx ^T× D and ∈ℝT×Cy ^T× C, T and D denote the time and feature dimensions, and C is the number of classes, mixup generates a new sample (′,′)(x ,y ) as ′=λi+(1−λ)jx = _i+(1-λ)x_j, and ′=λi+(1−λ)jy = _i+(1-λ)y_j, where λ∈[0,1]λ∈[0,1] is typically drawn from a Beta distribution. In semi-supervised learning, ~i y_i and ~j y_j denote the teacher predictions on ix_i and jx_j, respectively. The mixed pseudo-label is λ~i+(1−λ)~jλ y_i+(1-λ) y_j. Denote the student prediction on ′x as ^′ y . The corresponding pseudo-label loss is ℒICT=‖^′−(λ~i+(1−λ)~j)‖22, L_ICT= \| y -(λ y_i+(1-λ) y_j) \|_2^2, (3) which is the ICT loss [27]. Since this mixup preserves the content of both source clips, we refer to it as composition mixup. Audio contrastive self-supervised learning also adopts mixup [19, 14, 16], but treats the auxiliary sample as a perturbation to the input sample. In this context, λ is typically drawn from a restricted uniform distribution (e.g., in [19], λ∼U(0.6,1)λ (0.6,1)), which ensures that the contribution of the auxiliary sample remains smaller than that of the input sample. We refer to this use of mixup as perturbation mixup. When using it for semi-supervised learning, the mixed sample ′x is constructed in the same way, while only the target of the input sample is preserved, i.e., ′=iy =y_i and ~i y_i is used as the pseudo label in pseudo-label loss. To unify the two mixup methods, we propose conditional mixup. In audio mixup, the interpolation coefficient λ can be interpreted as the relative energy ratio between the input signal and the auxiliary signal, which is proportional to λ1−λ λ1-λ. When λ is close to 0.5, the two signals have similar energy, so events from both signals are treated as co-occurring in the mixture. When λ is large, the input signal dominates the mixture, and the auxiliary signal is treated as a perturbation. We therefore use λ∈[0.5,1]λ∈[0.5,1] together with a threshold τ to determine the mixup mode. When 0.5≤λ≤τ0.5≤λ≤τ, the mixed sample follows the composition view. In this case, we use label combination rather than interpolation to better represent event co-occurrence: the supervised target is the logical OR i∨jy_i _j, while the semi-supervised target is the sum ~i+~j y_i+ y_j, clamped by maximum of 1. Formally, for 0.5≤λc≤τ0.5≤ _c≤τ, we define c′ _c =λci+(1−λc)j, = _cx_i+(1- _c)x_j, (4) c′ _c =i∨j. =y_i _j. (5) The corresponding pseudo-label loss ℒc-psd L_c-psd is ℒc-psd=‖^c′−min(~i+~j,)‖22, L_c-psd= \| y _c-min( y_i+ y_j,1) \|_2^2, (6) When τ<λ≤1τ<λ≤ 1, the mixed sample follows perturbation mixup, and only the target of the input sample is preserved. For τ<λp≤1τ< _p≤ 1, we define p′ _p =λpi+(1−λp)j, = _px_i+(1- _p)x_j, (7) p′ _p =i. =y_i. (8) The corresponding pseudo-label loss ℒp-psd L_p-psd is ℒp-psd=‖^p′−~i‖22, L_p-psd= \| y _p- y_i \|_2^2, (9) The unified pseudo-label objective for conditional mixup is ℒcp-psd=12(ℒc-psd+ℒp-psd). L_cp-psd= 12 ( L_c-psd+ L_p-psd ). (10) I-C Embedding-Level Semi-Supervised Contrastive Losses We now introduce embedding-level semi-supervised contrastive losses. Following [16], contrastive learning forms positive pairs by applying data augmentation to the same sample and aligning the resulting embeddings. In our framework, conditional mixup produces two augmented views for each input, corresponding to the composition and perturbation cases. We therefore define two contrastive losses accordingly. As shown in Fig. 2, a projector and a predictor are attached after the ATST-SED encoder for contrastive learning. The contrastive loss is computed between the student predictor output and the teacher projector output, which helps prevent collapse in contrastive learning [4]. The extracted embeddings are ℓ2 _2-normalized along the hidden dimension. Perturbation case For the positive pair (i,p′)(x_i,x _p) created by perturbation mixup, the auxiliary sample is treated only as a perturbation to the input sample. Therefore, the contrastive objective is to maximize the similarity between the mixed sample and the input sample. Let ~i∈ℝT×H z_i ^T× H denote the teacher embedding of ix_i, and let p′∈ℝT×Hp _p ^T× H denote the student embedding of the mixed sample p′x _p. The frame-level contrastive loss is defined as ℒconp(i,p′)=1T∑t=1T‖~i(t,:)−p′(t,:)‖22, L_con^p(x_i,x _p)= 1T _t=1^T\| z_i(t,:)-p _p(t,:)\|_2^2, (11) where (t,:)(t,:) denotes the embedding at the t-th frame. As in [16], we also use the reversed positive pair by feeding p′x _p to the teacher and ix_i to the student, and denote the resulting loss by ℒcon′p L _con. The total perturbation contrastive loss is ℒp-con=12(ℒconp+ℒcon′p). L_p-con= 12( L_con^p+ L _con). (12) Composition case For the positive pair (i,c′)(x_i,x _c) created by composition mixup, both the input sample and the auxiliary sample contribute to the mixed content. Therefore, the contrastive objective is to maximize the similarity between the mixed sample and both source samples. Let ~i,~j∈ℝT×H z_i, z_j ^T× H denote the teacher embeddings of ix_i and jx_j, and let c′∈ℝT×Hp _c ^T× H denote the student embedding of the mixed sample c′x _c. We use the average of the two teacher embeddings as the contrastive target. The contrastive loss is defined as ℒconc(i,c′)=1T∑t=1T‖12(~i(t,:)+~j(t,:))−c′(t,:)‖22. L_con^c(x_i,x _c)= 1T _t=1^T \| 12 ( z_i(t,:)+ z_j(t,:) )-p _c(t,:) \|_2^2. (13) Similarly, we also use the reversed positive pair and denote the corresponding loss by ℒcon′c L _con. The total composition contrastive loss is ℒc-con=12(ℒconc+ℒcon′c). L_c-con= 12( L_con^c+ L _con). (14) The total contrastive loss used in this work is ℒcon=12(ℒc-con+ℒp-con). L_con= 12( L_c-con+ L_p-con). (15) I-D Overall training objective The proposed method is used for stage-2 fine-tuning of ATST-SED. The overall training objective is composed of three parts: the supervised BCE loss, the pseudo-label based loss, and the contrastive loss. We use the proposed conditional mixup for data augmentation. Frequency warping [16] is also applied to the mixed samples to further increase the difficulty of the semi-supervised task. The unmixed input samples remain unaugmented so that the teacher model can generate reliable pseudo labels, while the original Mean Teacher loss remains unchanged. The overall training objective is ℒtotal=ℒBCE+rMTℒMT+rcp-psdℒcp-psd+rconℒcon, L_total= L_BCE+r_MT L_MT+r_cp-psd L_cp-psd+r_con L_con, (16) where rcp-psdr_cp-psd and rconr_con are the weights for the pseudo-label loss and contrastive loss, respectively. I Experimental Results I-A Experiment setup Code is available on our website111https://github.com/Audio-WestlakeU/ATST-SED. All experiments are conducted on DESED [26]. The training set comprises 1,578 weakly labeled clips, 10,000 synthesized strongly labeled clips, 14,412 unlabeled clips, and 3,470 additional real strongly labeled clips from AudioSet [7]. The evaluation is performed on the DESED validation set with 1,168 strongly labeled audio clips. For preprocessing, all recordings are resampled to 16 kHz and truncated or padded to 10 seconds. For ATST-SED, the CNN branch uses 128-dimensional LogMel features with a frame length of 128 ms and a hop length of 16 ms. The ATST-Frame branch keeps its original preprocessing pipeline. When applying the proposed method, we add a projector and a predictor only during training for embedding-level contrastive regularization, while keeping the inference architecture unchanged. Both modules share the same architecture: a linear layer, batch normalization, a ReLU, and a final linear layer. The batch sizes for real strongly labeled, synthetic strongly labeled, weakly labeled, and unlabeled data are 24, 24, 48, and 48, respectively. We optimize the models with Adam [11]. For ATST-SED stage-2 fine-tuning, the learning-rate setup is the same as in [23]. A layer-wise decaying learning rate is used for the ATST blocks (initial rate 2e-4, decaying by 0.5 per layer), and a uniform initial rate of 2e-4 is used for the CNN, projector, and predictor. The RNN and classifier use a higher initial rate of 2e-3. Fine-tuning lasts for 20,000 steps. All learning rates follow an exponential ramp-up from 0 to their initial values during the first 290 steps. They then cosine-decay to α10 α10. The hyperparameters are empirically set as follows: rMT=70r_MT=70, rcp-psd=17.5r_cp-psd=17.5, rcon=3r_con=3, and the mixup-mode threshold τ=0.55τ=0.55. Post-processing applies median filtering and change-detection-based sound event bounding boxes (cSEBB) [6] to transform soft model predictions into binary decisions. Models are evaluated every 10 epochs on the DESED validation set using PSDS1 and PSDS2 [5]. PSDS1 emphasizes temporal stability, and PSDS2 emphasizes audio tagging accuracy, where higher values indicate better performance. Unless otherwise specified, all reported results use median-filter post-processing. I-B Comparison with the baseline model We first apply the proposed method to the baseline models, and the results are shown in Table I. We also train a CRNN model to show that the proposed method is effective on lightweight SED models. Its training follows the DCASE 2023 baseline setup. When applying our method to CRNN, we use rcp-psd=0.125r_cp-psd=0.125, rcon=0.1r_con=0.1, and τ=0.55τ=0.55. For clarity, we denote “ATST-SED + proposed method” as ATST-SEDv2. Our approach obtains consistent improvements across both architectures. For PSDS1, CRNN improves from 0.384 to 0.403, and ATST-SED improves from 0.583 to 0.607. These gains are non-trivial for PSDS1, because this metric strongly penalizes instability across classes [5]. For PSDS2, CRNN improves from 0.628 to 0.661, and ATST-SED improves from 0.810 to 0.817, indicating improved tagging capacity while maintaining temporal detection quality. The stronger gain on CRNN suggests that the method is especially helpful when the backbone capacity is limited, while the gain on ATST-SED shows that the method is still effective on a strong pretrained baseline. Per-class analysis in Fig. 3 compares our system with ATST-SED, showing that the improvement is distributed across classes rather than concentrated in only a few easy classes. The method particularly benefits challenging categories such as “dog”, “cat”, and “dishes”. These events are short and have high acoustic variability, so they are difficult for SED systems [20]. Overall, the proposed approach effectively enhances semi-supervised learning across both lightweight and pretraining-based SED models. Figure 3: Comparison between ATST-SED [23] and the proposed ATST-SEDv2 on per-class PSDS1 scores. TABLE I: Comparison with the previous baselines. Method PSDS1 PSDS2 CRNN 0.384 0.628 CRNN + proposed method 0.403 0.661 ATST-SED [23] 0.583 0.810 ATST-SED + proposed method (ATST-SEDv2) 0.607 0.817 I-C Ablation Study TABLE I: Ablation study of ATST-SEDv2 components. Method PSDS1 PSDS2 Stage 1 frozen training 0.529 0.778 Stage 2 w. ℒcon L_con 0.560 0.811 w. ℒMT+ℒcp-psd L_MT+ L_cp-psd 0.579 0.800 w. ℒMT+ℒp-psd+ℒp-con L_MT+ L_p-psd+ L_p-con 0.595 0.810 w. ℒMT+ℒc-psd+ℒc-con L_MT+ L_c-psd+ L_c-con 0.601 0.817 w. ℒMT+ℒcp-psd+ℒcon L_MT+ L_cp-psd+ L_con (ATST-SEDv2) 0.607 0.817 Table I reports the ablation results. Each row corresponds to an independent stage-2 fine-tuning setting, and the stage-1 result is listed for reference. Using only the contrastive loss (ℒcon L_con) improves performance over stage 1, which confirms that the contrastive objective is useful in semi-supervised fine-tuning. Pseudo-label losses (ℒMT+ℒcp-psd L_MT+ L_cp-psd) remain essential, and using them alone yields performance close to ATST-SED [23]. However, the gain from contrastive loss is clearly smaller than that from pseudo-label losses. A likely reason is that the contrastive objective and the SED objective do not necessarily share the same optimum. When contrastive loss is used alone, the available supervised data may be too limited to regularize the network toward the SED optimum. Therefore, contrastive loss should be used together with pseudo-label losses to better leverage unlabeled data during fine-tuning. For conditional mixup, we further ablate the composition and perturbation branches separately while keeping the Mean Teacher loss. Both the composition branch (ℒc-psd+ℒc-con L_c-psd+ L_c-con) and the perturbation branch (ℒp-psd+ℒp-con L_p-psd+ L_p-con) are effective when used separately. In particular, the composition-only setting reaches 0.601/0.817, which is clearly better than the original ATST-SED result of 0.583/0.810. This result shows that the composition branch, together with the introduced contrastive loss ℒc-con L_c-con, is effective. Similarly, the perturbation-only setting also yields strong results, validating perturbation mixup as an effective augmentation for semi-supervised learning. Overall, each proposed module improves the stage-1 model when used individually, and the modules are compatible. Combining all components yields the best performance. I-D Comparison with SOTA SED systems TABLE I: Comparison with SOTA SED models (MF: median filter). Methods marked with † use the scores reported in their original papers, and PSDS2 is not provided for them. Method PSDS1 PSDS2 cSEBB MF cSEBB MF ATST-SED [23] 0.618 0.583 0.813 0.810 MAT-SED† [2] 0.602 0.587 - - PMAMiter2_iter2† [1] 0.625 0.597 - - ATST-SEDv2 (ours) 0.645 0.607 0.822 0.817 Table I benchmarks ATST-SEDv2 against competing SOTA methods on the DESED validation set. To ensure a fair comparison, we apply cSEBB to both our model and the ATST-SED baseline. cSEBB uses adaptive thresholds, rather than a single threshold, to detect event boundaries from soft predictions. This allows both low-confidence and high-confidence events to be detected and generally yields more accurate results. We compare with two recent methods: MAT-SED [2] and PMAMiter2_iter2 [1]. Both methods apply self-supervised learning to AST-based [8] SED systems. They first pretrain with self-supervised tasks and then perform semi-supervised fine-tuning with pseudo-label losses. PMAM further applies iterative updates with prototypical modeling and achieves strong PSDS1 performance. Compared with these methods, ATST-SEDv2 uses a simpler strategy by jointly applying contrastive and pseudo-label losses during fine-tuning. Fine-tuning can be completed in a single run with lower training complexity. Even with this simpler design, the method leverages unlabeled data effectively and achieves better PSDS1 performance under both cSEBB and median-filter post-processing. Overall, ATST-SEDv2 establishes new state-of-the-art results on the DESED validation set, reaching 0.645 PSDS1 and 0.822 PSDS2. IV Conclusion In this paper, we extend ATST-SED by introducing an embedding-level self-supervised contrastive objective into semi-supervised fine-tuning. Since pseudo-label learning and contrastive learning favor different mixup semantics, we propose conditional mixup to combine composition mixup and perturbation mixup in one framework, together with the corresponding pseudo-label and embedding-level contrastive losses. In this way, decision-level pseudo-label supervision and embedding-level contrastive supervision complement each other during fine-tuning. This design improves the use of unlabeled data while keeping the inference-time model unchanged. On the DESED validation set, the proposed method achieves state-of-the-art performance, reaching 0.645 PSDS1 and 0.822 PSDS2. References [1] P. Cai, Y. Song, N. Jiang, Q. Gu, and I. McLoughlin (2025) Prototype based masked audio model for self-supervised learning of sound event detection. In ICASSP 2025 - 2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Vol. , p. 1–5. External Links: Document Cited by: §I-D, TABLE I. [2] P. Cai, Y. Song, K. Li, H. Song, and I. McLoughlin (2024) MAT-SED: A Masked Audio Transformer with Masked-Reconstruction Based Pre-training for Sound Event Detection. In Interspeech 2024, p. 557–561. External Links: Document, ISSN 2958-1796 Cited by: §I-D, TABLE I. [3] S. Chen, Y. Wu, C. Wang, S. Liu, D. Tompkins, Z. Chen, W. Che, X. Yu, and F. Wei (2023) BEATs: audio pre-training with acoustic tokenizers. In International Conference on Machine Learning (ICML), p. 5178–5193. Cited by: §I. [4] X. Chen and K. He (2021) Exploring simple siamese representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Cited by: §I, §I-C. [5] J. Ebbers, R. Haeb-Umbach, and R. Serizel (2022) Threshold independent evaluation of sound event detection scores. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), p. 1021–1025. Cited by: §I-A, §I-B. [6] J. Ebbers, F. G. Germain, G. Wichern, and J. Le Roux (2024) Sound Event Bounding Boxes. In Interspeech 2024, p. 562–566. External Links: Document, ISSN 2958-1796 Cited by: §I-A. [7] J. F. Gemmeke, D. P. Ellis, D. Freedman, A. Jansen, W. Lawrence, R. C. Moore, M. Plakal, and M. Ritter (2017) Audio set: an ontology and human-labeled dataset for audio events. In 2017 IEEE international conference on acoustics, speech and signal processing (ICASSP), p. 776–780. Cited by: §I, §I-A. [8] Y. Gong, Y. Chung, and J. Glass (2021) AST: Audio Spectrogram Transformer. In Interspeech 2021, p. 571–575. External Links: Document, ISSN 2958-1796 Cited by: §I, §I-D. [9] Y. Guan, J. Han, H. Song, S. Deng, G. Zheng, T. Zheng, and Y. He (2024) Sound activity-aware based cross-task collaborative training for semi-supervised sound event detection. IEEE/ACM Transactions on Audio, Speech, and Language Processing 32 (), p. 3947–3959. External Links: Document Cited by: §I. [10] L. JiaKai (2018) Mean teacher convolution system for dcase 2018 task 4. Technical report DCASE2018 Challenge. Cited by: §I, §I-A. [11] D. P. Kingma and J. Ba (2015) Adam: a method for stochastic optimization. In International Conference on Learning Representations (ICLR), Cited by: §I-A. [12] C.-Y. Koh, Y.-S. Chen, Y.-W. Liu, and M. R. Bai (2021) Sound event detection by consistency training and pseudo-labeling with feature-pyramid convolutional recurrent neural networks. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), p. 376–380. Cited by: §I, §I. [13] K. Li, P. Cai, and Y. Song (2023-06) LI USTC team’s submission for DCASE 2023 challenge task4a. Technical report DCASE2023 Challenge. Cited by: §I. [14] K. Li, Y. Song, L.-R. Dai, I. McLoughlin, X. Fang, and L. Liu (2023) AST-SED: an effective sound event detection method based on audio spectrogram transformer. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Vol. , p. 1–5. Cited by: §I, §I-B. [15] X. Li and X. Li (2022) ATST: audio representation learning with teacher-student transformer. In Interspeech, p. 4172–4176. Cited by: §I. [16] X. Li, N. Shao, and X. Li (2024) Self-supervised audio teacher-student transformer for both clip-level and frame-level tasks. IEEE/ACM Transactions on Audio, Speech, and Language Processing 32 (), p. 1336–1351. External Links: Document Cited by: §I, §I, §I, §I-A, §I-A, §I-B, §I-C, §I-C, §I-D, §I. [17] C. Liu, T. Kuo, C. Chen, C. Lu, B. Chan, Y. Cheng, and H. Chuang (2023-06) CHT+nsysu sound event detection system with pretrained embeddings extracted from beats model for dcase 2023 task 4. Technical report DCASE2023 Challenge. Cited by: §I. [18] H. Nam, S.-H. Kim, B.-Y. Ko, and Y.-H. Park (2022) Frequency Dynamic Convolution: Frequency-Adaptive Pattern Recognition for Sound Event Detection. In Interspeech, p. 2763–2767. External Links: Document Cited by: §I. [19] D. Niizumi, D. Takeuchi, Y. Ohishi, N. Harada, and K. Kashino (2021-07) BYOL for audio: self-supervised learning for general-purpose audio representation. In 2021 International Joint Conference on Neural Networks (IJCNN), External Links: Document, Link Cited by: §I, §I-B. [20] R. Serizel and N. Turpault (2019-06) Sound Event Detection from Partially Annotated Data: Trends and Challenges. In IcETRAN conference, Srebrno Jezero, Serbia. External Links: Link Cited by: §I, §I-B. [21] N. Shao, E. Loweimi, and X. Li (2022) RCT: Random consistency training for semi-supervised sound event detection. In Interspeech, p. 1541–1545. Cited by: §I. [22] N. Shao, X. Li, and X. Li (2022) ATST self-supervised plus rct semi-supervised sound event detection: submission to dcase 2022 challenge task 4. tech. rep.. Cited by: §I. [23] N. Shao, X. Li, and X. Li (2024) Fine-tune the pretrained atst model for sound event detection. In ICASSP 2024 - 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Vol. , p. 911–915. External Links: Document Cited by: §I, §I-A, Figure 3, §I-A, §I-C, TABLE I, TABLE I. [24] T-W. Su, J-Y. Liu, and Yi-H. Yang (2017) Weakly-supervised audio event detection using event-specific gaussian filters and fully convolutional networks. In ICASSP, p. 791–795. Cited by: §I. [25] A. Tarvainen and H. Valpola (2017) Mean teachers are better role models: weight-averaged consistency targets improve semi-supervised deep learning results. In Advances in Neural Information Processing Systems (NIPS), Vol. 30. Cited by: §I-A. [26] N. Turpault, R. Serizel, A. Shah, and J. Salamon (2019) SOUND event detection in domestic environments with weakly labeled data and soundscape synthesis. In Acoustic Scenes and Events 2019 Workshop (DCASE2019), p. 253. Cited by: §I, §I, §I-A. [27] V. Verma, A. Lamb, J. Kannala, Y. Bengio, and D. Lopez-Paz (2019) Interpolation consistency training for semi-supervised learning. In International Joint Conference on Artificial Intelligence (IJCAI), p. 3635–3641. Cited by: §I-A, §I-B. [28] H. Zhang, M. Cisse, Y. N. Dauphin, and D. Lopez-Paz (2018) Mixup: beyond empirical risk minimization. In International Conference on Learning Representations (ICLR), Cited by: §I-B. [29] X. Zheng, Y. Song, I. McLoughlin, L. Liu, and L.-R. Dai (2021) An improved mean teacher based method for large scale weakly labeled semi-supervised sound event detection. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), p. 356–360. Cited by: §I.