Paper deep dive
Finetuning Strategies for Querying Sounds by Vocal Imitation
Aditya Bhattacharjee, Christos Plachouras, Sungkyun Chang, Emmanouil Benetos
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 93%
Last extracted: 8/22/2026, 1:43:51 AM
Summary
This technical report details the winning submission to the AES AIMLA 2025 Challenge for Query by Vocal Imitation (QbVI). The authors propose two fine-tuning strategies: Submission #1 uses a frozen Consistent Ensemble Distillation (CED) encoder with supervised contrastive learning, while Submission #2 employs a shared MobileNetV3 encoder with a hybrid contrastive-triplet loss using semi-hard negatives from the VocalSketch dataset. Both methods utilize online data augmentation to improve robustness against vocal variation.
Entities (11)
Relation Signals (8)
Submission #2 → woncompetition → AES AIMLA 2025 Challenge
confidence 99% · Submission #2 is the winning submission in the AES AIMLA 2025 Challenge
Aditya Bhattacharjee → affiliatedwith → Queen Mary University of London
confidence 95% · Affiliation: School of Electronic Engineering and Computer Science, Queen Mary University of London, UK
Submission #2 → usesencoder → MobileNetv3
confidence 95% · Our second submission builds upon the MobileNetV3 architecture
AudioSet → usedforpretraining → MobileNetv3
confidence 93% · MobileNetV3 encoder pretrained on AudioSet [3]
Submission #1 → usesencoder → Consistent Ensemble Distillation
confidence 92% · Our first submission involves contrastive fine-tuning of a frozen Consistent Ensemble Distillation (CED) encoder
Submission #2 → usesdataset → VocalSketch Dataset
confidence 90% · The second submission incorporates both datasets [VimSketch and VocalSketch] to enable a hybrid training regime
Submission #1 → usesmethod → Contrastive Learning
confidence 90% · In Submission #1... A supervised contrastive loss is applied
Submission #2 → usesmethod → Triplet Loss
confidence 90% · In Submission #2... fine-tuning a shared MobileNetV3 encoder using a hybrid loss combining contrastive and triplet objectives.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:This technical report describes our winning submission to the AES AIMLA 2025 Challenge on querying sound effects by vocal imitation. We investigate two complementary fine-tuning strategies: contrastive learning with a frozen, pretrained CED encoder, and joint contrastive-triplet learning with semi-hard negatives using a MobileNetV3 encoder. This report has been updated for posterity to include details released after the challenge.
Tags
Links
- Source: https://arxiv.org/abs/2608.19174v1
- Canonical: https://arxiv.org/abs/2608.19174v1
Trouble viewing inline? Open PDF directly →
Full Text
13,911 characters extracted from source content.
Expand or collapse full text
Finetuning Strategies for Querying Sounds by Vocal Imitation Aditya Bhattacharjee, Christos Plachouras, Sungkyun Chang, Emmanouil Benetos Affiliation: School of Electronic Engineering and Computer Science, Queen Mary University of London, UK Abstract This technical report describes our winning submission to the AES AIMLA 2025 Challenge on querying sound effects by vocal imitation. We investigate two complementary fine-tuning strategies: contrastive learning with a frozen, pretrained CED encoder, and joint contrastive-triplet learning with semi-hard negatives using a MobileNetV3 encoder. This report has been updated for posterity to include details released after the challenge. I Introduction Query by Vocal Imitation (QbVI) involves retrieving audio samples from a reference collection based on a vocal imitation provided by the user. This problem is particularly challenging due to the variability in human vocalizations and the acoustic diversity of sound classes. Effective systems must learn representations that bridge the modality gap between vocal and non-vocal audio, while also being robust to the diverse ways humans imitate sounds. Early approaches to QbVI focused on latent bottleneck features from autoencoders [6] or semi-Siamese convolutional architectures trained with contrastive loss [7]. These methods typically relied on hand-crafted or moderately learned features, combined with traditional classifiers or similarity measures such as dynamic time warping or cosine distance. More recently, Greif et al. [4] demonstrated the effectiveness of contrastive learning with neural audio embeddings pre-trained on large-scale datasets like AudioSet. Their system employed a dual-tower MobileNetV3 architecture, fine-tuned using SimCLR-style contrastive objectives and an extensive augmentation pipeline. This approach set a strong baseline for neural QbVI systems. In this work, we focus specifically on neural network-based representations for QbVI and explore two complementary submission strategies for the AES-AIMLA 2025 Challenge. Our first submission involves contrastive fine-tuning of a frozen Consistent Ensemble Distillation (CED) encoder [2], which was originally trained for audio tagging via knowledge distillation. Our second submission builds upon the MobileNetV3 architecture and the setup of Greif et al., but integrates a triplet-based regularization objective. The motivation is to improve generalization by leveraging hard or semi-hard negatives sampled from a curated unpaired set of vocal imitations.11 1 Submission #2 is deployed on our website: https://thatsoundslike.me. I Datasets We use two primary datasets for training: • VimSketch Dataset [5]: This dataset provides supervised (reference, imitation) pairs with known correspondence. These pairs are used directly in our contrastive learning objective. • VocalSketch Dataset [1]: This dataset contains vocal imitations that were excluded from the curated VimSketch set. Although they do not have matched reference samples, they are labeled by sound class. We use these as structured negatives in a triplet learning framework, sampling same-class negatives when available and falling back to random sampling otherwise. In our first submission, only the VimSketch dataset is used. The second submission incorporates both datasets to enable a hybrid training regime that combines contrastive and triplet learning. We use an online augmentation methodology on both query and reference data. This strategy is used in both our submissions to improve robustness to variation in vocal imitations and reference recordings. At each training step, a random subset of transformations is applied to each audio sample. The augmentations include time-domain operations such as time shifting, gain adjustment, pitch shifting, and time stretching. In addition, we incorporate light additive Gaussian noise and frame-level corruptions to simulate temporal dropouts and distortions common in human vocalizations. Specifically, each sample is passed through up to one transformation randomly selected from a pool that includes: • Shift: Random temporal offset of the waveform within ± 30% of its length. • Gain: Random amplification or attenuation between −10-10 and +10+10 dB. • Pitch Shift: Semitone shift within a range of ±3± 3 semitones. • Time Stretch: Temporal scaling by a factor between 0.80.8 and 1.21.2. • Additive Gaussian Noise: Injects low-amplitude noise with amplitude sampled between 0.0010.001 and 0.0150.015. • Frame-Level Corruption: Applies stochastic frame-wise duplication, silencing, or removal to simulate human inconsistency. • Frequency and Time Masking: Randomly masks contiguous frequency bins or time frames in the spectrogram to increase robustness to partial input corruption. This online augmentation is applied independently to both the query and reference branches of the model during training. It plays a crucial role in improving generalization across different vocalization styles, recording conditions, and intra-class variation. For our validation dataset, we use the official qvim-dev set provided as part of the QVIM 2025 Challenge. Evaluation is performed using the Mean Reciprocal Rank (MRR) and Normalized Discounted Cumulative Gain (NDCG) metrics. I Submission #1 I-A Architecture and Training Setup Our first submission is based on the Consistent Ensemble Distillation (CED) framework, originally introduced by Dinkel et al. [2]. CED is a ViT-based audio encoder trained for AudioSet audio tagging using consistent ensemble distillation. It performed strongly on the HEAR 2021 benchmark, particularly for vocal imitation classification tasks, a motivation for its inclusion here. In this submission, we freeze the pretrained CED-base encoder (768-dimensional output) and train a lightweight MLP projection head to map embeddings to a 256-dimensional space suitable for retrieval. The encoder is shared across both the query and reference branches. Audio is resampled to 16 kHz and processed using the pretrained CED feature extractor. We extract the final encoder hidden representation, average-pool it over the sequence dimension to obtain a 768-dimensional embedding, and keep the CED encoder frozen during training. The training objective is a supervised contrastive loss applied to known (reference, imitation) pairs from the VimSketch dataset. In-batch negatives are used to structure the embedding space. We apply online data augmentation symmetrically to both the query and reference audio inputs. Each training example receives a randomly selected transformation from a curated pool of time-domain effects; namely, the ones discussed in Section I. I-B Implementation The system uses the AdamW optimizer with a cosine learning rate schedule (starting at 1×10−31× 10^-3), weight decay of 1×10−51× 10^-5, and a batch size of 128. The model is trained for 50 epochs, with evaluation performed after every epoch. The temperature τ for the contrastive loss is fixed at 0.07. The output projection is a linear layer mapping the encoder’s 768-dimensional output to a 256-dimensional embedding, which is then used to compute cosine similarities for retrieval. IV Submission #2 IV-A Architecture and Training Setup Our model architecture builds on the QbVI framework proposed by Greif et al. [4], using a MobileNetV3 encoder pretrained on AudioSet [3]. In contrast to the dual-tower encoder architecture of [4], we use a single shared encoder for both reference and imitation audio, promoting tighter alignment in the learned embedding space and reducing parameter overhead. The encoder outputs are L2-normalized, enabling cosine similarity as a metric for both the loss computation and retrieval. We adopt a hybrid learning strategy where supervised contrastive learning is regularized by a triplet loss. Contrastive loss is applied over positive pairs sampled from the VimSketch dataset, while the triplet loss uses negative examples drawn from the excluded subset of the VocalSketch dataset. These excluded examples consist of practice recordings and human-rejected imitations and do not participate in standard contrastive supervision. For triplet construction, negatives are sampled based on shared AudioSet ontology labels where available, falling back to random negatives otherwise. A semi-hard mining strategy is used, selecting negatives that are closer to the anchor than easy negatives but still harder than positives. To adaptively balance the contribution of both losses during training, we implement a dynamic weighting strategy where the weight assigned to the triplet loss is scaled based on the number of active semi-hard triplets in the batch. If a few valid triplets are found, the triplet loss is down-weighted to avoid destabilizing training. IV-B Implementation The input features closely follow AudioSet’s preprocessing pipeline, where all audio is resampled to 32 kHz and converted to 128-band log-Mel spectrograms using a 10-second duration, a window size of 800 samples, a hop size of 320, and an FFT size of 1024. Online data augmentation, as described earlier, is applied independently to both the query and reference inputs during training to increase robustness to vocal variation and noise. 960-dimensional embeddings are output from the penultimate layer of the encoder; the classifier head of the encoder is discarded. We train using the AdamW optimizer and a cosine learning rate schedule with one warmup epoch. The maximum learning rate is set to 2×10−42× 10^-4 and the minimum to 5×10−55× 10^-5. Training runs for 30 epochs with a batch size of 64. We fix the contrastive temperature to τ=0.07τ=0.07 and the triplet margin to 0.6. V Results The evaluation follows the official QVIM 2025 validation protocol, using the ‘qvim-dev‘ set provided by the organizers. The validation set includes both query-reference pairings and class metadata, enabling both pair-wise and class-wise evaluation. The primary evaluation metric is the Mean Reciprocal Rank (MRR) computed over known query-reference pairs. Secondary metrics include Normalized Discounted Cumulative Gain (NDCG) and class-wise MRR, which assess ranking quality and class-level retrieval consistency, respectively. Table I summarizes the validation performance for both of our submissions, in comparison to the baselines. For posterity, we report that Submission #2 is the winning submission in the AES AIMLA 2025 Challenge on querying sound effects by vocal imitation. Further details about the objective and subjective evaluation on test datasets can be found in the challenge website 22 2 https://qvim-aes.github.io/. TABLE I: Performance Metrics Model Name MRRex NDCGcat random 0.0444 0.337 2DFT 0.1262 0.4793 Greif et al. 0.2726 0.6463 Submission #1 0.2876 0.6600 Submission #2 0.2932 0.6468 VI Conclusion The goal of the Query by Vocal Imitation (QbVI) task is to retrieve audio samples from a reference database based on a vocal imitation provided by the user. As part of our participation in the AES-AIMLA 2025 Challenge, we explored two complementary approaches aimed at improving neural-network-based retrieval systems for QbVI. The submission differ in encoder architecture, training strategy, and supervision design. In Submission #1, we leverage a frozen CED-base encoder pretrained via knowledge distillation on AudioSet. A supervised contrastive loss is applied over known reference-imitation pairs, using in-batch negatives to structure the embedding space. This lightweight approach achieves strong retrieval performance with minimal fine-tuning. In Submission #2, we extend the training pipeline by fine-tuning a shared MobileNetV3 encoder using a hybrid loss combining contrastive and triplet objectives. Semi-hard negative mining is performed using excluded vocal imitations, which are sampled based on class labels. A dynamic weighting strategy balances the two losses based on the availability of active triplets. Together, these submissions highlight the value of both transfer learning and curriculum design in QbVI systems. Whether through careful pretraining or structured negative sampling, both strategies yield significant gains in retrieval performance over baseline systems. References [1] M. Cartwright and B. Pardo (2015) Vocalsketch: vocally imitating audio concepts. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems, Seoul, Republic of Korea, p. 43–46. Cited by: 2nd item. [2] H. Dinkel, Y. Wang, Z. Yan, J. Zhang, and Y. Wang (2024) CED: consistent ensemble distillation for audio tagging. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Seoul, Republic of Korea, p. 291–295. Cited by: §I, §I-A. [3] J. F. Gemmeke, D. P. Ellis, D. Freedman, A. Jansen, W. Lawrence, R. C. Moore, M. Plakal, and M. Ritter (2017) Audio set: an ontology and human-labeled dataset for audio events. In IEEE international conference on acoustics, speech and signal processing (ICASSP), New Orleans, USA, p. 776–780. Cited by: §IV-A. [4] J. Greif, F. Schmid, P. Primus, and G. Widmer (2024) Improving query-by-vocal imitation with contrastive learning and audio pretraining. In Detection and Classification of Acoustic Scenes and Events (DCASE), Tokyo, Japan. Cited by: §I, §IV-A. [5] F. Pishdadian, P. Seetharaman, B. Kim, and B. Pardo (2019) Classifying non-speech vocals: deep vs signal processing representations. Cited by: 1st item. [6] Y. Zhang and Z. Duan (2016) Supervised and unsupervised sound retrieval by vocal imitation. Journal of the Audio Engineering Society 64 (7/8), p. 533–543. Cited by: §I. [7] Y. Zhang and Z. Duan (2017) IMINET: Convolutional semi-Siamese networks for sound search by vocal imitation. In IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (WASPAA), New Paltz, USA, p. 304–308. Cited by: §I.