Paper deep dive
Pmeta-TLA: Backdoor Attacks for Speech Classification Models via Meta-Learning with Timbre Leakage Attack
Yueming Huang, Wenhan Yao, Fen Xiao, Xiarun Chen, Weiping Wen
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 96%
Last extracted: 7/5/2026, 11:06:45 AM
Summary
The paper introduces Pmeta-TLA, a novel backdoor attack mechanism for speech classification models, specifically targeting Keyword Spotting (KWS) tasks. It utilizes a Timbre Leakage Attack (TLA) as a trigger, which embeds timbre information at the frame level to create highly stealthy and natural-sounding poisoned samples. To enhance attack flexibility and complexity, the authors employ meta-learning combined with Projected Conflicting Gradients (PCGrad), enabling an 'all-to-all' multi-backdoor injection strategy. This approach allows a single model to host multiple backdoors that can be rapidly adapted to new triggers, significantly improving attack robustness and stealthiness compared to traditional perturbation-based or single-target component-based attacks.
Entities (7)
Relation Signals (5)
Pmeta-TLA → appliedto → Keyword Spotting
confidence 100% · We performed tests on data-poisoning backdoor attacks in keyword spotting tasks
Pmeta-TLA → uses → Timbre Leakage Attack
confidence 100% · We introduce Pmeta-TLA... and introduces TLA as a multi-target attack tool within it.
Pmeta-TLA → utilizes → Meta-Learning
confidence 100% · This method proposes a multi-backdoor injection training strategy using meta-learning
Pmeta-TLA → utilizes → Projected Conflicting Gradients
confidence 100% · using meta-learning and Projected Conflicting Gradients (PCGrad)
Timbre Leakage Attack → targets → Speech Classification
confidence 90% · The suggested trigger disseminates timbre information at the frame level within the deep self-supervised features
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Recently, speech classification methods have gained widespread adoption in intelligent gadgets. Current study indicates that backdoor attacks provide a substantial security concern to these models, underscoring the pressing necessity to investigate additional potential attack techniques to expose and prevent such risks. This work discusses the vulnerability of current speech triggers to detection by deep neural network defenders and introduces the Timbre Leakage Attack (TLA). The suggested trigger disseminates timbre information at the frame level within the deep self-supervised features, producing poisoned samples that appear natural to human perception. Furthermore, we introduce Pmeta-TLA, an innovative training mechanism for embedding numerous backdoors one time. This method proposes a multi-backdoor injection training strategy using meta-learning and Projected Conflicting Gradients (PCGrad) and introduces TLA as a multi-target attack tool within it. We performed tests on data-poisoning backdoor attacks in keyword spotting tasks utilizing some deep neural network models. Experimental results indicate that the proposed strategy attains superior Attack efficacy, enhanced stealthiness, robustness, and a reduced attack cost relative to baseline methods.
Tags
Links
- Source: https://arxiv.org/abs/2607.01702v1
- Canonical: https://arxiv.org/abs/2607.01702v1
Trouble viewing inline? Open PDF directly →
Full Text
72,032 characters extracted from source content.
Expand or collapse full text
Pmeta-TLA: Backdoor Attacks for Speech Classification Models via Meta-Learning with Timbre Leakage Attack Yueming Huang a , Wenhan Yao a , Fen Xiao a , Xiarun Chen b , Weiping Wen b,∗ a Xiangtan University, Yuhu District Xiangda Road, Xiangtan, 411100, Hunan, China b Peking University, No.5, Summer Palace Road, Haidian District, Beijing, China, Beijing, 100871, Beijing, China Abstract Recently, speech classification methods have gained widespread adoption in intelligent gadgets. Current study indicates that back- door attacks provide a substantial security concern to these models, underscoring the pressing necessity to investigate additional potential attack techniques to expose and prevent such risks. This work discusses the vulnerability of current speech triggers to detection by deep neural network defenders and introduces the Timbre Leakage Attack (TLA). The suggested trigger disseminates timbre information at the frame level within the deep self-supervised features, producing poisoned samples that appear natural to human perception. Furthermore, we introduce Pmeta-TLA, an innovative training mechanism for embedding numerous backdoors one time. This method proposes a multi-backdoor injection training strategy using meta-learning and Projected Conflicting Gradi- ents (PCGrad) and introduces TLA as a multi-target attack tool within it. We performed tests on data-poisoning backdoor attacks in keyword spotting tasks utilizing some deep neural network models. Experimental results indicate that the proposed strategy attains superior Attack efficacy, enhanced stealthiness, robustness, and a reduced attack cost relative to baseline methods. Keywords: Backdoor Attacks, Speech Classification, Meta-Learning, PCGrad, Triggers 1. Introduction The speech classification task entails developing classifiers capable of differentiating speakers or commands based on their utterances, which is essential for applications including intelli- gent security systems, personal device recognition, and human- computer interaction. However, the growing utilization of deep neural networks (DNNs) has afforded malicious actors the po- tential to perpetrate backdoor attacks against the classifiers. This threat creates an imperceptible "hidden backdoor" within the model [1, 2]. Attackers introduce meticulously designed poisoned samples into the training data, altering their labels to values predetermined by the attacker. Poisoned samples usually contain one or more triggers, which are typically data with the same form as the poisoned samples and are generated by attacker-designed trigger functions. Some examples of au- dio triggers include ultrasonic signals [3], one-hot frequency sounds [4], voice conversion triggers [5], or their combinations [6]. A common form of backdoor attack entails training the model on a poisoned dataset, known as a data-poisoning at- tack, which leads to the model becoming a backdoor variant. A backdoored model generally generates accurate and anticipated labels when presented with standard input samples. However, the backdoor threat may be activated without the user’s knowl- edge, resulting in the model producing an incorrect label pre- determined by the attacker when it processes an input sample containing a "trigger." Typically, attackers aim to satisfy the fol- lowing three characteristics. When numerous poisoned samples ∗ Corresponding author are undetectable by humans or models, the backdoor attack is characterized as stealthy. A backdoor attack is effective when a backdoored model consistently misclassifies nearly all poi- soned samples of the testing set. A backdoor attack is consid- ered robust when the backdoor model continues to be effective despite the application of backdoor defense techniques. Investigating backdoor attacks effectively exposes model vulnerabilities, supporting the enhancement of security mecha- nisms. While most backdoor attacks for speech classifiers are effective, we conclude that they are not sufficiently stealthy or robust, stemming from perceptible and detectable triggers. The present speech backdoor attack methods can be catego- rized into (1) perturbation triggers (p-tris) and (2) component triggers (c-tris). The existing trigger methods listed in Ta- ble 1 demonstrate this. Examples of perturbation triggers in- clude short noisy clips at any temporal position [7], music and other noisy sounds [8], one-hot spectrogram voices [4], and distorted speech [9]. However, some automatic speech quality assessment models, such as NISQA [10] and MosNet [11], can distinguish the quality differences between poisoned and clean samples, thereby revealing the low stealthiness of such perturbation-based triggers. The examples of component triggers include pitch-boosting sounds [12], timbre-converted speech [13, 5], rhythm-transferred utterances [14], etc. These triggers modify one or more components of clean utterances to form the poisoned utterances instead of distorting signals, and therefore they are more effective and stealthy. However, if the defender deliberately uses models such as Speaker Verification models (SVs) [15] or RMVPE [16] to detect whether the pitch and timbre components of speech are normal, these poisoned arXiv:2607.01702v1 [cs.CR] 2 Jul 2026 Table 1: Backdoor attacks in speech classification models. The p-tris signifies that the trigger is categorized as a perturbation trigger, whereas c-tris suggests that the trigger is classified as a component trigger. The table presents a clean mel-spectrogram sample alongside poisoned mel-spectrogram samples derived from multiple methods. AttackCategory TriggerASRExample Without attackWithout trigger 00.511.5 Time 0 512 1024 2048 4096 Hz Log-Mel Spectrogram -80 dB -70 dB -60 dB -50 dB -40 dB -30 dB -20 dB -10 dB +0 dB JingleBack [9]p-trisDistorted speech 00.511.5 Time 0 512 1024 2048 4096 Hz Log-Mel Spectrogram -80 dB -70 dB -60 dB -50 dB -40 dB -30 dB -20 dB -10 dB +0 dB PIBA [7]p-tris Short noisy clip at any temporal po- sition 00.511.5 Time 0 512 1024 2048 4096 Hz Log-Mel Spectrogram -80 dB -70 dB -60 dB -50 dB -40 dB -30 dB -20 dB -10 dB +0 dB DABA [8]p-trisMusic and noisy sounds 00.511.5 Time 0 512 1024 2048 4096 Hz Log-Mel Spectrogram -80 dB -70 dB -60 dB -50 dB -40 dB -30 dB -20 dB -10 dB +0 dB BAASV [4]p-trisOne-hot spectrogram voice 00.511.5 Time 0 512 1024 2048 4096 Hz Log-Mel Spectrogram -80 dB -70 dB -60 dB -50 dB -40 dB -30 dB -20 dB -10 dB +0 dB PBSM [12]c-trisPitch-boosting and sound-masking 00.511.5 Time 0 512 1024 2048 4096 Hz Log-Mel Spectrogram -80 dB -70 dB -60 dB -50 dB -40 dB -30 dB -20 dB -10 dB +0 dB VSVC [13]c-trisTimbre-converted speech 00.150.30.450.60.750.91.11.21.4 Time 0 512 1024 2048 4096 Hz Log-Mel Spectrogram -80 dB -70 dB -60 dB -50 dB -40 dB -30 dB -20 dB -10 dB +0 dB RSRT [14]c-trisSqueezed and stretched speech 0.0000.0600.1200.1800.2400.3000.3600.4200.4800.540 Time 0 512 1024 2048 4096 Hz Log-Mel Spectrogram (3-Column Aggregated) -80 dB -70 dB -60 dB -50 dB -40 dB -30 dB -20 dB -10 dB samples may still be detected before the model’s weight train- ing begins, thereby causing the attack to fail. So, is it possible to generate highly realistic poisoned sam- ples that enable low-cost backdoor attacks and evade defend- ers’ AI-based countermeasures? We propose a speech backdoor attack approach termed PC- Grad Meta-Learning with Timbre Leakage Attack (PMeta- TLA), which is predicated on a data-poisoning attack that uses Timbre Leakage Attack (TLA) as the backdoor attack trig- ger function. Initially, we introduce the timbre leakage trigger, which, in contrast to VSVC that alters the timbre of the entire utterance, executes voice conversion on a selected section of the utterance at the frame level. The trigger feature enables the correlation of the leaking timbre with a designated attack la- bel. To augment attack flexibility during inference, we propose employing meta-learning [17] to amplify the number of back- doors embedded in the victim model, facilitating an "all-to-all" backdoor attack approach. We perceive the execution of the backdoor attack as a multi-task learning process that completes both the clean task and the backdoor tasks. Within the inner loop of meta-learning, we utilize a multi-objective optimization algorithm known as Projecting Conflicting Gradients (PCGrad) [18] to equilibrate the weights of various tasks, with the objec- tive of concurrently embedding multiple backdoors into speech classifiers. Through meta-learning, the attacker compels the classifier to develop the meta-ability of "how to implant back- doors." Thus, when faced with robust defenses, the opponent can readily devise a new timbre and utilize it as an attack trig- ger to introduce a novel backdoor, thereby evading the defense. This significantly strengthens the attack’s resilience. This work’s primary contributions are as follows: • We presented a timbre leakage trigger that inherently al- ters the timbre of a segment clip. We examine the influence of the leaking timbre position on the efficacy of the attack via ablation studies. • We suggest employing meta-learning to augment the com- plexity of trigger embedding for the implantation of multi- ple backdoors. Furthermore, we present a PCGrad to equi- librate the balance between backdoor implantation tasks and clean sample prediction learning, hence enhancing the classifier’s performance across all tasks. • We perform backdoor attack studies on the typical speech classification tasks: Keyword Spotting (KWS). Experi- mental findings indicate that a singular leaking timbre trig- ger can attain attack performance equivalent to baseline methodologies. Following training with meta-learning and the PCGrad algorithm, the meta-backdoor speech classi- fication model can swiftly adjust to new triggers, hence augmenting the robustness of the attack. We employ vari- ous measures to illustrate the significant stealthiness of the trigger. 2 The subsequent sections of this work are structured as fol- lows. Section 2 delineates the background of the speech clas- sification task and backdoor attacks. Section 3 delineates the threat model and the execution of backdoor attacks. Section 4 presents the proposed approach, Pmeta-TLA. Section 5 de- lineates the experimental results and examines the evaluation measures and robustness. In Section 6, we conclude with a summary of the innovations and contributions of the proposed backdoor attack. 2. Background 2.1. Speech Classification Models This research focuses on speech classifiers for keyword spot- ting (KWS). The KWS classifiers predict keyword labels based on the speech waveforms or spectrograms. In recent years, deep neural networks (DNNs) have achieved optimal efficacy in the domains [19, 20, 21, 22, 23, 24], hence raising substantial se- curity issues. The classifier C θ is consistently tuned using the cross-entropy loss as outlined below: L (x,y)∈D = arg max θ p(y| C θ (x))(1) The (x, y) represents the model inputs and corresponding true labels. Training on the dataset D enables the classifier to dis- cern the correlations between a particular input feature and the potential attack labels, hence posing a risk of data poisoning backdoor attacks. 2.2. Backdoor Attacks 2.2.1. Present Backdoor Attacks. In preliminary studies, Battista et al. illustrated that the incor- poration of harmful input can cause machine learning models to develop biased probabilistic frameworks [25]. Subsequently, Gu et al. [26] superimposed many white pixels onto handwrit- ten digit images and altered their labels to a different class (des- ignated as the target label), so generating a poisoned dataset. Deep neural network models trained on this dataset were dis- covered to possess backdoors. According to a thorough and detailed survey [27], backdoor attacks can be classified into the following concurrent types: Poisoned-label and Clean-label Attack. Most poisoned-label attacks depend on the introduction of poisoned samples with altered labels to taint the dataset. To eliminate high-risk tag modification operations for the clean-label attack, Turner et al. [28] suggested that incorporating samples around the decision boundary of the DNN model into the dataset can facilitate a backdoor attack without altering the labels. Visible and Invisible Attack. When the poisoned samples cre- ated by the attacker, primarily comprising images, utterances, and texts, are distinctly perceptible to humans, such as eyeglass patterns superimposed on images [2], transient noise in speech [7], or text exhibiting a unique stylistic pattern [29], these back- door attacks are classified as the visible attacks. Otherwise, it is deemed an imperceptible Attack. Lin et al. [30] advocated the creation of composite triggers through the amalgamation of existing benign traits. This strategy is regarded as a form of invisible attack, as these visuals appear normal to human ob- servers. All-to-One and All-to-All Attack. Unlike all-to-one attacks [26, 31], when all poisoned samples have a uniform target la- bel, all-to-all attacks may involve poisoned samples with vary- ing target labels [32, 33]. They are thus designated as single- target backdoor attacks, which are characterized by the incor- poration of precisely one backdoor into the model, and multi- target backdoor attacks, wherein numerous distinct backdoors are concurrently embedded. Sample-Agnostic and Sample-Specific Attack.In the sample-agnostic Attack, every input data point possesses an identical trigger. This trait was extensively utilized in the for- mulation of backdoor defenses, including trigger synthesis- based defenses [34] and saliency-based defenses [32]. In the sample-specific attack, the trigger patterns are tailored to indi- vidual samples rather than being universally applicable. Attack- ers strategically include triggers according to specific property characteristics of the samples, such as vocal gender or object edges in photos [35]. Due to their capacity to circumvent the majority of current backdoor defenses, these attacks pose a con- siderable security risk and necessitate more scrutiny. Physical and Digital Backdoor Attacks. In contrast to prior digital attacks executed solely in the digital realm, physical at- tacks incorporate the tangible surroundings in the creation of poisoned samples. Chen et al. [2] were the pioneers in ex- amining this attack modality, employing a pair of glasses as a physical catalyst to mislead a hacked facial recognition sys- tem integrated into a camera. Wenger et al. [36] conducted an in-depth examination of physical-world attacks on facial recog- nition systems. Likewise, [26] employed a post-it note as a trigger to compromise traffic sign recognition in cameras. De- spite the great efficacy and challenging defense against physi- cal backdoor attacks, their deployment incurs significant costs; nevertheless, they warrant careful scrutiny. A high-risk backdoored model is generally characterized by poisoned labels (to guarantee attack efficacy), invisibility (not readily detectable), sample specificity, and an all-to-all con- figuration (complicating defense due to the presence of many backdoor kinds inside a single model). This research seeks to provide a backdoor model to expose the weaknesses of speech classification systems. 2.2.2. Speech Backdoor Attacks. Research on speech backdoor attacks seeks to uncover the se- curity vulnerabilities in speech classification methods. Speech backdoor attacks, informed by the acoustic characteristics of the trigger, concentrate on generating speech triggers that can be classified as perturbation triggers and component triggers. Perturbation Triggers. The perturbation trigger techniques predominantly emulate those employed in backdoor attacks of image classification. Attackers initially transform speech from the temporal domain into spectrograms for neural network in- puts, then introduce disturbances to the clean inputs. Zhai et al. [4] presented a clustering-based backdoor attack aimed at 3 speech recognition (SR) systems, utilizing low-volume one- hot-spectrum noise as the trigger pattern. DriNet [37] utilized a generative adversarial network to dynamically generate trig- gers, thereby undermining keyword spotting (KWS) models. Koffas et al. [3] utilized ultrasound as the triggering medium, facilitating both inaudible and physical attacks. Shi et al. [7] introduced a position-independent short-term noise trigger to improve the efficacy and adaptability of the attack. The DABA [8] and Cuckoo Attack [38] employed natural sound templates as triggers and introduced distinct selection techniques to ascer- tain the most efficacious noise samples. JingleBack [9] advo- cated employing six techniques, such as pitch shifting, spectral distortion, and others, to alter the style of clean utterances in a composite fashion. W. Zong et al. [39] introduced the Trojan- Model to incorporate auditory perturbations into the features, resulting in the speech-to-text model generating harmful text. Component Triggers. The perturbation trigger methods im- mediately alter the time-frequency characteristics of the signal, rendering them readily discernible to the human ear. Similar to the creation of invisible triggers to mitigate the conspicu- ousness of image classification attacks [40, 41, 42], the latest researchers have suggested "imperceptible" triggers for speech. These triggers alter the elements of speech—such as pitch, tim- bre, prosody, and content—without substantially compromis- ing the overall quality of the speech. Cai suggested employ- ing PBSM [12] to elevate the pitch of speech by several semi- tones, therefore generating tainted utterances and associating the altered pitch with a designated label. Nevertheless, owing to the restricted scope of pitch shifting, this method can estab- lish only a singular backdoor. Cai introduced VSVC, [13], to tackle the difficulty of implementing several unique backdoors by associating timbre with target labels using voice conversion on clean speech prior to training, hence utilizing specific tim- bres as triggers. Yao indicated that models can still discern al- terations in pitch and timbre and suggested RSRT [14] utilizing speech prosody as an alternative trigger. Yao proposed adjust- ing the speaking rate of an utterance while maintaining its tim- bre and pitch. Investigations into speech backdoor attacks have uncovered weaknesses in speech models. Therefore, this work continues to investigate component triggers to expose the vul- nerabilities of speech models. The previously reported audio-attribute trigger-based attacks demonstrate significant efficacy and subtlety. This research seeks to investigate effective multi-backdoor attack strategies inside the component trigger paradigm. 3. Threat Model This section presents the threat model. We delineate the poisoning-label and all-to-all backdoor attack issues and estab- lish assumptions regarding the attacker’s capability. 3.1. Backdoor Attack Pipeline Figure 1 depicts the fundamental procedure of a poisoned- label and all-to-all speech backdoor attack. The pipeline com- prises the attack stage, training stage, and inference stage. 3.1.1. Attack Stage In the attack stage, the attacker aims to stealthily inject poi- soned samples into the training set of the speech classifier. In a poisoning-label and all-to-all speech backdoor attack, the at- tacker can directly access to the clean dataset D c = (x i , y i ) N i=1 . Here, the class labels y i belong to LY = [1, 2,..., Y K ], and the attacker selects one or more attacked class set Y A ⊂ LY,|Y A | = TK to attack. The number of selected attacked labels is TK. Similarly, the attacker selects a target label set Y T ⊂ LY,|Y T | = TK for each label under attack. The attacked labels and the target labels are in a one-to-one correspondence. Subsequently, the attacker splits D c into two clean subsets: D s1 and D s2 , where the sample labels in D s2 belong to Y A . At this stage, we the at- tacker uses a pre-constructed trigger function FT (x,τ) to gen- erate poisoned samples from D s2 , forming the poisoned subset D s p = (FT (x j ),τ, y t ) M j=1 , x j ∈ D s2 . Each poisoned classifier in- put contains a trigger τ. By randomly mixing the clean training samples with the poisoned subset, a poisoned training dataset can be formed as D p = D s p ∪ D s1 . Typically, the number of poisoned samples for each attacked class is the same, denoted as M/TK, so the total number of samples in the D s2 is M, we refer to p r = M/N as the poisoning rate. Typically, the poisoning rate affects both the stealthiness and effectiveness of a backdoor attack, with opposing effects. 3.1.2. Training Stage During training, the clean validation dataset and test dataset D t can be sampled from the D s1 set. In the training dataset, the speech classifier C θ is optimized by the dataset D p . After back- door training, the clean classifier is transformed to backdoor classifier C ∗ θ . 3.1.3. Inference Stage During the inference stage, we select a subset of samples from D t that includes all attacked classes and modify their original labels to the corresponding target labels.The se- lected subset is referred to as the poisoned test dataset D pt = (FT (x k ),τ), y t ) K i=1 , x k ∈ D t , y t ∈ Y T . The Attack Success Ac- curacy (ASR) is equal to the calculated accuracy on this dataset with the backdoor model for attack effectiveness. 3.2. Attacker Capabilities In the contemporary landscape of swift open-source AI ad- vancement, a prevalent method for implementing services such as identity verification or speech classification involves acquir- ing pre-trained models or model architecture code from public repositories like Hugging Face or GitHub, followed by fine- tuning them on a designated dataset or training them from in- ception to achieve the desired classification objective. Malefac- tors can manipulate this process by introducing contaminated samples into the datasets utilized for training or fine-tuning, thereby executing backdoor attacks. [5, 43, 44, 45, 38]. We presume that the attack possesses read and write access to the dataset and can manipulate the specifics of the model train- ing procedure. In these conditions, the attack possesses nearly 4 (a-1) Trigger Function Clean Input Trigger Function Poisoned Input (a-2) Poisoned Dataset Generation True Label Target Label Label Shift 1 ( , ) N ci i Dxy = = Original Dataset 1121 ( ,),( ,) NMM sj jjsj j DxyDxy − == == 2 ( ( ,), ),1 sptkt DF x Try iN== 1pssp D= + Poisoned Training Dataset (a) Attack Stage (b) Training Stage (b-1) Clean Model Initialization (b-2) Backdoor Training Bad (c) Inference Stage (c-1) Test Poisoning Bad Label Clean Test Input Trigger Function Poisoned Test Input (c-2) Metrics Calculation ASR Bad Poisoned Inputs Target Label Clean Inputs Bad True Label (c-3) Backdoor Attack p D Poisoned Test Input Victim Model Public Repository API Backend Figure 1: The backdoor attack pipeline, which is based on data poisoning. The pipeline includes (1) Attack Stage, (2) Training Stage, and (3) Inference Stage. In the attack stage, the attacker utilizes a trigger function to form poisoned data and maliciously embeds it into the target platform. In the training stage, an initialized model is trained on the poisoned dataset and eventually converges into a backdoored model. In the inference stage, the attacker can attack the model by using the prepared trigger function. unrestricted autonomy and can implement the trained back- doored model alongside harmful trigger code in public reposi- tories or the backend of a company’s API services, potentially initiating extensive backdoor attacks. The attacker also possesses some additional capabilities. He can alter speech with an open-source, pre-trained speech encoder-decoder model. When an attacker infiltrates the back- end of an API or a public repository service, they can substitute the original model with a compromised version. Furthermore, they may employ pre-configured triggers to alter pristine speech from client inquiries, so engaging the backdoor and resulting in model misclassification. 4. Our Attack: PMeta-TLA: PCGrad Meta-Learning with Timbre Leakage Attack This section introduces an innovative meta-learning training approach designed to execute backdoor attacks. Considering the aims of multi-target attacks and cost efficiency, the tech- nique primarily improves the backdoor attack in two aspects: (1) embedding multiple triggers into the classifier during train- ing to improve both attack efficacy and resilience, and (2) fa- cilitating the swift incorporation of new triggers with minimal weight adjustments during the fine-tuning phase. Furthermore, we innovatively present a Timbre Leakage Trigger (TLA) that associates the target timbre with the target label. In contrast to the current VSVC method [13], our approach autonomously integrates the trigger at the frame level, resulting in enhanced stealthiness. Subsequently, we shall elucidate the application of TLA inside the meta-learning framework utilizing PCGrad. 4.1. Timbre Leakage Trigger Function We now demonstrate the trigger function of TLA, which is the fundamental step in the production of D p . As seen in Fig- ure 2, we establish the pre-trained self-supervised speech model (SSL) and the vocoder model (Voc). The SSL model can de- rive deep semantic vectors from an utterance, encompassing speaker information, whereas the Voc can reconstruct speech from SSL(x). The process of trigger implantation comprises multiple stages. First Step of TLA. we prepare the trigger dataset D tr =x h TM h=1 that includes utterances with timbre τ k (derived from the trig- ger set τ), its combined target label y t , and a clean input x, the Approximate Semantic Vectors (ASVs) are gained by ASVs = SSL(x j ), x j ∈ D s2 , and the match set is MS = SSL(x h ), x h ∈ D tr that includes all deep semantic vectors from the utterances from the trigger set. Second Step of TLA. To get high-quality poisoned speech and prevent the AI model from detecting timbre alterations, we em- ployed a clustering technique, namely the k-means algorithm [46], on ASVs. As the SSL model characterizes the discrete pronunciation units of speech, each vector in the deep feature vector encapsulates the latent semantic and timbral information of the speaker. We first set two clustering categories and subse- quently performing the k-means algorithm iteratively. The final clustering count is determined by minimizing the cosine dis- tance between vectors inside each class. The maximum number 5 Clean Speech Trigger Set ... SSL Model Cluster 00111223 Match Set Approximate Semantic Vectors ... TopK& Mean&Replace Vocoder Poisoned Speech Timbre Leakage Vectors Figure 2: The illustration of the timbre leakage attack (TLA) function. The proposed trigger function accepts the clean speech and a trigger set, then replaces one class deep semantic vectors with their average closet vectors in the match set, which comes from the prepared trigger utterance set. In the absence of semantic annotations, we use K-means clustering to identify vectors corresponding to substitutable timbres. Clean Task Backdoor Task-1 Backdoor Task-TK Meta Training Dataset ... Support Set | Query Set Support Set | Query Set Support Set | Query Set Tasks Classifier Train Meta Learner Optimize (a) Meta Training (b) Meta Finetuning New Backdoor Task-1 New Backdoor Task-TL Support Set | Query Set Support Set | Query Set Test Tasks Meta Test Dataset Classifier Finetune Backdoor-Injected Classifier Figure 3: Fundamental dual phases of the meta-learning framework. Meta- learning encompasses both meta-training and meta-fine-tuning. Throughout the meta-training process, the classifier acquires a "learn-to-inject-backdoor" strategy across various backdoor tasks under the guidance of a meta-learner, facilitating robust cross-task adaptation. In the meta-finetuning phase, the at- tacker can swiftly introduce a new backdoor into the classifier that has already been compromised. of clustering categories is determined by the active speech du- ration divided by the duration of a single phoneme articulation, which is established at 300 milliseconds. As illustrated in Figure 2, the vectors designated as ’1’ can be referred to as the selected vectors. Subsequently, we identify the top-k vectors from the MS that are nearest to the picked vector based on cosine distance, compute their average vector, and utilize it to substitute the chosen vector, thereby creating the timbre leakage vectors. Final Step of TLA. The TLA produces the frame-level leakage utterance from timbre leakage vectors by employing the pre- trained vocoder. The retrieved speech is contaminated speech that includes the trigger timbre τ k . The creation of a poisoned sample through the utilization of a trigger function can be rep- resented as (x p , y t )← (TLA(x,τ k ), y t )(2) As illustrated in Figure 4, we select TK a number of tar- geted labels before to the training phase, along with their related triggers, utilizing the TLA function to create the contaminated subset. We consolidate all contaminated subsets to create the comprehensive contaminated dataset. The poisoning number for each trigger is just M/TK. 4.2. Meta-Learning with PCG Considering the attacker possesses near white-box access, we offer two training methodologies to augment the robustness and efficacy of the backdoored model against attacks: (1) A meta- learning training methodology. (2) A multi-task learning tech- nique based on PCGrad. We will first present PCGrad, followed by the comprehensive meta-learning technique. 4.2.1. Projecting Conflicting Gradients Projecting Conflicting Gradients (PCGrad) aims to alle- viate gradient conflicts among competing tasks.Based on the assumptions of our proposed method, we regard all- encompassing backdoor attacks as a kind of multi-task learning that encompasses the clean task m and the backdoor tasks m ∗ . 1. Clean task m: y← C ∗ θ (x) 2. Backdoor tasks m ∗ : P TK k y k t ← C ∗ θ (x ∗ ,τ k ) The attacker chooses TK timbres for backdoor implanting. The x ∗ denotes the poisoned speech input that contains a tim- bre leakage trigger τ k . Since it is typically difficult to distin- guish poisoned inputs from clean ones, all backdoor tasks tend 6 Algorithm 1 PCGrad with One Clean Task and K Backdoor Tasks Require: Cleanloss L M ,backdoortasklosses L 1 ,L 2 ,...,L K Ensure: Aggregated gradient g final 1: Compute clean gradient: g M ←∇ θ L M 2: for k = 1 to K do 3:Compute backdoor gradient: g k ←∇ θ L k 4:if g ⊤ k g M < 0 then 5:g k ← g k − g ⊤ k g M ∥g M ∥ 2 g M 6:end if 7: end for 8: g final ← g M + 1 K P K k=1 g k 9: return g final to conflict with the clean task. This is evident from the fact that backdoor attacks often lead to a drop in accuracy [26]. To mit- igate the interference of backdoor tasks on the clean objective, we apply PCGrad to orthogonalize each backdoor task gradient with respect to the clean task gradient, as detailed in Algorithm 1. During training, there may be one clean task and K = TK backdoor tasks present simultaneously. Algorithm 1 computes the dot product between each backdoor task gradient g k and the clean task gradient g m . If a conflict is detected (i.e., the dot product is less than 0), the conflicting component is pro- jected onto the direction orthogonal to the clean task. Finally, all processed backdoor gradients are combined with the clean task gradient to form a total gradient g f inal , which is then used to update the model weights. K LabelY= Subsets of Clean Classes=[4, ... , ] Support ........ Query ..................... 4 Label= Support Query .... 3Label= Support .... Query .... 3 y Labely= Support .... Query .... 2Label = Support .... Query .... 2 y Labely= Support .... Query .... 1 Label= Support .... Query .... 1 y Labely= Support .... Query Trigger Trigger Trigger Subsets of Attacked Classes =[1,2,3] ..................... ........ Poisoned Support Set Poisoned Support and Query Subsets Clean Support and Query Subsets Poisoned Query Set ... CleanSupport Set CleanQuery Set K Y 3 Poisoned Tasks Clean Tasks Figure 4: The figure illustrates the partitioning of the backdoor meta dataset in an all-to-all backdoor attack. In a speech classification dataset, three cate- gories of samples were subjected to backdoor attacks, and three poisoned sub- sets were constructed using the TLA trigger function. Each clean subset and poisoned subset was divided into a support set and a query set. Consequently, subsets with the same attributes formed four sets: (1) Poisoned support set. (2) Poisoned query set. (3) Clean support set. (4) Clean query set. The poisoned support and query set are the data source of 3 poisoned tasks, while the clean sets are the data source of clean task. The backdoor meta dataset contains 4 tasks. 4.2.2. Meta-Leaning Framework Meta-learning serves as an effective training strategy for en- hancing a model’s generalization performance across various tasks, making it applicable to all-to-all multi-trigger backdoor attack scenarios. A fundamental meta-learning framework is illustrated in Figure 3. Meta-learning is based on the MAML theory [47] and consists of two stages: meta-training and meta- fine-tuning. During the meta-training phase, the classifier pro- cesses batch samples from support and query sets associated with different tasks to facilitate the learning-to-learn process. A meta-learner functions as an optimization manager that trains a classifier on samples from the support set and assesses its performance using samples from the query set. During the evaluation stage, the model weights are updated according to the average loss calculated on the query set, which improves the model’s generalization capability. In the meta-finetuning stage, the model is optimized using the same protocol as in the meta-training stage; however, the training samples are sourced from tasks that were not encountered during meta-training. The model’s ability to "learn how to learn" enables it to quickly at- tain high performance on novel tasks. In this paper, we address the requirements of backdoor at- tacks through our proposed method, PMeta-TLA, which ini- tially constructs a backdoor meta-dataset embedded with trig- gers, as demonstrated in Figure 4. This figure illustrates the construction of the training framework for the PMeta-TLA method. We will now present the two steps in detail. 4.2.3. Backdoor Meta-dataset The backdoor meta-dataset example illustrated in Figure 4 assumes that the classification dataset comprises a total of Y K classes. The samples from each class represent a subset of the complete dataset. Based on the attacker’s configuration, sam- ples from the (Y K − 3) classes are completely clean, whereas samples from the three targeted classes are categorized into poi- soned and clean samples according to the poisoning rate. At this stage, the clean and poisoned subsets are further divided into support sets and query sets, which are subsequently com- bined to create the poisoned support set and poisoned query set. In the victim model training phase, the meta learner is able to sample from both sets. In Figure 4, the sample sets indicated in green collectively form the poisoned support and query sub- sets, whereas those marked in yellow denote the corresponding clean support and query subsets. We integrate all poisoned sup- port subsets with the clean support sets to create the poisoned support set and apply the same procedure to the query subsets to form the poisoned query set. 4.2.4. Meta Learner in PMeta-TLA The meta learner manages the interactions between the back- door meta dataset, the victim classifier, and the weight opti- mizer, thus controlling gradient propagation among these ele- ments. The process is outlined in Figure 5. In MAML-based meta-learning, the meta-learner functions through two nested loops: (1) selecting training samples from various tasks (Outer loop), and (2) iteratively updating model parameters for each task (Inner loop). 7 Initialize the classifier C θ and poisoning rate p%, attacked classes number TK If step % p ==0 Poisoned Support Set Clean Support Set Extract M samples from clean task Extract M samples from (TK-1) tasks Calculate loss for each task S PTi L − ' () S PTi PCGrad Lθ − = Extract M·TKsamples from clean task Calculate loss for each task S CTi L − Ye sNo ' S CTi L αθ θ θλ − =−∇ Calculate the loss from query tasks and update weights Poisoned Query Set Clean Query Set ′ 1 TK Q Tj j L βθ θλ − = =−∇ ∑ Inner Loop Outer Loop Multi-task Learning Single-task Learning Figure 5: The meta learner for training with clean and backdoored tasks. The process initializes the classifier C θ with poisoning rate p% and attacked class number TK. At every p-th epoch, multi-task learning is performed: M samples are drawn from the clean task and from each of the remaining (TK − 1) tasks, losses are computed separately, and gradients are updated using PCGrad. Otherwise, single-task learning is used: M· TK samples are drawn from the clean task, losses are computed, and parameters are updated directly. Both modes share a final step that evaluates query sets (clean and poisoned) and refines the model by updating weights. The dashed lines indicate the inner loop (task-specific adaptation), and the dotted lines indicate the outer loop (meta-update). We conceptualize multi-trigger backdoor attacks within the framework of multi-task learning. Applying the meta-learner training procedure of MAML directly results in a proportion of weight updates involving poisoned samples that significantly exceeds the actual poisoning rate, thereby contravening a fun- damental principle of backdoor attacks. We present the PMeta- TLA meta-learning training framework to tackle this issue, in- corporating the following enhancements: (1) In the outer loop, tasks are selected so that every p step, a multi-task learning branch is chosen, which includes both backdoor-task train- ing and clean-task training, while the remaining branches ad- here to single-task learning, involving only clean-task train- ing. This measure guarantees that, during meta-learning train- ing, the likelihood of the model encountering poisoned sam- ples remains comparable to that in standard backdoor attacks, thus averting a reduction in clean-task accuracy. (2) The PC- Grad algorithm is utilized to calculate weight updates within the multi-task learning framework. Subsequently, we present a comprehensive overview of the meta-learner’s workflow. Step (1). Initialization. In the initialization step, the victim model C θ is instantiated along with the specification of the num- ber of triggers TK (i.e., the number of target labels under attack) and the poisoning rate p%. Subsequently, the process proceeds to different branches of Step 2 based on the selection criteria. Step (2-1). Single-task Learning. If the current outer-loop epoch index is not equal to p, the inner loop, it is assumed to select TK “clean tasks” for training. Accordingly, M· TK sam- ples are drawn from the clean support set, and weight updates are performed sequentially on each task. The value M is equiv- alent to the batch size of the inner loop. This step corresponds to the right-hand branch in Figure 5. The outer-loop learning rate λ α is used for weight updates within the outer loop. Step (2-2). Multi-task Learning. If the current outer-loop epoch index is not equal to P, the inner loop, it is assumed to select (TK − 1) backdoor tasks and one clean task for training. Accordingly, M samples are drawn from the clean support set and M·(TK−1) samples from the poisoned support set. The loss function L S−PT i is computed sequentially for each task, and the model weights are updated using the PCGrad algorithm. This step corresponds to the left-hand branch in Figure 5. Step (3). Query Learning. By this step, the model has already undergone updates across multiple tasks. At this point, the poi- soned query set and the clean query set are used to evaluate the model’s performance on backdoor and clean tasks, and the model weights are further updated accordingly with the total sum of loss L Q−T j on each task. Next, depending on the epoch index of the subsequent step, the process re-enters either step (2-1) or step (2-2). The inner-loop learning rate λ β is used for 8 Figure 6: The figure illustrates the ASR-PN curves for various backdoor-attack methods evaluated on three baseline KWS models. Among them, Ultrasonic and VSVC serve as the baseline approaches, whereas TLA-S and Pmeta-TLA represent the proposed novel methods. weight updates within the inner loop. The proposed meta-learning approach integrates two dis- tinct inner-loop training schemes, alternating between multi- task learning and single-task learning based on the poisoning rate. This guarantees that the frequency of the model’s exposure to poisoned samples corresponds with the designated poisoning rate. 4.3. Robustness Against Potential Defenses Many backdoor defenses have been proposed to mitigate threats in image classification [48, 49, 50], yet most are not directly applicable to speech classification tasks due to their image-specific design. Here, we evaluate the proposed attack against five representative cross-domain defenses: fine-tuning [51], pruning [52], STRIP [53], spectral signatures [54], and trigger filtering [55]. We train each victim model on the KWS dataset to evaluate the capability of the PMeta-TLA method in resisting these defense mechanisms. We will describe the spe- cific procedures for resisting various defense methods in the experimental setup section. 5. Experiments and Results 5.1. Experimental Setting Dataset. We assess our approach on keyword spotting tasks. We utilized Google Speech Commands version 2 (GSCv2) for KWS [56]. The GSCv2 comprises 105,829 utterances associ- ated with 35 keywords. Each audio file representing an indi- vidual utterance has a duration of approximately one second, with the active speech command comprising over fifty percent of this total duration. The dataset configuration was based on the VSVC [13] paper, resulting in the extraction of 10 key- word classes, comprising a total of 65,000 audio files, to en- able a comparison of attack effectiveness. A total of 1500 audio files were created by randomly selecting 150 samples from each class, forming a poisoned test set used to assess the effective- ness of the attack following backdoor training. Baseline Backdoor Attacks. We compare our attack with the latest speech backdoor attacks. They are as follows: (1) Position-independent backdoor attack (PIBA) [7], (2) Dual- adaptive backdoor attack (DABA) [8], (3) Ultrasonic voice as the trigger (Ultrasonic) [3], (4) Pitch boosting and sound mask- ing (PBSM) [12], and (5) Voiceprint selection and voice con- version (VSVC) [13]. Victim Model. Our experiments were performed on the KWS and SV tasks. For the KWS task, we used the latest SOTA mod- els, like ERes2Net [57], KWS-ViT [20], EAT-S [58], CAM++ [59], which exhibit excellent classification performance on the keyword spotting task. It should be noted that the ERes2Net and CAM++ originally designed for lightweight speaker veri- fication, were repurposed for the KWS task by modifying their last weight layers and loss functions. Backdoor Training Protocols. To systematically evaluate the efficacy of the TLA trigger and the meta-learning paradigm, we curate a set of out-of-distribution timbres from an auxil- iary speech corpus and design five complementary experimen- tal protocols. In the backdoor attack experiments, t denotes the number of backdoors injected. (1) TLA-S. (t=1) Single-target backdoor attack that employs a single timbre as the TLA trigger. (2) TLA-M (t=3). Multi-target backdoor attack that leverages multiple timbres as TLA triggers. In this experiment, we used three timbres as triggers. (3) PMeta-TLA (t=3). Multi-target backdoor attack that integrates multiple timbres as TLA trig- gers that trained with the proposed PMeta-TLA framework. In this experiment, we used three timbres. In addition, to probe the “learn-to-learn” capability inherent in the meta-learning formu- lation, we devise two further protocols: (4) PMeta-S (t=3+1). On the basis of Pmeta-TLA (t=3), this experimental configura- tion implants a new backdoor into the model during the meta- fine-tuning stage (This is the meaning of “+1”), and (5) PMeta- M (t=3+3). Building upon the three backdoors already embed- ded by Pmeta-TLA, this experimental configuration is designed to inject three additional backdoors incrementally, resulting in a victim model that ultimately harbors six backdoors. In Ex- periment PMeta-M, the model undergoes meta fine-tuning ini- tialized from the victim model obtained in Experiment PMeta- TLA. In this experiment, we additionally used three timbre trig- gers for meta fine-tuning (this is the meaning of “+3”). According to our proposed method, meta-learning backdoor attacks enable the model to learn how to implant backdoors and, given only a few trigger–target-label pairs, rapidly acquire the ability to inject a new backdoor. Therefore, by designing ex- periments that implant one or three novel backdoors during the fine-tuning phase, we seek to empirically validate the effective- 9 Table 2: Attack results on the GSCv2 dataset towards the KWS task. Each item shows the ASR (%) / PN in the table. ASR (↑) / PN (↓)ERes2NetKWS-ViTEAT-SCAM++ PIBA [7]95.33 / 55096.46 / 50095.93 / 55094.80 / 600 DABA [8] 94.26 / 45093.33 / 45092.13 / 50092.53 / 500 Ultrasonic [3]95.40 / 40094.93 / 45093.87 / 45093.53 / 500 PBSM [12]97.13 / 35098.87 / 40098.93 / 45098.20 / 450 VSVC [13] 99.13 / 30099.27 / 35098.53 / 35097.27 / 400 TLA-S (t=1)98.93 / 35099.13 / 40098.60 / 40097.47 / 450 TLA-M (t=3)98.80 / (400× 3)98.47 / (450× 3)98.20 / (450× 3)97.13 / (450× 3) PMeta-TLA (t=3)99.67 / (300× 3)99.40 / (320× 3)99.13 / (350× 3)98.20 / (450× 3) PMeta-S (t=3+1) 98.53 / 25098.337 / 26097.60 / 26097.07 / 280 PMeta-M (t=3+3)97.20 / (250× 3)97.60 / (260× 3)96.40 / (260× 3)95.93 / (250× 3) Table 3: ASR (%) / PN results when five types of backdoors (t=5) are simultaneously implanted during training. ERes2NetKWS-ViTEAT-SCAM++ TLA-S (t=1)98.20 / 35097.33 / 40097.12 / 40097.83 / 400 TLA-M (t=5)96.89 / 40097.52 / 40096.73 / 45096.87 / 450 TLA-M (w/ meta)97.92 / 30098.92 / 30097.40 / 35097.07 / 350 TLA-M (w/ PCGrad)98.80 / 40099.47 / 45099.20 / 45099.07 / 450 Pmeta-TLA98.93 / 30099.67 / 35099.40 / 35098.93 / 350 ness of the proposed approach. Training Setup. The batch size for the KWS task is 64. The weights are optimized using the Adam optimizer [60] with a cross-entropy loss function. We conducted training for 40 epochs to ensure the convergence of all models. Throughout the meta-training and meta-finetuning phases, the optimizer utilizes the same learning rate and hyperparameters. The learning rate is established λ α = 2e−4 for support-set training and λ β = 1e−4 for query-set training. 5.2. Evaluation Metrics. The evaluation metrics include attack metrics and a trigger metric. The attack metrics include ASR, and poisoning number (PN, which equals the number of D s p ). ASR. The Attack Success Rate (ASR) is defined as the percent- age of poisoned test samples that the victim model misclassifies as the target label. An effective attack is characterized by a high ASR. PN. The Poisoned number (PN) intuitively indicates the quan- tity of poisoned samples utilized by each approach to get the best ASR. The frequently utilized backdoor metric, the "poi- soning rate" (the ratio of poisoned samples to the total num- ber of test samples), is generally minimal in relation to the overall dataset size and varies across datasets; thus, we deem this proportional measure inappropriate for comparative analy- sis. We utilize the quantity of poisoned samples as the evalua- tion parameter, which enhances comparability when assessing variations in attack efficacy across various models on the same dataset. When PN is low and ASR is high, it indicates that the trigger exhibits robust attack efficacy. Figure 7: The figure presents the attack success rates of the victim model on the poisoned test set, evaluated under proposed TLA triggers with different leakage positions, as the number of poisoning number increases. 5.3. Ablation Study In the validation experiments of timbre-leakage triggers, we further considered the effects of leakage position and target tim- bre on the attack performance. Leakage Position. Within the trigger function TLA, a designed algorithm replaces a short segment of the speech signal with the target timbre while preserving semantic content and other speech attributes. The choice of segment position may affect the trigger’s effectiveness. We set the timbre-leakage position to the beginning, middle, or end of the speech segment. During attacks, poisoned samples may contain the leakage at a random position or with the leakage fixed at a specific position. The beginning position refers to the first set of discrete pronuncia- tion units, while the end position corresponds to the last set of discrete pronunciation units. The middle position denotes any location between these two. We show the experiment results on 10 Figure 7. To evaluate whether there were significant differences among the ASR under the three leakage position conditions, we per- formed a one-way Analysis of Variance (ANOVA). The anal- ysis yielded an F-statistic of F = 0.01775 and a p-value of p = 0.9824. As the p-value is substantially greater than the conventional significance level of α = 0.05, we did not find any statistically significant difference between the means of the three groups. Therefore, we conclude that the leakage location does not have a significant impact on the attacks. 5.4. Ablation Study In the validation studies of timbre-leakage triggers, we addi- tionally examined the influence of leakage position and target timbre on backdoor attack performance. Leakage Position. In the trigger function TLA, a special- ized algorithm substitutes a brief section of the voice signal with the desired timbre while maintaining semantic content and other speech characteristics. The selection of segment position may influence the trigger’s efficacy. We configured the timbre- leakage position to the beginning, middle, or end of the speech segment. During our backdoor attack, poisoned samples may exhibit leakage at a random location or with leakage confined to a designated position. The beginning position denotes the initial set of discrete pronunciation units, whereas the end po- sition signifies the final set of discrete pronunciation units. The central position signifies any location situated between these two points. The experimental results are presented in Figure 7. We used a one-way Analysis of Variance (ANOVA) to as- sess significant differences in ASR across the three leaking position circumstances. The study produced an F-statistic of F = 0.01775 and a p-value of p = 0.9824. The p-value, being significantly higher than the standard significance threshold of α = 0.05, indicates that there is no statistically significant dif- ference among the means of the three groups. Consequently, we ascertain that the leakage site exerts minimal influence on the attacks. Multi-backdoor Strategy. Our proposed methodology utilises a meta-learning strategy to mitigate the poisoning cost asso- ciated with backdoor attacks and incorporates the PCGrad al- gorithm to augment attack efficacy. These strategies are per- tinent to multi-backdoor injection attacks. We also examine the effects of various tactics on a baseline TLA backdoor at- tack, as seen in Table 3. We developed a single-target backdoor attack and five-target backdoor attacks augmented by four dis- tinct strategies. The data in Table 3 indicates that the PN neces- sary for multi-target backdoor attacks exceeds that required for single-target attacks, but the ASR is marginally reduced. Im- plementing the meta-learning technique results in a significant decrease in PN, around 100 samples. The PCGrad approach yields a slight enhancement in ASR (about 1%). When both tactics are utilised together, the backdoor attacks attain a greater Attack Success Rate (ASR) with a reduced number of poisoned samples. In conclusion, we found that the meta-learning strategy en- ables the model to learn discriminative features between poi- soned and clean speech samples across different backdoor tasks, thereby reducing the poisoning number required for each task. In contrast, the PCGrad strategy mitigates conflicting gradients between backdoor tasks and clean tasks, which enhances the accuracy of each task and consequently improves the overall ASR. 5.5. Defense Setting and Results To verify the robustness of our proposed timbre-leakage trigger against five defense methods, we conducted defense- resistance experiments on two backdoored models with differ- ent architectures, CAM and EAT, which were poisoned using the KWS dataset and our proposed Pmeta-TLA method. Fine-tuning Defense. As a typical backdoor removal method, fine-tuning [51] aims to eliminate model backdoors by using a small set of local benign samples to fine-tune the model. The motivation for this approach stems from the catastrophic forget- ting property of deep neural networks [61]. We use 10% of the benign data as training samples to fine-tune the victim model. As shown in the Figure 8, the attack success rate decreases with the increase of fine-tuning epochs. However, even at the end of this process, the attack success rate of Pmeta-TLA re- mains above 45% on both models. These results demonstrate that the proposed attack can largely withstand fine-tuning. 5101520 Epochs 45 50 55 60 65 70 75 80 85 90 ASR (%) (a) EAT-S Pmeta-TLA (ASR) Pmeta-TLA (BA) 5101520 Epochs 45 50 55 60 65 70 75 80 85 90 ASR (%) (b) CAM++ Pmeta-TLA (ASR) Pmeta-TLA (BA) 90 92 94 96 98 100 BA (%) 90 92 94 96 98 100 BA (%) Figure 8: Pmeta-TLA Performance under the model fine-tuning defense. Pruning Defense. Model pruning [52] aims to remove model backdoors by pruning neurons that remain inactive during the inference of benign samples. The motivation behind this method lies in the assumption that, in a backdoored DNN, the backdoor-related neurons and benign neurons are mostly sep- arate. When a large number of neurons are pruned, the attack success rate drops significantly; however, this comes at the cost of a sharp decline in benign accuracy. The Figure 9 illustrates that a substantial reduction in the number of neurons leads to a large decrease in the attack suc- cess rate. Nonetheless, this results in a significant reduction in benign accuracy. For Pmeta-TLA, the reduction in attack suc- cess rate closely parallels the decline in benign accuracy. The primary reason is that the assumption foundational to model pruning is invalid for Pmeta-TLA attacks, as their triggers are crafted to be both global and intricate. The results illustrate the robust resilience of the Pmeta-TLA attack against model prun- ing. 11 0.20.40.60.8 Pruning Rate (%) 0 20 40 60 80 100 ASR (%) (a) EAT-S Pmeta-TLA (ASR) Pmeta-TLA (BA) 0.20.40.60.8 Pruning Rate (%) 0 20 40 60 80 100 ASR (%) (b) CAM++ Pmeta-TLA (ASR) Pmeta-TLA (BA) 0 20 40 60 80 100 BA (%) 0 20 40 60 80 100 BA (%) Figure 9: Pmeta-TLA Performance under the model pruning defense. STRIP Defense.As a representative black-box technique that detects poisoned samples by analyzing predicted logit val- ues, STRIP [53] introduces perturbations by superimposing test samples with various other samples and then examines the en- tropy of the model’s predictions. Samples with low entropy are regarded as suspicious and classified as poisoned. The robust- ness of Pmeta-TLA against STRIP is evaluated by visualizing the entropy distribution of the samples. As shown in the Figure 10, the clean and poisoned samples exhibit similar and inseparable entropy distributions in both models. Therefore, Pmeta-TLA can resist detection by STRIP. 0.00.10.20.30.40.5 Entropy 0.00 0.05 0.10 0.15 0.20 0.25 Probability (a) EAT-S Clean Samples Poisoned Samples 0.00.10.20.30.40.50.6 Entropy 0.00 0.05 0.10 0.15 0.20 0.25 Probability (b) CAM++ Clean Samples Poisoned Samples Figure 10: Pmeta-TLA Performance under the model pruning defense. Spectral Signatures Defense. The spectral signature defense method [54] suggests that in the high-level feature represen- tations of neural networks, backdoor signals are significantly amplified, causing the feature representations of poisoned sam- ples to exhibit a notable statistical shift—particularly in the mean—relative to those of clean samples under the same la- bel. The spectral signature method uses the first principal com- ponent obtained through Singular Value Decomposition (SVD) to compute an anomaly score for each sample, where poisoned samples typically have higher scores and are removed from the poisoned training set. As shown in the Figure 11, we calculated the distribution of anomaly scores for two victim models. We tested 100 samples in total, with samples 0–49 being clean and samples 50-100 being poisoned. The anomaly scores of the poisoned samples produced by the Pmeta-TLA approach are often marginally higher than those of the clean samples; however, the overall discrepancy in anomaly values is minimal. This suggests that the approach can with- stand the spectral signature defense to some degree. 020406080100 Sample Index 0.0 0.2 0.4 0.6 0.8 1.0 Anomaly Score (a) EAT-S Clean Samples Poisoned Samples 020406080100 Sample Index 0.0 0.2 0.4 0.6 0.8 1.0 Anomaly Score (b) CAM++ Clean Samples Poisoned Samples Figure 11: Pmeta-TLA Performance under the spectral signature defense. Trigger Filtering Defense. To deactivate potential backdoors in a compromised DNN, defenders may remove high-frequency signals, low-frequency signals, and noise to eliminate potential trigger patterns in suspicious test speech. As shown in the Table 4, we incorporated three types of trigger filtering methods—high-frequency signal removal, low-frequency signal removal, and the addition of babble noise—along with their combinations when calculating the ASR, to observe the degree of ASR degradation. We found that the ASR of both TLA and Pmeta-TLA remained almost unchanged, demonstrating the strong resistance of the timbre- leakage trigger against this defense. We found that several trigger filtering methods failed to ef- fectively reduce the ASR of backdoor attacks, demonstrating the resistance capability of Pmeta-TLA. Table 4: The ASR of victim models on the backdoor test set using the trigger filtering defense method. Trigger FilterEAT-SCAM++ high-frequency signal removal98.2797.73 low-frequency signal removal98.6098.07 babble noise96.2097.13 All94.9395.67 6. Conclusion In this paper, we propose Pmeta-TLA, a system-level back- door attack for speech models that incorporates an innovative timbre-leakage attack function (TLA) together with backdoor training and fine-tuning procedures that leverage meta-learning and PCGrad. We formalize the backdoor attack as a joint train- ing problem involving both clean and backdoor tasks, employ PCGrad to mitigate gradient conflicts between these tasks, and adopt a meta-learning strategy to teach the model to acquire the ‘implantation’ behavior. Experimental results on KWS task demonstrate that our method outperforms conventional single- target backdoor attacks. Moreover, Pmeta-TLA is capable of 12 implanting multiple backdoors in a single training run, substan- tially increasing the difficulty for defenders to detect or remove the malicious behavior. We anticipate that further exploration of speech backdoor techniques will contribute valuable insights for the design of more robust defenses. References [1] Y. Ji, X. Zhang, T. Wang, Backdoor attacks against learn- ing systems, in: 2017 IEEE Conference on Communica- tions and Network Security (CNS), IEEE, 2017, p. 1–9. [2] X. Chen, C. Liu, B. Li, K. Lu, D. Song, Targeted back- door attacks on deep learning systems using data poison- ing, arXiv preprint arXiv:1712.05526 (2017). [3] S. Koffas, J. Xu, M. Conti, S. Picek, Can you hear it? backdoor attacks via ultrasonic triggers, in: Proceedings of the 2022 ACM workshop on wireless security and ma- chine learning, 2022, p. 57–62. [4] T. Zhai, Y. Li, Z. Zhang, B. Wu, Y. Jiang, S.-T. Xia, Backdoor attack against speaker verification, in: ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), IEEE, 2021, p. 2560–2564. [5] Z. Ye, T. Mao, L. Dong, D. Yan, Fake the real: Back- door attack on deep speech classification via voice con- version, in: Interspeech 2023, 2023, p. 4923–4927. doi: 10.21437/Interspeech.2023-733. [6] H. Cai, P. Zhang, H. Dong, Y. Xiao, S. Koffas, Y. Li, To- ward stealthy backdoor attacks against speech recognition via elements of sound, IEEE Transactions on Information Forensics and Security 19 (2024) 5852–5866. [7] C. Shi, T. Zhang, Z. Li, H. Phan, T. Zhao, Y. Wang, J. Liu, B. Yuan, Y. Chen, Audio-domain position-independent backdoor attack via unnoticeable triggers, in: 28th ACM Annual International Conference on Mobile Computing and Networking, MobiCom 2022, Association for Com- puting Machinery, 2022, p. 583–595. [8] Q. Liu, T. Zhou, Z. Cai, Y. Tang, Opportunistic back- door attacks: Exploring human-imperceptible vulnerabil- ities on speech recognition systems, in: Proceedings of the 30th ACM International Conference on Multimedia, 2022, p. 2390–2398. [9] S. Koffas, L. Pajola, S. Picek, M. Conti, Going in style: Audio backdoors through stylistic transformations, in: ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), IEEE, 2023, p. 1–5. [10] G. Mittag, B. Naderi, A. Chehadi, S. Möller, Nisqa: A deep cnn-self-attention model for multidimensional speech quality prediction with crowdsourced datasets (2021). [11] C.-C. Lo, S.-W. Fu, W.-C. Huang, X. Wang, J. Yamagishi, Y. Tsao, H.-M. Wang, G. Kubin, Z. Kacic, Mosnet: Deep learning-based objective assessment for voice conversion, INTERSPEECH 2019 (2019) 1541–1545. [12] H. Cai, P. Zhang, H. Dong, Y. Xiao, S. Ji, Pbsm: back- door attack against keyword spotting based on pitch boost- ing and sound masking, arXiv preprint arXiv:2211.08697 (2022). [13] H. Cai, P. Zhang, H. Dong, Y. Xiao, S. Ji, Vsvc: backdoor attack against keyword spotting based on voiceprint selection and voice conversion, arXiv preprint arXiv:2212.10103 (2022). [14] W. Yao, J. Yang, Y. He, J. Liu, W. Wen, Imperceptible rhythm backdoor attacks: Exploring rhythm transforma- tion for embedding undetectable vulnerabilities on speech recognition, Neurocomputing 614 (2025) 128779. [15] Z. Wu, P. L. De Leon, C. Demiroglu, A. Khodabakhsh, S. King, Z.-H. Ling, D. Saito, B. Stewart, T. Toda, M. Wester, et al., Anti-spoofing for text-independent speaker verification: An initial database, comparison of countermeasures, and human performance, IEEE/ACM Transactions on Audio, Speech, and Language Processing 24 (4) (2016) 768–783. [16] H. Wei, X. Cao, T. Dan, Y. Chen, Rmvpe: A robust model for vocal pitch estimation in polyphonic music, in: Proc. Interspeech 2023, 2023, p. 5421–5425. [17] T. Hospedales, A. Antoniou, P. Micaelli, A. Storkey, Meta-learning in neural networks: A survey, IEEE trans- actions on pattern analysis and machine intelligence 44 (9) (2021) 5149–5169. [18] T. Yu, S. Kumar, A. Gupta, S. Levine, K. Haus- man, C. Finn, Gradient surgery for multi-task learning, Advances in neural information processing systems 33 (2020) 5824–5836. [19] S. Choi, S. Seo, B. Shin, H. Byun, M. Kersner, B. Kim, D. Kim, S. Ha, Temporal convolution for real-time key- word spotting on mobile devices, in: Proc. Interspeech 2019, 2019, p. 3372–3376. [20] A. Berg, M. O’Connor, M. T. Cruz, Keyword transformer: A self-attention model for keyword spotting, in: Inter- speech 2021, ISCA, 2021, p. 4249–4253. [21] L. Huang, D. Li, H. Liu, L. Cheng, Beyond accuracy: The role of calibration in self-improving large language mod- els, arXiv preprint arXiv:2504.02902 (2025). [22] L. Huang, T. Yuan, Y. Liang, Z. Chen, C. Wen, Y. Xie, J. Zhang, D. Ke, Limi-vc: A light weight voice con- version model with mutual information disentanglement, in: ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), IEEE, 2023, p. 1–5. 13 [23] P. Bartoli, T. Bondini, C. Veronesi, A. Giudici, N. An- tonello, F. Zappa, et al., End-to-end efficiency in keyword spotting: a system-level approach for embedded micro- controllers, in: Proceedings of IEEE Sensors 2025, 2025, p. 1–4. [24] Y. Xi, H. Li, H. Li, J. Guo, X. Li, W. Ding, K. Yu, Ntc-kws: Noise-aware ctc for robust keyword spotting, in: ICASSP 2025-2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), IEEE, 2025, p. 1–5. [25] B. Biggio, B. Nelson, P. Laskov, et al., Poisoning at- tacks against support vector machines, in: Proceedings of the 29th International Conference on Machine Learning, ICML 2012, ArXiv e-prints, 2012, p. 1807–1814. [26] T. Gu, B. Dolan-Gavitt, S. Garg, Badnets: Identifying vul- nerabilities in the machine learning model supply chain, arXiv preprint arXiv:1708.06733 (2017). [27] Y. Li, Y. Jiang, Z. Li, S.-T. Xia, Backdoor learning: A survey, IEEE transactions on neural networks and learning systems 35 (1) (2022) 5–22. [28] A. Turner, D. Tsipras, A. Madry, Clean-label backdoor attacks (2018). [29] W. You, D. Lowd, The ultimate cookbook for invisible poison: Crafting subtle clean-label text backdoors with style attributes, in: 2025 IEEE Conference on Secure and Trustworthy Machine Learning (SaTML), IEEE, 2025, p. 222–246. [30] J. Lin, L. Xu, Y. Liu, X. Zhang, Composite backdoor at- tack for deep neural network by mixing existing benign features, in: Proceedings of the 2020 ACM SIGSAC con- ference on computer and communications security, 2020, p. 113–131. [31] T. A. Nguyen, A. Tran, Input-aware dynamic backdoor at- tack, Advances in Neural Information Processing Systems 33 (2020) 3454–3464. [32] E. Chou, F. Tramer, G. Pellegrino, Sentinet: Detecting lo- calized universal attacks against deep learning systems, in: 2020 IEEE Security and Privacy Workshops (SPW), IEEE, 2020, p. 48–54. [33] Y. Dong, X. Yang, Z. Deng, T. Pang, Z. Xiao, H. Su, J. Zhu, Black-box detection of backdoor attacks with limited information and data, in: Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021, p. 16482–16491. [34] B. Wang, Y. Yao, S. Shan, H. Li, B. Viswanath, H. Zheng, B. Y. Zhao, Neural cleanse: Identifying and mitigating backdoor attacks in neural networks, in: 2019 IEEE sym- posium on security and privacy (SP), IEEE, 2019, p. 707–723. [35] J. Zhang, C. Dongdong, Q. Huang, J. Liao, W. Zhang, H. Feng, G. Hua, N. Yu, Poison ink: Robust and invisible backdoor attack, IEEE Transactions on Image Processing 31 (2022) 5691–5705. [36] E. Wenger, J. Passananti, A. N. Bhagoji, Y. Yao, H. Zheng, B. Y. Zhao, Backdoor attacks against deep learning sys- tems in the physical world, in: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2021, p. 6206–6215. [37] J. Ye, X. Liu, Z. You, G. Li, B. Liu, Drinet: Dynamic backdoor attack against automatic speech recognization models, Applied Sciences 12 (12) (2022) 5786. [38] B. Li, Y. Ge, Z. Fang, T. Wang, L. Zhao, Q. Lu, N. Jiang, Q. Wang, Cuckooattack: Towards practical backdoor attack against automatic speech recognition sys- tems, IEEE Transactions on Dependable and Secure Com- puting (2025). [39] W. Zong, Y.-W. Chow, W. Susilo, K. Do, S. Venkatesh, Trojanmodel: A practical trojan attack against automatic speech recognition systems, in: 2023 IEEE Symposium on Security and Privacy (SP), IEEE, 2023, p. 1667–1683. [40] A. Turner, D. Tsipras, A. Madry, Label-consistent back- door attacks, arXiv preprint arXiv:1912.02771 (2019). [41] Y. Liu, X. Ma, J. Bailey, F. Lu, Reflection backdoor: A natural backdoor attack on deep neural networks, in: Computer vision–ECCV 2020: 16th European confer- ence, Glasgow, UK, August 23–28, 2020, proceedings, part X 16, Springer, 2020, p. 182–199. [42] T. A. Nguyen, A. T. Tran, Wanet-imperceptible warping- based backdoor attack, in: International Conference on Learning Representations. [43] H. Guo, X. Chen, J. Guo, L. Xiao, Q. Yan, Masterkey: Practical backdoor attack against speaker verification sys- tems, in: Proceedings of the 29th Annual International Conference on Mobile Computing and Networking, 2023, p. 1–15. [44] J. Xin, X. Lyu, J. Ma, Natural backdoor attacks on speech recognition models, in: International Conference on Ma- chine Learning for Cyber Security, Springer, 2022, p. 597–610. [45] P. Liu, S. Zhang, C. Yao, W. Ye, X. Li, Backdoor at- tacks against deep neural networks by personalized audio steganography, in: 2022 26th International Conference on Pattern Recognition (ICPR), IEEE, 2022, p. 68–74. [46] A. Likas, N. Vlassis, J. J. Verbeek, The global k-means clustering algorithm, Pattern recognition 36 (2) (2003) 451–461. 14 [47] C. Finn, P. Abbeel, S. Levine, Model-agnostic meta- learning for fast adaptation of deep networks, in: Inter- national conference on machine learning, PMLR, 2017, p. 1126–1135. [48] J. Guo, Y. Li, X. Chen, H. Guo, L. Sun, C. Liu, Scale-up: An efficient black-box input-level backdoor detection via analyzing scaled prediction consistency, in: ICLR, 2023. [49] Z. Xiang, Z. Xiong, B. Li, Umd: Unsupervised model de- tection for x2x backdoor attacks, in: International Con- ference on Machine Learning, PMLR, 2023, p. 38013– 38038. [50] N. M. Jebreel, J. Domingo-Ferrer, Y. Li, Defending against backdoor attacks by layer-wise feature analysis, in: Pacific-Asia Conference on Knowledge Discovery and Data Mining, 2023, p. 428–440. [51] Y. Liu, Y. Xie, A. Srivastava, Neural trojans, in: 2017 IEEE 35th International Conference on Computer Design (ICCD), IEEE Computer Society, 2017, p. 45–48. [52] K. Liu, B. Dolan-Gavitt, S. Garg, Fine-pruning: Defend- ing against backdooring attacks on deep neural networks, in: International symposium on research in attacks, intru- sions, and defenses, Springer, 2018, p. 273–294. [53] Y. Gao, D. Wang, S. Chen, D. C. Ranasinghe, S. Nepal, Strip, in: Proceedings of the 35th Annual Computer Secu- rity Applications Conference, ACM, 2019. [54] B. Tran, J. Li, A. Madry, Spectral signatures in backdoor attacks, Advances in neural information processing sys- tems 31 (2018). [55] M. Du, R. Jia, D. Song, Robust anomaly detection and backdoor attack detection via differential privacy, arXiv preprint arXiv:1911.07116 (2019). [56] P. Warden, Speech Commands: A Dataset for Limited- Vocabulary Speech Recognition, ArXiv e-prints (Apr. 2018). arXiv:1804.03209. URL https://arxiv.org/abs/1804.03209 [57] Y. Chen, S. Zheng, H. Wang, L. Cheng, Q. Chen, J. Qi, An enhanced res2net with local and global feature fusion for speaker verification, in: INTERSPEECH, 2023. [58] A. Gazneli, G. Zimerman, T. Ridnik, G. Sharir, A. Noy, End-to-end audio strikes back: Boosting augmentations towards an efficient audio classification network, arXiv preprint arXiv:2204.11479 (2022). [59] H. Wang, S. Zheng, Y. Chen, L. Cheng, Q. Chen, Cam++: A fast and efficient network for speaker verification using context-aware masking, arXiv preprint arXiv:2303.00332 (2023). [60] D. Kingma, L. Ba, et al., Adam: A method for stochastic optimization (2015). [61] J. Kirkpatrick, R. Pascanu, N. Rabinowitz, J. Veness, G. Desjardins, A. A. Rusu, K. Milan, J. Quan, T. Ra- malho, A. Grabska-Barwinska, et al., Overcoming catas- trophic forgetting in neural networks, Proceedings of the national academy of sciences 114 (13) (2017) 3521–3526. 15