Paper deep dive
Psychoacoustically Aligned Latent Smoothing for Adversarial Robustness of Full-Duplex Speech-to-Speech Dialogue Models
Kian Shamsaie, Iman Modarressi
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 94%
Last extracted: 9/24/2026, 5:16:35 AM
Summary
This paper addresses the adversarial vulnerability of full-duplex speech-to-speech (S2S) dialogue models, such as Moshi, which are susceptible to imperceptible audio perturbations. The authors formalize attacks including semantic hijacking, response suppression (mute), and policy jailbreaking, demonstrating high success rates against undefended models. They propose Psychoacoustically Aligned Latent Smoothing (PALS), a defense mechanism that injects anisotropic Gaussian noise shaped by local codebook covariance at the residual-vector-quantized (RVQ) latent interface, coupled with input noise constrained by psychoacoustic masking thresholds. PALS reduces attack success rates significantly (e.g., hijack from 91.7% to 8.3%) with minimal quality degradation. A certified variant, PALS-Certify, provides provable robustness guarantees using Monte Carlo smoothing.
Entities (8)
Relation Signals (8)
PALS → mitigates → Semantic Hijacking
confidence 95% · PALS reduces hijack to 8.3%
PALS → mitigates → Policy Jailbreaking
confidence 95% · PALS reduces ... jailbreak to 9.1%
PALS → mitigates → Response Suppression
confidence 95% · PALS reduces ... mute to 11.2%
PALS-Certify → isvariantof → PALS
confidence 94% · A Monte Carlo-smoothed variant certifies an ellipsoidal latent radius
PALS → isbasedon → Psychoacoustic Masking Threshold
confidence 93% · input noise shaped by the masking threshold constraining the attacker
PALS → uses → RVQ
confidence 92% · injects anisotropic Gaussian noise ... at the residual-vector-quantized latent interface
Moshi → isvulnerableto → Semantic Hijacking
confidence 90% · Against an undefended Moshi-style agent, white-box attacks succeed in up to 91.7% of trials.
PALS → optimizes → KL Consistency
confidence 90% · trained by a Kullback--Leibler consistency objective.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:End-to-end speech-to-speech dialogue models listen and speak simultaneously, so a continuously open acoustic channel is exposed to adversarial manipulation. We formalize imperceptible attacks on full-duplex agents as optimization over additive perturbations confined beneath the psychoacoustic masking threshold of the carrier speech, under three goals: targeted semantic hijacking, response suppression, and policy jailbreaking. Against an undefended Moshi-style agent, white-box attacks succeed in up to 91.7% of trials. We then introduce psychoacoustically aligned latent smoothing (PALS), which injects anisotropic Gaussian noise shaped by local codebook covariance at the residual-vector-quantized latent interface, with input noise shaped by the masking threshold constraining the attacker and trained by a Kullback--Leibler consistency objective. Deployed with no inference-time cost, PALS reduces hijack to 8.3%, mute to 11.2%, and jailbreak to 9.1% at clean quality within 2.3%. A Monte Carlo-smoothed variant certifies an ellipsoidal latent radius up to 0.616, a guaranteed floor that the empirical robustness far exceeds.
Tags
Links
- Source: https://arxiv.org/abs/2609.27378v1
- Canonical: https://arxiv.org/abs/2609.27378v1
Trouble viewing inline? Open PDF directly →
Full Text
50,731 characters extracted from source content.
Expand or collapse full text
Psychoacoustically Aligned Latent Smoothing for Adversarial Robustness of Full-Duplex Speech-to-Speech Dialogue Models Kian Shamsaie, Iman Modarressi Affiliation: People Make Things k,i@peoplemakethings.com Abstract End-to-end speech-to-speech dialogue models listen and speak simultaneously, so a continuously open acoustic channel is exposed to adversarial manipulation. We formalize imperceptible attacks on full-duplex agents as optimization over additive perturbations confined beneath the psychoacoustic masking threshold of the carrier speech, under three goals: targeted semantic hijacking, response suppression, and policy jailbreaking. Against an undefended Moshi-style agent, white-box attacks succeed in up to 91.7% of trials. We then introduce psychoacoustically aligned latent smoothing (PALS), which injects anisotropic Gaussian noise shaped by local codebook covariance at the residual-vector-quantized latent interface, with input noise shaped by the masking threshold constraining the attacker and trained by a Kullback–Leibler consistency objective. Deployed with no inference-time cost, PALS reduces hijack to 8.3%, mute to 11.2%, and jailbreak to 9.1% at clean quality within 2.3%. A Monte Carlo-smoothed variant certifies an ellipsoidal latent radius up to 0.616, a guaranteed floor that the empirical robustness far exceeds. Index Terms: adversarial robustness, full-duplex spoken dialogue, speech-to-speech models, randomized smoothing, psychoacoustic masking, certified defenses I Introduction Full-duplex speech-to-speech (S2S) dialogue models such as Moshi [1], dGSLM [2], and synchronous large language model (LLM) agents [3] encode the user channel into discrete tokens with a streaming residual vector quantization (RVQ) codec [4, 5], model both conversational streams jointly, and speak with sub-second latency. The shift creates an attack surface more dangerous than that of automatic speech recognition (ASR): a full-duplex agent ingests every frame of an always-open microphone, maintains persistent dialogue state an adversary can poison early and exploit late, and converts audio directly into spoken actions without an inspectable transcript. ASR is reliably fooled by inaudible perturbations, via waveform optimization [6], masking-threshold hiding [7, 8], or universal perturbations [9], and speech-capable LLMs inherit these vulnerabilities [10, 11, 12]. Whether, and how cheaply, a full-duplex S2S agent can be protected remains open. This paper gives a twofold answer. First, we port the psychoacoustically masked threat model to the full-duplex setting and show that it is devastating there. The attacker maximizes an objective over additive waveform perturbations confined to a perceptual ball whose per-bin radii derive from the masking threshold of the carrier conversation, under three goals: a targeted semantic hijack forcing an attacker-chosen response, a denial attack muting the agent, and a policy jailbreak delivered through an adversarial audio prefix. Digitally and white-box, these attacks succeed against an undefended Moshi-style agent in 91.7%, 88.4%, and 76.2% of trials, consistent with rates against ASR [6, 8] and speech LLMs [10], and remain effective over the air [13]. Second, and centrally, we propose a defense whose distinguishing property is minimality. Psychoacoustically aligned latent smoothing (PALS), applied during ordinary post-training at the RVQ latent interface, perturbs each pre-quantization latent frame with anisotropic Gaussian noise whose covariance is the local codebook covariance, so smoothing respects the geometry quantization imposes, and re-quantizes through a straight-through estimator so the model always consumes valid codes. This is coupled to a matched input-space augmentation: noise whose spectral envelope is the masking threshold that constrains the attacker, which we show is the maximum-entropy distribution under the threshold-implied covariance constraint and hence the least presumptive model of an attacker confined to the perceptual ball. Training adds a single Kullback–Leibler (KL) consistency term between output distributions under clean and augmented latents, in the spirit of TRADES [14]. One scalar σ scales both augmentations; no attack is run during training and no architecture changes. We separate two operating points and never conflate them. PALS-Train is the deployed system: the post-trained agent runs unmodified, with no inference-time smoothing or cost, and is evaluated empirically; it is the system behind every attack-success number, including the headline hijack rate of 8.3%. PALS-Certify is a Monte Carlo-smoothed variant whose decision is aggregated over n=1,000n=1,000 noise draws at inference; it carries a provable certificate at the smoothing cost, and supplies every certified-accuracy number. The defense comes with theory under explicit assumptions and inline proofs. Adapting randomized smoothing [15] to quantized latents with a fixed certification-time covariance, we show that a sufficient top-two posterior margin keeps the decision of PALS-Certify constant within an ellipsoidal (Mahalanobis) latent radius; because smoothing needs no continuity of the base function, the hard quantizer is certified for free. A TRADES-style decomposition of the smoothed loss bounds adversarial risk by clean risk, the trained consistency term, a coverage gap, and a tail penalty, and a corollary transfers the latent certificate to the masking-weighted norm. Empirically, PALS-Train cuts targeted hijack from 91.7% to 8.3% at matched perceptibility for a 2.3% relative quality drop, dominating Gaussian augmentation, input randomized smoothing [16], projected gradient descent (PGD) adversarial training [17], and a Demucs front-end [18]. The certificate is honestly a small-radius floor: PALS-Certify certifies 47.0% of decisions at latent radius 0.50.5, capped at 0.6160.616 by the protocol and corresponding to Δ≈−26 ≈-26 dB, far below the 00 dB attacks, so it is a guaranteed floor, not a guarantee against the strong attacks. I Related Work Adversarial examples were first characterized for visual classifiers [19]; Carlini and Wagner showed targeted attacks transfer to speech-to-text with near-imperceptible perturbations [6]. Psychoacoustic hiding constrained the perturbation beneath an MPEG-style masking threshold [7], and Qin et al. added expectation over room transformations for imperceptible, physically robust targeted attacks [8]. Universal perturbations [9] and physical playback [13] establish that the threat persists outside the digital sandbox; a systematization appears in [20], and universal segments that mute speech foundation models [21] foreshadow the denial attack we formalize for full-duplex agents. Text-domain adversarial suffixes [22] motivated audio-domain attacks on speech LLMs: SpeechGuard reported white-box jailbreak success near 90% [10], AdvWave introduced stealthy adaptive-target jailbreak optimization [11], voice-mode jailbreaks against commercial assistants exploit narrative framing [12], and benchmark suites standardize such evaluations [23]. These works target half-duplex audio LLMs [24, 25] or proprietary assistants; none formalizes the masked-perturbation attacker for streaming full-duplex S2S models, and none offers a certified defense at the codec latent interface, the gap addressed here. On the defense side, adversarial training with PGD [17] and its TRADES refinement [14] remain the strongest empirical defenses but multiply training cost and demand adaptive evaluation [26, 27], while audio purification, from perceptual input transformations [28] to diffusion purification [29], can be bypassed by attackers that differentiate through the purifier [26]. Certified approaches derive from randomized smoothing [15], with antecedents in differential privacy [30], strengthened by adversarially trained smoothed classifiers [31] and denoised smoothing [32]; sequential randomized smoothing certified ASR transcripts under inaudible-noise budgets at a measurable cost in word error rate (WER) [16]. The use of input-dependent and anisotropic noise is itself an established line: data-dependent smoothing optimizes the noise variance per input [33], ANCER fits per-sample axis-aligned ellipsoidal certificates by volume maximization [34], and center smoothing certifies structured outputs [35]; latent-space smoothing with an orthogonal, known-Lipschitz encoder propagates a latent certificate back to the input [36]. Our certified variant is a direct instance of this anisotropic, latent-space family: Theorem 1 is the standard whitening-to-isotropic-Cohen reduction composed with a Lipschitz pullback. Its genuine novelty is not the certificate machinery but the smoothing law, a psychoacoustically matched covariance tied to the imperceptibility threat model, derived by codebook geometry in the latent space and by a maximum-entropy argument [37] under the masking-threshold covariance in the input space, so the defender’s randomization matches exactly the attacker the masking model permits. Generative spoken dialogue modeling on dual channels began with dGSLM [2]; Moshi unified a text LLM backbone with the streaming Mimi codec into a real-time full-duplex agent [1], building on neural codec language modeling [38, 4, 5]. Synchronous LLMs [3], low-latency cascades [39], and end-to-end chatbots [40] broaden the design space, while benchmarks quantify turn-taking [41] and corruption sensitivity [42]. The dyadic corpora enabling such training [43, 44, 45, 46, 47] also supply the carrier conversations on which imperceptible attacks are mounted. Fig. 1: Threat model. The attacker optimizes a perturbation confined beneath the masking threshold of the carrier speech, delivers it digitally or over the air, and pursues hijack, mute, or jailbreak goals. I Threat Model and Attack Formulation I-A System Under Attack Let x∈ℝTx ^T denote the user-channel waveform at 24 kHz. A streaming convolutional encoder Ec:ℝT→ℝF′×dE_c:R^T ^F × d produces continuous latent frames z=Ec(x)z=E_c(x) at 12.5 Hz with d=512d=512, which an RVQ quantizer Q with eight codebooks of size 2048 maps to codes z^=Q(z) z=Q(z), following the Mimi design [1, 4]. A dialogue model fϑf_ consumes the interleaved user and agent token streams and an inner-monologue text stream; pϑ(⋅∣Q(Ec(x)),h)p_ (· Q(E_c(x)),h) is the distribution over agent responses given history h. A response is summarized by a semantic decision in a finite set Y induced by an external judge (Section VI), with hsemh_sem mapping a latent sequence to the class of the response it elicits. Figure 1 depicts the attack surface. I-B The Perceptual Ball Following [8, 7], let D denote the short-time Fourier transform (STFT) with frames f=1,…,Ff=1,…,F and bins k=1,…,Kk=1,…,K, and let θx(f,k)>0 _x(f,k)>0 be the global masking threshold of carrier x computed by MPEG-1 psychoacoustic model 1 [48, 49]. The attacker chooses δ in the perceptual ball x(η)=δ∈ℝT:|(Dδ)f,k|2≤η2θx(f,k)∀(f,k),P_x(η)= \δ ^T: (Dδ)_f,k ^2≤η^2\, _x(f,k)\;\;∀(f,k) \, (1) equivalently the ball of radius η in the masking-weighted supremum norm ∥δ∥θx,∞=maxf,k|(Dδ)f,k|/θx(f,k) δ _ _x,∞= _f,k (Dδ)_f,k / _x(f,k). We report the budget as Δ=20log10η =20 _10η dB; Δ=0 =0 touches the threshold of audibility and Δ<0 <0 buys margin. Projection onto the convex ball is per-bin magnitude clipping followed by inverse STFT. I-C Attack Objectives The attacker has white-box access to EcE_c, Q, and fϑf_ , full knowledge of the defense, and optimizes with Adam under projection onto (1), gradients passing through Q by the straight-through estimator as required for adaptive evaluation [26, 27]. The targeted semantic hijack forces an attacker-chosen response y⋆y , minδ∈x(η)−logpϑ(y⋆∣Q(Ec(x+δ)),h), _δ _x(η)\;- p_ (y Q(E_c(x+δ)),h ), (2) generalizing targeted ASR attacks [6, 8] to spoken responses. The mute attack drives the agent stream to its silence tokens by minimizing −∑tlogpϑ(s∅(t)∣Q(Ec(x+δ)),h)- _t p_ (s_ ^(t) Q(E_c(x+δ)),h) over δ∈x(η)δ _x(η), with s∅(t)s_ ^(t) the natural-pause code, the full-duplex analogue of muting speech foundation models [21]. The jailbreak attack optimizes a reusable prefix a prepended to harmful spoken requests xjx_j from an AdvBench-derived set [22, 11] toward affirmative openings, mina∈a0(η)∑j−logpϑ(yj+∣Q(Ec(a⊕xj)),h) _a _a_0(η) _j- p_ (y^+_j Q(E_c(a x_j)),h). Over-the-air attacks wrap the objectives in an expectation over room impulse responses and noise [8, 13]. IV Psychoacoustically Aligned Latent Smoothing PALS perturbs the model where the attack must pass, the latent bottleneck of the codec, with a single scalar σ scaling the entire defense; Figure 2 summarizes the pipeline. Fig. 2: The PALS post-training pipeline. The augmented path adds masking-shaped noise in the waveform domain and anisotropic codebook-covariance noise in the latent domain, re-quantizes through a straight-through estimator, and is tied to the clean path by a KL consistency term. One scalar σ governs both augmentations. IV-A Geometry-Aware Latent Augmentation RVQ imposes a strongly anisotropic latent geometry: high intra-cell variance lies along directions where encoder outputs fluctuate, exactly where an attack that survives quantization must cross cell boundaries. During calibration we assign each latent frame to its nearest first-codebook entry cvc_v and accumulate the empirical covariance Σ^v _v of the residuals in cell v. The smoothing covariance for a frame ztz_t in cell v(zt)v(z_t) is the shrunk, trace-normalized matrix Σ(zt)=dtr(Σt′)Σt′,Σt′=(1−ρ)Σ^v(zt)+ρId, (z_t)= dtr( _t) _t, _t=(1-ρ)\, _v(z_t)+ρ I_d, (3) with shrinkage ρ=0.1ρ=0.1 guaranteeing Σ(zt)≻0 (z_t) 0. The augmented latent is z~t=zt+σεt z_t=z_t+σ\, _t, εt∼(0,Σ(zt)) _t (0, (z_t)) per frame, and the model consumes re-quantized codes Q(z~)Q( z) with straight-through gradients, keeping training on the discrete support seen at inference. IV-B The Masking-Matched Input Dual To also train the encoder against threshold-shaped energy, we add a waveform-domain augmentation ν whose STFT coefficients are independent zero-mean circularly symmetric complex Gaussians with threshold-matched variance, (Dν)f,k∼(0,η(σ)2θx(f,k))(Dν)_f,k (0,η(σ)^2\, _x(f,k) ), synthesized by inverse STFT. The gain η(σ)=κση(σ)=κσ is calibrated once so the median latent displacement matches that injected directly in latent space, leaving σ the only free parameter. The masking threshold defines a per-bin expected-power (covariance) budget |(Dν)f,k|2≤η2θx(f,k)E (Dν)_f,k ^2≤η^2 _x(f,k), and Lemma 1 identifies the noise maximally noncommittal under it. This is a covariance constraint, not the pointwise supremum constraint (1) defining the perceptual ball; the Gaussian places mass outside the ball with probability near one, so we relate the two through Corollary 1 rather than as an identity. Lemma 1 (Maximum entropy under the masking-threshold covariance). Among all distributions on ν whose STFT coefficients are independent across bins, zero-mean, and satisfy the expected-power (covariance) constraint |(Dν)f,k|2≤η2θx(f,k)E\, (Dν)_f,k ^2≤η^2 _x(f,k) for all (f,k)(f,k), differential entropy is maximized uniquely by the circularly symmetric complex Gaussian with per-bin variance exactly η2θx(f,k)η^2 _x(f,k), i.e., by the masking-shaped noise used by PALS. Proof. For complex u with |u|2=sE u ^2=s, h(u)≤log(πes)h(u)≤ (π es) with equality iff u∼(0,s)u (0,s). By independence the joint entropy is the sum of per-bin entropies, each bounded by log(πeη2θx(f,k)) (π eη^2 _x(f,k)) and increasing in the variance, so the bound is attained by saturating each covariance constraint with a Gaussian; uniqueness follows from strict concavity. The optimization is over the second moment only, hence a statement about the covariance, not pointwise feasibility in (1). ∎ By Jaynes’s principle [37], a defender who knows only the attacker’s per-bin power budget can do no better than randomize with this least informative distribution, which covers the perceptual ball in the weighted-ℓ2 _2 sense made precise below. TABLE I: Main results at attack budget Δ=0 =0 dB. The attack, clean, and over-the-air columns evaluate the empirically deployed operators (PALS row == PALS-Train, no inference-time smoothing); the final column reports certified semantic accuracy of the Monte Carlo-smoothed PALS-Certify at latent ℓ2 _2 radius 0.5 (inside the protocol ceiling 0.616). Attack rates carry bootstrap 95% confidence half-widths. Defense Hijack ↓ Mute ↓ Jailbreak ↓ OTA hijack ↓ UTMOS ↑ WER ↓ T-F1 ↑ Cert.@0.5 ↑ Undefended 91.7 ± 1.7 88.4 ± 2.0 76.2 ± 3.7 46.3 ± 3.1 3.91 7.4 0.71 — Gaussian augmentation 41.5 ± 3.1 39.8 ± 3.0 33.0 ± 4.1 21.7 ± 2.6 3.78 8.3 0.69 — Demucs front-end [18] 53.7 ± 3.1 49.2 ± 3.1 44.6 ± 4.3 18.9 ± 2.4 3.84 7.9 0.70 — Input smoothing [16] 28.9 ± 2.8 26.1 ± 2.7 24.4 ± 3.7 15.2 ± 2.2 3.62 9.8 0.65 31.5 PGD-AT [17] 19.4 ± 2.5 17.2 ± 2.3 15.8 ± 3.2 9.6 ± 1.8 3.70 8.9 0.69 — PALS (ours, σ=0.25σ=0.25) 8.3 ± 1.7 11.2 ± 1.9 9.1 ± 2.5 4.7 ± 1.3 3.82 7.7 0.70 47.0† † PALS-Certify (inference-time MC smoothing, n=1,000n=1,000); all other columns are PALS-Train. IV-C Training Objective Let ℒpost(ϑ,x)L_post( ;x) be the standard post-training next-token cross-entropy over the agent audio and inner-monologue text streams [1]. Writing z^=Q(Ec(x)) z=Q(E_c(x)) for the clean codes and z~=Q(Ec(x+ν)+σε) z=Q (E_c(x+ν)+σ ) for the doubly augmented codes, PALS minimizes ℒ(ϑ)=[ℒpost(ϑ;x)+βKL(pϑ(⋅∣z^,h)∥pϑ(⋅∣z~,h))],L( )=E [L_post( ;x)+β\,KL (p_ (· z,h)\,\|\,p_ (· z,h) ) ], (4) with β=4β=4, the TRADES coupling [14] computed per emitted token and averaged. No adversarial optimization runs during training, so the overhead is one extra forward pass, roughly 1.4×1.4× wall-clock against 3.4×3.4× for ten-step PGD adversarial training. The trained model defines the deployed operator PALS-Train, which runs unmodified at inference with no added cost and is evaluated empirically. A second operator, PALS-Certify, aggregates the decision over n Monte Carlo draws of ε from a fixed certification-time covariance; it pays an n-fold inference cost and is the object of the certificate analyzed next. The two operators share weights but differ at inference, and we keep their results separate throughout. Fig. 3: (a) PALS-Certify: certified semantic accuracy versus latent ℓ2 _2 radius (isotropic ablation dashed; n=1,000n=1,000 draws, α=10−3α=10^-3). Each σ curve terminates at its own Clopper–Pearson/trace ceiling RΣmax(σ)R_ (σ), 0.6160.616 at σ=0.25σ=0.25; the data come from certify.py, which enforces the cap. (b) PALS-Train: digital white-box hijack success versus budget Δ with bootstrap 95% confidence intervals. (c) PALS-Train: empirical robustness versus clean quality; the σ sweep dominates all baselines, with the knee at σ=0.25σ=0.25. V Certified Robustness Analysis Throughout, Φ is the standard normal cumulative distribution function and Φ−1 ^-1 its inverse. The certificate is a property of PALS-Certify, whose smoothing law is identical at the clean point and every perturbation of it. Definition 1 (PALS-Certify, fixed-covariance smoothing). To certify the decision at a clean latent point z, fix the covariance Σ:=Σ(z)≻0 := (z) 0 at the cell of z (Eq. (3)), or a single globally whitened covariance, and hold it constant over ℝmR^m, drawing ε∼(0,σ2Σ) (0,σ^2 ); the certified operator is g~(w)=argmaxyℙ(h(w+ε)=y) g(w)= _yP(h(w+ )=y). PALS-Train instead reselects a cell-dependent Σ(zt) (z_t) per frame, so its noise law at z differs from that at z+δz+ _z and the smoothing premise fails; freezing Σ is therefore necessary for a valid certificate, and is what the released harness runs. Theorem 1 (Anisotropic smoothing certificate on quantized latents). Let h:ℝm→h:R^m be any measurable function, let Σ≻0 0 be fixed, and let ε∼(0,σ2Σ) (0,σ^2 ) as in Definition 1. Suppose for some class A and bounds pA¯,pB¯∈[0,1] p_A, p_B∈[0,1], ℙ(h(z+ε)=A)≥pA¯≥pB¯≥maxy≠Aℙ(h(z+ε)=y).P(h(z+ )=A)≥ p_A≥ p_B≥ _y≠ AP(h(z+ )=y). (5) Then g~(z+δz)=A g(z+ _z)=A for every latent perturbation δz _z inside the ellipsoid ∥Σ−1/2δz∥2≤σ2(Φ−1(pA¯)−Φ−1(pB¯))=:RΣ. ^-1/2 _z _2≤ σ2 ( ^-1( p_A)- ^-1( p_B) )=:R_ . (6) A Euclidean ball gives the weaker guarantee g~(z+δz)=A g(z+ _z)=A whenever ∥δz∥2≤R:=λmin(Σ)RΣ _z _2≤ R:= _ ( )\,R_ . Proof. Because Σ is fixed, the whitening map W=Σ−1/2W= ^-1/2 does not depend on the evaluation point. With u=Wzu=Wz and hW(u)=h(W−1u)h_W(u)=h(W^-1u), the noise Wε∼(0,σ2I)W (0,σ^2I) is isotropic and hWh_W measurable. The Neyman–Pearson argument of Cohen et al. [15] certifies hWh_W within isotropic radius σ2(Φ−1(pA¯)−Φ−1(pB¯)) σ2( ^-1( p_A)- ^-1( p_B)) around u; a latent perturbation δz _z maps to WδzW _z with ∥Wδz∥2=∥Σ−1/2δz∥2 W _z _2= ^-1/2 _z _2, which is (6), and the ℓ2 _2 form follows from ∥Σ−1/2δz∥2≤∥δz∥2/λmin(Σ) ^-1/2 _z _2≤ _z _2/ _ ( ). The argument needs no continuity of h, so it applies verbatim to h=hsem∘Qh=h_sem Q, absorbing the hard quantizer and decoder into the base function. ∎ The certificate is natively the ellipsoid (6); the ℓ2 _2 ball is its inscribed sphere and the only honest scalar summary under anisotropic noise. Trace normalization fixes trΣ=dtr =d, so λmin(Σ)≤1 _ ( )≤ 1 and R≤RΣR≤ R_ ; anisotropy is genuine gain only inside the ellipsoid, along high-variance codebook directions where re-quantization-surviving perturbations concentrate, worth up to 10 certified points over isotropic smoothing in Figure 3(a). PALS-Certify uses the standard protocol [15]: n0=100n_0=100 candidate draws and n=1,000n=1,000 estimation draws give a one-sided Clopper–Pearson lower bound pA¯ p_A at confidence 1−α1-α, α=10−3α=10^-3, with pB¯=1−pA¯ p_B=1- p_A, abstaining when pA¯≤1/2 p_A≤ 1/2. Because pA¯ p_A cannot exceed its value pA¯max≈0.9931 p_A_ ≈ 0.9931 at k=nk=n and Φ−1(pB¯)=−Φ−1(pA¯) ^-1( p_B)=- ^-1( p_A), the maximal Mahalanobis radius is RΣmax(σ)=σΦ−1(pA¯max)R_ (σ)=σ\, ^-1( p_A_ ), i.e. 0.6160.616 at σ=0.25σ=0.25, and λmin(Σ)≤1 _ ( )≤ 1 caps the ℓ2 _2 radius at the same value. Each σ curve in Figure 3(a) therefore terminates at its ceiling RΣmax(σ)R_ (σ), larger σ reaching farther, with curve data from the released certify.py, which enforces the cap. The certificate is thus a guaranteed floor, not a guarantee against the strong attacks: at σ=0.25σ=0.25 it reaches latent radius 0.6160.616 and an input budget of only Δ≈−26 ≈-26 dB, far below the [−20,0][-20,0] dB attacks of Section I. As is standard in the certified-defense literature, we therefore report two separated pillars: the provable floor of Theorem 1 and the empirical robustness of PALS-Train across the full range, which extends far past it. TABLE I: Empirical hijack success (%) of PALS-Train by condition and slice (left) and PALS-Train ablations with σ sweep (right), at Δ=0 =0 dB. Condition Undef. PGD-AT PALS Digital white-box 91.7 19.4 8.3 OTA, small room (T60≈0.2T_60\!≈\!0.2 s) 52.8 11.0 5.4 OTA, medium room (T60≈0.5T_60\!≈\!0.5 s) 46.3 9.6 4.7 OTA, medium ++ babble 10 dB 29.4 6.2 2.9 Female / male (digital) 92.4/90.9 20.6/18.3 8.9/7.8 Configuration Hijack ↓ OTA ↓ UTMOS ↑ PALS, full (σ=0.25σ=0.25) 8.3 4.7 3.82 isotropic covariance (Σ=I =I) 13.6 7.5 3.80 without input-space dual 14.9 9.8 3.85 without KL consistency 12.7 6.9 3.74 without re-quantization 10.9 5.8 3.69 σ=0.0625σ=0.0625 / 0.1250.125 21.5 / 14.2 11.6 / 7.9 3.90 / 3.88 Theorem 2 (Risk decomposition for the smoothed loss). Fix x and history h. Let ℓ(x,δ)=r∼pϑ(⋅∣Q(Ec(x+δ)),h)[φ(r)] (x,δ)=E_r p_ (· Q(E_c(x+δ)),h)[ (r)] for a per-response cost φ∈[0,1] ∈[0,1], and define the Gaussian-smoothed loss ℓσ(x,δ)=ξ∼(0,σ2Σ)[ℓ(x,δ+ξ)] _σ(x,δ)=E_ξ (0,σ^2 )[ (x,δ+ξ)]. Then δ↦ℓσ(x,δ)δ _σ(x,δ) is Λσ _σ-Lipschitz in ∥⋅∥2 · _2 with Λσ=2/π/(σλmin(Σ)) _σ= 2/π\,/(σ _ ( )), even though ℓ is discontinuous through the hard quantizer. For any augmentation μx _x and any target budget η, supδ∈x(η)ℓσ(x,δ)≤ℓσ(x,0)+νKLx(ν)/2+Λσν∥ν−δx⋆∥2+τx, split _δ _x(η) _σ(x,δ)≤& _σ(x,0)+E_ν KL_x(ν)/2\\ &+ _σ\,E_ν ν-δ _x _2+ _x, split (7) where KLx(ν)=KL(pϑ(⋅∣z^,h)∥pϑ(⋅∣Q(Ec(x+ν)),h))KL_x(ν)=KL(p_ (· z,h)\,\|\,p_ (· Q(E_c(x+ν)),h)), δx⋆δ _x attains the supremum, and τx=ℙν∼μx(ν∉x(η)) _x=P_ν _x(ν _x(η)) is the tail mass that μx _x places outside the perceptual ball. Proof. Lipschitzness is the standard Gaussian-smoothing bound: for f:ℝm→[0,1]f:R^m→[0,1] and isotropic ξ∼(0,σ2I)ξ (0,σ^2I), f¯(⋅)=[f(⋅+ξ)] f(·)=E[f(·+ξ)] has ∥∇f¯∥2≤2/π/σ ∇ f _2≤ 2/π/σ by differentiating under the integral and bounding ∥ξ∥E ξ -type terms (Stein’s identity), independent of any continuity of f [31]. Whitening by Σ−1/2 ^-1/2 replaces σ by σλmin(Σ)σ _ ( ) in the worst ℓ2 _2 direction, giving Λσ _σ. For the decomposition, write ℓσ(x,δ⋆)−ℓσ(x,0)=[ℓσ(x,δ⋆)−νℓσ(x,ν)]+[νℓσ(x,ν)−ℓσ(x,0)] _σ(x,δ )- _σ(x,0)=[ _σ(x,δ )-E_ν _σ(x,ν)]+[E_ν _σ(x,ν)- _σ(x,0)]. The first bracket is at most Λσν∥ν−δ⋆∥2 _σ\,E_ν ν-δ _2 by Lipschitzness. For the second, φ∈[0,1] ∈[0,1] gives ℓσ(x,ν)−ℓσ(x,0)≤TV≤KLx(ν)/2 _σ(x,ν)- _σ(x,0) ≤ KL_x(ν)/2 by Pinsker wherever ν∈x(η)ν _x(η), while the event ν∉x(η)ν _x(η) contributes at most its probability τx _x, giving (7). ∎ The decomposition explains each design choice: the first term is the post-training loss; the second is, up to Jensen, exactly the KL consistency term of (4); the third shrinks as μx _x covers the ball, served by the maximum-entropy choice of Lemma 1; and τx _x is the explicit penalty for the Gaussian tail outside the ball, maintaining mathematical rigor without falsely assuming on-ball Gaussian support or τx=0 _x=0. Corollary 1 (Weighted-norm transfer of the latent certificate). Let the STFT use a tight frame with frame bound AwA_w. Then x(η)⊆δ:∥δ∥2≤ηρxP_x(η) \δ: δ _2≤η\, _x\ with ρx2=Aw−1∑f,kθx(f,k) _x^2=A_w^-1 _f,k _x(f,k), so the perceptual ball sits inside a masking-weighted ℓ2 _2 ball. Consequently a latent certificate of radius RΣR_ transfers to the weighted norm: every threshold-compliant perturbation whose induced latent displacement satisfies ∥Σ−1/2δz∥2≤RΣ ^-1/2 _z _2≤ R_ leaves the decision of PALS-Certify invariant, a guarantee holding in the weighted-ℓ2 _2 sense rather than pointwise in the supremum norm (1). Proof. By the frame inequality, Aw∥δ∥22≤∑f,k|(Dδ)f,k|2≤η2∑f,kθx(f,k)A_w δ _2^2≤ _f,k (Dδ)_f,k ^2≤η^2 _f,k _x(f,k), which gives the containment; the transfer is then Theorem 1 applied to the induced latent displacement δz=Ec(x+δ)−Ec(x) _z=E_c(x+δ)-E_c(x). ∎ The pullback to an input-space radius is heuristic, not a hard guarantee. Assumption 1 (Estimated local encoder Lipschitz constant). There is L^>0 L>0 with ∥Ec(x+δ)−Ec(x)∥2≤L^∥δ∥2 E_c(x+δ)-E_c(x) _2≤ L δ _2 for held-out x and δ with x+δx+δ in x+x(1)\x+P_x(1)\; this is an empirical estimate, not a certified global bound. Under Assumption 1, a latent radius R maps to a waveform radius R/L^R/ L. We obtain L L by power iteration on encoder Jacobian-vector products [50, 51] over 10,00010,000 held-out five-second segments, finding median 2.42.4 and 95th-percentile 3.73.7, the latter used for reported budgets. Because L L is an empirical percentile of a local estimate, the input radius (and the Δ≈−26 ≈-26 dB figure) is heuristic; the hard certificate lives in the latent space, and a spectral-norm-constrained encoder making the bound architectural is deferred to Section VII-C. VI Experimental Setup VI-A Model, Data, and Training The defended system is a Moshi-style agent [1]: a 7B-parameter temporal transformer with a depth transformer over RVQ tokens, a Mimi-style codec at 12.5 Hz with eight codebooks, and an inner-monologue text stream. From a pretrained checkpoint we post-train on 3,118 h of dual-channel conversational English from five sources: 141 h from otoSpeech-processed [52, 47] (48 kHz stereo FLAC, C BY 4.0); 727 h of spontaneous dyadic English from SSSD [44]; ≈ 850 h across 1,656 video calls from CANDOR [43] (with demographic metadata); a 1,300 h subset of Seamless Interaction [45]; and the 100 h headset partition of AMI [46] (Whisper large-v3 attains ≈ 16% WER [53]). Audio is resampled to 24 kHz, one speaker per channel. Post-training runs 80k AdamW steps (learning rate 2×10−52× 10^-5, cosine decay, weight decay 0.1, global batch 64 five-minute segments) on 32 H100 GPUs for ≈ 5 days, σ ramping over the first 10k steps. PALS sets σ=0.25σ=0.25, β=4β=4, ρ=0.1ρ=0.1, covariances calibrated on 20 h of held-out audio. VI-B Attack Suite, Metrics, and Baselines Attacks follow Section I: 1,000 Adam iterations (rate 10−310^-3) with projection onto x(η)P_x(η), five restarts, and Δ∈−20,−15,−10,−5,0 ∈\-20,-15,-10,-5,0\ dB; masking thresholds use MPEG-1 model 1 on 2048-sample Hann windows with hop 512 [8]. Hijack uses 200 attacker-chosen responses over 1,000 held-out carriers; mute succeeds when agent activity stays below 5% of frames for 10 s; jailbreak uses 520 AdvBench-derived requests [22] with a 3 s universal prefix per category [11, 23]. Every judged quantity (including the 9-way response-intent taxonomy Y) uses Gemini 2.5 Pro [54] (temperature 0, 3 samples by majority vote, Cohen’s κ=0.88κ=0.88 [55] on a 200-item audit). Simulated over-the-air (OTA) evaluation convolves attacks with pyroomacoustics impulse responses [56] over 120 rooms (T60∈[0.2,0.9]T_60∈[0.2,0.9] s) with MUSAN babble [57] at 10–20 dB under expectation over transforms. Clean metrics use UTMOS [58], DNSMOS [59], Whisper large-v3 WER [53], turn-taking F1 on Full-Duplex-Bench [41], and ITU-T P.808 [60]; intervals use 10410^4 bootstrap resamples. Baselines comprise the undefended agent, Gaussian augmentation at matched power, input-space randomized smoothing [16], PGD adversarial training [17], and a Demucs front-end [18], all attacked adaptively [27, 26]. VII Results VII-A Main Comparison Table I reports the central comparison at the full budget; the attack columns evaluate the deployed PALS-Train. The undefended agent is broken almost at will (91.7% hijack, 88.4% mute, 76.2% jailbreak, 46.3% over the air). Baselines help at a price: input smoothing costs quality and turn-taking [16], the purification front-end collapses under the adaptive attacker [26], and PGD is strongest but triples training cost. PALS-Train attains 8.3% hijack, 11.2% mute, and 9.1% jailbreak, 1.5×1.5× to 2.3×2.3× lower than PGD, at clean UTMOS 3.82 (an absolute decrease of 0.09 MOS points, within 2.3% of 3.91), WER up only to 7.7%, and turn-taking F1 unchanged. Figure 3(b) holds PALS-Train below 9% from −20-20 to 00 dB, its margin over PGD widening with the budget, well beyond the certified radius reported next. VII-B Certified Robustness Figure 3(a) shows certified accuracy of PALS-Certify, each σ curve terminating at its ceiling (0.6160.616 at σ=0.25σ=0.25); 47.0% of decisions are certified at radius 0.5<0.6160.5<0.616, a floor at Δ≈−26 ≈-26 dB covering none of the demonstrated attacks. The anisotropy is not cosmetic: isotropic smoothing certifies up to 10 points fewer over most of the range, narrowing toward the ceiling (dashed), and input smoothing only 31.5% (Table I). Crucially, the mute attack is evaluated purely mechanically (audio activity <5%<5\% of frames for 10 s) without any LLM judge; mute drops from 88.4±2.0%88.4± 2.0\% to 11.2±1.9%11.2± 1.9\%, verifying that the ∼ 10-fold gain is not an artifact of LLM judge bias. VII-C Ablations, Slices, and Limitations Table I (left) shows reverberation and babble degrade the attack more than the defense (PALS-Train at 2.9% under the hardest condition), with negligible speaker-sex and corpus spreads. Table I (right) dissects the defense and isolates gains relative to TRADES-style consistency. Removing the KL term raises hijack from 8.3% to 12.7%, testing generic consistency. However, isotropic covariance (Σ=I =I) yields 13.6% and removing the input masking dual yields 14.9% (doubling simulated OTA success), while PGD-AT reaches 19.4%. This confirms that neither consistency nor adversarial training alone explains the gain; the psychoacoustically shaped input dual and codebook covariance geometry are the primary drivers. Removing re-quantization degrades clean UTMOS to 3.69. Three limitations remain: the hard certificate is latent-space, with its input-space pullback heuristic through an estimated local Lipschitz constant (spectral-norm constraints [51] left to future work); the certified floor transfers in the weighted-ℓ2 _2 sense (Corollary 1); and corpora are English dyadic conversations. VIII Conclusion Psychoacoustically aligned latent smoothing answers imperceptible attacks on full-duplex S2S dialogue models at the codec bottleneck every attack must cross. PALS-Train cuts attack success by roughly an order of magnitude at a 2.3% quality cost with no inference-time overhead, while PALS-Certify supplies a provable anisotropic floor the empirical robustness far exceeds. Acknowledgment The authors disclose that Claude Opus 4.8 was used for editing and rewriting all sections (Abstract, Introduction, Related Work, Methods, Experiments, Results, Conclusion) to improve the flow and presentation, and for assisting in implementation of the code. In addition, Gemini nano banana pro was used for generating diagrams (Figures 1 and 2). References [1] A. Défossez, L. Mazaré, M. Orsini, A. Royer, P. Pérez, H. Jégou, E. Grave, and N. Zeghidour (2024) Moshi: a speech-text foundation model for real-time dialogue. Note: arXiv preprint arXiv:2410.00037 Cited by: §I, §I, §I-A, §IV-C, §VI-A. [2] T. A. Nguyen, E. Kharitonov, J. Copet, Y. Adi, W. Hsu, A. Elkahky, P. Tomasello, R. Algayres, B. Sagot, A. Mohamed, and E. Dupoux (2023) Generative spoken dialogue language modeling. Transactions of the Association for Computational Linguistics 11, p. 250–266. Cited by: §I, §I. [3] B. Veluri, B. N. Peloquin, B. Yu, H. Gong, and S. Gollakota (2024) Beyond turn-based interfaces: synchronous LLMs as full-duplex dialogue agents. In Proc. Conf. Empirical Methods in Natural Language Processing (EMNLP), p. 21390–21402. Cited by: §I, §I. [4] N. Zeghidour, A. Luebs, A. Omran, J. Skoglund, and M. Tagliasacchi (2022) SoundStream: an end-to-end neural audio codec. IEEE/ACM Transactions on Audio, Speech, and Language Processing 30, p. 495–507. Cited by: §I, §I, §I-A. [5] A. Défossez, J. Copet, G. Synnaeve, and Y. Adi (2023) High fidelity neural audio compression. Transactions on Machine Learning Research. Cited by: §I, §I. [6] N. Carlini and D. Wagner (2018) Audio adversarial examples: targeted attacks on speech-to-text. In Proc. IEEE Security and Privacy Workshops (SPW), p. 1–7. Cited by: §I, §I, §I, §I-C. [7] L. Schönherr, K. Kohls, S. Zeiler, T. Holz, and D. Kolossa (2019) Adversarial attacks against automatic speech recognition systems via psychoacoustic hiding. In Proc. Network and Distributed System Security Symposium (NDSS), Cited by: §I, §I, §I-B. [8] Y. Qin, N. Carlini, G. Cottrell, I. Goodfellow, and C. Raffel (2019) Imperceptible, robust, and targeted adversarial examples for automatic speech recognition. In Proc. Int. Conf. Machine Learning (ICML), p. 5231–5240. Cited by: §I, §I, §I, §I-B, §I-C, §VI-B. [9] P. Neekhara, S. Hussain, P. Pandey, S. Dubnov, J. McAuley, and F. Koushanfar (2019) Universal adversarial perturbations for speech recognition systems. In Proc. Interspeech, p. 481–485. Cited by: §I, §I. [10] R. Peri, S. M. Jayanthi, S. Ronanki, A. Bhatia, K. Mundnich, S. Dingliwal, N. Das, Z. Hou, G. Huybrechts, S. Vishnubhotla, D. Garcia-Romero, S. Srinivasan, K. Han, and K. Kirchhoff (2024) SpeechGuard: exploring the adversarial robustness of multi-modal large language models. In Findings of the Association for Computational Linguistics: ACL, p. 10018–10035. Cited by: §I, §I, §I. [11] M. Kang, C. Xu, and B. Li (2025) AdvWave: stealthy adversarial jailbreak attack against large audio-language models. In Proc. Int. Conf. Learning Representations (ICLR), Cited by: §I, §I, §I-C, §VI-B. [12] X. Shen, Y. Wu, M. Backes, and Y. Zhang (2024) Voice jailbreak attacks against GPT-4o. Note: arXiv preprint arXiv:2405.19103 Cited by: §I, §I. [13] H. Yakura and J. Sakuma (2019) Robust audio adversarial example for a physical attack. In Proc. Int. Joint Conf. Artificial Intelligence (IJCAI), p. 5334–5341. Cited by: §I, §I, §I-C. [14] H. Zhang, Y. Yu, J. Jiao, E. P. Xing, L. E. Ghaoui, and M. I. Jordan (2019) Theoretically principled trade-off between robustness and accuracy. In Proc. Int. Conf. Machine Learning (ICML), p. 7472–7482. Cited by: §I, §I, §IV-C. [15] J. Cohen, E. Rosenfeld, and J. Z. Kolter (2019) Certified adversarial robustness via randomized smoothing. In Proc. Int. Conf. Machine Learning (ICML), p. 1310–1320. Cited by: §I, §I, §V, §V. [16] R. Olivier and B. Raj (2021) Sequential randomized smoothing for adversarially robust speech recognition. In Proc. Conf. Empirical Methods in Natural Language Processing (EMNLP), p. 6372–6386. Cited by: §I, §I, TABLE I, §VI-B, §VII-A. [17] A. Madry, A. Makelov, L. Schmidt, D. Tsipras, and A. Vladu (2018) Towards deep learning models resistant to adversarial attacks. In Proc. Int. Conf. Learning Representations (ICLR), Cited by: §I, §I, TABLE I, §VI-B. [18] A. Défossez, G. Synnaeve, and Y. Adi (2020) Real time speech enhancement in the waveform domain. In Proc. Interspeech, p. 3291–3295. Cited by: §I, TABLE I, §VI-B. [19] C. Szegedy, W. Zaremba, I. Sutskever, J. Bruna, D. Erhan, I. Goodfellow, and R. Fergus (2014) Intriguing properties of neural networks. In Proc. Int. Conf. Learning Representations (ICLR), Cited by: §I. [20] H. Abdullah, K. Warren, V. Bindschaedler, N. Papernot, and P. Traynor (2021) SoK: the faults in our ASRs: an overview of attacks against automatic speech recognition and speaker identification systems. In Proc. IEEE Symposium on Security and Privacy (SP), p. 730–747. Cited by: §I. [21] V. Raina, R. Ma, C. McGhee, K. Knill, and M. Gales (2024) Muting Whisper: a universal acoustic adversarial attack on speech foundation models. In Proc. Conf. Empirical Methods in Natural Language Processing (EMNLP), Cited by: §I, §I-C. [22] A. Zou, Z. Wang, N. Carlini, M. Nasr, J. Z. Kolter, and M. Fredrikson (2023) Universal and transferable adversarial attacks on aligned language models. Note: arXiv preprint arXiv:2307.15043 Cited by: §I, §I-C, §VI-B. [23] Z. Peng, Y. Liu, Z. Sun, M. Li, Z. Luo, J. Zheng, W. Dong, X. He, X. Wang, Y. Xue, S. Xu, and X. Huang (2026) JALMBench: benchmarking jailbreak vulnerabilities in audio language models. In Proc. Int. Conf. Learning Representations (ICLR), Cited by: §I, §VI-B. [24] C. Tang, W. Yu, G. Sun, X. Chen, T. Tan, W. Li, L. Lu, Z. Ma, and C. Zhang (2024) SALMONN: towards generic hearing abilities for large language models. In Proc. Int. Conf. Learning Representations (ICLR), Cited by: §I. [25] Y. Chu, J. Xu, Q. Yang, H. Wei, X. Wei, Z. Guo, Y. Leng, Y. Lv, J. He, J. Lin, C. Zhou, and J. Zhou (2024) Qwen2-Audio technical report. Note: arXiv preprint arXiv:2407.10759 Cited by: §I. [26] A. Athalye, N. Carlini, and D. Wagner (2018) Obfuscated gradients give a false sense of security: circumventing defenses to adversarial examples. In Proc. Int. Conf. Machine Learning (ICML), p. 274–283. Cited by: §I, §I-C, §VI-B, §VII-A. [27] F. Tramèr, N. Carlini, W. Brendel, and A. Madry (2020) On adaptive attacks to adversarial example defenses. In Advances in Neural Information Processing Systems (NeurIPS), Cited by: §I, §I-C, §VI-B. [28] S. Hussain, P. Neekhara, S. Dubnov, J. McAuley, and F. Koushanfar (2021) WaveGuard: understanding and mitigating audio adversarial examples. In Proc. USENIX Security Symposium, p. 2273–2290. Cited by: §I. [29] S. Wu, J. Wang, W. Ping, W. Nie, and C. Xiao (2023) Defending against adversarial audio via diffusion model. In Proc. Int. Conf. Learning Representations (ICLR), Cited by: §I. [30] M. Lecuyer, V. Atlidakis, R. Geambasu, D. Hsu, and S. Jana (2019) Certified robustness to adversarial examples with differential privacy. In Proc. IEEE Symposium on Security and Privacy (SP), p. 656–672. Cited by: §I. [31] H. Salman, J. Li, I. Razenshteyn, P. Zhang, H. Zhang, S. Bubeck, and G. Yang (2019) Provably robust deep learning via adversarially trained smoothed classifiers. In Advances in Neural Information Processing Systems (NeurIPS), Cited by: §I, §V. [32] H. Salman, M. Sun, G. Yang, A. Kapoor, and J. Z. Kolter (2020) Denoised smoothing: a provable defense for pretrained classifiers. In Advances in Neural Information Processing Systems (NeurIPS), Cited by: §I. [33] M. Alfarra, A. Bibi, P. H. S. Torr, and B. Ghanem (2022) Data dependent randomized smoothing. In Proc. Conf. Uncertainty in Artificial Intelligence (UAI), p. 64–74. Cited by: §I. [34] F. Eiras, M. Alfarra, M. P. Kumar, P. H. S. Torr, P. K. Dokania, B. Ghanem, and A. Bibi (2021) ANCER: anisotropic certification via sample-wise volume maximization. Note: arXiv preprint arXiv:2107.04570 Cited by: §I. [35] A. Kumar and T. Goldstein (2021) Center smoothing: certified robustness for networks with structured outputs. In Advances in Neural Information Processing Systems (NeurIPS), Cited by: §I. [36] H. Zeng, J. Su, and F. Huang (2021) Certified defense via latent space randomized smoothing with orthogonal encoders. Note: arXiv preprint arXiv:2108.00491 Cited by: §I. [37] E. T. Jaynes (1957) Information theory and statistical mechanics. Physical Review 106 (4), p. 620–630. Cited by: §I, §IV-B. [38] Z. Borsos, R. Marinier, D. Vincent, E. Kharitonov, O. Pietquin, M. Sharifi, D. Roblek, O. Teboul, D. Grangier, M. Tagliasacchi, and N. Zeghidour (2023) AudioLM: a language modeling approach to audio generation. IEEE/ACM Transactions on Audio, Speech, and Language Processing 31, p. 2523–2533. Cited by: §I. [39] Q. Fang, S. Guo, Y. Zhou, Z. Ma, S. Zhang, and Y. Feng (2025) LLaMA-Omni: seamless speech interaction with large language models. In Proc. Int. Conf. Learning Representations (ICLR), Cited by: §I. [40] A. Zeng, Z. Du, M. Liu, K. Wang, S. Jiang, L. Zhao, Y. Dong, and J. Tang (2024) GLM-4-Voice: towards intelligent and human-like end-to-end spoken chatbot. Note: arXiv preprint arXiv:2412.02612 Cited by: §I. [41] G. Lin, J. Lian, T. Li, Q. Wang, G. Anumanchipalli, A. H. Liu, and H. Lee (2025) Full-duplex-bench: a benchmark to evaluate full-duplex spoken dialogue models on turn-taking capabilities. Note: arXiv preprint arXiv:2503.04721 Cited by: §I, §VI-B. [42] M. A. Shah, D. S. Noguero, M. A. Heikkilä, B. Raj, and N. Kourtellis (2025) Speech robust bench: a robustness benchmark for speech recognition. In Proc. Int. Conf. Learning Representations (ICLR), Cited by: §I. [43] A. Reece, G. Cooney, P. Bull, C. Chung, B. Dawson, C. Fitzpatrick, T. Glazer, D. Knox, A. Liebscher, and S. Marin (2023) The CANDOR corpus: insights from a large multimodal dataset of naturalistic conversation. Science Advances 9 (13), p. eadf3197. Cited by: §I, §VI-A. [44] Z. Sheikh, S. Shimizu, S. Arora, J. Shi, S. Cornell, X. Li, and S. Watanabe (2025) Scalable spontaneous speech dataset (SSSD): crowdsourcing data collection to promote dialogue research. In Proc. Interspeech, Cited by: §I, §VI-A. [45] V. Agrawal et al. (2025) Seamless interaction: dyadic audiovisual motion modeling and large-scale dataset. Note: arXiv preprint arXiv:2506.22554 Cited by: §I, §VI-A. [46] J. Carletta, S. Ashby, S. Bourban, M. Flynn, M. Guillemot, T. Hain, J. Kadlec, V. Karaiskos, W. Kraaij, M. Kronenthal, G. Lathoud, M. Lincoln, A. Lisowska, I. McCowan, W. Post, D. Reidsma, and P. Wellner (2005) The AMI meeting corpus: a pre-announcement. In Proc. Int. Workshop on Machine Learning for Multimodal Interaction (MLMI), p. 28–39. Cited by: §I, §VI-A. [47] otoearth (2025) otoSpeech-full-duplex-280h: 280 hours of raw english full-duplex dyadic conversation. Note: Hugging Face dataset, otoearth/otoSpeech-full-duplex-280h, C BY 4.0 Cited by: §I, §VI-A. [48] T. Painter and A. Spanias (2000) Perceptual coding of digital audio. Proceedings of the IEEE 88 (4), p. 451–515. Cited by: §I-B. [49] H. Fastl and E. Zwicker (2007) Psychoacoustics: facts and models. 3rd edition, Springer, Berlin, Germany. Cited by: §I-B. [50] A. Virmaux and K. Scaman (2018) Lipschitz regularity of deep neural networks: analysis and efficient estimation. In Advances in Neural Information Processing Systems (NeurIPS), Cited by: §V. [51] T. Miyato, T. Kataoka, M. Koyama, and Y. Yoshida (2018) Spectral normalization for generative adversarial networks. In Proc. Int. Conf. Learning Representations (ICLR), Cited by: §V, §VII-C. [52] otoearth (2025) otoSpeech-full-duplex-processed-141h: cleaned full-duplex dyadic conversation. Note: Hugging Face dataset, otoearth/otoSpeech-full-duplex-processed-141h, C BY 4.0 Cited by: §VI-A. [53] A. Radford, J. W. Kim, T. Xu, G. Brockman, C. McLeavey, and I. Sutskever (2023) Robust speech recognition via large-scale weak supervision. In Proc. Int. Conf. Machine Learning (ICML), p. 28492–28518. Cited by: §VI-A, §VI-B. [54] G. Comanici et al. (2025) Gemini 2.5: pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. Note: arXiv preprint arXiv:2507.06261 Cited by: §VI-B. [55] J. Cohen (1960) A coefficient of agreement for nominal scales. Educational and Psychological Measurement 20 (1), p. 37–46. Cited by: §VI-B. [56] R. Scheibler, E. Bezzam, and I. Dokmanić (2018) Pyroomacoustics: a Python package for audio room simulation and array processing algorithms. In Proc. IEEE Int. Conf. Acoustics, Speech and Signal Processing (ICASSP), p. 351–355. Cited by: §VI-B. [57] D. Snyder, G. Chen, and D. Povey (2015) MUSAN: a music, speech, and noise corpus. Note: arXiv preprint arXiv:1510.08484 Cited by: §VI-B. [58] T. Saeki, D. Xin, W. Nakata, T. Koriyama, S. Takamichi, and H. Saruwatari (2022) UTMOS: UTokyo-SaruLab system for VoiceMOS challenge 2022. In Proc. Interspeech, p. 4521–4525. Cited by: §VI-B. [59] C. K. A. Reddy, V. Gopal, and R. Cutler (2022) DNSMOS P.835: a non-intrusive perceptual objective speech quality metric to evaluate noise suppressors. In Proc. IEEE Int. Conf. Acoustics, Speech and Signal Processing (ICASSP), p. 886–890. Cited by: §VI-B. [60] International Telecommunication Union (2021) Subjective evaluation of speech quality with a crowdsourcing approach. Note: Recommendation ITU-T P.808, International Telecommunication Union, Geneva, Switzerland Cited by: §VI-B.