Paper deep dive
NOPE-HYPE: A Structured Simulation Workflow for Robust Speech-to-Text Across Diverse Acoustic Environments
Niramay M. Patel, Bibek Behera, Raksha Sharma
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 93%
Last extracted: 9/10/2026, 3:41:02 AM
Summary
The paper introduces NOPE-HYPE, a structured simulation workflow for improving robust speech-to-text (S2TT) translation across diverse acoustic environments. It utilizes a controllable environment simulator (SIM) that generates noise based on Power Spectral Density (PSD) templates, envelope dynamics, and transient activity. The workflow includes environment reduction via prototype selection and a hyperparameter search to identify optimal simulator configurations. Experiments on Whisper and SeamlessM4T models demonstrate that simulator-generated noise achieves performance comparable to balanced real-noise training, outperforming simple Gaussian or pink noise augmentations.
Entities (10)
Relation Signals (8)
NOPE-HYPE → evaluatedon → Whisper
confidence 95% · We evaluate robustness on Whisper [13] and SeamlessM4T [14]
NOPE-HYPE → evaluatedon → SeamlessM4T
confidence 95% · We evaluate robustness on Whisper [13] and SeamlessM4T [14]
NOPE-HYPE → uses → SIM
confidence 95% · NOPE-HYPE is a structured workflow... combines a controllable environment simulator... SIM: A Controllable Environment Audio Simulator
SIM → basedon → Power Spectral Density
confidence 92% · The simulator supports robust training... matching stable statistical properties... PSD estimation and PSD-matched bed.
NOPE-HYPE → appliedto → English to Hindi
confidence 90% · We test our framework on two language pairs: English to Hindi and English to German.
NOPE-HYPE → appliedto → English to German
confidence 90% · We test our framework on two language pairs: English to Hindi and English to German.
SIM → generates → simulator-generated noise
confidence 90% · We show that simulator-generated noise achieves performance comparable to balanced realnoise training
SIM → usesdatafrom → DEMAND
confidence 85% · We use real environmental recordings from the DEMAND dataset... to match the stable spectral and temporal properties of real environments
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Robust speech-to-text translation systems should perform reliably across diverse acoustic conditions, yet practical pipelines lack controllable tools for systematic environment exploration. Large speech models remain sensitive to unseen acoustic conditions, as training data rarely cover the full range of real this http URL present NOPEHYPE, a structured training workflow that combines a controllable environment simulator, coverage-optimal environment reduction on Power Spectral Density (PSD) templates, and a small, interpretable hyperparameter search over simulator knobs. We show that simulator-generated noise achieves performance comparable to balanced realnoise training across Whisper and SeamlessM4T models, provide principled environment prototype sets, and identify practical default simulator configurations from a structured 27-run hyperparameter sweep.
Tags
Links
- Source: https://arxiv.org/abs/2609.10058v1
- Canonical: https://arxiv.org/abs/2609.10058v1
Trouble viewing inline? Open PDF directly →
Full Text
44,825 characters extracted from source content.
Expand or collapse full text
NOPE-HYPE: A Structured Simulation Workflow for Robust Speech-to-Text Across Diverse Acoustic Environments Niramay M. Patel Affiliation: IISER Bhopal, India niramay23@iiserb.ac.in Bibek Behera Affiliation: IIT Bombay, India bibek.iitkgp@gmail.com Raksha Sharma Affiliation: IIT Roorkee, India raksha.sharma@cs.iitr.ac.in Abstract Robust speech-to-text translation systems should perform reliably across diverse acoustic conditions, yet practical pipelines lack controllable tools for systematic environment exploration. Large speech models remain sensitive to unseen acoustic conditions, as training data rarely cover the full range of real environments. We present NOPE-HYPE, a structured training workflow that combines a controllable environment simulator, coverage-optimal environment reduction on Power Spectral Density (PSD) templates, and a small, interpretable hyperparameter search over simulator knobs. We show that simulator-generated noise achieves performance comparable to balanced real-noise training across Whisper and SeamlessM4T models, provide principled environment prototype sets, and identify practical default simulator configurations from a structured 27-run hyperparameter sweep. Index Terms: speech recognition, noise augmentation, acoustic robustness, environment simulation, hyperparameter search I Introduction Speech-to-text (S2T) translation systems often operate outside curated studio audio settings. The audio of the deployment often includes transportation noise, indoor hums, transient events, and speech-like interference from nearby people. Even strong speech foundation models can degrade under such shifts because training data and augmentation choices rarely cover the full space of real acoustic variability in a controlled way. As shown in Table I, models trained only on clean speech perform worse than those trained with noise injection. A common mitigation is noise augmentation. In practice, this collapses into a single axis, such as Gaussian noise, a single colored-noise type, or a narrow SNR range [1, 2]. These choices are convenient, but they do not isolate factors that matter in real environments — the spectral shape differs by scene, energy changes over time, transient activity varies, and human activity adds interference that overlaps with speech. Learned audio generators can increase realism, but their latent factors are often hard to interpret and sweep independently, limiting their usefulness for a targeted robustness search [7, 8, 9]. In this paper, we shift the focus from adding arbitrary noise to systematically modeling environment-specific noise to build an S2T system that performs well under both clean and noisy conditions. We propose NOPE-HYPE, a practical workflow designed to improve systematic robustness through interpretable control and structured search. We test our framework on two language pairs: English to Hindi and English to German. The workflow has three parts: • SIMULATOR (SIM): A parameterized simulator for environmental noise and noise-in-speech mixtures. Rather than aiming for exactly realistic noise, it enables structured modeling of acoustic factors relevant to speech-to-text performance. • Environment Reduction: When many environmental categories exist, uniform sampling becomes computationally expensive because each environment requires a full hyperparameter sweep of the simulator. We summarize each environment using a PSD template and select a small subset of prototypes that covers the full set under explicit distances. • Hyperparameter Search: We perform a sequential and interpretable search over simulator parameters that yield a robust default configuration and clarify which acoustic factors strongly influence model performance. The paper is organized as follows. Section I reviews robust speech recognition and augmentation. Section I summarizes the NOPE-HYPE workflow, including the controllable noise simulator (SIM) and environment reduction. Section VI describes experiments evaluating Whisper and SeamlessM4T, including ablations and a 27-run hyperparameter sweep. Sections X and XI discuss the results and summarize the main findings. I Related Work Robust automatic speech recognition (ASR) has often used acoustic augmentation to improve performance in noisy conditions, including additive noise [1, 2], reverberation [3], speed and pitch perturbation [1], and time-frequency masking [4]. SpecAugment [4] established structured masking as a strong baseline, with later extensions to mixed-sample and hidden-space augmentation [5, 6]. However, these methods typically rely on coarse heuristics and do not allow systematic exploration of environment structure. Synthetic data pipelines generate diverse acoustic conditions via simulated noise and reverberation, using either simple RIR/noise addition [1, 3] or generative models for realistic noise synthesis [10, 11]. While effective for robustness, they often trade simplicity for controllability, making structured ablation difficult [12]. Large-scale S2T systems such as Whisper [13] and SeamlessM4T [14] achieve strong baseline robustness, but remain sensitive to training distributions and unseen noise. Evaluations typically focus on fixed benchmarks rather than systematic augmentation design or environment reduction, leaving limited understanding of how environment structure impacts robustness. I Overview of NOPE-HYPE NOPE-HYPE is a structured workflow for improving and analyzing acoustic robustness in speech-to-text systems, where robustness refers to maintaining strong performance under changing noise conditions and environment types. Instead of treating noise augmentation as a fixed pre-processing step, it decomposes robustness into three controllable stages. We use real environmental recordings from the DEMAND dataset [15] with the environment categories listed in supplementary material. First, each environment is summarized by a Power Spectral Density (PSD) template on a shared frequency grid (SFG). A SFG is used so that all environments are represented on the same frequency bins, allowing their PSD curves to be compared directly. Second, when many environment categories exist, we reduce the set by selecting k prototype environments using a coverage objective over PSD templates. Third, we train with SIM using fitted targets from selected environments, generating controlled environment noise mixed with clean speech at sampled SNR levels, and run structured sweeps over simulator parameters to select robustness-oriented configurations. As summarized in Algorithm 1, this workflow enables direct comparison across environments under a unified training protocol. Algorithm 1 Robustness Workflow 1: Compute PSD template xex_e for each environment e 2: Optionally select few noise environments by coverage optimization on xe\x_e\ 3: for each environment e in P (or all environments) do 4: Fit simulator targets: PSD, envelope stats, transient rate, optional tonal peaks 5: end for 6: for each simulator configuration θ (default or sweep) do 7: Generate noise with simulator using θ 8: Mix with clean speech at sampled SNR (optional RIR) 9: Train S2T model with fixed compute budget 10: Evaluate on clean, real-noise, and sim-noise test sets 11: end for 12: Select θ by robustness objective and report ablations IV SIM: A Controllable Environment Audio Simulator The simulator supports robust training and structured parameter search while providing direct control over key acoustic factors. It produces deterministic outputs under a fixed seed, enabling reliable ablations and controlled comparisons. It targets realistic environmental noise by matching stable statistical properties of real audio rather than trying to sound indistinguishable from real recordings. IV-A Interface and Modes SIM provides three commands: gen (ambience-only), mix (speech plus ambience at target SNR), and eval (noise similarity statistics). Synthesis uses fs=16kHzf_s=16\,kHz by default. It supports two modes: • Preset mode: A fully procedural synthesis pipeline built from layered primitives. It generates a colored-noise bed (white, pink, or brown), applies multi-band spectral shaping with per-band gain control, and introduces slow stochastic modulation to create nonstationarity. Event primitives such as band-limited transients or ramps can also be added. This mode requires no real recordings and provides full parameter control, making it suitable for bootstrap experiments and factorized sweeps. • Generated mode: Anchors synthesis to real recordings through targets estimated from real ambience, including a mean PSD curve, envelope log-RMS statistics with a slow-dynamics rate hint, transient-rate estimates from onset peaks, and optional tonal peaks. This mode aims to match the stable spectral and temporal properties of real environments while retaining explicit control over each component. Fig. 1: SIM model pipeline. IV-B Generated Mode Core Building Blocks The generated mode synthesizes noise from real-environment recordings by matching PSD structure, envelope dynamics, transient activity, chatter, and RIR. Fig. 1 summarizes the pipeline. PSD estimation and PSD-matched bed. We estimate a target PSD using Welch averaging [17] and synthesize a stationary background bed by matching this target spectrum in the Fourier domain, using target magnitudes with randomized phases and a small number of calibration passes to reduce residual PSD mismatch. This design is motivated by the fact that, for stationary or slowly varying background noise, the PSD captures the dominant spectral structure of the environment, while phase randomization provides a simple way to generate noise in the PSD. During calibration, the PSD correction ratios are clipped to bounded ranges to maintain numerical stability and reproducibility. Envelope shaping. We apply a smooth multiplicative envelope a(t)a(t) to the stationary PSD-matched bed to model nonstationarity. We generate a low-frequency control signal u(t)=LPFre(ϵ(t)),u(t)=LPF_r_e(ε(t)), where ϵ(t)ε(t) is white noise and LPFre(⋅)LPF_r_e(·) denotes low-pass filtering with rate parameter rer_e. The resulting signal is then exponentiated to produce a positive envelope, a(t)=exp(σeu(t)),a(t)= ( _eu(t)), where rer_e controls how quickly the envelope varies and σe _e controls the magnitude of energy fluctuations. This stage allows the simulator to improve beyond a purely stationary background bed and capture the slow amplitude variation observed in real environments. Transient injection. Real environments contain short impulsive events such as clicks or impacts. We model these by injecting band-limited transient bursts at a target rate λ events per second, shaped by a smooth envelope and filtered within an environment-specific frequency band. Chatter injection. We model human background activity by mixing short speech segments sampled from a separate speech pool, band-limited and scheduled in burst-like intervals to mimic background murmur or conversation. RIR. For indoor environments, we optionally apply a room impulse response during mixing to simulate reverberant propagation. This introduces reflections characteristic of enclosed spaces and provides explicit control over reverberation. IV-C Mixing Given clean speech s(t)s(t) and simulated ambience xe(t)x_e(t), we form a noisy utterance at a target SNR: y(t)=s(t)+gxe(t),y(t)=s(t)+g\,x_e(t), where the scaling factor is g=RMS(s)RMS(xe) 10−SNR/20.g= RMS(s)RMS(x_e)\,10^-SNR/20. so that the resulting mixture matches the target SNR. The derivation of this expression is provided in supplementary material. After mixing, we apply mild soft clipping and peak normalization to keep the waveform within a stable numeric range and avoid occasional amplitude spikes. V Reducing Environment Sets When many environments are available, sweeping simulator hyperparameters across all environments becomes computationally expensive and often redundant due to similar spectral structure. We reduce the environment set by treating it as a coverage problem in PSD-template space and selecting representative prototypes. PSD templates and distances. Each environment eie_i is represented by a centered PSD template xi∈ℝFx_i ^F on a SFG. We compute distances between PSD templates using two complementary measures: • Correlation distance: This distance captures whether two environments have similar spectral patterns. dcorr(x,y)=1−corr(x,y).d_corr(x,y)=1-corr(x,y). • RMSE: This distance measures how close two PSD templates are in absolute magnitude across frequencies. drmse(x,y)=1F∑f(xf−yf)2.d_rmse(x,y)= 1F _f(x_f-y_f)^2. Objectives of prototype selection. Let E=e1,…,eNE=\e_1,…,e_N\ denote the set of environments represented by PSD templates. Our goal is to select a small subset of environments P⊆EP E of size k, which will serve as prototypes. For any environment eie_i, we measure its distance from the selected prototype set as d(ei,P)=minp∈Pd(ei,p),d(e_i,P)= _p∈ Pd(e_i,p), which assigns each environment to its closest prototype. Since the number of environments is small (N=12N=12), we evaluate candidate prototype sets using exhaustive search. For each value of k, we enumerate all possible subsets P⊆EP E with |P|=k|P|=k and evaluate each subset under both distance metrics. We consider two coverage objectives. Worst-case coverage, or minimax coverage: As illustrated in the left panel of Fig. 2, this objective is determined by the farthest environment from its nearest prototype. It encourages every environment to remain close to at least one selected prototype: minP⊆E,|P|=kmaxei∈Ed(ei,P). _P E,\;|P|=k _e_i∈ Ed(e_i,P). Average-case coverage: As illustrated in the right panel of Fig. 2, this objective depends on the mean distance from all environments to their nearest prototypes. It favors prototype sets that provide good overall coverage on average: minP⊆E,|P|=k1N∑ei∈Ed(ei,P). _P E,\;|P|=k 1N _e_i∈ Ed(e_i,P). (1) This process is conceptually related to clustering methods such as k-means, where cluster centers represent groups of data points. In our setting, the selected environments act as prototype centers, and each environment is assigned to its nearest prototype according to the chosen PSD-template distance. Fig. 2: Prototype-selection objectives: minimax (left) and average-case (right). Selection of prototypes and choice of k. Using the coverage objectives defined above, we evaluate candidate prototype sets as a function of the number of selected environments k. As shown in Fig. 3, the largest reduction in representation error occurs between k=1k=1 and k=3k=3, after which improvements diminish. Based on worst-case coverage, we treat k=3k=3 and k=4k=4 as the main operating points. The set k=3k=3 provides compact coverage suitable for efficient experimentation, while k=4k=4 slightly improves worst-case coverage. The exact prototype sets and coverage metrics obtained through exhaustive evaluation are reported in supplementary material. Fig. 3: Prototype representation quality vs. number of selected environments k. Both RMSE and correlation-distance exhibit diminishing returns after k∈3,4k∈\3,4\. VI Experimental Protocol We used the Indic-ST corpus [16], focusing on 50 hours of English-Hindi speech for training and a 5-hour held-out evaluation set, with all audio processed at 16 kHz. We also use the same setting for English-German speech from the CoVoST dataset [18]. We evaluate robustness on Whisper [13] and SeamlessM4T [14], fine-tuning medium-sized pretrained checkpoints and evaluating Hindi and German generation on clean, real-noise, and simulator-noise conditions. We report BLEU, chrF, and WER as evaluation metrics. Full training and decoding details are provided in supplementary material. VII Primary Robustness Comparison We benchmark the primary robustness comparison on the English-Hindi dataset under five training noise conditions using the same 50-hour training budget. The regimes include clean-only training, Gaussian noise, pink noise, balanced real environment noise (uniformly sampled from 12 environments), and SIM-generated noise. The key outcome is the close similarity between real-noise and simulator-noise training (Table I). In SeamlessM4T, the simulator slightly improves WER relative to real-noise training while maintaining comparable BLEU and chrF. On Whisper, the two are essentially tied. These results show that SIM-generated noise achieves performance comparable to balanced real-noise training under the evaluated conditions. TABLE I: Main robustness comparison (50h train, 5h test). Seamless Whisper Training condition BLEU↑ WER↓ chrF↑ BLEU↑ WER↓ chrF↑ Clean only 47.62847.628 0.43220.4322 66.40066.400 34.16834.168 0.57160.5716 53.03453.034 Gaussian 46.52346.523 0.44670.4467 65.60965.609 29.99429.994 0.62650.6265 46.34046.340 Pink (10 dB) 47.60647.606 0.43100.4310 66.84866.848 34.01934.019 0.57000.5700 53.11753.117 Real env (12) 48.95948.959 0.42930.4293 67.89367.893 35.554 0.55640.5564 54.14354.143 SIM noise 48.996 0.4221 67.896 35.49435.494 0.5549 54.160 As shown in Table I, Gaussian noise is a weak proxy for real environments, with especially large degradation on Whisper. Pink noise is stronger than Gaussian, but it still remains consistently worse than real-noise and simulator-noise training on both models. This suggests that matching only coarse spectral color is insufficient, and that the consistent gains of SIM noise over Gaussian and pink noise provide a proof of concept that more structured and realistic environment simulation is beneficial for robust training. It motivates simulator components that go beyond stationary PSD matching to envelope dynamics, transient activity, and chatter. TABLE I: Performance gap relative to SIM noise. Negative BLEU/chrF and positive WER indicate worse performance than the simulator. Model Baseline Δ Δ Δ SeamlessM4T Gaussian −2.473-2.473 0.02460.0246 −2.287-2.287 SeamlessM4T Pink −1.390-1.390 0.00890.0089 −1.048-1.048 Whisper Gaussian −5.500-5.500 0.07160.0716 −7.820-7.820 Whisper Pink −1.475-1.475 0.01510.0151 −1.043-1.043 TABLE I: Effect of clean:noise mixture ratio. Entries are mean ± std over seeds. Setting Noisy BLEU↑ Noisy WER↓ Noisy chrF↑ clean20_noise80 46.1446.14± 0.430.43 0.4650.465± 0.0080.008 66.6366.63± 0.230.23 clean40_noise60 46.4546.45± 1.201.20 0.4690.469± 0.0190.019 66.4366.43± 0.260.26 clean50_noise50 46.9946.99± 1.581.58 0.4600.460± 0.0260.026 66.5766.57± 0.310.31 clean70_noise30 47.1247.12± 0.480.48 0.4570.457± 0.0080.008 66.4266.42± 0.150.15 Setting Clean BLEU↑ Clean WER↓ Clean chrF↑ clean20_noise80 48.7648.76± 0.190.19 0.4360.436± 0.0030.003 67.8867.88± 0.110.11 clean40_noise60 48.4748.47± 0.630.63 0.4430.443± 0.0120.012 67.7067.70± 0.200.20 clean50_noise50 48.7248.72± 0.860.86 0.4400.440± 0.0140.014 67.8067.80± 0.030.03 clean70_noise30 49.9149.91± 0.190.19 0.4250.425± 0.0030.003 67.9567.95± 0.130.13 VIII Impact of Simulator Components We evaluate whether the simulator controls correspond to meaningful factors for robust training on SeamlessM4T. In each ablation, we keep the training pipeline fixed and vary only one control group. Chatter ON/OFF. Enabling chatter improves both noisy and clean performance (Table IV), suggesting that speech-like interference is an important factor not well captured by stationary background noise alone. TABLE IV: Chatter ablation. Entries are mean ± std over seeds Setting Noisy BLEU↑ Noisy WER↓ Noisy chrF↑ CHATTER_OFF 46.4546.45± 0.100.10 0.4640.464± 0.0060.006 66.6166.61± 0.320.32 CHATTER_ON 47.8747.87± 0.140.14 0.4500.450± 0.0010.001 67.4467.44± 0.860.86 Setting Clean BLEU↑ Clean WER↓ Clean chrF↑ CHATTER_OFF 48.2448.24± 0.240.24 0.4450.445± 0.0020.002 67.6367.63± 0.400.40 CHATTER_ON 49.3649.36± 0.030.03 0.4310.431± 0.0000.000 68.5268.52± 0.170.17 RIR ON/OFF. Enabling RIR reduces performance on both noisy and clean evaluation (Table V), consistent with reverberation acting as a domain shift when training and test reverberation are mismatched [3]. RIR should be treated as an explicit control rather than a default. TABLE V: Additional ablations across simulator components. Setting Noisy BLEU↑ Noisy WER↓ Noisy chrF↑ RIR off 45.86 0.4593 65.62 RIR on 44.90 0.4798 64.36 Bursty 45.40 0.4676 65.10 Continuous 45.48 0.4658 65.49 Full sim 46.97 0.4464 66.90 PSD only 45.07 0.4645 65.78 Setting Clean BLEU↑ Clean WER↓ Clean chrF↑ RIR off 47.73 0.4367 67.10 RIR on 47.21 0.4542 65.96 Bursty 47.04 0.4482 66.80 Continuous 47.53 0.4405 67.13 Full sim 48.94 0.4316 67.90 PSD only 47.58 0.4399 67.10 Continuous vs. bursty chatter. The difference between continuous and bursty chatter is small, as shown in Table V. This suggests that the presence of chatter and its overall level matter more than its fine-grained temporal scheduling in this setting. PSD-only vs. full simulator. The PSD-only variant performs worse than the full simulator, shown in Table V. This indicates that stationary spectral matching alone is insufficient; temporal dynamics and structured interference contribute meaningfully to robustness. IX Hyperparameter Search Over Simulator Knobs After establishing that the simulator does not degrade performance and testing the importance of the knobs, we conduct a structured sweep over the simulator hyperparameters to systematically assess how different knob configurations affect performance. We fine-tune only the SeamlessM4T model, as it performs better than Whisper based on previous results. We conduct a structured sweep consisting of 27 experiments on both the language pair datasets: 9 simulator configurations (Table VI) evaluated across 3 fitted PSD templates—cafe, washing (laundry), and car (Section V). Each configuration specifies a set of simulator controls governing background energy statistics, temporal dynamics, and speech-to-noise mixing. The simulator configurations are defined by a set of parameters; the full list and definitions are provided in supplementary material. We select the best configurations by minimizing the robustness objective AvgWERrob=12(WERreal+WERsim),AvgWER_rob= 12 (WER_real+WER_sim ), (2) while monitoring clean WER to avoid sacrificing clean accuracy. TABLE VI: Sweep configurations (shared across PSD templates). Cfg μlogRMS _ σlogRMS _ rer_e (Hz) λ σλ _λ μSNR _SNR σSNR _SNR cfg01 −6.6-6.6 0.30.3 0.015 2590.015\,259 2.22.2 0.300.30 00 33 cfg02 −6.6-6.6 0.30.3 0.045 7760.045\,776 3.63.6 0.600.60 1010 44 cfg03 −6.6-6.6 0.30.3 0.106 8120.106\,812 4.84.8 0.250.25 2020 55 cfg04 −5.2-5.2 0.50.5 0.015 2590.015\,259 3.63.6 0.600.60 2020 55 cfg05 −5.2-5.2 0.50.5 0.045 7760.045\,776 4.84.8 0.250.25 00 33 cfg06 −5.2-5.2 0.50.5 0.106 8120.106\,812 2.22.2 0.300.30 1010 44 cfg07 −3.3-3.3 0.70.7 0.015 2590.015\,259 4.84.8 0.250.25 1010 44 cfg08 −3.3-3.3 0.70.7 0.045 7760.045\,776 2.22.2 0.300.30 2020 55 cfg09 −3.3-3.3 0.70.7 0.106 8120.106\,812 3.63.6 0.600.60 00 33 IX-A Sweep results and selected default We summarize the sweep outcomes by aggregating performance across the three PSD templates. The goal is to identify configurations that perform consistently well under the robustness objective in Equation 2. As shown in Table VII and Table VIII, configuration 3 (cfg03) and configuration 6 (cfg06) achieve the strongest robustness outcomes in this sweep, with consistently low variance across environments: • cfg03: μlogRMS=−6.6 _ =-6.6, σlogRMS=0.3 _ =0.3, re=0.106812Hzr_e=0.106812\,Hz, λ=4.8λ=4.8, σλ=0.25 _λ=0.25, SNR ∼(20,5) (20,5) dB. • cfg06: μlogRMS=−5.2 _ =-5.2, σlogRMS=0.5 _ =0.5, re=0.106812Hzr_e=0.106812\,Hz, λ=2.2λ=2.2, σλ=0.30 _λ=0.30, SNR ∼(10,4) (10,4) dB. TABLE VII: Sweep results aggregated over PSD templates (cafe, washing, car). Entries are mean ± std over environments. Cfg Clean WER↓ Real WER↓ Sim WER↓ AvgWERrob↓AvgWER_rob cfg01 0.42710.4271± 0.00260.0026 0.45000.4500± 0.00520.0052 0.44240.4424± 0.00200.0020 0.44620.4462± 0.00310.0031 cfg02 0.42500.4250± 0.00160.0016 0.44350.4435± 0.00460.0046 0.44260.4426± 0.01060.0106 0.44300.4430± 0.00730.0073 cfg03 0.42170.4217± 0.00130.0013 0.43710.4371± 0.00490.0049 0.43860.4386± 0.00950.0095 0.43790.4379± 0.00720.0072 cfg04 0.43440.4344± 0.01150.0115 0.45170.4517± 0.01620.0162 0.44580.4458± 0.01610.0161 0.44870.4487± 0.01610.0161 cfg05 0.43230.4323± 0.00900.0090 0.44880.4488± 0.00870.0087 0.44720.4472± 0.00700.0070 0.44800.4480± 0.00760.0076 cfg06 0.42780.4278± 0.00120.0012 0.43650.4365± 0.00100.0010 0.43330.4333± 0.00250.0025 0.43490.4349± 0.00090.0009 cfg07 0.43800.4380± 0.00750.0075 0.45560.4556± 0.00540.0054 0.45610.4561± 0.01300.0130 0.45590.4559± 0.00870.0087 cfg08 0.42600.4260± 0.01280.0128 0.44990.4499± 0.00890.0089 0.44280.4428± 0.00980.0098 0.44640.4464± 0.00840.0084 cfg09 0.43800.4380± 0.00980.0098 0.45410.4541± 0.00920.0092 0.44900.4490± 0.00820.0082 0.45160.4516± 0.00850.0085 TABLE VIII: Sweep results for English to German aggregated over PSD templates. Entries are mean ± std over environments. Cfg Clean WER↓ Real WER↓ Sim WER↓ AvgWERrob↓AvgWER_rob cfg01 0.5540.554± 0.0060.006 0.5800.580± 0.0030.003 0.5820.582± 0.0040.004 0.5810.581± 0.0030.003 cfg02 0.5530.553± 0.0030.003 0.5820.582± 0.0030.003 0.5830.583± 0.0020.002 0.5830.583± 0.0030.003 cfg03 0.5490.549± 0.0020.002 0.5780.578± 0.0010.001 0.5800.580± 0.0030.003 0.5790.579± 0.0020.002 cfg04 0.5530.553± 0.0060.006 0.5810.581± 0.0060.006 0.5830.583± 0.0040.004 0.5820.582± 0.0050.005 cfg05 0.5520.552± 0.0020.002 0.5810.581± 0.0040.004 0.5820.582± 0.0040.004 0.5810.581± 0.0040.004 cfg06 0.5500.550± 0.0010.001 0.5790.579± 0.0020.002 0.5810.581± 0.0000.000 0.5800.580± 0.0010.001 cfg07 0.5520.552± 0.0040.004 0.5790.579± 0.0030.003 0.5820.582± 0.0050.005 0.5810.581± 0.0040.004 cfg08 0.5520.552± 0.0060.006 0.5810.581± 0.0050.005 0.5850.585± 0.0080.008 0.5830.583± 0.0070.007 cfg09 0.5510.551± 0.0020.002 0.5800.580± 0.0050.005 0.5810.581± 0.0010.001 0.5800.580± 0.0030.003 Fig. 4: Sweep sensitivity across language pairs. Values show the WER increase relative to the best configuration for each metric and language pair, using the clean WER and AvgWERrobAvgWER_rob columns from Tables VII and VIII. Lower is better. Fig. 4 shows that cfg03 and cfg06 are the most stable choices across both language pairs and across clean and robust evaluation. We do not claim that cfg03 or cfg06 is globally optimal. Rather, they are strong defaults within this small structured search space and across the three PSD templates studied. The comparison suggests that faster envelope dynamics (rer_e) is consistently helpful, while robustness can be obtained either through a higher-SNR level with higher event rates (cfg03) or through a moderate-SNR level with lower event rates (cfg06). These trends are empirical for our setting and may change across deployment conditions. To examine whether the selected default varies by environment, we also report the best run for each PSD template separately. TABLE IX: Best run per PSD template (min AvgWERrobAvgWER_rob) for en-hi. Env Best exp Clean WER Real WER Sim WER AvgWERrobAvgWER_rob cafe exp02 0.42380.4238 0.43820.4382 0.43340.4334 0.43580.4358 washing exp12 0.42040.4204 0.43170.4317 0.42850.4285 0.43010.4301 car exp24 0.42640.4264 0.43750.4375 0.43050.4305 0.43400.4340 TABLE X: Best run per PSD template (min AvgWERrobAvgWER_rob) for en-de. Env Best exp Clean WER Real WER Sim WER AvgWERrobAvgWER_rob cafe exp04 0.554 2140.554\,214 0.578 4580.578\,458 0.580 6120.580\,612 0.579 5350.579\,535 washing exp12 0.547 7520.547\,752 0.575 9620.575\,962 0.580 0310.580\,031 0.577 9970.577\,997 car exp19 0.547 2050.547\,205 0.576 6110.576\,611 0.576 8170.576\,817 0.576 7140.576\,714 The best configuration varies slightly by PSD template, as shown in Table IX and Table X, which is expected because different spectral shapes and temporal properties imply different masking regimes. In practice, cfg03 and cfg06 provide good general defaults, while per-environment tuning can be treated as a second-stage refinement when deployment is dominated by a single environment type. X Discussion The central design choice in NOPE-HYPE is controllability, so that changes in robustness can be attributed to explicit simulator controls or selection objectives. Training with simulator-generated noise matches balanced real-noise training under the same data budget and evaluation conditions, indicating that the simulator captures the statistics that matter for robustness in the studied settings. This framework highlights that prototype selection is most useful when the environment catalog is large and training budgets are limited. Compute PSD templates, select k environments via the coverage objective, and focus fitting and tuning on those prototypes. A conservative workflow is to start with k=3k=3 prototypes for rapid iteration, fit simulator targets from real recordings for those prototypes, train with cfg03 or cfg06 as robust default configurations, and then use ablations to decide whether chatter and RIR are appropriate for the intended deployment domain. When deployment is dominated by a single environment type, the same procedure can be refined by tuning knobs per PSD template using the per-environment selection approach. XI Conclusion We introduced NOPE-HYPE, a structured robustness workflow for speech-to-text training that combines a controllable environment simulator, coverage-optimal environment reduction on PSD templates, and a structured hyperparameter search over simulator knobs. We demonstrated performance comparable to balanced real-noise training across Whisper and SeamlessM4T. Through ablations, we showed that dynamics matter beyond stationary spectral matching and provided principled environment prototype sets. Finally, we selected practical default simulator configurations from a 27-run sweep. Together, these components form a reproducible and interpretable path toward robust S2T training. References [1] T. Ko, V. Peddinti, D. Povey, and S. Khudanpur, “Audio augmentation for speech recognition,” in Proc. Interspeech, 2015, p. 3586–3589, doi: 10.21437/Interspeech.2015-711. [2] A. Hannun, C. Case, J. Casper, B. Catanzaro, G. Diamos, E. Elsen, R. Prenger, S. Satheesh, S. Sengupta, A. Coates, and A. Y. Ng, “Deep Speech: Scaling up end-to-end speech recognition,” arXiv:1412.5567, 2014. [3] T. Ko, V. Peddinti, D. Povey, M. L. Seltzer, and S. Khudanpur, “A study on data augmentation of reverberant speech for robust speech recognition,” in Proc. IEEE Int. Conf. Acoust., Speech Signal Process. (ICASSP), 2017, p. 5220–5224, doi: 10.1109/ICASSP.2017.7953152. [4] D. S. Park, W. Chan, Y. Zhang, C.-C. Chiu, B. Zoph, E. D. Cubuk, and Q. V. Le, “SpecAugment: A simple data augmentation method for automatic speech recognition,” in Proc. Interspeech, 2019, p. 2613–2617, doi: 10.21437/Interspeech.2019-2680. [5] G. Kim, D. K. Han, and H. Ko, “SpecMix: A mixed sample data augmentation method for training with time-frequency domain features,” arXiv:2108.03020, 2021. [6] H. Wang, Y. Zou, and W. Wang, “SpecAugment++: A hidden space data augmentation method for acoustic scene classification,” arXiv:2103.16858, 2021. [7] A. van den Oord, S. Dieleman, H. Zen, K. Simonyan, O. Vinyals, A. Graves, N. Kalchbrenner, A. Senior, and K. Kavukcuoglu, “WaveNet: A generative model for raw audio,” arXiv:1609.03499, 2016. [8] K. Kumar, R. Kumar, T. de Boissiere, L. Gestin, W. Z. Teoh, J. Sotelo, A. de Brebisson, Y. Bengio, and A. Courville, “MelGAN: Generative adversarial networks for conditional waveform synthesis,” arXiv:1910.06711, 2019. [9] H. Liu, Z. Chen, Y. Yuan, X. Mei, X. Liu, D. Mandic, W. Wang, and M. D. Plumbley, “AudioLDM: Text-to-audio generation with latent diffusion models,” arXiv:2301.12503, 2023. [10] A. Fazel, W. Yang, Y. Liu, R. Barra-Chicote, Y. Meng, R. Maas, and J. Droppo, “SynthASR: Unlocking synthetic data for speech recognition,” arXiv:2106.07803, 2021. [11] E. Casanova, C. Shulby, A. Korolev, A. Candido Junior, A. da Silva Soares, S. Aluísio, and M. A. Ponti, “ASR data augmentation in low-resource settings using cross-lingual multi-speaker TTS and cross-lingual voice conversion,” arXiv:2204.00618, 2023. [12] D. Liu, A. Nassereldine, C. Xu, and J. Xiong, “Towards pretraining robust ASR foundation model with acoustic-aware data augmentation,” arXiv:2505.20606, 2025. [13] A. Radford, J. W. Kim, T. Xu, G. Brockman, C. McLeavey, and I. Sutskever, “Robust speech recognition via large-scale weak supervision,” arXiv:2212.04356, 2022. [14] Seamless Communication et al., “SeamlessM4T: Massively multilingual & multimodal machine translation,” arXiv:2308.11596, 2023. [15] J. Thiemann, N. Ito, and E. Vincent, “DEMAND: A collection of multi-channel recordings of acoustic noise in diverse environments,” in Proc. Int. Congr. Acoust. (ICA), 2013. [16] S. Shah, K. R. Saxena, K. M. Bharadwaj, S. Adavanne, and N. Adiga, “IndicST: Indian multilingual translation corpus for evaluating speech large language models,” in Proc. IEEE Int. Conf. Acoust., Speech Signal Process. Workshops (ICASSPW), 2025, p. 1–5, doi: 10.1109/ICASSPW65056.2025.11011192. [17] P. Welch, “The use of fast Fourier transform for the estimation of power spectra: A method based on time averaging over short, modified periodograms,” IEEE Trans. Audio Electroacoust., vol. 15, no. 2, p. 70–73, 1967, doi: 10.1109/TAU.1967.1161901. [18] C. Wang, J. Pino, A. Wu, and J. Gu, “CoVoST: A diverse multilingual speech-to-text translation corpus,” arXiv:2002.01320, 2020. Acknowledgment AI-Generated Content Disclosure: The authors used AI-based writing and research-assistance tools, including OpenAI ChatGPT, Anthropic Claude Sonnet, Google Gemini 3.1 Pro, and xAI Grok 4.3, to assist with manuscript review, language polishing, grammar correction, organization suggestions, LaTeX formatting suggestions, and drafting of explanatory text. AI assistance was applied to portions of the Abstract, Introduction, Related Work, Method description, Experimental Protocol, Discussion, Conclusion, and Acknowledgment sections. The AI systems were not used to fabricate experimental results, datasets, evaluation metrics, figures, tables, or citations. All experimental design choices, simulator implementation, training runs, evaluation results, analyses, and final scientific claims were reviewed, edited, and verified by the authors, who take full responsibility for the final content of the paper. Supplementary Material Appendix A Method Details A-A Derivation of Noise Scaling Factor Let s(t)s(t) denote the clean speech signal and xe(t)x_e(t) the simulated ambience. The mixed signal is y(t)=s(t)+gxe(t),y(t)=s(t)+g\,x_e(t), (3) where g is chosen to achieve a target signal-to-noise ratio (SNR). We define the SNR in decibels using RMS amplitudes: SNR=20log10(RMS(s)RMS(gxe)).SNR=20 _10 ( RMS(s)RMS(gx_e) ). (4) Using the scaling property of RMS, RMS(gxe)=gRMS(xe),RMS(gx_e)=g\,RMS(x_e), (5) we obtain SNR =20log10(RMS(s)gRMS(xe)), =20 _10 ( RMS(s)g\,RMS(x_e) ), (6) SNR20 SNR20 =log10(RMS(s)gRMS(xe)), = _10 ( RMS(s)g\,RMS(x_e) ), (7) 10SNR/20 10^SNR/20 =RMS(s)gRMS(xe). = RMS(s)g\,RMS(x_e). (8) Solving for g gives g=RMS(s)RMS(xe)10−SNR/20.g= RMS(s)RMS(x_e)10^-SNR/20. (9) This is the scaling factor used during speech-noise mixing in SIM. Appendix B Experimental Details B-A Training and Decoding Details Table XI summarizes the principal training and decoding settings used for the evaluated models. TABLE XI: Training and decoding settings for SeamlessM4T and Whisper. Setting SeamlessM4T Whisper Checkpoint hf-seamless-m4t-medium whisper-medium Trainer HuggingFace Trainer HuggingFace Seq2SeqTrainer Audio sampling rate 16 kHz 16 kHz Batch size 4 8 Gradient accumulation 4 4 Optimizer AdamW AdamW Learning rate 3×10−53× 10^-5 3×10−53× 10^-5 Scheduler Cosine Cosine Warmup ratio 0.1 0.1 Weight decay 0.03 0.03 Maximum gradient norm 0.5 0.5 Epoch budget Up to 10 Up to 10 Early stopping Patience 5, threshold 0.005 Patience 5, threshold 0.005 Precision fp16 bf16 if available, otherwise fp16 Gradient checkpointing Yes Yes Target decoding language Hindi (en-hi); German (en-de) Hindi (en-hi); German (en-de) Beam size 4 4 Maximum new tokens 160 160 B-B Coverage-Based Environment Reduction Table XII reports the environment prototypes obtained under the PSD-based coverage analysis described in the main paper. The prototypes are actual environments rather than synthetic cluster centroids. TABLE XII: Prototype environments selected using PSD coverage. k Selected prototypes max dcorrd_corr mean dcorrd_corr RMSEμ (dB) RMSEmax (dB) Corrμ Corrmin 1 bus 0.077 0.034 1.986 3.073 0.966 0.923 2 cafe, office 0.040 0.021 1.316 2.302 0.979 0.966 3 cafe, washing, car 0.032 0.015 1.074 2.089 0.987 0.969 4 washing, station, car, meeting 0.028 0.012 0.748 1.897 0.992 0.977 5 cafe, living, washing, bus, meeting 0.027 0.010 0.670 2.077 0.994 0.973 6 living, washing, field, station, car, meeting 0.020 0.008 0.489 1.327 0.996 0.984 Increasing k improves spectral coverage of the environment set. The reduction is largest over the first few prototypes and becomes smaller after approximately k=3k=3–44, consistent with the knee observed in the coverage curve in the main paper. We therefore use the k=3k=3 set, consisting of cafe, washing, and car, as the compact set for the subsequent simulator sweep. B-C Full Simulator Parameter Sweep For each language pair, the structured sweep contains nine simulator configurations evaluated over three PSD templates: cafe, washing, and car. This gives 27 experiments per language pair. The full English-to-Hindi run-level results are reported below. To maintain readable text, the results are divided into clean/real-noise and simulator-noise tables rather than compressed into a single extremely wide table. TABLE XIII: Full English-to-Hindi sweep: clean and real-noise evaluation. Env Exp Cfg Clean BLEU Clean chrF Clean WER Real BLEU Real chrF Real WER cafe exp01 cfg01 48.668 67.761 0.425 47.421 66.771 0.444 cafe exp02 cfg02 48.762 68.015 0.424 47.736 66.782 0.438 cafe exp03 cfg03 48.959 67.903 0.423 47.422 66.582 0.438 cafe exp04 cfg04 47.807 68.204 0.447 46.102 66.876 0.470 cafe exp05 cfg05 47.869 67.975 0.443 46.771 66.912 0.457 cafe exp06 cfg06 48.696 67.701 0.429 47.923 66.819 0.436 cafe exp07 cfg07 48.491 67.617 0.435 47.037 66.399 0.456 cafe exp08 cfg08 49.695 68.499 0.412 47.612 67.057 0.440 cafe exp09 cfg09 47.261 67.436 0.449 46.143 66.484 0.463 washing exp10 cfg01 48.315 68.126 0.430 46.949 67.160 0.452 washing exp11 cfg02 49.264 68.164 0.424 47.855 66.812 0.446 washing exp12 cfg03 49.125 68.263 0.420 47.745 66.681 0.432 washing exp13 cfg04 48.520 67.527 0.433 47.059 66.057 0.443 washing exp14 cfg05 48.445 68.079 0.428 47.361 67.011 0.440 washing exp15 cfg06 48.642 68.168 0.429 47.983 67.127 0.436 washing exp16 cfg07 47.692 67.853 0.446 46.554 66.637 0.461 washing exp17 cfg08 48.764 67.964 0.428 46.976 66.352 0.454 washing exp18 cfg09 48.325 67.860 0.429 47.144 66.718 0.445 car exp19 cfg01 48.480 68.323 0.426 46.801 66.980 0.454 car exp20 cfg02 48.777 67.943 0.427 47.328 66.603 0.447 car exp21 cfg03 49.161 67.941 0.422 47.349 66.567 0.441 car exp22 cfg04 48.911 68.028 0.424 47.445 66.541 0.442 car exp23 cfg05 48.638 67.574 0.427 47.131 66.293 0.450 car exp24 cfg06 48.852 68.243 0.426 47.752 66.862 0.438 car exp25 cfg07 48.664 68.078 0.432 47.051 66.564 0.450 car exp26 cfg08 48.169 67.857 0.438 46.718 66.479 0.456 car exp27 cfg09 48.223 68.011 0.436 46.984 66.863 0.455 TABLE XIV: Full English-to-Hindi sweep: simulator-noise evaluation. Env Exp Cfg Sim BLEU Sim chrF Sim WER cafe exp01 cfg01 47.707 66.971 0.441 cafe exp02 cfg02 48.112 67.316 0.433 cafe exp03 cfg03 47.487 66.710 0.440 cafe exp04 cfg04 46.550 67.108 0.464 cafe exp05 cfg05 46.913 67.113 0.455 cafe exp06 cfg06 47.955 66.902 0.435 cafe exp07 cfg07 46.512 66.485 0.466 cafe exp08 cfg08 47.726 67.340 0.437 cafe exp09 cfg09 46.450 66.725 0.458 washing exp10 cfg01 47.294 67.329 0.445 washing exp11 cfg02 47.914 66.940 0.440 washing exp12 cfg03 48.077 67.086 0.429 washing exp13 cfg04 47.414 66.524 0.438 washing exp14 cfg05 47.256 67.191 0.442 washing exp15 cfg06 48.097 67.359 0.434 washing exp16 cfg07 46.540 66.982 0.461 washing exp17 cfg08 47.751 66.959 0.437 washing exp18 cfg09 47.475 67.104 0.443 car exp19 cfg01 47.416 67.355 0.441 car exp20 cfg02 46.788 66.647 0.454 car exp21 cfg03 47.350 66.523 0.447 car exp22 cfg04 47.774 66.792 0.435 car exp23 cfg05 47.520 66.540 0.444 car exp24 cfg06 48.148 67.303 0.430 car exp25 cfg07 47.563 66.882 0.441 car exp26 cfg08 46.970 66.703 0.454 car exp27 cfg09 47.647 67.278 0.445 Each experiment combines one of the nine configurations with one of the three PSD templates. These run-level results are aggregated by configuration in the main paper to compute clean, real-noise, simulator-noise, and robustness-oriented WER statistics. Appendix C Data and Environment Details C-A DEMAND Environment Categories The DEMAND dataset contains recordings from 18 real-world acoustic environments spanning indoor, outdoor, public, and transportation conditions. The environment categories considered in our data preparation are: • Indoor: kitchen, living room, washing area, hallway, meeting room, office, cafeteria, and restaurant. • Outdoor and public spaces: open field, park, river, station, street cafe, and public square. • Transportation and traffic: traffic, bus, car, and metro. The main experiments use a fixed subset of 12 environments from this catalog for the real-noise comparison and PSD-based environment reduction. The prototype-selection procedure is therefore performed with N=12N=12, as stated in the main paper. The subsequent structured hyperparameter sweep uses the three selected PSD templates: cafe, washing, and car. C-B Spectral Examples from DEMAND Figure 5 provides qualitative examples of acoustic structure in two DEMAND environments using mean Welch PSD curves and mean STFT spectrograms. The PSD summarizes the long-term frequency-wise energy distribution, while the spectrogram shows how spectral energy evolves over time. The examples illustrate that different environments exhibit distinct spectral profiles and temporal structures, motivating the use of environment-specific PSD templates in NOPE-HYPE. Fig. 5: Qualitative examples of acoustic structure for two DEMAND environments, shown using mean Welch PSD curves and mean STFT spectrograms. C-C Sweep Parameter Definitions The simulator configurations are defined using the following parameters: • μlogRMS _ : mean log-RMS level controlling the characteristic background noise energy. • σlogRMS _ : variability of the log-RMS background energy. • rer_e: rate parameter controlling slow temporal variation of the ambience envelope. • λ: mean rate of transient acoustic events. • σλ _λ: variability of the transient-event rate. • μSNR _SNR: mean SNR used when mixing clean speech with environmental noise. • σSNR _SNR: variability of the sampled SNR across utterances.