Paper deep dive
FFASR: Benchmarking Far-Field Automatic Speech Recognition using High-Fidelity Simulated RIRs
Shivam Saini, Eric Bezzam, Georg Götz, Alessia Milo, Steinar Guðjónsson, Konstantinos Gkanos, Finnur Pind, Daniel Gert Nielsen
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 90%
Last extracted: 10/4/2026, 3:28:15 AM
Summary
The paper introduces FFASR, a benchmark for far-field automatic speech recognition (ASR) designed to address the limitations of existing close-microphone benchmarks. It comprises 15,637 utterances convolved with high-fidelity hybrid wave/geometrical-acoustics room impulse responses (RIRs) from 14 furnished rooms. The benchmark features nine conditions varying acoustic factors such as SNR and talker motion, while keeping transcripts and talkers constant to isolate degradation effects. Results show significant WER degradation in far-field conditions compared to near-field, with simulated RIRs providing a scalable proxy for measured far-field evaluation.
Entities (8)
Relation Signals (6)
FFASR → evaluates → Word Error Rate
confidence 95% · Across contemporary systems, mean word error rate (WER) rises from 4.4% near-field to 41.3% in the static low-SNR condition
FFASR → uses → Room Impulse Response
confidence 95% · Dry speech from 15 talkers is convolved with hybrid wave/geometrical-acoustics room impulse responses from 14 furnished rooms
Hugging Face → hosts → FFASR
confidence 90% · Submissions arrive through a public Hugging Face Space
FFASR → varies → Signal-to-Noise Ratio
confidence 90% · static far-field mixtures at high/mid/low signal-to-noise ratio(SNR)
Whisper → isevaluatedin → FFASR
confidence 85% · The systems we evaluate are drawn from across the current architecture space: web-scale weakly supervised encoder–decoders (Whisper [31])
Conformer → isevaluatedin → FFASR
confidence 85% · Conformer [34]/FastConformer [35] encoders with CTC or transducer [36] decoders... The systems we evaluate are drawn from across the current architecture space
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Far-field automatic speech recognition(ASR) degrades under reverberation, noise, and talker motion, yet the benchmarks that drive model selection emphasize close-microphone speech. We present FFASR, a held-out corpus of 15,637 utterances and an open leaderboard spanning nine conditions, each varying a single acoustic factor: anechoic near-field speech, a measured-versus-simulated office-lab pair, static far-field mixtures at high/mid/low signal-to-noise ratio(SNR), and moving-talker variants at matched SNR. Dry speech from 15 talkers is convolved with hybrid wave/geometrical-acoustics room impulse responses from 14 furnished rooms; because the speech is newly recorded and the test waveforms are never released, the corpus resists training-data contamination. Across contemporary systems, mean word error rate (WER) rises from 4.4% near-field to 41.3% in the static low-SNR condition; a moving talker adds a small but consistent penalty at matched SNR; and on the office-lab pair, measured and simulated WER agree to within about 1.7 pp on average. These results support high-fidelity simulation as a scalable proxy for measured far-field evaluation under the conditions we test.
Tags
Links
- Source: https://arxiv.org/abs/2609.38897v1
- Canonical: https://arxiv.org/abs/2609.38897v1
Trouble viewing inline? Open PDF directly →
Full Text
43,884 characters extracted from source content.
Expand or collapse full text
FFASR: Benchmarking Far-Field Automatic Speech Recognition using High-Fidelity Simulated RIRs Shivam Saini1, Eric Bezzam2, Georg Götz1, Alessia Milo1, Steinar Guðjónsson1, Konstantinos Gkanos1, Finnur Pind1, Daniel Gert Nielsen1 Affiliation: 2Hugging Face, Paris, France *Correspondence: shivam@treble.tech Affiliation: 1Treble Technologies, Reykjavík, Iceland Abstract Far-field automatic speech recognition (ASR) degrades under reverberation, noise, and talker motion, yet the benchmarks that drive model selection emphasize close-microphone speech. We present FFASR, a held-out corpus of 15,637 utterances and an open leaderboard spanning nine conditions, each varying a single acoustic factor: anechoic near-field speech, a measured-versus-simulated office-lab pair, static far-field mixtures at high/mid/low signal-to-noise ratio (SNR), and moving-talker variants at matched SNR. Dry speech from 15 talkers is convolved with hybrid wave/geometrical-acoustics room impulse responses from 14 furnished rooms; because the speech is newly recorded and the test waveforms are never released, the corpus resists training-data contamination. Across contemporary systems, mean word error rate (WER) rises from 4.4% near-field to 41.3% in the static low-SNR condition; a moving talker adds a small but consistent penalty at matched SNR; and on the office-lab pair, measured and simulated WER agree to within about 1.7 p on average. These results support high-fidelity simulation as a scalable proxy for measured far-field evaluation under the conditions we test. Index Terms: Far-Field Speech Recognition, ASR, benchmark, leaderboard, moving sources I Introduction Consumer and enterprise voice interfaces increasingly operate at meter-scale distances from the microphone: smart speakers, conference systems, and wearable assistants capture speech only after room reflections, diffraction, and competing noise have reshaped the waveform. The resulting mismatch is well documented, reverberation smears phonetic cues over hundreds of milliseconds while additive noise suppresses low-energy consonants, and has motivated a long line of dedicated evaluations, from the REVERB challenge [1] to the CHiME series [2, 3], AMI [4], and VOiCES [5]. Yet the benchmarks that drive day-to-day model selection remain dominated by close-microphone read speech such as LibriSpeech [6] and its derivatives [7], and modern foundation models are routinely trained on simulated reverberant data [8, 9] without a corresponding standardized far-field test. Constructing such a test set is a balancing act: it needs physical realism, broad and controllable acoustic coverage, and source material novel enough to resist training-data contamination, and these goals pull against one another. Measured corpora [10, 11, 12] offer physical ground truth but cover few geometries and fixed SNR statistics, and once they have been public for several years their audio and transcripts leak into web-scale training crawls, raising contamination risk. Classical simulators based on geometrical acoustics (GA), such as the image-source method [13, 14, 15], scale arbitrarily but omit diffraction, scattering, and low-frequency modal behavior, exactly the wave phenomena that dominate below the Schroeder frequency of typical rooms. Hybrid engines that couple wave-based solvers [16] with GA above a crossover frequency have narrowed this realism gap [17, 18, 19, 20], and recent hybrid higher-order Ambisonics RIR datasets [18, 21] demonstrated that such room impulse responses (RIRs) support far-field ASR research at scale. In our own preliminary leaderboard runs, several systems that were almost indistinguishable on near-field speech separated sharply once the same utterances were rendered in rooms, which is what convinced us a dedicated far-field test set was needed. No existing resource combines all of this: a fixed far-field test set whose audio stays private, separate SNR and source-motion factors, a measured anchor for the sim-to-real gap, and shared infrastructure that scores different model families under identical decoding and normalization rules. FFASR is built to close that gap; Fig. 1 gives a one-view overview and Table I lists the nine conditions. We contribute: A benchmark and open infrastructure. A nine-condition far-field corpus (Table I) built from newly recorded anechoic speech (889 utterances, 15 talkers) convolved with hybrid wave/GA RIRs from 14 furnished rooms (53–367 m3), plus a measured office-lab triplet for sim-to-real calibration. A deterministic, seeded pipeline shares transcripts, talkers, and noise draws across conditions, so per-condition WER differences come only from the acoustic factor being varied. An open leaderboard scores each submission in an isolated GPU sandbox [22] and never releases the test waveforms (Sec. I–IV). Empirical findings from evaluating contemporary systems, from compact CTC models to large speech LLMs (Sec. V): clean-speech rankings hide large far-field differences, separating systems that look equivalent on clean speech and triggering insertion-dominated hallucination at low SNR; source motion adds a smaller additional cost at matched SNR; and the measured and simulated office-lab renders give similar WER across architectures, suggesting the simulated renders are close enough to measured ones for this evaluation. Fig. 1: Data-generation pipeline. Newly recorded anechoic speech is convolved with hybrid wave/GA room impulse responses and seeded noise draws; each mixture is labelled by its post-render SNR. TABLE I: FFASR evaluation conditions. N is the number of clips in each packed split. Condition N Description Near field 889 Dry anechoic speech Lab simulated 2000 Stage A hybrid render Lab measured 2000 Stage A measured mono High SNR 1746 Static, SNR ≥14≥ 14 dB Mid SNR 1938 Static, 8–12 dB Low SNR 1647 Static, ≤6≤ 6 dB Moving high 1808 Trajectory + high SNR Moving mid 1816 Trajectory + mid SNR Moving low 1793 Trajectory + low SNR I Related Work I-A Far-Field Corpora and Challenges The REVERB challenge [1] established reverberant evaluation with simulated and measured single- and eight-channel data. CHiME-5/6 [2, 3] moved to unsegmented multi-talker dinner parties, while AMI [4] and LibriCSS [23] target meetings and overlapped speech. These corpora combine many degradation factors at once, such as overlap, disfluency, and distant arrays, which makes them excellent integration tests but poor instruments for isolating a single far-field factor; convolution with measured RIRs is controllable but static. FFASR complements these with single-talker, factorized conditions with matched lexical content across all nine splits. I-B Simulation for Far-Field Speech Processing Simulated RIRs are a standard tool for training robust ASR [8, 9] and for privacy-preserving on-device development [24]; our concern here is instead their fidelity for evaluation. The hybrid wave/GA rendering in an industrial hybrid wave/GA simulator(Sec. I-B) has been validated against measurement across several tasks [25, 26, 27, 28, 18, 29, 30]. Most relevant to the present evaluation, [26] investigates the simulation-to-real gap when evaluating audio algorithms on simulated data with varying fidelity rather than measurements, while [29] and [30] report that the added physical fidelity of hybrid RIR training data over image-source augmentation transfers to a downstream ASR metric, reducing median WER on measured test data by up to 38% relative. FFASR builds on this line, using 14 furnished scenes with many source–receiver positions and directional noise sources. I-C Robust ASR Systems The systems we evaluate are drawn from across the current architecture space: web-scale weakly supervised encoder–decoders (Whisper [31]) and compressed variants [32, 33]; Conformer [34]/FastConformer [35] encoders with CTC or transducer [36] decoders, including the token-and-duration transducer (TDT) [37] used by Parakeet; Canary [38]; OWSM [39, 40]; speech-aware LLMs (Granite Speech [41]); wav2vec 2.0 [42] CTC baselines via SpeechBrain [43]; and streaming-oriented compact models (Moonshine [44]). We treat these as the systems under test rather than as related methods, and report their behavior in Sec. V. I-D Leaderboards and Evaluation Methodology The Open ASR Leaderboard [7] standardized text normalization and RTFx reporting for predominantly close-talk English test sets and documented how normalization choices alone shift WER rankings. FFASR adopts the same normalizer and efficiency metric so that scores are directly comparable, while adding the far-field axes those suites lack. Prior work also documents Whisper-family hallucination on degraded or silent input [45]; our low-SNR conditions elicit this failure mode systematically and expose it as WER exceeding 100% (Sec. V-A). I The FFASR Benchmark Fig. 1 summarizes the generation flow and Algorithm 1 states it step by step. A few deliberate choices shape the corpus. All source speech is newly recorded and the test waveforms are never distributed, so a model cannot score well simply by having seen the test data in training. Each condition changes a single acoustic factor while holding transcripts, talkers, and seeded noise draws fixed, so a WER gap between two conditions can be attributed to that factor rather than to a change in content. And rendering uses a hybrid wave/GA solver that has been checked against measurement, which keeps the simulated rooms physically grounded rather than idealized. The nine resulting conditions are listed in Table I. Algorithm 1 FFASR scene generation (one mixture) 0: one dry utterance s, a pool of anechoic interferers, the RIR collections, and a shared random seed 1: draw 11–22 interferers nk\n_k\ using the shared seed, so the same draw recurs across conditions 2: pick a room and the target/receiver positions, and load the target’s RIR hsh_s and one RIR hkh_k per interferer 3: place each source in the room by convolution: target ys=hs∗sy_s=h_s s, background yn=∑khk∗nky_n= _kh_k n_k time-varying hs(t,τ)h_s(t,τ) if the target moves 4: pass every render through the device front-end: a 7070 Hz high-pass plus added pink self-noise d(t)d(t) 5: form the mixture y=ys+yn+dy=y_s+y_n+d, and mark the speech-active region A of ysy_s with an ITU-T P.56 detector 6: measure the SNR of y over A using (2) 7: if the SNR lands in the High, Mid, or Low band then 8: keep y and label it with that band 9: else 10: discard y: it fell in a guard band between bands 11: end if 11: mono 1616 kHz mixture y with its SNR-band label I-A Anechoic Source Speech Clean speech is recorded in the anechoic chamber at an anonymized university. Fifteen subjects (6 female, 9 male; who gave informed consent for research use of the recordings, who gave informed consent for research use of the recordings) each read 80 English sentences from a fixed prompt list. Fifteen talkers is a compromise: enough to average over individual voice and accent, but small enough that every talker appears in every condition, so no split is confounded by a change in voices. Two talkers are native English speakers and thirteen are non-native, spanning different native-language backgrounds, so the pool reflects the accent diversity of deployed voice interfaces. FFASR isolates the far-field acoustic factor rather than accent: all nine conditions share this single talker pool, so each model’s own near-field score is its clean-speech reference. Accent thus varies across talkers but is held constant across conditions (Sec. VI). Of the 1,200 prompt recordings (15 talkers × 80 prompts), a single annotator screened every recording and discarded 311 (26%) for mispronunciations, disfluencies, truncated prompts, or audible recording defects, leaving 889 utterances. Because no public read-speech test set is reused, leaderboard scores reflect acoustic generalization rather than memorized lexicon or speaker profiles. Each utterance is normalized and rendered at 60 dB SPL free-field level. I-B Hybrid Room Acoustic Simulation Let s(t)s(t) denote the dry target utterance, nk(t)k=1K\n_k(t)\_k=1^K the anechoic interferer signals, and hs,hkh_s,h_k the RIRs to the receiver from the target speaker (subscript ss) and the k-th noise source. Each scene uses up to K≤2K≤ 2 interferers, and each static far-field mixture is y(t)=(hs∗s)(t)+∑k=1K(hk∗nk)(t)+d(t),y(t)=(h_s s)(t)+ _k=1^K(h_k n_k)(t)+d(t), (1) where ∗ denotes convolution and d(t)d(t) is device self-noise. RIRs are drawn from three precomputed collections: • Static Sources: 14 scenes spanning living spaces, offices, meeting rooms, classrooms, and restaurants, with volumes 53–367 m3. • Moving Sources: paired moving variants of the same rooms with time-varying source paths. • Measured-Simulated pairs: 44 measured RIRs and their simulated counterparts over six receivers in a single office, in three configurations that change only the acoustic treatment: bare brick walls(RT60≈0.9RT_60≈ 0.9 s), the same room furnished (≈0.6≈ 0.6 s), and the room fitted with wall absorber panels (≈0.3≈ 0.3 s). All furnished-room IRs use hybrid simulation in an industrial hybrid wave/GA simulator [17]: a wave-based discontinuous Galerkin finite-element solver [16] up to a 2 kHz crossover and GA above, computed at 32 kHz and stored as 8th-order ambisonics (81 channels) before rendering to mono 16 kHz. The 2 kHz crossover sits below most consonantal energy (2–8 kHz), so the wave solver mainly improves the low- and mid-frequency modal and diffraction behavior. The external validations above [18, 29] and the measured anchor (Sec. V-C) confirm that this added fidelity still transfers to ASR-level WER. Receivers are restricted to heights 1.0<z<2.31.0<z<2.3 m, consistent with tabletop and wall-mounted devices; elevated sources (z>1.9z>1.9 m) model HVAC-like emitters. Each scene places one directive target talker and up to two noise paths: a mixture pool and static-like ambient noises from AID [46]. I-C Noise, SNR Ranges, and Device Chain Reverberant energy builds up differently depending on room geometry and absorption, so the source-level ratio set in advance is a poor predictor of the SNR observed at the receiver. We therefore measure SNR after rendering. A fixed device chain is first applied at the listener to emulate the analog front-end of consumer hardware: a 4th-order Butterworth high-pass at 70 Hz and pink device self-noise d(t)d(t) at 35–40 dB SPL. The 70 Hz corner approximates the low-frequency rolloff of the small transducers in smart speakers and phones and removes sub-band rumble those devices would not capture, so the signal reaching the recognizer matches deployed hardware rather than an idealized full-band render. The SNR is then the energy ratio between the target render ys=hs∗sy_s=h_s s and the full background, namely the interferer render yn=∑khk∗nky_n= _kh_k n_k plus the device noise d(t)d(t). Both energy terms are accumulated only over the speech-active region A of the target render ysy_s, where A is the set of samples lying in frames flagged active by an ITU-T P.56 detector [47], so that silent and inter-word pauses do not contribute: SNR=10log10∑t∈ys2(t)∑t∈(yn(t)+d(t))2.SNR=10 _10 _t y_s^2(t) _t (y_n(t)+d(t) )^2. (2) Keeping d(t)d(t) in the denominator means the label reflects the SNR actually presented to the recognizer, not an idealized speech-to-interferer ratio. Each mixture is assigned to one of three SNR ranges, Low [−2,6][-2,6] dB, Mid [8,12][8,12] dB, and High ≥14≥ 14 dB. The bands are chosen to bracket qualitatively different listening regimes, adverse, effortful, and comfortable, rather than to trace a fine SNR curve, which keeps the conditions interpretable and the per-condition splits large. The transition bands (6,8)(6,8) and (12,14)(12,14) dB act as guard bands and are excluded, so labels stay unambiguous under small changes in the render. For reproducibility, a shared integer seed fixes the interferer draw, room, and source/receiver positions so any scene can be regenerated, mixtures are peak-limited below 0 dBFS to prevent clipping before (2) is evaluated, and all renders are written as 16 kHz mono WAV. I-D Moving Talker Conditions To simulate a moving talker, the static target RIR is replaced by a time-varying RIR that evolves along a predefined source trajectory, while keeping the interferers fixed. Therefore, the static convolution in (1) becomes a time-varying convolution, y(t)=∑τhs(t,τ)s(t−τ)+∑k=1K(hk∗nk)(t)+d(t),y(t)= _τh_s(t,τ)\,s(t-τ)+ _k=1^K(h_k n_k)(t)+d(t), (3) where τ is the discrete time lag and hs(t,τ)h_s(t,τ) is the target impulse response at time t, interpolated along the source trajectory; the interferers stay static. We synthesize hs(t,τ)h_s(t,τ) with the frequency-domain interpolation method of Mullins and Sampedro Llopis [48]. That work compared three offline movement-simulation algorithms. Frequency-domain interpolation achieved the most accurate interaural-time-difference reproduction and the highest perceptual (MUSHRA) similarity to real moving-source recordings, outperforming time-domain crossfading and nearest-neighbor switching. We reuse the same talker and interferer signals as in the static scenes. The static target RIR is replaced by a trajectory that traverses the complete moving-IR path over 6 s of speech within a 15 s scene, corresponding to an effective spatial update rate of ≈ 15.6 Hz.We define Moving High, Mid, and Low using the same SNR ranges as the corresponding static conditions; the labels refer to SNR rather than movement speed. The moving and static splits are matched by SNR range but not perfectly seed-aligned: because SNR is measured after rendering and motion changes the source–receiver distance within an utterance, a seed in one static range can land in an adjacent range once it moves, so a naive moving-minus-static difference mixes the motion effect with a small reshuffling across ranges. At the scale of FFASR this is minor: each moving range holds ≈ 1,800 utterances from the same talker, room, and noise seeds as its static counterpart, the guard bands limit boundary leakage, and the matched splits’ post-render SNR distributions stay closely aligned. The aggregate comparison is therefore stable enough for the leaderboard-level analysis reported here, and the motion penalty is consistently positive across competitive models and SNR ranges (Sec. V). I-E Dataset Statistics and Splits Table I lists the packed evaluation splits and Table I summarizes the acoustic statistics of the furnished-room renders. Up to 2000 samples per condition are kept by subsampling that keeps the 15 talkers balanced; the anechoic split keeps all 889 dry utterances. In total 15,637 clips are drawn from a pool of 17,700 renders, and a full nine-condition run transcribes ≈ 54.9 h of audio. The talkers contribute roughly evenly (per-talker counts 821–992). At the clip level the corpus is about 60% male / 40% female and 86% non-native / 14% native, tracking the 9:6 and 13:2 splits of the talker pool. The renders span the 14 rooms of Sec. I-B and more than 40 receiver positions (seated and standing listeners, wall-, shelf-, and counter-mounted devices), with post-render SNR roughly balanced across the three ranges. Fig. 2 shows an example scene. TABLE I: Acoustic statistics of the FFASR furnished-room renders. Quantity Min Max Mean SD Room volume (m3) 17.47 367.41 128.52 77.26 T20T_20 (s) 0.19 1.29 0.60 0.3 Source–receiver dist. (m) 0.3 13.8 3.98 2.2 C50C_50 (dB) -7.8 19.61 5.3 4.5 Post-render SNR (dB) -1.9 88.9 10.5 6.4 Fig. 2: Example scene from the dataset. Left: the target speech source, the directional transient and ambient noise sources, and the receiver. Right: an example trajectory of the moving speech source. IV Evaluation Protocol and Metrics IV-A Leaderboard Infrastructure Submissions arrive through a public Hugging Face Space [49]. Each job clones the evaluation harness inside an isolated UV sandbox on an NVIDIA L4 GPU [22], loads the user-specified checkpoint, and writes a versioned JSON artifact to a private bucket; the orchestrating process never imports the inference stack, avoiding dependency collisions across model families. The packed test waveforms are never exposed to submitters, which is the mechanism that resists contamination. For the same reason we do not accept closed, API-only systems, since scoring them would require sending the held-out audio to a third party. TABLE I: Distribution of WER (%) across the 15 leading evaluated systems, summarized per condition. The lower panel reports the two derived effects in percentage points (p). Condition Mean Med SD Min Max Near field 4.4 4.3 0.4 3.8 5.4 Lab measured 29.0 27.8 6.7 20.0 45.0 Lab simulated 27.7 26.9 6.9 18.6 42.6 Static High SNR 10.1 10.2 2.2 6.7 15.1 Static Mid SNR 20.5 20.3 4.7 13.9 29.5 Static Low SNR 41.3 40.2 8.3 28.4 57.0 Moving High SNR 11.5 11.5 2.3 8.2 16.8 Moving Mid SNR 23.2 23.5 5.1 15.3 32.3 Moving Low SNR 43.5 42.4 8.6 30.7 60.0 Derived effects (p) Sim2real Δ (meas-sim) 1.3 1.3 1.2 −2.5-2.5 2.8 Motion penalty, High SNR 1.3 1.5 0.5 0.1 1.9 Motion penalty, Mid SNR 2.7 2.6 0.9 1.2 4.7 Motion penalty, Low SNR 2.2 2.3 1.0 0.1 3.8 IV-B Decoding and Metrics Every utterance is transcribed at 16 kHz with batch size 1. References and hypotheses pass through the Whisper EnglishTextNormalizer [31] (lowercasing, punctuation removal, contraction expansion) before scoring, following the Open ASR Leaderboard protocol [7]. WER is the length-normalized Levenshtein cost WER=S+D+IN,WER= S+D+IN, (4) with substitutions S, deletions D, insertions I, and reference length N [50]. WERWER can exceed 1 when insertions dominate; rather than clipping these values, we report them because they are diagnostic of hallucination [45]. The primary ranking metric is Average WER over four core scenarios: near-field speech plus the three static SNR ranges. This ranking should be read together with the per-condition results on the live leaderboard, since changing which conditions enter the average will in general change the ordering. Throughput is reported as RTFxRTFx batch size 1 on the reference GPU, matching [7]. Fig. 3: Average WER vs. RTFx on NVIDIA L4 (batch size 1). Efficient CTC/TDT models occupy the upper-left region. V Results and Discussion At the time of submission the leaderboard holds 24 publicly released systems spanning CTC, RNN-T/TDT, encoder–decoder, and speech-LLM architectures, all scored under the same harness, normalizer, and decoding settings. We summarize the field through the per-condition distribution in Table I and report what we find across conditions rather than a ranking of named systems, since that ranking will change as new models appear; model identities and their efficiency are shown in Fig. 3. Where we quantify an effect below, we give its mean across the 15 systems together with a 95% interval, mean±1.96SD/15mean± 1.96\,SD/ 15, that reflects how tightly the across-system mean is pinned down by our sample of systems (not a per-utterance bootstrap, which would need the seed-paired data deferred to a future release, Sec. VI). V-A Far-field speech is much harder than near-field Near-field WER is uniformly low and tightly clustered (mean 4.4%, all systems within 3.8–5.4%), so clean speech barely separates the field. Reverberation and noise change this: mean WER rises to 10.1% at high SNR, 20.5% at mid SNR, and 41.3% at low SNR, and the spread across systems widens from under 2 p near-field to roughly 30 p at low SNR. The largest single step is from mid to low SNR (≈ 21 p in the mean). We read this as the regime where the noise floor approaches speech energy and masks low-energy consonant cues, though we infer this from the error pattern rather than measure it directly. The degradation is also architecture-dependent. Aggressively compressed encoder–decoders (distilled and “lite” Whisper variants, and the most compact Canary checkpoints) match the rest of the field near-field yet fall furthest behind at low SNR, while large speech-LLM and well-trained transducer systems degrade more gracefully. For the checkpoints we evaluate this suggests that compression trades away reverberation robustness first; we cannot rule out that better-trained small models would behave differently. The most extreme case is outright hallucination: at low SNR the smallest Whisper checkpoints exceed 100% WER (whisper-base 117%, whisper-tiny 137%) [31] even though the same checkpoints score below 10% on clean speech [7]. Inspecting these outputs, the errors are insertion-dominated: during noise-only or low-energy segments the model emits repeated phrases or fabricated sentences [45], so the hypothesis runs far longer than the reference and WER passes 100%. Far-field WER therefore separates robust from brittle systems where clean-speech WER cannot. V-B Source motion adds a consistent penalty Adding source motion at matched SNR raises WER across the board: the static-to-moving penalty (Table I, lower panel) is positive for essentially every competitive system in every SNR range. The penalty is not uniform with SNR. It is largest at mid SNR (mean +2.7+2.7 p, 95% interval [2.2,3.2][2.2,3.2]), smaller at low SNR (+2.2+2.2 p, [1.7,2.7][1.7,2.7]), and smallest at high SNR (+1.3+1.3 p, [1.0,1.6][1.0,1.6]); all three intervals exclude zero. We had expected the penalty to grow as SNR fell, tracking the overall error rate. Instead the largest average penalty sits at mid SNR. A plausible reading is masking: at low SNR additive noise already dominates the errors and hides the kinematic contribution, while at high SNR there is little accuracy left to lose, leaving the mid-SNR regime, where the within-utterance change in direct-to-reverberant ratio is the main remaining difficulty, as the place a moving talker costs most. We offer this as an interpretation, not a measured mechanism. Because the moving conditions are released as a beta and are matched by SNR range rather than seed-paired (Sec. I-D), we report these as an aggregate trend rather than a per-model paired estimate. V-C High-fidelity simulation matches measurement On the matched office-lab pair, where only the rendering engine differs, measured and simulated WER agree closely: across systems the mean absolute difference is 1.7 p and the mean signed difference is +1.3+1.3 p (95% interval [0.7,1.9][0.7,1.9]). The direction is not consistent, measured is marginally harder for most systems but easier for some, so the residual gap behaves like small, roughly zero-mean differences between two realistic renderings rather than a systematic bias of the simulator. The gap is also model-dependent: robust systems sit within about 1 p, while brittle ones amplify small statistical differences between the measured and simulated RIRs into large WER differences (up to +12.7+12.7 p), which is why we report it per system rather than as a single constant. The agreement is not from one easy operating point: the anchor spans three configurations (RT60≈0.3RT_60≈ 0.3–0.90.9 s) and 44 RIR pairs over six receivers, and it reproduces on a larger model set the downstream-ASR fidelity reported for wave-based/hybrid rendering [26]. Because the anchor is a single office lab (Sec. VI), we read this as site-specific evidence: within those bounds, the simulated office-lab renders are close enough to the measured ones for this evaluation. V-D Speed-accuracy trade-off Fig. 3 plots Average WER against RTFx and shows a clear accuracy–throughput tension. The most accurate systems are large and comparatively slow, anchoring the low-WER but not the high-throughput end of the Pareto frontier. The efficient end is dominated by frame-synchronous transducer and CTC decoders [37, 35], which reach competitive WER at one to two orders of magnitude higher throughput, whereas autoregressive speech-LLMs fall well below real-time. The frontier reads as a deployment guide: transducer/CTC models for latency-bound use, attention-decoder or speech-LLM systems when accuracy is paramount. VI Limitations and Future Work Mono input. FFASR currently scores single-channel renders even though the underlying RIRs are 8th-order ambisonic. We chose mono because all 24 evaluated systems accept only single-channel input, so a mono harness is what makes them directly comparable; the cost is that beamforming and array front-ends are not yet exercised. Measured-room coverage. The measured anchor is one office, rendered in three configurations spanning RT60≈0.3RT_60≈ 0.3–0.90.9 s (absorber panels, furnishing, bare brick). This covers a wide range of reverberation at a single site, so the sim-to-real agreement we report should not be read as a general bound. We used the rooms we had measured pairs for; extending to more measured rooms is the clearest next step. Speaker and accent coverage. Source material is English read speech from 15 talkers. Accent varies across talkers but is held constant across conditions by design, which is what lets each near-field score serve as a clean reference; the trade-off is that FFASR measures far-field robustness, not accent robustness, and broader demographic coverage would need a larger recording effort. Read vs. conversational speech. The prompts are read sentences. This keeps lexical content matched across conditions, but omits the disfluency, overlap, and turn-taking of spontaneous speech, which corpora such as CHiME [3] and AMI [4] target directly. Moving-source beta. The moving conditions are a beta. The splits are matched by SNR range rather than seed-paired, and the rendering may still be revised, so we report the motion penalty as an aggregate trend rather than a per-model paired estimate. Future work. Planned extensions include multichannel and multi-speaker input that uses the existing ambisonic RIRs, moving toward joint ASR–diarization evaluation [3, 23]; more measured rooms; a seed-paired motion metric with confidence intervals; and streaming latency tiers that report word-emission delay alongside RTFx [44]. VII Conclusion FFASR is a far-field ASR benchmark whose source speech is newly recorded and whose test audio stays private, built so that a single acoustic factor changes between conditions. The practical aim of FFASR is to make far-field robustness visible in comparisons that look saturated on near-field speech. Two patterns stand out from the current field. Clean-speech WER has saturated, the systems we evaluate sit within 2 p of each other near-field, yet the same systems spread by more than 20 p once reverberation and noise are added, and the smallest models tip into insertion-dominated hallucination at low SNR. Moreover, on the office-lab anchor, measured and simulated WER track each other to within about 1.7 p across architectures, close enough for simulation to stand in for measurement at that site. References [1] K. Kinoshita, M. Delcroix, S. Gannot, E. A. P. Habets, R. Haeb-Umbach, W. Kellermann, V. Leutnant, R. Maas, T. Nakatani, B. Raj, A. Sehr, and T. Yoshioka (2016) A summary of the REVERB challenge: state-of-the-art and remaining challenges in reverberant speech processing research. EURASIP J. Adv. Signal Process. 2016 (1), p. 7. External Links: Document Cited by: §I, §I-A. [2] J. Barker, S. Watanabe, E. Vincent, and J. Trmal (2018) The fifth ‘CHiME’ speech separation and recognition challenge: dataset, task and baselines. In Proc. Interspeech, p. 1561–1565. External Links: Document Cited by: §I, §I-A. [3] S. Watanabe, M. Mandel, J. Barker, E. Vincent, et al. (2020) CHiME-6 challenge: tackling multispeaker speech recognition for unsegmented recordings. In Proc. 6th Int. Workshop on Speech Processing in Everyday Environments (CHiME), p. 1–7. External Links: Document Cited by: §I, §I-A, §VI, §VI. [4] J. Carletta, S. Ashby, S. Bourban, M. Flynn, et al. (2006) The AMI meeting corpus: a pre-announcement. In Proc. Int. Workshop Machine Learning for Multimodal Interaction (MLMI), LNCS, Vol. 3869, p. 28–39. External Links: Document Cited by: §I, §I-A, §VI. [5] C. Richey, M. A. Barrios, Z. Armstrong, C. Bartels, H. Franco, M. Graciarena, A. Lawson, M. K. Nandwana, A. Stauffer, J. van Hout, P. Gamble, J. Hetherly, C. Stephenson, and K. Ni (2018) Voices obscured in complex environmental settings (VOiCES) corpus. In Proc. Interspeech, p. 1566–1570. External Links: Document Cited by: §I. [6] V. Panayotov, G. Chen, D. Povey, and S. Khudanpur (2015) LibriSpeech: an ASR corpus based on public domain audio books. In Proc. IEEE Int. Conf. Acoust., Speech, Signal Process. (ICASSP), p. 5206–5210. External Links: Document Cited by: §I. [7] V. Srivastav, S. Zheng, E. Bezzam, E. L. Bihan, A. Moumen, and S. Gandhi (2025) Open asr leaderboard: towards reproducible and transparent multilingual speech recognition evaluation. arXiv preprint arXiv:2510.06961. Cited by: §I, §I-D, §IV-B, §IV-B, §V-A. [8] T. Ko, V. Peddinti, D. Povey, M. L. Seltzer, and S. Khudanpur (2017) A study on data augmentation of reverberant speech for robust speech recognition. In Proc. IEEE Int. Conf. Acoust., Speech, Signal Process. (ICASSP), p. 5220–5224. External Links: Document Cited by: §I, §I-B. [9] C. Kim, A. Misra, K. Chin, T. Hughes, A. Narayanan, T. N. Sainath, and M. Bacchiani (2017) Generation of large-scale simulated utterances in virtual rooms to train deep-neural networks for far-field speech recognition in Google Home. In Proc. Interspeech, p. 379–383. External Links: Document Cited by: §I, §I-B. [10] J. Eaton, N. D. Gaubitch, A. H. Moore, and P. A. Naylor (2016) Estimation of room acoustic parameters: the ACE challenge. IEEE/ACM Trans. Audio, Speech, Lang. Process. 24 (10), p. 1681–1693. External Links: Document Cited by: §I. [11] I. Szöke, M. Skácel, L. Mošner, J. Paliesek, and J. Černocký (2019) Building and evaluation of a real room impulse response dataset. IEEE J. Sel. Topics Signal Process. 13 (4), p. 863–876. External Links: Document Cited by: §I. [12] E. Hadad, F. Heese, P. Vary, and S. Gannot (2014) Multichannel audio database in various acoustic environments. In Proc. Int. Workshop Acoust. Signal Enhancement (IWAENC), p. 313–317. External Links: Document Cited by: §I. [13] J. B. Allen and D. A. Berkley (1979) Image method for efficiently simulating small-room acoustics. J. Acoust. Soc. Am. 65 (4), p. 943–950. External Links: Document Cited by: §I. [14] R. Scheibler, E. Bezzam, and I. Dokmanić (2018) Pyroomacoustics: a python package for audio room simulation and array processing algorithms. In Proc. IEEE Int. Conf. Acoust., Speech, Signal Process. (ICASSP), p. 351–355. External Links: Document Cited by: §I. [15] D. Díaz-Guerra, A. Miguel, and J. R. Beltrán (2021) gpuRIR: a python library for room impulse response simulation with GPU acceleration. Multimedia Tools and Applications 80 (4), p. 5653–5671. External Links: Document Cited by: §I. [16] F. Pind, A. P. Engsig-Karup, C. Jeong, J. S. Hesthaven, M. S. Mejling, and J. Strømann-Andersen (2019) Time domain room acoustic simulations using the spectral element method. J. Acoust. Soc. Am. 145 (6), p. 3299–3310. External Links: Document Cited by: §I, §I-B. [17] Treble Technologies (2025) Treble software development kit. Note: https://w.treble.tech/software-development-kit Cited by: §I, §I-B. [18] S. S. Mullins, G. Götz, E. Bezzam, S. Zheng, and D. G. Nielsen (2025) Treble10: a high-quality dataset for far-field speech recognition, dereverberation, and enhancement. arXiv preprint arXiv:2510.23141. Cited by: §I, §I-B, §I-B. [19] Z. Tang, R. Aralikatti, A. Ratnarajah, and D. Manocha (2022) GWA: a large high-quality acoustic dataset for audio processing. In Proc. ACM SIGGRAPH, External Links: Document Cited by: §I. [20] C. Chen, C. Schissler, S. Garg, P. Kobernik, A. Clegg, P. Calamia, D. Batra, P. Robinson, and K. Grauman (2022) SoundSpaces 2.0: a simulation platform for visual-acoustic learning. In Advances in Neural Information Processing Systems (NeurIPS), Cited by: §I. [21] S. Saini and J. Peissig (2026) HiFi-harp: a high-fidelity 7th-order ambisonic room impulse response dataset. In ICASSP 2026-2026 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), p. 14632–14636. Cited by: §I. [22] Hugging Face (2024) Hugging face jobs documentation. Note: https://huggingface.co/docs/huggingface_hub/en/guides/jobs Cited by: §I, §IV-A. [23] Z. Chen, T. Yoshioka, L. Lu, T. Zhou, Z. Meng, Y. Luo, J. Wu, X. Xiao, and J. Li (2020) Continuous speech separation: dataset and analysis. In Proc. IEEE Int. Conf. Acoust., Speech, Signal Process. (ICASSP), p. 7284–7288. External Links: Document Cited by: §I-A, §VI. [24] N. Khan, S. Nisar, M. A. Khan, F. Afghah, Y. A. U. Rehman, and G. Barb (2025) A novel privacy-preserving framework for low-resource asr using federated self-supervised learning. IEEE Access 13, p. 217156–217175. Cited by: §I-B. [25] A. Milo, J. F. Einarsson, Ú. Einarsson, and F. Pind (2023) Treble auralizer: a real time web audio engine enabling 3dof auralization of simulated room acoustics designs. In 2023 Immersive and 3D Audio: from Architecture to Automotive (I3DA), p. 1–8. External Links: Document Cited by: §I-B. [26] G. Götz, D. G. Nielsen, S. Gudjónsson, and F. Pind (2025) Room-acoustic simulations as an alternative to measurements for audio-algorithm evaluation. IEEE Access 13, p. 214000–214008. Cited by: §I-B, §V-C. [27] F. Pind, J. F. Einarsson, S. Guðjónsson, M. Cosnefroy, J. Pedersen, J. B. Stefánsson, and A. Milo (2022) A novel wave-based virtual acoustics and spatial audio framework. In Audio Engineering Society Conference: 2022 AES International Conference on Audio for Virtual and Augmented Reality, Cited by: §I-B. [28] P. Puzalowski, J. Pedersen, and C. JEONG (2024) Optimizing a small room for critical listening with treble. In INTER-NOISE and NOISE-CON Congress and Conference Proceedings, Vol. 270, p. 2317–2328. Cited by: §I-B. [29] G. Götz, A. Milo, S. Guðjónsson, D. G. Nielsen, J. Pedersen, and F. Pind (2026) Improving multichannel speech enhancement through accurate room-acoustic simulations. In Proc. Interspeech, Cited by: §I-B, §I-B. [30] A. Milo, G. Götz, S. Guðjónsson, D. G. Nielsen, J. Pedersen, and F. Pind (2026) TRAINING deepfilternet with accurate room acoustic simulations improves single-channel speech enhancement. In Proc. Int. Workshop Acoust. Signal Enhancement (IWAENC), Cited by: §I-B. [31] A. Radford, J. W. Kim, T. Xu, G. Brockman, C. McLeavey, and I. Sutskever (2023) Robust speech recognition via large-scale weak supervision. In Proc. Int. Conf. Machine Learning (ICML), Vol. 202, p. 28492–28518. Cited by: §I-C, §IV-B, §V-A. [32] S. Gandhi, P. von Platen, and A. M. Rush (2023) Distil-whisper: robust knowledge distillation via large-scale pseudo labelling. Note: arXiv:2311.00430 Cited by: §I-C. [33] K. Kamahori, J. Kasai, N. Kojima, and B. Kasikci (2025) Liteasr: efficient automatic speech recognition with low-rank approximation. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, p. 3430–3442. Cited by: §I-C. [34] A. Gulati, J. Qin, C. Chiu, N. Parmar, Y. Zhang, J. Yu, W. Han, S. Wang, Z. Zhang, Y. Wu, and R. Pang (2020) Conformer: convolution-augmented transformer for speech recognition. In Proc. Interspeech, p. 5036–5040. External Links: Document Cited by: §I-C. [35] D. Rekesh, N. R. Koluguri, S. Kriman, S. Majumdar, V. Noroozi, H. Huang, O. Hrinchuk, K. C. Puvvada, A. Kumar, J. Balam, and B. Ginsburg (2023) Fast Conformer with linearly scalable attention for efficient speech recognition. In Proc. IEEE Automat. Speech Recognit. Understanding Workshop (ASRU), p. 1–8. External Links: Document Cited by: §I-C, §V-D. [36] A. Graves (2012) Sequence transduction with recurrent neural networks. Note: arXiv:1211.3711 Cited by: §I-C. [37] H. Xu, F. Jia, S. Majumdar, H. Huang, S. Watanabe, and B. Ginsburg (2023) Efficient sequence transduction by jointly predicting tokens and durations. In Proceedings of the 40th International Conference on Machine Learning, A. Krause, E. Brunskill, K. Cho, B. Engelhardt, S. Sabato, and J. Scarlett (Eds.), Proceedings of Machine Learning Research, Vol. 202, p. 38462–38484. External Links: Link Cited by: §I-C, §V-D. [38] K. C. Puvvada, P. Żelasko, H. Huang, O. Hrinchuk, N. R. Koluguri, K. Dhawan, S. Majumdar, E. Rastorgueva, Z. Chen, V. Lavrukhin, J. Balam, and B. Ginsburg (2024) Less is more: accurate speech recognition & translation without web-scale data. In Proc. Interspeech, p. 3964–3968. External Links: Document Cited by: §I-C. [39] Y. Peng, M. Shakeel, Y. Sudo, W. Chen, J. Tian, C. Lin, and S. Watanabe (2025) OWSM v4: improving open whisper-style speech models via data scaling and cleaning. In Proc. Interspeech, p. 2225–2229. External Links: Document Cited by: §I-C. [40] Y. Peng, Y. Sudo, M. Shakeel, and S. Watanabe (2024) OWSM-CTC: an open encoder-only speech foundation model for speech recognition, translation, and language identification. In Proc. Annual Meeting Assoc. Comput. Linguistics (ACL), External Links: Link Cited by: §I-C. [41] G. Saon, A. Dekel, A. Brooks, T. Nagano, et al. (2025) Granite-speech: open-source speech-aware LLMs with strong English ASR capabilities. Note: arXiv:2505.08699 Cited by: §I-C. [42] A. Baevski, Y. Zhou, A. Mohamed, and M. Auli (2020) Wav2vec 2.0: a framework for self-supervised learning of speech representations. In Advances in Neural Information Processing Systems (NeurIPS), Vol. 33, p. 12449–12460. Cited by: §I-C. [43] M. Ravanelli, T. Parcollet, A. Moumen, S. de Langen, C. Subakan, P. Plantinga, et al. (2024) Open-source conversational AI with SpeechBrain 1.0. J. Machine Learning Research 25 (333), p. 1–11. Cited by: §I-C. [44] N. Jeffries, E. King, M. Kudlur, G. Nicholson, J. Wang, and P. Warden (2024) Moonshine: speech recognition for live transcription and voice commands. Note: arXiv:2410.15608 Cited by: §I-C, §VI. [45] A. Koenecke, A. S. G. Choi, K. X. Mei, H. Schellmann, and M. Sloane (2024) Careless whisper: speech-to-text hallucination harms. In Proc. ACM Conf. Fairness, Accountability, and Transparency (FAccT), External Links: Document Cited by: §I-D, §IV-B, §V-A. [46] P. Götz, C. Tuna, A. Walther, and E. A. P. Habets (2022) AID: open-source anechoic interferer dataset. In Proc. Int. Workshop Acoust. Signal Enhancement (IWAENC), p. 1–5. External Links: Document Cited by: §I-B. [47] ITU-T (2011) Objective measurement of active speech level. Recommendation Technical Report ITU-T P.56, International Telecommunication Union. Cited by: §I-C. [48] S. S. Mullins and H. Sampedro Llopis (2026) Comparing three movement simulation algorithms from discrete impulse responses. In Proc. AES Int. Conf. on Audio for Virtual and Augmented Reality and Immersive Games, Paris, France. Cited by: §I-D. [49] Treble Technologies (2026) FFASR evaluation leaderboard. Note: Hugging Face Space External Links: Link Cited by: §IV-A. [50] A. C. Morris, V. Maier, and P. Green (2004) From WER and RIL to MER and WIL: improved evaluation measures for connected speech recognition. In Proc. Interspeech, p. 2765–2768. External Links: Document Cited by: §IV-B.