Paper deep dive

Understanding Multilingual Medical ASR Adaptation Through Layer-Wise Analysis

Souranil Kahali, Rituparna Bose, Abner Hernandez, Tomas Arias-Vergara, Andreas Maier, Ning Ma, Paula Andrea Perez-Toro

Year: 2026Venue: arXiv preprintArea: cs.CLType: PreprintEmbeddings: 42

Intelligence

Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 92%

Last extracted: 8/20/2026, 5:12:17 AM

Summary

This paper investigates the internal representation changes in Whisper models during multilingual medical automatic speech recognition (MedASR) adaptation. Through layer-wise encoder analysis, the authors find that English medical fine-tuning causes the dominant shift in encoder representations, while subsequent multilingual (English-German) continuation largely preserves this adapted space. Performance-wise, Whisper-Medium achieves the best results for English and combined EN+DE tasks, whereas German-only adaptation is limited by data scarcity. The study highlights that domain and language information remain recoverable across layers, but error-predictive cues weaken as WER improves.

Entities (9)

Whisper · model · 98%MedASR · task · 95%WER · metric · 95%Layer-wise Analysis · method · 93%Whisper-Medium · modelvariant · 92%Whisper-Small · modelvariant · 92%Kaggle Medical Speech · dataset · 90%PoCaP · dataset · 90%Linear CKA · metric · 88%

Relation Signals (7)

PoCaP → usedfor → German MedASR training

confidence 95% · German data are drawn from the PoCaP Corpus... for German-only diagnostic fine-tuning

Kaggle Medical Speech → usedfor → English MedASR training

confidence 95% · We use the publicly available Kaggle Medical Speech... dataset... for English-only fine-tuning

English medical fine-tuning → causes → dominant encoder shift

confidence 92% · English medical fine-tuning produces the dominant encoder shift, whereas multilingual continuation largely preserves the adapted representation space.

Whisper-Medium → achievesbestperformancefor → Combined EN+DE MedASR

confidence 90% · ...and the lowest combined EN+DE WER under direct EN+DE training (26.30%)

Whisper-Medium → achievesbestperformancefor → English MedASR

confidence 90% · Whisper-Medium gives the lowest English WER (7.72%)...

Multilingual continuation → preserves → adapted representation space

confidence 90% · multilingual continuation largely preserves the adapted representation space.

Linear CKA → measures → representational geometry changes

confidence 85% · Linear Centered Kernel Alignment (CKA) [23], which captures changes in representational geometry.

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Medical automatic speech recognition (MedASR) requires adaptation to specialised terminology, limited annotated clinical data, and multilingual use cases. Although large-scale pretrained ASR models such as Whisper achieve strong generalisation, their behaviour after medical and multilingual adaptation remains insufficiently understood beyond word error rate (WER). This paper investigates how multilingual medical adaptation reshapes the internal representations of Whisper models through layer-wise encoder analysis. We compare zero-shot decoding, English-only fine-tuning, German-only diagnostic fine-tuning, two-stage EN->EN+DE continuation, and direct EN+DE fine-tuning across Whisper model sizes. Fine-tuning substantially improves MedASR performance, but the best model depends on the adaptation setting: Whisper-Medium gives the lowest English WER (7.72%) and the lowest combined EN+DE WER under direct EN+DE training (26.30%); German-only Whisper-Large-v3 gives the lowest German WER (44.96%), but as a within-corpus diagnostic on 86 single-speaker training utterances rather than robust generalisation. Layer-wise analysis of the two-stage Whisper-Small trajectory shows that English medical fine-tuning produces the dominant encoder shift, whereas multilingual continuation largely preserves the adapted representation space. Domain and language information remain highly recoverable across layers, while linearly recoverable error-predictive cues weaken as WER improves.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

38,743 characters extracted from source content.

Expand or collapse full text

Understanding Multilingual Medical ASR Adaptation Through Layer-Wise Analysis Souranil Kahali 1⋆ , Rituparna Bose 1⋆ , Abner Hernandez 1 , Tomas Arias-Vergara 1 , Andreas Maier 1 , Ning Ma 2 , Paula A. Perez-Toro 1,3 1 Pattern Recognition Lab, Friedrich-Alexander-Universit ̈ at Erlangen-N ̈ urnberg (FAU), Erlangen, Germany 2 School of Computer Science, University of Sheffield, Sheffield, UK 3 Chair for AI in Healthcare and Medicine, Technical University of Munich (TUM), Munich, Germany ⋆ Equal contribution Abstract—Medical automatic speech recognition (MedASR) requires adaptation to specialised terminology, limited annotated clinical data, and multilingual use cases. Although large-scale pretrained ASR models such as Whisper achieve strong general- isation, their behaviour after medical and multilingual adaptation remains insufficiently understood beyond word error rate (WER). This paper investigates how multilingual medical adaptation reshapes the internal representations of Whisper models through layer-wise encoder analysis. We compare zero-shot decoding, English-only fine-tuning, German-only diagnostic fine-tuning, two-stage EN→EN+DE continuation, and direct EN+DE fine- tuning across Whisper model sizes. Fine-tuning substantially improves MedASR performance, but the best model depends on the adaptation setting: Whisper-Medium gives the lowest English WER (7.72%) and the lowest combined EN+DE WER under direct EN+DE training (26.30%); German-only Whisper- Large-v3 gives the lowest German WER (44.96%), but as a within-corpus diagnostic on 86 single-speaker training utterances rather than robust generalisation. Layer-wise analysis of the two- stage Whisper-Small trajectory shows that English medical fine- tuning produces the dominant encoder shift, whereas multilingual continuation largely preserves the adapted representation space. Domain and language information remain highly recoverable across layers, while linearly recoverable error-predictive cues weaken as WER improves. Index Terms—Medical ASR, Whisper Fine-tuning, Multilin- gual Adaptation, Layer-wise Analysis I. INTRODUCTION Automatic speech recognition (ASR) systems [1]–[3] have made substantial gains in recent years, through Transformer- based architectures and large-scale pretraining. Models such as Whisper [4] and Wav2Vec2 [5] achieve strong performance on general-purpose benchmarks like LibriSpeech [6] and Com- monVoice [7]. However, adapting these systems to domain- specific applications-particularly healthcare-presents distinct challenges. Medical speech contains rare and specialised ter- minology, recorded under varied speaker and acoustic condi- tions, and high accuracy demands where a single transcription error can carry clinical consequences. Generic ASR models trained on general speech corpora perform poorly in these settings, which motivates targeted domain adaptation. Prior work has shown that fine-tuning pretrained medical ASR (MedASR) models on domain-specific corpora substan- tially improves recognition accuracy [8], [9]. Multilingual MedASR has also been explored through initiatives such as MultiMed [10], which introduced benchmarks across clinical languages. In parallel, interpretability research has established that Transformer-based speech encoders organise information hierarchically: lower layers tend to encode acoustic and pho- netic information, while deeper layers capture abstract lin- guistic representations [11], [12]. However, these two research directions, domain adaptation and representation analysis, have largely developed independently. In particular, it remains un- clear how medical and multilingual fine-tuning reorganise the internal encoder representations of pretrained ASR models. Our work aims to close that gap by combining multi-size Whisper evaluation with systematic layer-wise analysis of how medical and multilingual adaptation reshape encoder represen- tations. We define the two-stage EN→EN+DE setting used throughout the paper as English medical fine-tuning followed by continued fine-tuning on combined English–German med- ical speech initialised from the English checkpoint. The main contributions are: (i) A systematic comparison of zero-shot, monolingual, continued EN→EN+DE, and direct EN+DE Whisper adaptation across four model sizes; (i) A layer-wise analysis showing that English medical fine-tuning produces the dominant representational change, whereas multilingual continuation largely preserves the English-adapted represen- tations; and (i) Evidence that encoder representations be- come less linearly predictive of transcription errors following adaptation, even as decoder WER improves. Fig. 1 provides a visual overview of the study design. It summarises the data and adaptation pipeline, the main ASR results, and the layer-wise analyses used to interpret the adapted encoder representations. I. RELATED WORK Transformer-based ASR. Large-scale pretrained ASR models are strong starting points for domain adaptation, but their WER performance depends heavily on the target domain. Whisper [4] is an encoder–decoder Transformer trained on large-scale weakly supervised multilingual audio, giving it strong zero-shot and cross-lingual ASR capabili- arXiv:2608.18825v1 [cs.CL] 19 Aug 2026 English medical speech German medical speech Pretrained Whisper Whisper encoder (L0--L12) Monolingual FT EN-only FT + DE-only diagnostic EN+DE FT Two-stage EN à EN+DE + Direct EN+DE L12 L11 L10 L9 L8 L7 L6 L5 L4 L3 L2 L1 L0 High-level semantics Low-level acoustics Whisper Encoder Linear CKA Layer DomainLanguage Layer Layer-wise Insights Data & Adaptation Pipeline Largest drift during EN medical fine-tuning Multilingual continuation causes only mild extra drift Domain and language information remain recoverable Error-predictive signal weakness after adaptation English German Upper layers (L8-L12) Lower layers (L0-L7) text EN-FT/DE-only = monolingual FT EN à EN+DE /Direct EN+DE = multilingual FT Layer Cross-lingual centroid alignment,but geometry remains language-sensitive Pretrained After Stage2 (EN+DE) Pretained After Stage 1(EN FT) After Stage 2(EN +DE) After Stage1 (EN FT) After Stage2 (EN+DE) Macro-F1 Macro-F1 Fig. 1. Overview of the multilingual MedASR adaptation and analysis pipeline. English and German medical speech are used for monolingual fine-tuning, two-stage EN→EN+DE continuation, and direct EN+DE fine-tuning. Adapted models are evaluated with WER, and Whisper-Small encoder representations are analysed through drift and probing tasks. ties. Wav2Vec2 [5] uses self-supervised contrastive pretrain- ing and achieves low WER on standard benchmarks after supervised fine-tuning. However, strong general-domain per- formance does not necessarily transfer to medical speech, where specialised terminology and variable acoustic conditions require targeted adaptation [8], [13]. Medical ASR. Fine-tuning pretrained ASR models on clin- ical corpora has consistently improved medical transcription performance [8], [10]. Adedeji et al. [8] showed that domain- specific fine-tuning, combined with large language model post- processing, improves medical ASR accuracy. The MultiMed project [10] introduced multilingual medical ASR across five languages, demonstrating the feasibility of adapting general ASR models to clinical speech. Its results are not directly comparable to ours because the datasets, languages, clinical scenarios, train/test splits, and decoding setup differ from the English Kaggle and German PoCaP evaluation used here. We therefore restrict direct empirical comparisons to models evaluated on the same test utterances under the same decoding and metric pipeline. Our work builds on this line by studying English–German medical adaptation across multiple Whisper fine-tuning strategies and by providing layer-level evidence of how adaptation changes encoder representations. Probing and layer-wise analysis. Belinkov and Glass [12] surveyed probing methods for analysing neural language mod- els, establishing the use of lightweight classifiers on frozen representations. Pasad et al. [11] applied layer-wise analysis to self-supervised speech models, showing that different encoder depths encode different phonological and linguistic properties. Prasad and Jyothi [14] probed accent information in end- to-end ASR systems. Wiepert et al. [15] examined layer selection for pathological speech feature prediction. To our knowledge, no prior work has jointly examined two-stage multilingual medical ASR adaptation and layer-wise probing of domain, language, and error-predictability signals in a fine- tuned encoder–decoder ASR model. Clinical speech analysis. Prior work on pathological speech has often combined acoustic and linguistic embeddings within Transformer-based pipelines [15], [16]. These approaches focus on supervised classification tasks. Our work takes a different angle: we analyse the encoder’s layer-wise behaviour for interpretability, without relying on diagnostic labels. I. DATA AND PRE-PROCESSING This section describes the medical speech datasets and pre- processing pipeline used for the monolingual and multilingual Whisper fine-tuning experiments. English Medical Corpus. We use the publicly available Kaggle Medical Speech, Transcription, and Intent dataset [17], comprising approximately 8.5 hours of spoken medical phrases with human-written transcriptions. Audio files are resampled to 16 kHz mono before fine-tuning. The dataset is partitioned into speaker-disjoint training, validation, and test splits using speaker identifiers: training (997 samples, 23 speakers), validation (136 samples, 3 speakers), and test (104 samples, 3 speakers). No speaker appears in more than one split, enabling speaker-disjoint evaluation. Mean transcript length is 49.4 characters in training and 52.1 characters in testing. German Medical Corpus. German data are drawn from the PoCaP Corpus [18], which contains port catheter placement procedures recorded at the Radiology Department of Univer- sity Hospital Erlangen, Germany. The dataset is not publicly available due to patient privacy regulations. We use audio from the operating physician only. Training uses operation OP- 001 (86 samples), while validation (37 samples) and test (38 samples) are file-disjoint halves of OP-002, ensuring no audio overlap. The same resampling and normalisation pipeline is applied as for the English corpus. Because the German training set is small and single-speaker, German results reflect within- corpus adaptation rather than cross-speaker generalisation. We therefore report German-only fine-tuning only as a diagnostic baseline, while the main multilingual comparisons use com- bined EN+DE training. Multilingual Dataset. The multilingual dataset is created by merging the English and German splits while preserving text labels and language identifiers. It contains 1,083 training, 173 validation, and 142 test samples, including 104 English and 38 German test utterances. During evaluation, each sample is decoded using language-specific forced decoder identifiers. Pre-processing and normalisation. Audio is converted to mono and resampled to 16 kHz. We use the corresponding Whisper feature extractor [4], [19] for log-Mel features (80-bin for Base/Small/Medium, 128-bin for Large-v3). Transcripts are kept in their original casing and punctuation; no text normalisation, denoising, voice-activity detection, or augmen- tation is applied. General English Contrast Set For the domain probing task, a 100-sample subset of LibriSpeech test-clean [6] is used as a general English contrast. Selecting samples with the same language as the medical set controls for language effects, ensuring that domain probe performance reflects domain- specific encoder content rather than language discrimination. Reproducibility. We will release the English split mani- fests, preprocessing scripts, evaluation code, and experiment configurations. The German PoCaP audio cannot be redis- tributed due to privacy restrictions, but split identifiers and evaluation scripts will be provided. IV. FINE-TUNING METHODOLOGY A. Training Configuration All fine-tuning experiments are implemented using Hugging Face Transformers [19] and PyTorch [20] on a single NVIDIA A100 GPU. We evaluate four fine-tuning settings: English- only medical fine-tuning, German-only diagnostic fine-tuning, two-stage EN→EN+DE continuation, and direct EN+DE fine- tuning from pretrained weights. Training uses Seq2SeqTrainer with cross-entropy loss, greedy decoding for validation WER, and a maximum generation length of 225 tokens. All runs use an effective batch size of 32, mixed precision, gradient checkpointing, and early stopping with patience 5 based on validation WER. Checkpoints are saved every 200 steps, and the best checkpoint is selected by lowest validation WER. We compute word error rate (WER), character error rate (CER), and SemScore [21], where SemScore is the mean cosine similarity between reference and prediction sentence embeddings using paraphrase-multilingual-MiniLM-L12-v2[22]. For validation and final evaluation, Whisper outputs are generated with greedy decoding and language-specific forced decoder identifiers implemented through the Hugging Face Transformers interface [4], [19]. English-only and German-only checkpoints are decoded on the corresponding monolingual test sets, while EN+DE checkpoints are decoded on language-filtered subsets of the multilingual test set and then pooled for the combined score. Wav2Vec2 baselines use CTC argmax decoding [5]. WER and CER are computed with jiwer on the decoded strings without additional case-folding or punctuation removal. Whisper- Small, Whisper-Medium, and Whisper-Large-v3 are trained under the same configuration; the selected checkpoints are step 400, step 800, and step 400, respectively. B. Monolingual Medical Fine-Tuning For English-only fine-tuning, each Whisper variant is trained on the English medical training set using the shared configuration, with best-checkpoint selection by validation WER. The processor language is fixed to English. The Whisper-Small EN-FT checkpoint used for layer-wise analysis is step 400 (11.98% English test WER); Whisper-Small is chosen because it balances adapted ASR performance with the compute needed for repeated hidden-state extraction across encoder layers. The German-only diagnostic follows the same configuration on the 86 German training utterances, with best-checkpoint selection by validation WER. Because this setting uses a small single-speaker German training set, only German test perfor- mance is reported and interpreted as within-corpus adaptation evidence. C. EN+DE Adaptation For two-stage adaptation, the English fine-tuned check- point is used to initialise continued training on the combined English–German dataset. This setting tests whether English medical adaptation can be preserved while introducing German medical speech. As a second multilingual baseline, direct EN+DE fine-tuning starts from pretrained Whisper weights and trains on the same combined English–German data in one stage. In both settings, no validation or test samples are used for gradient updates, batches mix both languages, and samples are decoded with language-specific forced decoder identifiers during evaluation. For the layer-wise analysis, we use the two-stage Whisper-Small multilingual fine-tuned (ML- FT) checkpoint from step 1,400, which achieves 10.91% English and 53.87% German WER. V. LAYER-WISE ANALYSIS We analyse Whisper-Small encoder representations to ex- amine how medical and multilingual fine-tuning affect the internal representation space. The analysis combines hidden- state extraction, representation-drift measurement, and probing classifiers for domain, language, and error-related information. A. Hidden-State Extraction Whisper-small has 12 encoder Transformer blocks. Includ- ing the input embedding output, we extract 13 hidden states per sample (L0–L12). Each hidden state has shape T × 384, where T is the sequence length. We mean-pool each layer over time to obtain one 384-dimensional vector per audio sample and layer. Hidden states are extracted independently from the Pretrained, EN-FT, and multilingual ML-FT checkpoints. B. Representation Drift Representation drift measures how encoder hidden states change after fine-tuning without using task labels. For the same 100 English audio samples, we compute per-layer similarity between checkpoints using two complementary metrics: co- sine similarity between mean representations, which captures directional alignment, and Linear Centered Kernel Alignment (CKA) [23], which captures changes in representational ge- ometry. We analyse three pairs: Pretrained → EN-FT, EN- FT → ML-FT, and ML-FT English versus ML-FT German. C. Probing Classifiers. We evaluated three binary probing tasks to analyse what information is encoded across layers. Domain probe classifies English medical speech (positive) vs. LibriSpeech [6] general English speech. This controls for language but may also reflect corpus-specific differences in accent, speaker population, recording conditions, and back- ground noise. TABLE I ASR WER (%) ON ENGLISH AND GERMAN MEDICAL SPEECH TEST SETS. BASE (B), SMALL (S), MEDIUM (M) AND LARGE (L) WHISPER VERSIONS WHERE ADAPTED. ModelAdaptEnglish↓German↓Combined↓ Zero-shot Whisper B None21.3070.4244.65 Whisper S None17.5762.9839.15 Whisper M None16.7771.99 † 43.02 Whisper L None16.5056.5135.52 Monolingual (Mono) medical fine-tuning Whisper B Mono15.0058.86– Whisper S Mono11.9857.69– Whisper M Mono7.7245.94– Whisper L Mono14.3744.96– Two-stage EN→EN+DE (Two-stage) medical fine-tuning Whisper B Two-stage14.9158.8635.80 Whisper ‡ S Two-stage10.9153.8731.33 Whisper M Two-stage7.9957.4931.52 Whisper L Two-stage13.2253.0932.17 Direct multilingual (Multi) medical fine-tuning Whisper B Multi14.6461.9037.10 Whisper S Multi9.9452.2030.03 Whisper M Multi7.8146.7226.30 Whisper L Multi15.3547.5030.63 Mono: EN-only (English) / DE-only diagnostic (German). Multi: direct EN+DE. † Wide CI. ‡ Whisper-S two-stage = layer-wise analysis checkpoint. Language probe classifies English medical vs. German med- ical speech. Because the two language subsets come from different corpora, the contrast may also reflect speaker and acoustic-condition differences. These confounds should be considered when interpreting the near-ceiling probe F1 scores reported in Sec. VI.C. WER probe classifies low- vs. high-WER utterances. WER is computed per checkpoint using greedy decoding on 100 En- glish test clips and labelled by rank-based tertile split: bottom- third (low-WER, label=0) vs. top-third (high-WER, label=1), middle third excluded (n=33 per class, applied identically across checkpoints). For each encoder layer, variable-length hidden-state se- quences were converted into fixed-dimensional representations by mean-pooling across the temporal dimension. Each task uses Logistic Regression, Linear SVC, and a one-hidden-layer MLP trained on frozen encoder features with stratified 5-fold cross-validation [24]. Panel-mean macro-F1 is the primary metric. Features are MinMax-scaled and reduced to 50 princi- pal components to limit overfitting under small sample sizes. To assess robustness, representation drift and probing analyses are repeated over five bootstrap seeds: 42, 123, 456, 789, and 2024. Layer-wise differences are tested using Kruskal– Wallis tests with Bonferroni and FDR-BH correction [25]. Within each fold, MinMax scaling and PCA reduction to 50 components are fitted only on the training split and then applied to the held-out validation split, preventing validation- fold information from entering preprocessing. VI. EXPERIMENTS & RESULTS This section reports ASR performance and representation- level analyses. We first compare zero-shot and fine-tuned Whisper models with Wav2Vec2 baselines, then analyse Whisper-Small encoder representations to study how medical and multilingual adaptation affect layer-wise information. A. ASR Performance: Zero-Shot and Fine-Tuned Tab. I reports zero-shot and fine-tuned MedASR WER results across all models. Among zero-shot Whisper mod- els, Whisper-Large-v3 performs best on all subsets (English 16.50%, German 56.51%, combined 35.52%). The Wav2Vec2 CTC baselines show poor out-of-domain transfer, with English WER above 92% and German XLSR-53 reaching 76.00%, confirming the need for domain adaptation in this medical setting. Fine-tuning changes the model ranking. English-only fine- tuning gives the best English result with Whisper-Medium (7.72% WER), but German transfer without German training is inconsistent: Whisper-Small degrades on German, whereas Whisper-Medium improves relative to its zero-shot baseline. The German-only diagnostic shows useful within-corpus adap- tation signal, with Whisper-Large-v3 reaching 44.96% WER and Whisper-Medium 45.94%. Since this setting uses only 86 German training utterances, we do not interpret it as robust German generalisation. For multilingual training, two-stage EN→EN+DE adap- tation improves over zero-shot for all Whisper sizes, with Whisper-Small giving the lowest two-stage combined WER (31.33%). Direct EN+DE fine-tuning gives the best over- all combined result, with Whisper-Medium reaching 26.30% TABLE I REPRESENTATIVE TRANSCRIPTION ERRORS. ZS-SMALL / S2-SMALL = WHISPER-SMALL ZERO-SHOT / TWO-STAGE EN→EN+DE; DE-LARGE = GERMAN-ONLY WHISPER-LARGE-V3. ITALICS MARK THE CONTESTED TERM. TypeUtterances ENsymptom term REF: stomach pain and bloating; ZS: stomach pain and rotting; S2: stomach pain and bloating. EN clinical term REF: could I have a concussion; ZS: could I have a competition; S2: could I have a concussion. DEimaging term REF: R ̈ ontgenthorax, Pneu; ZS: R ̈ ontgen Torax, Pneuer; S2: r ̈ ontgen torax, panor; DE-Large: r ̈ ontgen thorax, pneu. DEanatomy terms REF: Klavikola, Vene, Terumo-Nadel; ZS: Gravikula, Wene, Terumo; S2: gravikulare, vene, pteromunadel; DE-Large: klavikula, vene, terumonadel. DE hallucination REF: die Prostater dr ̈ uben; ZS/S2: Brotzeit verdr ̈ ucken/verdr ̈ uben; DE-Large: die Prostata dr ̈ uben. combined WER and 46.72% German WER. This may re- flect the strong EN–DE data imbalance (997 vs. 86 train- ing utterances): direct EN+DE training learns a shared mul- tilingual representation from the start, whereas continued EN→EN+DE fine-tuning starts from an English-specialised checkpoint. Overall, model scale alone does not determine adapted MedASR performance; the strongest model depends on whether the objective is zero-shot robustness, English adaptation, German within-corpus adaptation, or combined EN+DE performance. Because the layer-wise analysis below is conducted on the two-stage Whisper-Small trajectory, the representation- level findings should be interpreted as explaining that specific adaptation path; comparable analyses of direct EN+DE and German-only encoders are left for future work. Qualitative error audit. To complement aggregate WER, Tab. I shows representative examples from three checkpoints: zero-shot Whisper-Small (ZS-Small), two-stage Whisper- Small EN→EN+DE (S2-Small, the layer-wise model), and German-only Whisper-Large-v3 (DE-Large, 44.96% German WER). English examples compare ZS-Small and S2-Small; DE-Large is included for German rows only, where it achieves the best German WER. B. Representation Drift The drift analysis in Fig. 2 reveals a clear asymmetry between the two adaptation stages. Pretrained→EN-FT pro- duces the largest representational shift, most pronounced in upper layers (L8–L12), indicating that English medical fine- tuning drives the main encoder reorganisation. In contrast, EN-FT→ML-FT shows substantially smaller drift, suggesting that multilingual continuation largely preserves the English- adapted representation space. The ML-FT English vs. German comparison yields high cosine similarity (0.980–0.994) but low Linear CKA (0.040– 0.107), indicating centroid-level cross-lingual alignment while the full representation geometry remains language-sensitive. Tab. I summarises the bootstrap-averaged per-layer ranges. 0123456789101112 Encoder layer 0.88 0.90 0.92 0.94 0.96 0.98 1.00 Cosine similarity Representation drift — Cosine similarity Pretrained → EN-FT EN-FT → ML-FT ML-FT: EN vs DE 0123456789101112 Encoder layer 0.0 0.2 0.4 0.6 0.8 1.0 Linear CKA Representation drift — Linear CKA Pretrained → EN-FT EN-FT → ML-FT ML-FT: EN vs DE Fig. 2. Bootstrap-averaged representation drift across encoder layers (L0– L12) over five seeds; shading denotes ±1 std. Cosine similarity is shown on the left and Linear CKA on the right. Pretrained→EN-FT produces the largest upper-layer shift, whereas EN-FT→ML-FT remains close to the adapted encoder. ML-FT EN vs. DE shows high cosine similarity but low Linear CKA, indicating centroid alignment with language-sensitive geometry. TABLE I REPRESENTATION DRIFT SUMMARY OVER FIVE BOOTSTRAP SEEDS. VALUES REPORT THE PER-LAYER RANGE ACROSS L0–L12. Drift pairCosine rangeLin. CKA range Pretrained → EN-FT0.906–1.0000.946–1.000 EN-FT → ML-FT0.989–1.0000.988–1.000 ML-FT: EN vs. DE0.980–0.9940.040–0.107 C. Probing Results Domain probe. Fig. 3(a) shows that domain information re- mains highly separable across all checkpoints and layers, with panel-mean macro-F1 at or near ceiling (≥ 0.984; see Tab. IV). Because many layers achieve nearly identical F1, exact peak- layer claims are not meaningful. The main finding is that medical vs. general speech remains consistently recoverable from frozen encoder representations across adaptation stages. For ML-FT, silhouette scores decrease from 0.33 at L0 to 0.11 at L11, suggesting that deeper-layer domain representations remain classifiable but less geometrically compact. Language Probe. Fig. 3(b) and Tab. IV show that language- probe F1 remains near ceiling under bootstrap aggrega- tion. Language identity is almost perfectly separable across checkpoints and layers, with bootstrap mean F1 ≥ 0.990 from L1 onward across all checkpoints. This indicates that Whisper’s multilingual pretraining encodes robust language- discriminative features that are preserved through both fine- tuning stages. Since performance is at ceiling across nearly all depths, the result is best interpreted as evidence of preserved language-discriminative structure rather than a meaningful single-layer peak. WER/Error-PredictabilityProbe.TheWER/error- predictability probe in Fig. 3(c) and Tab. IV shows an overall decrease in linearly recoverable error information after adaptation, although the layer-wise curves fluctuate across encoder depth. Using rank-based low-/high-WER tertile labels, the best-layer panel-mean macro-F1 drops from 0.721± 0.028 at L2 for the Pretrained encoder (mean across layers: 0.666), to 0.619 ± 0.039 at L6 after EN-FT (mean: 0.569), and to 0.556 ± 0.033 at L11 after ML-FT 0123456789101112 Encoder layer 0.86 0.88 0.90 0.92 0.94 0.96 0.98 1.00 1.02 Probe macro-F1 (a) Domain probe Medical English vs. LibriSpeech English Pretrained EN-FT ML-FT 0123456789101112 Encoder layer 0.86 0.88 0.90 0.92 0.94 0.96 0.98 1.00 1.02 Probe macro-F1 (b) Language probe Medical English vs. Medical German Pretrained EN-FT ML-FT 0123456789101112 Encoder layer 0.3 0.4 0.5 0.6 0.7 Probe macro-F1 (c) Error-predictability probe Low-WER vs. High-WER English utterances Pretrained EN-FT ML-FT Fig. 3. Layer-wise probe macro-F1 across encoder layers (L0–L12) for the three probing tasks, bootstrap-aggregated over five seeds (shading: ±1 std). (a) Domain probe (medical English vs. LibriSpeech) and (b) Language probe (English medical vs. German medical) remain near ceiling across all checkpoints and layers, so peak-layer claims are not meaningful here. (c) Error-predictability probe uses rank-based low-/high-WER tertile labels. The pretrained encoder carries the strongest linearly recoverable error signal at the best layer; this signal weakens after English fine-tuning and multilingual continuation while decoder WER improves on the same 100 clips (19.87%→13.61%→12.48%). (mean: 0.509, close to binary-chance level). In parallel, decoder WER on the same 100 English clips decreases from 19.87% to 13.61% and 12.48%. Thus, as decoding improves, residual errors become less linearly predictable from frozen encoder activations, suggesting that adaptation reorganises failure modes that were more separable in the pretrained representation space. Kruskal–Wallis tests pooling 13 layers over five bootstrap seeds (n=65 observations per test) show significant layer-wise variation for all domain and language probes after Bonferroni and FDR-BH correction (all adjusted q < 10 −4 ). For the WER probe, only the pretrained checkpoint remains significant after correction (p = 1.47× 10 −3 ; Bonferroni q = 1.32× 10 −2 ; FDR-BH q = 1.89× 10 −3 ), whereas EN-FT (p = 0.051) and ML-FT (p = 0.213) are not significant under either correction. Domain and language information thus remain robustly layer- dependent, while error-predictive information becomes weaker and no longer significantly layer-localised after fine-tuning. TABLE IV PROBE SUMMARY OVER FIVE BOOTSTRAP SEEDS. DOMAIN AND LANGUAGE REPORT THE MINIMUM MACRO-F1 ACROSS LAYERS; WER F1 REPORTS THE BEST-LAYER LOW-/HIGH-WER PROBE RESULT UNDER A RANK-BASED TERTILE SPLIT (n=33 PER CLASS). Checkpoint Domain F1 Lang. F1WER F1Dec. WER minminbest layer(%) Pretrained ≥ 0.984 ≥ 0.990 0.721± 0.028 (L2)19.87 EN-FT ≥ 0.984 ≥ 0.993 0.619± 0.039 (L6)13.61 ML-FT ≥ 0.985 ≥ 0.993 0.556± 0.033 (L11)12.48 Dec. WER is computed on the same 100 English utterances used to label the WER probe (greedy decoding) and is not layer-wise. VII. DISCUSSION & CONCLUSION This work investigates medical ASR using multi-size Whis- per evaluation, English–German adaptation strategies, and layer-wise encoder analysis. Fine-tuning substantially im- proves MedASR over zero-shot decoding, but model scale alone does not determine the best adapted model under limited clinical data: Whisper-Medium is strongest under English- only and direct EN+DE fine-tuning, while Whisper-Large-v3 leads under the German-only diagnostic, which we interpret cautiously as within-corpus adaptation on 86 single-speaker training utterances. The layer-wise analysis yields three messages. (i) Med- ical adaptation dominates representational change. En- glish medical fine-tuning produces the largest encoder drift, concentrated in upper layers. (i) Multilingual continua- tion preserves learned representations. EN-FT→ML-FT drift is substantially smaller, and the domain and language probe contrasts remain highly separable across layers; their near-ceiling separability partly reflects corpus, speaker, and acoustic-condition confounds (Sec. V-C), so exact peak-layer claims should be treated cautiously. (i) Improved ASR does not imply stronger linear error signals. The WER probe weakens after each adaptation step even as decoder WER improves, suggesting that fine-tuning reduces simple linearly recoverable failure cues in the encoder. These findings complement standard WER evaluation and suggest several directions. The persistence of domain and lan- guage information across layers motivates parameter-efficient or selective fine-tuning. The weakening of WER-predictive signals suggests a role for difficulty-aware data selection using pretrained encoder representations. The strong separability of the English–German medical contrast motivates further study of shared multilingual encoder architectures for clinical ASR. Limitations. The medical corpora are small, particularly the single-procedure German subset (86 train / 38 test), and probing uses 100–138 utterances per task; the layer-wise anal- ysis covers only Whisper-Small. Fine-grained layer-specific, cross-speaker, and cross-scale conclusions should therefore be treated cautiously. Future work should validate these findings on larger multilingual clinical corpora and extend the layer- wise analysis to larger Whisper variants, direct EN+DE, German-only, and parameter-efficient adaptation strategies. VIII. GENERATIVE AI USE DISCLOSURE Generative artificial intelligence tools were used to assist with language editing, clarity of presentation, and code draft- ing/debugging. All research ideas, methodology, experiments, analyses, and interpretations were conceived, verified, and carried out by the authors, who take full responsibility for the originality, validity, and integrity of the work. REFERENCES [1] A. Graves, A.-r. Mohamed, and G. Hinton, “Speech recognition with deep recurrent neural networks,” in 2013 IEEE International Conference on Acoustics, Speech and Signal Processing, 2013, p. 6645–6649. [2] W. Chan, N. Jaitly, Q. Le, and O. Vinyals, “Listen, attend and spell: A neural network for large vocabulary conversational speech recognition,” in 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).IEEE Press, 2016, p. 4960–4964. [Online]. Available: https://doi.org/10.1109/ICASSP.2016.7472621 [3] Q. Zhang, H. Lu, H. Sak, A. Tripathi, E. McDermott, S. Koo, and S. Kumar, “Transformer transducer: A streamable speech recognition model with transformer encoders and rnn-t loss,” in ICASSP 2020- 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2020, p. 7829–7833. [4] A. Radford, J. W. Kim, T. Xu, G. Brockman, C. McLeavey, and I. Sutskever, “Robust speech recognition via large-scale weak supervi- sion,” in International conference on machine learning. PMLR, 2023, p. 28 492–28 518. [5] A. Baevski, Y. Zhou, A. Mohamed, and M. Auli, “wav2vec 2.0: A framework for self-supervised learning of speech representations,” Advances in neural information processing systems, vol. 33, p. 12 449– 12 460, 2020. [6] V. Panayotov, G. Chen, D. Povey, and S. Khudanpur, “Librispeech: an asr corpus based on public domain audio books,” in 2015 IEEE international conference on acoustics, speech and signal processing (ICASSP). IEEE, 2015, p. 5206–5210. [7] R. Ardila, M. Branson, K. Davis, M. Henretty, M. Kohler, J. Meyer, R. Morais, L. Saunders, F. M. Tyers, and G. Weber, “Common voice: A massively-multilingual speech corpus,” 2020. [Online]. Available: https://arxiv.org/abs/1912.06670 [8] A. Adedeji, S. Joshi, and B. Doohan, “The sound of healthcare: Im- proving medical transcription asr accuracy with large language models,” arXiv preprint arXiv:2402.07658, 2024. [9] A. Hernandez, T. Arias-Vergara, A. Maier, and P. A. P ́ erez-Toro, “Enhancing ASR accuracy for speakers with parkinson’s disease using instruction-tuned LLMs,” in International Conference on Text, Speech, and Dialogue. Springer, 2025, p. 153–164. [10] K. Le-Duc, P. Phan, T.-H. Pham, B. P. Tat, M.-H. Ngo, T. Nguyen-Tang, and T.-S. Hy, “Multimed: Multilingual medical speech recognition via attention encoder decoder,” in Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 6: Industry Track), 2025, p. 1113–1150. [11] A. Pasad, J.-C. Chou, and K. Livescu, “Layer-wise analysis of a self-supervised speech representation model,” in 2021 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU). IEEE, 2021, p. 914–921. [12] Y. Belinkov and J. Glass, “Analysis methods in neural language processing:Asurvey,”TransactionsoftheAssociationfor Computational Linguistics, vol. 7, p. 49–72, 2019. [Online]. Available: https://aclanthology.org/Q19-1004/ [13] Y. Liu, X. Yang, and D. Qu, “Exploration of whisper fine-tuning strategies for low-resource asr,” EURASIP Journal on Audio, Speech, and Music Processing, vol. 2024, no. 1, p. 29, 2024. [14] A. Prasad and P. Jyothi, “How accents confound: Probing for accent in- formation in end-to-end speech recognition systems,” in Proceedings of the 58th annual meeting of the association for computational linguistics, 2020, p. 3739–3753. [15] D. A. Wiepert, R. L. Utianski, J. R. Duffy, J. L. Stricker, L. R. Barnard, D. T. Jones, and H. Botha, “Speech foundation models in healthcare: Effect of layer selection on pathological speech feature prediction,” arXiv preprint arXiv:2402.01796, 2024. [16] P. A. P ́ erez-Toro, S. P. Bayerl, T. Arias-Vergara, J. C. V ́ asquez-Correa, P. Klumpp, M. Schuster, E. N ̈ oth, J. R. Orozco-Arroyave, and K. Ried- hammer, “Influence of the interviewer on the automatic assessment of alzheimer’s disease in the context of the adresso challenge.” in Interspeech, 2021, p. 3785–3789. [17] P.Mooney,“Medicalspeech,transcription,andintent dataset,”https://w.kaggle.com/datasets/paultimothymooney/ medical-speech-transcription-and-intent, 2020, accessed: June 2026. [18] K. C. Demir, M. May, A. Schmid, M. Uder, K. Breininger, T. Weise, A. Maier, and S. H. Yang, “Pocap corpus: A multimodal dataset for smart operating room speech assistant using interventional radiology workflow analysis,” in International conference on text, speech, and dialogue. Springer, 2022, p. 464–475. [19] T. Wolf, L. Debut, V. Sanh, J. Chaumond, C. Delangue, A. Moi, P. Cistac, T. Rault, R. Louf, M. Funtowicz, J. Davison, S. Shleifer, P. von Platen, C. Ma, Y. Jernite, J. Plu, C. Xu, T. L. Scao, S. Gugger, M. Drame, Q. Lhoest, and A. M. Rush, “Huggingface’s transformers: State-of-the-art natural language processing,” 2020. [Online]. Available: https://arxiv.org/abs/1910.03771 [20] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. K ̈ opf, E. Yang, Z. DeVito, M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner, L. Fang, J. Bai, and S. Chintala, “Pytorch: An imperative style, high-performance deep learning library,” 2019. [Online]. Available: https://arxiv.org/abs/1912.01703 [21] A. Aynetdinov and A. Akbik, “Semscore: Automated evaluation of instruction-tuned llms based on semantic textual similarity,” 2024. [Online]. Available: https://arxiv.org/abs/2401.17072 [22] N. Reimers and I. Gurevych, “Sentence-bert: Sentence embeddings using siamese bert-networks,” 2019. [Online]. Available: https: //arxiv.org/abs/1908.10084 [23] S. Kornblith, M. Norouzi, H. Lee, and G. Hinton, “Similarity of neural network representations revisited,” in International conference on machine learning. PMlR, 2019, p. 3519–3529. [24] G. Alain and Y. Bengio, “Understanding intermediate layers using linear classifier probes,” 2018. [Online]. Available: https://arxiv.org/abs/ 1610.01644 [25] Y. Benjamini and Y. Hochberg, “Controlling the false discovery rate: A practical and powerful approach to multiple testing,” Journal of the Royal Statistical Society. Series B (Methodological), vol. 57, no. 1, p. 289–300, 1995. [Online]. Available: http://w.jstor.org/stable/2346101

Instant research discovery