Paper deep dive
CAL-MOS: Bridging Layers with Adapters for Robust MOS Prediction Across Speech Foundation Models
Alef Iury Siqueira Ferreira, Pedro Lustosa Rege Botelho, Fernanda Silva, Daniel Casanova, Rafael Faustino, Frederico Oliveira, Arlindo Galvão Filho, Anderson da Silva Soares
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Speech Quality Assessment (SQA) is essential for modern speech technologies, and recent non-intrusive SQA predictors increasingly rely on Speech Foundation Models (SFMs). However, because SFMs expose representations from many layers, it remains unclear which depths are most informative for MOS prediction and how multi-layer information should be combined reliably across backbones and datasets. We benchmark ten SFMs on four MOS datasets under three regimes: full fine-tuning, last-layer probing with a frozen encoder, and naive cross-layer weighted aggregation. We find that the best layer is strongly backbone- and dataset-dependent, and that naive weighted fusion can be unstable across settings. We further evaluate a layer-calibrated aggregation variant that applies per-layer adapters before pooling, which improves the robustness of multi-layer fusion and narrows the gap to full fine-tuning while keeping the backbone frozen.
Tags
Links
- Source: https://arxiv.org/abs/2609.14956v1
- Canonical: https://arxiv.org/abs/2609.14956v1
Trouble viewing inline? Open PDF directly →
Full Text
38,768 characters extracted from source content.
Expand or collapse full text
Ferreira Lustosa Rege Botelho Silva Casanova Faustino Oliveira Filho da Silva Soares CAL-MOS: Bridging Layers with Adapters for Robust MOS Prediction Across Speech Foundation Models Alef Iury Pedro Fernanda Daniel Rafael Frederico Arlindo Galvão Anderson Abstract Speech Quality Assessment (SQA) is essential for modern speech technologies, and recent non-intrusive SQA predictors increasingly rely on Speech Foundation Models (SFMs). However, because SFMs expose representations from many layers, it remains unclear which depths are most informative for MOS prediction and how multi-layer information should be combined reliably across backbones and datasets. We benchmark ten SFMs on four MOS datasets under three regimes: full fine-tuning, last-layer probing with a frozen encoder, and naive cross-layer weighted aggregation. We find that the best layer is strongly backbone- and dataset-dependent, and that naive weighted fusion can be unstable across settings. We further evaluate a layer-calibrated aggregation variant that applies per-layer adapters before pooling, which improves the robustness of multi-layer fusion and narrows the gap to full fine-tuning while keeping the backbone frozen. keywordsSpeech Quality Assessment, Self-Supervised Learning, Speech Foundation Models †address: 1 AKCIT, Brazil 2 Federal University of Goiás (UFG), Brazil 3 Federal University of Rio Grande do Norte (UFRN), Brazil 4 Federal University of Technology (UTFPR), Brazil †email: alef_iury_c.c@discente.ufg.br 1 Introduction Recent progress in speech synthesis, enhancement, and related speech generation technologies has increased the need for reliable quality evaluation methods [1, 2]. Although objective metrics are efficient, they often fail to reflect the nuances of human perception [1, 3, 4]. As a result, subjective Mean Opinion Score (MOS) listening tests remain the standard protocol for quality assessment [5]. However, MOS testing is notoriously expensive, time-consuming, and difficult to scale. It is also affected by annotator variability and other sources of bias, especially for high-quality modern systems [1, 6, 2]. These limitations motivate automatic, non-intrusive MOS prediction as a practical alternative. Much of the recent progress in non-intrusive MOS prediction has been driven by large pre-trained speech encoders, or Speech Foundation Models (SFMs). Prior work has either fine-tuned the full encoder with a regression head, achieving strong performance at a higher computational cost [7, 8, 9], or trained lightweight predictors on frozen SFM representations for greater efficiency [10]. Additional studies have combined SFM embeddings with complementary acoustic features to improve robustness [11]. Together, these results establish SFMs as strong backbones for MOS prediction, but leave open the question of how best to exploit their internal representations. This problem arises because different layers capture fundamentally different types of information. Earlier and intermediate layers often retain stronger acoustic and phonetic cues, whereas deeper layers are increasingly shaped by the pre-training objective [12, 13]. For MOS prediction, where cues such as noise, distortion, articulation, and prosody are all relevant, this distinction is especially important [14]. Recent studies have indeed shown that early-to-mid layers can outperform the final layer for MOS prediction across several backbones [15, 16]. Motivated by this, we revisit MOS prediction from SFMs as a practical problem of layer utilization. We benchmark ten SFMs on four MOS datasets under a unified protocol, comparing last-layer probing, best-layer selection, naive cross-layer weighted aggregation, and full fine-tuning. We further evaluate a layer-calibrated aggregation variant based on per-layer adapters, inspired by prior work in speaker verification [17, 18], to study whether calibrating layer outputs before fusion improves the robustness of MOS prediction. Our contributions are threefold: (1) an empirical study of layer utilization for MOS prediction across diverse speech foundation models, datasets, and training regimes, offering practical guidance on when to use the last layer, an intermediate layer, cross-layer aggregation, or full fine-tuning; (2) showing that the most informative depth is strongly backbone- and dataset-dependent, and that naive weighted-sum fusion is not a reliable general solution across backbones and datasets; and (3) showing that applying prior layer calibration before aggregation provides a stronger and more robust frozen-backbone alternative than naive weighted aggregation, often narrowing the gap to full fine-tuning. 2 Related Works SQA has shifted from traditional signal-based methods to data-driven deep learning systems. While early models like MOSNet [4] marked significant progress, the current paradigm is dominated by SFMs models. Fine-tuning self-supervised models such as wav2vec 2.0 [19] was shown to surpass earlier approaches for MOS prediction [19, 7], and subsequent studies confirmed the effectiveness of other large speech encoders, including HuBERT [8] and supervised models like Whisper-based [20] systems [10, 11]. Stronger systems such as UTMOS [9] further improved performance through more elaborate training pipelines, while MOSA-Net+ [11] combined Whisper embeddings with spectral features to improve robustness. A related line of work has examined which internal representations of pre-trained speech models are most useful for perceptual prediction. Layer-wise studies in speech representation learning show that intermediate layers often retain stronger acoustic and phonetic information, whereas deeper layers become increasingly specialized to the pre-training objective [12, 13, 14]. In MOS prediction, recent studies by Liang et al. [15] and Cooper et al. [16] similarly found that early-to-mid layers can outperform the final layer across several backbones and evaluation settings. Our work builds most directly on this line of research, but moves beyond single-layer selection to study how multiple layers can be used robustly across backbones and datasets. This connects to prior cross-layer aggregation work in other speech tasks, particularly speaker verification, where learnable layer weighting, multi-scale aggregation, and per-layer adapters have been used to better align intermediate representations before fusion [17, 18]. Motivated by these ideas, we compare last-layer probing, best-layer selection, naive weighted aggregation, and full fine-tuning under a unified MOS benchmark and further examine whether per-layer calibration can make cross-layer fusion more reliable for MOS prediction. 3 Methods We study non-intrusive MOS prediction using SFMs under a unified framework designed to isolate the effect of layer utilization. Given an input utterance, an SFM produces hidden representations at multiple depths. Our central question is how these layer-wise representations should be used for MOS prediction. We evaluate a diverse set of publicly available SFMs: wav2vec 2.0 Large [19], XLS-R [21] 300M and 1B, MMS [22] 300M and 1B, Wav2BERT 2.0 [23], WavLM Large [24], HuBERT Large [8], data2vec Large [25], and Whisper Large-v3 [20]. These models differ in training objective, supervision, scale, and pre-training data, making them suitable for studying whether layer-selection trends generalize across backbones. Our first strategy is Last Layer (L), where only the final hidden representation of the frozen backbone is used, serving as a standard feature-extraction baseline. Our second strategy is Best Layer (BL), which uses a single frozen intermediate layer selected from layer-wise analysis, testing whether careful layer selection alone is sufficient. Our third strategy is Weighted Sum (WS), where all layer representations are combined via learnable scalar weights normalized during training like done by [26], providing a simple form of cross-layer aggregation that assumes representations from different depths are already compatible enough to be fused directly. We also evaluate Full Fine-Tuning (FT), where the entire SFM and regression head are optimized jointly, serving as a strong baseline. In addition, motivated by the instability observed in naive cross-layer fusion, we introduce Adapter + Mean (A+M), a layer-calibrated aggregation strategy. Here, each layer is first passed through an independent adapter consisting of a linear projection, layer normalization, ReLU activation, and a second linear layer. The adapted sequences are then concatenated across layers and mean-pooled to form a fixed-dimensional utterance representation, allowing each layer to be transformed into a more task-aligned space before aggregation without requiring full backbone adaptation. After layer selection or aggregation, masked mean pooling over time produces an utterance-level embedding, which is passed to a Multi-Layer Perceptron (MLP) regression head consisting of a linear projection, a ReLU activation, and a final linear output layer; predictions are clipped to the [1,5][1,5] interval to match the MOS scale. All models are trained by minimizing Mean Squared Error (MSE) between predicted and reference MOS values, with a batch size of 64 and the AdamW optimizer (β1=0.9 _1=0.9, β2=0.98 _2=0.98, ϵ=1×10−8ε=1× 10^-8), a weight decay of 1×10−61× 10^-6, and norm-based gradient clipping (maximum threshold of 10). A cosine learning rate scheduler with a 500-step linear warm-up adjusts learning rates between 1×10−51× 10^-5 and 5×10−55× 10^-5. We evaluate on four MOS datasets: BVCC [8], BRSpeechMOS [10], SingMOS [27], and TMHINT-QI [28]. These datasets cover different languages, speech generation conditions, and rating scenarios. We report utterance-level and system-level results using MSE (Mean Square Error) and Spearman Rank Correlation Coefficient (SRCC). To compare models under a consistent budget, we first run a 20-epoch screening stage across all candidate backbones and training regimes to identify strong models and characterize broad trends. We then perform detailed layer-wise analysis on the most competitive backbones and extend training to 100 epochs for the final comparison. Table 1: Results of the 20-epoch screening stage, comparing model families using utterance-level and system-level MSE and SRCC on BRSpeech (BRS), BVCC (BVC), SingMOS (SM), and TMHINT-QI (TMH). Bold indicates the best result within each model family, while blue and gray cell backgrounds denote the overall best and second-best entries, respectively. Avg. Utt. and Avg. Sys. report averages across the four datasets at the utterance and system levels. L = Last Layer, WS = Weighted Sum, and FT = Fine-Tuning. BRS BVC SM TMH Avg. Method Utt. Sys. Utt. Sys. Utt. Sys. Utt. Sys. Utt. Sys. MSE SRCC MSE SRCC MSE SRCC MSE SRCC MSE SRCC MSE SRCC MSE SRCC MSE SRCC MSE SRCC MSE SRCC w2v2-L L 0.9900.990 0.3690.369 0.3540.354 0.7140.714 0.7750.775 0.4890.489 0.5890.589 0.6610.661 0.5780.578 0.1140.114 0.1830.183 0.4030.403 0.6070.607 0.5810.581 0.1340.134 0.7180.718 0.7380.738 0.3880.388 0.3150.315 0.6240.624 w2v2-L WS 0.7900.790 0.5920.592 0.072 0.943 0.2280.228 0.8560.856 0.1340.134 0.8980.898 0.5160.516 0.3810.381 0.1520.152 0.5620.562 0.378 0.743 0.040 0.935 0.478 0.643 0.099 0.835 w2v2-L FT 0.783 0.665 0.1010.101 0.943 0.190 0.881 0.100 0.924 0.395 0.591 0.040 0.876 1.5581.558 0.3950.395 0.6360.636 0.2080.208 0.7320.732 0.6330.633 0.2190.219 0.7380.738 XLSR-300M L 1.0071.007 0.4540.454 0.2870.287 0.6570.657 0.5360.536 0.6610.661 0.3990.399 0.7180.718 0.5460.546 0.2080.208 0.1830.183 0.1520.152 0.4770.477 0.6860.686 0.1000.100 0.8340.834 0.6410.641 0.5020.502 0.2420.242 0.5900.590 XLSR-300M WS 0.9650.965 0.5790.579 0.2620.262 0.8860.886 0.2280.228 0.8580.858 0.1240.124 0.9140.914 0.5690.569 0.4070.407 0.1660.166 0.6550.655 0.3560.356 0.7520.752 0.030 0.940 0.5290.529 0.6490.649 0.1460.146 0.8490.849 XLSR-300M FT 0.731 0.693 0.055 0.943 0.170 0.894 0.096 0.935 0.395 0.585 0.053 0.819 0.350 0.762 0.0410.041 0.9240.924 0.411 0.733 0.061 0.905 XLSR-1B L 0.9730.973 0.5130.513 0.2580.258 0.8860.886 0.2950.295 0.8220.822 0.1630.163 0.8730.873 0.4210.421 0.5320.532 0.1040.104 0.7520.752 0.3750.375 0.7520.752 0.0480.048 0.9300.930 0.5160.516 0.6550.655 0.1430.143 0.8600.860 XLSR-1B WS 0.8190.819 0.6040.604 0.2010.201 0.8860.886 0.2260.226 0.8650.865 0.1360.136 0.9180.918 0.4200.420 0.5280.528 0.0970.097 0.6990.699 0.3780.378 0.7590.759 0.0610.061 0.939 0.4610.461 0.6890.689 0.1240.124 0.8610.861 XLSR-1B FT 0.744 0.668 0.044 0.943 0.178 0.892 0.099 0.937 0.404 0.587 0.081 0.782 0.352 0.768 0.037 0.9360.936 0.419 0.729 0.065 0.899 MMS-300M L 1.2011.201 0.5230.523 0.4050.405 0.8860.886 0.5380.538 0.7600.760 0.3730.373 0.7800.780 0.5800.580 0.3860.386 0.2030.203 0.6350.635 0.4190.419 0.7270.727 0.0730.073 0.8610.861 0.6840.684 0.5990.599 0.2640.264 0.7900.790 MMS-300M WS 0.8450.845 0.5540.554 0.1890.189 0.8860.886 0.2140.214 0.8640.864 0.0960.096 0.9210.921 0.4480.448 0.4910.491 0.075 0.7210.721 0.345 0.7650.765 0.027 0.946 0.4630.463 0.6690.669 0.097 0.8690.869 MMS-300M FT 0.685 0.689 0.026 0.943 0.172 0.894 0.085 0.940 0.416 0.606 0.0940.094 0.811 0.5100.510 0.773 0.1930.193 0.9350.935 0.446 0.741 0.1000.100 0.907 MMS-1B L 1.0691.069 0.4770.477 0.3610.361 0.8860.886 0.3080.308 0.8090.809 0.2030.203 0.8630.863 0.4800.480 0.4540.454 0.1410.141 0.5750.575 0.3730.373 0.7530.753 0.0340.034 0.9430.943 0.5570.557 0.6230.623 0.1850.185 0.8170.817 MMS-1B WS 0.7880.788 0.6140.614 0.1200.120 0.8860.886 0.2370.237 0.8620.862 0.1370.137 0.9230.923 0.403 0.5650.565 0.070 0.8200.820 0.349 0.761 0.027 0.950 0.444 0.7010.701 0.089 0.8950.895 MMS-1B FT 0.758 0.678 0.111 0.943 0.170 0.893 0.076 0.942 0.4800.480 0.588 0.1620.162 0.828 0.4390.439 0.7570.757 0.1240.124 0.9340.934 0.4620.462 0.729 0.1180.118 0.912 w2v-BERT L 1.4301.430 −0.045-0.045 0.6180.618 −0.029-0.029 0.4860.486 0.7050.705 0.3320.332 0.7950.795 0.7180.718 −0.086-0.086 0.2540.254 −0.126-0.126 0.4690.469 0.7060.706 0.0820.082 0.8900.890 0.7760.776 0.3200.320 0.3220.322 0.3830.383 w2v-BERT WS 0.694 0.6530.653 0.0250.025 0.9430.943 0.2310.231 0.8580.858 0.1380.138 0.9050.905 0.4100.410 0.5280.528 0.0850.085 0.7520.752 0.348 0.767 0.028 0.955 0.421 0.7020.702 0.0690.069 0.8890.889 w2v-BERT FT 0.7030.703 0.682 0.011 1.000 0.199 0.886 0.115 0.928 0.386 0.604 0.054 0.826 0.4460.446 0.7600.760 0.0590.059 0.9390.939 0.4330.433 0.733 0.060 0.923 WavLM-L L 0.7930.793 0.6060.606 0.133 0.8860.886 0.2730.273 0.8240.824 0.1650.165 0.8790.879 0.5160.516 0.3960.396 0.1650.165 0.5950.595 0.3750.375 0.7530.753 0.0410.041 0.9290.929 0.4890.489 0.6450.645 0.1260.126 0.8220.822 WavLM-L WS 1.3331.333 0.5240.524 0.5310.531 0.8860.886 0.3080.308 0.8190.819 0.1390.139 0.8870.887 1.0401.040 0.3170.317 0.3880.388 0.6220.622 0.4460.446 0.7060.706 0.0470.047 0.9170.917 0.7820.782 0.5910.591 0.2760.276 0.8280.828 WavLM-L FT 0.746 0.703 0.1530.153 0.943 0.198 0.878 0.105 0.931 0.363 0.622 0.063 0.823 0.351 0.767 0.039 0.936 0.414 0.742 0.090 0.908 HuBERT-L L 0.7900.790 0.6210.621 0.0930.093 0.886 0.3190.319 0.8030.803 0.1780.178 0.8800.880 0.4740.474 0.4390.439 0.126 0.6140.614 0.3910.391 0.7350.735 0.0450.045 0.9220.922 0.4930.493 0.6490.649 0.1110.111 0.8260.826 HuBERT-L WS 1.5361.536 0.4390.439 0.7810.781 0.7710.771 0.3470.347 0.7910.791 0.1950.195 0.8650.865 1.0731.073 0.2860.286 0.3820.382 0.4600.460 0.4270.427 0.7030.703 0.042 0.9180.918 0.8460.846 0.5550.555 0.3500.350 0.7540.754 HuBERT-L FT 0.749 0.664 0.052 0.8290.829 0.230 0.886 0.160 0.933 0.438 0.623 0.1290.129 0.808 0.376 0.750 0.0460.046 0.928 0.448 0.731 0.097 0.875 d2v-L L 1.0821.082 0.4630.463 0.3970.397 0.8290.829 0.7950.795 0.2920.292 0.5760.576 0.4510.451 0.5360.536 0.2710.271 0.1620.162 0.2920.292 0.6940.694 0.5320.532 0.1950.195 0.5710.571 0.7770.777 0.3900.390 0.3330.333 0.5360.536 d2v-L WS 0.9680.968 0.5070.507 0.2830.283 0.8860.886 0.4980.498 0.7270.727 0.3680.368 0.8040.804 0.4930.493 0.4010.401 0.1520.152 0.5520.552 0.450 0.6970.697 0.068 0.8950.895 0.6020.602 0.5830.583 0.2180.218 0.7840.784 d2v-L FT 0.907 0.614 0.036 0.943 0.222 0.863 0.106 0.922 0.448 0.557 0.059 0.837 0.4920.492 0.753 0.1160.116 0.950 0.517 0.697 0.079 0.913 Whisper-Lv3 L 0.784 0.6250.625 0.030 0.8860.886 0.2610.261 0.8360.836 0.1460.146 0.8890.889 0.4250.425 0.5440.544 0.0990.099 0.7670.767 0.344 0.768 0.023 0.954 0.454 0.6930.693 0.075 0.8740.874 Whisper-Lv3 WS 1.0111.011 0.5130.513 0.2720.272 0.8290.829 0.3510.351 0.7870.787 0.2280.228 0.8490.849 0.4790.479 0.4370.437 0.1430.143 0.5500.550 0.3810.381 0.7450.745 0.0370.037 0.9370.937 0.5550.555 0.6210.621 0.1700.170 0.7910.791 Whisper-Lv3 FT 0.8090.809 0.640 0.0410.041 0.943 0.201 0.885 0.133 0.930 0.378 0.604 0.061 0.794 0.4940.494 0.7570.757 0.1560.156 0.9270.927 0.4700.470 0.722 0.0980.098 0.898 Figure 1: Layer-wise MOS prediction performance for five selected speech foundation models, reported as normalized SRCC across BRSpeech, BVCC, SingMOS, TMHINT-QI, and the average across datasets. Columns denote encoder layers and rows denote datasets. Markers indicate the top three layers in each row: best ( ), second-best (⧫ ), and third-best (▲ ). The results show substantial variation across backbones and datasets, with strong layers often appearing in the early-to-mid portion of the network rather than consistently at the final layer. Figure 2: Mean normalized layer-importance curves for five SFMs. Colored lines denote model-specific averages, the black line shows the global mean, and the gray band indicates variability across models. Table 2: Results of the 100-epoch main comparison on the five selected model families, evaluated with utterance-level and system-level MSE and SRCC on BRSpeech (BRS), BVCC (BVC), SingMOS (SM), and TMHINT-QI (TMH). Bold indicates the best result within each model family, while blue and gray cell backgrounds denote the overall best and second-best results, respectively. Avg. Utt. and Avg. Sys. report averages across the four datasets at the utterance and system levels. L = Last Layer, BL = Best Layer, WS = Weighted Sum, FT = Fine-Tuning, and A+M = Adapters + Mean. BRS BVC SM TMH Avg. Method Utt. Sys. Utt. Sys. Utt. Sys. Utt. Sys. Utt. Sys. MSE SRCC MSE SRCC MSE SRCC MSE SRCC MSE SRCC MSE SRCC MSE SRCC MSE SRCC MSE SRCC MSE SRCC WavLM-L L 0.7030.703 0.6860.686 0.057 0.943 0.2420.242 0.8490.849 0.1290.129 0.9100.910 0.3570.357 0.6080.608 0.0560.056 0.8350.835 0.347 0.7640.764 0.026 0.946 0.4120.412 0.7270.727 0.0670.067 0.9080.908 WavLM-L BL 0.7030.703 0.6860.686 0.057 0.943 0.2340.234 0.8650.865 0.1400.140 0.9140.914 0.3570.357 0.6080.608 0.0560.056 0.8350.835 0.347 0.7640.764 0.026 0.946 0.4100.410 0.7310.731 0.0700.070 0.9090.909 WavLM-L WS 0.8680.868 0.6250.625 0.0770.077 0.9430.943 0.2700.270 0.8450.845 0.1400.140 0.9080.908 0.4460.446 0.5530.553 0.0480.048 0.8470.847 0.4070.407 0.7300.730 0.0420.042 0.9220.922 0.4980.498 0.6880.688 0.0770.077 0.9050.905 WavLM-L FT 0.7200.720 0.6890.689 0.0670.067 0.9430.943 0.188 0.885 0.098 0.930 0.3760.376 0.6020.602 0.0500.050 0.8490.849 0.3840.384 0.772 0.0820.082 0.9430.943 0.4170.417 0.7370.737 0.0740.074 0.9160.916 WavLM-L A+M 0.691 0.698 0.0720.072 0.9430.943 0.1890.189 0.8830.883 0.1030.103 0.9250.925 0.342 0.636 0.041 0.864 0.347 0.7720.772 0.0460.046 0.9340.934 0.392 0.747 0.065 0.917 MMS-300M L 0.7590.759 0.6210.621 0.0780.078 0.8860.886 0.2480.248 0.8390.839 0.1420.142 0.8840.884 0.4230.423 0.5100.510 0.0940.094 0.6490.649 0.3450.345 0.7640.764 0.0260.026 0.9480.948 0.4440.444 0.6840.684 0.0850.085 0.8420.842 MMS-300M BL 0.9960.996 0.684 0.3740.374 0.943 0.2110.211 0.8780.878 0.1370.137 0.9140.914 0.380 0.6110.611 0.050 0.841 0.3450.345 0.7640.764 0.0290.029 0.9450.945 0.4830.483 0.7340.734 0.1480.148 0.911 MMS-300M WS 0.750 0.6550.655 0.044 0.8290.829 0.2070.207 0.8760.876 0.1280.128 0.9240.924 0.4070.407 0.5740.574 0.0740.074 0.7800.780 0.3590.359 0.7570.757 0.0420.042 0.9250.925 0.431 0.7160.716 0.072 0.8650.865 MMS-300M FT 0.8210.821 0.6520.652 0.1020.102 0.943 0.169 0.894 0.072 0.944 0.4280.428 0.5540.554 0.0960.096 0.7490.749 0.339 0.775 0.025 0.954 0.4390.439 0.7190.719 0.0740.074 0.8970.897 MMS-300M A+M 0.8050.805 0.6790.679 0.1240.124 0.943 0.2510.251 0.8880.888 0.2020.202 0.9300.930 0.4860.486 0.611 0.1560.156 0.7870.787 0.4010.401 0.7710.771 0.0840.084 0.9520.952 0.4860.486 0.737 0.1420.142 0.9030.903 XLSR-300M L 0.7930.793 0.6120.612 0.0980.098 0.8860.886 0.2810.281 0.8180.818 0.1740.174 0.8720.872 0.4330.433 0.5080.508 0.1130.113 0.6690.669 0.3670.367 0.7470.747 0.0320.032 0.9490.949 0.4690.469 0.6710.671 0.1040.104 0.8440.844 XLSR-300M BL 0.7340.734 0.687 0.039 0.9430.943 0.1980.198 0.8750.875 0.1050.105 0.9160.916 0.3710.371 0.6300.630 0.0600.060 0.864 0.3640.364 0.7540.754 0.0300.030 0.9480.948 0.4170.417 0.7370.737 0.0580.058 0.9180.918 XLSR-300M WS 0.722 0.6740.674 0.0750.075 0.9430.943 0.2020.202 0.8760.876 0.1040.104 0.9230.923 0.3850.385 0.6070.607 0.0580.058 0.8150.815 0.3600.360 0.7600.760 0.0430.043 0.9480.948 0.4170.417 0.7290.729 0.0700.070 0.9070.907 XLSR-300M FT 0.7350.735 0.6780.678 0.0550.055 1.000 0.1730.173 0.8920.892 0.082 0.940 0.3840.384 0.5940.594 0.0540.054 0.7990.799 0.3370.337 0.7700.770 0.023 0.953 0.407 0.7330.733 0.053 0.923 XLSR-300M A+M 0.7710.771 0.6860.686 0.1480.148 0.9430.943 0.172 0.893 0.0900.090 0.9330.933 0.352 0.631 0.046 0.8260.826 0.337 0.772 0.0280.028 0.9500.950 0.4080.408 0.745 0.0780.078 0.9130.913 Wav2BERT L 0.7520.752 0.6380.638 0.1620.162 0.7710.771 0.3020.302 0.8010.801 0.1550.155 0.8830.883 0.4450.445 0.4490.449 0.1170.117 0.6590.659 0.3660.366 0.7520.752 0.0290.029 0.9540.954 0.4660.466 0.6600.660 0.1160.116 0.8170.817 Wav2BERT BL 0.679 0.6850.685 0.042 0.9430.943 0.2000.200 0.8800.880 0.1270.127 0.9270.927 0.3550.355 0.6120.612 0.0460.046 0.8670.867 0.3460.346 0.7620.762 0.0290.029 0.9440.944 0.3950.395 0.7350.735 0.0610.061 0.9200.920 Wav2BERT WS 0.6990.699 0.6860.686 0.0620.062 0.9430.943 0.1980.198 0.8780.878 0.1060.106 0.9290.929 0.3470.347 0.6250.625 0.0440.044 0.8470.847 0.3410.341 0.7690.769 0.026 0.956 0.3960.396 0.7390.739 0.0590.059 0.9190.919 Wav2BERT FT 0.8270.827 0.6740.674 0.1510.151 1.000 0.1880.188 0.8860.886 0.1090.109 0.9280.928 0.4400.440 0.5450.545 0.0730.073 0.7910.791 0.3750.375 0.7570.757 0.0470.047 0.9050.905 0.4580.458 0.7160.716 0.0950.095 0.9060.906 Wav2BERT A+M 0.6990.699 0.691 0.0500.050 0.9430.943 0.178 0.890 0.092 0.934 0.341 0.644 0.039 0.902 0.335 0.773 0.0260.026 0.9490.949 0.388 0.749 0.052 0.932 HuBERT-L L 0.691 0.6850.685 0.0320.032 0.9430.943 0.2540.254 0.8410.841 0.1230.123 0.9120.912 0.377 0.5870.587 0.0730.073 0.7870.787 0.3670.367 0.7480.748 0.0350.035 0.940 0.4220.422 0.7150.715 0.0660.066 0.8950.895 HuBERT-L BL 0.691 0.6850.685 0.0320.032 0.9430.943 0.2160.216 0.8720.872 0.1490.149 0.8960.896 0.377 0.5870.587 0.0730.073 0.7870.787 0.3670.367 0.7480.748 0.0350.035 0.940 0.413 0.7230.723 0.0720.072 0.8910.891 HuBERT-L WS 0.8060.806 0.6570.657 0.028 1.000 0.2590.259 0.8550.855 0.1510.151 0.9180.918 0.4670.467 0.5190.519 0.051 0.8440.844 0.4070.407 0.7260.726 0.032 0.9380.938 0.4850.485 0.6890.689 0.0650.065 0.925 HuBERT-L FT 0.7460.746 0.6770.677 0.0520.052 0.9430.943 0.191 0.888 0.1010.101 0.942 0.3820.382 0.6050.605 0.0610.061 0.8110.811 0.3850.385 0.7540.754 0.0650.065 0.9210.921 0.4260.426 0.7310.731 0.0700.070 0.9040.904 HuBERT-L A+M 0.6920.692 0.690 0.0300.030 0.9430.943 0.2090.209 0.8780.878 0.100 0.9200.920 0.3940.394 0.623 0.0710.071 0.856 0.360 0.760 0.0470.047 0.9310.931 0.4140.414 0.738 0.062 0.9130.913 4 Results and Discussion Table 1 first provides a 20-epoch screening across ten SFMs and three training regimes. A clear pattern is that performance depends strongly on both the backbone and the layer-utilization strategy. FT is a strong baseline overall, but frozen-backbone approaches remain competitive when layers are used appropriately. At the same time, the screening results already show that naive WS is not uniformly reliable: it is competitive for some backbones, such as XLSR-300M, MMS-300M, and Wav2BERT, but clearly degrades for others, notably WavLM-L, HuBERT-L, and Whisper-Lv3. This is an important early finding: simply using and aggregating more layers does not guarantee better MOS prediction. The screening stage was also used to select the backbones for the subsequent layer-wise study and the 100-epoch comparison in Table 2. We retained WavLM-L, HuBERT-L, XLSR-300M, MMS-300M, and Wav2BERT because they were the strongest model families overall in terms of average SRCC in the 20-epoch screening. We also preferred the 300M multilingual variants over the corresponding 1B models because the larger models did not show a sufficiently clear SRCC advantage to justify the substantially higher cost of exhaustive per-layer analysis and longer training. The layer-wise heatmaps in Fig. 1 and the mean importance curves in Fig. 2 help explain these trends. Across models, the strongest layers are often found in the early-to-mid portion of the encoder rather than consistently at the final layer. The global mean importance curve shows the same tendency. However, the figures also make clear that there is no universal best depth: the strongest layer varies across backbones and datasets, and some models exhibit broad high-performing regions while others show sharper peaks and valleys. This explains both why L can be suboptimal and why WS can be unstable: the most useful information is distributed differently across depth and is not necessarily aligned well enough for direct fusion. The 100-epoch results confirm that better layer selection often matters more than simply defaulting to the final layer. The clearest gains from BL over L appear for XLSR-300M and Wav2BERT. For XLSR-300M, the average utterance-level MSE decreases from 0.469 to 0.417 and the average system-level MSE from 0.104 to 0.058. For Wav2BERT, the corresponding improvements are from 0.466 to 0.395 and from 0.116 to 0.061. These results reinforce the main conclusion of the layer-wise analysis: the final layer is often not the strongest default representation for MOS prediction. For WavLM-L and HuBERT-L, however, BL brings smaller gains, indicating that the usefulness of intermediate layers is backbone-dependent rather than universal. A second key finding is that naive cross-layer aggregation is not a reliable substitute for careful layer selection. WS remains competitive for Wav2BERT and reasonably strong for MMS-300M and XLSR-300M, but it is clearly weaker for WavLM-L and HuBERT-L. In these cases, directly combining hidden states from different depths with scalar weights appears too restrictive. The issue is not only which layers are useful, but also whether their representations are compatible enough to be fused without adaptation. This is where A+M strategy becomes most useful. Across the final comparison, A+M is the strongest and most consistent frozen-backbone aggregation method, and the best overall configuration is Wav2BERT A+M, which reaches 0.388/0.749 in average utterance-level MSE/SRCC and 0.052/0.932 in average system-level MSE/SRCC. The figures suggest a plausible reason for this behavior: Wav2BERT exhibits a relatively broad band of strong middle-to-late layers rather than a single narrow optimum. This makes it a good candidate for calibrated multi-layer fusion, since several layers appear informative but not identical. In this setting, the adapters can help reshape these useful representations before pooling, making aggregation more effective than either L, BL, or naive WS. WavLM-L shows a different but still favorable pattern. Its heatmap suggests that useful information is present at multiple depths, but with a more irregular structure, including weaker late-layer regions. This likely makes naive WS vulnerable to mixing strong and weak representations. Consistent with this interpretation, WavLM-L A+M is the second-best overall configuration and improves markedly over WS, while also remaining slightly stronger than FT on the average utterance-level metrics. HuBERT-L behaves similarly, although the gains from A+M are more moderate. By contrast, XLSR-300M already has a relatively stable high-performing region, so BL is already strong and the gains from A+M are smaller. MMS-300M is the main mixed case: its strongest layers shift more across datasets, and A+M improves some ranking metrics but is less consistently favorable on MSE, suggesting stronger cross-dataset variation in where useful information resides. Overall, the results support three main conclusions. First, MOS-relevant information is distributed across the depth of speech foundation models, and the best-performing layer is strongly backbone- and dataset-dependent. Second, naive weighted fusion is not a reliable general solution for exploiting multiple layers. Third, lightweight per-layer calibration makes cross-layer aggregation substantially more robust, with the largest benefits appearing when several layers are informative but not trivially compatible, as seen most clearly for Wav2BERT and WavLM-L. 5 Conclusion This work revisited non-intrusive MOS prediction from speech foundation models through the lens of layer utilization. Across ten backbones and four datasets, we showed that the final layer is often not the most informative representation for MOS prediction, with strong performance frequently emerging in early-to-mid layers instead. We also found that naive cross-layer weighted fusion is not a reliable general solution, as its effectiveness depends strongly on the backbone. To address this, we introduced a layer-calibrated aggregation strategy based on per-layer adapters followed by pooling, which consistently improves robustness of multi-layer fusion and yields strongest overall frozen-backbone results, most notably with Wav2BERT and WavLM-L. Overall, our findings suggest that progress in MOS prediction depends not only on stronger backbones, but also on using their hierarchical representations more effectively. Future work will explore alternative compact fusion designs and evaluate generalization under broader cross-domain settings. 6 Use of Generative AI Disclosure The authors used generative AI tools, specifically large language models, only to polish the writing of portions of the manuscript. These tools were used for minor editing tasks such as improving grammar, clarity, and phrasing. The authors take full responsibility for the content of the manuscript. 7 Acknowledgements This work has been fully/partially funded by the project Research and Development of Algorithms for Construction of Digital Human Technological Components supported by Advanced Knowledge Center in Immersive Technologies (AKCIT), with financial resources from the PPI IoT of the MCTI grant number 057/2023, signed with EMBRAPII. The authors also wish to acknowledge the Artificial Intelligence Lab at Recod.ai and the Institute of Computing, University of Campinas, for granting access to the computational infrastructure required for the experiments. References [1] P. Wagner, J. Beskow, S. Betz, J. Edlund, J. Gustafson, G. Eje Henter, S. Le Maguer, Z. Malisz, É. Székely, C. Tånnander, and J. Voße (2019) Speech synthesis evaluation — state-of-the-art assessment and suggestion for a novel research program. In 10th ISCA Workshop on Speech Synthesis (SSW 10), p. 105–110. External Links: Document Cited by: §1. [2] O. Perrotin, B. Stephenson, S. Gerber, G. Bailly, and S. King (2025) Refining the evaluation of speech synthesis: a summary of the blizzard challenge 2023. Computer Speech & Language 90, p. 101747. External Links: ISSN 0885-2308, Document, Link Cited by: §1. [3] K. Shen, D. Yan, J. Hu, and Z. Ye (2024) Non-intrusive speech quality assessment: a survey. Neurocomputing 580, p. 127471. External Links: ISSN 0925-2312, Document, Link Cited by: §1. [4] C. Lo, S. Fu, W. Huang, X. Wang, J. Yamagishi, Y. Tsao, and H. Wang (2019) MOSNet: deep learning-based objective assessment for voice conversion. In Interspeech 2019, p. 1541–1545. External Links: Document, ISSN 2958-1796 Cited by: §1, §2. [5] P. C. Loizou (2011) Speech quality assessment. In Multimedia Analysis, Processing and Communications, W. Lin, D. Tao, J. Kacprzyk, Z. Li, E. Izquierdo, and H. Wang (Eds.), p. 623–654. External Links: ISBN 978-3-642-19551-8, Document, Link Cited by: §1. [6] S. Le Maguer, S. King, and N. Harte (2024) The limits of the mean opinion score for speech synthesis evaluation. Computer Speech & Language 84, p. 101577. External Links: ISSN 0885-2308, Document, Link Cited by: §1. [7] W. Tseng, C. Huang, W. Kao, Y. Y. Lin, and H. Lee (2021) Utilizing self-supervised representations for mos prediction. In Interspeech 2021, p. 2781–2785. External Links: Document, ISSN 2958-1796 Cited by: §1, §2. [8] E. Cooper, W. Huang, T. Toda, and J. Yamagishi (2022) Generalization ability of mos prediction networks. In ICASSP 2022 - 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Vol. , p. 8442–8446. External Links: Document Cited by: §1, §2, §3, §3. [9] T. Saeki, D. Xin, W. Nakata, T. Koriyama, S. Takamichi, and H. Saruwatari (2022) UTMOS: utokyo-sarulab system for voicemos challenge 2022. In Interspeech 2022, p. 4521–4525. External Links: Document, ISSN 2958-1796 Cited by: §1, §2. [10] F. S. Oliveira, E. Casanova, A. C. Junior, L. R. S. Gris, A. S. Soares, and A. R. Galvão Filho (2023) Evaluation of speech representations for mos prediction. In Text, Speech, and Dialogue: 26th International Conference, TSD 2023, Pilsen, Czech Republic, September 4–6, 2023, Proceedings, Berlin, Heidelberg, p. 270–282. External Links: ISBN 978-3-031-40497-9, Link, Document Cited by: §1, §2, §3. [11] R. E. Zezario, Y. Chen, S. Fu, Y. Tsao, H. Wang, and C. Fuh (2024) A study on incorporating whisper for robust speech assessment. In 2024 IEEE International Conference on Multimedia and Expo (ICME), Vol. , p. 1–6. External Links: Document Cited by: §1, §2. [12] A. Pasad, J. Chou, and K. Livescu (2021) Layer-wise analysis of a self-supervised speech representation model. In 2021 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), p. 914–921. Cited by: §1, §2. [13] A. Pasad, B. Shi, and K. Livescu (2023) Comparative layer-wise analysis of self-supervised speech models. In ICASSP 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Vol. , p. 1–5. External Links: Document Cited by: §1, §2. [14] G. Close, W. Ravenscroft, T. Hain, and S. Goetze (2023) Perceive and predict: self-supervised speech representation based loss functions for speech enhancement. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), p. 1–5. Cited by: §1, §2. [15] X. Liang, F. Cumlin, V. Ungureanu, C. K. Reddy, C. Schuldt, and S. Chatterjee (2025) Selection of layers from self-supervised learning models for predicting mean-opinion-score of speech. arXiv preprint arXiv:2508.08962. Cited by: §1, §2. [16] E. Cooper, T. Okamoto, Y. Ohtani, T. Toda, and H. Kawai (2025) Layer-wise analysis for quality of multilingual synthesized speech. arXiv preprint arXiv:2509.04830. Cited by: §1, §2. [17] Z. Li, M. Cheng, and M. Li (2025) Enhancing speaker verification with w2v-bert 2.0 and knowledge distillation guided structured pruning. arXiv preprint arXiv:2510.04213. Cited by: §1, §2. [18] D. Cai and M. Li (2024) Leveraging asr pretrained conformers for speaker verification through transfer learning and knowledge distillation. IEEE/ACM Transactions on Audio, Speech, and Language Processing 32, p. 3532–3545. Cited by: §1, §2. [19] A. Baevski, Y. Zhou, A. Mohamed, and M. Auli (2020) Wav2vec 2.0: a framework for self-supervised learning of speech representations. In Advances in Neural Information Processing Systems, H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (Eds.), Vol. 33, p. 12449–12460. External Links: Link Cited by: §2, §3. [20] A. Radford, J. W. Kim, T. Xu, G. Brockman, C. Mcleavey, and I. Sutskever (2023) Robust speech recognition via large-scale weak supervision. In Proceedings of the 40th International Conference on Machine Learning, A. Krause, E. Brunskill, K. Cho, B. Engelhardt, S. Sabato, and J. Scarlett (Eds.), Proceedings of Machine Learning Research, Vol. 202, p. 28492–28518. External Links: Link Cited by: §2, §3. [21] A. Babu, C. Wang, A. Tjandra, K. Lakhotia, Q. Xu, N. Goyal, K. Singh, P. von Platen, Y. Saraf, J. Pino, A. Baevski, A. Conneau, and M. Auli (2022) XLS-r: self-supervised cross-lingual speech representation learning at scale. In Interspeech 2022, p. 2278–2282. External Links: Document, ISSN 2958-1796 Cited by: §3. [22] V. Pratap, A. Tjandra, B. Shi, P. Tomasello, A. Babu, S. Kundu, A. Elkahky, Z. Ni, A. Vyas, M. Fazel-Zarandi, A. Baevski, Y. Adi, X. Zhang, W. Hsu, A. Conneau, and M. Auli (2024) Scaling speech technology to 1,000+ languages. Journal of Machine Learning Research 25 (97), p. 1–52. External Links: Link Cited by: §3. [23] L. Barrault, Y. Chung, M. C. Meglioli, D. Dale, N. Dong, M. Duppenthaler, P. Duquenne, B. Ellis, H. Elsahar, J. Haaheim, et al. (2023) Seamless: multilingual expressive and streaming speech translation. arXiv preprint arXiv:2312.05187. Cited by: §3. [24] S. Chen, C. Wang, Z. Chen, Y. Wu, S. Liu, Z. Chen, J. Li, N. Kanda, T. Yoshioka, X. Xiao, J. Wu, L. Zhou, S. Ren, Y. Qian, Y. Qian, J. Wu, M. Zeng, X. Yu, and F. Wei (2022) WavLM: large-scale self-supervised pre-training for full stack speech processing. IEEE Journal of Selected Topics in Signal Processing 16 (6), p. 1505–1518. External Links: Document Cited by: §3. [25] A. Baevski, W. Hsu, Q. Xu, A. Babu, J. Gu, and M. Auli (2022) Data2vec: a general framework for self-supervised learning in speech, vision and language. In International conference on machine learning, p. 1298–1312. Cited by: §3. [26] S. Yang, P. Chi, Y. Chuang, C. J. Lai, K. Lakhotia, Y. Y. Lin, A. T. Liu, J. Shi, X. Chang, G. Lin, T. Huang, W. Tseng, K. Lee, D. Liu, Z. Huang, S. Dong, S. Li, S. Watanabe, A. Mohamed, and H. Lee (2021) SUPERB: Speech Processing Universal PERformance Benchmark. In Interspeech 2021, p. 1194–1198. External Links: Document, ISSN 2958-1796 Cited by: §3. [27] Y. Tang, J. Shi, Y. Wu, and Q. Jin (2024) Singmos: an extensive open-source singing voice dataset for mos prediction. arXiv preprint arXiv:2406.10911. Cited by: §3. [28] Y. Chen and Y. Tsao (2022) InQSS: a speech intelligibility and quality assessment model using a multi-task learning network. In Interspeech 2022, p. 3088–3092. External Links: Document, ISSN 2958-1796 Cited by: §3.