Paper deep dive
Measuring Fairness in Large Audio Language Models via Semantic-Aware Bias Estimation
Zhe Liu
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 93%
Last extracted: 8/17/2026, 3:52:47 AM
Summary
This paper introduces a semantic-aware mixed-effects regression framework to evaluate fairness in Large Audio Language Models (LALMs). It addresses confounding factors in spoken-input settings by incorporating sentence-level semantic embeddings (extracted from the LALM itself) as covariates and modeling speaker identity as a random effect. Experiments on simulated and real-world data (LibriSpeech, AIR-Bench-Chat) demonstrate that this approach reduces spurious fairness findings compared to vanilla estimation or models ignoring semantic variation.
Entities (10)
Relation Signals (8)
Semantic-Aware Mixed-Effects Regression → models → Speaker Identity
confidence 95% · models speaker identity as a random effect
Semantic-Aware Mixed-Effects Regression → uses → Semantic Embeddings
confidence 95% · Our approach incorporates sentence-level semantic embeddings of reference text as covariates
Semantic-Aware Mixed-Effects Regression → reduces → Spurious Fairness Findings
confidence 93% · demonstrate that the proposed approach substantially reduces spurious fairness findings
LibriSpeech → usedin → ASR Fairness Evaluation
confidence 92% · We evaluate ASR fairness on gender... LibriSpeech
AIR-Bench-Chat → usedin → Audio Question Answering Fairness Evaluation
confidence 92% · AIR-Bench-Chat... evaluate the ability of LALMs to comprehend complex audio inputs... assess gender fairness
Qwen2-Audio → usedfor → embedding extraction
confidence 90% · semantic representations are extracted from the same LALM under evaluation... We use the Qwen2-Audio model
Embed-AGG → partof → Semantic-Aware Mixed-Effects Regression
confidence 88% · For the semantic-aware regression approach, we extract sentence embeddings using the same Qwen2-Audio model with both the Embed-AGG and Embed-EOWL methods.
Embed-EOWL → →
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Large Audio Language Models (LALMs) have seen increasing use for audio understanding tasks such as speech recognition and audio question answering, raising concerns about fairness across demographic subgroups. Fairness evaluation in spoken-input settings is challenging due to confounding factors, including semantic variation in spoken content and speaker-specific characteristics. Ignoring these factors can result in misleading conclusions about model bias. We propose a semantic-aware mixed-effects regression framework for fairness evaluation in LALMs that explicitly accounts for these confounders. Our approach incorporates sentence-level semantic embeddings of reference text as covariates and models speaker identity as a random effect. Notably, semantic representations are extracted from the same LALM under evaluation, enabling semantic control over variation as perceived by the model itself. Experiments on simulated data and real-world benchmarks demonstrate that the proposed approach substantially reduces spurious fairness findings and yields more robust and interpretable estimates of subgroup performance differences.
Tags
Links
- Source: https://arxiv.org/abs/2608.13624v1
- Canonical: https://arxiv.org/abs/2608.13624v1
Trouble viewing inline? Open PDF directly →
Full Text
32,776 characters extracted from source content.
Expand or collapse full text
Measuring Fairness in Large Audio Language Models via Semantic-Aware Bias Estimation Zhe Liu Affiliation: Meta Platforms, Inc. Menlo Park, USA Abstract Large Audio Language Models (LALMs) have seen increasing use for audio understanding tasks such as speech recognition and audio question answering, raising concerns about fairness across demographic subgroups. Fairness evaluation in spoken-input settings is challenging due to confounding factors, including semantic variation in spoken content and speaker-specific characteristics. Ignoring these factors can result in misleading conclusions about model bias. We propose a semantic-aware mixed-effects regression framework for fairness evaluation in LALMs that explicitly accounts for these confounders. Our approach incorporates sentence-level semantic embeddings of reference text as covariates and models speaker identity as a random effect. Notably, semantic representations are extracted from the same LALM under evaluation, enabling semantic control over variation as perceived by the model itself. Experiments on simulated data and real-world benchmarks demonstrate that the proposed approach substantially reduces spurious fairness findings and yields more robust and interpretable estimates of subgroup performance differences. Index Terms: large audio language models, fairness evaluation, semantic confounding, mixed-effects regression. I Introduction Over the past few years, Large language models (LLMs) have achieved strong performance across a wide range of language understanding and generation tasks [2, 13]. More recently, Large Audio Language Models (LALMs), which extend LLMs to process spoken inputs and generate textual or spoken outputs, have emerged as a powerful paradigm for audio understanding and spoken interaction [23, 6, 7, 8]. These models enable a variety of real-world applications, including automatic speech recognition (ASR), audio question answering, and conversational assistants. A growing body of studies has shown that LLMs can exhibit biases associated with demographic and social attributes [16]. When extended to spoken-input scenarios, these concerns are further amplified [17, 24, 3]. Spoken audio conveys rich paralinguistic information, including speaker gender, age, and accent, while also exhibiting substantial semantic variation. Thus, observed performance disparities across subgroups may arise not only from biases inherited from the underlying LALMs, but also from confounding factors such as semantic differences in spoken content and correlations among utterances produced by the same speaker. For example, if one demographic group is disproportionately associated with semantically more complex or information-dense utterances, a model may exhibit higher error rates for that group even in the absence of any intrinsic bias. If these factors are not properly controlled, fairness evaluations may overstate or misattribute disparities. Motivated by these challenges, this paper focuses on measuring audio understanding fairness in LALMs while explicitly controlling for semantic variation in spoken content. We propose a semantic-aware mixed-effects regression method that incorporates sentence-level semantic embedding of reference text as covariates while modeling speaker identity as a random effect. This approach enables more reliable assessment of subgroup performance gaps and helps disentangle the contributions of semantic content, speaker characteristics, and model behavior in observed disparities. Existing work on statistical model-based fairness evaluation in audio understanding has largely focused on ASR tasks using traditional ASR systems [18]. In these settings, semantic information is typically obtained from external representations, such as fastText [5] or BERT [9]. In contrast, our work targets LALMs and considers a broader class of audio understanding tasks beyond ASR. Moreover, rather than relying on external semantic estimators, we propose a novel approach that leverages the same LALM under evaluation to project reference text into a semantic vector space. By doing so, the semantic embeddings are directly aligned with the model’s internal representation of the input, providing a more accurate and model-consistent control for semantic variation in fairness evaluation. To the best of our knowledge, this work is the first to introduce a fairness evaluation framework for LALMs that explicitly accounts for semantic variation in spoken content. Our main contributions are threefold: (1) we propose a semantic-aware mixed-effects regression that includes semantic sentence embeddings as covariates while modeling speaker identity as a random effect, enabling more reliable fairness measurement in audio understanding tasks; (2) we develop new strategies that extract sentence embeddings of reference text from the same LALMs under evaluation; compared with external embedding estimators, the resulting embeddings are more closely aligned with the models’ own perception and judgment of spoken content; and (3) we conduct extensive experiments on both ASR and audio question answering benchmarks, demonstrating that controlling for semantic and speaker-level confounding factors substantially reduces spurious fairness findings. The rest of this paper is structured as follows. Section I discusses related work. Section I introduces the semantic-aware mixed-effects regression method for fairness evaluations of LALMs in audio understanding tasks. Section IV and Section V demonstrate the validity of the proposed approach on synthetic and real-world speech data, respectively. We conclude in Section VI. I Related Work I-A Fairness Evaluation in Audio Understanding Fairness in speech and audio systems has been extensively studied in the context of ASR, where performance disparities across demographic groups such as gender, age, and accent have been investigated [21, 15]. Authors in [18] propose a model-based statistical framework for ASR fairness evaluation that accounts for speaker-level dependencies using mixed-effects regression. Our work builds upon this framework and extends it in two key directions: (1) we target LALMs and consider a broader class of audio understanding tasks, beyond ASR with traditional hybrid or RNN-T speech models, and (2) we introduce strategies for extracting semantic representations from the same LALM under evaluation, providing model-consistent control for semantic variation rather than relying on external embeddings. I-B Sentence Embeddings Extracting sentence representations has been a longstanding research topic [5, 9, 20]. In contrast to these approaches, our work extracts sentence representations directly from the LALM under evaluation. This design choice ensures that the resulting embeddings reflect the model’s own perception of semantic content, which is critical for controlling semantic confounding in fairness evaluation. I Methods Suppose we aim to investigate fairness in audio understanding tasks for LALMs with respect to a factor variable of primary interest (e.g., speaker gender). For the s-th utterance in the evaluation dataset, we denote its factor level by f(s)f(s) (e.g., male or female), where f is a deterministic mapping. Our goal is to test whether the effect of this factor is statistically significant on the measured task performance across its different levels. We propose a semantic-aware mixed-effects regression and demonstrate how it can be utilized to quantify any performance gaps between subgroups in disparity and fairness analyses. I-A Semantic-Aware Mixed-Effects Regression For illustration, we use the ASR task to demonstrate the proposed framework. After running speech recognition on an evaluation dataset using LALMs and obtaining the corresponding hypotheses, we compute utterance-level error statistics for subsequent analysis. Specifically, let CijinsC^ins_ij and Cijdel+subC^del+sub_ij denote the number of insertion errors and the sum of deletion and substitution errors, respectively, for utterance j from speaker i; let NijN_ij be the number of words in the corresponding ground-truth transcription. We model insertion errors using a Poisson regression, and the sum of deletion and substitution errors using a Binomial regression. This choice reflects the distinct nature of these error types: insertion errors correspond to count data without a fixed upper bound for a given utterance, making the Poisson distribution a natural modeling choice, whereas deletion and substitution errors are bounded by the number of reference words and can be viewed as Bernoulli trials at the word level, which are appropriately modeled using a Binomial distribution. Let xijx_ij denote the semantic sentence embedding of reference text for utterance j from speaker i, which accounts for semantic variation across utterances in the evaluation set. Here, the reference text is defined as the concatenation of the ground-truth transcription of audio input and all accompanying textual content (e.g., instructions or prompts) provided to the LALM under evaluation. We include xijx_ij as a fixed-effect covariate in the model, and will discuss its computation in the next section. In addition, we incorporate speaker-level random effects [4, 12] to capture speaker-specific characteristics. This modeling choice reflects the assumption that speakers are randomly sampled from a larger population and that our primary interest lies in estimating population-level effects rather than making inferences about individual speakers. To measure the effect of factor f(⋅)f(·) on word-error-rate (WER) results across different subgroups, the proposed model is described as follows: ri r_i ∼(0,σ2), (0,σ^2), (1) Cijins|λij C^ins_ij _ij ∼Poisson(λij), ( _ij), (2) Cijdel+sub|pij C^del+sub_ij p_ij ∼Binomial(Nij,pij), (N_ij,p_ij), (3) log(λij) ( _ij) =log(Nij)+μf(i)ins+ri+θins⊤xij, = (N_ij)+μ^ins_f(i)+r_i+θ^ins x_ij, (4) logit(pij) (p_ij) =μf(i)del+sub+ri+θdel+sub⊤xij, =μ^del+sub_f(i)+r_i+θ^del+sub x_ij, (5) where rir_i represents the speaker-level random effect that is independently sampled from a Gaussian distribution with mean 0 and variance σ2σ^2 which is learnable; λij _ij denotes the expected number of insertion errors for utterance j from speaker i; pijp_ij denotes the per-word probability of a deletion or substitution error for the same utterance; μf(i)ins,μf(i)del+subμ^ins_f(i),μ^del+sub_f(i) are fixed effects of the factor of interest f(⋅)f(·) for insertions and the sum of deletions and substitutions, respectively; θins,θdel+subθ^ins,θ^del+sub are the corresponding regression coefficients for the semantic covariates. The model can be fitted via maximum likelihood, with the likelihood expressed as an integral over the random effects, which can be approximated using adaptive Gauss-Hermite quadrature [1]. After fitting the model, we assess the effect of f(⋅)f(·) by leveraging the estimated fixed-effects coefficients and their variance-covariance matrices. Fairness disparities are then quantified by the ratio of the resulting expected WERs among different levels of f(⋅)f(·). Statistical inference on the WER ratios can be conducted by delta method [10] or bootstrap [11]. Although we utilize ASR task as an example for illustration, the proposed framework is readily extensible to other audio understanding tasks for LALMs by adopting appropriate statistical distributional assumptions in the regression model for the task-specific response variable. By including the semantic embeddings xijx_ij as covariates, this framework explicitly controls for utterance-level semantic variation. This allows us to isolate the contribution of f(⋅)f(·) to audio understanding capabilities, providing a more precise and semantic-aware measure of fairness across subgroups. In practice, when the original embedding dimensionality is high, we optionally apply principal component analysis (PCA) to project the embeddings onto a lower-dimensional subspace that preserves the dominant semantic structure while ensuring stable and well-conditioned regression fitting. I-B Extract Sentence Representations with LALMs We investigate two approaches for computing semantic embeddings of reference text using the same LALMs under evaluation. The resulting representations are incorporated as additional explanatory covariates in the regression models introduced in the previous section, enabling semantic-aware bias estimation. I-B1 Prompt LALMs with reference text For any reference text =(x1,…,xn)x=(x_1,…,x_n), passing it to the LALM under evaluation yields the hidden states 0,1,…,L=LALM(), _0,h_1,…,h_L=LALM(x), (6) where 0h_0 denotes the hidden state of the embedding layer, lh_l for l∈1,…,Ll∈1,…,L denotes the hidden state of the l-th Transformer [22] layer, and L is the total number of Transformer layers. Let the final hidden state be written as L=(hL1,…,hLn′)h_L=(h_L1,…,h_Ln ), where n′n is the number of tokens after tokenization. The hidden state of the final token, hLn′h_Ln , can be used as the sentence embedding of x. Alternatively, other aggregation or pooling strategies over the hidden states can be employed. In this work, we consider an aggregation method that averages the hidden states from the first, middle, and final Transformer layers across all tokens 13(1n′∑k=1n′h1k+1n′∑k=1n′h⌊L/2⌋,k+1n′∑k=1n′hLk). 13 ( 1n _k=1^n h_1k+ 1n _k=1^n h_ L/2 ,k+ 1n _k=1^n h_Lk ). (7) This aggregation is motivated by the observation that earlier Transformer layers tend to encode lower-level lexical features, while deeper layers progressively capture higher-level semantic abstractions [26]. By combining representations across different depths, the resulting embedding provides a more comprehensive summary of the input. We refer to this method as “Embed-AGG”. I-B2 Prompt LALMs with explicit one word limitation The prompt template with an explicit one-word limitation (EOWL) was first introduced in [14]. It provides a simple and direct way to instruct LLMs to summarize the meaning of a sentence using a single word. Under this template, for any reference text x, we query the LALM under evaluation as LALM(This sentence: ``" means in one word:). ( This sentence: ``x" means in one word:). (8) In contrast to [14], which leverages hidden states, we use the logits of the LALM’s next-token prediction as the representation of x. The motivation is that the EOWL prompt encourages the model to condense the semantic content of the entire sentence into a single-word prediction. Thus, the associated output logits can be viewed as a compact semantic representation of the sentence. We refer to this method as “Embed-EOWL”. Note that extracting semantic embeddings from the same LALM under evaluation is a deliberate design choice. The goal of the semantic covariate is to control for variation in how the model perceives input difficulty; using the model’s own representations provides the most faithful characterization of this variation. IV Simulation Experiments Under the scenario where the factor of interest (e.g., speaker gender) has no true effect on the response quality of LALMs, substantial differences in the semantic content of the audio inputs across factor levels can nevertheless lead to spurious findings of unfairness if not properly accounted for. In this section, we perform simulation experiments to demonstrate that the proposed semantic-aware regression framework effectively addresses semantic confounding in fairness measurements for LALMs, thereby reducing bias in effect estimation and mitigating false positive conclusions of unfairness. IV-A Setups Suppose we want to study the effect of speaker gender on the response accuracy in audio question answering task for LALMs. First, we prompt the Llama-3 70B model [13] to generate 2,000 text questions. Using different prompt instructions, half of the questions are designed to be short, clear, and primarily based on common-sense knowledge, while the other half are longer, more detailed, and require deeper reasoning that draws on knowledge from multiple domains. We refer to these two subsets as the simple question set and the hard question set, respectively. We then apply a text-to-speech (TTS) system to convert the text questions into audio questions. Specifically, we randomly select 200 questions from the simple set and 800 questions from the hard set to be synthesized using female speakers, and 800 questions from the simple set and 200 questions from the hard set to be synthesized using male speakers. We use the Qwen2-Audio model [7] as the LALM to generate textual responses to these audio questions. In this simulation, assume that the response accuracy scores follow a (5,1)N(5,1) distribution for questions from the hard set and a (6,1)N(6,1) distribution for questions from the simple set. Therefore, under this data-generating process, speaker gender has no effect on response accuracy. This design intentionally induces correlation between gender and semantic difficulty while ensuring that gender has no causal effect on response accuracy. IV-B Methods and Results As a vanilla approach, we compute the average response scores for the male and female groups and report their ratio, with 95% confidence intervals estimated via bootstrap. For the semantic-aware regression approach, we extract sentence embeddings using the same Qwen2-Audio model with both the Embed-AGG and Embed-EOWL methods. PCA is then applied to reduce the embedding dimensionality to 8, which retains the majority (around 83%) of the total variance and yield stable regression estimates in practice. We subsequently fit a linear regression model that includes the reduced semantic representations as covariates and estimate the ratio of response scores between the male and female groups, as well as the corresponding confidence intervals. For comparison, we also include a regression method that excludes semantic information although it is equivalent to the vanilla estimation. TABLE I: Comparison of vanilla and semantic-aware methods for gender fairness evaluation on simulated data. Simulated Data Male-Female Score Comparison Method Ratio Confidence Interval Is False Positive? Vanilla Estimation 1.124 (1.103, 1.144) Y Regression w/o Semantics 1.124 (1.104, 1.144) Y Semantic-Aware: Embed-AGG 1.003 (0.983, 1.024) N Semantic-Aware: Embed-EOWL 1.000 (0.980, 1.021) N Table I summarizes the estimated male-female response accuracy score ratios obtained using different evaluation methods. Both the vanilla estimation approach and the regression model without semantic covariates yield identical ratio estimates (1.124) with confidence intervals that exclude 1, leading to false positive conclusions of gender unfairness despite the ground truth being gender-neutral. In contrast, the proposed semantic-aware regression methods produce ratio estimates close to 1 with confidence intervals covering 1, correctly indicating no statistically significant performance gap between male and female speakers. These results demonstrate that failing to control for semantic variation can induce spurious fairness violations, while incorporating semantic sentence embeddings effectively mitigates semantic confounding and yields more reliable fairness assessments. Fig. 1: PCA visualization of sentence embeddings. After applying PCA to further project the 8-dimensional Embed-AGG into 2 dimensions, Figure 1 illustrates that the embeddings extracted from Qwen2-Audio clearly differentiate between simple and hard questions. This separation indicates that the extracted sentence embeddings successfully capture the underlying semantic difficulty of the questions. V Real Data Experiments In this section, we apply the proposed semantic-aware mixed-effects regression to real speech data for fairness analysis. V-A Setups We consider the following two evaluation datasets in this study: • LibriSpeech [19]. It is a widely used ASR benchmark. We use the evaluation splits, including Test-Clean, which contains 2,620 utterances from 40 speakers (20 male), and Test-Other, which contains 2,939 utterances from 33 speakers (16 male). We evaluate ASR fairness on gender; • AIR-Bench-Chat [25]. It is a benchmark of audio understanding questions designed to evaluate the ability of LALMs to comprehend complex audio inputs and follow human instructions. In this experiment, we select the subset with available gender annotations, consisting of 436 utterances from male speakers and 163 utterances from female speakers. For evaluation, we use Llama-3 70B model as a judge to score the generated answers from LALMs. The scores range from 1 to 10 and are based on usefulness, relevance, accuracy, and comprehensiveness. We then analyze these scores to assess gender fairness. Since the LLM judge evaluates only textual responses without access to speaker information, its scoring is applied uniformly across subgroups and is unlikely to confound gender comparisons. For both datasets, we use Qwen2-Audio as the LALM to generate textual responses, either by transcribing audio inputs or by answering audio-based questions. V-B Methods and Results Table I presents the aggregated performance metrics for male and female speakers across the evaluated datasets. For the ASR task on LibriSpeech datasets, we present the average WER, while for AIR-Bench-Chat we report the average response accuracy score. The corresponding male-female ratios provide a naive measure of group-level performance differences. While such aggregated statistics are intuitive and commonly adopted, they do not account for potential confounding factors such as speaker-specific characteristics or semantic variation. TABLE I: Aggregated male and female performance comparison across evaluation datasets. Aggregated WER or Score Male Female Ratio LibriSpeech Test-Clean (WER) 3.21 4.47 0.719 LibriSpeech Test-Other (WER) 6.89 5.93 1.162 AIR-Bench-Chat (Score) 6.02 6.21 0.969 We examine regression-based models that progressively incorporate additional structure, including fixed-effect regression without speaker- or semantic-level controls, mixed-effects regression with speaker-level random effects (when speaker identifiers are available), and our proposed semantic-aware mixed-effects regression models that further include the semantic embeddings of references as covariates. For the semantic-aware models, we extract sentence embeddings using Qwen2-Audio with both the Embed-AGG and Embed-EOWL methods. PCA is then applied to reduce the embedding dimensionality to 8, which retains the majority of the total variance while ensuring stable and well-conditioned regression estimation. This systematic comparison allows us to isolate the impact of speaker-level dependencies and semantic confounding on fairness measurements. Tables I and IV show the comparison of different methods for gender fairness evaluation on LibriSpeech data. On the Test-Clean and Test-Other sets, vanilla estimation and regression models without speaker or semantic controls indicate significant male-female WER gaps. In contrast, models that account for speaker-level random effects eliminate much of this bias, and the proposed semantic-aware mixed-effects regression further attenuates the estimated gaps, rendering them statistically insignificant. This suggests that previously observed disparities are largely driven by speaker-specific characteristics and sentence-level linguistic differences rather than gender itself. TABLE I: Comparison of different methods for gender fairness evaluation on LibriSpeech Test-Clean data. LibriSpeech Test-Clean Male-Female WER Comparison Method Ratio Confidence Interval Is Stat. Significant? Vanilla Estimation 0.719 (0.553, 0.938) Y Regression w/o Spk. w/o Sem. 0.719 (0.658, 0.786) Y Regression w/ Spk. w/o Sem. 0.777 (0.600, 1.011) N Spk. Sem.-Aware: Embed-AGG 0.802 (0.619, 1.042) N Spk. Sem.-Aware: Embed-EOWL 0.822 (0.634, 1.073) N TABLE IV: Comparison of different methods for gender fairness evaluation on LibriSpeech Test-Other data. LibriSpeech Test-Other Male-Female WER Comparison Method Ratio Confidence Interval Is Stat. Significant? Vanilla Estimation 1.162 (1.013, 1.329) Y Regression w/o Spk. w/o Sem. 1.162 (1.088, 1.244) Y Regression w/ Spk. w/o Sem. 1.125 (0.875, 1.450) N Spk. Sem.-Aware: Embed-AGG 1.020 (0.811, 1.289) N Spk. Sem.-Aware: Embed-EOWL 1.017 (0.801, 1.294) N Table V shows the comparison of different methods for gender fairness evaluation on the AIR-Bench-Chat dataset. All methods show no statistically significant gender differences in response quality. Notably, the semantic-aware regression yields ratios closest to unity, confirming that controlling for semantic content leads to more stable and interpretable fairness assessments in audio understanding tasks. Note that speaker-level random effects cannot be applied to this dataset, as speaker identifiers are not provided in the original benchmark. TABLE V: Comparison of different methods for gender fairness evaluation on AIR-Bench-Chat data. AIR-Bench-Chat Male-Female Score Comparison Method Ratio Confidence Interval Is Stat. Significant? Vanilla Estimation 0.969 (0.893, 1.055) N Regression w/o Semantics 0.969 (0.891, 1.055) N Semantic-Aware: Embed-AGG 1.004 (0.920, 1.101) N Semantic-Aware: Embed-EOWL 1.002 (0.924, 1.093) N The Embed-AGG and Embed-EOWL methods perform similarly in practice; however, Embed-AGG is more effective, as its lower-dimensional PCA projection explains a larger proportion of the variance in the original embedding space. Overall, these results demonstrate that failing to control for semantic and speaker-level confounding factors can lead to false conclusions of unfairness. The proposed semantic-aware mixed-effects framework provides a more reliable and principled approach for measuring fairness in LALMs across audio understanding tasks. While our experiments focus on Qwen2-Audio and gender as the primary attribute, we emphasize that the proposed framework is model-agnostic and attribute-agnostic by design. It is natural to extend the evaluation to additional models and demographic attributes. VI Conclusions In this work, we present a semantic-aware mixed-effects regression model for measuring fairness in LALMs. By controlling for sentence-level semantic variation, the proposed approach provides more reliable estimates of subgroup performance gaps than conventional methods. Experimental results show that many apparent disparities can be attributed to semantic or speaker confounding rather than intrinsic model bias. This paper highlights the importance of semantic control in fairness evaluations and offers a general framework applicable to a wide range of audio understanding tasks. References [1] M. Abramowitz and I. A. Stegun (1964) Handbook of mathematical functions with formulas, graphs, and mathematical tables. Vol. 55, US Government printing office. Cited by: §I-A. [2] J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat, et al. (2023) GPT-4 technical report. arXiv preprint arXiv:2303.08774. Cited by: §I. [3] S. Arora, K. Chang, C. Chien, Y. Peng, H. Wu, Y. Adi, E. Dupoux, H. Lee, K. Livescu, and S. Watanabe (2025) On the landscape of spoken language models: a comprehensive survey. arXiv preprint arXiv:2504.08528. Cited by: §I. [4] B. Baltagi (2008) Econometric analysis of panel data. John Wiley & Sons. Cited by: §I-A. [5] P. Bojanowski, E. Grave, A. Joulin, and T. Mikolov (2017) Enriching word vectors with subword information. Transactions of the Association for Computational Linguistics 5, p. 135–146. Cited by: §I, §I-B. [6] Z. Borsos, R. Marinier, D. Vincent, E. Kharitonov, O. Pietquin, M. Sharifi, D. Roblek, O. Teboul, D. Grangier, M. Tagliasacchi, and N. Zeghidour (2023) AudioLM: a language modeling approach to audio generation. IEEE/ACM Transactions on Audio, Speech, and Language Processing 31, p. 2523–2533. Cited by: §I. [7] Y. Chu, J. Xu, Q. Yang, H. Wei, X. Wei, Z. Guo, Y. Leng, Y. Lv, J. He, J. Lin, C. Zhou, and J. Zhou (2024) Qwen2-Audio technical report. arXiv preprint arXiv:2407.10759. Cited by: §I, §IV-A. [8] A. Défossez, L. Mazaré, M. Orsini, A. Royer, P. Pérez, H. Jégou, E. Grave, and N. Zeghidour (2024) Moshi: a speech-text foundation model for real-time dialogue. arXiv preprint arXiv:2410.00037. Cited by: §I. [9] J. Devlin, M. Chang, K. Lee, and K. Toutanova (2019) BERT: pre-training of deep bidirectional transformers for language understanding. In Proceedings of NAACL, Cited by: §I, §I-B. [10] J. L. Doob (1935) The limiting distributions of certain statistics. The Annals of Mathematical Statistics 6 (3), p. 160–169. Cited by: §I-A. [11] B. Efron and R. J. Tibshirani (1994) An introduction to the bootstrap. CRC Press. Cited by: §I-A. [12] J. J. Faraway (2016) Extending the linear model with r: generalized linear, mixed effects and nonparametric regression models. CRC press. Cited by: §I-A. [13] A. Grattafiori, A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Vaughan, et al. (2024) The Llama 3 herd of models. arXiv preprint arXiv:2407.21783. Cited by: §I, §IV-A. [14] T. Jiang, S. Huang, Z. Luan, D. Wang, and F. Zhuang (2024) Scaling sentence embeddings with large language models. In Findings of EMNLP, p. 3182–3196. Cited by: §I-B2, §I-B2. [15] A. Koenecke, A. Nam, E. Lake, J. Nudell, M. Quartey, Z. Mengesha, C. Toups, J. R. Rickford, D. Jurafsky, and S. Goel (2020) Racial disparities in automated speech recognition. Proceedings of the National Academy of Sciences 117 (14), p. 7684–7689. Cited by: §I-A. [16] Y. Li, M. Du, R. Song, X. Wang, and Y. Wang (2023) A survey on fairness in large language models. arXiv preprint arXiv:2308.10149. Cited by: §I. [17] Y. Lin, T. Lin, C. Yang, K. Lu, W. Chen, C. Kuan, and H. Lee (2024) Listen and speak fairly: a study on semantic gender bias in speech integrated large language models. In IEEE SLT, p. 439–446. Cited by: §I. [18] Z. Liu, I. Veliche, and F. Peng (2022) Model-based approach for measuring the fairness in ASR. In Proceedings of ICASSP, Cited by: §I, §I-A. [19] V. Panayotov, G. Chen, D. Povey, and S. Khudanpur (2015) LibriSpeech: an ASR corpus based on public domain audio books. In Proceedings of ICASSP, Cited by: 1st item. [20] N. Reimers and I. Gurevych (2019) Sentence-BERT: sentence embeddings using siamese BERT-networks. In Proceedings of EMNLP-IJCNLP, p. 3982–3992. Cited by: §I-B. [21] R. Tatman (2017) Gender and dialect bias in YouTube’s automatic captions. In Proceedings of the First ACL Workshop on Ethics in Natural Language Processing, p. 53–59. Cited by: §I-A. [22] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin (2017) Attention is all you need. Advances in Neural Information Processing Systems 30. Cited by: §I-B1. [23] J. Wu, Y. Gaur, Z. Chen, L. Zhou, Y. Zhu, T. Wang, J. Li, S. Liu, B. Ren, L. Liu, et al. (2023) On decoder-only architecture for speech-to-text and large language model integration. In IEEE ASRU, p. 1–8. Cited by: §I. [24] Y. Wu, T. Wang, Y. Peng, Y. Chao, X. Zhuang, X. Wang, S. Yin, and Z. Ma (2025) Evaluating bias in spoken dialogue LLMs for real-world decisions and recommendations. arXiv preprint arXiv:2510.02352. Cited by: §I. [25] Q. Yang, J. Xu, W. Liu, Y. Chu, Z. Jiang, X. Zhou, Y. Leng, Y. Lv, Z. Zhao, C. Zhou, and J. Zhou (2024) AIR-Bench: benchmarking large audio-language models via generative comprehension. In Proceedings of ACL, p. 1979–1998. Cited by: 2nd item. [26] W. X. Zhao, K. Zhou, J. Li, T. Tang, X. Wang, Y. Hou, Y. Min, B. Zhang, J. Zhang, Z. Dong, et al. (2023) A survey of large language models. arXiv preprint arXiv:2303.18223. Cited by: §I-B1.