Paper deep dive
RAS: a Reliability Oriented Metric for Automatic Speech Recognition
Wenbin Huang, Yuhang Qiu, Bohan Li, Yiwei Guo, Jing Peng, Hankun Wang, Xie Chen, Kai Yu
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 97%
Last extracted: 6/21/2026, 8:14:49 AM
Summary
The paper introduces RAS (Reliability-Oriented Metric), a novel evaluation metric for Automatic Speech Recognition (ASR) that balances transcription usefulness and error aversion. Unlike standard Word Error Rate (WER), RAS accounts for an 'abstention' mechanism where models use a special placeholder (PH) to signal uncertainty. The authors propose an abstention-aware ASR framework using a two-stage training process: supervised bootstrapping (PH-Supv) to teach the model to use the PH token, followed by Reinforcement Learning (using GRPO) with RAS as the reward signal. Experiments on LibriSpeech and TALCS datasets demonstrate that this approach significantly improves transcription reliability and robustness, especially in noisy or code-switching environments, compared to standard ASR models and logit-based confidence methods.
Entities (9)
Relation Signals (5)
PH → enables → Abstention
confidence 100% · the PH represents abstention: when the model encounters low-quality or ambiguous speech... it outputs PH
RAS → evaluates → Automatic Speech Recognition
confidence 100% · To evaluate reliability under abstention, we propose RAS, a reliability-oriented metric
RAS → usedasrewardfor → GRPO
confidence 100% · using utterance-level RAS proposed in Section 2.2 as the reward signal
LibriSpeech → usedforexperiment → RAS
confidence 100% · We conduct experiments on two datasets: LibriSpeech and the TALCS Corpus.
GRPO → optimizes → Whisper
confidence 90% · we employ Group Relative Policy Optimization (GRPO) algorithm to optimize its output policy... developing an abstention-aware Whisper model
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Automatic speech recognition systems often produce confident yet incorrect transcriptions under noisy or ambiguous conditions, which can be misleading for both users and downstream applications. Standard evaluation based on Word Error Rate focuses solely on accuracy and fails to capture transcription reliability. We introduce an abstention-aware transcription framework that enables ASR models to explicitly abstain from uncertain segments. To evaluate reliability under abstention, we propose RAS, a reliability-oriented metric that balances transcription informativeness and error aversion, with its trade-off parameter calibrated by human preference. We then train an abstention-aware ASR model through supervised bootstrapping followed by reinforcement learning. Our experiments demonstrate substantial improvements in transcription reliability while maintaining competitive accuracy.
Tags
Links
- Source: https://arxiv.org/abs/2604.24278v1
- Canonical: https://arxiv.org/abs/2604.24278v1
Trouble viewing inline? Open PDF directly →
Full Text
33,104 characters extracted from source content.
Expand or collapse full text
Huang Qiu Li Guo Peng Wang Chen Yu RAS: a Reliability Oriented Metric for Automatic Speech Recognition Wenbin Yuhang Bohan Yiwei Jing Hankun Xie Kai 1 X-LANCE Lab, School of Computer Science, Shanghai Jiao Tong University, China 1 MoE Key Lab of Artificial Intelligence; Jiangsu Key Lab of Language Computing, China hartmann_psi, qiuyuhang, kai.yu@sjtu.edu.cn Abstract Automatic speech recognition systems often produce confident yet incorrect transcriptions under noisy or ambiguous conditions, which can be misleading for both users and downstream applications. Standard evaluation based on Word Error Rate focuses solely on accuracy and fails to capture transcription reliability. We introduce an abstention-aware transcription framework that enables ASR models to explicitly abstain from uncertain segments. To evaluate reliability under abstention, we propose RAS, a reliability-oriented metric that balances transcription informativeness and error aversion, with its trade-off parameter calibrated by human preference. We then train an abstention-aware ASR model through supervised bootstrapping followed by reinforcement learning. Our experiments demonstrate substantial improvements in transcription reliability while maintaining competitive accuracy. keywords: speech recognition, reliability, abstention, selective prediction, uncertainty modeling 1 Introduction Modern automatic speech recognition (ASR) systems achieve high accuracy in clean acoustic conditions, yet they often still produce superficially fluent transcripts in the presence of noise, overlapping speech, signal degradation, or low-resource settings. These outputs are frequently the result of forced decoding under weak acoustic evidence, yielding errors that appear confident rather than explicitly uncertain. Such plausible-but-wrong transcriptions can mislead downstream decision-making and, due to their readability, may further reduce human vigilance during review. The risk is especially acute in high-stakes applications with stringent transcription requirements, such as medical documentation and legal records. However, dominant evaluation paradigms do not adequately characterize this failure mode. One natural direction is to borrow reliability tools from broader machine learning. While abstention (i.e., learning with a reject option) and selective prediction have been shown to improve reliability in machine learning by explicitly trading coverage for reduced risk [1, 2, 3, 4, 5, 6], these approaches are most often formulated at the instance level, making a decision to either accept or reject an entire input. This formulation is ill-suited to ASR, where uncertainty is highly localized and utility depends on token-/word-level decisions. A related line of work, ASR confidence estimation [7, 8, 9, 10, 11] constitutes a viable avenue for addressing this issue. However, most existing methods still follow a two-stage, post-hoc paradigm: they first generate a transcription and only subsequently attach uncertainty scores as a separate layer of metadata. As a result, confidence is modeled implicitly rather than integrated into the decoding process, leaving the system without an internal mechanism to actively opt out of unreliable segments during inference. Meanwhile, standard Word-Error-Rate(WER) [12, 13] and its edit-distance variants[14, 15, 16], as well as semantic metrics[17, 18, 19], all implicitly assume that the model must produce a complete transcription. As a result, they quantify only how much a hypothesis differs from the reference, not whether the system should output anything (or how much it should output) when evidence is insufficient. Consequently, these metrics cannot assess whether an ASR system makes an appropriate trade-off between informativeness and reliability under uncertainty. In this work, we argue that reliable ASR should be able to to explicitly abstain from localized uncertain predictions. We introduce a fine-grained abstention paradigm that augments the output space with a dedicated placeholder, as shown in Figure 1. Unlike full-sentence rejection, our approach allows the model to selectively abstain on ambiguous segments while transcribing the remaining content with high commitment. This transition from passive scoring to active, fine-grained abstention offers significant advantages for downstream applications: it prevents error propagation by providing clear, localized signals of uncertainty, allowing downstream modules to treat these placeholders as missing information rather than misleading hallucinations. Figure 1: Conventional v.s. abstention-aware hypothesis. To evaluate this paradigm, we propose Reliability-Aware Score (RAS), a novel metric derived from a modified edit distance that accounts for placeholder marks. RAS explicitly balances transcription usefulness and error aversion through a trade-off hyperparameter calibrated via human listening tests. This provides a principled objective for both the evaluation and optimization of reliability-aware ASR. We further demonstrate the practical utility of our framework by developing an abstention-aware Whisper [20] model. By employing a two-stage training pipeline—comprising supervised bootstrapping followed by reinforcement learning (RL) [21] using RAS as the reward—we significantly enhance system reliability. Experimental results demonstrate that our approach substantially improves transcription trustworthiness, particularly in low-resource and noisy conditions, while maintaining competitive accuracy. Our main contributions are summarized as follows: • We extend selective prediction to sequential ASR, enabling segment-level abstention instead of full-utterance rejection. • We propose RAS, a reliability-oriented metric calibrated with human preference data to quantify the trade-off between informativeness and error aversion. • We establish a robust training pipeline combining supervised learning and RL, which significantly elevates the reliability of ASR models in challenging acoustic environments. 2 Reliability-Aware Score To equip ASR models with explicit rejection capability, we extend the original vocabulary with a special placeholder mark ℋPH. Unlike ordinary lexical words, the ℋPH represents abstention: when the model encounters low-quality or ambiguous speech and cannot reliably determine the underlying content, it outputs ℋPH to indicate uncertainty rather than producing a potentially misleading guess. With the introduction of ℋPH, we now modify the standard WER into RAS, an abstention-aware metric, calculated by a proposed dynamic programming formulation that extends the definition of standard edit distance [14]. 2.1 Dynamic programming formulation Let the reference text ref of length N>0N>0 contains no ℋPHs, while the hypothesis ℎhyp of length M may contain ℋPHs. Let rir_i and hjh_j denote the i-th and j-th words (0-indexing) from ref and ℎhyp, respectively. Unlike standard edit distance, a single ℋPH may align to zero or multiple consecutive reference words, reflecting abstention over an uncertain acoustic segment. Consecutive ℋPHs are merged to avoid redundancy. When extending the definition of edit distance, it is inappropriate to treat errors introduced by ℋPHs in the same manner as ordinary word errors. Although failing to provide useful information, a ℋPH does not assert an incorrect lexical item and therefore avoids introducing potentially misleading content. In this sense, ℋPH-related errors are comparatively less harmful and more reliability-preserving. To reflect this distinction, we assign a cost factor α∈(0,1)α∈(0,1) to all ℋPH-related operations. We therefore define gi,jαg^α_i,j as the minimum weighted edit distance from [0:i)ref[0:i) to ℎ[0:j)hyp[0:j) under an abstention-aware alignment scheme, with boundary conditions: g0,0α=0,gi,0α=i,g0,jα=∑t=0j−11,ht≠ℋ,α,ht=ℋ,g^α_0,0=0, g^α_i,0=i, g^α_0,j= _t=0^j-1 cases1,&h_t ,\\ α,&h_t=PH, cases (1) and define the transition as the minimum of three potential costs: gi,jα=minSub,Del,Ins,g_i,j^α= \Sub,Del,Ins\, (2) where Sub =gi−1,j−1α+ri−1≠hj−1,hj−1≠ℋmin0≤k<igk,j−1α+α(i−k),hj−1=ℋ, = casesg_i-1,j-1^α+I_r_i-1≠ h_j-1,&h_j-1 \\ _0≤ k<i\g_k,j-1^α+α(i-k)\,&h_j-1=PH cases~, (3) Del =gi−1,jα+1, =g_i-1,j^α+1, (4) Ins =gi,j−1α+1,hj−1≠ℋgi,j−1α+α,hj−1=ℋ, = casesg_i,j-1^α+1,&h_j-1 \\ g_i,j-1^α+α,&h_j-1=PH cases~, (5) which corresponds to standard edit operations when hj−1≠ℋh_j-1 , but introduces a flexible alignment for ℋPHs. When hj−1=ℋh_j-1=PH, the substitution term Sub allows ℋPH to align with an arbitrary-length segment r[k:i)r_[k:i) in the reference, effectively ``absorbing′ i−ki-k deletions at a discounted cost of α per word. This corresponds to a many-to-one mapping where ℋPH covers a contiguous span of uncertain content. When no ℋPH appears in ℎhyp, the formulation reduces exactly to standard edit distance. We further define the weighted edit distance from ref to hyp as gα(ref,hyp):=gN,Mαg^α(ref,hyp):=g^α_N,M, computed in O(N2M)O(N^2M) time. Figure 2: Relationship between WER and RAS under different α settings on LibriSpeech-test-clean [22] (Base+PH-Supv+RL) as in Section 4.4. Each point corresponds to one utterance. 2.2 Beyond WER: The definition of RAS From the optimal DP alignment, we obtain the counts of correct matches C(ref,hyp)C(ref,hyp) and gα(ref,hyp)g^α(ref,hyp). Inspired by existing utility formulations for reliability-aware decision making in NLP [23], we define RAS R(ref,hyp)R(ref,hyp) as: R(ref,hyp)=Usefulness−Cost,R(ref,hyp)=Usefulness-Cost, (6) where usefulness measures the proportion of correctly transcribed words, and cost penalizes alignment errors with differential weighting as described in Section 2.1: Usefulness=C(ref,hyp)N,Cost=gα(ref,hyp)N,Usefulness= C(ref,hyp)N, = g^α(ref,hyp)N, (7) with α∈(0,1)α∈(0,1). This formulation directly balances informative content against erroneous commitments under the proposed alignment. It has a maximum of 1, and a higher RAS implies a more reliable and useful ASR model. When no ℋPH appears in ℎhyp, RAS reduces to be similar to standard WER: R(ref,hyp)=1−2(S+D)+IN,WER(ref,hyp)=S+D+IN,R(ref,hyp)=1- 2(S+D)+IN, \\ WER(ref,hyp)= S+D+IN, (8) both RAS and WER are monotonic functions of the total number of word errors (S+D+I)(S+D+I). Therefore, in the absence of ℋPHs, maximizing RAS is equivalent to minimizing WER in terms of optimization direction. The proposed objective holds a clear negative correlation with WER as shown in Figure 2. In practice, α should be calibrated to align with human preferences over reliability in specific application contexts, rather than selected heuristically. 2.3 Human preference alignment via listening tests Listening tests were conducted using the BeaqleJS framework [24]. For each audio sample i, participants were presented with the ground-truth transcript (GiG_i) and two simulated ASR outputs: a conventional transcription without ℋPHs (AiA_i) and an abstention-aware transcription (BiB_i). Transcription BiB_i was constructed by replacing recognition errors in AiA_i with ℋPHs and additionally masking a small fraction of correct words to simulate conservative abstention. Participants were asked to select the transcription they considered more reliable, with an additional ``Can’t Decide′ option. We estimate α from collected human preferences. Let the number of audios be K. For each audio i, let kiAk_i^A, kiBk_i^B, and kiCk_i^C denote the numbers of subjects preferring conventional transcription AiA_i, abstention-aware transcription BiB_i, and indicating indifference, respectively, with si=kiA+kiB+kiCs_i=k_i^A+k_i^B+k_i^C. Let PiP_i denote the probability of human preferring BiB_i for audio i, and R, C, gαg^α, N have the same meanings as in Section 2.2. We use notation ΔMi:=M(Gi,Bi)−M(Gi,Ai) M_i:=M(G_i,B_i)-M(G_i,A_i) where M∈R,C,gαM∈\R,C,g^α\. Following the Bradley–Terry model [25], PiP_i is modeled as a function of ΔRi R_i. Note that R(Gi,Ai)R(G_i,A_i), R(Gi,Bi)R(G_i,B_i) share the same denominator: Pi=σ(ΔRi)=σ[1Ni(ΔCi−Δgiα)],P_i=σ( R_i)=σ [ 1N_i ( C_i- g^α_i ) ], (9) where σ(⋅)σ(·) denotes the logistic function. We define the preference likelihood loss and incorporate indifference responses as a regularization encouraging small RAS differences. The final objective is the weighted sum of the two: ℒpref=−1K∑i[kiBsilogPi+kiAsilog(1−Pi)], _pref=- 1K _i [ k^B_is_i P_i+ k^A_is_i (1-P_i) ], (10) ℒtie=1K∑ikiCsi⋅(ΔRi)2, _tie= 1K _i k^C_is_i·( R_i)^2, (11) α∗=argminα∈(0,1)ℒpref+λ⋅ℒtie, α^*= α∈(0,1) ~L_pref+λ·L_tie, (12) where λ controls the strength of the indifference constraint. This formulation enables robust calibration of α to align the proposed RAS with human reliability preferences. 3 Training Abstention-Aware ASR Our training pipeline is designed in two consecutive stages to enhance the model's uncertainty awareness and self-correction capabilities. 3.1 Stage 1: placeholder supervision (PH-Supv) The primary objective of this stage is to construct a dataset that guides the base model to identify and flag prediction errors. y_gt: chronic disease of hair follicles and sebaceous gland y_hyp: the chronic disease of her and spoculus gland type: Insert Match Substitute Delete Match Substitute Match y_ph: <ph> chronic disease of <ph> <ph><ph><ph> and <ph><ph><ph> gland Figure 3: Strategy for generating Stage 1 yphy_ph. Incorrectly predicted segments are replaced with ℋPH. 3.1.1 Training set construction Given an audio input a and its ground truth transcription ygty_gt, we first conduct inference using the base model ℳM to generate a hypothesis yhypy_hyp. To locate recognition errors, we align yhypy_hyp with ygty_gt using the standard Word Error Rate (WER) calculation. This alignment yields a sequence of operations, where each operation is one of Match, Substitute, Insert, Delete\Match, Substitute, Insert, Delete\. We define a GT-guided ℋPH-replacement sequence yphy_ph, constructed from these operations as follows: Match: The corresponding text in yhypy_hyp is retained in yphy_ph. Substitute / Insert: The erroneous text segment terrt_err in yhypy_hyp is replaced by the special token ℋPH. To maintain relative sequence length, the number of ℋPH tokens is determined by the tokenizer T of the base model. Specifically, we insert N tokens of ℋPH, where N=|(terr)|N=|T(t_err)|. Delete: In this case, yhypy_hyp omits text present in ygty_gt. We insert ℋPH tokens into yphy_ph corresponding to the missing text segment tmisst_miss from ygty_gt. Similar to the above, the number of ℋPH tokens is set to N=|(tmiss)|N=|T(t_miss)|. Figure 3 shows an example of the process above. This process results in a refined training set consisting of pairs (a,yph)(a,y_ph). 3.1.2 Model training Prior to training, we expand the vocabulary of the base model ℳM to include the new token ℋPH, ensuring that the tokenizer treats it as a single, indivisible token. The model is then fine-tuned on the constructed dataset using the same ASR objective as ℳM. In this work, we employ Whisper [20] as the base model ℳM, and thus use a cross entropy objective in this stage. 3.2 Stage 2: group relative policy optimization (RL) Once the model acquires the capability to output ℋPH, we employ Group Relative Policy Optimization (GRPO) [26] algorithm to optimize its output policy, using utterance-level RAS proposed in Section 2.2 as the reward signal. Specifically, for each input prompt q, GRPO samples a group of G outputs o1,o2,…,oG\o_1,o_2,…,o_G\ from the current policy πθold _ _old. Each output oio_i is evaluated by the RAS reward function to obtain a reward rir_i. The group-relative advantage is then computed as: A^i=ri−mean(rjj=1G)std(rjj=1G). A_i= r_i-mean(\r_j\_j=1^G)std(\r_j\_j=1^G). The policy is optimized by maximizing the following objective: ℒGRPO(θ)=[1G∑i=1G1|oi|∑t=1|oi|(L^i,t−βDKLi,t)],L_GRPO(θ)=E [ 1G _i=1^G 1|o_i| _t=1^|o_i| ( L_i,t-β\,D_KL^i,t ) ], (13) L^i,t=minρi,tA^i,clip(ρi,t,1−ϵ,1+ϵ)A^i, L_i,t= \ _i,t A_i,~clip( _i,t,1-ε,1+ε) A_i \, (14) where ρi,t=πθ(oi,t∣q,oi,<t)πθold(oi,t∣q,oi,<t) _i,t= _θ(o_i,t q,\,o_i,<t) _ _old(o_i,t q,\,o_i,<t) is the importance sampling ratio, ϵε is the clipping parameter, and DKLi,tD_KL^i,t is the per-token KL divergence against the reference policy πref _ref as defined in [26]. 4 Experiments 4.1 Datasets We conduct experiments on two datasets: LibriSpeech [22], a widely used English audiobook corpus, and the TALCS Corpus [27], an English-Mandarin code-switching dataset. For LibriSpeech, we utilize train-clean-360 for training and test-clean for evaluation. To assess ASR reliability under adverse acoustic conditions, we simulate a noisy variant (Noisy LibriSpeech) by injecting Additive White Gaussian Noise on the total duration of original audio samples. Four distinct train and test subsets were generated with Signal-to-Noise Ratios (SNRs) fixed at 0,5,10,20\0,5,10,20\ dB, respectively. For all other datasets, we adhere to the official training, development, and test partitions. 4.2 Evaluation metric and human alignment results We adopt RAS (defined in Section 2.2) as our evaluation criterion, with α estimated by human alignment listening test. The test design follows Section 2.3. Audio samples were manually selected from the Medical ASR Recording Dataset [28] (173 samples) and the AMI Corpus [29] (191 samples), covering terminology-intensive medical speech and noise-corrupted conversational speech. For each sample, two transcript variants were constructed as described. The full sample list and transcript pairs are provided in the supplementary material. After validity filtering, 980 preference annotations were collected from 42 participants. The human oracle upper bound, computed via majority voting and annotator agreement with the consensus, reached 78.11%, indicating strong inter-annotator consistency. With λ=0.1λ=0.1 (corresponding to a low tie rate of 6.63%), minimizing the objective in Equation (12) yields α=0.5064α=0.5064 and an average RAS gap of ΔU=0.0461 U=0.0461, demonstrating alignment between the proposed RAS formulation and human reliability judgments. 4.3 Experimental setup We use the following experiment notations: Base & Base+Logit: We use Whisper-Tiny as the baseline(Base). Base+Logit applies logit-based ℋPH-replacement using token-level confidence aggregated multiplicatively (word-level for English; character/word-level for Chinese), following official heuristics111github.com/openai/whisper/discussions/1183. For each dataset, we replace tokens below a confidence bar with ℋPH, tuning bar (typically in [0.1,0.3][0.1,0.3]) to maximize RAS. Base+PH-Supv: This stage uses ℋPH replacements from Base predictions on the training set (details in Secion 3.1). We fine-tune the decoder and embeddings for 8 epochs using AdamW [30] with a batch size of 64, a 1.0×10−51.0× 10^-5 learning rate, 1,000 warmup steps, and linear decay. Base+PH-Supv+RL: Initialized from Base+PH-Supv, we further optimize ℋPH supervision via GRPO (Section 3.2). We unfreeze the decoder/embeddings and train with a global batch size of 64. For each prompt, G=8G=8 responses are sampled (512 samples/step) using temperature 0.7 and top-p 0.95. We employ an adaptive KL penalty [31] with β0=0.2 _0=0.2, updated every 50 steps: βt+1=βtexp(η(KLt−KLtarget)) _t+1= _t (η(KL_t-KL_target)), where η=0.02η=0.02 and KLtarget=30KL_target=30. We use the Adam optimizer [32] with a 2×10−62× 10^-6 peak learning rate (1,000-step warmup and linear decay) and early stopping based on the stabilization of the reward mean. GT-guided ℋPH-replacement: We construct ℋPH-replaced sequences by applying the algorithm in Secion 3.1.1 to Base outputs, guided by test set ground-truth. 4.4 Main results Table 1: RAS performance on LibriSpeech and TALCS. Boldface marks the best result per dataset, excluding GT-guided ℋPH-replacement systems. Method LibriSpeech TALCS RAS↑ Usefulness↑ Cost↓ RAS↑ Usefulness↑ Cost↓ Base 0.8603 0.9362 0.0759 −-0.1093 0.5874 0.6968 Base+Logit 0.8650 0.9349 0.0698 −-0.0650 0.5595 0.6245 Base+PH-Supv+RL (Ours) 0.8811 0.9376 0.0565 0.4786 0.7391 0.2940 GT-guided ℋPH-replacement 0.9031 0.9361 0.0329 0.3772 0.5874 0.2103 Figure 4: RAS on Noisy LibriSpeech. Table 1 reports results under clean (LibriSpeech) and code-switching (TALCS) conditions. Our method Base+PH-Supv+RL consistently outperforms both baselines across all metrics. On TALCS, where the base model yields a negative RAS (−0.11-0.11), our method achieves 0.480.48, demonstrating that ℋPH-supervised pretraining followed by RL is especially effective in challenging code-switching scenarios. The logit-only baseline (Base+Logit) provides marginal gains, confirming that naive confidence thresholding is insufficient. GT-guided ℋPH-replacement serves as a near-oracle upper bound guided by ground-truth. Figure 4 reports the performance under noisy conditions (Noisy LibriSpeech). At SNR=0SNR=0 dB, our method improves RAS by 0.26570.2657 over Base, compared to 0.02080.0208 improvement at clean conditions, highlighting the robustness benefit in highly noisy environments. The improvement in RAS brought by our method becomes increasingly pronounced as the SNR decreases, demonstrating that the proposed training pipeline, i.e., PH-Supv+RL, confers greater robustness gains in more challenging acoustic environments. 4.5 Ablation study Table 2: Ablation study for the two-stage training pipeline. Method LibriSpeech TALCS RAS↑ Usefulness↑ Cost↓ RAS↑ Usefulness↑ Cost↓ Base+PH-Supv 0.8696 0.9277 0.0581 0.4054 0.6520 0.2466 Base+PH-Supv+RL 0.8811 0.9376 0.0565 0.4786 0.7391 0.2940 Table 2 presents an ablation study isolating the contribution of GRPO-based RL training. Comparing Base+PH-Supv and Base+PH-Supv+RL, we observe that RL consistently improves RAS and Usefulness on both datasets. On TALCS, while Cost slightly increases, this reflects the RL objective's optimization toward RAS: the model learns that the Usefulness gain from predicting more words outweighs the Cost incurred when some ℋPH-replaced tokens are instead decoded incorrectly. The net effect is a substantially higher RAS, confirming that the GRPO-based RL stage provides complementary benefits on top of supervised placeholder training. Notably, on TALCS both Base+PH-Supv and Base+PH-Supv+RL in Table 2 surpass the GT-guided ℋPH-replacement upper bound in Table 1. Since Base performs poorly there, GT-guided replacement is fundamentally limited by its weak code-switching capability. PH-Supv training, by contrast, genuinely improves the model: even without full GT transcripts, the correct tokens retained after replacing Base's errors provide sufficient supervision to substantially boost TALCS performance, with RL further amplifying this advantage. 5 Conclusion This work redefines ASR reliability by introducing placeholder-based abstention, shifting the paradigm from speculative transcription to risk-aware reporting. We propose the RAS, a principled metric calibrated via human preferences to balance informativeness with error aversion. By implementing an abstention-aware training pipeline, we demonstrate that models can substantially improve reliability without sacrificing competitive accuracy. Our framework establishes a new criterion for trustworthy speech processing. We expect the RAS framework to provide a robust evaluation foundation for future research in trustworthy speech processing, particularly in serving as a reliable reward signal for optimizing model behavior through reinforcement learning. 6 Generative AI Use Disclosure Generative AIs are used in this work for manuscript polishing and code troubleshooting. We utilize these tools to improve linguistic clarity and assist in technical debugging. The conceptual framework, experimental design, and final writing are entirely conducted by human authors, who take full responsibility for the content and consent to its submission. References [1] C. K. Chow, ``An optimum character recognition system using decision functions,′ IRE Transactions on Electronic Computers, vol. EC-6, no. 4, p. 247–254, 1957. [2] V. Kuleshov and P. Liang, ``Calibrated structured prediction,′ in Proceedings of the 29th International Conference on Neural Information Processing Systems - Volume 2, ser. NIPS'15. Cambridge, MA, USA: MIT Press, 2015, p. 3474–3482. [3] H. Xu, Z. Zhu, L. Pan, Z. Wang, S. Zhu, D. Ma, R. Cao, L. Chen, and K. Yu, ``Reducing tool hallucination via reliability alignment,′ in Proceedings of the 42nd International Conference on Machine Learning, ser. Proceedings of Machine Learning Research, vol. 267. PMLR, 13–19 Jul 2025, p. 69 992–70 006. [4] H. Xu, Z. Zhu, S. Zhang, D. Ma, S. Fan, L. Chen, and K. Yu, ``Rejection improves reliability: Training LLMs to refuse unknown questions using RL from knowledge feedback,′ in First Conference on Language Modeling, 2024. [5] E. Schönwälder, C. Falkenberg, C. Hartmann, and W. Lehner, ``Abstention is all you need,′ 2025 IEEE 12th International Conference on Data Science and Advanced Analytics (DSAA), p. 1–10, 2025. [6] H. Zheng, H. Xu, Y. Liu, S. Fan, L. Chen, P. Fung, and K. Yu, ``Enhancing LLM reliability via explicit knowledge boundary modeling,′ in Second Conference on Language Modeling, 2025. [7] H. Jiang, ``Confidence measures for speech recognition: A survey,′ Speech Communications, vol. 45, p. 455–470, 2005. [8] D. Oneaţă, A. Caranica, A. Stan, and H. Cucu, ``An evaluation of word-level confidence estimation for end-to-end automatic speech recognition,′ 2021 IEEE Spoken Language Technology Workshop (SLT), p. 258–265, 2021. [9] H. Futami, H. Inaguma, M. Mimura, S. Sakai, and T. Kawahara, ``ASR rescoring and confidence estimation with electra,′ 2021 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), p. 380–387, 2021. [10] B. Naowarat, T. Kongthaworn, and E. Chuangsuwanich, ``Word-level Confidence Estimation for CTC Models,′ in Interspeech 2023, 2023, p. 3297–3301. [11] M. Huo, Y. Zhang, and Y. Tang, ``Identifying and calibrating overconfidence in noisy speech recognition,′ arXiv preprint arXiv:2509.07195, 2025. [12] L. R. Rabiner, A tutorial on hidden Markov models and selected applications in speech recognition. San Francisco, CA, USA: Morgan Kaufmann Publishers Inc., 1990, p. 267–296. [13] National Institute of Standards and Technology, ``The 1997 hub-5ne evaluation plan for recognition of conversational speech over the telephone,′ https://catalog.ldc.upenn.edu/docs/LDC2002S25/hub5nev3.htm, 1997, accessed Feb 14, 2026. [14] V. I. Levenshtein, ``Binary codes capable of correcting deletions, insertions, and reversals,′ Soviet physics. Doklady, vol. 10, p. 707–710, 1965. [15] D. Thennal, J. James, D. P. Gopinath et al., ``Advocating character error rate for multilingual ASR evaluation,′ in Findings of the Association for Computational Linguistics: NAACL 2025, 2025, p. 4926–4935. [16] A. C. Morris, V. Maier, and P. Green, ``From WER and RIL to MER and WIL: improved evaluation measures for connected speech recognition,′ in Interspeech 2004, 2004, p. 2765–2768. [17] K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu, ``BLEU: a method for automatic evaluation of machine translation,′ in Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics. Philadelphia, Pennsylvania, USA: Association for Computational Linguistics, Jul. 2002, p. 311–318. [18] C.-Y. Lin, ``ROUGE: A package for automatic evaluation of summaries,′ in Text Summarization Branches Out. Barcelona, Spain: Association for Computational Linguistics, Jul. 2004, p. 74–81. [19] T. Zhang, V. Kishore, F. Wu, K. Q. Weinberger, and Y. Artzi, ``BERTScore: Evaluating text generation with BERT,′ in International Conference on Learning Representations, 2020. [20] A. Radford, J. W. Kim, T. Xu, G. Brockman, C. Mcleavey, and I. Sutskever, ``Robust speech recognition via large-scale weak supervision,′ in Proceedings of the 40th International Conference on Machine Learning, ser. Proceedings of Machine Learning Research, vol. 202. PMLR, 23–29 Jul 2023, p. 28 492–28 518. [21] R. S. Sutton, A. G. Barto et al., Reinforcement learning: An introduction. MIT press Cambridge, 1998, vol. 1, no. 1. [22] V. Panayotov, G. Chen, D. Povey, and S. Khudanpur, ``Librispeech: An asr corpus based on public domain audio books,′ in 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2015, p. 5206–5210. [23] H. Xu, Z. Wang, Z. Zhu, L. Pan, X. Chen, S. Fan, L. Chen, and K. Yu, ``Alignment for efficient tool calling of large language models,′ in Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, C. Christodoulopoulos, T. Chakraborty, C. Rose, and V. Peng, Eds. Suzhou, China: Association for Computational Linguistics, Nov. 2025, p. 17 776–17 792. [24] S. Kraft and U. Zölzer, ``Beaqlejs: Html 5 and javascript based framework for the subjective evaluation of audio quality.′ Linux Audio Conference, 05 2014. [25] R. A. Bradley and M. E. Terry, ``Rank analysis of incomplete block designs: I. the method of paired comparisons,′ Biometrika, vol. 39, no. 3/4, p. 324–345, 1952. [26] Z. Shao, P. Wang, Q. Zhu, R. Xu, J. Song, X. Bi, H. Zhang, M. Zhang, Y. Li, Y. Wu et al., ``Deepseekmath: Pushing the limits of mathematical reasoning in open language models,′ arXiv preprint arXiv:2402.03300, 2024. [27] C. Li, S. Deng, Y. Wang, G. Wang, Y. Gong, C. Chen, and J. Bai, ``TALCS: An open-source Mandarin-English code-switching corpus and a speech recognition baseline,′ in Interspeech 2022, 2022, p. 1741–1745. [28] Hani89, ``Medical ASR recording dataset,′ 2023. [Online]. Available: https://huggingface.co/datasets/Hani89/medical_asr_recording_dataset [29] I. Mccowan, J. Carletta, W. Kraaij, S. Ashby, S. Bourban, M. Flynn, M. Guillemot, T. Hain, J. Kadlec, V. Karaiskos, M. Kronenthal, G. Lathoud, M. Lincoln, A. Lisowska Masson, W. Post, D. Reidsma, and P. Wellner, ``The AMI meeting corpus,′ Int'l. Conf. on Methods and Techniques in Behavioral Research, 01 2005. [30] I. Loshchilov and F. Hutter, ``Decoupled weight decay regularization,′ in International Conference on Learning Representations, 2019. [31] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, ``Proximal policy optimization algorithms,′ arXiv preprint arXiv:1707.06347, 2017. [32] D. P. Kingma and J. Ba, ``Adam: A method for stochastic optimization,′ in International Conference on Learning Representations (ICLR), 2015.