Paper deep dive
Which Medical Questions Deserve Rationales? Perturbation-Sensitive Selection for Robust QA
Yuexin Wu, Dayou Yu, Vasile Rus
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 91%
Last extracted: 9/10/2026, 3:11:02 AM
Summary
The paper introduces RMS-RSP, a perturbation-sensitive method for selecting high-quality rationales for medical question-answering models under a fixed token budget. By perturbing hidden states at rationale tokens and measuring shifts in the gold-versus-distractor margin, RMS-RSP identifies supervision that improves robustness to answer-option reordering and semantic consistency, though it does not universally improve standard accuracy compared to random selection.
Entities (10)
Relation Signals (9)
RMS-RSP → evaluatedon → MedMCQA
confidence 95% · Across five medical QA datasets... RMS-RSP provides a deliberately qualified result
RMS-RSP → evaluatedon → AfriMed-QA
confidence 95% · Across five medical QA datasets... RMS-RSP provides a deliberately qualified result... with a statistically resolved gain only on AfriMed-QA
RMS-RSP → evaluatedon → MedExpQA
confidence 95% · Across five medical QA datasets... RMS-RSP provides a deliberately qualified result
RMS-RSP → evaluatedon → MedExQA
confidence 95% · Across five medical QA datasets... RMS-RSP provides a deliberately qualified result
RMS-RSP → evaluatedon → PubMedQA
confidence 95% · Across five medical QA datasets... RMS-RSP provides a deliberately qualified result
MedGemma-4B-IT → usedwith → RMS-RSP
confidence 92% · All runs use MedGemma-4B-IT... For RMS-RSP... We compare against Random... RMS-RSP provides a deliberately qualified result.
RMS-RSP → improves → ReAcc
confidence 90% · RMS-RSP has the best macro ReAcc (0.4260) and ReCon (0.5694), improving over Random by 1.91 and 2.85 percentage points.
RMS-RSP → improves → ReCon
confidence 90% · RMS-RSP has the best macro ReAcc (0.4260) and ReCon (0.5694), improving over Random by 1.91 and 2.85 percentage points.
Signed-RSP → isvariantof → RMS-RSP
confidence 85% · Our Signed-RSP ablation instead uses the mean signed drop... This tradeoff supports preserving perturbation magnitude when the goal is invariance
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Medical question-answering datasets often contain answer labels, whereas high-quality rationales remain scarce, noisy, or costly to validate. This changes the acquisition question: rather than asking which questions should be labeled, we ask which already-labeled questions should receive rationale supervision under a fixed token budget. We study an offline version of this problem in which candidate rationales are visible to the selector but withheld from downstream training unless selected. We propose root-mean-square Robustness-based Sample Prioritization (RMS-RSP), which perturbs hidden states only at rationale tokens and measures the resulting shift in the gold-versus-best-distractor margin. Across five medical QA datasets, MedGemma-4B-IT, three training seeds, ten budgeted non-RSP selectors, and an unbudgeted full-supervision reference, RMS-RSP provides a deliberately qualified result. Its locked-budget accuracy is 60.61% on average versus 60.08% for Random, with a statistically resolved gain only on AfriMed-QA (+1.44 points). Its full-budget accuracy area is not better than Random. However, after three answer-option reorderings, RMS-RSP improves robust accuracy and semantic consistency by 1.91 and 2.85 points on average, respectively, with the same direction on all five datasets. Training on every pool rationale raises macro accuracy to 63.74%, but consumes 29--254 times more rationale tokens and does not uniformly improve robustness. These findings do not establish universal accuracy gains; they instead suggest that rationale-local boundary sensitivity can identify supervision that improves invariance to semantically equivalent formatting changes.
Tags
Links
- Source: https://arxiv.org/abs/2609.09684v1
- Canonical: https://arxiv.org/abs/2609.09684v1
Trouble viewing inline? Open PDF directly →
Full Text
36,115 characters extracted from source content.
Expand or collapse full text
Which Medical Questions Deserve Rationales? Perturbation-Sensitive Selection for Robust QA Yuexin Wu Dayou Yu Vasile Rus Affiliation: Department of Computer Science, University of Memphis Affiliation: ywu10@memphis.edu dayou.yu@memphis.edu vrus@memphis.edu Abstract Medical question-answering datasets often contain answer labels, whereas high-quality rationales remain scarce, noisy, or costly to validate. This changes the acquisition question: rather than asking which questions should be labeled, we ask which already-labeled questions should receive rationale supervision under a fixed token budget. We study an offline version of this problem in which candidate rationales are visible to the selector but withheld from downstream training unless selected. We propose root-mean-square Robustness-based Sample Prioritization (RMS-RSP), which perturbs hidden states only at rationale tokens and measures the resulting shift in the gold-versus-best-distractor margin. Across five medical QA datasets, MedGemma-4B-IT, three training seeds, ten budgeted non-RSP selectors, and an unbudgeted full-supervision reference, RMS-RSP provides a deliberately qualified result. Its locked-budget accuracy is 60.61% on average versus 60.08% for Random, with a statistically resolved gain only on AfriMed-QA (+1.44 points). Its full-budget accuracy area is not better than Random. However, after three answer-option reorderings, RMS-RSP improves robust accuracy and semantic consistency by 1.91 and 2.85 points on average, respectively, with the same direction on all five datasets. Training on every pool rationale raises macro accuracy to 63.74%, but consumes 29–254 times more rationale tokens and does not uniformly improve robustness. These findings do not establish universal accuracy gains; they instead suggest that rationale-local boundary sensitivity can identify supervision that improves invariance to semantically equivalent formatting changes. 1 Introduction Medical QA datasets commonly provide answer labels at much greater scale than carefully checked explanations. Rationales are longer, require domain expertise to write or verify, and may contain irrelevant or incorrect intermediate statements even when the final answer is correct. The relevant allocation problem is therefore not only which questions should be labeled, as in conventional active learning (Settles, 2009), but which already-labeled questions deserve additional rationale supervision. This distinction matters in medical settings: applying all available rationales indiscriminately can spend substantial annotation or training budget while exposing the model to noisy reasoning traces. Recent reasoning-data selectors rank traces by answer uncertainty, likelihood, local step compatibility, or early training dynamics (Goncharov et al., 2026; Yang et al., 2026; Just et al., 2026; Wang et al., 2026; Jin et al., 2026). These methods have advanced data-efficient reasoning distillation, but most were developed for mathematics, coding, or general science. Moreover, their scores do not directly ask whether the rationale is coupled to the medical decision boundary. In multiple-choice medical QA, this boundary is especially important: a useful rationale should support the correct clinical option over its strongest distractor rather than merely be fluent or difficult. We revisit Robustness-based Sample Prioritization (RSP) with a normalized, rationale-local score. Given an answer-trained model and a candidate rationale, we inject RMS-scaled Gaussian noise into the rationale-token hidden states at several late layers. We then measure how much the gold-versus-best-distractor log-probability margin changes. The canonical RMS-RSP score is the root mean square of these shifts. It is scale-normalized, sensitive to both positive and negative boundary changes, and tied to the tokens that would be acquired as supervision. Our evaluation is designed around the limits of the current evidence. We use five medical QA datasets spanning African medical examinations, multilingual medical exams, underrepresented specialties, biomedical literature, and Indian entrance examinations (Nimo et al., 2025; Alonso et al., 2024; Kim et al., 2024; Jin et al., 2019; Pal et al., 2022). We compare against Random, answer entropy and margin, rationale length, and five recent reasoning-data selectors. We report locked-budget accuracy, macro-F1, full-curve Token-AUBC, and invariance to answer-option permutations. Our contributions are: • a precise formulation of budgeted rationale selection for answer-labeled medical QA, separating answer supervision from rationale supervision; • RMS-RSP, a rationale-local, relative-scale perturbation score based on gold–distractor margin shifts, together with a signed ablation; • a five-dataset comparison showing heterogeneous standard accuracy but consistent improvements in option-order robust accuracy and semantic consistency, together with a high-resource all-rationales reference. 2 Related work Rationale supervision in medical QA. Chain-of-thought prompting and rationale fine-tuning can improve multi-step reasoning (Wei et al., 2022; Zelikman et al., 2022), but generated explanations need not faithfully describe the computation that produced an answer (Turpin et al., 2023). This concern is amplified in healthcare, where an incorrect intermediate claim can be consequential. Medical benchmarks vary widely in explanation provenance: MedMCQA supplies short explanations, MedExpQA provides physician-written reference explanations, MedExQA provides explanation pairs, and PubMedQA pairs decisions with article conclusions (Pal et al., 2022; Alonso et al., 2024; Kim et al., 2024; Jin et al., 2019). We treat these rationales as an offline acquisition oracle rather than assuming that every rationale is equally useful. Reasoning-data selection. Uncertainty sampling selects difficult inputs from model outputs. Complexity-aware fine-tuning uses answer entropy to reserve reasoning supervision for complex items (Goncharov et al., 2026). RSR balances token rank and surprisal to identify trajectories aligned with but informative to a student (Yang et al., 2026). LALP scores each step from a restricted local context (Just et al., 2026), while ASLEC-DROP removes low-probability first-step tokens that confound naturalness scores with step length (Wang et al., 2026). TEMP uses losses under random and fine-tuning-direction parameter perturbations and observes that useful reasoning traces can be identified early (Jin et al., 2026). These are close baselines because they address the same supervision-allocation question. RSP differs by perturbing activations at rationale positions and measuring an answer-boundary response. Multiple-choice robustness. LLMs can change predictions when answer options are reordered even though their semantic content is unchanged (Pezeshkpour and Hruschka, 2024; Zheng et al., 2024). We therefore evaluate more than original-order accuracy. Option permutations are not claimed to simulate every clinical distribution shift; they are a controlled invariance test aligned with RSP’s decision-boundary motivation. 3 Budgeted rationale selection 3.1 Problem setup Let the pool be =(xi,i,yi,ri)i=1ND=\(x_i,O_i,y_i,r_i)\_i=1^N, where xix_i is a medical question (and context, when present), iO_i is its option set, yiy_i is the known correct option, and rir_i is a candidate rationale. We first train an answer-only model fθ0f_ _0 on all (xi,yi)(x_i,y_i) pairs. Rationales are not used in this stage. Each rationale has token cost cic_i. A selector ranks candidates and chooses S⊆1,…,NS \1,…,N\ such that ∑i∈Sci≤B _i∈ Sc_i≤ B. Only ri:i∈S\r_i:i∈ S\ are unlocked for the rationale-training branch. In our offline experiments, rir_i is visible to rationale-aware scoring functions; consequently, the setup models selection from existing or cheaply generated candidate explanations for validation/training, not selection before any rationale has been produced. This scope distinction is central to our claims. 3.2 Rationale-local perturbations For candidate i, we append the rationale and the suffix “Final Answer:” to the question and options. Let aija_ij be the logit for option j at the answer position. The clean gold–distractor margin is mi=aiyi−maxj≠yiaij.m_i=a_iy_i- _j≠ y_ia_ij. (1) At transformer layer ℓ , let Hi(ℓ)∈ℝT×dH_i^( ) ^T× d be the hidden states and Mi,R∈0,1T×1M_i,R∈\0,1\^T× 1 mask only the rationale tokens. We perturb H~i(ℓ,k)=Hi(ℓ)+αRMS(Hi,R(ℓ))(Mi,R⊙Zi(ℓ,k)),Zi(ℓ,k)∼(0,I), H_i^( ,k)=H_i^( )+α\,RMS\! (H_i,R^( ) ) (M_i,R Z_i^( ,k) ), Z_i^( ,k) (0,I), (2) where RMS(Hi,R(ℓ))RMS(H_i,R^( )) is computed across masked tokens and hidden dimensions. Relative scaling avoids comparing a fixed absolute noise level across layers with different activation magnitudes. Restricting noise to rationale positions asks how strongly the answer boundary depends on that candidate explanation rather than on the question generally. Let m~i(ℓ,k) m_i^( ,k) be the perturbed margin and di(ℓ,k)=mi−m~i(ℓ,k)d_i^( ,k)=m_i- m_i^( ,k). For layers ℒL and K perturbations per layer, the canonical score is siRMS-RSP=1|ℒ|K∑ℓ∈ℒ∑k=1K(di(ℓ,k))2.s_i^RMS-RSP= 1|L|K _ _k=1^K (d_i^( ,k) )^2. (3) We select larger scores first, greedily skipping an item when its cost would exceed the remaining token budget. RMS magnitude is direction-agnostic: either a reduction or increase in the gold margin indicates that the boundary is locally sensitive to the rationale representation. Our Signed-RSP ablation instead uses the mean signed drop, 1|ℒ|K∑ℓ,kdi(ℓ,k) 1|L|K _ ,kd_i^( ,k). 3.3 Downstream rationale training Starting from the answer-only adapter, each selected item contributes an answer replay record and a rationale-generation record ending in the correct answer. We additionally sample one answer-only replay item from the unselected pool per selected item. This paired design reduces the chance that a selector wins merely by changing the amount of answer supervision. At inference, the model directly scores answer letters without generating a rationale. 4 Experimental design 4.1 Datasets and splits Table 1 summarizes the frozen splits. We use the English subset of MedExpQA and the expert multiple-choice portion of AfriMed-QA. Multi-answer AfriMed-QA rows are removed. MedExQA and PubMedQA use deterministic derived splits after exact deduplication; the others preserve official test partitions where available. No test item is used to choose a token budget. The MedMCQA test set had been evaluated in earlier exploratory work, so its result is protocol-aligned rather than a pristine confirmation. Table 1: Datasets and frozen protocol. “RSP n” is the number of rationales selected by canonical RMS-RSP at the development-locked token budget. Dataset Pool Dev Test Options Tokens B RSP n AfriMed-QA 1,500 559 892 4–5 512 4 MedExpQA 434 63 125 4–5 256 5 MedExQA 600 164 200 4 512 7 PubMedQA 600 200 200 3 1,024 20 MedMCQA 512 256 4,096 4 1,024 10 4.2 Model, acquisition, and training All runs use MedGemma-4B-IT (Sellergren et al., 2025) with LoRA adapters (Hu et al., 2022). The answer-only adapter is trained for one epoch on every pool answer (r=16r=16, LoRA α=32α=32, learning rate 2×10−42× 10^-4). Each rationale branch starts from that adapter and trains for two epochs at 10−410^-4 with sequence length 1,024, batch size 1, gradient accumulation 8, and one-to-one unselected answer replay. We use training seeds 13, 23, and 37. For RMS-RSP, ℒ=−2,−4,−8L=\-2,-4,-8\, K=4K=4, and perturbation scale α=0.10α=0.10. Candidate budgets are 256, 512, and 1,024 rationale tokens. For each dataset, development data select one shared budget by maximizing RMS-RSP accuracy minus the strongest non-RSP selector at that budget, breaking ties toward fewer tokens. We then evaluate all methods once at the locked budget. Because this rule is centered on RMS-RSP, we also report Token-AUBC over all budgets to prevent a favorable single budget from carrying the conclusion. Random uses three acquisition seeds crossed with the three training seeds (nine runs per budget). Deterministic selectors use three training seeds. The same selected set is used across downstream seeds; random acquisition varies independently. As a high-resource reference, we additionally train from the same answer-only adapters on every pool rationale for the same two epochs. This full-supervision condition consumes 29,399–129,935 scored rationale tokens, depending on the dataset, and has no unselected replay pool. It is neither token- nor update-matched to the budgeted branches and is therefore reported as an unbudgeted reference rather than a competing selector. 4.3 Baselines We compare with answer entropy, negative top-two answer margin, rationale length, and Random. Five recent selectors are adapted to the shared candidate-rationale pool: • Complexity-aware FT ranks by answer-position vocabulary entropy (Goncharov et al., 2026). • RSR uses token-rank over surprisal, with the sign reversed so larger scores are selected (Yang et al., 2026). • LALP averages step likelihood from the question and the preceding 25% of steps (Just et al., 2026). • ASLEC-DROP averages rationale-token likelihood after dropping the first token of each step (Wang et al., 2026). • TEMP combines losses under calibrated random parameter perturbations and checkpoints extrapolated along a small rationale-LoRA direction (Jin et al., 2026). Because the medical datasets lack consistent step annotations, LALP and ASLEC share a deterministic newline-and-sentence segmentation. These are protocol adaptations rather than claims of exact reproduction under the original models and datasets. 4.4 Metrics and uncertainty The primary task metric is multiple-choice accuracy; macro-F1 is secondary. We also compute Brier score (Brier, 1950) and expected calibration error (Guo et al., 2017) in the released result artifacts. Token-AUBC is trapezoidal area under test accuracy at 0, 256, 512, and 1,024 acquired rationale tokens, normalized by 1,024; the zero-token point is the answer-only model. For option robustness, each item receives three deterministic, distinct, non-identity permutations. Predictions are mapped back to the original semantic options. ReAcc requires a correct prediction on the original and all three permutations. ReCon requires the same semantic prediction across all four versions, irrespective of correctness. No model is retrained for this evaluation. Paired 95% intervals use 10,000 hierarchical bootstrap draws over downstream seeds and test items; comparisons with Random additionally resample acquisition seeds. 5 Results 5.1 Standard accuracy is positive in some settings, not universal Table 2 reports the shared development-locked budget for each dataset. Canonical RMS-RSP is best on AfriMed-QA (64.16%) and exceeds Random by 1.44 points with a 95% interval of [0.21,2.77][0.21,2.77]. Its point estimate also exceeds Random on MedExpQA (+0.09) and PubMedQA (+1.61), ties on MedExQA, and trails by 0.46 on MedMCQA. The five-dataset macro average is 60.61% versus 60.08% for Random. Only AfriMed-QA resolves a nonzero accuracy difference, so the evidence does not support a claim of consistent accuracy dominance. Table 2: Test accuracy (%, mean ± standard deviation across three training seeds). Random averages 3×33× 3 acquisition/training runs. Bold marks the best token-budgeted method; All rationales is an unbudgeted reference. Method AfriMed MedExp MedEx PubMed MedMCQA Macro Random ×33\!×\!3 ±0.8562.72\!±\!0.85 ±4.8054.84\!±\!4.80 ±0.8361.83\!±\!0.83 ±0.4269.06\!±\!0.42 ±2.1151.93\!±\!2.11 60.08 Answer entropy ±1.2061.85\!±\!1.20 ±7.4355.73\!±\!7.43 ±0.5060.50\!±\!0.50 ±0.8769.50\!±\!0.87 ±2.0151.52\!±\!2.01 59.82 Answer margin ±0.2363.53\!±\!0.23 ±5.6055.20\!±\!5.60 ±2.2963.00\!±\!2.29 ±4.1968.33\!±\!4.19 ±0.7052.12\!±\!0.70 60.44 Rationale length ±0.3063.12\!±\!0.30 ±6.5552.00\!±\!6.55 ±1.3262.50\!±\!1.32 ±0.8770.50\!±\!0.87 ±0.9653.85\!±\!0.96 60.39 Complexity-aware FT ±2.4461.25\!±\!2.44 ±7.2655.47\!±\!7.26 ±1.2661.83\!±\!1.26 ±0.2969.67\!±\!0.29 ±2.5051.03\!±\!2.50 59.85 RSR ±0.6862.52\!±\!0.68 ±5.6255.73\!±\!5.62 ±2.2560.67\!±\!2.25 ±0.7668.67\!±\!0.76 ±1.8252.48\!±\!1.82 60.01 ASLEC-DROP ±1.1860.76\!±\!1.18 ±3.3354.67\!±\!3.33 ±2.0062.50\!±\!2.00 ±0.5869.33\!±\!0.58 ±2.9150.06\!±\!2.91 59.46 TEMP ±0.7863.68\!±\!0.78 ±8.0550.67\!±\!8.05 ±2.0260.83\!±\!2.02 ±6.0665.50\!±\!6.06 ±0.2852.69\!±\!0.28 58.67 Signed-RSP ±0.8363.98\!±\!0.83 ±5.3354.13\!±\!5.33 ±1.4462.67\!±\!1.44 ±1.0471.17\!±\!1.04 ±1.0252.38\!±\!1.02 60.8760.87 RMS-RSP ±0.5164.16\!±\!0.51 ±6.9954.93\!±\!6.99 ±2.5761.83\!±\!2.57 ±0.5870.67\!±\!0.58 ±0.4951.46\!±\!0.49 60.61 All rationales ±0.5664.61\!±\!0.56 ±3.0361.07\!±\!3.03 ±1.6163.17\!±\!1.61 ±1.2672.17\!±\!1.26 ±0.8357.67\!±\!0.83 63.74 Macro-F1 tells a similarly restrained story. RMS-RSP averages 55.71, essentially matching Random at 55.68. Signed-RSP reaches 56.50, while answer margin is best at 56.86. Thus the canonical score’s main empirical advantage is not general discrimination performance. 5.2 Boundary-sensitive selection improves option-order robustness Table 3 shifts from one locked operating point to full-budget efficiency and controlled invariance. LALP has the best accuracy Token-AUBC (0.6045); canonical RMS-RSP is below Random (0.5960 versus 0.6021). In contrast, RMS-RSP has the best macro ReAcc (0.4260) and ReCon (0.5694), improving over Random by 1.91 and 2.85 percentage points. The robustness gains are directionally positive on every dataset (Figure 1). AfriMed-QA resolves both differences; MedExQA’s ReAcc interval touches zero at its lower endpoint. The remaining intervals include zero. Table 3: Five-dataset macro averages. Token-AUBC uses original-order accuracy. Bold marks the best budgeted method; All rationales is an unbudgeted reference and has no Token-AUBC. Method Token-AUBC ReAcc ReCon Answer-only 0.5996 0.3915 0.5115 Random 3×33× 3 0.6021 0.4070 0.5408 Answer entropy 0.5919 0.3869 0.5105 Answer margin 0.5994 0.4016 0.5290 Rationale length 0.6018 0.4055 0.5361 Complexity-aware FT 0.5919 0.3972 0.5245 RSR 0.6014 0.4074 0.5449 LALP 0.6045 0.4039 0.5322 ASLEC-DROP 0.6009 0.3893 0.5098 TEMP 0.5954 0.3592 0.4663 Signed-RSP 0.6025 0.4023 0.5315 RMS-RSP 0.5960 0.4260 0.5694 All rationales – 0.4539 0.5861 Figure 1: Canonical RMS-RSP minus Random in percentage points. Points are means; bars are paired 95% hierarchical bootstrap intervals over training seeds, Random acquisition seeds, and test items. Robustness differences are positive on all five datasets, whereas original-order accuracy is heterogeneous. The metric separation is informative. RMS magnitude rewards rationales whose representations are strongly coupled to the gold–distractor boundary, regardless of the direction of a particular perturbation. That coupling is plausibly useful for learning a less position-dependent answer rule, but it does not guarantee that every selected rationale improves clean accuracy. We treat this as an empirical interpretation, not a causal proof. 5.3 Signed versus RMS aggregation The signed ablation is better on locked-budget macro accuracy (60.87 versus 60.61) and macro-F1 (56.50 versus 55.71), while canonical RMS is substantially better on robustness (ReAcc 42.60 versus 40.23; ReCon 56.94 versus 53.15). This tradeoff supports preserving perturbation magnitude when the goal is invariance, but also shows that the aggregation choice should be tied to the deployment objective. A single RSP variant is not uniformly best. 5.4 Full supervision improves average performance at much higher cost The all-rationales reference reaches 63.74% macro accuracy and 59.84 macro-F1, exceeding RMS-RSP by 3.12 and 4.13 points, respectively. It also improves macro ReAcc by 2.79 points and ReCon by 1.67 points. These averages conceal substantial heterogeneity (Table 4). Only MedMCQA resolves a nonzero all-rationales gain over RMS-RSP on all three metrics: +6.21+6.21 accuracy points (95%95\% CI [4.93,7.45][4.93,7.45]), +8.41+8.41 ReAcc points ([5.47,11.03][5.47,11.03]), and +10.31+10.31 ReCon points ([6.37,13.63][6.37,13.63]). Conversely, on PubMedQA the full reference gains 1.50 accuracy points but loses 4.83 ReAcc and 6.83 ReCon points; all three intervals include zero, and robustness varies strongly across seeds. Table 4: Unbudgeted all-rationales reference. Token multiple is relative to the locked RMS-RSP budget; deltas are All rationales minus RMS-RSP in percentage points. Only the three MedMCQA intervals exclude zero. Dataset Rationales Tokens Token multiple Accuracy Δ ReAcc Δ ReCon Δ AfriMed-QA 1,500 129,935 254×254× +0.45+0.45 +0.41+0.41 −0.04-0.04 MedExpQA 434 45,361 177×177× +6.13+6.13 +8.27+8.27 +4.27+4.27 MedExQA 600 62,209 122×122× +1.33+1.33 +1.67+1.67 +0.67+0.67 PubMedQA 600 29,399 29×29× +1.50+1.50 −4.83-4.83 −6.83-6.83 MedMCQA 512 59,121 58×58× +6.21+6.21 +8.41+8.41 +10.31+10.31 Macro – – – +3.12+3.12 +2.79+2.79 +1.67+1.67 6 Discussion and limitations What the evidence supports. The strongest defensible claim is narrow: under a fixed rationale-token budget, selecting examples whose rationale-token representations exert a large local effect on the gold–distractor boundary can improve invariance to answer-option reorderings. The evidence does not support “RSP consistently improves medical QA accuracy.” Accuracy gains are dataset- and budget-dependent, and only AfriMed-QA has a clearly nonzero locked-budget gain over Random. Practical interpretation. RSP is best viewed as a curation layer for candidate explanations that already exist—for example, archived dataset rationales or inexpensive machine-generated drafts—before scarce expert validation or downstream training budget is spent. The present experiment measures the value of selecting rationale tokens for training; it does not measure clinician annotation time or the quality improvement from expert rewriting. A prospective study should compare total generation, review, and correction cost. Full-supervision reference. Using every rationale improves average discrimination and robustness, showing that additional rationale supervision can be valuable when cost is unconstrained. However, it requires 29–254 times the locked token budget, its advantage is concentrated in MedExpQA and MedMCQA, and it reduces the PubMedQA robustness point estimates. The appropriate claim is therefore efficiency under scarcity: RMS-RSP often approaches the high-resource reference with far fewer rationale tokens, not that selection universally outperforms full supervision. Candidate-rationale visibility. The selector reads each rationale before deciding whether to unlock it for downstream training. This is appropriate for offline curation but not for classic active learning in which the annotation does not yet exist. Calling the method pre-annotation acquisition without this qualification would overstate its scope. A future proxy that scores questions without candidate rationales, or uses cheap drafts before expert review, is needed for that setting. Experimental limits. We evaluate one 4B medical model, three training seeds, and relatively small acquisition pools. The token budgets sometimes unlock very few rationales (four on AfriMed-QA at 512 tokens), increasing sensitivity to individual traces. The all-rationales reference is neither token- nor update-matched: processing many more rationales also entails many more gradient updates, so it is a high-resource comparison rather than a causal estimate of selection quality. MedExQA and PubMedQA use derived splits, and MedMCQA is not a pristine confirmatory test. Development selection explicitly favors the canonical method; Token-AUBC mitigates but does not remove that concern. We do not correct for multiple comparisons. Option permutations probe a real MCQ failure mode but are narrower than clinical distribution shift, factuality, harm, or calibration under deployment. Dataset rationales are treated as supervision without new clinician auditing, so we cannot attribute gains to clinical explanation quality. Finally, these are benchmark experiments and do not validate the model for diagnosis or patient care. 7 Conclusion We formulated budgeted rationale selection for answer-labeled medical QA and evaluated a rationale-local perturbation score against recent reasoning-data selectors on five datasets. Canonical RMS-RSP does not dominate standard accuracy or Token-AUBC. Training on every rationale improves average performance but costs 29–254 times more rationale tokens and is not uniformly more robust. Within the low-budget comparison, RMS-RSP’s reproducible advantage is a consistent increase in robustness and semantic consistency under answer-option permutations. This result suggests a promising, appropriately limited role for representation sensitivity: not as a universal difficulty score, but as a mechanism-aligned signal for selecting rationale supervision when stable medical decisions matter. References Alonso et al. (2024) I. Alonso, M. Oronoz, and R. Agerri MedExpQA: multilingual benchmarking of large language models for medical question answering. Artificial Intelligence in Medicine 155, p. 102938. External Links: Document Cited by: §1, §2. Brier (1950) G. W. Brier Verification of forecasts expressed in terms of probability. Monthly Weather Review 78 (1), p. 1–3. Cited by: §4.4. Goncharov et al. (2026) A. Goncharov, D. Vyazhev, P. Sychev, E. Khalafyan, and A. Zaytsev Complexity-aware fine-tuning. In Findings of the Association for Computational Linguistics: EACL 2026, p. 682–696. External Links: Document Cited by: §1, §2, 1st item. Guo et al. (2017) C. Guo, G. Pleiss, Y. Sun, and K. Q. Weinberger On calibration of modern neural networks. In Proceedings of the 34th International Conference on Machine Learning, p. 1321–1330. Cited by: §4.4. Hu et al. (2022) E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen LoRA: low-rank adaptation of large language models. In International Conference on Learning Representations, Cited by: §4.2. Jin et al. (2026) H. H. Jin, W. Yang, M. Ghaffari, C. Morato, and B. Mirzasoleiman Reasoning quality emerges early: data curation for reasoning models. In Proceedings of the 43rd International Conference on Machine Learning, Note: arXiv:2606.26797 Cited by: §1, §2, 5th item. Jin et al. (2019) Q. Jin, B. Dhingra, Z. Liu, W. W. Cohen, and X. Lu PubMedQA: a dataset for biomedical research question answering. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, p. 2567–2577. External Links: Document Cited by: §1, §2. Just et al. (2026) H. A. Just, M. Ko, and R. Jia The signal is in the steps: local scoring for reasoning data selection. arXiv preprint arXiv:2510.03988. Note: Version 2 Cited by: §1, §2, 3rd item. Kim et al. (2024) Y. Kim, J. Wu, Y. Abdulle, and H. Wu MedExQA: medical question answering benchmark with multiple explanations. In Proceedings of the 23rd Workshop on Biomedical Natural Language Processing, p. 167–181. External Links: Document Cited by: §1, §2. Nimo et al. (2025) C. Nimo, T. Olatunji, A. T. Owodunni, T. Abdullahi, E. Ayodele, M. Sanni, E. C. Aka, F. Omofoye, F. Yuehgoh, T. Faniran, et al. AfriMed-QA: a pan-african, multi-specialty, medical question-answering benchmark dataset. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 1948–1973. External Links: Document Cited by: §1. Pal et al. (2022) A. Pal, L. K. Umapathi, and M. Sankarasubbu MedMCQA: a large-scale multi-subject multi-choice dataset for medical domain question answering. In Proceedings of the Conference on Health, Inference, and Learning, Proceedings of Machine Learning Research, Vol. 174, p. 248–260. Cited by: §1, §2. Pezeshkpour and Hruschka (2024) P. Pezeshkpour and E. Hruschka Large language models sensitivity to the order of options in multiple-choice questions. In Findings of the Association for Computational Linguistics: NAACL 2024, p. 2006–2017. External Links: Document Cited by: §2. Sellergren et al. (2025) A. Sellergren, S. Kazemzadeh, T. Jaroensri, A. Kiraly, M. Traverse, T. Kohlberger, S. Xu, F. Jamil, C. Hughes, C. Lau, et al. MedGemma technical report. arXiv preprint arXiv:2507.05201. Note: Revised April 2026 Cited by: §4.2. Settles (2009) B. Settles Active learning literature survey. Computer Sciences Technical Report 1648, University of Wisconsin–Madison. Cited by: §1. Turpin et al. (2023) M. Turpin, J. Michael, E. Perez, and S. R. Bowman Language models don’t always say what they think: unfaithful explanations in chain-of-thought prompting. In Advances in Neural Information Processing Systems, Vol. 36, p. 74952–74965. Cited by: §2. Wang et al. (2026) B. Wang, R. Miao, C. Shen, S. Yan, K. Liu, X. Li, X. Yuan, S. Fan, J. Zhang, and J. Ye On the step length confounding in LLM reasoning data selection. In Findings of the Association for Computational Linguistics: ACL 2026, p. 18443–18457. External Links: Document Cited by: §1, §2, 4th item. Wei et al. (2022) J. Wei, X. Wang, D. Schuurmans, M. Bosma, F. Xia, E. Chi, Q. V. Le, and D. Zhou Chain-of-thought prompting elicits reasoning in large language models. In Advances in Neural Information Processing Systems, Vol. 35, p. 24824–24837. Cited by: §2. Yang et al. (2026) Y. Yang, M. Lai, W. Zhao, X. Fan, Z. Xi, M. Wu, C. Huang, J. Zhao, H. Lv, J. Tong, et al. Which reasoning trajectories teach students to reason better? a simple metric of informative alignment. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 42123–42150. External Links: Document Cited by: §1, §2, 2nd item. Zelikman et al. (2022) E. Zelikman, Y. Wu, J. Mu, and N. Goodman STaR: bootstrapping reasoning with reasoning. In Advances in Neural Information Processing Systems, Vol. 35, p. 15476–15488. Cited by: §2. Zheng et al. (2024) C. Zheng, H. Zhou, F. Meng, J. Zhou, and M. Huang Large language models are not robust multiple choice selectors. In International Conference on Learning Representations, Cited by: §2. Appendix A Additional implementation details Rationale steps are segmented deterministically using newline boundaries followed by a sentence heuristic. At most 384 rationale tokens are scored. RSP uses one clean pass and 12 noisy passes per candidate (three layers, four perturbations each). A fixed suffix requests the final answer, and only single-token option letters are scored. The answer-only and rationale branches share the same maximum sequence length (1,024), optimizer family (AdamW), cosine learning-rate schedule, and 3% warmup. Selected rationale branches contain three records per selected item: selected answer replay, rationale target, and one randomly sampled unselected answer replay. Random acquisition seeds are 13, 23, and 37, crossed with downstream seeds 13, 23, and 37. Appendix B Complete macro-F1 results Table 5: Test macro-F1 (%, mean across three downstream seeds; Random averages nine runs). Bold marks the best budgeted method; All rationales is unbudgeted. Method AfriMed MedExp MedEx PubMed MedMCQA Macro Answer-only 61.54 52.36 58.80 51.89 51.80 55.28 Random 3×33× 3 60.86 53.82 60.78 51.19 51.76 55.68 Answer entropy 60.40 54.97 58.96 51.89 51.42 55.53 Answer margin 61.51 54.64 61.50 55.26 51.38 56.86 Rationale length 61.07 48.87 61.65 54.37 53.66 55.92 Complexity-aware FT 59.76 54.74 60.78 51.57 50.87 55.54 RSR 60.34 54.46 59.48 48.52 52.14 54.99 LALP 59.96 55.25 62.05 53.13 50.78 56.24 ASLEC-DROP 59.20 53.08 61.60 48.76 49.85 54.50 TEMP 61.53 48.34 59.62 44.20 52.25 53.19 Signed-RSP 61.71 53.15 61.33 54.57 51.74 56.50 RMS-RSP 62.42 53.84 60.61 50.39 51.28 55.71 All rationales 62.94 59.04 62.41 57.29 57.51 59.84 Appendix C Dataset-level Token-AUBC and robustness Table 6: Accuracy Token-AUBC over 0/256/512/1,024 rationale tokens. Method AfriMed MedExp MedEx PubMed MedMCQA Macro Answer-only .6353 .5333 .5983 .7117 .5193 .5996 Random 3×33× 3 .6283 .5489 .6176 .6958 .5199 .6021 Entropy .6237 .5410 .5996 .6875 .5078 .5919 Margin .6322 .5363 .6198 .6877 .5211 .5994 Rationale length .6328 .5347 .6154 .6992 .5269 .6018 Complexity-aware FT .6168 .5340 .6162 .6821 .5102 .5919 RSR .6290 .5540 .6081 .6981 .5175 .6014 LALP .6195 .5723 .6208 .6987 .5110 .6045 ASLEC-DROP .6187 .5613 .6212 .6933 .5097 .6009 TEMP .6369 .5267 .6038 .6827 .5270 .5954 Signed-RSP .6325 .5467 .6185 .6975 .5171 .6025 RMS-RSP .6304 .5317 .6108 .6944 .5129 .5960 Table 7: Option-order robust accuracy (ReAcc). Method AfriMed MedExp MedEx PubMed MedMCQA Macro Answer-only .4623 .3760 .4167 .4183 .2843 .3915 Random 3×33× 3 .4588 .3893 .4506 .4617 .2747 .4070 Entropy .4439 .4000 .3917 .4250 .2738 .3869 Margin .4383 .4160 .4367 .4533 .2636 .4016 Rationale length .4570 .2987 .4383 .5350 .2984 .4055 Complexity-aware FT .4324 .4027 .4433 .4667 .2410 .3972 RSR .4439 .4000 .4583 .4583 .2764 .4074 LALP .4178 .3920 .4333 .5117 .2646 .4039 ASLEC-DROP .4174 .3600 .4333 .4900 .2457 .3893 TEMP .4638 .2933 .4083 .3500 .2807 .3592 Signed-RSP .4656 .3867 .4667 .4100 .2826 .4023 RMS-RSP .4791 .4000 .4767 .4933 .2812 .4260 All rationales .4832 .4827 .4933 .4450 .3652 .4539 Table 8: Semantic consistency across option orders (ReCon). Method AfriMed MedExp MedEx PubMed MedMCQA Macro Answer-only .6005 .5440 .5067 .4983 .4080 .5115 Random 3×33× 3 .5977 .5742 .5722 .5722 .3878 .5408 Entropy .5658 .6107 .4733 .5167 .3863 .5105 Margin .5613 .6213 .5250 .5667 .3706 .5290 Rationale length .5916 .4293 .5650 .6700 .4247 .5361 Complexity-aware FT .5549 .6080 .5567 .5667 .3360 .5245 RSR .5695 .5707 .6017 .5933 .3891 .5449 LALP .5362 .5733 .5383 .6417 .3713 .5322 ASLEC-DROP .5299 .5120 .5467 .6133 .3472 .5098 TEMP .6024 .4080 .4933 .4300 .3977 .4663 Signed-RSP .6005 .5653 .5950 .4983 .3984 .5315 RMS-RSP .6274 .6027 .6050 .6133 .3984 .5694 All rationales .6271 .6453 .6117 .5450 .5015 .5861 Appendix D Canonical RMS-RSP versus Random Table 9: Paired differences from Random. Values and intervals are percentage points. Dataset Accuracy Δ [95% CI] ReAcc Δ [95% CI] ReCon Δ [95% CI] AfriMed-QA +1.44+1.44 [+0.21,+2.77][+0.21,+2.77] +2.03+2.03 [+0.26,+4.12][+0.26,+4.12] +2.98+2.98 [+0.19,+5.83][+0.19,+5.83] MedExpQA +0.09+0.09 [−3.73,+3.56][-3.73,+3.56] +1.07+1.07 [−2.93,+4.89][-2.93,+4.89] +2.84+2.84 [−2.93,+8.80][-2.93,+8.80] MedExQA 0.000.00 [−3.67,+3.89][-3.67,+3.89] +2.61+2.61 [0.00,+5.17][0.00,+5.17] +3.28+3.28 [−0.11,+6.67][-0.11,+6.67] PubMedQA +1.61+1.61 [−1.67,+4.83][-1.67,+4.83] +3.17+3.17 [−10.00,+15.00][-10.00,+15.00] +4.11+4.11 [−14.50,+21.06][-14.50,+21.06] MedMCQA −0.46-0.46 [−2.39,+2.27][-2.39,+2.27] +0.65+0.65 [−0.94,+2.71][-0.94,+2.71] +1.06+1.06 [−1.27,+3.74][-1.27,+3.74] Appendix E Reproducibility and artifact scope The experiment artifacts retain selected UID order, token cost, acquisition score, predictions, per-run metrics, and command arguments. The present anonymous draft omits identifying repository links. Code and processed split manifests should be released with a non-identifying archive at submission time. Dataset licenses and model terms remain those of the original resources. No new patient data or clinical records were collected.