Paper deep dive
Cross-lingual Calibration of Pre-Generation Success Probes for Multilingual LLM Routing
Andrea Paganelli, Stefano Civelli, Pietro Bernardelle, Gianluca Demartini
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Pre-generation success probes estimate response correctness from a language model's hidden activations before decoding, enabling cost-aware routing. While prior work has demonstrated their utility primarily on English inputs, we study their reliability across languages along three dimensions: (1) whether they preserve the ranking of likely successes and failures (DISCRIMINATION); (2) whether they retain probabilities that match observed success frequencies (CALIBRATION); and (3) whether they produce scores comparable enough across candidate models for cost-aware multilingual routing (UTILITY). Using 3,000 MATH problems in 10 languages and 8 open-weight model configurations, we compare cross-lingual transfer from English-trained probes and equal-budget pooled multilingual probes. English-trained probes retain useful cross-lingual discrimination but become less well calibrated after transfer. Pooled multilingual supervision improves both properties and yields more reliable estimates of success. In routing experiments, the pooled router achieves a 0.7% higher test success rate while reducing modeled cost by 13.0% relative to always selecting the model with the highest average success. These results show that multilingual routing requires success estimates that remain well calibrated and comparable across languages and models.
Tags
Links
- Source: https://arxiv.org/abs/2610.06216v1
- Canonical: https://arxiv.org/abs/2610.06216v1
Trouble viewing inline? Open PDF directly ā
Full Text
79,684 characters extracted from source content.
Expand or collapse full text
Cross-lingual Calibration of Pre-Generation Success Probes for Multilingual LLM Routing Andrea Paganelli 1,2 , Stefano Civelli 1 , Pietro Bernardelle 1 , Gianluca Demartini 1 1 The University of Queensland 2 Polytechnic University of Milan Correspondence: s.civelli@uq.edu.au Abstract Pre-generation success probes estimate re- sponse correctness from a language modelās hidden activations before decoding, enabling cost-aware routing.While prior work has demonstrated their utility primarily on English inputs, we study their reliability across lan- guages along three dimensions: (1) whether they preserve the ranking of likely successes and failures (DISCRIMINATION); (2) whether they retain probabilities that match observed success frequencies (CALIBRATION); and (3) whether they produce scores compara- ble enough across candidate models for cost- aware multilingual routing (UTILITY). Using 3,000 MATH problems in 10 languages and 8 open-weight model configurations, we com- pare cross-lingual transfer from English-trained probes and equal-budget pooled multilingual probes. English-trained probes retain useful cross-lingual discrimination but become less well calibrated after transfer. Pooled multilin- gual supervision improves both properties and yields more reliable estimates of success. In routing experiments, the pooled router achieves a 0.7% higher test success rate while reducing modeled cost by 13.0% relative to always se- lecting the model with the highest average suc- cess. These results show that multilingual rout- ing requires success estimates that remain well calibrated and comparable across languages and models. 1 Introduction Large language models (LLMs) do not perform consistently across languages. A model may solve a mathematical problem in English and fail on a semantically equivalent formulation in another lan- guage (Ahuja et al., 2023; Ko et al., 2025; Shi et al., 2022; Son et al., 2025). At the same time, always using the strongest available model is costly and often unnecessary, since a cheaper model may be equally capable of solving the same problem (Ag- garwal et al., 2024; Chen et al., 2023, 2024; Ong 010203040506070 Cost saving vs. anchor (%) 72 74 76 78 80 82 84 86 Rollout success (%) English English + calibration Pooled Always oss-high Figure 1: Multilingual routing with pre-generation suc- cess probes. Test rollout success versus modeled cost saving relative to always selectinggpt-oss-high, on the balanced ten-language test set. Each point corre- sponds to a different value of the routing trade-off coef- ficientĪ». Curves compare English-trained, recalibrated English-trained, and equal-budget pooled multilingual probes used for routing. et al., 2024). Cost-aware model routing aims to address this trade-off by selecting, for each input, the least expensive model likely to produce a cor- rect response. Such a decision requires estimating, before decoding, how likely each candidate model is to answer correctly. Recent work has shown that a modelās hidden activations contain information about the correct- ness of its future response (Azaria and Mitchell, 2023; Lugoloobi et al., 2026). Linear probes can extract this information after the prompt has been processed but before the first output token is gen- erated. These pre-generation success probes can therefore support routing without first generating and evaluating a candidate response (Di Cicco, 2026; Lugoloobi et al., 2026; Varshney et al., 2026). However, existing evidence is largely limited to En- glish inputs, leaving unclear whether these probes remain reliable across languages and whether their arXiv:2610.06216v1 [cs.CL] 5 Oct 2026 estimates are sufficiently comparable for multilin- gual routing. We therefore ask: (RQ1) How reli- ably do pre-generation success probes transfer from English to non-English target languages? (RQ2) How effectively do the resulting success estimates support cost-aware routing? We investigate these questions using 3,000 prob- lems from the MATH dataset (Hendrycks et al., 2021), translated from English into 9 additional lan- guages, and 8 open-weight model configurations. For each problem, language, and model configura- tion, we collectn = 5sampled generations to esti- mate empirical success and extract residual-stream activations at every model depth before decoding. We train layer-wise ridge-regression probes under monolingual training and equal-budget pooled mul- tilingual training. We then compare English-to- target cross-lingual transfer with pooled multilin- gual supervision, using a probe trained directly on labelled data in the target language as an oracle reference. We assess probesā reliability through three increasingly demanding requirements: (1) probes should preserve the ordering of likely suc- cesses and failures (DISCRIMINATION); (2) their predicted probabilities should remain aligned with observed success frequencies (CALIBRATION); and (3) their scores should be sufficiently comparable across candidate models to support effective cost- aware choices (UTILITY). Our results show that English-trained probes re- tain substantial cross-lingual discrimination, partic- ularly at intermediate layers (Civelli et al., 2026). Their scores, however, do not retain a consistent probabilistic meaning across languages. Pooled multilingual supervision reduces expected calibra- tion error by 38ā69% across models and produces a stronger successācost frontier in routing (see Fig- ure 1). Overall, our findings show that preserving cross-lingual rankings is only one requirement for reliable multilingual routing. Effective routing also requires success probabilities whose scale remains meaningful across languages and sufficiently com- parable across candidate models. 2 Related Work Pre-generation prediction from activations. Prior work has investigated whether a modelās in- ternal state contains information about properties of its eventual response before generation begins. Early studies identify activation patterns associ- ated with truthfulness and correctness (Azaria and Mitchell, 2023). Related work shows that human- annotated problem difficulty is linearly decodable from pre-generation representations (Civelli et al., 2026; Lugoloobi and Russell, 2025). Moving from these general response and problem-level proper- ties to model-specific success, Cencerrado et al. (2025) estimate forthcoming answer correctness from question-only activations by constructing cor- rectness directions from the difference between the mean activations of successful and unsuccessful examples. These directions transfer well across factual knowledge tasks, but their predictive per- formance is weaker on mathematical reasoning. Lugoloobi et al. (2026) extend this line of work by training supervised linear probes directly on la- belled successes and failures. They show that these probes can predict model success on mathematical and coding tasks and, crucially, use the resulting es- timates for cost-aware model routing. Subsequent approaches similarly exploit prefilling-time signals for self-evaluation (Kasnavieh et al., 2026) or use an open-weight modelās activations to predict the correctness of separate target models for routing (Varshney et al., 2026). Together, these studies es- tablish that pre-generation representations encode signals correlated with correctness and that these signals can support routing decisions. Cross-lingual transfer of internal signals.Mul- tilingual models have been shown to map seman- tically equivalent inputs in different languages to partially aligned representation spaces (Bernardelle et al., 2026; Kim and Lee, 2025; Li et al., 2025; Schut et al., 2025; Wendler et al., 2024), en- abling zero-shot cross-lingual transfer (Conneau et al., 2020; LibovickĆ½ et al., 2020; Pires et al., 2019). Nevertheless, multilingual representation differences in tokenisation, pretraining data, and language-specific model competence can produce substantial variation across languages (Rust et al., 2021). Moreover, representational alignment alone does not guarantee that task-relevant information will transfer. In a controlled multilingual setting, Hua et al. (2024) show that strongly aligned repre- sentations can coexist with poor cross-lingual trans- fer, demonstrating that shared geometry is not suf- ficient for transferring a learned predictor. This dis- tinction is particularly important for meta-cognitive signals. Concurrent work finds that problem diffi- culty has a partly shared cross-lingual geometry, es- pecially in intermediate model layers (Civelli et al., 2026). Similarly, Datta et al. (2026) report that activation-based hallucination detectors trained in English deteriorate under zero-shot cross-lingual transfer and benefit from multilingual supervision. Calibration under distributional shifts. Dis- crimination and calibration describe complemen- tary properties of a predictor. A predictor may rank successful examples above unsuccessful ones while assigning scores that do not correspond to empirical outcome frequencies (Guo et al., 2017; Naeini et al., 2015; Platt, 1999; Zadrozny and Elkan, 2002). Prior work has shown that cross- lingual transfer can substantially degrade the cal- ibration of output-level predictions (Jiang et al., 2022). More recently, Zhou et al. (2025) study mul- tilingual confidence signals in internal representa- tions and find that intermediate layers can provide better-calibrated confidence estimates than final- layer representations, particularly outside English. This result connects multilingual calibration to rep- resentational depth, but does not establish whether supervised pre-generation success probes preserve calibration when transferred between languages or whether multilingual supervision improves their downstream routing utility. Cost-aware LLM routing.LLM routing and cas- cading seek to reduce inference cost by assigning each query to the least expensive model expected to answer it correctly (Chen et al., 2023; Dekon- inck et al., 2025; Ding et al., 2024; Jitkrittum et al., 2025; Ong et al., 2024). Most routing methods estimate model suitability from external signals such as query embeddings, preference data, previ- ously observed outputs, or uncertainty measures. Activation-based routers instead use internal rep- resentations computed during prompt prefilling to estimate model-specific success before generation begins (Lugoloobi et al., 2026; Varshney et al., 2026). For routing, however, predictive ranking alone may be insufficient. Model scores must be comparable across candidate models and operat- ing conditions if they are used to set thresholds or optimise expected success under a cost constraint. Existing activation-based routing work has not examined whether success estimates or routing thresholds remain valid under language shift. 3 Methodology 3.1 Formulation Letx i denote a mathematical problem,āthe lan- guage in which it is presented, andman evaluated language model. We writex (ā) i for the correspond- ing language-specific prompt. LetĻ m (r | x (ā) i ) denote the probability that modelm, under the fixed decoding policy, produces responserwhen givenx (ā) i . Accordingly,Ļ m (Ā· | x (ā) i )denotes the full distribution over possible responses. We write R m,ā,i ā¼ Ļ m (Ā· | x (ā) i )to denote one response sampled from this distribution. The correspond- ing single-generation success probability is p m,ā,i = Pr [Correct(R m,ā,i ) = 1].(1) Following Lugoloobi et al. (2026), for every modelāproblemālanguage tuple, we samplen = 5 independent responses and define k m,ā,i = n X r=1 I h Correct R (r) m,ā,i = 1 i as the number of correct responses. The empirical success score is therefore defined as y m,ā,i = k m,ā,i n ā0, 0.2, 0.4, 0.6, 0.8, 1. (2) This score estimates the probability in Equation 1, since E[y m,ā,i ] = p m,ā,i . Sincep m,ā,i is conditional on both the model and the language of the prompt, the same underlying problem can have different success probabilities across models and translations. 3.2 Data We sample 3,000 English-language problems from the MATH dataset (Hendrycks et al., 2021) and translate each into Arabic, Chinese, French, Ital- ian, Russian, Swahili, Telugu, Thai, and Turkish using DeepL (DeepL SE, 2026). To preserve the mathematical content and structure of the source problems, we protect mathematical notation and structured blocks during translation. We apply regular-expression checks to detect malformed no- tation, and use COMET-Kiwi (Rei et al., 2022) as a reference-free estimate of translation adequacy. For further details on translation quality see Ap- pendix A. We partition the 3,000 underlying MATH prob- lems into training, validation, and test sets using a 70/15/15 split. We split by underlying prob- lem rather than by translated prompt, so all ten language-specific versions of a problem remain in the same partition, with the resulting assign- ment shared across all models. The training split is used to fit probe parameters, the validation split is used for regularisation, reporting-layer, recalibra- tion, and routing-policy selection, and the test split is reserved for final evaluation. 3.3 Models and Generation Our experiments include 8 configurations of instruction-tuned (Ouyang et al., 2022), open- weight models:Qwen3at0.6B,1.7B,4B, and8Bpa- rameters (Yang et al., 2025);Qwen3.5at4Band9B parameters (Qwen Team, 2026); andgpt-oss-20b (OpenAI, 2025), evaluated under low- and high- reasoning settings, denotedgpt-oss-lowand gpt-oss-high, respectively. To simulate a realistic usage setting, each model receives only the mathematical problem, formatted using its native chat template, with no additional instructions. For each modelāproblemālanguage tuple, we samplen = 5responses at a tempera- ture of 0.7, extract the final answer, and aggregate the resulting binary correctness labels according to Equation 2. 3.4 Layer-wise Representation Extraction For each modelāproblemālanguage tuple, we per- form a single forward pass using the Hugging Face Transformers library (Wolf et al., 2020) and extract residual-stream activations at every model depth. For a model withLtransformer blocks, we consider L + 1representation depths. Depth0denotes the initial residual-stream state obtained by mapping the input tokens through the modelās embedding matrix, before any transformer block is applied. Depthsj = 1,...,Ldenote the residual-stream states after the corresponding transformer blocks. At each depth, we record the activationh (j) m,ā,i ā R d m at the final prompt token, before the gener- ation of the first output token, whered m is the hidden dimension of model m. 1 3.5 Success-Probes Training and Selection Probes training. Using the layer-wise represen- tations defined above, we train a separate linear ridge-regression probe for each model, layer, and supervision regime. Under (1) monolingual super- vision, the probe is trained on the complete train- ing set in a single language, whereas under (2) pooled multilingual supervision, the same overall 1 We select the final prompt token because it can attend to the entire input, making its activation a representation of the model state from which generation begins. training budget is distributed evenly across the ten languages. Before probe training, each activation dimen- sion is standardised by removing its mean and scaling it to unit variance. Both statistics are es- timated exclusively from the corresponding train- ing split. The probe produces the raw prediction bp (j),raw m,ā,i = w ā¤ e h (j) m,ā,i + b, where e h (j) m,ā,i is the stan- dardised activation. For each candidate ridge coeffi- cientĪ±ā10, 10 2 , 10 3 , 10 4 , 10 5 we fit a separate set of probe parameterswandbon the training split by minimizing X i y m,ā,i ā bp (j),raw m,ā,i 2 + Ī±ā„wā„ 2 2 . Thus,wandbare optimized for a fixed value ofĪ±; the value ofĪ±itself is subsequently se- lected using the validation procedure described in the next paragraph. To constrain the regres- sion output, we project its predictions onto[0, 1]: bp (j) m,ā,i = min n 1, max n 0,bp (j),raw m,ā,i o .This pro- jection is applied when computing validation and test metrics and when using predictions for recali- bration or routing. Validation-based probe selection. We use val- idation Brier score to select the ridge coefficient and reporting layer because the probes are intended to estimate single-generation success probabilities. For an evaluation setD, letbp i denote the projected predicted probability,k i the number of successful rollouts, andn i the total number of retained roll- outs for problemi. We compute the rollout-level Brier score as BS(D) = P iāD k i (1ā bp i ) 2 + (n i ā k i )bp 2 i P iāD n i . (3) In our experiments,n i = 5for every problem. This score is the mean squared error between the predicted probability and the individual binary roll- out outcomes, and therefore evaluates the same single-generation success probability defined in Equation 1. It summarizes overall probabilistic quality and may improve through better discrimi- nation, better calibration, or both. Selection proceeds in two stages. First, as stated in the previous paragraph, we fit one probe for each candidateĪ±ā10, 10 2 , 10 3 , 10 4 , 10 5 on the training split for every model, layer, and supervi- sion regime, and select the coefficient that mini- mizes validation Brier score: Ī± ā m,j,s = arg min Ī± BS val m,j,s (Ī±), wheresdenotes the supervision regime. For a monolingual probe, including the English probe and each same-language reference probe, the can- didate coefficients are evaluated on validation ex- amples in the probeās training language. For the pooled multilingual probe, they are evaluated on the union of the validation sets from all ten lan- guages. The resulting layer-wise ridge-coefficient selections are reported in Appendix B. Second, after selectingĪ± ā m,j,s at every layer, we select one shared reporting layer for each model. Using a different layer for each supervision regime or evaluation metric could selectively favour indi- vidual conditions. Fors ā English, Pooled,we therefore define macroBS val s,m,j = 1 |L target | X āāL target BS val s,m,j,ā , where each score is computed using the corre- sponding validation-selected coefficientĪ± ā m,j,s , and Lā targetcontains the nine non-English target lan- guages. We select j ā m = arg min j 1 2 macroBS val English,m,j + 1 2 macroBS val Pooled,m,j . (4) Same-language probes do not influence the layer choice. Once selected,j ā m is frozen across all eval- uation regimes (Section 3.6) and used for all scalar test metrics and downstream analyses. 3.6 Cross-lingual Probe Reliability To answer RQ1, we evaluate probe quality on the same held-out examples from each of the nine non- English target languages. Excluding English en- sures that all reported comparisons measure perfor- mance under language transfer. We compare three regimes: (a) English trans- fer (ENG.), where a probe trained under monolin- gual supervision on English is applied to the target language; (b) pooled multilingual supervision (POOL), where a probe trained under pooled multi- lingual supervision is evaluated separately on each target language; and (c) same-language reference (ORACLE), where a probe trained under monolin- gual supervision in the target language is evaluated on that language. The oracle therefore uses labelled target-language training data and serves as a refer- ence for the other two conditions. Discrimination. We evaluate discrimination us- ing rollout-weighted AUROC. A problem withk i successful andn i āk i failed generations contributes its scorebp i with weightk i to the positive class and weightn i ā k i to the negative class. AUROC therefore measures the probability that a randomly selected successful rollout receives a higher score than a randomly selected failed rollout. AUROC depends only on the ordering of predic- tions and is invariant to strictly monotonic trans- formations of their values. A probe can therefore achieve high AUROC while being systematically overconfident or underconfident. Calibration. Calibration measures whether pre- dicted probabilities agree with observed success frequencies.We diagnose this using sample- weighted Expected Calibration Error (ECE) with ten equal-width bins and reliability diagrams. ECE summarizes the average absolute difference be- tween predicted and observed success rates across bins, while reliability diagrams show where these differences occur. Calibration does not measure whether a probe meaningfully separates easy and difficult problems. For example, a probe that always predicts the over- all success rate may be perfectly calibrated while having no discriminative ability. Probabilistic quality. We evaluate the overall quality of the predicted success probabilities using the rollout-level Brier score in Equation 3 and ag- gregated binomial negative log-likelihood (NLL). Both are strictly proper scoring rules: in expecta- tion, they are minimized by reporting the true suc- cess probability. They jointly reward predictions that are correctly scaled and informative. Brier score applies a quadratic penalty to probability er- rors and is bounded, whereas NLL applies a loga- rithmic penalty and therefore penalizes confidently incorrect predictions more strongly. Neither metric isolates calibration from discrim- ination: an improvement may result from a better probability scale, a better ordering of examples, or both. We consequently use AUROC to measure discrimination, calibration diagnostics to evaluate probability scale, and Brier score and NLL to sum- marize their combined effect on probability quality (more details can be found in Appendix C). 3.7 Multilingual Routing Utility To answer RQ2, we evaluate routing over a language-balanced mixture of all ten languages. This scope differs deliberately from the nine- language probe-quality evaluation since in a real routing setting prompts may come from any of the supported languages, including English. We use the predicted success probabilities to select a target model before generation. Given a set of candidate modelsM, the router chooses m ā (x) = arg max māM [bp m (x)ā Ī»c m ],(5) wherec m is the estimated inference cost (re- fer to Appendix D) andĪ»controls the successā cost trade-off. We evaluate the fixed gridĪ» ā 0, .01, .02, .05, .10, .20, .35, .50, .75, 1, with each value defining one operating point on the successācost frontier. Our main analysis uses encoderātarget decou- pled routing (ETD) (Varshney et al., 2026), in which a fixed encoder supplies the representation used to predict the success of every candidate tar- get model. We useQwen3-4Bas the shared encoder (refer to Appendix E) and train a separate target- specific ridge head for each candidate model at every encoder layer. We select one ETD report- ing layer using the joint validation-Brier rule in Equation 4, additionally macro-averaged over the 8 target models. Predictions from this frozen layer are used in all subsequent recalibration and routing conditions. 4 Results Before evaluating probe DISCRIMINATION, CAL- IBRATION, and ROUTING UTILITY, we first ex- amine the extent to which model success varies across prompt languages. ForQwen3-4B, the em- pirical success rate ranges from 0.145 in Swahili to 0.716 in English (refer to Table 1), showing that success depends jointly on the mathematical problem, the evaluated model, and the language in which the problem is presented. Similar cross- lingual variation is observed across the other eval- uated models, with complete model-by-language success rates reported in Table 9. A useful multi- lingual probe must therefore recover a model- and language-conditional success signal. We subsequently evaluate whether English- trained probes preserve this ordering across lan- guages, whether their predictions remain proba- bilistically calibrated, and whether they support Language Qwen3-4B English0.716 Chinese0.630 Italian0.621 French0.613 Russian0.611 Arabic0.537 Thai0.526 Turkish0.526 Telugu0.322 Swahili0.145 Mean0.525 Table 1: Empirical success rates ofQwen3-4Bacross languages. Each value is the average empirical success score over the test split, where each problem-level score is computed from five sampled generations. effective cost-aware routing. Probe-quality results are macro-averaged over the nine non-English tar- get languages, whereas routing is evaluated over all ten languages, including English. Throughout this section, we reportQwen3-4Bas the representative model; the corresponding results for all other tested models exhibit consistent trends and are reported in Appendix F. 4.1 Cross-Lingual Discrimination Table 2 shows that the English-trainedQwen3-4B probe retains substantial discrimination when trans- ferred unchanged to the nine non-English lan- guages. It achieves a macro-average rollout AU- ROC of 0.729, only 2.9 percentage points below the 0.758 obtained by probes trained directly in each evaluation language (oracle). Thus, English supervision retains enough success-relevant infor- mation to preserve much of the ordering between more and less likely successes after language shift. Equal-budget pooled multilingual supervision further increases AUROC to 0.741, reducing the gap to the same-language reference from 2.9 to 1.7 percentage points. Because the English and pooled Evaluation regime Rollout AUROCā Rollout Brierā ECEā English0.7290.2130.125 Pooled0.7410.1900.043 Oracle0.7580.1830.031 Table 2: Self-probe quality forQwen3-4Bat the shared reporting layer (L23). Values are macro-averages over the nine non-English test languages. AUROC treats the five rollouts as weighted binary outcomes. Qwen3-4B 0.00.51.0 0.0 0.5 1.0 ECE=0.125 L23 Arabic 0.51.0 ECE=0.187 L23 Chinese 0.51.0 ECE=0.064 L23 English 0.51.0 ECE=0.091 L23 French 0.51.0 ECE=0.122 L23 Italian 0.00.51.0 0.0 0.5 1.0 ECE=0.053 L23 Russian 0.51.0 ECE=0.247 L23 Swahili 0.51.0 ECE=0.071 L23 Telugu 0.51.0 ECE=0.063 L23 Thai 0.51.0 ECE=0.167 L23 Turkish Empirical success rate Predicted success probability (a) English-trained transfer 0.00.51.0 0.0 0.5 1.0 ECE=0.050 L23 Arabic 0.51.0 ECE=0.043 L23 Chinese 0.51.0 ECE=0.060 L23 English 0.51.0 ECE=0.045 L23 French 0.51.0 ECE=0.025 L23 Italian 0.00.51.0 0.0 0.5 1.0 ECE=0.037 L23 Russian 0.51.0 ECE=0.042 L23 Swahili 0.51.0 ECE=0.054 L23 Telugu 0.51.0 ECE=0.054 L23 Thai 0.51.0 ECE=0.041 L23 Turkish Empirical success rate Predicted success probability (b) Pooled multilingual Figure 2: Per-language reliability diagrams forQwen3-4Bat its shared reporting layer. The English-trained probe is transferred unchanged to each target language, whereas the pooled probe uses the same total training budget distributed equally across the ten languages. The diagonal denotes perfect calibration. conditions both use 2,100 training prompts, this improvement results from the linguistic composi- tion of the supervision rather than a larger annota- tion budget. The pattern is consistent across all 8 model configurations (refer to Table 10). Pooled supervision improves AUROC for every configura- tion and raises the cross-model mean from 0.697 to 0.724, thereby reducing the average gap to the same-language reference from 4.9 to 2.2 percent- age points. Cross-lingual ranking information is therefore broadly transferable from English, al- though multilingual supervision recovers a substan- tial portion of the remaining discrimination loss. 4.2 Ranking Transfers More Reliably Than Probability Scales As introduced in Section 3.6, AUROC measures whether a probe orders examples correctly, but not whether its scores match the underlying success probabilities. Table 2 shows that English-trained probes retain useful discrimination after transfer, while their Brier scores and ECE deteriorate sub- stantially relative to the labelled same-language oracles. Equal-budget pooled supervision recov- ers a considerable portion of this loss: it reduces mean Brier score from 0.171 to 0.154 and lowers ECE for all 8 models (refer to Table 10), bringing both metrics closer to the oracle reference. Because Brier score measures overall probabilistic quality, this recovery may reflect improvements in discrim- ination and calibration; ECE more directly iden- tifies improved alignment between predicted and observed probabilities. These aggregate metrics do not show whether the calibration loss is uniform across languages or driven by particular language shifts. The per-language reliability diagrams in Figure 2 therefore examine how the relationship between predicted and observed success changes PredictorAUROCā Brierā NLLā ECEā English raw0.7290.2130.6240.125 English + cal.0.7290.2080.6140.099 Pooled raw0.7410.190 0.573 0.043 Table 3: Prediction quality of theQwen3-4BETD rout- ing head at the shared encoderās reporting layer (L23), macro-averaged over the nine non-English test lan- guages. Bold marks the best value for each metric. for each target language in the Qwen3-4B model. The English-trained probe remains capable of ranking many target-language problems, but the mapping from its scores to observed success rates varies across languages. In particular, the same numerical prediction can correspond to different empirical success frequencies after language shift. Distributing the same supervision budget across languages produces scores that lie closer to the reliability diagonal and are more comparable across the covered language distribution. For reliability diagrams for the other models see Appendix F. 4.3 ETD Success-Prediction Performance Same-model probe performance is not sufficient to characterize routing because the ETD router (Varshney et al., 2026) does not use each target modelās own representations. Instead, to save cost, Qwen3-4Bsupplies a shared representation, and a separate linear head predicts the success of each candidate target model. We therefore evaluate the ETD heads whose predictions are actually passed to the router. The shared ETD layer is layer 23, selected using the joint Englishāpooled validation- Brier criterion and then frozen for calibration, test- ing, and routing. Table 3 shows that English-trained ETD heads retain some useful ranking information, with mean RouterVal. choice Ī» Test success [95% CI] Saving [95% CI] English raw Max. success0.0080.8 [78.8,82.8]33.7 [31.8,35.5] Anchor matchāNot attained on validation English + cal. Max. success0.0085.0 [83.1,86.8]10.8 [9.0,12.6] Anchor matchāNot attained on validation Pooled raw Max. success0.0086.6 [84.8,88.3]13.0 [10.2,15.9] Anchor match 0.1085.5 [83.6,87.2]22.9 [19.8,26.3] Always oss-high Baselineā85.90.0 Table 4: Validation-selected routing operating points. Anchor matching allows at most 0.1 percentage point validation success loss, then minimizes cost; unattained rows mean no fixed-grid point met that constraint. Brackets are paired bootstrap intervals over base prob- lem IDs; choices are frozen before test. rollout AUROC 0.684. Nevertheless, pooled su- pervision improves mean AUROC to 0.734 and reduces mean Brier score from 0.181 to 0.152. The routing advantage of pooled supervision therefore cannot be attributed to calibration alone: pooling improves the overall probabilistic predictions, in- cluding their discrimination. To isolate the contribution of probability scal- ing, we apply a positive-slope variant of Platt scal- ing (Platt, 1999) to each English-trained target- model head using pooled multilingual validation counts (Appendix G). This transformation pre- serves within-model rankings, and hence AUROC, while allowing the probability scale to change. Re- calibration reduces mean Brier score from 0.181 to 0.169, but pooled raw predictions remain better at 0.152. Calibration metrics generally improve af- ter recalibration, although not uniformly for every target model. This ablation therefore shows that correcting score scales helps, but does not repro- duce the full effect of multilingual supervision. 4.4 Pooled Supervision Produces Stronger Routing Trade-offs We finally apply the ETD predictions to the cost- aware policy in Equation 5. We compare the English-trained probe, its monotonically recali- brated variant, the pooled multilingual probe, and the baseline that always selectsgpt-oss-high. Each reported operating point is selected on valida- tion and frozen before test evaluation. Figure 1 shows the test-set successācost frontiers obtained by sweeping the fixedĪ»grid. The figure displays all test outcomes, whereas Table 4 reports two deployment points whoseĪ»values are selected exclusively on validation: maximum success and anchor matching. The selected values are then frozen and evaluated on test. At its validation-selected maximum-success point, the pooled router obtains 86.6% test success while saving 13% of the anchor cost. Its anchor- matching selection saves 22.9% on test and obtains 85.5% success. This test success is slightly below the anchor because the matching constraint is en- forced on validation; it does not guarantee an exact match after transfer to the test set. Neither the English probe nor its recalibrated variant meets the anchor-matching criterion on val- idation. Nevertheless, recalibration increases their validation-selected maximum test success from 80.8% to 85.0%. Correcting the probability scales therefore improves routing substantially, but does not fully recover the pooled routerās performance. Across the routing grid, recalibration closes 26% of the English-to-pooled utility-regret gap (95% CI: 10.4ā38.9%). Calibration thus explains part, but less than half, of the pooled advantage. Because pooling also improves ETD AUROC, its routing benefit comes from both better rankings and more reliable probability estimates. 5 Conclusion This work investigated whether pre-generation suc- cess probes can support cost-aware model rout- ing across languages. Using 3,000 mathemati- cal problems in ten languages and 8 open-weight model configurations, we found that English- trained probes retain useful cross-lingual discrim- ination, meaning they can continue to rank likely successes above failures after language transfer. Their numerical predictions, however, do not pre- serve the same probabilistic meaning across lan- guages. Equal-budget pooled multilingual super- vision improves both discrimination and calibra- tion, producing success estimates that are more reliable across languages and candidate models. These improvements translate into stronger routing decisions. The pooled multilingual router reaches 86.6% test success while saving 13.0% relative to always using the highest-average-success model, and saves 22.9% at its validation-selected anchor- matching operating point. Monotonic recalibration substantially improves English-trained predictions but recovers only about one quarter of the routing- utility gap, indicating that the benefit of multilin- gual supervision cannot be explained by calibration alone. Overall, our results demonstrate that trans- ferable rankings are necessary but insufficient for multilingual routing. Limitations Our study is limited to mathematical reasoning, these findings might not extend to other domains. Nine of our ten languages are machine-translated rather than natively authored, so results reflect transfer across translated prompts. We restrict our analysis to linear probes at a single post-prompt position and do not probe during generation, which may have an impact, especially under extended reasoning traces. We evaluate only open-weight models up to 20B parameters, and larger or closed- weight models may behave differently. Finally, our routing costs are estimated under a simplified model rather than measured from a live deploy- ment. References Pranjal Aggarwal, Aman Madaan, Ankit Anand, Sriv- idya Pranavi Potharaju, Swaroop Mishra, Pei Zhou, Aditya Gupta, Dheeraj Rajagopal, Karthik Kappagan- thu, Yiming Yang, et al. 2024. Automix: Automati- cally mixing language models. Advances in Neural Information Processing Systems, 37:131000ā131034. Kabir Ahuja, Harshita Diddee, Rishav Hada, Milli- cent Ochieng, Krithika Ramesh, Prachi Jain, Ak- shay Nambi, Tanuja Ganu, Sameer Segal, Mohamed Ahmed, Kalika Bali, and Sunayana Sitaram. 2023. MEGA: Multilingual evaluation of generative AI. In Proceedings of the 2023 Conference on Empir- ical Methods in Natural Language Processing, pages 4232ā4267, Singapore. Association for Computa- tional Linguistics. Amos Azaria and Tom Mitchell. 2023. The internal state of an LLM knows when itās lying. In Find- ings of the Association for Computational Linguis- tics: EMNLP 2023, pages 967ā976. Pietro Bernardelle, Samaneh Mohtadi, Stefano Civ- elli, Joel Mackenzie, and Gianluca Demartini. 2026. Llms encode relevance as a layer-wise cross-lingual signal. arXiv preprint arXiv:2607.15555. IvĆ”n Vicente Moreno Cencerrado, Arnau PadrĆ©s Masde- mont, Anton Gonzalvez Hawthorne, David Demitri Africa, and Lorenzo Pacchiardi. 2025.No an- swer needed:Predicting llm answer accuracy from question-only linear probes. arXiv preprint arXiv:2509.10625. Lingjiao Chen, Matei Zaharia, and James Zou. 2023. FrugalGPT: How to use large language models while reducing cost and improving performance. Preprint, arXiv:2305.05176. Shuhao Chen, Weisen Jiang, Baijiong Lin, James Kwok, and Yu Zhang. 2024. Routerdc: Query-based router by dual contrastive learning for assembling large lan- guage models. Advances in Neural Information Pro- cessing Systems, 37:66305ā66328. Stefano Civelli, Pietro Bernardelle, NicolĆ² Brunello, and Gianluca Demartini. 2026. A shared geometry of difficulty in multilingual language models. In Proceedings of the 64th Annual Meeting of the As- sociation for Computational Linguistics (Volume 2: Short Papers), pages 796ā807. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco GuzmĆ”n, Edouard Grave, Myle Ott, Luke Zettle- moyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In Pro- ceedings of the 58th Annual Meeting of the Asso- ciation for Computational Linguistics, pages 8440ā 8451. Debtanu Datta, Mohan Kishore Chilukuri, Yash Ku- mar, Saptarshi Ghosh, and Muhammad Bilal Zafar. 2026. Do LLM hallucination detectors suffer from low-resource effect? Preprint, arXiv:2601.16766. Accepted at EACL 2026. DeepL SE. 2026. DeepL Translator.https://w. deepl.com/translator. Accessed: 2026-07-09. Jasper Dekoninck, Maximilian Baader, and Martin Vechev. 2025. A unified approach to routing and cascading for LLMs. In Proceedings of the 42nd International Conference on Machine Learning. Carlo Di Cicco. 2026. Code correctness signals in llm hidden states: Pre-generation probing and repair ge- ometry. arXiv preprint arXiv:2606.14530. Dujian Ding, Ankur Mallick, Chi Wang, Robert Sim, Subhabrata Mukherjee, Victor RĆ¼hle, Laks Laksh- manan, and Ahmed H Awadallah. 2024. Hybrid llm: Cost-efficient and quality-aware query routing. In In- ternational Conference on Learning Representations, volume 2024, pages 41348ā41366. Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Wein- berger. 2017. On calibration of modern neural net- works. In Proceedings of the 34th International Con- ference on Machine Learning, pages 1321ā1330. Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. 2021. Measuring mathematical problem solving with the MATH dataset. In Proceed- ings of the Neural Information Processing Systems Track on Datasets and Benchmarks. Tianze Hua, Tian Yun, and Ellie Pavlick. 2024. moth- ello: When do cross-lingual representation alignment and cross-lingual transfer emerge in multilingual models? In Findings of the Association for Computa- tional Linguistics: NAACL 2024, pages 1585ā1598. Zhengping Jiang, Anqi Liu, and Benjamin Van Durme. 2022.Calibrating zero-shot cross-lingual (un- )structured predictions. In Proceedings of the 2022 Conference on Empirical Methods in Natural Lan- guage Processing, pages 2648ā2674. WittawatJitkrittum,HarikrishnaNarasimhan, Ankit Singh Rawat, Jeevesh Juneja, Congchao Wang, Zifeng Wang, Alec Go, Chen-Yu Lee, Pradeep Shenoy, Rina Panigrahy, Aditya Krishna Menon, and Sanjiv Kumar. 2025. Universal model routing for efficient LLM inference.Preprint, arXiv:2502.08773. Hossein Hosseini Kasnavieh, Gholamreza Haffari, Christopher Leckie, and Adel N Toosi. 2026. In- trolm: Introspective language models via prefilling- time self-evaluation. In Findings of the Association for Computational Linguistics: ACL 2026, pages 12313ā12326. JaeSeong Kim and Suan Lee. 2025. How language directions align with token geometry in multilingual llms. arXiv preprint arXiv:2511.16693. Hyunwoo Ko, Guijin Son, and Dasol Choi. 2025. Un- derstand, solve and translate: Bridging the multilin- gual mathematical reasoning gap. In Proceedings of the 5th Workshop on Multilingual Representation Learning (MRL 2025), pages 78ā95, Suzhuo, China. Association for Computational Linguistics. Daoyang Li, Haiyan Zhao, Qingcheng Zeng, and Meng- nan Du. 2025. Exploring multilingual probing in large language models: A cross-language analysis. In Proceedings of the 1st Joint Workshop on Large Language Models and Structure Modeling (XLLM 2025), pages 61ā70. Jind Ė rich LibovickĆ½, Rudolf Rosa, and Alexander Fraser. 2020. On the language neutrality of pre-trained mul- tilingual representations. In Findings of the Associ- ation for Computational Linguistics: EMNLP 2020, pages 1663ā1674. William Lugoloobi, Thomas Foster, William Bankes, and Chris Russell. 2026. LLMs encode their failures: Predicting success from pre-generation activations. Preprint, arXiv:2602.09924. William Lugoloobi and Chris Russell. 2025. LLMs encode how difficult problems are.Preprint, arXiv:2510.18147. Mahdi Pakdaman Naeini, Gregory F. Cooper, and Milos Hauskrecht. 2015. Obtaining well calibrated proba- bilities using Bayesian binning. In AAAI Conference on Artificial Intelligence (AAAI). Isaac Ong, Amjad Almahairi, Vincent Wu, Wei-Lin Chi- ang, Tianhao Wu, Joseph E. Gonzalez, M. Waleed Kadous, and Ion Stoica. 2024. RouteLLM: Learn- ing to route LLMs with preference data. Preprint, arXiv:2406.18665. OpenAI. 2025. gpt-oss-120b & gpt-oss-20b model card. Preprint, arXiv:2508.10925. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instruc- tions with human feedback. Advances in neural in- formation processing systems, 35:27730ā27744. Telmo Pires, Eva Schlinger, and Dan Garrette. 2019. How multilingual is multilingual BERT? In Proceed- ings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4996ā5001. John C. Platt. 1999. Probabilistic outputs for support vector machines and comparisons to regularized like- lihood methods. Advances in Large Margin Classi- fiers, 10(3):61ā74. Qwen Team. 2026. Qwen3.5: Towards native multi- modal agents. Ricardo Rei, Marcos Treviso, Nuno M. Guerreiro, Chrysoula Zerva, Ana C. Farinha, Christine Maroti, JosĆ© G. C. de Souza, Taisiya Glushkova, Duarte M. Alves, Alon Lavie, Luisa Coheur, and AndrĆ© F. T. Martins. 2022. CometKiwi: IST-Unbabel 2022 sub- mission for the quality estimation shared task. In Proceedings of the Seventh Conference on Machine Translation. Phillip Rust, Jonas Pfeiffer, Ivan Vuli Ģ c, Sebastian Ruder, and Iryna Gurevych. 2021. How good is your tok- enizer? On the monolingual performance of multi- lingual language models. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Confer- ence on Natural Language Processing (Volume 1: Long Papers), pages 3118ā3135. Lisa Schut, Yarin Gal, and Sebastian Farquhar. 2025. Do multilingual llms think in english? In ICLR 2025 Workshop on Building Trust in Language Models and Applications. Freda Shi, Mirac Suzgun, Markus Freitag, Xuezhi Wang, Suraj Srivats, Soroush Vosoughi, Hyung Won Chung, Yi Tay, Sebastian Ruder, Denny Zhou, Dipanjan Das, and Jason Wei. 2022. Language models are multilingual chain-of-thought reasoners. Preprint, arXiv:2210.03057. Guijin Son, Jiwoo Hong, Hyunwoo Ko, and James Thorne. 2025. Linguistic generalizability of test-time scaling in mathematical reasoning. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 14333ā14368, Vienna, Austria. Association for Computational Linguistics. Tanay Varshney, Annie Surla, Michelle Xu, Go- mathy Venkata Krishnan, Maximilian Jeblick, David Austin, Neal Vaidya, and Davide Onofrio. 2026. Llm router: Rethinking routing with prefill activations. arXiv preprint arXiv:2603.20895. Chris Wendler, Veniamin Veselovsky, Giovanni Monea, and Robert West. 2024. Do llamas work in english? on the latent language of multilingual transformers. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15366ā15394. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier- ric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, and 3 others. 2020. Trans- formers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38ā45, Online. Association for Computational Linguistics. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. 2025. Qwen3 technical report. arXiv preprint arXiv:2505.09388. Bianca Zadrozny and Charles Elkan. 2002. Transform- ing classifier scores into accurate multiclass prob- ability estimates. In ACM SIGKDD International Conference on Knowledge Discovery and Data Min- ing (KDD). Ej Zhou, Caiqi Zhang, Tiancheng Hu, Chengzu Li, Nigel Collier, Ivan Vuli Ģ c, and Anna Korhonen. 2025. Beyond the final layer: Intermediate representations for better multilingual calibration in large language models. In Mechanistic Interpretability Workshop at NeurIPS 2025. A Translation Quality Assessment We compare translation quality using the reference- freeUnbabel/wmt22-cometkiwi-damodel (Rei et al., 2022). COMET-Kiwi assigns a relative qual- ity score based on the semantic adequacy of a trans- lation with respect to its English source. The scores are useful for comparing and ranking the two trans- lation systems, but their absolute values do not correspond to an independently interpretable level of translation quality. We score all 3,000 problems in each target language for both DeepL (DeepL SE, 2026) and Qwen3.5-27B (Qwen Team, 2026). The latter was prompted using the below provided prompt. DeepL obtains a slightly higher mean COMET-Kiwi score than Qwen3.5-27B (see Table 5), and its translation were therefore selected for our experiments. <System Prompt> You are a professional technical translator spe- cializing in mathematics competition problems. You will be given a math problem written in English, from the MATH benchmark. Translate ONLY the natural-language prose into[TARGET LANGUAGE]. Do not translate, rename, reformat, or otherwise modify: ā¢Asymptote code blocks, delimited by [asy] ... [/asy]; ā¢L A T E X math, delimited by$...$,$$...$$, \[...\], or \(...\); ā¢ L A T E Xcommands(e.g., , , , , , and ); ā¢ variable names, numbers, units, and math- ematical symbols that appear inside for- mulas. Preserve the original paragraph structure and line breaks. Do not solve the problem. Do not add commentary, notes, or explanations of any kindātranslate what is given, nothing more. Respond with a single JSON object of the form "translation": "<translated text>" and nothing else. <User Prompt> Translate the following math problem into [TARGET LANGUAGE]. Ā«PROBLEMĀ» [SOURCE PROBLEM] Ā«ENDĀ» Language DeepL Qwen3.5-27B Arabic0.77580.7677 Chinese0.80500.8058 French0.82930.8286 Italian0.82950.8278 Russian0.81390.8121 Swahili0.79130.7834 Telugu0.79340.7826 Thai0.79210.7942 Turkish0.82840.7940 Overall0.80650.7996 Table 5: Mean COMET-Kiwi scores over 3,000 MATH problems per target language and translation system. Higher is better; bold indicates the stronger system within each row. Qwen3-0.6B Qwen3-1.7B Qwen3-4B Qwen3-8B Qwen3.5-4B Qwen3.5-9B gpt-oss-low gpt-oss-high Self-probes: EnglishSelf-probes: pooled 0.00.20.40.60.81.0 Relative depth j/L Qwen3-0.6B Qwen3-1.7B Qwen3-4B Qwen3-8B Qwen3.5-4B Qwen3.5-9B gpt-oss-low gpt-oss-high ETD heads: English 0.00.20.40.60.81.0 Relative depth j/L ETD heads: pooled 10 10 2 10 3 10 4 10 5 Selected Ī± Figure 3: Ridge coefficient selected at every relative rep- resentation depth. The top row shows the same-model probe and the bottom row shows ETD target-specific heads; columns separate English and pooled supervision. In each cell, the selectedĪ±minimizes rollout-level vali- dation Brier score over10, 10 2 , 10 3 , 10 4 , 10 5 . Each row denotes the model supplying the representation for the same-model probe and the predicted target model for ETD; each cell denotes one layer. ETD heads use the shared Qwen3-4B representation. B Layer-wise Ridge Regularization Selection For every model, representation layer, and super- vision regime, we fit one ridge probe for each Ī± ā 10, 10 2 , 10 3 , 10 4 , 10 5 . Predictions are pro- jected onto[0, 1], and the coefficient with the low- est rollout-level Brier score on the corresponding validation set is selected. English probes use En- glish validation examples, whereas pooled probes use the combined, equally sized validation sets from all ten languages. ETD heads apply the same rule separately to each target model at every layer of the shared Qwen3-4B encoder. Exact validation- Brier ties are resolved in favour of the smaller value ofĪ±. Figure 3 focuses on the English and pooled regimes used for shared-layer selection and rout- ing; the same-language probes use the identical selection rule. C Why Discrimination, Calibration, and Probabilistic Quality Differ Consider a balanced dataset containing two equally large groups of problems. Generations for the āeasyā group succeed with probability0.75, while generations for the āhardā group succeed with prob- ability0.25. The overall success rate is therefore 0.5. Table 6 compares three hypothetical probes. The overconfident and well-specified probes have identical AUROC because they produce the Probebp easy bp hard AUROC ECE BrierNLL Constant0.50 0.500.50 0.00 0.250 0.693 Overconfident 1.00 0.000.75 0.25 0.250 ā Well specified 0.75 0.250.75 0.00 0.188 0.562 Table 6: Expected metrics for three hypothetical probes. The dataset contains equally many easy and hard prob- lems with true success probabilities0.75and0.25, re- spectively. The infinite NLL for the overconfident probe is the theoretical value before numerical clipping. same ordering: both assign larger scores to easy problems than to hard problems. AUROC cannot reveal that the first probeās probabilities are too extreme. ECE distinguishes them by comparing their numerical predictions with the observed group frequencies. Conversely, the constant and well-specified probes both have zero ECE. The constant probe predicts the correct overall prevalence and is there- fore calibrated in aggregate, but it cannot distin- guish easy from hard problems and has AUROC 0.5. Thus, calibration alone does not establish that a predictor is informative. Finally, the constant and overconfident probes have the same expected Brier score despite fail- ing for different reasons. The constant probe is calibrated but uninformative, whereas the overcon- fident probe is discriminative but miscalibrated. AUROC and ECE reveal this distinction. NLL additionally exposes the severity of the overconfi- dent errors: assigning probability zero to an event that occurs produces an infinite logarithmic penalty. In practice, we clip probabilities only for numer- ical evaluation, resulting in a finite but still large penalty. D Routing Cost Estimation Because the evaluated open-weight models do not have directly comparable first-party API prices, we construct a deployment-aware proxy for their infer- ence costs. The proxy prices reflect each modelās memory footprint under the precision used for in- ference, rather than its nominal parameter count alone. LetM m denote the deployed memory footprint of modelm, in GB. We assign the output-token price according to P out m = Ī±M m , Ī± = 0.10, whereP out m is expressed in US dollars per million output tokens. This proxy is intended to preserve ModelInput price Output price Avg. input Avg. output Exp. cost Relative ($/M tok.)($/M tok.)tokenstokens($/k gen.) Exp. cost Qwen3-0.6B0.0100.060205.12593.450.03770.0111 Qwen3-1.7B0.0280.170205.122361.620.40730.1196 Qwen3-4B0.0670.400205.122056.090.83610.2456 gpt-oss-low0.2041.224226.57946.031.20420.3537 Qwen3.5-4B0.0670.400186.493042.491.22940.3611 Qwen3-8B0.1330.800205.121545.651.26390.3712 Qwen3.5-9B0.1500.900184.492538.442.31230.6791 gpt-oss-high0.2041.224226.572743.933.40481.0000 Table 7: Proxy prices used in the routing simulation. Average input lengths are computed over the balanced pooled training sample used for routing-cost estimation (210 prompts per language and 2,100 prompts per model), using each modelās native tokenizer and chat template. Average output lengths are computed over the same sample. Relative expected costs are normalized by the average standalone input-plus-output cost of gpt-oss-high. relative deployment costs across models rather than reproduce the prices of a particular inference provider. The deployed memory footprint is par- ticularly relevant forgpt-oss-20b, whose weights are natively distributed in a quantized format. 2 Its price is therefore based on its quantized inference footprint rather than its nominal parameter count, resulting in a lower estimated cost than would be assigned to an unquantized 20-billion-parameter model. For every model, the input-token price is set to one sixth of its output-token price: P in m = 1 6 P out m . This ratio is similar to the input-to-output pricing ratio used by several hosted closed-source models. 3 Thegpt-oss-lowandgpt-oss-highconfigu- rations are assigned the same input and output prices because they use the same model weights and have the same memory footprint. Their only difference is the configured reasoning effort. The resulting difference in computational cost is there- fore captured through their observed input and out- put token counts: the high-reasoning configuration processes and generates more tokens on average, despite having the same per-token price. For each model, letP in m andP out m denote the in- put and output prices reported in Table 7. LetT in m,i andT out m,i denote the corresponding input and out- put token counts when modelmprocesses example i. The estimated generation cost is c m,i = P in m T in m,i + P out m T out m,i 10 6 . 2 https://openai.com/index/introducing-gpt-oss/ 3 https://developers.openai.com/api/docs/pricing Thus, the cost difference betweengpt-oss-low andgpt-oss-higharises from their different ob- served token counts rather than from different per- token prices. For ETD routing, each query is first processed by the sharedQwen3-4Bencoder before a target model is selected. The total cost of routing exampleito modelmtherefore additionally includes the input- prefilling cost of this encoder: c ETD m,i = ( c m,i ,m = Qwen3-4B, c encoder i + c m,i , mĢø= Qwen3-4B. where c encoder i = P in Qwen3-4B T in Qwen3-4B,i 10 6 . Because the shared encoder is used only to extract the internal representation required by the routing policy, it does not generate output tokens and there- fore incurs no output-token cost. When the selected target isQwen3-4B, its encoder prefill is reused for generation rather than charged a second time. Expected costs are obtained by averaging the per-example costs after applying the routing policy. For a policyĻthat selects modelĻ(i)for example i, the expected ETD routing cost over an evaluation set containing N examples is C ETD (Ļ) = 1 N N X i=1 c ETD Ļ(i),i . Relative costs are reported with respect to always selecting gpt-oss-high: C rel (Ļ) = C ETD (Ļ) 1 N P N i=1 c gpt-oss-high,i . The reference-system denominator excludes the shared encoder cost because the always- gpt-oss-highpolicy invokes the target model directly and does not perform an ETD routing step. E Shared Encoder Selection Search procedure. We selected the shared en- coder using a validation-only diagnostic conducted before the final probe-selection pipeline was fixed. Starting from the same 8 model configurations con- sidered as routing targets, we treat each configu- ration in turn as the ETD encoder. For every en- coder, we train pooled multilingual target-specific ridge heads at each available representation layer. We then evaluate the resulting router on the com- mon validation split, using the same candidate set, cost model, routing policies,Ī»values, and anchor- margin grid for every encoder. This diagnostic differs from the final probe- training protocol in one respect that matters for reproducibility.Its ridge grid isĪ± ā 10, 10 2 , 10 3 , 10 4 , 10 5 , and it selectsĪ±separately for every encoder layer and target head by maxi- mum validation AUROC, breaking ties by valida- tion RMSE and then smallerĪ±. The main pipeline, by contrast, selects ridge coefficients by validation Brier score as described in Section 3.5. We there- fore use the all-encoder scan only to motivate the encoder architecture; all reportedQwen3-4Bprobe, calibration, and routing results are retrained under the final protocol. Because the architectures have different num- bers of layers, comparing all encoders at one ab- solute layer would be arbitrary. We therefore first select, separately for each encoder, the nonzero layer whose pooled validation routing frontier at- tains the highest success; layer 0 is evaluated but excluded from this maximum-success selection. Ta- ble 8 reports that layer and the maximum validation success attained on its frontier. This criterion is used only for the encoder comparison. After fixing Qwen3-4B, the main pipeline selects the reporting layer using the joint validation-Brier rule in Equa- tion 4; that separate rule also selects layer 23. The encoder and reporting layer are frozen before any test evaluation. Interpretation.Qwen3-4Bobtains the highest ob- served maximum validation success, 88.3%, with a nominal prefill cost equal to 0.40% of the an- chor cost. The comparison is close:Qwen3.5-4B obtains 88.2% at a slightly lower prefill cost of Shared encoder Layer Max. validation success Prefill / anchor Qwen3-0.6B2188.1%0.06% Qwen3-1.7B1987.9%0.17% Qwen3-4B2388.3% 0.40% Qwen3-8B2688.1%0.80% Qwen3.5-4B1088.2%0.37% Qwen3.5-9B1388.2%0.81% gpt-oss-high688.1%1.36% gpt-oss-low687.9%1.36% Table 8: Validation-only shared-encoder comparison un- der pooled ETD supervision. For each encoder, the table reports its validation-selected layer and the maximum success attained over the fixed validation routing grid. No test outcomes enter the comparison. Prefill cost is the encoderās standalone input-only cost as a percentage of the standalonegpt-oss-highanchor cost, using the prices and training-mean input lengths in Appendix D. 0.37%, whileQwen3.5-9Balso obtains 88.2% but has roughly twice the prefill cost ofQwen3-4B. Smaller Qwen3 encoders are cheaper, but have lower maximum validation success in this exper- iment. We therefore chooseQwen3-4Bas the ob- served validation leader with a moderate prefill requirement. Since the differences among the lead- ing encoders are small and the comparison has no uncertainty intervals, it should not be interpreted as showing thatQwen3-4Bis intrinsically better than the alternatives. The prefill column provides a common cost ref- erence rather than the realized overhead of every route. When the router selectsQwen3-4Bitself as the target model, its encoder prefill is reused for generation and is not charged a second time. F Additional Results F.1 Empirical Success Rates Table 9 reports the empirical success rates of the evaluated models across all ten languages. Each value is obtained by averaging the empirical suc- cess scores over the test split. Since each problem- level score is estimated from five sampled genera- tions, these values estimate the probability that a model solves a problem under the decoding proce- dure used in this work, rather than the outcome of a single deterministic generation. Success varies substantially across both mod- els and languages. English has the highest av- erage success rate across models, followed by Chinese and French, whereas Swahili and Telugu have the lowest averages. Increasing model ca- pacity or reasoning effort generally improves suc- cess: performance rises across theQwen3model Language Qwen3-0.6B Qwen3-1.7B Qwen3-4B Qwen3-8B Qwen3.5-4B Qwen3.5-9B gpt-oss-low gpt-oss-high Mean English0.2790.5080.7160.7280.9160.9350.8950.9250.738 Chinese0.2200.4080.6300.6380.8510.8980.7540.8860.661 French0.1790.3880.6130.6630.8480.8950.7890.9110.661 Russian0.1700.3840.6110.6610.8370.8890.7610.8840.650 Italian0.1600.3780.6210.6540.8350.8800.7550.8970.648 Arabic0.1340.3120.5370.6000.7850.8410.7490.885 0.605 Turkish0.1070.2820.5260.5740.7600.8210.7540.8820.588 Thai0.1380.3240.5260.5780.5900.7150.6950.8260.549 Telugu0.0560.1530.3220.4350.4970.5800.7160.8490.451 Swahili0.0340.1240.1450.4180.5360.6830.5870.6470.397 Mean0.1480.3260.5250.5950.7460.8140.7460.8590.595 Table 9: Empirical success rates of the evaluated models across languages. Each value is the average empirical success score over the test split, where each problem-level score is computed from five sampled generations. Languages are ordered by decreasing average success rate across models. The final row reports the average success rate for each model across languages, and the final column reports the average success rate for each language across models. Rollout AUROCāRollout BrierāECEā ModelLayer Eng. (ā p ) Pool (ā p ) Oracle Eng. (ā rel ) Pool (ā rel ) Oracle Eng. (ā rel ) Pool (ā rel ) Oracle Qwen3-0.6B140.710 (ā5.9) 0.718 (ā5.1)0.7690.120 (+26%) 0.099 (+4%)0.0950.116 (+274%) 0.038 (+23%)0.031 Qwen3-1.7B200.741 (ā3.7) 0.766 (ā1.2)0.7780.190 (+21%) 0.161 (+3%)0.1570.129 (+249%) 0.042 (+14%)0.037 Qwen3-4B230.729 (ā2.9) 0.741 (ā1.7)0.7580.213 (+16%) 0.190 (+4%)0.1830.125 (+303%) 0.043 (+39%)0.031 Qwen3-8B210.739 (ā1.6) 0.743 (ā1.2)0.7550.212 (+10%) 0.196 (+2%)0.1920.118 (+237%)0.037 (+6%)0.035 Qwen3.5-4B190.684 (ā7.2) 0.737 (ā1.9)0.7560.178 (+19%) 0.155 (+3%)0.1500.083 (+186%) 0.042 (+45%)0.029 Qwen3.5-9B140.690 (ā4.9) 0.723 (ā1.6)0.7390.148 (+15%) 0.133 (+3%)0.1290.060 (+107%) 0.036 (+24%)0.029 gpt-oss-low170.681 (ā4.0) 0.695 (ā2.6)0.7210.183 (+6%)0.178 (+3%)0.1720.053 (+83%)0.033 (+14%)0.029 gpt-oss-high70.603 (ā8.7) 0.669 (ā2.1)0.6900.125 (+12%) 0.116 (+4%)0.1120.078 (+136%)0.030 (ā9%)0.033 Meanā0.697 (ā4.9) 0.724 (ā2.2)0.7460.171 (+15%) 0.154 (+3%)0.1490.095 (+197%) 0.038 (+19%)0.032 Table 10: Self-probe quality at the shared reporting layer. Values are macro-averages over the nine non-English test languages. Eng. uses 2,100 English prompts, Pool uses 210 prompts per language (2,100 total), and Oracle uses 2,100 prompts in each target language. Parentheses report deltas relative to Oracle. For AUROC,ā p = 100(Xā Oracle) is the absolute difference in percentage points. For Brier and ECE,ā rel = 100(Xā Oracle)/Oracleis the relative percentage difference. Red indicates worse performance than Oracle and blue indicates better performance. AUROC treats the five rollouts as weighted binary outcomes. sizes, fromQwen3.5-4BtoQwen3.5-9B, and from gpt-oss-lowtogpt-oss-high. Nevertheless, the language gaps remain visible even for the strongest configurations. For example,gpt-oss-highde- creases from 0.925 in English to 0.647 in Swahili. These results reinforce that success is conditional not only on the underlying problem and target model, but also on the language in which the prob- lem is presented. F.2 Same-model Probe Discrimination and Calibration The complete same-model probe results (Table 10) show that the main findings are consistent across model configurations. English-trained probes re- tain useful discrimination after transfer to the nine non-English languages, obtaining a mean roll- out AUROC of 0.697. Thus, the representations learned from English supervision preserve a sub- stantial amount of information about which target- language generations are likely to succeed. Equal-budget pooled multilingual supervision improves mean AUROC to 0.724, bringing discrim- ination closer to the labelled same-language refer- ence. The improvement indicates that the pooled probe does not merely adjust the numerical scale of the predictions: multilingual supervision also im- proves the ordering of successful and unsuccessful generations. The difference between supervision regimes is clearer for probabilistic quality. English transfer ob- tains a mean Brier score of 0.171, whereas pooled supervision reduces it to 0.154. Pooled supervision also lowers ECE for all 8 models, moving the pre- dictions toward the same-language reference. En- glish supervision therefore transfers more reliably as a ranking function than as a probability estima- tor. A score learned in English may still identify relatively easy and difficult target-language prob- lems, while no longer corresponding to the same TargetPredictorAUROCā Brierā NLLā ECEā Qwen3-0.6B English raw0.7690.1090.4650.112 English + cal.0.7690.106 0.3850.093 Pooled raw0.773 0.0880.400 0.035 Qwen3-1.7B English raw0.7590.1760.5470.103 English + cal.0.7590.1720.5390.085 Pooled raw0.777 0.158 0.520 0.051 Qwen3-4B English raw0.7290.2130.6240.125 English + cal.0.7290.2080.6140.099 Pooled raw0.741 0.190 0.573 0.043 Qwen3-8B English raw0.7230.2260.6480.138 English + cal.0.7230.2080.6090.074 Pooled raw0.733 0.200 0.606 0.038 Qwen3.5-4B English raw0.6920.1930.5710.133 English + cal.0.6920.1900.5660.141 Pooled raw0.727 0.158 0.492 0.040 Qwen3.5-9B English raw0.6680.1610.5030.098 English + cal.0.6680.1460.4550.061 Pooled raw0.715 0.134 0.426 0.032 gpt-oss-low English raw0.5840.2322.1100.189 English + cal.0.5840.1960.5810.061 Pooled raw0.712 0.174 0.522 0.032 gpt-oss-high English raw0.5450.1401.3500.123 English + cal.0.5450.126 0.4190.057 Pooled raw0.693 0.1140.427 0.048 Shared encoder for all rows: Qwen3-4B, layer 23. A separate ridge head predicts each target modelās success. Table 11: Prediction quality of the ETD routing heads, macro-averaged over the nine non-English test lan- guages. English calibration is fit on multilingual vali- dation data with a positive slope. Bold marks the best value per metric within each target modelās three predic- tor settings. empirical success frequency after language shift. The same-language probes generally remain the strongest reference because they are trained with the full labelled training set from the evaluation lan- guage. Importantly, however, pooled supervision uses the same total number of training examples as the English-only condition. Its advantage there- fore results from the linguistic composition of the supervision rather than from a larger annotation budget. F.3 EncoderāTarget Decoupled Prediction Quality The ETD results evaluate the predictions that are di- rectly consumed by the router. Unlike same-model probes, these heads predict the success of every tar- get model using a shared representation extracted fromQwen3-4B. This is a more demanding setting because the encoder representation must contain information not only about its own expected suc- cess, but also about the behaviour of several distinct target models. English-trained ETD heads retain some discrim- inative information, with mean rollout AUROC 0.684. Pooled multilingual supervision improves this value to 0.734 and reduces mean Brier score from 0.181 to 0.152. The improvement in AUROC shows that the pooled routing advantage cannot be explained solely by better calibration: pooled supervision also produces more informative target- model rankings. Monotonic recalibration reduces the English headsā mean Brier score to 0.169. Because this transformation is strictly increasing within each target-model head, it preserves the corresponding AUROC while correcting systematic shifts in prob- ability scale. Its improvement therefore isolates the contribution of calibration. Nevertheless, the raw pooled heads retain the lowest mean Brier score, and recalibration does not improve every calibra- tion diagnostic for every target model. Multilin- gual supervision therefore contributes information that cannot be recovered by applying a single post- hoc monotonic correction to English-trained pre- dictions. This distinction is important for routing. The policy compares predicted success values across candidate models rather than ranking examples for one model in isolation. Consequently, even an English-trained head with useful within-model AU- ROC can produce poor decisions when its score scale is not comparable with those of the other target-model heads. F.4 Reliability Diagrams The main text presents the reliability diagrams for Qwen3-4B. The figures below report the correspond- ing per-language diagrams for the remaining seven model configurations. Each pair compares the unchanged English-trained probe with the equal- budget pooled multilingual probe at the modelās frozen shared reporting layer. Across models, the English-trained diagrams show that calibration errors are language depen- dent: predicted values that are appropriate for one language can correspond to different observed suc- cess frequencies in another. This explains why English transfer can preserve useful AUROC while obtaining worse ECE and Brier scores. Pooled supervision generally moves the curves closer to the diagonal and reduces their variation across lan- guages, although it does not eliminate all model- and language-specific calibration error. The diagrams also show why a single macro- averaged calibration statistic is insufficient on its own. Similar aggregate ECE values can arise from different patterns, including systematic overpre- diction, systematic underprediction, or opposing errors across languages that partially cancel when averaged. We therefore interpret these figures to- gether with AUROC, Brier score, NLL, and the calibration coefficients reported in the complete tables. G Monotonic Recalibration To isolate the effect of probability scaling, we recal- ibrate the predictions of each English-trained ETD target-model head using the pooled multilingual validation set. We fit a positive-slope Platt-scaling transformation (Platt, 1999): bp cal = Ļ (a m + softplus(b m ) logit (bp raw )), wherea m andb m are fitted separately for each tar- get modelm. Predictions are clipped to[10 ā7 , 1ā 10 ā7 ] before applying the logit. The calibration parameters are estimated by min- imizing binomial negative log-likelihood on the nine non-English validation languages, using the observed numbers of successful and unsuccessful rollouts. They are then frozen and applied un- changed to the test predictions and routing sim- ulation. Becausesoftplus(b m ) > 0, the transformation is strictly increasing. It therefore preserves the ordering of predictions within each target-model head and cannot change its AUROC, apart from numerical ties. However, since a different transfor- mation is fitted for each target model, recalibration can change the relative probability scales across models and therefore affect routing decisions. Qwen3-0.6B 0.00.51.0 0.0 0.5 1.0 ECE=0.072 L14 Arabic 0.51.0 ECE=0.329 L14 Chinese 0.51.0 ECE=0.040 L14 English 0.51.0 ECE=0.116 L14 French 0.51.0 ECE=0.065 L14 Italian 0.00.51.0 0.0 0.5 1.0 ECE=0.084 L14 Russian 0.51.0 ECE=0.033 L14 Swahili 0.51.0 ECE=0.171 L14 Telugu 0.51.0 ECE=0.094 L14 Thai 0.51.0 ECE=0.079 L14 Turkish Empirical success rate Predicted success probability (a) English-trained transfer 0.00.51.0 0.0 0.5 1.0 ECE=0.054 L14 Arabic 0.51.0 ECE=0.041 L14 Chinese 0.51.0 ECE=0.045 L14 English 0.51.0 ECE=0.051 L14 French 0.51.0 ECE=0.055 L14 Italian 0.00.51.0 0.0 0.5 1.0 ECE=0.032 L14 Russian 0.51.0 ECE=0.013 L14 Swahili 0.51.0 ECE=0.027 L14 Telugu 0.51.0 ECE=0.036 L14 Thai 0.51.0 ECE=0.034 L14 Turkish Empirical success rate Predicted success probability (b) Pooled multilingual Figure 4: Per-language reliability diagrams forQwen3-0.6Bat its shared reporting layer. The English-trained probe is transferred unchanged to each target language, whereas the pooled probe uses the same total training budget distributed equally across the ten languages. The diagonal denotes perfect calibration. Qwen3-1.7B 0.00.51.0 0.0 0.5 1.0 ECE=0.105 L20 Arabic 0.51.0 ECE=0.156 L20 Chinese 0.51.0 ECE=0.036 L20 English 0.51.0 ECE=0.081 L20 French 0.51.0 ECE=0.064 L20 Italian 0.00.51.0 0.0 0.5 1.0 ECE=0.145 L20 Russian 0.51.0 ECE=0.254 L20 Swahili 0.51.0 ECE=0.242 L20 Telugu 0.51.0 ECE=0.066 L20 Thai 0.51.0 ECE=0.049 L20 Turkish Empirical success rate Predicted success probability (a) English-trained transfer 0.00.51.0 0.0 0.5 1.0 ECE=0.028 L20 Arabic 0.51.0 ECE=0.052 L20 Chinese 0.51.0 ECE=0.051 L20 English 0.51.0 ECE=0.042 L20 French 0.51.0 ECE=0.053 L20 Italian 0.00.51.0 0.0 0.5 1.0 ECE=0.048 L20 Russian 0.51.0 ECE=0.030 L20 Swahili 0.51.0 ECE=0.037 L20 Telugu 0.51.0 ECE=0.055 L20 Thai 0.51.0 ECE=0.037 L20 Turkish Empirical success rate Predicted success probability (b) Pooled multilingual Figure 5: Per-language reliability diagrams forQwen3-1.7Bat its shared reporting layer. The English-trained probe is transferred unchanged to each target language, whereas the pooled probe uses the same total training budget distributed equally across the ten languages. The diagonal denotes perfect calibration. Qwen3-8B 0.00.51.0 0.0 0.5 1.0 ECE=0.112 L21 Arabic 0.51.0 ECE=0.085 L21 Chinese 0.51.0 ECE=0.045 L21 English 0.51.0 ECE=0.127 L21 French 0.51.0 ECE=0.157 L21 Italian 0.00.51.0 0.0 0.5 1.0 ECE=0.216 L21 Russian 0.51.0 ECE=0.057 L21 Swahili 0.51.0 ECE=0.097 L21 Telugu 0.51.0 ECE=0.147 L21 Thai 0.51.0 ECE=0.068 L21 Turkish Empirical success rate Predicted success probability (a) English-trained transfer 0.00.51.0 0.0 0.5 1.0 ECE=0.031 L21 Arabic 0.51.0 ECE=0.043 L21 Chinese 0.51.0 ECE=0.046 L21 English 0.51.0 ECE=0.026 L21 French 0.51.0 ECE=0.028 L21 Italian 0.00.51.0 0.0 0.5 1.0 ECE=0.032 L21 Russian 0.51.0 ECE=0.042 L21 Swahili 0.51.0 ECE=0.064 L21 Telugu 0.51.0 ECE=0.035 L21 Thai 0.51.0 ECE=0.029 L21 Turkish Empirical success rate Predicted success probability (b) Pooled multilingual Figure 6: Per-language reliability diagrams forQwen3-8Bat its shared reporting layer. The English-trained probe is transferred unchanged to each target language, whereas the pooled probe uses the same total training budget distributed equally across the ten languages. The diagonal denotes perfect calibration. Qwen3.5-4B 0.00.51.0 0.0 0.5 1.0 ECE=0.084 L19 Arabic 0.51.0 ECE=0.026 L19 Chinese 0.51.0 ECE=0.023 L19 English 0.51.0 ECE=0.054 L19 French 0.51.0 ECE=0.080 L19 Italian 0.00.51.0 0.0 0.5 1.0 ECE=0.074 L19 Russian 0.51.0 ECE=0.134 L19 Swahili 0.51.0 ECE=0.120 L19 Telugu 0.51.0 ECE=0.124 L19 Thai 0.51.0 ECE=0.048 L19 Turkish Empirical success rate Predicted success probability (a) English-trained transfer 0.00.51.0 0.0 0.5 1.0 ECE=0.041 L19 Arabic 0.51.0 ECE=0.021 L19 Chinese 0.51.0 ECE=0.050 L19 English 0.51.0 ECE=0.025 L19 French 0.51.0 ECE=0.019 L19 Italian 0.00.51.0 0.0 0.5 1.0 ECE=0.020 L19 Russian 0.51.0 ECE=0.073 L19 Swahili 0.51.0 ECE=0.097 L19 Telugu 0.51.0 ECE=0.045 L19 Thai 0.51.0 ECE=0.034 L19 Turkish Empirical success rate Predicted success probability (b) Pooled multilingual Figure 7: Per-language reliability diagrams forQwen3.5-4Bat its shared reporting layer. The English-trained probe is transferred unchanged to each target language, whereas the pooled probe uses the same total training budget distributed equally across the ten languages. The diagonal denotes perfect calibration. Qwen3.5-9B 0.00.51.0 0.0 0.5 1.0 ECE=0.055 L14 Arabic 0.51.0 ECE=0.037 L14 Chinese 0.51.0 ECE=0.020 L14 English 0.51.0 ECE=0.063 L14 French 0.51.0 ECE=0.058 L14 Italian 0.00.51.0 0.0 0.5 1.0 ECE=0.042 L14 Russian 0.51.0 ECE=0.072 L14 Swahili 0.51.0 ECE=0.134 L14 Telugu 0.51.0 ECE=0.044 L14 Thai 0.51.0 ECE=0.030 L14 Turkish Empirical success rate Predicted success probability (a) English-trained transfer 0.00.51.0 0.0 0.5 1.0 ECE=0.012 L14 Arabic 0.51.0 ECE=0.018 L14 Chinese 0.51.0 ECE=0.022 L14 English 0.51.0 ECE=0.022 L14 French 0.51.0 ECE=0.020 L14 Italian 0.00.51.0 0.0 0.5 1.0 ECE=0.019 L14 Russian 0.51.0 ECE=0.073 L14 Swahili 0.51.0 ECE=0.086 L14 Telugu 0.51.0 ECE=0.048 L14 Thai 0.51.0 ECE=0.032 L14 Turkish Empirical success rate Predicted success probability (b) Pooled multilingual Figure 8: Per-language reliability diagrams forQwen3.5-9Bat its shared reporting layer. The English-trained probe is transferred unchanged to each target language, whereas the pooled probe uses the same total training budget distributed equally across the ten languages. The diagonal denotes perfect calibration. gpt-oss-20b-low 0.00.51.0 0.0 0.5 1.0 ECE=0.021 L17 Arabic 0.51.0 ECE=0.062 L17 Chinese 0.51.0 ECE=0.043 L17 English 0.51.0 ECE=0.027 L17 French 0.51.0 ECE=0.043 L17 Italian 0.00.51.0 0.0 0.5 1.0 ECE=0.048 L17 Russian 0.51.0 ECE=0.058 L17 Swahili 0.51.0 ECE=0.086 L17 Telugu 0.51.0 ECE=0.041 L17 Thai 0.51.0 ECE=0.094 L17 Turkish Empirical success rate Predicted success probability (a) English-trained transfer 0.00.51.0 0.0 0.5 1.0 ECE=0.032 L17 Arabic 0.51.0 ECE=0.026 L17 Chinese 0.51.0 ECE=0.032 L17 English 0.51.0 ECE=0.026 L17 French 0.51.0 ECE=0.023 L17 Italian 0.00.51.0 0.0 0.5 1.0 ECE=0.034 L17 Russian 0.51.0 ECE=0.041 L17 Swahili 0.51.0 ECE=0.061 L17 Telugu 0.51.0 ECE=0.036 L17 Thai 0.51.0 ECE=0.017 L17 Turkish Empirical success rate Predicted success probability (b) Pooled multilingual Figure 9: Per-language reliability diagrams forgpt-oss-20b-lowat its shared reporting layer. The English-trained probe is transferred unchanged to each target language, whereas the pooled probe uses the same total training budget distributed equally across the ten languages. The diagonal denotes perfect calibration. gpt-oss-20b-high 0.00.51.0 0.0 0.5 1.0 ECE=0.099 L7 Arabic 0.51.0 ECE=0.065 L7 Chinese 0.51.0 ECE=0.018 L7 English 0.51.0 ECE=0.088 L7 French 0.51.0 ECE=0.097 L7 Italian 0.00.51.0 0.0 0.5 1.0 ECE=0.026 L7 Russian 0.51.0 ECE=0.130 L7 Swahili 0.51.0 ECE=0.069 L7 Telugu 0.51.0 ECE=0.040 L7 Thai 0.51.0 ECE=0.088 L7 Turkish Empirical success rate Predicted success probability (a) English-trained transfer 0.00.51.0 0.0 0.5 1.0 ECE=0.020 L7 Arabic 0.51.0 ECE=0.014 L7 Chinese 0.51.0 ECE=0.036 L7 English 0.51.0 ECE=0.045 L7 French 0.51.0 ECE=0.024 L7 Italian 0.00.51.0 0.0 0.5 1.0 ECE=0.006 L7 Russian 0.51.0 ECE=0.050 L7 Swahili 0.51.0 ECE=0.038 L7 Telugu 0.51.0 ECE=0.045 L7 Thai 0.51.0 ECE=0.030 L7 Turkish Empirical success rate Predicted success probability (b) Pooled multilingual Figure 10: Per-language reliability diagrams forgpt-oss-20b-highat its shared reporting layer. The English- trained probe is transferred unchanged to each target language, whereas the pooled probe uses the same total training budget distributed equally across the ten languages. The diagonal denotes perfect calibration.