Paper deep dive
What We Observe as LLM Behavior Can Be a Side-effect of Inference Backend
Shahed Masoudian, Passant Shafaei, Monorama Swain, Markus Schedl
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Benchmark scores are reported as properties of a model, yet the inference framework used to produce them, such as HuggingFace, vLLM, or Ollama, are considered non-influential and their names and versions are almost never disclosed. In this work we investigate how much this choice can influence the model output. In a fully-crossed study (three instruction-tuned models x five inference frameworks x six benchmarks x four generation modes) we investigate how different tools (wrappers/backend) influence benchmark scores and how their score changes is influenced by generation hyper-parameters. We find backend to be a non-negligible factor where even under greedy, sampling-noise-free decoding, changing the backend can significantly alter models performance and this effect is structural and strongly model-dependent. Decomposing the variance according to generation mode reveal that considerable portion of the variability (roughly 39\%) a practitioner sees out-of-the-box can stem from the backend, while the remaining stems from sampling noise and each framework's default generation parameters, both of which are avoidable by disclosing and matching the generation configuration. These divergences are more pronounced on factual than on social-bias benchmarks. Overall, benchmark numbers are not backend-agnostic therefore, we recommend disclosing the backend, its version, and the full generation configuration, also using deterministic decoding for cross-backend comparison.
Tags
Links
- Source: https://arxiv.org/abs/2608.04714v1
- Canonical: https://arxiv.org/abs/2608.04714v1
PDF not stored locally. Use the link above to view on the source site.
Full Text
63,116 characters extracted from source content.
Expand or collapse full text
What We Observe as LLM Behavior Can Be a Side-effect of Inference Backend Shahed Masoudian 1 , Passant Elchafei 1 , Monorama Swain 1 , Markus Schedl 1 1 Johannes Kepler University Linz Correspondence: shahed.masoudian@jku.at Abstract Benchmark scores are reported as properties of a model, yet the inference framework used to produce them, such as HuggingFace, vLLM, or Ollama, are considered non-influential and their names and versions are almost never disclosed. In this work we investigate how much this choice can influence the model out- put. In a fully-crossed study (three instruction- tuned models × five inference frameworks × six benchmarks × four generation modes) we investigate how different tools (wrapper- s/backend) influence benchmark scores and how their score changes is influenced by gen- eration hyper-parameters. We find backend to be a non-negligible factor where even under greedy, sampling-noise-free decoding, chang- ing the backend can significantly alter mod- els performance and this effect is structural and strongly model-dependent. Decompos- ing the variance according to generation mode reveal that considerable portion of the vari- ability (roughly 39%) a practitioner sees out- of-the-box can stem from the backend, while the remaining stems from sampling noise and each framework’s default generation parame- ters, both of which are avoidable by disclos- ing and matching the generation configuration. These divergences are more pronounced on fac- tual than on social-bias benchmarks. Overall, benchmark numbers are not backend-agnostic therefore, we recommend disclosing the back- end, its version, and the full generation config- uration, also using deterministic decoding for cross-backend comparison. 1 Introduction With the growing body of LLM analysis papers, reproducibility of results requires special attention. Prior works demonstrated that LLM score can be in- fluenced by factors other than model itself namely prompt formatting and few-shot ordering (Zhao et al., 2021; Lu et al., 2022; Sclar et al., 2024), the version of the evaluation harness (Biderman et al., 2024), and even precision of weights (Mekala et al., 2025). The obvious remark is that a reported per- formance is not just a property of a model alone but rather a joint property of a model, a prompt, an evaluation harness, and a possibly computational environment. What has received far less systematic evaluation is the framework through which model weights are loaded, tokenized, and prompted to generate output. Researchers routinely evaluate LLMs using a col- lection of frameworks whose properties may differ substantially. While HuggingFace Transformers, the HuggingFace pipeline are considered as most direct generation implementation of pytorch, Kwon et al. (2023) introduce PagedAttention and con- tinuous batching in vLLM, making its sampling implementation structurally different from Hug- gingFace’s sequentialmodel.generate(). Ollama wraps llama.cpp, which independently implements the GGUF format and its own tokenization pipeline. LangChain (Chase, 2022) adds a prompt-template and chain abstraction layer to the process. In major published papers the choice among these frame- works is either not reported or vaguely reported, and the assumption that they are interchangeable has not been empirically justified. When frame- works differ in how they render chat templates, tok- enization, set sampling parameter, or post-process outputs, then the same model evaluated with the same query may produce numerically different scores. This difference causes reproducibility is- sues specially in case authors do not disclose their exact experimental setup as well as inference frame- work used for their evaluation. This word addresses the gap with a controlled evalu- ation over four dimensions namely inference frame- work (Huggingface, Pipeline, Langchain, vllm and Ollama), generation mode (the hyper-parameters used for generation), benchmark, and model fam- ily (LLAMA-3.2-1B-INST, QWEN2.5-1.5B-INST, arXiv:2608.04714v1 [cs.SE] 5 Aug 2026 GEMMA-3-1B-IT), on a single consumer GPU without quantization. Rather than reporting only backend gaps, we treat the backend as one fac- tor among four and quantify investigate behavioral changes with respect to model, benchmark and gen- eration modes. Formally we investigate: RQ1:To which extent does the inference back- end alter benchmark scores, and under which generation modes does this effect emerge? RQ2:Are backend differences large enough to change evaluation conclusions or merely sta- tistically detectable? RQ3:How are these effects distributed across task families (factual vs. social bias)? Overall our work result in the following contri- butions: (1) We introduce a fully-crossed, repro- ducible test-bed (models×backends×bench- marks×generation modes) that isolates how the inference backend alone affects evaluation out- comes 1 . (2) We demonstrate that backend influ- ence even though a real factor and it can signifi- cantly alter model behavior even when sampling- noise doesn’t exist. (3) Finally we show that these effects are strongly model-dependent and, more pronounced on factual than on bias benchmarks. 2 Related Work The reproducibility of NLP results has been the community concern since at least Fokkens et al. (2013). Dodge et al. (2020) demonstrate that vari- ance due to random seeds in training and fine- tuning can render result rankings unstable across papers. Bouthillier et al. (2021) provide a taxon- omy of reproducibility sources in machine learning experiments more broadly. Crane (2018) and Wiel- ing et al. (2018) document systematic failures to replicate published NLP results. Zhao et al. (2021) show that few-shot in-context learning is sensitive to example order. Closer to our setting, Bider- man et al. (2024) show that version changes to the EleutherAI LM Evaluation Harness produce non-trivial score shifts for the same model on the same benchmark. A parallel line of work estab- lishes that model outputs are highly sensitive to prompt formulation. Blodgett et al. (2020) argue that bias benchmarks embed assumptions that are rarely made explicit. Lu et al. (2022) demonstrate 1 The code and analysis results will be available in our public repository upon acceptance. that permuting answer choices in multiple-choice prompts changes model rankings. Sclar et al. (2024) systematically vary format tokens (spacing, capi- talization, separator style) and find accuracy differ- ences of up to 80 p on some benchmarks. Liang et al. (2023) benchmark models using a standard- ized harness specifically to reduce this form of variation. Wang et al. (2024) characterize LLMs as sensitive to instruction noise and show that incon- sistent prompt templates compromise benchmark validity. Selvam et al. (2023) also show that Stere- oSet scores are sensitive to prompt phrasing and demonstrate that the generation-based variant of the benchmark produces different model rankings than the log-probability variant. Parrish et al. (2022) note that BBQ results can be confounded by model refusal behavior. On weight representation, Mekala et al. (2025) evaluate INT4 and INT8 quantized models drop accuracy of 2–8 p relative to full-precision base- lines. Singh and Sajjad (2025) show that models reasoning performance are more sensitive to pre- cision loss than factual recall tasks when quan- tized. Yuan et al. (2025) demonstrate that even greedy decoding is not fully reproducible. Due to non-associative floating-point arithmetic and fi- nite numerical precision, model outputs can vary across GPU architectures, batch sizes, and numer- ical precisions, leading to measurable differences in evaluation results. Our experimental setup con- siders and controls these known sources of vari- ability. all experiments are conducted with a fixed computational environment, GPU type, numerical configuration, and deterministic decoding settings, to ensure minimal external influence. Most closely related to our work, Pape et al. (2026) identify the inference backend as a silent hyperparameter and quantify its impact on LLM reproducibility, showing that the serving frame- work is a consequential and under-reported source of score variation. Our work differntiate from theirs in scope and objectives. We study cross influence of backend with model, benchmark and generation modes, allowing us to quantify the proportion of total score variance in interac- tions, rather than examining backend differences in isolation. Second, we introduce a controlled generation-mode axis, ranging from greedy decod- ing through parameter-matched stochastic decod- ing to framework-specific defaults. This enables us to dissect backends influence in comparison in relation to sampling noise and general generation parameter (e.g., max_new_token), a question that cannot be answered by a single-condition backend comparison. Third, we compare the effect across two task families (factual and social bias) and ana- lyze how backend sensitivity varies across model families. 3 Experimental Setup Our experimental design holds the weight preci- sion, hardware, and evaluation protocol constant while varying (1) the inference backend, (2) the generation modes (generation hyper parameters), (3) the benchmark, and (4) the model family. 3.1 Model and Computational Environment All experiments run models in 16-bit floating point on a single NVIDIA RTX 3090 (24 GiB VRAM), CUDA 12.6, PyTorch 2.6. Models are drawn from three instruction-tuned families at the ap- proximately 1B parameter scale: LLAMA-3.2-1B- INST (Meta), GEMMA-3-1B-IT (Google), and QWEN2.5-1.5B-INST (Alibaba). We intentionally selected models around the 1B parameter scale be- cause they allow exhaustive evaluation across five backends, six benchmarks, four generation modes, and multiple repetitions while keeping computa- tional requirements manageable and constant. 3.2 Inference Backends Five serving frameworks are evaluated, or- deredbyincreasingseparationfromthe raw PyTorch computation graph.hf_raw. Serves as the reference baseline and calls AutoModelForCausalLM.generate()directly with no wrapper, formats prompts via the tokenizer’s apply_chat_template method. hf_pipeline.Wrapsthesamein- process model object with HuggingFace’s pipeline("text-generation")abstraction. Since it shares the exact same model object ashf_raw, any divergence can be attributed to pipeline abstraction (e.g., preprocessing, prompt handling, generation defaults, or output post-processing). langchain_hf.Wraps that pipeline further with LangChain’s HuggingFacePipeline (Chase, 2022), introducing an additional prompt-formatting and orchestration layer. vllm.(version 0.8.5, V0 engine) runs in a separate CUDA context, loads an independent copy of the model, uses PagedAttention (Kwon et al., 2023) and an independent implementation of decoding and sampling routines. ollama. Maximally separated from the raw com- putation while it runs as a standalone OS process backed by llama.cpp using a GGUF model repre- sentation and its own tokenizer/runtime stack. Sys- tem prompts are delivered as a structuredsystem- role message via the Ollama chat API. For the four HuggingFace-ecosystem backends namelyhf_raw,hf_pipeline,langchainand for vllm, weights are loaded as fp16 SafeTensors di- rectly from the HuggingFace Hub cache. All four introduced inference backends operate on numeri- cally identical weight matrices. However,ollama loads a fp16 GGUF variant from Ollama’s own model registry hence not identical weights to the other four. Although fp16 precision is preserved, the GGUF export process may introduce small nu- merical discrepancies relative to the original check- point. Observed Ollama score differences therefore reflect a combination of serving-stack behavior and weight-representation artifacts. However, GGUF conversion is an unavoidable consequence of choos- ing Ollama, so its effect is part of the backend cost a practitioner incurs, not a confound to be removed. 3.3 Generation Modes We investigate backend effects under four gener- ation modes designed to progressively control or introduce additional sources of variability. Deterministic.Greedydecodingwith temperature=0,do_sample=False,seed=42, andmax_new_tokens=256. This mode controls stochasticity leaving only structural backend effect. Since model weights, prompts, and generation parameters are set with no sampling, any observed divergence reflects differences introduced by the inference stack itself, such as tokenization, chat- template rendering, numerical implementation, or other backend-specific processing. Fix.Sampling is enabled while gener- ationparametersarefixedacrossback- ends(temperature=0.7,top_p=0.9, top_k=50,repetition_penalty=1.1, max_new_tokens=256).This mode introduces sampling noise while keeping generation settings constant, allowing us to assess how sampling interacts with backend. Token=256. Only the generation length is standard- ized (max_new_tokens=256) while all remaining parameters are left at each framework’s defaults. This mode captures the combined effect of backend- specific defaults and sampling stochasticity while controlling for output-length differences. Default. Each backend is evaluated using its gen- eration configuration. This setting reflects the way practitioners frequently interact with models resort- ing to default parameters. It therefore represents the least controlled but most realistic deployment scenario(Specific values in Table 6). In Deterministic mode we collect one response from the model while for other modes, five re- sponse per query is collected to account for sam- pling noise. Note that the purpose of these modes is to understand how backends interact with gen- eration hyper-parameters. Deterministic isolates the structural backend component, whereas the re- maining modes progressively introduce additional sources of variability that practitioners commonly report in real-world usage. 3.4 Evaluation Suite We select our evaluation benchmarks with two task families in mind: 1) factual knowledge 2) social bias. Each task contains both classification-style and open-ended generation sub-tasks to cover vari- ations of generation styles. For each benchmark we evaluate models output based on metrics provided by the respective benchmark unless stated. MMLU (Hendrycks et al., 2021): 14,042 multiple- choice questions spanning 57 academic subjects. The model is presented with a four-way choice (A–D) with a format instruction in the system role. Metric: accuracy. TriviaQA (Joshi et al., 2017): 11,313 open-domain factual questions evaluated with the full set of accept- able aliases. Metrics:exact_match(binary, af- ter canonical normalization) andtoken_f1(token- level F1 using Counter intersection, maximized over all aliases). TruthfulQA MC1 (Lin et al., 2022): 817 adver- sarial questions designed to elicit common false beliefs, presented in multiple-choice format with up to eight options (A–H). Metric: accuracy. TruthfulQA-Gen: the generation variant of Truth- fulQA. The model produces a free-form answer scored against both the curatedbest_answer and the full set ofcorrect_answers. Metrics: rouge_l_best(mean ROUGE-L against the single best answer) androuge_l_correct(mean maxi- mum ROUGE-L against any correct answer). Note that these are reported as surface-overlap proxies while the original fine-tuned classifier evaluation (Lin et al., 2022) is not backend-agnostic and is therefore not used here. BBQ (Parrish et al., 2022): 58,492 questions testing social bias across eleven demographic categories.Each item appears in two con- texts:disambiguated (factual context identi- fies the correct answer) and ambiguous (the only correct answer is “Unknown”).Met- rics:accuracy: overall fraction of correct an- swers.disambig_accuracy: accuracy on dis- ambiguated items.ambig_unknown_rate: frac- tion of ambiguous items answered “Unknown”. ambig_bias_score: the directional bias metric from Parrish et al. (2022), defined as(n biased − n counter )/(n biased +n counter )over non-Unknown re- sponses to ambiguous items. Range[−1, +1]; 0 = unbiased; positive = stereotypically biased. StereoSet (Nadeem et al., 2021): 4,229 items (2,106 intra-sentence, 2,123 inter-sentence) test- ing whether models prefer stereotypical sentence completions. We adapt StereoSet to a generation- based multiple-choice format(A/B/C, shuffled with a fixed random seed across all backends) so scores are not directly comparable to the StereoSet leader- board. Metrics:lms(Language Model Score): frac- tion of items where the model selected a meaning- ful sentence over the unrelated option.s(Stereo- type Score): among meaningful choices, the frac- tion that were stereotypical. Target: 0.50 (unbi- ased).icat:lms× 2× min(s, 1− s); penalizes low lms and biased s.invalid_rate: fraction of items producing no parseable A/B/C response. All StereoSet metrics are computed from pooled counts across both sub-tasks (Nadeem et al., 2021). Overall, roughly7.2M generations per model to cover six benchmarks, five backends, and four gen- eration modes with repetitions. This budget makes the full-factorial design sensible at the∼1B scale. 3.5 Score Calibration The six benchmarks produce in total 14 metrics with different scales, optimization directions, and ideal target values. To enable meaningful aggre- gation during reporting, all scores are calibrated to a common[0, 1]scale in which1always in- dicates the best attainable value and0indicates the worst.For higher-is-better metrics (accu- racy, exact match, token F1, ROUGE-L, LMS, ICAT,ambig_unknown_rate) we apply no calibra- tion. Forinvalid_rateandambig_bias_score (ideal 0), calibration is1− raw. Forss(ideal 0.5), calibration is 1− 2|raw− 0.5|. 3.6 Metrics Here we define every quantity reported in Sec- tion 4. We define a cell as a unique combination (m,b,d,c)of model familym, inference backend b, benchmarkd, and generation modec. Bench- markdhasN d items; letx (m,b,d,c) i ∈ [0, 1]be the calibrated score of itemi(Section 3.5), where1is the best attainable value, and for binary-outcome metricsx i = v i ∈0, 1is the correctness verdict. The cell score is the average over all items: s m,b,d,c = 1 N d N d X i=1 x (m,b,d,c) i Considering the baseline backend isb 0 = hf_raw, we hold some subset of the indices(m,d,c)fixed and compares target backendbagainstb 0 , and ag- gregate over the remaining free indices either by averaging (e.g. the mean|∆|over benchmarks and metrics, or the mean variance over model bench- mark pairs) or by counting (e.g. how many compar- isons are significant). Divergence (∆).is the score divergence of back- end b from the baseline b 0 , ∆ b = s b − s b 0 Note that|∆|is the absolute divergence and repre- sents the magnitude. Disagreement rate. In binary verdicts, it is the fraction of items on which b and b 0 differ, dis b = 1 N d N d X i=1 1 h v b i ̸= v b 0 i i . Severity. A (backend, mode) disagreement is considered severe when a given model and mode flips more than5%of the task’s items, i.e.dis b > 0.05 .This is the magnitude at which prior Model hf_pipeline langchain_hf vllm ollama LLAMA-3.2-1B-INST0.000 (0)0.049 (6)0.001 (0)0.055 (9) GEMMA-3-1B-IT0.000 (0)0.002 (2)0.000 (0)0.010 (4) QWEN2.5-1.5B-INST0.000 (0)0.009 (8)0.025 (10)0.011 (7) Table 1: Mean|∆|fromhf_rawacross the 14 primary metrics, with the number BH-significant out of 11 in parentheses, per model and backend for deterministic mode which is sampling noise free. work treats benchmark differences as consequen- tial (Mekala et al., 2025; Sclar et al., 2024). Variance and Standard deviation. Due to va- riety of individual model performance as well as unbalanced datasets for each benchmark, comput- ing the variance simply by aggregating over all benchmarks or over all models is not revealing. We measure the backend’s contribution as variance of the cell score (σ 2 m,d (c) = 1 |B| P b∈B s b − ̄s m,d 2 ) at a fixed modecwhere ̄s m,d is the mean over backends. We then average over all models and benchmark pairs (standard deviation is p ̄σ 2 (c)): ̄σ 2 (c) = 1 |M||D| X m,d σ 2 m,d (c), Significance test.On the binary-outcome bench- marks (MMLU, TruthfulQA, BBQ, TriviaQA exact-match) we comparebagainstb 0 per item with McNemar’s paired test. Continuous metrics use the Wilcoxon signed-rank test on the paired differences x b i − x b 0 i . Across theMtests in a comparison we control the false-discovery rate with Benjamini– Hochberg. We sortp (1) ≤ · ≤ p (M) , and reject every(k)withp (k) ≤ k M α(α = 0.05) and call such results BH-significant. To test whether the five backends differ jointly within a mode we use the Friedman omnibus test with(m,d)pairs. We also report Effect size and Cohen Kappa agreement test are also provided in Appendix H and C. 4 Results and Discussion Before we dive into the results please note that dur- ing reporting we utilize all 14 metrics provided in Section 3.4 while for the significance test we ex- clude (ambig_bias_score, S, and ICAT) which are calculated from other independent metrics and uti- lize the remaining 11 independent per-item metrics . BackendDeterm.FixToken=256Default hf_pipeline0/332/3317/3319/33 langchain_hf16/3316/3315/3318/33 vllm10/3312/3319/3322/33 ollama20/3323/3323/3327/33 Table 2: Number of significant divergences from hf_raw, out of 30 primary-metric comparisons (3 mod- els×11 metrics), for each backend and generation mode. 4.1 RQ1: To which extent does the inference backend alter benchmark scores? Table 1 reports the mean divergence (|∆|, Sec- tion 3.6) from thehf_rawbaseline under determin- istic generation for each model/backend pair, aver- aged across all 14 evaluation metrics. Parentheses indicate the number of statistically significant diver- gences among the 11 per-item metrics. Overall the backend effect is model-dependent.hf_pipeline is an exact replica ofhf_rawwith0.0divergence, as expected. GEMMA-3-1B-IT is least affected by the backend where no divergence exceeds0.010. LLAMA-3.2-1B-INST is the most affected, with ollama(0.055) andlangchain_hf(0.049) diverg- ing substantially. Forlangchain_hfthe mech- anism is prompt handling on a shared, weight- identical engine, and forollamait mixes GGUF weight conversion with template and tokenization handling, which cannot be separated.vllm, nu- merically identical tohf_rawfor LLAMA-3.2-1B- INST and GEMMA-3-1B-IT (0.000), instead di- verges for QWEN2.5-1.5B-INST (0.025, signifi- cant on10/11metrics). Considering that models are suing the exact same model running vLLM in the same bfloat16 ashf_rawleaves the divergence essentially from vLLM’s distinct kernel implemen- tation (PagedAttention) under non-associative arith- metic (Yuan et al., 2025). Effect-size analysis (Ta- ble 7) shows these differences reach|r| = 0.38 forvllmon QWEN2.5-1.5B-INST. Since greedy decoding removes sampling noise, the observed di- vergences reflect structural backend effects. These results establish the presence of a backend effect even in the absence of stochasticity, a pattern that becomes relevant when considering more stochas- tic generation modes. Table 2 counts, for each backend and mode, how many of the33comparisons (3 models×11 per-item metrics) significantly differ fromhf_raw. Moving from Deterministic,hf_pipelineshows low divergence fromhf_raw(2/33under Fix with matched parameters, averaged over five repeti- tions), which we attribute to the sampling-noise while for others this is the mixed influence of sam- pling noise and backend differences. As we let backend default generation modes take over the divergence grows to17/33and19/33under To- ken=256 and Default, a direct result of variation in the backends’ default generation parameters. Note that Token=256 controls onlymax_new_tokens and consistently shows less significant divergence compared to Default, where all parameters are pre- determined (Table 6). 4.2 RQ2: Are backend differences large enough to change evaluation conclusions? Table 3 reports the variance attributable to backend choice under different generation modes, aggre- gated across models and benchmarks. Determinis- tic isolates the structural backend effect, Fix adds sampling noise while keeping all generation hyper- parameters matched, and Default further introduces each framework’s own default hyper-parameters. Under this attribution, the structural backend effect alone accounts for a variance of0.00074. Adding sampling noise increases the variance to0.00112, while framework-specific default hyper-parameters increase it further to0.00190. Thus, even when evaluating the same model on the same benchmark, the choice of serving framework and generation configuration can affect the reported results. Under this incremental attribution, the structural backend effect accounts for approximately 39% of the variance observed with Default settings, while the additional variability introduced by sampling and framework-specific defaults accounts for the remaining 61%. These values represent sequential contributions rather than an exact causal decompo- sition, as interactions between backend behavior and generation settings may exist. Mode (variance sources)VarianceStd Deterministic (backend only)0.000740.017 Fix (+ sampling noise)0.001120.016 Token=256 (+ default hyper-params)0.001180.019 Default (+ all default hyper-params)0.001900.026 Table 3: Backend-induced variance computed within each (model, benchmark) and averaged over all models and benchmark pairs, by generation mode. Comput- ing the variance within a single model and benchmark removes differences in benchmark scale and difficulty, revealing the backend’s contribution. 0.00.20.4 |Δ calibrated score| 0.0 0.1 0.2 0.3 0.4 0.5 disagreement rate not severe Gemma-3-1B Backend (shape) hf_pipeline langchain_hf vllm ollama 0.00.20.4 |Δ calibrated score| Llama-3.2-1B 0.00.20.4 |Δ calibrated score| Qwen2.5-1.5B Task (colour) MMLU TriviaQA TruthfulQA TruthfulQA-Gen BBQ StereoSet Figure 1:|∆|fromhf_raw(xaxis) versus the per-item disagreement rate (yaxis) for each (model, backend, dataset, mode) cell. Marker shape encodes the backend and colour the benchmark. The shaded quarter-circle (radius0.05) is the “not-severe” zone and a divergence that flips fewer than5%of items. The dashed line isy = xwhere rate of flips is equivalent to|∆| score. The variance decomposition establishes that backend-related factors considerably contribute to benchmark variability. The remaining question is whether this variability is merely detectable or it is large enough to alter evaluation conclusions. To answer this, we move from variance attribution to practical significance. We look at severe cases of divergence (introduced in Section 3.6) with5% threshold. Figure 1 plots the disagreement rate (Section 3.6) against the|∆|for every (model, back- end, metric, mode) cell (points inside the shaded quarter-circle are not severe), and Table 4 counts the severe configurations for each model and task out of the sixteen backend×mode combinations. Note that hf_raw is the baseline of comparison. TaskLlamaGemmaQwen MMLU1449 TriviaQA14015 TruthfulQA1448 TruthfulQA-Gen845 BBQ1444 StereoSet940 Table 4: Number of the16(backend (4)×mode (4)) configurations that are severely influenced for each model and task (i.e. that flip the correct/incorrect verdict on more than 5% of the task’s items). Our results indicate a model-dependence behavior not limited to a single benchmark. LLAMA-3.2- 1B-INST is affected by backend and mode on al- most every task namely14of the16configurations on MMLU, TriviaQA, TruthfulQA and BBQ, and 8–9on the two generation-based tasks, with per- item disagreement reaching48%on StereoSet and 40%on TruthfulQA. GEMMA-3-1B-IT, in con- trast, is severe on at most4of16configurations on any task (none on TriviaQA) and its disagree- ment never exceeds13%. QWEN2.5-1.5B-INST sits between the two and its severity seems to be highly task dependent (15/16on TriviaQA,9/16 on MMLU) while it stays robust to Stereoset (0/16) and 4/16 on BBQ. Two patterns hold across all models. First, almost all severe cells lie above they = xdiagonal in Figure 1, so the aggregate divergence understates how many individual answers change. Second, per-question agreement corroborates the ranking. Further analysis of Cohen’sκagainsthf_raw(Ap- pendix D) averages0.84forhf_pipelinebut falls to0.45forollamaon LLAMA-3.2-1B-INST, ver- sus0.79on GEMMA-3-1B-IT the same ordering appears in the error-set overlap (Appendix E), and the score-space clustering (Appendix F). 4.3 RQ3: How are these effects distributed across task families ModeQWEN2.5-1.5B-INSTGEMMA-3-1B-ITLLAMA-3.2-1B-INST Fact.BiasFact.BiasFact.Bias Deterministic0.0168 ∗ 0.00690.00250.00400.0123 ∗ 0.0368 ∗ Fix0.00740.00550.00540.00410.0214 ∗ 0.0413 ∗ Token=2560.00820.00420.00420.00400.0948 ∗ 0.0407 ∗ Default0.0467 ∗ 0.00440.0420 ∗ 0.00400.0789 ∗ 0.0380 ∗ Table 5: Average|∆|fromhf_raw, split by task family (Factual: MMLU, TriviaQA, TruthfulQA, TruthfulQA- Gen; Bias: BBQ, StereoSet). Values are averaged across all non-baseline backends and all primary metrics within each group. Higher = greater divergence. ∗ = significant and at least one benchmark×backend pair BH-adjusted significant (p < 0.05). Backend effects are not uniformly distributed across task families. Table 5 report divergence fromhf_rawaccording to factual and social-bias benchmark families. Under deterministic decoding, task sensitivity differs across models. LLAMA-3.2- 1B-INST and GEMMA-3-1B-IT exhibit higher di- vergence on bias benchmarks, whereas QWEN2.5- 1.5B-INST shows higher divergence on factual benchmarks. Once generation modes are intro- duced, factual benchmarks become more sensitive, particularly under default settings. Interestingly, LLAMA-3.2-1B-INST exhibits larger|∆|under Token=256 than under Default generation, espe- cially for factual benchmarks (|∆| = 0.0948vs. 0.0789). Since Token=256 forces all frameworks to generate up to 256 new tokens, this increase may reflect the sensitivity of factual generation to longer decoding trajectories, where backend-specific stop- ping behavior and generation dynamics have more opportunity to diverge. In contrast, QWEN2.5- 1.5B-INST and GEMMA-3-1B-IT show the op- posite trend, with Default producing larger diver- gence than Token=256, indicating that generation length alone does not explain the observed differ- ences but interacts with model characteristics and framework-specific defaults. Across the 12 model–mode configurations, fac- tual benchmarks exhibit significant divergence in 7 cases, compared with 4 cases for bias benchmarks. These results indicate that backend effects more than generation mode and model behavior interact with benchmark characteristics. Taken together, these results support a consistent conclusion. The inference backend is a genuine factor influencing benchmark measurements rather than a negligible implementation detail. Even un- der greedy decoding, where sampling variability is removed, changing only the inference stack can alter measured performance. The magnitude and direction of these changes depend jointly on the model, benchmark, and generation configuration, meaning that a single aggregate score can hide important sources of evaluation variability. We therefore recommend that evaluation studies re- port the backend, its exact version, and the com- plete generation configuration, ensuring that avoid- able sources of variation, such as sampling parame- ters and framework-specific defaults, remain repro- ducible rather than implicit. Furthermore, our anal- ysis shows that sampling variability remains non- negligible even when averaging across five gen- erations, suggesting that stochastic decoding can obscure backend-induced effects. Greedy decod- ing therefore provides a more reproducible regime. While bias benchmarks exhibited generally smaller backend-induced divergence in our evaluation, fac- tual benchmarks showed greater sensitivity to gen- eration configuration. Overall, backend choice represents a measurable source of evaluation vari- ability where some components can be controlled through standardized generation settings, while the remaining backend-induced differences should be explicitly measured and reported rather than left unspecified. 5 Conclusion In this work we presented a systematic evalua- tion of how five LLM inference frameworks affect benchmark scores when the model, data, and evalu- ation protocol are held constant. Under greedy decoding the structural backend effect amounts to a variance of0.00074where adding sampling raises it to0.00112, and each framework’s de- fault hyper-parameters raise it further to0.00190. Even under greedy decoding, where no sampling noise is present, the backend still changes which items a model answers correctly, and its effect is strongly model-dependent. We relate these diver- gences to differences in each framework’s serving stack namely the tokenization, system template, and default paramters suited for the backend or even weight of the model in case of Ollama rather than to a single cause. The configuration-driven part can be neutralized by matching the genera- tion settings, whereas these structural, weight- and template-level differences can only be disclosed and considered consequential backend choice ef- fecting reproducibility. For the evaluation com- munity, the central implication is that benchmark numbers are not backend-agnostic. We therefore recommend a minimum reporting standard where the backend name and its version, the complete gen- eration configuration, and, for any cross-backend comparison, the use of deterministic (greedy) de- coding as the only regime free of sampling noise. Finally we recommend for bias benchmarks, report- ing format-compliance alongside the bias scores. 6 Limitations On the generation axis, our four modes probe only a few points in a large space. The Fix and Token=256 modes each fix a single hyper-parameter configura- tion, and we do not sweep temperature, top-p, top- k, or the repetition penalty to map how backend divergence varies across the sampling landscape; nor do we vary batch size, the attention backend, KV-cache settings, or the random seed beyond a single value. We run every model in float16 without quantization, whereas quantization format and compute precision are themselves likely sources of cross-backend dis- agreement that our design holds constant. The stochastic modes use five generations per prompt, which bounds the precision of our sampling-noise and variance estimates. A larger number of repe- titions would tighten the decomposition between structural and sampling-driven variance, and the Deterministic mode rests on a single greedy run whose reproducibility we assume within, but do not test beyond, our environment. That environ- ment is a single consumer GPU (RTX 3090) with one CUDA and driver version. Since floating-point non-associativity is hardware-dependent, the ab- solute magnitudes we report though not severely, we expect, the qualitative model-dependent pattern may differ slightly on other accelerators or preci- sions. Furthermore, we provide 3 different models from different family but do not try out scaling factor in backend influence. Eventhough (Pape et al., 2026) utilizes models of higher sizes and demonstrate they are backend dependent we are not independently investigating this matter. Finally, the evaluation and analysis are scoped in ways that bound the conclusions. Our six bench- marks are English-only and dominated by multiple- choice and short-form generation. We do not cover very long-form generation, code, mathematical reasoning, multi-turn dialogue, or agentic tasks, where output-length and sampling effects could compound differently. Bias is represented by two US- and English-centric social-bias suites (BBQ and StereoSet), and StereoSet’s composite score conflates format compliance with bias direction, a confound we report but cannot fully disentangle. Analytically, we treathf_rawas the reference back- end, so all divergences are relative rather than mea- sured against an external ground truth. We reduce each benchmark to one calibrated primary metric in[0, 1]and our within-cell variance attribution assumes that the ladder of modes cleanly and ad- ditively separates structural, sampling, and hyper- parameter sources, an approximation rather than an exact decomposition. Our per-item flip metric is binary and its numerical value depends on each benchmark’s scoring rule, and we rely throughout on automatic metrics without human evaluation of the generated text. All in all our analysis brings forth signs of backend dependency and further investigation of severity of influence per task or scaling of the results to larger models requires further investigation which we leave for future works. References Stella Biderman, Hailey Schoelkopf, Lintang Sutawika, Leo Gao, Jonathan Tow, Baber Abbasi, Alham Fikri Aji, Phat Alexandrov, Anton Arkil, Stella Biderman, and 1 others. 2024. Lessons from the Trenches on Reproducible Evaluation of Language Models. arXiv preprint arXiv:2405.14782. Su Lin Blodgett, Solon Barocas, Hal Daumé I, and Hanna Wallach. 2020. Language (technology) is power: A critical survey of “bias” in NLP. In Proceedings of the 58th Annual Meeting of the Association for Compu- tational Linguistics, pages 5454–5476, Online. Associa- tion for Computational Linguistics. Xavier Bouthillier, Pierre Delaunay, Mirko Bronzi, Assya Trofimov, Brennan Nichyporuk, Justin Szeto, Nazanin Mohammadi Sepahvand, Edward Raff, Kanika Madan, Vikram Voleti, Samira Ebrahimi Kahou, Vin- cent Michalski, Tal Arbel, Chris Pal, Gael Varoquaux, and Pascal Vincent. 2021. Accounting for variance in machine learning benchmarks. In Proceedings of Ma- chine Learning and Systems, volume 3, pages 747–769. Harrison Chase. 2022. LangChain.https://github. com/langchain-ai/langchain. Matt Crane. 2018. Questionable answers in question answering research: Reproducibility and variability of published results. Transactions of the Association for Computational Linguistics, 6:241–252. Jesse Dodge, Gabriel Ilharco, Roy Schwartz, Ali Farhadi, Hannaneh Hajishirzi, and Noah A. Smith. 2020. Fine-tuning pretrained language models: Weight initial- izations, data orders, and early stopping. arXiv preprint arXiv:2002.06305. Antske Fokkens, Marieke van Erp, Marten Postma, Ted Pedersen, Piek Vossen, and Nuno Freire. 2013. Off- spring from reproduction problems: What replication failure teaches us. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguis- tics (Volume 1: Long Papers), pages 1691–1701, Sofia, Bulgaria. Association for Computational Linguistics. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. Measuring massive multitask language under- standing. In 9th International Conference on Learn- ing Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net. Mandar Joshi, Eunsol Choi, Daniel S. Weld, and Luke Zettlemoyer. 2017. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehen- sion. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, ACL 2017, Vancouver, Canada, July 30 - August 4, Volume 1: Long Papers, pages 1601–1611. Association for Computa- tional Linguistics. Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gon- zalez, Hao Zhang, and Ion Stoica. 2023. Efficient mem- ory management for large language model serving with pagedattention. In Proceedings of the 29th Symposium on Operating Systems Principles, SOSP 2023, Koblenz, Germany, October 23-26, 2023, pages 611–626. ACM. Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, Ben- jamin Newman, Binhang Yuan, Bobby Yan, Ce Zhang, Christian Cosgrove, Christopher D. Manning, Christo- pher Ré, Diana Acosta-Navas, Drew A. Hudson, and 31 others. 2023. Holistic evaluation of language models. Trans. Mach. Learn. Res., 2023. Stephanie Lin, Jacob Hilton, and Owain Evans. 2022. Truthfulqa: Measuring how models mimic human false- hoods. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022, pages 3214–3252. Association for Computational Linguistics. Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. 2022.Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022, pages 8086–8098. Associa- tion for Computational Linguistics. Anmol Mekala, Anirudh Atmakuru, Yixiao Song, Marzena Karpinska, and Mohit Iyyer. 2025. Does quan- tization affect models’ performance on long-context tasks? In Proceedings of the 2025 Conference on Em- pirical Methods in Natural Language Processing, pages 9422–9470, Suzhou, China. Association for Computa- tional Linguistics. Moin Nadeem, Anna Bethke, and Siva Reddy. 2021. Stereoset: Measuring stereotypical bias in pretrained language models. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguis- tics and the 11th International Joint Conference on Nat- ural Language Processing, ACL/IJCNLP 2021, (Volume 1: Long Papers), Virtual Event, August 1-6, 2021, pages 5356–5371. Association for Computational Linguistics. Pape, Evertz, and Schönherr. 2026. The silent hy- perparameter: Quantifying the impact of inference backends on LLM reproducibility.arXiv preprint arXiv:2605.19537. Alicia Parrish, Angelica Chen, Nikita Nangia, Vishakh Padmakumar, Jason Phang, Jana Thompson, Phu Mon Htut, and Samuel R. Bowman. 2022. BBQ: A hand- built bias benchmark for question answering. In Find- ings of the Association for Computational Linguistics: ACL 2022, Dublin, Ireland, May 22-27, 2022, volume ACL 2022 of Findings of ACL, pages 2086–2105. Asso- ciation for Computational Linguistics. Melanie Sclar, Yejin Choi, Yulia Tsvetkov, and Alane Suhr. 2024. Quantifying language models’ sensitivity to spurious features in prompt design or: How I learned to start worrying about prompt formatting. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenRe- view.net. Nikil Selvam, Sunipa Dev, Daniel Khashabi, Tushar Khot, and Kai-Wei Chang. 2023. The tail wagging the dog: Dataset construction biases of social bias bench- marks. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 1373–1386, Toronto, Canada. Association for Computational Linguistics. Manpreet Singh and Hassan Sajjad. 2025. Interpreting the effects of quantization on LLMs. In Proceedings of the 14th International Joint Conference on Natural Language Processing and the 4th Conference of the Asia-Pacific Chapter of the Association for Computa- tional Linguistics, pages 2267–2281, Mumbai, India. The Asian Federation of Natural Language Processing and The Association for Computational Linguistics. Ruochen Wang, Sohyun An, Minhao Cheng, Tianyi Zhou, Sung Ju Hwang, and Cho-Jui Hsieh. 2024. One prompt is not enough: Automated construction of a mixture-of-expert prompts. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024, volume 235 of Proceedings of Machine Learning Research, pages 50043–50064. PMLR / OpenReview.net. Martijn Wieling, Josine Rawee, and Gertjan van No- ord. 2018. Squib: Reproducibility in computational linguistics: Are we willing to share? Computational Linguistics, 44(4):641–649. Jiayi Yuan, Hao Li, Xinheng Ding, Wenya Xie, Yu-Jhe Li, Wentian Zhao, Kun Wan, Jing Shi, Xia Hu, and Zirui Liu. 2025. Understanding and mitigating numer- ical sources of nondeterminism in LLM inference. In Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2025, NeurIPS 2025, San Diego, CA, USA, De- cember 2-7, 2025 / Mexico City, Mexico, November 30 - December 5, 2025. Zihao Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. 2021. Calibrate before use: Improving few-shot performance of language models. In Proceed- ings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Re- search, pages 12697–12706. PMLR. Table 6: Default-mode generation parameters, per backend and model. BackendModel temp. top_p top_k rep. pen.max new tok. ollamaLLAMA-3.2-1B-INST 0.80.90401.1−1 (unbounded) GEMMA-3-1B-IT1.00.95641.1−1 (unbounded) QWEN2.5-1.5B-INST 0.80.90401.1−1 (unbounded) vllmLLAMA-3.2-1B-INST 1.01.00−1 (off)1.016 GEMMA-3-1B-IT1.01.00−1 (off)1.016 QWEN2.5-1.5B-INST 1.01.00−1 (off)1.016 hf_pipelineLLAMA-3.2-1B-INST 0.60.90501.020 GEMMA-3-1B-IT1.00.95641.020 QWEN2.5-1.5B-INST 0.70.80201.120 hf_rawLLAMA-3.2-1B-INST 0.60.90501.020 GEMMA-3-1B-IT1.00.95641.020 QWEN2.5-1.5B-INST 0.70.80201.120 langchain_hfLLAMA-3.2-1B-INST 0.60.90501.020 GEMMA-3-1B-IT1.00.95641.020 QWEN2.5-1.5B-INST 0.70.80201.120 A Default generation parameters Table 6 shows the default parameters of the models for each backend.Each cell is the effective generation configuration a back- end applies when the harness supplies no parameter, read directly from the running stack.For the HuggingFace-family backends from each model’smodel.generation_config (transformers 4.57), for vLLM0.8.5from SamplingParams(), and for Ollama0.31.1from ollama show --parameterson the exact fp16 tags we serve.The three HuggingFace back- ends dispatch to the samemodel.generate()and therefore share one configuration, which is the model’s own tuned one; vLLM ignores the model’s generation_configand samples from a fixed temperature= 1.0; Ollama reads the model’s Modelfile and otherwise falls back to its runner defaults. Values not set by the model or Modelfile inherit the framework fallback (transformers: top_k= 50,rep. pen.= 1.0,max_length= 20; Ollama:temp.= 0.8,top_p= 0.9,top_k= 40, rep. pen.= 1.1). All configurations sample (do_sample=truein every model’s config; vLLM and Ollama always sample). B Per-Model Signed Divergence from hf_raw Figure 2 reveals the direction of backend effects that an absolute-value (unsigned) view conceals. For LLAMA-3.2-1B-INST, TruthfulQA cells show mixed positive and negative divergence (Ollama above, vLLM belowhf_raw), explaining why the absolute divergence is large while individual back- ends go in opposite directions. For Gemma and Qwen, the vast majority of cells are near-zero and non-significant. C Per-Backend Signed Divergence by Task Figures 3 and 4 give the most granular view behind RQ1 and RQ2. For each model, and for the Deter- ministic and Default modes, the signed difference of every non-baseline backend fromhf_rawon each task’s primary metric, with∗marking tasks on which at least one metric is BH-significant. Under Deterministic decoding (Figure 3)hf_pipeline is uniformly near zero, Gemma’s rows are pale throughout, and Llama shows the strongest struc- tural divergence on the TruthfulQA family. Under Default decoding (Figure 4) the cells deepen most sharply forvllmandollamaon Llama’s factual tasks reflecting the framework-default sampling gaps and added sampling noise discussed in Sec- tion 4.3. D Inter-Backend Agreement Analysis To complement aggregate mean-divergence met- rics, we assess whether backends agree on which questions a model answers correctly. For each (model, mode, dataset) we compute whether mod- els agree on correctness of the answer as a bi- nary measure, correct/wrong and calculate Fleissκ across all five backends jointly, and Cohen’sκfor each non-baseline backend paired againsthf_raw. Figure 5 shows compound Cohenκviews. The Backend×mode panel (right) confirms the sen- MMLU TriviaQA-EM TriviaQA-F1 TruthfulQA TQAGen-Rb TQAGen-Rc BBQ-Acc BBQ-Dis BBQ-Unk BBQ-Bias S-LMS S-Inv S-S S-ICAT Metric Deterministic Default Fix Token=512 Mode +0.01*+0.01*+0.01*-0.00*+0.02*+0.02*-0.00*+0.00*-0.01*-0.01+0.00*+0.00+0.00+0.00 +0.00*+0.00*-0.00*-0.01*-0.09*-0.11*-0.01*-0.00*-0.01*-0.00+0.00*+0.00+0.00+0.01 +0.01*+0.01*+0.01*-0.00*+0.00*+0.00*-0.00*+0.00*-0.01*-0.01+0.00*+0.00+0.00+0.01 +0.00*+0.00*-0.00*-0.01*+0.00*+0.01*-0.01*-0.00*-0.01*+0.00+0.00*-0.00+0.00+0.00 Qwen MMLU TriviaQA-EM TriviaQA-F1 TruthfulQA TQAGen-Rb TQAGen-Rc BBQ-Acc BBQ-Dis BBQ-Unk BBQ-Bias S-LMS S-Inv S-S S-ICAT Metric +0.00-0.00*-0.00+0.00-0.00*-0.00+0.00*+0.00*+0.00*-0.00+0.01*+0.00-0.01+0.00 +0.00*+0.00*+0.00*+0.01*-0.10*-0.13*+0.00*+0.00*+0.00*-0.00+0.01*-0.00-0.01+0.00 +0.00*+0.00*+0.00+0.01*+0.01*+0.01*+0.00*+0.00*+0.00*-0.00+0.01*+0.00*-0.01+0.00 +0.00*+0.00*+0.00*+0.00+0.00*+0.01*+0.00*+0.00*+0.00*-0.00+0.01*+0.00-0.01+0.01 Gemma MMLU TriviaQA-EM TriviaQA-F1 TruthfulQA TQAGen-Rb TQAGen-Rc BBQ-Acc BBQ-Dis BBQ-Unk BBQ-Bias S-LMS S-Inv S-S S-ICAT Metric -0.01*-0.01*-0.01*-0.00*-0.01*-0.01*+0.00*-0.03*+0.04*+0.01-0.07*-0.06*-0.00-0.07 -0.02*-0.03*-0.04*-0.02*-0.14*-0.17*-0.00-0.03*+0.02*+0.00-0.08*-0.09*-0.00-0.08 +0.02*-0.00*-0.01*+0.07*+0.01*+0.01*+0.01*+0.01*+0.00*-0.00+0.09*+0.11*+0.01+0.09 -0.02*-0.03*-0.03*-0.03*-0.18*-0.22*+0.00*-0.03*+0.03*+0.00-0.08*-0.10*+0.00-0.08 Llama −0.2 −0.1 0.0 0.1 0.2 Avg. signed diff vs hf_raw Figure 2: Average divergence fromhf_rawon the calibrated primary metric of each benchmark, for each generation mode. Note that contrary to main body of paper here we report the∆instead of|Delta|to capture negative and positive influences. ∗ = at least one backend BH-adjusted significant (p < 0.05). MMLU TriviaQA-EM TriviaQA-F1 TruthfulQA TQAGen-Rb TQAGen-Rc BBQ-Acc BBQ-Dis BBQ-Unk BBQ-Bias S-LMS S-Inv S-S S-ICAT Metric hf_pipeline langchain_hf vllm ollama Backend +0.00+0.00+0.00+0.00+0.00+0.00+0.00+0.00+0.00+0.00+0.00+0.00+0.00+0.00 -0.00-0.01*-0.01*+0.02*-0.01*-0.01*+0.00*-0.00+0.01*+0.01-0.02*+0.00+0.01-0.01 +0.02*+0.03*+0.03*-0.04*+0.07*+0.08*-0.01*+0.01*-0.03*-0.02+0.01*+0.00+0.00+0.01 +0.01*+0.02*+0.02*+0.01+0.00+0.00-0.01*+0.01*-0.03*-0.02+0.01*+0.00-0.00+0.01 Qwen, Deterministic −0.075 −0.050 −0.025 0.000 0.025 0.050 0.075 Signed diff vs hf_raw MMLU TriviaQA-EM TriviaQA-F1 TruthfulQA TQAGen-Rb TQAGen-Rc BBQ-Acc BBQ-Dis BBQ-Unk BBQ-Bias S-LMS S-Inv S-S S-ICAT Metric hf_pipeline langchain_hf vllm ollama Backend +0.00+0.00+0.00+0.00+0.00+0.00+0.00+0.00+0.00+0.00+0.00+0.00+0.00+0.00 +0.00-0.00*-0.00-0.01-0.01*-0.01+0.00+0.00-0.00-0.00+0.00+0.00+0.00+0.00 +0.00-0.00+0.00-0.00+0.00+0.00+0.00+0.00+0.00+0.00-0.00+0.00-0.00-0.00 +0.01+0.00+0.00+0.02+0.00+0.00+0.01*+0.02*+0.00*-0.00+0.04*+0.00-0.03+0.01 Gemma, Deterministic −0.04 −0.02 0.00 0.02 0.04 Signed diff vs hf_raw MMLU TriviaQA-EM TriviaQA-F1 TruthfulQA TQAGen-Rb TQAGen-Rc BBQ-Acc BBQ-Dis BBQ-Unk BBQ-Bias S-LMS S-Inv S-S S-ICAT Metric hf_pipeline langchain_hf vllm ollama Backend +0.00+0.00+0.00+0.00+0.00+0.00+0.00+0.00+0.00+0.00+0.00+0.00+0.00+0.00 -0.01-0.04*-0.04*-0.04*-0.00-0.00+0.02*-0.12*+0.15*+0.02-0.11-0.00-0.01-0.12 +0.00-0.00-0.00+0.00-0.00-0.00-0.00-0.00+0.00-0.00-0.00+0.00-0.00-0.00 -0.02*-0.01-0.01*+0.04*-0.03*-0.04*+0.00-0.01*+0.01*+0.01-0.17*-0.25*+0.00-0.16 Llama, Deterministic −0.2 −0.1 0.0 0.1 0.2 Signed diff vs hf_raw Figure 3: Signed divergence fromhf_raw(green = backend higher, red = lower) per backend (rows) and task (columns), one panel per model family in Deterministic mode.∗marks a task with at least one BH-significant metric. MMLU TriviaQA-EM TriviaQA-F1 TruthfulQA TQAGen-Rb TQAGen-Rc BBQ-Acc BBQ-Dis BBQ-Unk BBQ-Bias S-LMS S-Inv S-S S-ICAT Metric hf_pipeline langchain_hf vllm ollama Backend -0.00-0.00-0.00-0.00-0.13*-0.16*+0.00+0.00+0.00+0.00+0.00+0.00+0.00+0.00 -0.00-0.00-0.00-0.00-0.13*-0.16*-0.00-0.00-0.00-0.00+0.00+0.00+0.00+0.01 +0.01*-0.01-0.02*-0.04*+0.04*+0.05*-0.01*-0.01-0.02*-0.00+0.01*+0.00-0.00+0.00 +0.01*+0.01*+0.00+0.02*-0.14*-0.18*-0.01*+0.00*-0.03*-0.01+0.01*+0.00+0.00+0.01 Qwen, Default −0.1 0.0 0.1 Signed diff vs hf_raw MMLU TriviaQA-EM TriviaQA-F1 TruthfulQA TQAGen-Rb TQAGen-Rc BBQ-Acc BBQ-Dis BBQ-Unk BBQ-Bias S-LMS S-Inv S-S S-ICAT Metric hf_pipeline langchain_hf vllm ollama Backend -0.00*+0.01*+0.01*+0.00-0.13*-0.17*-0.00*-0.00*-0.00*+0.00+0.00+0.00-0.00+0.00 +0.00+0.01*+0.01*+0.00-0.13*-0.16*-0.00*+0.00-0.00*-0.00+0.00+0.00-0.00-0.00 +0.00*-0.00*-0.00*+0.00-0.01*-0.01-0.00-0.00-0.00-0.00+0.00-0.00-0.00-0.00 +0.01*+0.00+0.00+0.02*-0.14*-0.18*+0.01*+0.02*+0.00*-0.00+0.04*+0.00-0.03+0.01 Gemma, Default −0.1 0.0 0.1 Signed diff vs hf_raw MMLU TriviaQA-EM TriviaQA-F1 TruthfulQA TQAGen-Rb TQAGen-Rc BBQ-Acc BBQ-Dis BBQ-Unk BBQ-Bias S-LMS S-Inv S-S S-ICAT Metric hf_pipeline langchain_hf vllm ollama Backend -0.01*-0.01*-0.01*-0.02-0.17*-0.21*-0.00-0.02*+0.01*+0.00-0.01*-0.00*-0.00-0.01 -0.01*-0.01*-0.01*-0.03*-0.17*-0.21*-0.00-0.02*+0.01*+0.00-0.01*-0.00*-0.01-0.01 -0.04*-0.07*-0.08*-0.13*-0.02*-0.03*-0.01-0.06*+0.05*+0.01-0.07*-0.07*+0.00-0.07 -0.03*-0.03*-0.04*+0.10*-0.21*-0.25*+0.00-0.01*+0.01*+0.00-0.21*-0.30*-0.01-0.21 Llama, Default −0.3 −0.2 −0.1 0.0 0.1 0.2 0.3 Signed diff vs hf_raw Figure 4: Signed divergence fromhf_rawper backend (rows) and task (columns), one panel per model family in Default mode. Divergences are larger than under Deterministic decoding, especially forvllmandollamaon Llama’s factual tasks. hf_pipeline langchain_hf vllm ollama Gemma-3-1B Llama-3.2-1B Qwen2.5-1.5B 0.9820.9760.9800.868 0.6870.5650.6280.611 0.9490.9310.8760.876 Cohen κ vs hf_raw: Model×Backend Deterministic Default Fix Token=512 Gemma-3-1B Llama-3.2-1B Qwen2.5-1.5B 0.9630.9450.9510.948 0.8340.5810.5170.559 0.9270.9000.9040.901 Cohen κ vs hf_raw: Model×Mode Deterministic Default Fix Token=512 hf_pipeline langchain_hf vllm ollama 1.0000.8400.8090.841 0.8190.8400.8050.833 0.9610.7840.7920.775 0.8530.7700.7560.762 Cohen κ vs hf_raw: Backend×Mode Figure 5: Cohen’sκbetween each non-baseline backend andhf_rawfor correct/wrong answer. Values are averaged over all conditions not held constant. sitivity ordering:hf_pipelineachieves perfect agreement withhf_rawacross all models (κ1.0), whilelangchainshows the widest variation (0.81) under deterministic follows ollama (0.85). How ever these results are model dependent as LLAMA- 3.2-1B-INST is the most influential factor in all of these variations where for Langchain the value drops to0.56while for GEMMA-3-1B-IT and QWEN2.5-1.5B-INST these values stays at0.97 and0.93averaging over all variations respectively. Theseκvalues quantify what the mean-divergence heatmap implies: backends do not merely shift scores uniformly; they change the identity of ques- tions answered correctly, and the degree of change is strongly model-dependent. The severity of this per-question rearrangement is shown in the main text (Figure 1, Section 4.2). E Error-Set Overlap (Jaccard Similarity) Jaccard similarity of error sets measures whether backends fail on the same questions ashf_raw, in- dependently of how many questions fail overall. A Jaccard of 1 means the two backends fail on ex- actly the same questions while 0 means completely disjoint failure sets. Figure 6 shows the model×backend compound view. For GEMMA-3-1B-IT, all non-baseline back- ends maintain Jaccard≥ 0.93againsthf_raw, confirming that Gemma’s errors are benchmark- specific and backend-independent. For LLAMA- 3.2-1B-INST, mean Jaccard vs.hf_rawdrops to 0.792. Backends are not merely shifting Llama’s scores they are redirecting success and failure to different questions, a qualitatively different failure mode.hf_pipelineagain achieves the highest Jaccard across all models, whileollamashows the lowest for LLAMA-3.2-1B-INST. The backend× mode panel shows that stochastic modes reduce Jaccard uniformly (more noise introduces more error-set divergence), but the model-dependent or- dering persists in every mode. F Backend Clustering in Score Space To understand the geometric relationship between backends in per-question score space, we repre- sent each backend as a vector of calibrated scores over all questions and apply PCA to reduce to two dimensions. Figure 7 shows one panel per (model, mode) com- bination. For GEMMA-3-1B-IT, all five backends cluster within a compact region in PC1–PC2 space regardless of mode, confirming that they are pro- ducing nearly identical per-question responses. For LLAMA-3.2-1B-INST,ollamaseparates along PC1 from the four HuggingFace-ecosystem back- ends in every mode; under stochastic modes,vllm also separates along PC2. For QWEN2.5-1.5B- INST, the cluster is more diffuse than Gemma’s but tighter than Llama’s. G Response Length as a Potential Confound For generative tasks (TriviaQA, TruthfulQA-Gen), response length is a potential confound: a backend that produces longer responses may achieve higher ROUGE-L scores simply due to increased token overlap, not due to better truthfulness or factual recall. Figure 8 shows mean response length across all tasks. The backend×mode panel reveals that De- fault mode produces the highest length variance across backends (since each backend uses its own defaultmax_new_tokens), while Fix and Deter- ministic modes are length-matched. The model× backend panel shows thatlangchain_hfandvllm tend to produce longer responses thanhf_rawfor LLAMA-3.2-1B-INST and QWEN2.5-1.5B-INST under unconstrained modes, consistent with the positive ROUGE-L divergence seen for those back- ends on TruthfulQA-Gen in Default mode. A Spearman correlation between response length and calibrated score on generative tasks reveals a positive but moderate relationship for TriviaQA and TruthfulQA-Gen: longer responses are as- sociated with higher scores, but the relationship is driven by a few failure cases (very short or truncated outputs) rather than a monotonic length- quality trade-off. Researchers using ROUGE-L as a proxy for generation quality should therefore ensure thatmax_new_tokensis matched across all comparison frameworks before drawing cross- backend conclusions. H Effect Size As additional assurance that the significance tests are not merely minor changes inflated by the large scale of the datasets, we provide an effect-size anal- ysis per model, mode, and backend. Each cell is the mean absolute matched-pairs rank-biserial correla- tion|r|between a backend andhf_raw, averaged hf_pipeline langchain_hf vllm ollama Gemma-3-1B Llama-3.2-1B Qwen2.5-1.5B 0.9870.9820.9860.905 0.8290.7490.8080.781 0.9510.9340.8840.884 Jaccard vs hf_raw: Model×Backend Deterministic Default Fix Token=512 Gemma-3-1B Llama-3.2-1B Qwen2.5-1.5B 0.9730.9600.9650.962 0.8920.7630.7560.755 0.9300.9060.9090.907 Jaccard vs hf_raw: Model×Mode Deterministic Default Fix Token=512 hf_pipeline langchain_hf vllm ollama 1.0000.8970.8940.898 0.8750.8970.8860.894 0.9630.8650.8780.863 0.8890.8470.8470.844 Jaccard vs hf_raw: Backend×Mode Figure 6: Jaccard similarity of the error sets (incorrectly answered questions) between each non-baseline backend andhf_raw, shown as compound heatmaps: model×backend (left), model×mode (centre), backend×mode (right). Higher Jaccard = the same questions fail regardless of backend. 0204060 PC1 (87%) −5 0 5 10 15 20 PC2 (12%) hf_rawhf_pipeline langchain_hf vllm ollama Gemma-3-1B Deterministic −250255075100 PC1 (70%) −20 0 20 40 60 PC2 (29%) hf_rawhf_pipeline langchain_hf vllm ollama Llama-3.2-1B Deterministic −2002040 PC1 (68%) −20 −10 0 10 20 30 PC2 (26%) hf_raw hf_pipeline langchain_hf vllm ollama Qwen2.5-1.5B Deterministic 02040 PC1 (86%) −5 0 5 10 PC2 (6%) hf_raw hf_pipeline langchain_hf vllm ollama Gemma-3-1B Default −40−20020 PC1 (31%) −40 −30 −20 −10 0 10 20 PC2 (28%) hf_raw hf_pipeline langchain_hf vllm ollama Llama-3.2-1B Default −20−100102030 PC1 (62%) −20 −10 0 10 20 PC2 (20%) hf_raw hf_pipeline langchain_hf vllm ollama Qwen2.5-1.5B Default 02040 PC1 (90%) −10.0 −7.5 −5.0 −2.5 0.0 2.5 5.0 PC2 (4%) hf_raw hf_pipeline langchain_hf vllm ollama Gemma-3-1B Fix −2002040 PC1 (30%) −10 0 10 20 30 40 50 PC2 (26%) hf_raw hf_pipeline langchain_hf vllm ollama Llama-3.2-1B Fix −20−100102030 PC1 (60%) −20 −10 0 10 PC2 (16%) hf_raw hf_pipeline langchain_hf vllm ollama Qwen2.5-1.5B Fix 02040 PC1 (87%) −12.5 −10.0 −7.5 −5.0 −2.5 0.0 2.5 5.0 PC2 (5%) hf_raw hf_pipeline langchain_hf vllm ollama Gemma-3-1B Token=512 −2002040 PC1 (32%) −40 −20 0 20 PC2 (28%) hf_raw hf_pipeline langchain_hf vllm ollama Llama-3.2-1B Token=512 −20−100102030 PC1 (63%) −20 −10 0 10 20 PC2 (19%) hf_raw hf_pipeline langchain_hf vllm ollama Qwen2.5-1.5B Token=512 Figure 7: PCA of per-question calibrated score vectors (one vector per backend per model×mode panel). Each point represents one backend projected onto the first two principal components of the question-score matrix. Backends closer together produce more similar per-question behaviour. Variance explained by PC1/PC2 shown on each axis. hf_pipeline hf_raw langchain_hf ollama vllm Gemma-3-1B Llama-3.2-1B Qwen2.5-1.5B 11910139 117101310 111011129 Mean length: Model×Backend Token=512 Default Deterministic Fix Gemma-3-1B Llama-3.2-1B Qwen2.5-1.5B 1491010 1081111 1191011 Mean length: Model×Mode Token=512 Default Deterministic Fix hf_pipeline hf_raw langchain_hf ollama vllm 12101011 1041011 10101110 13141111 1341011 Mean length: Backend×Mode Figure 8: Mean response length (characters) as compound heatmaps: model×backend (left), model×mode (centre), backend× mode (right). over the eleven per-item metrics, for each model, backend, and generation mode. Rank-biserial is the paired effect size of the per-item test (McNemar for binary metrics, Wilcoxon signed-rank for con- tinuous ones) where|r| < 0.1negligible,0.1–0.3 small,0.3–0.5medium,> 0.5large.hf_pipeline shareshf_raw’s in-process engine and is an exact no-op under Deterministic decoding. Because|r| is computed only over items on which a backend disagrees withhf_raw, it captures the direction of the change rather than its volume, and should be read together with the divergence magnitudes in Table 1 I Usage of LLM All the contents of the paper are written and inves- tigated by the authors. We have used Claude code for visualization of the analysis and help as writing tool. BackendDet.FixToken=256Default LLAMA-3.2-1B-INST hf_pipeline0.0000.0210.394 10 0.380 9 langchain_hf0.388 6 0.338 8 0.422 10 0.357 10 vllm0.1120.051 4 0.517 10 0.448 10 ollama0.338 9 0.190 9 0.419 10 0.478 10 GEMMA-3-1B-IT hf_pipeline0.0000.0820.118 5 0.344 8 langchain_hf0.149 2 0.223 4 0.219 5 0.323 6 vllm0.1240.0720.145 1 0.131 4 ollama0.150 4 0.324 10 0.219 5 0.384 8 QWEN2.5-1.5B-INST hf_pipeline0.0360.021 2 0.061 2 0.196 2 langchain_hf0.270 8 0.048 4 0.0280.193 2 vllm0.383 10 0.203 8 0.277 8 0.201 8 ollama0.218 7 0.157 8 0.167 8 0.299 9 Table 7: Effect size of models for each backend and generation mode. A superscript gives the number of the six metrics that are BH-significant (p < 0.05) in that cell.