Paper deep dive
Same Query, Different Answer: Run-to-Run Variance in Semantic Query Processing
Oktie Hassanzadeh, Dharmashankar Subramanian
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 95%
Last extracted: 10/13/2026, 3:07:17 AM
Summary
This paper investigates run-to-run variance in semantic query processing engines that utilize Large Language Models (LLMs). By re-executing benchmarks (SemBench, LRO-Bench, UDA-Bench) and logging 1.2 million model calls, the authors demonstrate that even at temperature 0, outputs vary significantly. A 'possible-worlds' model predicts that query plans amplify item-level variance exponentially based on the number of rows exposed. The study finds that benchmark comparisons are often unstable, and proposes remedies such as voting, calibrated cascades, and recording operator outputs to ensure reproducibility.
Entities (17)
Relation Signals (10)
Dharmashankar Subramanian → affiliatedwith → IBM
confidence 99% · Affiliation: IBM, Yorktown Heights, NY, USA
Oktie Hassanzadeh → affiliatedwith → IBM
confidence 99% · Affiliation: IBM, Yorktown Heights, NY, USA
Semantic Query Engines → uses → LLM
confidence 98% · Semantic query engines ask a large language model (LLM) to judge individual rows
Plans → amplifies → Variance
confidence 95% · Plans amplify these changes exponentially in the number of rows an answer exposes
SemBench → uses → BigQuery
confidence 95% · SemBench’s published results, which contain five executions of every query on LOTUS, Palimpzest, ThalamusDB, and BigQuery.
SemBench → uses → LOTUS
confidence 95% · SemBench’s published results, which contain five executions of every query on LOTUS
SemBench → uses → ThalamusDB
confidence 95% · SemBench’s published results, which contain five executions of every query on LOTUS, Palimpzest, ThalamusDB, and BigQuery.
SemBench → uses → Palimpzest
confidence 95% · SemBench’s published results, which contain five executions of every query on LOTUS, Palimpzest, ThalamusDB, and BigQuery.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Semantic query engines ask a large language model (LLM) to judge individual rows, and benchmarks compare these engines on the quality of a few executions at temperature 0. We measure how much another execution changes. We re-execute the text and tabular workloads of SemBench, LRO-Bench, and UDA-Bench through their own harnesses, log every one of 1.2 million model calls, and reanalyze SemBench's published executions. At temperature 0, between 0.07% and 10.2% of rows change their operator output over five executions, depending on the model, and up to 24.0% on one task. Plans amplify these changes exponentially in the number of rows an answer exposes, as a possible-worlds model predicts: on UDA-Bench, a few percent of extracted values change, but about two thirds of the answers do. In SemBench's published results, 21% of text results change quality between executions, including on engines whose plans are deterministic. Several per-query system comparisons, and 16 of 28 tests of LRO-Bench's implementation recommendations, are not settled by the executions the benchmarks run. Changing rows are rows on which models disagree with each other; identical requests within one execution disagree as often as across executions; and one execution's decision margin or reasoning length predicts which rows change. Voting reduces the variance but does not remove it, and a calibrated cascade keeps its guarantee while the rows it returns change. We derive reporting rules for benchmarks and design rules for engines.
Tags
Links
- Source: https://arxiv.org/abs/2610.13823v1
- Canonical: https://arxiv.org/abs/2610.13823v1
PDF not stored locally. Use the link above to view on the source site.
Full Text
84,578 characters extracted from source content.
Expand or collapse full text
Same Query, Different Answer: Run-to-Run Variance in Semantic Query Processing CCS: Information systems Query optimizationCCS: Information systems Database query processing Oktie Hassanzadeh email: hassanzadeh@us.ibm.com Affiliation: IBM, Yorktown Heights, NY, USA and Dharmashankar Subramanian email: dharmash@us.ibm.com Affiliation: IBM, Yorktown Heights, NY, USA © none Abstract. Semantic query engines ask a large language model (LLM) to judge individual rows, and benchmarks compare these engines on the quality of a few executions at temperature 0. We measure how much another execution changes. We re-execute the text and tabular workloads of SemBench, LRO-Bench, and UDA-Bench through their own harnesses, log every one of 1.2 million model calls, and reanalyze SemBench’s published executions. At temperature 0, between 0.07% and 10.2% of rows change their operator output over five executions, depending on the model, and up to 24.0% on one task. Plans amplify these changes exponentially in the number of rows an answer exposes, as a possible-worlds model predicts: on UDA-Bench, a few percent of extracted values change, but about two thirds of the answers do. In SemBench’s published results, 21% of text results change quality between executions, including on engines whose plans are deterministic. Several per-query system comparisons, and 16 of 28 tests of LRO-Bench’s implementation recommendations, are not settled by the executions the benchmarks run. Changing rows are rows on which models disagree with each other; identical requests within one execution disagree as often as across executions; and one execution’s decision margin or reasoning length predicts which rows change. Voting reduces the variance but does not remove it, and a calibrated cascade keeps its guarantee while the rows it returns change. We derive reporting rules for benchmarks and design rules for engines. Keywords: semantic operators, LLM-powered query processing, benchmarking, reproducibility, nondeterminism 1. Introduction Semantic operators extend relational query processing with predicates and functions that a large language model (LLM) evaluates. A semantic filter, map, join, ranking, or aggregation sends each input row, or each pair of rows, to the model together with a natural-language instruction (Patel et al., 2024; Liu et al., 2024b; Shankar et al., 2024; Liskowski et al., 2025; Dorbani et al., 2025). The filter the review is clearly positive has no definition in the query itself; a row qualifies if the model says it does. Benchmarks for these systems (Lao et al., 2025; Su et al., 2026; Deng et al., 2026) report the quality, cost, and latency of each query on each system and compare systems on these numbers. Optimizers calibrate proxy models and cascades against the model’s outputs (Kang et al., 2020; Zeighami et al., 2025a; Russo et al., 2025), and engines store semantic outputs for later reuse (Dai et al., 2024). The model’s outputs thus become query answers, materialized columns, and the labels optimizers calibrate on, so whether they repeat is a question about the data that query processing produces. All of these uses treat an execution of a semantic query as a property of the system: running the query again returns the same answer. The assumption appears safe because benchmarks execute queries at temperature 0, which makes decoding greedy. For LLMs in general, however, outputs at temperature 0 vary between calls (Ouyang et al., 2023; Song et al., 2025; Atil et al., 2025; He and Thinking Machines Lab, 2025; Yuan et al., 2025). A general-purpose semantic query engine cannot assume deterministic model endpoints. Hosted APIs give no such guarantee (OpenAI, 2025), and the deployment behind an endpoint changes over time; one of ours changed between two of our campaigns (Section 8). Deterministic modes of open-source servers are opt-in, restricted to some GPUs, and cost substantial throughput (He and Thinking Machines Lab, 2025; vLLM Team, 2025; SGLang Team, 2025), and outputs still change with GPU type and count (Yuan et al., 2025). What is not known is what this variation means for query processing: how many outputs change, whether the changes survive the operators above, whether they change benchmark conclusions, what causes them, and what benchmarks and engines can do. Example 1.1. SemBench Movie Q3 counts the positive reviews of one film among 120 candidates (Figure 1). Executed 10 times on gpt-oss-120B with SemBench’s own LOTUS plan and prompt at temperature 0, it returned 4 different counts, and SemBench’s quality metric ranged from 0.64 to 0.93. Llama-3.3-70B returned the same count in every execution on one inference service (quality 0.43), and the same count in every execution on a second stack serving the same weights; but that count was a different one (quality 0.36). Dot plot of the counts returned by repeated executions of one query on six model deployments, with the gold count marked. Figure 1. The count that Movie Q3 returns in each execution at temperature 0, per deployment. Marker labels give the number of executions that returned the count; grey deployments returned one count in every execution.Dot plot of the counts returned by repeated executions of one query on six model deployments, with the gold count marked. This study. We measure run-to-run variance of semantic queries on text and tabular data at two levels. At the benchmark level, we reanalyze SemBench’s published results, which contain five executions of every query on LOTUS, Palimpzest, ThalamusDB, and BigQuery. At the operator level, we re-execute three benchmarks through their own harnesses, without changing plans, prompts, or scoring, and log every model call: SemBench’s LOTUS implementations of the Movie and Medical text queries, including three semantic joins; LRO-Bench’s five LLM-enhanced relational operators in each of their implementations; and UDA-Bench’s pipelines that extract attributes from documents and query them with SQL. The logs give the output of every input row in every execution, which no published benchmark records, and let us attribute each change of an answer to changes of individual operator outputs. We add two commercial models and a temperature control. The central observation is that the plan, not the model alone, decides how much an answer varies: a possible-worlds model predicts that reproduction decays exponentially in the number of items an answer exposes, and the measurements follow it. An item is one unit of work for a semantic operator: a row judged by a filter or map, a pair judged by a join, or one value extracted from one document. A deployment is a model served by a particular inference stack. Contributions. This Experiments & Analysis paper is an experimental analysis with a reproducibility component. Its contributions are insights into existing engines, operator implementations, plans, remedies, and benchmark practices, not new methods: • A possible-worlds model that separates item-level variance from answer reproduction and predicts, per plan operator, how plans amplify item changes (Section 3). • A test of published benchmark results and recommendations against repeated executions (Sections 5 and 7). • Per-item logs of three benchmarks on eight deployments, with an analysis of how changes propagate through plans and what quality metrics hide (Sections 6 and 7), and of which items change and why (Section 8). • A replay of voting and guaranteed proxy cascades on the recorded outputs, and guidelines for benchmark designers and engine builders (Sections 9 and 10). Summary of findings. F1 (Section 5). About one in five published SemBench results changes quality across executions, on every engine and also when the plan is deterministic. Some per-query comparisons between systems need more executions than the benchmark runs; scenario rankings are stable. F2 (Section 6). The share of items that change spans two orders of magnitude across models, commercial ones included, and grows with the number of executions observed. Graded judgments and free text change more than lookups of stated facts. F3 (Section 7). Plans amplify item changes exponentially in the number of items an answer exposes: a few percent of changed extractions change most SQL answers. LIMIT absorbs changes, and quality metrics hide many of the rest; averaged over deployments, only 32% of SemBench queries reproduce every operator output and 48% their answer. Many of LRO-Bench’s implementation recommendations are not settled by its executions. F4 (Section 8). Temperature 0 reduces the variance but does not remove it. The variance belongs to each call: identical requests within one execution disagree as often as across executions. It acts on items close to a tie, which are items models disagree about, and a single execution’s decision margin or reasoning length identifies them. F5 (Section 8). The deployment is part of the result: the same model on two deployments returns different answers, variance, and failure rates, and one deployment shifted between campaigns. None of the client settings or operator implementations we varied removes the variance. F6 (Section 9). Voting reduces but does not remove variance; re-asking only the items a first call flags gets most of its benefit at a fraction of its cost. A cascade’s statistical guarantee survives the oracle’s variance on our queries, but the rows it returns do not. Recording operator outputs is the remedy that makes an answer reproducible. 2. Background and Related Work Semantic query engines and their optimizers. Engines such as LOTUS (Patel et al., 2024), Palimpzest (Liu et al., 2024b), DocETL (Shankar et al., 2024), FlockMTL (Dorbani et al., 2025), Snowflake Cortex (Liskowski et al., 2025), UQE (Dai et al., 2024), Sema (Qi et al., 2026), and iPDB (Kumarasinghe et al., 2026) offer semantic filters, maps, joins, rankings, and aggregations over tables and documents; Evaporate (Arora et al., 2023) extracts structured views from documents. Optimizers place semantic operators in plans (Mang et al., 2026), compile them into operator graphs or deterministic code (Ha et al., 2026; Dong and Wang, 2026), order and estimate predicates (Zhao et al., 2026; Kim et al., 2026), choose listwise ranking and serving strategies (Shin et al., 2026; Son et al., 2026; Sanmartino et al., 2026), and reduce cost with proxies and cascades, several with statistical guarantees (Kang et al., 2020; Zeighami et al., 2025a; Zeighami et al., 2025b; Zhu et al., 2026; Chung et al., 2026; Russo et al., 2025; Liu et al., 2024a). All of them treat the model as an oracle with one answer per item; we measure how far that holds and replay a guaranteed cascade when it does not. Benchmarks and evaluation. SemBench (Lao et al., 2025) fixes queries, data, and ground truth for several scenarios and implements each query natively in each system; LRO-Bench (Su et al., 2026) classifies LLM-enhanced relational operators and compares implementations of each; UDA-Bench (Deng et al., 2026) evaluates document analysis against an annotated relational view. They report one execution or the mean of several, without per-item variance or a test of whether their comparisons are stable. Closest to our work, Alrashed et al. (Alrashed et al., 2026) argue that strict execution equality is the wrong criterion for evaluating generated text-to-SQL queries with AI operators, because their results are stochastic. We hold the query and plan fixed, as the engine benchmarks do, measure the variance of their execution per item and per answer, trace it to its sources, and test which published benchmark conclusions survive it. Database benchmarking has long recognized that results depend on hidden setup choices (Raasveldt et al., 2018; Manolescu et al., 2008); we show that for semantic queries the serving stack and the number of executions are such choices. Nondeterminism at temperature 0. Greedy decoding is deterministic only if the logits are. Inference servers batch concurrent requests dynamically, and batch composition changes the reduction order of floating-point kernels, so the logits of a request depend on the requests that share its batch (He and Thinking Machines Lab, 2025); reduced precision amplifies the effect (Yuan et al., 2025). These fixes act inside the server, at its operator’s cost; we measure what the variation of the endpoints engines use does to query answers. Studies of code generation (Ouyang et al., 2023) and of LLM benchmarks (Song et al., 2025; Atil et al., 2025) report output variation at temperature 0, and evaluation guidance asks for error bars over repeated executions (Miller, 2024; Bouthillier et al., 2021). Self-consistency and semantic entropy use disagreement among samples drawn at higher temperatures as a signal (Wang et al., 2023; Farquhar et al., 2024; Kadavath et al., 2022). These works measure free-form outputs or aggregate accuracy; we measure operator outputs inside query plans, where relational operators combine many small judgments into one answer. Uncertain and crowdsourced data. A semantic operator whose output varies across executions defines a distribution over relations, as in probabilistic databases (Dalvi and Suciu, 2007; Suciu et al., 2011; Benjelloun et al., 2006; Jampani et al., 2008; Huang et al., 2009); our propagation model (Section 3) is the possible-worlds semantics of a tuple-independent database, applied to the question whether two worlds give the same answer. Crowdsourced query processing (Franklin et al., 2011; Marcus et al., 2011; Parameswaran et al., 2012) faced noisy human judgments of rows and pairs and answered them with redundancy, voting, and worker models (Dawid and Skene, 1979); an LLM operator is a worker that disagrees with itself, and we measure what redundancy buys. Storing semantic outputs for reuse is a form of view materialization (Gupta and Mumick, 1995), and recording which model outputs produced an answer is a form of provenance (Cheney et al., 2009). 3. A Possible-Worlds Model and Measures Consider a semantic operator applied to n items. In each execution, item i receives an output YiY_i drawn from a distribution pip_i over possible outputs. An execution is thus a possible world Y=(Y1,…,Yn)Y=(Y_1,…,Y_n), and the plan above the operator maps it to an answer Q(Y)Q(Y). We define the answer reproduction of a query as the probability that two independent executions return the same answer, R=Pr[Q(Y)=Q(Y′)]R= [Q(Y)=Q(Y )], and the disagreement of item i as the probability that two executions give it different outputs, δi=Pr[Yi≠Yi′]=1−∑opi(o)2 _i= [Y_i≠ Y _i]=1- _op_i(o)^2. If items vary independently, Y is a tuple-independent probabilistic relation, and R follows from the pip_i and the plan: • If the answer determines every item’s output, as for a per-row map, a filter returning its full result, or a join returning every match, then R=∏i(1−δi)≈e−nδ¯R= _i(1- _i)≈ e^-n δ. Reproduction decays exponentially in the number of items the answer exposes: with δ¯=0.1% δ=0.1\%, a join over n=1,560n=1,560 pairs reproduces with probability about 0.210.21. • For COUNT, R=∑cPr[C=c]2R= _c [C=c]^2 for the Poisson-binomial count C; opposite changes cancel, so R decreases only with the spread of C: for Boolean items with σ2=∑ipi(1−pi)σ^2= _ip_i(1-p_i) large, R≈1/(2σπ)R≈ 1/(2σ π), which depends on n only through σ. • Under LIMIT k, the answer depends only on the items up to the k-th qualifying row, so n is replaced by that prefix. For other plans, such as a ranking by mean score, we evaluate R by simulating pairs of worlds through the plan. Table 1 lists, for each plan operator in our workloads, which items decide the answer, and how often the prediction falls within 0.15 of the reproduction we measure (Section 7). Table 1. Which items decide the answer under each plan operator, and how often the possible-worlds prediction is within 0.15 of measured reproduction: the closed form e−xe^-x (C) or the simulation through the plan (S). SemBench rows count query–deployment cells; the last row counts UDA-Bench query–deployment pairs. Plan operator Items deciding the answer Within Full output (map, set, join) all n items C 23/24 LIMIT k prefix up to the k-th match S 22/24 COUNT, ratio, counts spread of the count S 18/18 AVG spread of the mean S 6/6 Rank by group mean groups near a tie S 6/6 SQL over extractions values the SQL reads C 202/369 The model also shows why the usual measure of variance is not a property of the operator: the share of items whose output is not identical in k executions, 1−1n∑i∑opi(o)k1- 1n _i _op_i(o)^k, grows with k. Measures. Let yiry_ir be the parsed output of item i in execution r of k; we parse outputs exactly as the benchmark’s runner does, including its default for an unparsable reply. The item flip rate is the share of items whose k outputs are not all identical. Because it grows with k, we compare deployments at k=5k=5; for a deployment with ten executions we report the exact expectation over all subsets of five. The pairwise disagreement estimates δ¯ δ without depending on k, as the mean over items and pairs of executions of [yir≠yis]1[y_ir≠ y_is]. Answer reproduction is the share of pairs of executions whose final answers are identical after canonical ordering; for LRO-Bench and UDA-Bench, with hundreds of queries, we report the share of queries whose answer is not identical in all executions. The quality range is the maximum minus the minimum of the benchmark’s own quality metric across executions. The independence prediction of R draws each item from its empirical output distribution and evaluates the plan on 2,000 simulated pairs of executions. The decision margin is the log-probability gap between the two most likely tokens at the first token that determines the output, taken from the first execution. An item is contested if the majority outputs of the other deployments disagree on it; because it excludes the deployment’s own output, contestedness is a proxy for the ambiguity of the item itself. We report Wilson intervals (Brown et al., 2001) for proportions and percentile bootstrap intervals otherwise. 4. Study Design We ask five research questions. RQ1: How much do published benchmark results change across executions, and do system comparisons change with them? RQ2: For what share of items does an operator’s output change, and how does it depend on the model and the operator? RQ3: How often does the answer change, does the item level predict it, and does the quality metric reveal it? RQ4: Which items change, and which properties of the execution environment matter? RQ5: What do repetition and statistically calibrated plans buy under this variance? 4.1. Workloads We restrict the study to text and tabular data and run each benchmark through its own harness, so that prompts, operators, plans, and scoring are the benchmark’s. We list every deviation. SemBench. For RQ1 we use SemBench’s published results (Lao et al., 2025): five executions with Gemini 2.5 Flash of every query on LOTUS, Palimpzest, ThalamusDB, and BigQuery at the scale factors of SemBench’s cross-system comparison. Headline numbers use the 28 queries whose inputs are only text or tables according to SemBench’s coverage file. For RQ2–RQ5 we execute SemBench’s LOTUS runner on the Movie scenario and on the three text queries of Medical (Table 2). They cover Boolean filters under LIMIT, COUNT, a ratio, a set result, and AVG; a two-way label grouped by label; a 1–5 score per review and per movie followed by a ranking; a 24-way disease extraction; and three semantic self-joins. SemBench keeps every review of the joined film at every scale factor, so the joins (Movie Q5–Q7) require 65,536 comparisons each. We keep a seeded sample of 40 of the film’s 128 distinct reviews, leave every other row unchanged, evaluate SemBench’s gold SQL on the same files, and execute the join exactly (1,560 pairs of distinct reviews) rather than with SemBench’s default cascade, which samples pairs at random and would add algorithmic randomness to the model variance we measure. Table 2. SemBench queries executed repeatedly. Items is the number of semantic-operator outputs analysed per execution; for the joins, on a 40-review sample, it excludes each review paired with itself. Query Output Answer Items Movie Q1 bool first 5 positive reviews 2,000 Movie Q2 bool first 5 qualifying reviews 120 Movie Q3 bool count of positive reviews 120 Movie Q4 bool share of positive reviews 120 Movie Q8 label reviews per sentiment 120 Movie Q9 score score per review 256 Movie Q10 score movies ranked by mean score 2,000 Medical Q1 bool patients with allergy symptoms 1,200 Medical Q4 bool mean age of patients with acne 1,200 Medical Q10 extract disease per symptom text 1,200 Movie Q5 pair first 10 same-sentiment pairs 1,560 Movie Q6 pair first 10 opposite-sentiment pairs 1,560 Movie Q7 pair all opposite-sentiment pairs 1,560 LRO-Bench. LRO-Bench (Su et al., 2026) defines five LLM-enhanced relational operators (select, match, impute, cluster, order) over 27 databases, in single- and multi-operator queries, and implements each operator with one call for the whole operand (ALL), one call per element (ONE), or a hybrid (SEMI). We run its pipelines unchanged on every scored query in each of the 14 operator–implementation settings its scripts evaluate, without chain-of-thought prompting: 735 query–setting combinations per deployment. LRO-Bench retries a failed query and averages up to three valid executions; we record every execution and count an unparsable reply as a failure. We use one fixed judge, gpt-oss-120B, for impute answers so that scores are comparable across deployments. UDA-Bench. UDA-Bench (Deng et al., 2026) poses SQL queries over attributes that must first be extracted from documents. We use its Player dataset (225 Wikipedia documents about players, teams, owners, and cities) and 132 queries covering selection, filtering, aggregation, joins, and mixtures. Nine queries name a column differently from the ground truth and fail even on it, which leaves 123 queries. We follow UDA-Bench’s LOTUS plan: one extraction call per document and attribute with the plan’s instruction to keep values concise, then the query’s SQL over the extracted tables, run and scored lexically by UDA-Bench’s own code. Values are truncated to their first line and cast to the ground-truth type, as in UDA-Bench; a rejected request yields NULL. The 22 queries with an empty ground truth count for reproduction but not for quality. 4.2. Models, serving, and protocol We call the open-weight models through an OpenAI-compatible gateway (Table 3): one reasoning model, three instruction-tuned models with 70–72B parameters, one 24B model, and one 8B model. The gateway serves Llama-3.3-70B from a managed inference service (M) and from a vLLM deployment (V). The two deployments share the model name and the prompt: on all 7,544 requests both received, the prompt token counts are identical. They may still differ in hardware, software, numeric precision, and load, so a difference between them is a difference of deployments. Through a second gateway we run two commercial models, GPT-4o and Claude Sonnet 5, on the SemBench filters and maps. The newest commercial reasoning models we could reach reject a temperature of 0, and older ones accept the parameter without reporting whether it was applied, so neither can be studied at temperature 0. Every execution uses the benchmark’s runner, plan, and prompt at temperature 0. SemBench and UDA-Bench run on LOTUS with its cache disabled and SemBench’s default of 20 concurrent requests; LRO-Bench uses its own client with up to ten concurrent requests and five attempts per call. We checksum the inputs before each execution and disable SemBench’s data generation, so every execution reads identical inputs in identical order. A wrapper around each harness’s model client records a hash of the request and the output of every call, and on LOTUS also the finish reason, token counts, reasoning length, and, where served, the top-5 log-probabilities of the first output tokens; it changes no request except to ask for them. A SemBench execution with a failed call is discarded and rerun; on UDA-Bench, a transport failure is retried for the affected document only. We use LOTUS 1.1.3 and the benchmark versions c814e38 (SemBench), 6eb2930 (LRO-Bench), and f2da981 (UDA-Bench). The main executions ran on 2026-10-02 (SemBench and UDA-Bench) and 2026-10-02 to 2026-10-03 (LRO-Bench); additional executions, the commercial models, and the temperature control ran on 2026-10-09. We call the main executions and each later batch of executions a campaign. The study logged 1,219,338 model calls. Table 3. Model deployments. M is a managed inference service, V a vLLM deployment, and C a commercial API. The last columns give executions per query on SemBench (S, including joins), LRO-Bench (L), and UDA-Bench (U); later executions of some deployments are used only where stated. Model Type Stack Logprobs S L U gpt-oss-120B reasoning M no 10 5 5 Llama-3.3-70B-Instruct instruct M no 5 5 5 Llama-3.3-70B-Instruct instruct V yes 5 5 – Qwen2.5-72B-Instruct instruct V yes 5 – – Mistral-Small-3.1-24B instruct M no 5 – – Granite-3.3-8B-Instruct instruct V yes 5 5 5 GPT-4o instruct C yes 5 – – Claude Sonnet 5 instruct C no 5 – – 5. RQ1: Published Benchmark Results Table 4. SemBench’s published repeats: five executions of each query with Gemini 2.5 Flash, at the scale factor SemBench uses for its repeat tables. A cell is one system on one query. Moves is the share of cells whose quality differs across the five runs. Range is the largest difference between two runs of a cell. Best −- mean is how far the luckiest run sits above the published mean. System Cells Moves Range ≥0.1≥ 0.1 Max range Best −- mean LOTUS 49 24% 6% 0.41 0.010 Palimpzest 64 16% 5% 0.30 0.007 ThalamusDB 47 19% 9% 0.97 0.016 BigQuery 64 27% 8% 1.00 0.035 Text queries 101 21% 5% 0.44 0.007 All queries 224 21% 7% 1.00 0.017 Left: step curves of the share of cells above each quality range, per system. Right: step curves of the share of system pairs needing more than n executions, for SemBench's systems and our deployments. Figure 2. Left: share of SemBench’s published text cells whose quality range over five executions exceeds x. Right: share of pairs of systems on a query whose mean qualities differ and whose comparison needs more than n executions per system to be separated at 95%; dotted line at SemBench’s five.Left: step curves of the share of cells above each quality range, per system. Right: step curves of the share of system pairs needing more than n executions, for SemBench's systems and our deployments. Table 4 summarizes SemBench’s five published executions per query; we call one system on one query a cell. Of the 101 cells on text queries, 21 (21%) report a different quality in at least one execution, 5 by at least 0.1, and up to 0.44 (Figure 2, left). The share ranges from 1 of 27 cells on MMQA to 6 of 11 on Medical. Including image and audio queries, the share is the same (21% of 224), and over every configuration SemBench publishes it is 22% of 1023. Every system is affected, including BigQuery, which calls its model inside the service. These ranges include every source of variance, and some engines add their own: ThalamusDB evaluates a sample of rows, Palimpzest’s optimizer chooses physical operators, and LOTUS runs semantic joins with a sampled cascade. Engine randomness does not explain the result, however. Among cells with a deterministic plan (LOTUS outside its joins, and BigQuery), 12 of 50 change quality; among the others, 9 of 51 do. In 3 cells, none on a text query, one execution returns no rows while the others do, which indicates a failed execution recorded as a result; excluding them, 20% of all cells still change. System comparisons. On text queries, 7 of the 112 pairs of systems whose mean quality on a query differs are ordered the other way by at least one execution; over all queries, 18 of 220, and 2.8% of all single-execution comparisons. Five executions do not always suffice to settle a comparison either. For 12 text pairs at least one system varies; a 95% interval of the difference of their means excludes zero only after a median of 9 executions per system, and 7 pairs need more than SemBench’s five (5 more than ten; Figure 2, right). We compute this number from the observed standard deviations and mean difference of each pair, so it understates the need where five executions happened to show no variance. Scenario-level results, which average over many queries, are stable: in all 6 scenarios, the best system by mean quality is best in every execution, and the full ranking of a single execution matches the ranking of the means in 26 of 29 executions. Finding. About one published cell in five changes quality across executions, on every engine and also when the engine’s plan is deterministic. A per-query comparison drawn from one execution can be reversed by another, and some need more executions than the benchmark runs. Scenario rankings, which average over many queries, are stable, so benchmark designers should report per-query results with their spread. 6. RQ2: How Many Items Change? Published results contain one quality value per execution; our logs contain every operator output. Table 5 reports the item flip rate per SemBench query and deployment. Table 5. Repeated SemBench executions at temperature 0. F: item flip rate (%), the share of items whose output is not identical in five executions (for gpt-oss-120B, the expected rate over five of its ten executions). R: answer reproduction, the share of pairs of executions that return an identical answer (with five executions, ten pairs, so R moves in steps of 0.1; for gpt-oss-120B, 45 pairs of ten). Δ : range of SemBench’s quality metric across executions. Shaded: the answer changes but the quality does not. gpt-oss-120B Llama-70B (M) Llama-70B (V) Qwen-72B Mistral-24B Granite-8B Query F R Δ F R Δ F R Δ F R Δ F R Δ F R Δ Movie Q1 filter, LIMIT 5 4.2 .53 .00 0.1 1 .00 0.1 1 .00 0.1 1 .00 0.1 1 .00 3.2 1 .00 Movie Q2 filter, LIMIT 5 4.4 .62 .20 0.0 1 .00 0.0 1 .00 0.0 1 .00 0.0 1 .00 2.5 1 .00 Movie Q3 filter, COUNT 2.8 .36 .29 0.0 1 .00 0.0 1 .00 0.0 1 .00 0.0 1 .00 2.5 .30 .14 Movie Q4 filter, ratio 3.1 .27 .21 0.0 1 .00 0.0 1 .00 0.0 1 .00 0.0 1 .00 5.0 .10 .21 Movie Q8 label, GROUP BY 3.3 .29 .08 0.0 1 .00 0.0 1 .00 0.0 1 .00 0.0 1 .00 1.7 .40 .04 Movie Q9 score per review 19.4 .00 .03 0.0 1 .00 0.8 .20 .00 0.4 .40 .01 0.0 1 .00 3.1 .00 .01 Movie Q10 score, AVG, rank 17.3 .00 .03 0.7 .10 .00 0.6 .00 .01 0.1 1 .00 0.2 .30 .00 3.1 .00 .01 Medical Q1 filter, set 4.5 .00 .04 0.4 .00 .01 0.0 1 .00 0.0 1 .00 0.0 1 .00 2.7 .00 .04 Medical Q4 filter, AVG(age) 0.8 .00 .02 0.1 .60 .01 0.1 .40 .00 0.1 .60 .01 0.0 1 .00 0.1 .40 .00 Medical Q10 24-way extract 24.0 .00 .03 7.8 .00 .04 0.7 .00 .00 0.0 1 .00 0.1 .60 .00 3.8 .00 .01 The model sets the order of magnitude. Pooled over the ten SemBench filter and map queries, the flip rate over five executions ranges from 0.07% for Mistral-24B to 10.2% (95% CI [9.55%, 10.85%]) for gpt-oss-120B (Table 5). Averaged over queries instead of pooled over items, the range is 0.03% to 8.4%. The reasoning model changes items on every query, and every one of its replies parses under the runner’s rules. The rate is not a fixed property of a deployment: for gpt-oss-120B it grows from 5.2% over two executions to 10.2% over five and 13.0% over ten, without levelling off (Figure 4, left); on 1–5 scores it grows from 8.8% to 22.3%. A few executions therefore reveal only part of the items that can change. The pairwise disagreement, which does not depend on the number of executions, is 5.19% for gpt-oss-120B and 1.38% for Granite-8B. On the joins (Table 8), gpt-oss-120B and Granite-8B change 15.2% and 15.6% of the Q7 verdicts, against at most 0.3% for Mistral-24B, Qwen-72B, and Llama-70B (V). On UDA-Bench (Table 9), gpt-oss-120B changes 7.5% of 2,208 extracted values and Llama-70B (M) 4.0%. 31% of gpt-oss-120B’s changes are between a value and an empty reply; counting only changes between two values, the rates are 6.1%, 3.8%, and 4.5% for gpt-oss-120B, Llama-70B (M), and Granite-8B. Table 6. Item flip rate over five executions, in percent, of the commercial models on the SemBench filters and maps, by output type, next to the lowest and highest rate among the six open-weight deployments. Movie Q1, Medical Q1, Q4, and Q10 are excluded for all deployments: the content filter of GPT-4o rejected every attempt. Output GPT-4o Claude Sonnet 5 Open-weight Boolean filters 0.00 2.22 0.00–3.5 Two-way label 0.83 0.00 0.00–3.3 1–5 scores 9.04 4.65 0.13–17.5 Pooled 7.49 4.13 0.11–15.1 Commercial models vary too. On the same filters and maps, GPT-4o changes 7.5% of items over five executions and Claude Sonnet 5 4.1% (Table 6). Both rates lie inside the range of the open-weight deployments, and for both models the 1–5 scores change most. Their answers are not identical in all executions on 3 and 5 of the 6 queries. The content filter in front of GPT-4o rejected every attempt on Movie Q1, Medical Q1, Q4, and Q10. On Movie Q10, it rejected some executions and passed others with the same prompts. A filter outside the model is thus one more source of executions that differ. Grouped bar chart of item flip rates per model for Boolean, label, score, and extraction outputs. Figure 3. Item flip rate over five executions by the output type of the operator, pooled over SemBench queries (log scale; zero shown at 0.01%).Grouped bar chart of item flip rates per model for Boolean, label, score, and extraction outputs. The task sets the rest. Within a model, the flip rate depends on what the operator asks (Figure 3). For gpt-oss-120B it is 3.4% on Boolean filters and 3.3% on the two-way label. On 1–5 scores it is 17.5%, and on the 24-way diagnosis 24.0%. Llama-70B (M) flips 0.19% of Boolean outputs but 7.8% of extractions. A model that appears deterministic on filters can vary considerably on maps. The size of the output space does not explain this on its own. UDA-Bench’s extractions have open output spaces, yet most values are stable; the attributes that change most are those for which a document mentions several candidates, such as a team’s owners or a city’s population. In a separate control, five executions of gpt-oss-120B on thirteen filter and map tasks of our own over 100 synthetic product reviews, copying a city never changed, whereas a six-way sentiment score changed on 11% of reviews and a free-text summary on 82%. Finding. At temperature 0, the share of items whose output changes over five executions ranges from 0.07% to 10.2%, and to 24.0% on one query, and keeps growing with more executions. The model sets the order of magnitude; within a model, graded judgments and free text change more than lookups of stated facts, whatever the size of the output space. 7. RQ3: From Items to Answers A changed item matters only if it changes the answer. We measure answer reproduction on SemBench’s filters and maps, its joins, LRO-Bench’s operators, and SQL over UDA-Bench’s extractions. 7.1. SemBench filters and maps Left: line plot of flip rate rising with the number of executions for four output types. Right: scatter of reproduction against expected disagreements on a log axis, with an exponential decay curve. Figure 4. Left: item flip rate of gpt-oss-120B against the number of executions observed, by output type (exact expectation over subsets of the ten executions). Right: answer reproduction against the expected number of items on which two executions disagree, one point per SemBench query (joins included) and deployment. Filled points are answers that expose every item, for which the possible-worlds model predicts R≈e−xR≈ e^-x; hollow points are answers under LIMIT, an aggregate, or a ranking.Left: line plot of flip rate rising with the number of executions for four output types. Right: scatter of reproduction against expected disagreements on a log axis, with an exponential decay curve. The plan sets the effect of a change. Answers that depend on every item, such as a per-review score or a set of patients, change whenever any item does (Table 5). On none of the 10 queries does gpt-oss-120B return the same answer in every pair of executions. Rankings are fragile: even Mistral-24B, which flips 0.2% of the Movie Q10 items, reproduces its ranking of movies in only 0.30 of execution pairs, because movies with nearly equal means swap on a single change. Item distributions predict answer stability, with a caveat. Figure 4 (right) places every SemBench cell by x=∑iδix= _i _i, the number of items on which two executions are expected to disagree; cells in which no item changes have x=0x=0 and are omitted. For the 17 cells whose answer exposes every item, the closed form e−xe^-x alone is within 0.15 of measured reproduction in 16 cases (mean error 0.04): one expected disagreement already cuts reproduction to about a third, and no such answer with three or more was reproduced in any pair of executions. 32 of the other 36 cells lie on or above the curve, because LIMIT, aggregates, and rankings absorb some changes. For these plans the full independence prediction, which accounts for the plan, is close to measured reproduction for every SemBench plan operator (Table 1). Nor do changes concentrate in one anomalous execution: in every cell with at least ten deviations from the item’s majority output, the execution with the most deviations accounts for at most 40% of them. A permutation test of the execution labels of each item agrees: of 18 such cells, 2 have p<0.05p<0.05 and 0 have p<0.01p<0.01, as expected if executions are independent. Because this in-sample fit uses the same executions it predicts, we also fit the item distributions of gpt-oss-120B on its first five executions and predict reproduction among the other five. The prediction is within 0.15 on 8 of 10 queries. On Movie Q2, the first five executions show no change among the rows that decide its LIMIT, so the model predicts 1.00, but the later five reproduce in only 0.30 of pairs: rare changes take more executions to observe than to matter. Quality metrics hide answer changes. In the shaded cells of Table 5, the answer changes while quality does not. On Movie Q1, gpt-oss-120B scores 1.00 in all 10 executions, yet only 0.53 of execution pairs return the same five reviews, because any five qualifying reviews earn full credit. When the metric does react, the effect can be large: the quality of Movie Q3 on gpt-oss-120B ranges from 0.64 to 0.93. Changed answers are usually close to each other. On gpt-oss-120B, two executions of Movie Q9 differ in 10.0% of the review scores, and two executions of Medical Q10 in 12.8% of the diagnoses, on average; the movie rankings of Q10 differ by 1−τb=0.081- _b=0.08. Treating the six deployments as the systems of a benchmark, 22 of the 126 deployment pairs whose mean quality on a query differs are reversed by some pair of single executions, and 17 would need more than five executions per deployment to be separated at 95% (Figure 2, right, dashed). Table 7. LRO-Bench, five executions of every scored query in each implementation. Per deployment: queries with a failed execution, answered queries whose answer differs between executions, and the mean range of LRO-Bench’s quality metric over answered queries. ALL issues one call for the operand, ONE one call per element, SEMI a hybrid. gpt-oss-120B (M) Llama-70B (V) Llama-70B (M) Granite-8B (V) Operator Impl. Fail Differ Range Fail Differ Range Fail Differ Range Fail Differ Range Select ALL 0 21/60 0.056 0 7/60 0.008 27 7/33 0.049 12 14/48 0.033 ONE 1 25/60 0.055 6 9/55 0.008 35 2/27 0.002 13 5/47 0.017 Match ALL 0 30/60 0.145 3 9/59 0.021 41 11/26 0.089 17 9/43 0.050 ONE 9 33/55 0.074 20 19/40 0.013 58 2/3 0.077 12 29/49 0.023 SEMI 5 39/57 0.106 0 14/60 0.010 15 28/46 0.057 3 34/58 0.016 Impute ALL 0 52/75 0.121 1 25/74 0.013 24 33/55 0.024 5 29/70 0.006 ONE 0 41/60 0.066 2 26/58 0.012 60 1/1 0.000 30 15/31 0.004 Cluster ALL 0 33/60 0.084 2 9/59 0.029 19 19/47 0.085 7 9/53 0.032 ONE 0 42/60 0.095 0 6/60 0.007 30 14/30 0.026 38 9/25 0.043 Order ALL 0 16/30 0.100 0 3/30 0.034 29 1/2 0.167 27 2/4 0.500 ONE (heap) 0 13/30 0.050 5 2/25 0.040 29 1/1 0.100 20 2/11 0.000 ONE (sort) 0 17/30 0.113 5 8/25 0.042 29 1/1 0.100 21 4/9 0.041 SEMI 0 29/30 0.244 13 4/18 0.011 30 0/0 – 12 10/20 0.058 Multi ALL 1 22/59 0.220 7 28/54 0.130 11 24/51 0.115 60 0/0 – All 16 413/726 0.107 64 169/677 0.026 437 144/323 0.061 277 171/468 0.029 7.2. Semantic joins Table 8. Exact semantic self-joins on 40 reviews (1,560 pairs of distinct reviews). Flip is the share of pairs whose verdict changes across executions (%); Repro is answer reproduction. Q5 and Q6 return the first ten same- and opposite-sentiment pairs, Q7 all opposite-sentiment pairs. Flip rates are over five executions; for gpt-oss-120B, the expected rate over five of its ten. Q5 Q6 Q7 Deployment Flip Repro Flip Repro Flip Repro F1 range Mistral-24B (M) 0.1 1.00 0.0 1.00 0.0 1.00 0.000 Qwen-72B (V) 0.1 1.00 0.2 1.00 0.3 0.10 0.002 Llama-70B (V) 0.3 1.00 0.2 1.00 0.2 0.20 0.001 Llama-70B (M) 2.7 0.10 1.4 0.30 1.5 0.00 0.009 gpt-oss-120B (M) 13.5 0.00 15.1 0.00 15.2 0.00 0.019 Granite-8B (V) 14.8 0.00 14.8 0.00 15.6 0.00 0.027 Table 9. UDA-Bench Player, five executions of 123 queries. Values is the share of extracted (document, attribute) values that change (%). Per query class, the number of queries whose result table differs between executions. F1 is UDA-Bench’s quality, mean over queries with a non-empty ground truth, with the mean range across executions. Answers differ Deployment Values Sel. Filt. Agg. Join Mix. All F1 Range Queries 8 60 10 13 32 123 gpt-oss-120B (M) 7.5 6 44 8 13 13 84 0.44 0.031 Llama-70B (M) 4.0 8 40 10 10 15 83 0.51 0.011 Granite-8B (V) 5.2 8 48 9 6 13 84 0.38 0.021 A join multiplies the items an answer depends on (Table 8). Q7 returns every opposite-sentiment pair among 1,560, so a single changed verdict changes the answer, and only Mistral-24B, which changes none, reproduces it. Qwen-72B and Llama-70B (V) change only 0.3% and 0.2% of the verdicts, yet reproduce in only 0.10 and 0.20 of pairs, close to the independence predictions of 0.16 and 0.15. Q5 and Q6 return only the first ten matches and reproduce in every execution on the same deployments. The F1 score of Q7 barely moves, whereas the precision of Q5’s ten pairs on gpt-oss-120B ranges from 0.20 to 0.70. 7.3. Relational operators and their implementations On LRO-Bench (Table 7), a query is answered by a deployment if at least two of its executions succeed. The answer of gpt-oss-120B differs between executions on 413 of its 726 answered queries (57%), and its quality changes on 343. Every operator is affected: 38% of answered queries for select, 59% for match, 69% for impute, 62% for cluster, 62% for order, and 37% for multi-operator pipelines. The instruction-tuned models change less often (25% for Llama-70B (V), 37% for Granite-8B). Llama-70B (M) fails on most queries (Section 8), so its rate covers a different set of queries. The mean quality range is 0.107 for gpt-oss-120B and 0.026 for Llama-70B (V). Table 10. LRO-Bench’s implementation recommendations on our executions. ✓ : the 95% interval of the mean quality difference over queries is above zero and every execution agrees; ∼ : the interval contains zero; –: fewer than three queries. Recommendation gpt-oss Llama (M) Llama (V) Granite Filter, rows: ONE >> ALL ∼ – ∼ ✓ Filter, columns, tables: ALL >> ONE ✓ ✓ ✓ ✓ Match, rows: SEMI ≥ ALL ∼ – ∼ ∼ Match, cells, columns: ALL >> SEMI ✓ ∼ ∼ ✓ Impute, columns: ONE >> ALL ∼ – ∼ ∼ Impute, cells, rows: ALL >> ONE ✓ – ✓ ∼ Cluster: ALL >> ONE ✓ ✓ ∼ ∼ Order: ALL >> SEMI ✓ – ∼ – Order: ALL >> ONE (heap) ∼ – ∼ – Do LRO-Bench’s recommendations survive? LRO-Bench recommends an implementation per operator and operand type. We test 9 of these recommendations on each deployment (Table 10), comparing the mean quality of two implementations over queries with a bootstrap interval, and checking each execution on its own. Of 28 tests, 12 hold in every execution and none is reversed, but in 16 the interval contains zero. In 6 tests, single executions disagree on which implementation is better, so a single execution could support either conclusion. Only the recommendation of ALL over ONE for filters on columns and tables holds on every deployment. Our models differ from those LRO-Bench used, so this tests the stability of the recommendations, not the published numbers. 7.4. SQL over extracted tables UDA-Bench separates extraction from querying and shows the amplification the model predicts (Table 9). Llama-70B (M) changes 4.0% of the extracted values, yet the result table of 83 of 123 queries (67%) differs between executions; for gpt-oss-120B, 7.5% of values and 68% of answers change. Most queries read one attribute of every document in a table, so one changed value among a hundred suffices. On gpt-oss-120B, the answer changes for most aggregation queries and all 13 join queries. Each query names the attributes its SQL reads, so we can sum the disagreement of every extracted value it reads into an exposure xqx_q, as in Section 3. Because a filter or aggregate may absorb a change, e−xqe^-x_q is a lower bound on reproduction. It is within 0.15 of measured reproduction on 70 of 123 queries for gpt-oss-120B and on every query that only selects attributes (8 of 8); 2 queries reproduce less often than it predicts. Quality changes on 78 queries, but only slightly (mean range 0.031, at most 0.17): a user who reruns a query is far more likely to see a different result table than a different score. 7.5. Three levels of reproducibility Slope chart with three panels, one per benchmark; each line is a deployment rising from outputs to answer to quality. Figure 5. Share of queries whose operator outputs (Out.), final answer (Ans.), and benchmark quality (Qual.) are identical over five executions, per deployment. All levels count the same queries, those with a quality score: on UDA-Bench, 101 of 123; on LRO-Bench, those answered in at least two executions. LRO-Bench compares raw replies; UDA-Bench compares the extracted values each query reads.Slope chart with three panels, one per benchmark; each line is a deployment rising from outputs to answer to quality. Figure 5 summarizes the three benchmarks at three levels, over the same scored queries and five executions. Averaged over deployments, every operator output is identical on 32% of SemBench queries, every raw reply on 44% of LRO-Bench queries (a stricter test, since replies may differ in wording only), and every read value on 2% of UDA-Bench queries. The answer is identical on 48%, 59%, and 19%, and the quality on 52%, 71%, and 33%. Each level hides changes from the next: the gap between outputs and answers is what plans absorb, and the gap between answers and quality is what the metric does not see. The levels also order deployments differently. On SemBench, Llama-70B (M) and (V) reproduce their operator outputs equally often, but (V) reproduces more answers; on UDA-Bench, all three deployments reproduce about a fifth of their answers, but their quality reproduces on between 23% and 42%. A benchmark that reports reproducibility should therefore state the level. Finding. Whether an item change reaches the answer depends on the plan, as a possible-worlds model predicts. Aggregates, rankings, full joins, and SQL over extracted tables turn a few percent of changed items into changes of most answers; LIMIT absorbs many changes, and quality metrics hide many of the rest. Answer stability must be measured on answers. Item distributions predict it per plan operator, but only once enough executions have revealed the rare changes. Implementation recommendations inherit the variance: many of LRO-Bench’s are not settled by its executions. 8. RQ4: Which Items Change, and Why The variance is not sampling at a hidden temperature. A gateway could ignore the requested temperature, or a reasoning model could sample regardless of it. For each of Movie Q3 and Q1, we replayed the recorded requests of 40 items, half that changed in the main executions and half that did not, 20 times each at temperature 0, at temperature 0 with a fixed seed, and at temperature 1, interleaved in time. At temperature 0, the items disagree in 2.5% of pairs of executions; at temperature 1, in 7.5%. On the items that changed in the main executions, the rates are 12.1% and 41.1%. Temperature 0 is therefore applied and reduces the variance, but does not remove it. A fixed seed has no consistent effect: the interval of its difference from temperature 0 is [+0.7, +3.5] points on Q3 and [-7.2, -0.4] on Q1. The gateway returns no system fingerprint that would reveal a change of deployment. Changing items are contested items. On SemBench, 77% of changing items are contested, that is, the majority outputs of the other deployments disagree on them, compared with 34% of stable items. The relation is graded (Figure 6, right): gpt-oss-120B changes 3.2% of the items on which all other deployments agree, but 37.5% of those on which three dissent from the others’ plurality; for Llama-70B (M), the rate rises from 0.19% to 10.7%. Run-to-run variance thus concentrates on items that models disagree about, which suggests that the items themselves are ambiguous. Typical cases are mixed reviews, such as “Taken 3 isn’t a bad time; it’s often laughable without veering totally into self-parody” for the filter clearly positive, and symptoms that fit several of the 24 diseases; on each, the other deployments disagree with one another as well. Variance belongs to the call, not the execution. SemBench’s inputs contain 533 duplicated rows, which LOTUS sends as separate, identical requests within one execution. If the variance came from state that persists through an execution, the two copies would agree within an execution more often than across executions. They do not. On gpt-oss-120B the two copies differ in 6.0% of executions (95% CI 5.4–6.7%). Across executions, the pairwise disagreement is 6.1%. On Granite-8B the two rates are 1.1% and 1.3%, and on the other deployments both are smaller. One Q9 review receives two different scores within the same execution in 6 of 10 executions, and 91 of the 533 duplicated inputs receive two different outputs within at least one execution of gpt-oss-120B, so a single answer can rest on contradictory outputs for the same input. Left: observed share of differing later outputs falling with the decision margin, with fitted curves. Right: flip rates rising with cross-model dissent. Figure 6. Left: share of later executions whose output differs from the first, by the first execution’s decision margin (log scale), with the fitted noise model (dotted), for deployments with log-probabilities and at least ten changing items. Right: item flip rate over five executions against the number of other deployments whose majority output differs from the plurality of the others; with five other deployments, at most three can dissent from a plurality (bins of at least 20 items; zero shown at 0.01%).Left: observed share of differing later outputs falling with the decision margin, with fitted curves. Right: flip rates rising with cross-model dissent. A single execution’s margin predicts later changes. The decision margin of the first execution separates items that change in the later executions from items that do not (Figure 6, left). For Granite-8B the AUC is 0.93 (95% CI [0.90, 0.96], 80 changing items). Llama-70B (V) and Qwen-72B reach 1.00 and 1.00, on only 17 and 5 changing items. gpt-oss-120B’s stack returns no log-probabilities, but its reasoning trace serves the same purpose: the length of the first execution’s trace predicts which items change in the other nine with AUC 0.84 (95% CI [0.83, 0.86], 1,057 changing items). Because traces are longer on some operators than on others, we also compute the AUC within each of the 6 queries with at least ten changing and ten stable items; weighted by changing items it is 0.79 (from 0.73 to 1.00). Nor does contestedness explain it: within items on which zero, one, two, or three other deployments dissent, the AUC is 0.91, 0.77, 0.71, and 0.70. We do not read the length as deliberation. The temperature control above logs full traces of Movie Q3: two executions of an item give identical traces in 23.5% of pairs, and identical traces never give different answers. Items with longer traces have more diverging pairs (rank correlation 0.57). Two traces fork at their first differing token. Pairs whose answers differ fork earlier, at 0.18 of the trace’s length against 0.43 (interval of the difference over items [-0.20, -0.02]; Q1 is alike). A longer trace thus offers more positions to fork, and an early fork changes the answer more often. Either way, a single execution can flag the outputs to store, re-ask, or route elsewhere. One noise scale explains the margins. Suppose each call perturbs an item’s decision margin m by noise of scale σ, so that a later execution differs from the first with probability λ+(1−λ)Φ(−m/σ)λ+(1-λ) (-m/σ) for a small floor λ. Fitted by maximum likelihood, Granite-8B gives σ=0.28σ=0.28 nats (95% CI [0.23, 0.33]) with λ=0.26%λ=0.26\%, and Llama-70B (V) σ=0.37σ=0.37 (Figure 6, left). The noise is small against typical margins: items with a margin below one nat make up 7.8% of Granite-8B’s items but 78% of its changes. Run-to-run variance thus acts on items close to a tie, which is why it concentrates on contested items and why voting cannot remove it (Section 9). The deployment changes the answer, its variance, and whether there is one. With the same model name and prompts, the majority outputs of Llama-3.3-70B on the managed service (M) and on vLLM (V) differ on 83 of 4,736 Movie items (1.8%), whereas only 31 items (0.7%) change within either stack. Each stack is nearly deterministic on Movie, but the two are deterministically different: Movie Q3 scores 0.43 on one and 0.36 on the other in every execution (Figure 1). The stacks also differ in variance: on the 24-way extraction, M flips 7.8% of items and V 0.7%; on join Q5, M 2.7% and V 0.3%. On LRO-Bench, the stack decides whether there is an answer at all: at least one execution fails for 437 of 735 queries (59%) on M but 64 (9%) on V. M’s replies are often not in the form LRO-Bench’s code expects: of its failed executions, 39% precede the JSON with text, 2% wrap it in a tool-call envelope, 42% return JSON of the wrong shape, and 17% return no JSON. A lenient parser could rescue at most the first two groups, so M would still fail far more often than V. These failures are reproducible: only 43 of the 437 failing queries succeed in another execution. The deployment also changes over time. In the main executions, M returned the same output in every execution for all 736 items of Movie Q2–Q4, Q8, and Q9; across these and a later campaign, pairs of executions disagree on 1.9% of items (excess over pairs within a campaign [+0.9, +2.6] points). Granite-8B on V shows no such shift ([-0.1, +0.1]). No operator implementation is stable. Across LRO-Bench’s whole-operand, per-element, and hybrid implementations, the answer of gpt-oss-120B differs on between 35% and 97% of answered queries, and no implementation is the most stable for every operator. The implementation does change how often the other deployments fail (Table 7). Client settings have no detectable effect. Batch composition changes the logits (He and Thinking Machines Lab, 2025), so we varied one factor at a time on Movie Q3, Q8, and Q9, with ten interleaved executions per setting (Table 11): concurrency to Granite-8B, input order, and gpt-oss-120B’s reasoning effort. In all 6 comparisons with the reference, the 95% bootstrap interval of the change in disagreement contains zero. The intervals rule out only large effects, but one request at a time does not make Granite-8B deterministic (disagreement 1.65% on Q3), consistent with a server that batches the requests of all its tenants. Reasoning effort changes what the model answers but not how stably: moving from medium to low or high effort changes the majority output of 14.1% and 10.9% of the Q9 items, more than the within-setting disagreement of 8.82%. Table 11. Cause probes on Movie Q3, Q8, and Q9. Each setting changes one factor. Disagreement is the pairwise item disagreement within the setting (%); moved is the share of items whose majority output differs from the reference setting of the same model (%); repeats in parentheses. Movie Q3 Movie Q8 Movie Q9 Pooled Setting Disagree Moved Disagree Moved Disagree Moved Δ disagree [95% CI] Granite-8B: Concurrency 1 1.65 (10) 0.0 0.17 (10) 0.8 1.68 (10) 0.4 -0.23 [-0.66, +0.18] Granite-8B: Concurrency 8 1.74 (10) 0.8 0.46 (10) 0.8 1.55 (10) 0.4 -0.20 [-0.65, +0.21] Granite-8B: Concurrency 20 (ref.) 2.56 (10) – 0.61 (10) – 1.48 (10) – – Granite-8B: Concurrency 64 1.85 (10) 0.0 0.17 (10) 0.8 1.52 (10) 0.0 -0.26 [-0.62, +0.07] Granite-8B: Shuffled input order 2.65 (10) 0.8 0.46 (10) 0.8 1.40 (10) 0.4 -0.06 [-0.38, +0.26] gpt-oss-120B: Reasoning effort low 1.56 (10) 5.0 0.00 (10) 1.7 7.39 (10) 14.1 -0.97 [-2.28, +0.51] gpt-oss-120B: Reasoning effort medium (ref.) 0.63 (10) – 1.87 (10) – 8.82 (10) – – gpt-oss-120B: Reasoning effort high 1.74 (10) 0.8 1.74 (10) 1.7 7.40 (10) 10.9 -0.49 [-1.73, +0.65] Finding. Changes are not uniform noise, and they are not a property of the execution: identical requests within one execution disagree as often as across executions. They concentrate on items that different models disagree about, and one execution’s decision margin or reasoning length predicts them, because the variance acts on items close to a tie. Temperature 0 reduces the variance but does not remove it. The deployment changes which answer a model gives, how much it varies, and whether it parses, and it can shift between campaigns. None of the settings we varied, which are client concurrency, input order, reasoning effort, a seed, and the operator implementation, removes the variance, so engine builders cannot configure it away. 9. RQ5: What Repetition and Calibration Buy Voting. We replay repetition on the ten recorded SemBench executions of gpt-oss-120B without new model calls, splitting them into disjoint groups and combining each group item by item by majority (median for scores). A vote of three equals execute twice and ask a third time only where the two disagree, at 2+p2+p calls per item for a disagreement share p. Averaged over queries, a vote of three removes 36% of the pairwise disagreement, at most 2.13 calls per item, and a vote of five 48% (Table 12). Neither makes the aggregate, ranking, or Medical answers reproducible: no two combined executions return the same answer. The output distributions explain why. For 48% of changing items, the less common output occurs in at least three of ten executions. With 40 executions of Granite-8B on Movie Q2–Q4, Q8, and Q9, a vote of three removes 33.2% of the disagreement on Boolean items, close to the 33.6% that independent executions with each item’s minority share predict. Voting changes item accuracy against SemBench’s labels by at most 1.8 percentage points on any query, because variance is the smaller part of the error. The majority output is wrong on between 17.9% and 30.7% of labelled items, depending on the deployment, whereas at most 9.9% of labelled items change. For the count of Movie Q3, the variance across executions makes up 7% of gpt-oss-120B’s squared error and nothing for most deployments; the rest is bias. Selective voting. Because the reasoning length of a first call predicts later changes (Section 8), an engine can re-ask only the items it flags. We replay this on three disjoint groups of three executions on the SemBench filters and maps: each item takes the output of the group’s first execution, except the share q of items with the longest first reasoning trace, which take the vote of all three, at 1+2q1+2q calls per item. Pooled over items, a vote of three on every item removes 30% of the disagreement. Re-asking the longest-reasoning 20% of items achieves 64% of that reduction at 1.4 calls per item, and 30% achieves 80% at 1.6; re-asking a random 20% achieves 29%. Table 12. Repetition replayed on ten executions of gpt-oss-120B. Disagreement is the probability that two combined executions differ on an item (%). Vote of 3 costs 2+p2+p calls per item when the third call is made only where the first two disagree. Ten executions form only two groups of five, so Vote 5 rests on a single pair and can exceed Vote 3. Minority ≥3≥ 3 is the share of changing items whose less common output occurs in at least three of the ten executions, so that a vote of three often still returns it. Query One Vote 3 Vote 5 Cost of 3 Minority ≥3≥ 3 Movie Q1 2.1 1.5 1.1 2.02 46% Movie Q2 2.1 1.1 0.8 2.02 29% Movie Q3 1.2 0.6 0.8 2.02 17% Movie Q4 1.4 0.6 0.8 2.02 17% Movie Q8 1.6 1.1 0.0 2.01 40% Movie Q9 9.9 5.7 5.5 2.10 51% Movie Q10 8.7 5.8 5.0 2.09 49% Medical Q1 2.2 1.4 1.2 2.02 43% Medical Q4 0.4 0.4 0.2 2.00 55% Medical Q10 12.8 8.9 8.4 2.13 50% Calibrated cascades. Proxy cascades with guarantees (Kang et al., 2020; Zeighami et al., 2025a) label a sample with the expensive model, the oracle, and choose a threshold on a cheap proxy’s score so that recall or precision with respect to the oracle meets a target with high probability, for one oracle labelling. We replay a recall-target cascade on Medical Q1 and Q4 (1200 rows each), with Granite-8B’s signed first-execution decision margin as the proxy score and gpt-oss-120B or Llama-70B (M) as the oracle. We calibrate for 90% recall with 95% confidence on 400 oracle-labelled rows, once per oracle execution, for 300 samples, and measure recall against every oracle execution (Table 13). On Medical Q1 with gpt-oss-120B as oracle, the same 400 rows labelled by two different oracle executions lead to the same returned rows in only 24% of cases, and the share of calibrations that miss the target rises from 1.0% against the labelling execution to 2.2% against another one. Both stay below the 5% the guarantee allows, because the slack of the confidence bound exceeds the oracle’s disagreement of 2.2%. A precision target was infeasible with this proxy: no threshold met it with confidence, and the cascade returned almost no rows. Table 13. Proxy cascades calibrated for 90% recall with 95% confidence on 400 oracle-labelled rows (300 samples), with Granite-8B as proxy. Dis.: pairwise disagreement of the oracle’s executions. Rows: mean rows returned. Violation: share of calibrations whose recall falls below 90% against the oracle execution that labelled the sample (same) or another one (other). Repro: for a fixed sample, the share of pairs of oracle executions whose calibrations return the same rows. All rates in %. Violation Query Oracle Dis. Rows Same Other Repro Medical Q1 gpt-oss-120B 2.21 321 1.0 2.2 24 Medical Q1 Llama-70B (M) 0.18 350 1.1 1.1 84 Medical Q4 gpt-oss-120B 0.39 1065 0.1 0.2 94 Medical Q4 Llama-70B (M) 0.03 836 0.9 0.9 98 Answers with uncertainty. Instead of one answer, an engine could return what the possible-worlds model implies about all of them (Dalvi and Suciu, 2007), fitted on the first five executions of gpt-oss-120B and tested on the other five. For the count of Movie Q3, the central 95% interval [9, 13] contains every later count but not the gold count of 14: it describes another execution, not the truth. For set answers, the rows returned by all five executions (certain (Imieliński and Lipski, 1984; Libkin, 2016)) and by any (possible) do not bracket later executions: on Medical Q1, each later execution omits on average 1.6 certain patients and adds 3.0 that are not even possible. Five worlds miss the rare changes: of the Granite-8B items unanimous in five executions, 2.2% change in a later one, and only 74/105 later Boolean answers lie between the bounds. Uncertain answers need item probabilities from more executions or from a per-call signal such as the margin. Recording outputs. The remedy that does make an answer reproducible is to record the operator outputs it came from, as response caches do (Bang, 2023), and it is cheap. A stored output takes about 39 bytes, so one gpt-oss-120B execution of the ten SemBench filters and maps produces 348 KB of outputs. Deduplicating identical requests within an execution saves 6.5% of its 8,336 calls and removes the contradictory outputs of Section 8. Token cost varies little between executions: completion tokens vary by a median coefficient of variation of 0.9% across executions, while wall time varies by a median of 20% on gpt-oss-120B, and the slowest execution of a query takes up to 17.0 times as long as the fastest on Llama-70B (V). Finding. Repetition reduces variance but cannot remove it, because many changing items are close to a tie. Re-asking the items with the longest first reasoning trace achieves most of a full vote’s reduction at a fraction of its calls. A calibrated cascade keeps its guarantee under the oracle’s variance on these queries, but which rows it returns depends on which oracle execution labelled its sample. Most error against gold is bias, not variance. An answer is reproducible only if the operator outputs that produced it are recorded, which costs tens of bytes per output. 10. Lessons For benchmark designers. (1) Report per-query quality with its spread over several executions, and choose the number of executions per comparison from the observed variance; five did not suffice for 7 of 112 published text comparisons. (2) Report answer reproduction next to quality. (3) Report flip rates at a stated number of executions, or pairwise disagreement. (4) Do not report the best of several executions, and report failed executions and their rate as a result rather than retrying them away. (5) Record per-item outputs, so that others can attribute variance and replay remedies without model calls. (6) Report the deployment of every open-weight model. (7) State a recommendation between methods only if it holds across executions, not only in one; 6 of our tests of LRO-Bench’s recommendations would go either way on a single execution. For engine builders. (1) Treat a stored semantic column as one sample from a distribution, and store the decision margin or the reasoning length with it. An engine that wants one answer per input must deduplicate calls within an execution, not only across executions. (2) Estimate a plan’s answer stability from item disagreement and the plan’s shape: answers that expose n items reproduce with probability about e−nδ¯e^-n δ, so full joins and per-row outputs need far lower item disagreement than LIMIT queries. (3) Optimizers that calibrate on one oracle execution inherit its variance: the guarantee may survive, but the plan does not. Record the calibration labels with the plan. (4) Choose operator implementations for cost and robustness to reply formats, not for stability. (5) To reproduce an answer, reuse its recorded operator outputs. Voting at about twice the cost removes 36% of the disagreement (averaged over queries), not all of it; if repetition is needed, spend it on the items whose margin or reasoning length flags them. 11. Threats to Validity Models, serving, and engines. We could not run Gemini 2.5 Flash, so RQ2–RQ5 rest on six open-weight deployments and, for the SemBench filters and maps, two commercial models; RQ1 covers Gemini 2.5 Flash and four engines through SemBench’s own results. No reachable model runs with and without reasoning at temperature 0, so we cannot separate reasoning from the model. Per-item logs need a hook into an engine’s model client, which we built for LOTUS and LRO-Bench’s client; other engines enter only through RQ1. We do not control deployment load; probe and control settings are interleaved in time. The deployment effect rests on one model served by two deployments, the only pair available to us. Decision margins exist only for deployments that return log-probabilities. Workloads and replays. The study covers text and tabular data only. The joins use a 40-review sample with an exact plan, and LRO-Bench and UDA-Bench run with the deviations of Section 4.1 on one of UDA-Bench’s datasets. LRO-Bench’s impute scores come from a model judge that may itself vary; in all 2,506 pairs of executions with identical answers, it gave identical scores. The cascade replay fixes the proxy’s scores to one execution and so understates the variance of a real cascade. Number of executions. Five executions bound the smallest flip rates we can detect and give answer reproduction a granularity of 0.1, so we report intervals. Flip rates keep growing: on five Movie queries, Granite-8B flips 3.1% of items at five executions and 5.2% at 40. 12. Conclusion Temperature 0 does not make a semantic query reproducible. Item outputs change at rates set by the model and the operator, plans amplify the changes exponentially in the items an answer exposes, and the changes unsettle per-query comparisons and implementation recommendations. Benchmarks should report quality spreads, answer reproduction, and failure rates; engines that need stable answers should record the operator outputs that produced them. Artifact Availability The drivers, the per-call logs of every execution, and the analysis scripts that produce every number, table, and figure in this paper are available at https://github.com/semviews/sqp-variance-study. References Alrashed et al. (2026) Tarfah Alrashed, Fatma Özcan, Per Jacobsson, Tal Neiman, and Xianshun Chen. 2026. The Stochastic Shift: A New Evaluation Paradigm for Text-to-SQL with AI Operators. arXiv:2609.21133 [cs.DB] https://arxiv.org/abs/2609.21133 Arora et al. (2023) Simran Arora, Brandon Yang, Sabri Eyuboglu, Avanika Narayan, Andrew Hojel, Immanuel Trummer, and Christopher Ré. 2023. Language Models Enable Simple Systems for Generating Structured Views of Heterogeneous Data Lakes. Proceedings of the VLDB Endowment 16, 11 (2023). Atil et al. (2025) Berk Atil, Sarp Aykent, Alexa Chittams, Lisheng Fu, Rebecca J. Passonneau, Evan Radcliffe, Guru Rajan Rajagopal, Adam Sloan, Tomasz Tudrej, Ferhan Ture, Zhe Wu, Lixinyu Xu, and Breck Baldwin. 2025. Non-Determinism of “Deterministic” LLM Settings. arXiv:2408.04667 [cs.CL] https://arxiv.org/abs/2408.04667 Bang (2023) Fu Bang. 2023. GPTCache: An Open-Source Semantic Cache for LLM Applications Enabling Faster Answers and Cost Savings. In Proceedings of the 3rd Workshop for Natural Language Processing Open Source Software (NLP-OSS). 212–218. Benjelloun et al. (2006) Omar Benjelloun, Anish Das Sarma, Alon Halevy, and Jennifer Widom. 2006. ULDBs: Databases with Uncertainty and Lineage. In Proceedings of the 32nd International Conference on Very Large Data Bases (VLDB). 953–964. Bouthillier et al. (2021) Xavier Bouthillier, Pierre Delaunay, Mirko Bronzi, Assya Trofimov, Brennan Nichyporuk, Justin Szeto, Naz Sepah, Edward Raff, Kanika Madan, Vikram Voleti, Samira Ebrahimi Kahou, Vincent Michalski, Dmitriy Serdyuk, Tal Arbel, Chris Pal, Gaël Varoquaux, and Pascal Vincent. 2021. Accounting for Variance in Machine Learning Benchmarks. In Proceedings of Machine Learning and Systems (MLSys). Brown et al. (2001) Lawrence D. Brown, T. Tony Cai, and Anirban DasGupta. 2001. Interval Estimation for a Binomial Proportion. Statist. Sci. 16, 2 (2001), 101–133. Cheney et al. (2009) James Cheney, Laura Chiticariu, and Wang-Chiew Tan. 2009. Provenance in Databases: Why, How, and Where. Foundations and Trends in Databases 1, 4 (2009), 379–474. Chung et al. (2026) Yeounoh Chung, Rushabh Desai, Jian He, Yu Xiao, Thibaud Hottelier, Yves-Laurent Kom Samo, Pushkar Khadilkar, Xianshun Chen, Sam Idicula, Fatma Özcan, Alon Halevy, and Yannis Papakonstantinou. 2026. 100x Cost and Latency Reduction: Performance Analysis of AI Query Approximation using Lightweight Proxy Models. Proceedings of the ACM on Management of Data (2026). doi:10.1145/3802002 Dai et al. (2024) Hanjun Dai, Bethany Yixin Wang, Xingchen Wan, Bo Dai, Sherry Yang, Azade Nova, Pengcheng Yin, Phitchaya Mangpo Phothilimthana, Charles Sutton, and Dale Schuurmans. 2024. UQE: A Query Engine for Unstructured Databases. arXiv:2407.09522 [cs.DB] https://arxiv.org/abs/2407.09522 Dalvi and Suciu (2007) Nilesh Dalvi and Dan Suciu. 2007. Efficient Query Evaluation on Probabilistic Databases. The VLDB Journal 16, 4 (2007), 523–544. Dawid and Skene (1979) A. P. Dawid and A. M. Skene. 1979. Maximum Likelihood Estimation of Observer Error-Rates Using the EM Algorithm. Journal of the Royal Statistical Society: Series C (Applied Statistics) 28, 1 (1979), 20–28. Deng et al. (2026) Qiyan Deng, Jianhui Li, Chengliang Chai, Ye Yuan, Jinqi Liu, Junzhi She, Kaisen Jin, Zhaoze Sun, Yuhao Deng, Jia Yuan, Yuping Wang, Xu Zhou, Guoren Wang, and Lei Cao. 2026. Unstructured Data Analysis Using LLMs: A Comprehensive Benchmark. Proc. VLDB Endow. 19, 9 (2026), 2398–2410. Dong and Wang (2026) Wenkai Dong and Yifan Wang. 2026. From Interpretation to Compilation: A Compilation-Based Execution Engine for Semantic Operator Systems. arXiv:2608.06677 [cs.DB] https://arxiv.org/abs/2608.06677 Dorbani et al. (2025) Anas Dorbani, Sunny Yasser, Jimmy Lin, and Amine Mhedhbi. 2025. Beyond Quacking: Deep Integration of Language Models and RAG into DuckDB. arXiv:2504.01157 [cs.DB] https://arxiv.org/abs/2504.01157 Farquhar et al. (2024) Sebastian Farquhar, Jannik Kossen, Lorenz Kuhn, and Yarin Gal. 2024. Detecting Hallucinations in Large Language Models Using Semantic Entropy. Nature 630 (2024), 625–630. Franklin et al. (2011) Michael J. Franklin, Donald Kossmann, Tim Kraska, Sukriti Ramesh, and Reynold Xin. 2011. CrowdDB: Answering Queries with Crowdsourcing. In Proceedings of the 2011 ACM SIGMOD International Conference on Management of Data. 61–72. Gupta and Mumick (1995) Ashish Gupta and Inderpal Singh Mumick. 1995. Maintenance of Materialized Views: Problems, Techniques, and Applications. IEEE Data Engineering Bulletin 18, 2 (1995), 3–18. Ha et al. (2026) Jaehyun Ha, Yongjoo Park, and Wook-Shin Han. 2026. CADENZA: Compiling Natural-Language Intent into Task-Specific Operator DAGs for Semantic Query Processing. arXiv:2606.29151 [cs.DB] https://arxiv.org/abs/2606.29151 He and Thinking Machines Lab (2025) Horace He and Thinking Machines Lab. 2025. Defeating Nondeterminism in LLM Inference. Thinking Machines Lab: Connectionism. September 10, 2025. https://thinkingmachines.ai/blog/defeating-nondeterminism-in-llm-inference/ Huang et al. (2009) Jiewen Huang, Lyublena Antova, Christoph Koch, and Dan Olteanu. 2009. MayBMS: A Probabilistic Database Management System. In Proceedings of the ACM SIGMOD International Conference on Management of Data. 1071–1074. Imieliński and Lipski (1984) Tomasz Imieliński and Witold Lipski. 1984. Incomplete Information in Relational Databases. J. ACM 31, 4 (1984), 761–791. Jampani et al. (2008) Ravi Jampani, Fei Xu, Mingxi Wu, Luis Leopoldo Perez, Christopher Jermaine, and Peter J. Haas. 2008. MCDB: A Monte Carlo Approach to Managing Uncertain Data. In Proceedings of the 2008 ACM SIGMOD International Conference on Management of Data. 687–700. Kadavath et al. (2022) Saurav Kadavath et al. 2022. Language Models (Mostly) Know What They Know. arXiv:2207.05221 [cs.CL] Kang et al. (2020) Daniel Kang, Edward Gan, Peter Bailis, Tatsunori Hashimoto, and Matei Zaharia. 2020. Approximate Selection with Guarantees using Proxies. Proceedings of the VLDB Endowment 13, 12 (2020), 1990–2003. Kim et al. (2026) Kyoungmin Kim, Martin Catheland, and Anastasia Ailamaki. 2026. Fast LLM-Based Semantic Filtering: From a Unified Framework to an Adaptive Two-Phase Method. arXiv:2606.08090 [cs.DB] https://arxiv.org/abs/2606.08090 Kumarasinghe et al. (2026) Udesh Kumarasinghe, Tyler Liu, Ahmed R. Mahmood, Chunwei Liu, and Walid G. Aref. 2026. iPDB: Optimizing Semantic SQL Queries. arXiv:2601.16432 [cs.DB] https://arxiv.org/abs/2601.16432 Lao et al. (2025) Jiale Lao, Andreas Zimmerer, Olga Ovcharenko, Tianji Cong, Matthew Russo, Gerardo Vitagliano, Michael Cochez, Fatma Özcan, Gautam Gupta, Thibaud Hottelier, H. V. Jagadish, Kris Kissel, Sebastian Schelter, Andreas Kipf, and Immanuel Trummer. 2025. SemBench: A Benchmark for Semantic Query Processing Engines. arXiv:2511.01716 [cs.DB] https://arxiv.org/abs/2511.01716 Libkin (2016) Leonid Libkin. 2016. SQL’s Three-Valued Logic and Certain Answers. ACM Transactions on Database Systems 41, 1 (2016), 1:1–1:28. Liskowski et al. (2025) Paweł Liskowski, Benjamin Han, Paritosh Aggarwal, Bowei Chen, Boxin Jiang, Nitish Jindal, Zihan Li, Aaron Lin, Kyle Schmaus, Jay Tayade, Weicheng Zhao, Anupam Datta, Nathan Wiegand, and Dimitris Tsirogiannis. 2025. Cortex AISQL: A Production SQL Engine for Unstructured Data. arXiv:2511.07663 [cs.DB] https://arxiv.org/abs/2511.07663 Liu et al. (2024b) Chunwei Liu, Matthew Russo, Michael Cafarella, Lei Cao, Peter Baille Chen, Zui Chen, Michael Franklin, Tim Kraska, Samuel Madden, and Gerardo Vitagliano. 2024b. A Declarative System for Optimizing AI Workloads. arXiv:2405.14696 [cs.DB] https://arxiv.org/abs/2405.14696 Liu et al. (2024a) Shu Liu, Asim Biswal, Amog Kamsetty, Audrey Cheng, Luis Gaspar Schroeder, Liana Patel, Shiyi Cao, Xiangxi Mo, Ion Stoica, Joseph E. Gonzalez, and Matei Zaharia. 2024a. Optimizing LLM Queries in Relational Data Analytics Workloads. arXiv:2403.05821 [cs.DB] https://arxiv.org/abs/2403.05821 Mang et al. (2026) Qiuyang Mang, Yufan Xiang, Hangrui Zhou, Runyuan He, Jiaxiang Yu, Hanchen Li, Aditya Parameswaran, and Alvin Cheung. 2026. PLOP: Cost-Based Placement of Semantic Operators in Hybrid Query Plans. arXiv:2604.09944 [cs.DB] https://arxiv.org/abs/2604.09944 Manolescu et al. (2008) Ioana Manolescu, Loredana Afanasiev, Andrei Arion, Jens Dittrich, Stefan Manegold, Neoklis Polyzotis, Karl Schnaitter, Pierre Senellart, Spyros Zoupanos, and Dennis Shasha. 2008. The Repeatability Experiment of SIGMOD 2008. SIGMOD Record 37, 1 (2008), 39–45. Marcus et al. (2011) Adam Marcus, Eugene Wu, David Karger, Samuel Madden, and Robert Miller. 2011. Human-powered Sorts and Joins. Proceedings of the VLDB Endowment 5, 1 (2011), 13–24. Miller (2024) Evan Miller. 2024. Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations. arXiv:2411.00640 [stat.AP] https://arxiv.org/abs/2411.00640 OpenAI (2025) OpenAI. 2025. Create Chat Completion: the seed Parameter. OpenAI API reference. https://developers.openai.com/api/reference/python/resources/chat/subresources/completions/methods/create/ Ouyang et al. (2023) Shuyin Ouyang, Jie M. Zhang, Mark Harman, and Meng Wang. 2023. LLM is Like a Box of Chocolates: The Non-determinism of ChatGPT in Code Generation. arXiv:2308.02828 [cs.SE] https://arxiv.org/abs/2308.02828 Parameswaran et al. (2012) Aditya G. Parameswaran, Hector Garcia-Molina, Hyunjung Park, Neoklis Polyzotis, Aditya Ramesh, and Jennifer Widom. 2012. CrowdScreen: Algorithms for Filtering Data with Humans. In Proceedings of the 2012 ACM SIGMOD International Conference on Management of Data. 361–372. Patel et al. (2024) Liana Patel, Siddharth Jha, Parth Asawa, Melissa Pan, Carlos Guestrin, and Matei Zaharia. 2024. Semantic Operators: A Declarative Model for Rich, AI-based Analytics Over Text Data. arXiv:2407.11418 [cs.DB] https://arxiv.org/abs/2407.11418 Qi et al. (2026) Kangkang Qi, Dongyang Xie, Wenbo Li, Hao Zhang, Yuanyuan Zhu, Jeffrey Xu Yu, and Kangfei Zhao. 2026. Sema: A High-performance System for LLM-based Semantic Query Processing. arXiv:2603.11622 [cs.DB] https://arxiv.org/abs/2603.11622 Raasveldt et al. (2018) Mark Raasveldt, Pedro Holanda, Tim Gubner, and Hannes Mühleisen. 2018. Fair Benchmarking Considered Difficult: Common Pitfalls in Database Performance Testing. In Proceedings of the Workshop on Testing Database Systems (DBTest). 2:1–2:6. Russo et al. (2025) Matthew Russo, Chunwei Liu, Sivaprasad Sudhir, Gerardo Vitagliano, Michael Cafarella, Tim Kraska, and Samuel Madden. 2025. Abacus: A Cost-Based Optimizer for Semantic Operator Systems. arXiv:2505.14661 [cs.DB] https://arxiv.org/abs/2505.14661 Sanmartino et al. (2026) Gabriele Sanmartino, Matthias Urban, Paolo Papotti, and Carsten Binnig. 2026. The Stretto Execution Engine for LLM-Augmented Data Systems. arXiv:2602.04430 [cs.DB] https://arxiv.org/abs/2602.04430 SGLang Team (2025) SGLang Team. 2025. Towards Deterministic Inference in SGLang and Reproducible RL Training. LMSYS Org blog. September 22, 2025. https://w.lmsys.org/blog/2025-09-22-sglang-deterministic/ Shankar et al. (2024) Shreya Shankar, Tristan Chambers, Tarak Shah, Aditya G. Parameswaran, and Eugene Wu. 2024. DocETL: Agentic Query Rewriting and Evaluation for Complex Document Processing. arXiv:2410.12189 [cs.DB] https://arxiv.org/abs/2410.12189 Shin et al. (2026) Jason Shin, Jiwon Chang, and Fatemeh Nargesian. 2026. ListK: Semantic ORDER BY and LIMIT K with Listwise Prompting. arXiv:2603.17223 [cs.DB] https://arxiv.org/abs/2603.17223 Son et al. (2026) Hojae Son, Md Ashraful Islam, Huy Gia Cao, Hui Guan, and Marco Serafini. 2026. Kalypso: Relational LLM Serving. arXiv:2607.23815 [cs.DB] https://arxiv.org/abs/2607.23815 Song et al. (2025) Yifan Song, Guoyin Wang, Sujian Li, and Bill Yuchen Lin. 2025. The Good, The Bad, and The Greedy: Evaluation of LLMs Should Not Ignore Non-Determinism. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics (NAACL). 4195–4206. doi:10.18653/v1/2025.naacl-long.211 Su et al. (2026) Yunxiang Su, Tianjing Zeng, Zhongjun Ding, Yin Lin, Rong Zhu, Zhewei Wei, Bolin Ding, and Jingren Zhou. 2026. Large Language Model-Enhanced Relational Operators: Taxonomy, Benchmark, and Analysis. arXiv:2603.02537 [cs.DB] https://arxiv.org/abs/2603.02537 Suciu et al. (2011) Dan Suciu, Dan Olteanu, Christopher Ré, and Christoph Koch. 2011. Probabilistic Databases. Morgan & Claypool. vLLM Team (2025) vLLM Team. 2025. Batch Invariance. vLLM documentation. https://docs.vllm.ai/en/stable/features/batch_invariance/ Wang et al. (2023) Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V. Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2023. Self-Consistency Improves Chain of Thought Reasoning in Language Models. In International Conference on Learning Representations (ICLR). Yuan et al. (2025) Jiayi Yuan, Hao Li, Xinheng Ding, Wenya Xie, Yu-Jhe Li, Wentian Zhao, Kun Wan, Jing Shi, Xia Hu, and Zirui Liu. 2025. Understanding and Mitigating Numerical Sources of Nondeterminism in LLM Inference. In Advances in Neural Information Processing Systems (NeurIPS). https://arxiv.org/abs/2506.09501 Zeighami et al. (2025a) Sepanta Zeighami, Shreya Shankar, and Aditya Parameswaran. 2025a. Cut Costs, Not Accuracy: LLM-Powered Data Processing with Guarantees. Proceedings of the ACM on Management of Data 3, 6 (2025). Zeighami et al. (2025b) Sepanta Zeighami, Shreya Shankar, and Aditya Parameswaran. 2025b. Featurized-Decomposition Join: Low-Cost Semantic Joins with Guarantees. arXiv:2512.05399 [cs.DB] https://arxiv.org/abs/2512.05399 Zhao et al. (2026) Fuheng Zhao, Pawel Liskowski, Zihan Li, Benjamin Han, Puxuan Yu, Varich Boonsanong, Dimitris Tsirogiannis, and Anupam Datta. 2026. Larch: Learned Query Optimization for Semantic Predicates. arXiv:2606.07923 [cs.DB] https://arxiv.org/abs/2606.07923 Zhu et al. (2026) Yuxuan Zhu, Tengjun Jin, Chenghao Mo, and Daniel Kang. 2026. Accelerating Approximate Analytical Join Queries over Unstructured Data with Statistical Guarantees. Proceedings of the ACM on Management of Data (2026). doi:10.1145/3802004