Paper deep dive

Semantic Bandits: In-Context Exploration-Exploitation is Biased by Semantic Priors

David Eric Austin, Kaheer Suleman, Jackie Chi Kit Cheung

Year: 2026Venue: arXiv preprintArea: cs.CLType: PreprintEmbeddings: 93

Intelligence

Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 94%

Last extracted: 8/23/2026, 2:37:23 AM

Summary

This paper introduces the 'semantic bandit,' an extension of the multi-armed bandit problem, to investigate how Large Language Models (LLMs) balance exploration and exploitation when exposed to semantic priors (inductive biases from pre-training). The study finds that semantically informative action labels (e.g., sentiment, ordinal, world knowledge) significantly bias LLMs toward exploitation, improving performance when aligned with reward structures but degrading it when misaligned. Additionally, negative rewards trigger more exploration than positive ones, suggesting an expected-scale bias. These findings highlight that language-based representations introduce unavoidable biases in LLM decision-making agents.

Entities (11)

LLM · model · 99%Semantic Bandit · concept · 98%Multi-Armed Bandit · problem · 97%Semantic Priors · concept · 96%Cumulative Regret · metric · 95%UCB1 · algorithm · 95%In-Context Reinforcement Learning · task · 94%Action Nomenclature · concept · 93%Gemini 3.1 Flash-Lite · model · 92%OLMo-3.1-32B · model · 92%Qwen3-32b · model · 92%

Relation Signals (10)

Semantic Bandit → extends → Multi-Armed Bandit

confidence 98% · We introduce the semantic bandit, an extension of the multi-armed bandit setting that explicitly considers the textual labels assigned to actions

Semantic Priors → bias → LLM

confidence 95% · semantically informative action labels reduce exploration in favour of exploitation... shape LLM exploration behaviour.

Gemini 3.1 Flash-Lite → evaluatedin → Semantic Bandit

confidence 95% · We evaluate three instruction-tuned LLMs: ... Gemini 3.1 Flash Lite.

Qwen3-32b → evaluatedin → Semantic Bandit

confidence 95% · We evaluate three instruction-tuned LLMs: ... Qwen3-32B ... in our experiments

OLMo-3.1-32B → evaluatedin → Semantic Bandit

confidence 95% · We evaluate three instruction-tuned LLMs: ... OLMo-3.1-32B-Instruct

Action Nomenclature → influences → Exploration Behaviour

confidence 94% · Action nomenclatures — the textual labels assigned to bandit arms — significantly influence exploration behaviour

Negative Rewards → trigger → Exploration

confidence 93% · negative rewards trigger substantially more exploration than equivalent positive rewards

UCB1 → comparewith → LLM

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Large language models (LLMs) are increasingly deployed as decision-making agents in settings that require sophisticated environmental exploration. However, existing work has raised questions about how LLMs actually balance exploration and exploitation. Unlike classical agents, LLM agents engage with tasks through natural language, exposing them to semantic information with no formal counterpart in the task structure. We introduce the semantic bandit, an extension of the multi-armed bandit setting that explicitly considers the textual labels assigned to actions, and use it to study how semantic priors --- inductive biases arising from associations between language and expected reward learned during pre-training, shape LLM exploration behaviour. We find that semantically informative action labels reduce exploration in favour of exploitation, improving performance when aligned with the reward structure and severely degrading it when misaligned. We further find that negative rewards trigger substantially more exploration than equivalent positive rewards, consistent with an expected-scale bias induced by reward conventions common in pre-training data. Overall, we argue that the use of language to define the environment and rewards introduces unavoidable biases derived from the fact that the model is trained on word co-occurence, with implications for the reliability and robustness of LLM agents in real-world decision-making settings.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

90,020 characters extracted from source content.

Expand or collapse full text

Semantic Bandits: In-Context Exploration-Exploitation is Biased by Semantic Priors David Eric Austin Affiliation: Mila – Quebec AI Institute Kaheer Suleman Affiliation: Skyfall AI Jackie Chi Kit Cheung Affiliation: Mila – Quebec AI Institute Affiliation: Canada CIFAR AI Chair, Mila [4pt] McGill University Abstract Large language models (LLMs) are increasingly deployed as decision-making agents in settings that require sophisticated environmental exploration. However, existing work has raised questions about how LLMs actually balance exploration and exploitation. Unlike classical agents, LLM agents engage with tasks through natural language, exposing them to semantic information with no formal counterpart in the task structure. We introduce the semantic bandit, an extension of the multi-armed bandit setting that explicitly considers the textual labels assigned to actions, and use it to study how semantic priors — inductive biases arising from associations between language and expected reward learned during pre-training, shape LLM exploration behaviour. We find that semantically informative action labels reduce exploration in favour of exploitation, improving performance when aligned with the reward structure and severely degrading it when misaligned. We further find that negative rewards trigger substantially more exploration than equivalent positive rewards, consistent with an expected-scale bias induced by reward conventions common in pre-training data. Overall, we argue that the use of language to define the environment and rewards introduces unavoidable biases derived from the fact that the model is trained on word co-occurence, with implications for the reliability and robustness of LLM agents in real-world decision-making settings. 1 Introduction Large language models (LLMs) exhibit sophisticated capabilities on a wide variety of tasks (Bubeck et al. 2023; Srivastava et al. 2023). This has led to widespread deployment of LLM-based decision-making agents in in-context reinforcement learning (ICRL) tasks, which require acting on incomplete information and adapting across sequential turns based on feedback (Shinn et al. 2023; Zhou et al. 2023; Drouin et al. 2024). Any such decision-making agent must balance between exploring new actions to gather information and exploiting past actions known to yield high reward (Sutton & Barto 2020). While prior work (Krishnamurthy et al. 2024; Monea et al. 2024) evaluates ICRL using frameworks from classical reinforcement learning (RL), we argue that these frameworks are incomplete. Classical decision-making agents operate on symbolic problem representations. These representations are deliberately minimal: they preserve the formal structure of the task (states, actions, rewards) and discard everything else. In contrast, LLMs capture additional semantic information beyond the formal task structure. The name assigned to an action (Monea et al. 2024; Min et al. 2022), the magnitude and sign of numeric values (Mirzadeh et al. 2025), and the domain in which a problem is situated (Kambhampati et al. 2024) are all semantically meaningful to a language model in ways they are not to a classical agent. This additional information may reflect genuine regularities in the world or it may consist of inscrutable associations that are not human-interpretable, learned as word co-occurrence statistics during pre-training. We refer to the inductive biases that arise from these learned associations as semantic priors. These biases have no formal counterpart in classical RL. Due to this fundamental difference, evaluating ICRL using frameworks from classical RL leaves out crucial explanatory variables. To address this shortcoming, we introduce the semantic multi-armed bandit (Figure 1), a novel extension of the canonical bandit problem that explicitly parametrizes key aspects of the textual representation alongside the standard reward distributions, and use it to demonstrate how ICRL exploration is biased by semantic priors. Figure 1: LLM exploration in multi-armed bandit problems is systematically biased by associations between language and expected reward learned in pre-training. In this example, the model is biased towards selecting actions with positive-sentiment labels (”divine”). It repeatedly exploits this action when the observed reward is positive (left), missing potentially higher-reward actions with neutral or negative sentiment labels. This label bias is nullified when the observed reward is negative, leading to significantly more exploration. We report three primary findings: • Action nomenclatures — the textual labels assigned to bandit arms — significantly influence exploration behaviour, even when they are not relevant to the task space. When the nomenclature is aligned with the underlying reward structure (e.g., positive-sentiment labels assigned to high-reward arms), models exploit semantic priors to achieve low regret with little exploration. When nomenclature is misaligned, the same priors misdirect the model toward low-reward actions, dramatically increasing cumulative regret. • Reward polarity acts as a strong contextual cue: negative reward values trigger substantially more exploration than positive values, suggesting that LLMs interpret reward not as an abstract numerical signal but relative to an expected scale. • These biases interact — when the reward is negative, models are much less likely to exploit biases from action nomenclature (Figure 1). While the impact of semantic priors on classification and question answering tasks has been established (Jiang et al. 2024; Mirzadeh et al. 2025; Cheng et al. 2025b; Shojaee et al. 2025), their impact on decision-making tasks remains underexplored. Monea et al. 2024 identified that semantic information plays a role in LLM exploration, but to our knowledge we are the first to characterize the potentially misleading semantic biases involved. We argue that the use of language as a representation space in ICRL tasks introduces semantic priors that are not derived from formal problem structure, leading to helpful or harmful effects depending on the alignment between the prior and the true reward distribution. Our most surprising finding is that the influence of these semantic priors is strong enough to nullify principled exploration behaviour in one of the simplest decision-making problems that can be constructed. As LLMs are increasingly deployed in real-world decision settings like recommendation systems and autonomous agents, understanding when semantic context aids adaptation and when it induces systematic errors is critical for safe and reliable deployment. 2 Related work In-context reinforcement learning (ICRL). In-context learning is a paradigm in which pre-trained LLMs solve novel problems given a task description and a few examples in the prompt (Brown et al. 2020; Dong et al. 2024). Min et al. 2022 argue that LLMs do not learn from in-context examples. Rather, the reasoning logic is learned implicitly in pre-training and adapted in-context to match the demonstration format. We focus on in-context reinforcement learning (ICRL), a special class of in-context learning, in which LLMs act as decision-making agents — adapting their performance entirely from in-context reward signal across sequential turns (Krishnamurthy et al. 2024; Monea et al. 2024; Nie et al. 2025). While there is a significant body of work investigating LLM decision-making in application contexts (Shinn et al. 2023; Zhou et al. 2023; Wang et al. 2023), fewer works have attempted to make claims about generalized decision-making concepts like exploration, planning, and generalization. We focus on exploration, which is traditionally assessed through the multi-armed bandit (MAB) problem (Sutton & Barto 2020). In contrast to Pan et al. 2025, we operationalize exploration through arm coverage, switching, and cumulative regret rather model-based analyses that separate directed exploration, random exploration, perseveration, and semantic-prior-driven choice. While some prior work has explored fine-tuning approaches (Nie et al. 2025; Laskin et al. 2022; Lee et al. 2023) for improving exploration behaviour, we follow Krishnamurthy et al. 2024, who focus on identifying failure modes without fine-tuning. We also extend work by Monea et al. 2024, who identify that LLMs are slower to learn in ICRL tasks when semantic labels are removed. We extend this body of work by exploring the relationship between semantic information and ICRL behaviour. Reasoning in LLMs. LLMs have demonstrated impressive performance on reasoning benchmarks, prompting claims of generalized abstract reasoning capabilities (Bubeck et al. 2023; Wei et al.). However, performance on logical and mathematical reasoning tasks remains highly sensitive to structure-preserving linguistic perturbations (Shi et al. 2023; Jiang et al. 2024; Mirzadeh et al. 2025; Tang et al. 2023; Cheng et al. 2025a), potentially indicating probabilistic pattern-matching from immense, closed-source training corpora, rather than principled reasoning (Kambhampati et al. 2024; Jiang et al. 2024; Mirzadeh et al. 2025; Zhang et al. 2023) . Tang et al. 2023 argue that LLM reasoning is fundamentally semantic, unlike in human reasoning, which is related to but distinct from language (Mahowald et al. 2024). Most prior work has focused on classification and question answering task. We extend this by exploring how these effects manifest in ICRL settings. 3 Problem definition Classical reinforcement learning We define an environment as the tuple S,A,R,T\S,A,R,T\, where S is the state space, A is the action set, R:A→Δ⁡(ℝ)R:A→ (R) is the reward function and T is the episode length.11 1 As the MAB is single-state, we simplify the Markov decision process formulation by removing the state transition function and discount factor. Also, Δ denotes the probability simplex. A classical agent operates on a symbolic rendering generated by ρΣ:S×A×T→Σ _ :S× A× T→ , where Σ is an abstract representation space designed to preserve only formal structure. Its policy πΣ:Σ→A _ : → A is invariant to everything outside this structure. Classical multi-armed bandit (MAB) The canonical problem for measuring exploration is the multi-armed bandit (MAB). A MAB instance has a single state and is fully defined by a set of k arms A=[a1,…,ak]A=[a_1,…,a_k], each associated with an unknown reward distribution rir_i. Over T turns, the agent selects an arm and observes the resulting reward, aiming to maximize cumulative reward. Since the distributions are unknown, the agent must balance exploration of the distributions with exploitation of promising arms. LLMs as decision-making agents Unlike the symbolic agent, an LLM-based agent M operates on a textual rendering (Tang et al. 2023) generated by ρV∗:S×A×T×C→V∗ _V^*:S× A× T× C→ V^*, where V∗V^* is the space of all token sequences and C is the set of scenarios. This rendering includes aspects of how the environment is rendered in language as well as additional instructions and prompting details. Its policy πM:V∗→V∗ _M:V^*→ V^* maps a linguistic problem description to a linguistic action. This output is passed to a parsing function ϕ:V∗→Aφ:V^*→ A which converts the linguistic output to a concrete action. Crucially, ρV∗ _V^* and πM _M encode information that is not derived from S,A,R,T\S,A,R,T\, such as the semantic associations about the scenario or between action labels and real-world outcomes. This information is legible to M as semantic priors. Symbolic agents are blind to it by construction; LLM agents are not. This offers a potential advantage for LLM agents when the semantic prior aligns with the true task structure. The semantic bandit While a classical agent engages only with the formal MAB instance, an LLM agent engages with a natural language description of it. This means the problem is no longer fully specified the reward distributions [r1,…,rk][r_1,…,r_k] and T. We introduce the semantic bandit as an extension of the MAB that makes this additional structure explicit. A semantic bandit instance is defined by the standard MAB tuple, an action nomenclature, and a scenario. The action nomenclature is the set of textual labels L=[l1,…,lk]⊂V∗L=[l_1,…,l_k]⊂ V^* assigned to arms [a1,…,ak][a_1,…,a_k]. These labels have no bearing on the formal reward structure, but carry semantic associations that may predispose an LLM toward or away from particular arms. The scenario c∈Cc∈ C is the domain in which the problem is grounded. In the classical MAB, the only information relevant to an optimal policy is the history of (action,reward)(action,reward) pairs. Any influence of semantic details on behaviour therefore constitutes a deviation from normative reward-driven decision-making. 4 Experimental design 4.1 Semantic bandit specification We instantiate our semantic bandit S,A,R,L,c\S,A,R,L,c\ as follows. We define a state space S with just a single state, as is standard for MAB. We define a set of k=3k=3 actions A=[aH,aM,aL]A=[a_H,a_M,a_L], denoting the high, medium, and low reward actions respectively 22 2 For a comparison of results with 3 arms vs 5 arms, see Appendix B.4. Each of these actions has an associated label in L=[lH,lM,lL]L=[l_H,l_M,l_L] and an associated Gaussian reward distribution ri∼⁡(μi,σ2)r_i ( _i,σ^2) with means μH,μM,μL _H, _M, _L respectively and shared variance σ2σ^2. We select c from a set of three scenarios: bandit, farm, and clothing recommendation. We design our experiments to answer two research questions about LLM-based agents deployed on semantic bandit problems: • RQ1: How does action nomenclature impact exploration behaviour and cumulative regret? • RQ2: How does the observed reward value impact exploration behaviour, and are there threshold effects consistent with an expected-scale bias? 4.2 Experimental Conditions 4.2.1 Action nomenclature (RQ1) We evaluate four action nomenclatures, each designed to engage a different type of semantic prior. Table 4 shows examples of each for the farming scenario. The alphanumeric nomenclature uses randomly sampled six-character strings (e.g., f4tjo5) and serves as a control condition with minimal semantic content. The sentiment nomenclature assigns adjectives of positive, neutral, and negative valence to the high-, medium-, and low-reward arms respectively, targeting the positivity bias documented in LLMs (Sharma et al. 2024); labels are sampled at the start of each run from Taboada et al. 2011. The ordinal nomenclature uses explicit rank labels (Highest, Intermediate, Lowest), inducing a direct semantic ordering. The world knowledge nomenclature uses domain-specific labels whose relative value can be inferred from pre-training knowledge. We do not include a world nomenclature for the bandit scenario (see Section 4.3). To disentangle the effect of nomenclature from the effect of reward feedback, we evaluate each nomenclature (besides alphanumeric) in both a helpful configuration — where the semantically favoured label is assigned to the highest-reward arm — and a misleading configuration, where the labels for the highest- and lowest-reward actions are swapped . Mean Reward World Knowledge Ordinal Sentiment Alphanumeric 75 Productive Grassland Highest Phenomenal f4tjo5 50 Rocky Hills Intermediate Typical 88pdak 25 Arid Desert Lowest Repulsive 4y11s9 Table 1: Helpful action nomenclature examples for the High+ reward scale. In the misleading case, the labels for the highest and lowest rewards are swapped. Note that the labels for sentiment and alphanumeric are examples - we randomly sample these labels from a set at the start of each run to reduce the bias from a single label. 4.2.2 Reward scale (RQ2.1) To investigate the influence of observed reward values on exploration, our main experiments contrast two reward conditions: High+, with mean rewards (μH,μM,μL)=(75,50,25)( _H, _M, _L)=(75,50,25), and High-, with (μH,μM,μL)=(−25,−50,−75)( _H, _M, _L)=(-25,-50,-75). We also evaluated a Low+ and Low- nomenclature, where all values are scaled down by a factor of 100. For space, we move these results to Appendix B. We use a shared variance σ2σ^2 across all arms. We evaluate three levels of variance, σ∈12.5,6.25,0σ∈\12.5,6.25,0\, referred to as high, low, and no variance, as a robustness check. 4.2.3 Reward scale sweep (RQ2.2) The main reward experiment does not allow us to isolate the effects of polarity and magnitude. To isolate their effects, we run a scale sweep that varies a single observed reward value r∈[−2,2]r∈[-2,2] in increments of 0.10.1 and measures exploration behaviour as a function of r alone. The protocol is as follows: the model’s first action is forced to the semantically favoured arm, which returns the specified reward r. We then record the number of subsequent turns until the model selects a different arm. This is repeated for each value of r, allowing us to characterize both the polarity threshold (is the model more likely to explore after observing negative values?) and magnitude effects (are there non-monotonic threshold effects at particular reward values?). 4.3 Scenarios We evaluate all conditions across a set of three scenarios C that vary the thematic framing of the prompt: (i) a bandit scenario, in which the agent is explicitly told it is solving a multi-armed bandit problem; (i) a farming scenario, in which the agent selects between agricultural fields to maximise crop yield; and (i) a clothing recommendation scenario, in which the agent recommends clothing items to users. The three scenarios are not independent of task difficulty. In the clothing scenario, stable contextual information — describing user preferences and climate conditions — is provided at the start of each replicate. This context is fixed within a replicate but varies across replicates. We treat the three scenarios as a robustness check on the generality of our findings rather than as a primary experimental variable. 4.4 Models and baselines We evaluate three instruction-tuned LLMs: (i) OLMo-3.1-32B-Instruct, (i) Qwen3-32B33 3 We ran additional experiments with other Qwen3 sizes. See Appendix B.3, and (i) Gemini 3.1 Flash Lite. Qwen3 and Gemini offer native support for thinking mode, while we use CoT prompting for OLMo. We run 10 replicates per condition, resampling action labels and shuffling arm order at the start of each replicate. We compare LLM agents against a classical MAB baseline: UCB1 (Auer et al. 2002) 44 4 UCB observes only scalar rewards, and so is label-agnostic. We ran 10,000 replicates per condition, with arm-to-mean assignments shuffled independently at the start of each replicate.. We adapt our prompt structure from the best-performing prompt identified by Krishnamurthy et al. 2024, with the interaction history summarized as the observed mean reward. Additionally, we experiment with two alternative prompting strategies intended to mitigate bias from the labels: 1) an explicit warning that labels may be misleading (”Be careful about bias in the action labels. Action names may be misleading and are not guaranteed to correlate with reward.”) and 2) the explicit warning plus an explicit instruction to explore (”Make sure to explore”). The full prompt and the prompt variations are provided in Appendix A.2. 4.5 Metrics Cumulative regret We report normalized cumulative regret, defined as the cumulative difference between the reward of the optimal arm and the reward received, normalized to [0,1][0,1]. Lower regret indicates more effective exploration–exploitation behaviour. Let ata_t denote the action chosen by the agent at timestep t, and let a∗=arg⁡maxa∈A⁡r⁡(a)a^*= _a∈ Ar(a) denote the optimal action with the highest expected reward. The instantaneous regret at timestep t is defined as ℓt=r⁡(a∗)−r⁡(at) _t=r(a^*)-r(a_t). The cumulative regret over a horizon of T timesteps is then RT=∑t=1Tℓt=∑t=1T(r⁡(aH)−r⁡(at))R_T= _t=1^T _t= _t=1^T (r(a_H)-r(a_t) ) We normalize by the maximum possible regret to get the normalized cumulative regret: R^T=RT∑t=1T(r⁡(aH)−r⁡(aL)) R_T= R_T _t=1^T (r(a^H)-r(a^L) ). Exploration count We track the number of distinct arms selected across the episode. Since k=3k=3, this value ranges from 1 (no exploration) to 3 (full exploration). We report exploration count per turn to characterize the trajectory of exploration behaviour over the episode rather than just its aggregate. Exploration Probability In the scalesweep experiments, we manually set the interaction history such that at turn 0, the semantically favoured arm was selected. Let this action be a0a_0. We define P⁡(exploration)=P⁡(a1≠a0)P(exploration)=P(a_1≠ a_0), i.e. the probability that the model selects a different arm at turn 1. 4.6 Additional implementation details To account for stochasticity in model outputs, we run 10 replicates of each main experiment configuration. For the the reward scale sweep, we ran 20 replicates per configuration. To reduce variance from any single label choice within a nomenclature, arm order in the prompt is shuffled and labels are resampled at the start of each replicate for sentiment and alphanumeric, as well as for world on the clothing domain. Bandit Farm Clothing Model Nomenclature Helpful Misleading Helpful Misleading Helpful Misleading Qwen3-32B Ordinal 0.00 0.79 0.03 0.71 0.01 0.90 World – – 0.10 0.28 0.04 0.86 Sentiment 0.09 0.29 0.14 0.29 0.10 0.34 Alphanumeric 0.16 – 0.15 – 0.16 – OLMo-3.1 32B Ordinal 0.04 0.79 0.03 0.91 0.00 1.00 World – – 0.01 0.97 0.07 0.91 Sentiment 0.28 0.51 0.43 0.55 0.10 0.90 Alphanumeric 0.29 – 0.47 – 0.47 – Gemini 3.1 Flash Lite Ordinal 0.13 0.27 0.15 0.17 0.15 0.22 World – – 0.15 0.15 0.07 0.56 Sentiment 0.15 0.15 0.15 0.15 0.15 0.15 Alphanumeric 0.15 – 0.15 – 0.15 – UCB1 0.15 – 0.15 – 0.15 – Table 2: Normalized cumulative regret at the final turn, by model and nomenclature type with high reward scale and no variance. We bold the lowest cumulative regret for each model in the helpful case and the highest in the misleading case. These always occur from the same nomenclature. 5 Results Figure 2: Impact of nomenclature on exploration count. Each row reflects a different LLM. Alphanumeric (black) tends to result in normative exploration behaviour (comparable to UCB1 baseline in yellow). In contrast, ordinal, world, and sentiment nomenclatures bias the model towards exploitation of semantically favoured actions. These results are with positive reward, helpful nomenclatures, and no variance. See full results in Appendix B. 5.1 RQ1 - How does action nomenclature impact performance and exploration behaviour? Action nomenclatures can significantly bias the model away from normative reward-driven behaviour. The alphanumeric nomenclature tends to result in exploration behaviour that closely matches the UCB symbolic baseline (Figure 2). In contrast, the LLM tends to exploit semantic biases when the nomenclature is semantically meaningful. Exploiting helpful semantic biases can lead to much lower regret than symbolic methods (Table 2). However, this behaviour can lead to dismal performance if the semantic prior is not aligned with the underlying reward structure. The model has a strong bias in cases where the arms are explicitly ordered (ordinal in Figure 2). There is a weaker but still impactful bias towards words with positive sentiment. The bias from world knowledge varies, but is stronger in the clothing recommendation task than the farm task. Each LLM displayed distinct exploration behaviour (Figure 2). OLMo struggled to explore at all, except when explicitly informed that the task was a MAB. Qwen3 displayed both consistent exploration in the alphanumeric case and pure exploitation in the ordinal case. Gemini tended to explore quickly, matching reward-normative behaviour on most tasks. The notable exception was the world-knowledge nomenclature on the clothing recommendation task. This task was slightly more complex than the rest, since it required additional contextual knowledge in the prompt. This may indicate that the other task-nomenclature pairs are too easy for the Gemini model and it may increasingly rely on semantic priors as task complexity increases. We provide a sample reasoning trace in Appendix B.5 which demonstrates that while semantic priors may be reduced in large frontier reasoning models, they are still present and impact decision-making. Figure 3: Impact of Explicit Debiasing on Exploration Count When the prompt includes both the warning and the instruction to explore, exploration coverage is raised significantly. When the prompt includes only the warning that biases may be misleading, exploration count tends to slightly increase. These results are with helpful nomenclatures, no variance, and high scales. See Appendix A.4 for detailed results. Prompt debiasing interventions can reduce the impact of action nomenclature bias. When the prompt includes both the warning and the instruction to explore, exploration coverage is raised significantly (Figure 3). This is unsurprising. The model no longer needs to determine how to balance between exploration and exploitation, as it has been directly instructed to explore. When the prompt includes only the warning that biases may be misleading, exploration count tends to slightly increase. The impact on cumulative regret depends on whether the nomenclature is helpful or misleading. When the nomenclature is helpful, both the instruction to explore and the warning usually increase cumulative regret. When the nomenclature is misleading, both usually decrease cumulative regret. The strength of this impact is highly inconsistent across domains, nomenclatures, and models. Figure 4: Impact of Reward Scale on Exploration Count We observe more exploration when reward is negative (blue) than positive (brown). These results are with no variance and High scales. See Appendix B for results with variance and with Low scales. 5.2 RQ2 - How does the observed reward value impact exploration behaviour, and are there threshold effects consistent with an expected-scale bias? Observed reward polarity is a major factor in determining model exploration behaviour. Negative rewards lead to significantly more exploration than positive rewards (blue and brown respectively in Figure 4 in body and Figure 9 in the appendix) The exception is when the positive reward scale has already saturated the task and reached full exploration. To disentangle the impact of reward sign from reward magnitude, we performed scale sweep experiments to evaluate the probability of immediate exploration given an observed reward value (Figure 5(a)). We find a strong bias from reward sign — the probability of exploring is much higher for values less than zero. This effect is larger and more sudden for Qwen3 and Olmo than for Gemini. The impact of reward magnitude is unclear and highly non-monotonic. There was weak evidence for other threshold effects around 1.0, but it was inconsistent across models and domains. (a) Averaged across nomenclatures (b) Gemini-Clothing. Figure 5: Probability of exploration at turn 2 given the observed reward at turn 1. We see a significantly more exploration for negative values than for positive values with OLMo and Qwen3 across all domains (left, averaged across nomenclatures). For Gemini, the effect is limited to the world nomenclature on the clothing scenario (right). 6 Discussion LLMs are highly sensitive to surface details, sometimes overriding reward signals in a near-trivial decision-making task. The most important finding in this paper is not that LLMs are influenced by semantic context — that is perhaps unsurprising. It is that this influence is strong enough to completely determine behaviour on one of the simplest decision-making problems that can be constructed: a stationary bandit problem with three arms and no variance. There is no noise to confuse the signal, no complexity to overwhelm the reasoner, no ambiguity about which arm is best once it has been sampled. A classical agent identifies the optimal arm within a handful of turns and never deviates. Yet changing only the textual labels assigned to arms — while holding the formal task structure constant — is sufficient to drive LLM behaviour from near-optimal to near-worst-case. While we did not explore realistic deployment contexts, failure on such a simple synthetic task raises concerns about the stability of exploration in more complex environments. The assumption of semantic alignment in evaluation Our results have important implications for practitioners in environments with potentially misleading semantic biases. Practitioners who evaluate LLM agents exclusively on tasks with naturalistic, semantically coherent labels are measuring a mixture of reward-driven reasoning and semantic prior exploitation, without knowing the proportion of each. This is a concern for the generalizability of evaluation results: a model that scores well because its priors happen to align with the reward structure in the evaluation environment may fail silently in a deployment environment where that alignment does not hold. Through prompt interventions, the model can be steered away from semantic prior exploitation and towards exploration. However, these semantic priors are often desirable and may be the reason for using an LLM over a classical solver. The practitioner must determine whether semantic priors act as a help or a hinderance in their context. 7 Limitations and Future Work A major limitation of our work is the limited number of replicates we performed due to computational constraints. Further experimentation is necessary to make conclusive statistical claims. More limitations arise due to the simplicity of our semantic bandit environment. While it is effective for demonstrating the sensitivity of LLMs to semantic priors in even simple contexts, it also limits the scope of our claims. Future work could evaluate more complex bandit environments, exploring the impact of semantic priors introduced by context in contextual bandits. An important line of future work would be to ground these effects concretely in realistic agentic deployment settings. Finally, our work only considers a few avenues for semantic priors to manifest. There are a number of avenues for future work to extend this, such as the form of numeric rewards (e.g. number of decimal points) or the prompt theme itself. 8 Conclusion We introduced the semantic bandit, a formal extension of the multi-armed bandit that makes explicit the action nomenclature and scenario context through which LLM agents engage with decision tasks. We found that (1) semantically informative labels reduce exploration in favour of exploitation, improving performance under helpful alignment and severely degrading it under misalignment and (2) negative reward values trigger substantially more exploration than formally equivalent positive values, consistent with an expected-scale bias arising from pre-training conventions. Understanding when semantic context aids adaptation and when it induces systematic errors is a prerequisite for reliable deployment of LLM agents in real-world settings. We hope the semantic bandit framework provides a useful tool for future work on the robustness and calibration of LLM decision-making. Ethics Statement The findings of this paper raise concerns about the deployment of LLM-based agents. We identify that semantic biases can be used to manipulate model behaviour. While we present this as a caution to practitioners, we recognize that this could also be used adversarially. Finally, we want to re-iterate that these findings correspond to a small set of experiments in a controlled environment. Practitioners should run evaluations in their own environments to determine how they are impacted. Acknowledgments The authors would like to thank the reviewers and area chairs for their valuable feedback. This work was supported by the Natural Sciences and Engineering Research Council of Canada (NSERC). Jackie Chi Kit Cheung is supported by a Canada CIFAR AI Chair. We acknowledge material support from NVIDIA Corporation in the form of computational resources provided to Mila. We would like to thank Google for providing free Gemini credits as well as the Digital Research Alliance of Canada and Mila for providing additional compute. References Auer et al. (2002) Peter Auer, Nicolò Cesa-Bianchi, and Paul Fischer. Finite-time Analysis of the Multiarmed Bandit Problem. Machine Learning, 47(2):235–256, May 2002. ISSN 1573-0565. doi: 10.1023/A:1013689704352. Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901, 2020. Bubeck et al. (2023) Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, Harsha Nori, Hamid Palangi, Marco Tulio Ribeiro, and Yi Zhang. Sparks of artificial general intelligence: Early experiments with GPT-4. March 2023. Cheng et al. (2025a) Ziling Cheng, Meng Cao, Leila Pishdad, Yanshuai Cao, and Jackie CK Cheung. Can llms reason abstractly over math word problems without cot? disentangling abstract formulation from arithmetic computation. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, p. 14317–14344, 2025a. Cheng et al. (2025b) Ziling Cheng, Meng Cao, Marc-Antoine Rondeau, and Jackie CK Cheung. Stochastic chameleons: Irrelevant context hallucinations reveal class-based (mis)generalization in LLMs. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (eds.), Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 30187–30214, Vienna, Austria, July 2025b. Association for Computational Linguistics. ISBN 979-8-89176-251-0. doi: 10.18653/v1/2025.acl-long.1458. URL https://aclanthology.org/2025.acl-long.1458/. Dong et al. (2024) Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Jingyuan Ma, Rui Li, Heming Xia, Jingjing Xu, Zhiyong Wu, Baobao Chang, Xu Sun, Lei Li, and Zhifang Sui. A survey on in-context learning. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, p. 1107–1128, Miami, Florida, USA, November 2024. Association for Computational Linguistics. doi: 10.18653/v1/2024.emnlp-main.64. URL https://aclanthology.org/2024.emnlp-main.64/. Drouin et al. (2024) Alexandre Drouin, Maxime Gasse, Massimo Caccia, Issam H. Laradji, Manuel Del Verme, Tom Marty, Léo Boisvert, Megh Thakkar, Quentin Cappart, David Vazquez, Nicolas Chapados, and Alexandre Lacoste. WorkArena: How Capable Are Web Agents at Solving Common Knowledge Work Tasks?, 2024. Jiang et al. (2024) Bowen Jiang, Yangxinyu Xie, Zhuoqun Hao, Xiaomeng Wang, Tanwi Mallick, Weijie J Su, Camillo Jose Taylor, and Dan Roth. A peek into token bias: Large language models are not yet genuine reasoners. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, p. 4722–4756, Miami, Florida, USA, November 2024. Association for Computational Linguistics. doi: 10.18653/v1/2024.emnlp-main.272. Kambhampati et al. (2024) Subbarao Kambhampati, Karthik Valmeekam, Lin Guan, Mudit Verma, Kaya Stechly, Siddhant Bhambri, Lucas Saldyt, and Anil Murthy. Position: LLMs can’t plan, but can help planning in LLM-modulo frameworks. In Proceedings of the 41st International Conference on Machine Learning, ICML’24, Vienna, Austria, 2024. JMLR.org. Krishnamurthy et al. (2024) Akshay Krishnamurthy, Keegan Harris, Dylan J. Foster, Cyril Zhang, and Aleksandrs Slivkins. Can large language models explore in-context? In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang (eds.), Advances in Neural Information Processing Systems, volume 37, p. 120124–120158. Curran Associates, Inc., 2024. doi: 10.52202/079017-3818. Laskin et al. (2022) Michael Laskin, Luyu Wang, Junhyuk Oh, Emilio Parisotto, Stephen Spencer, Richie Steigerwald, DJ Strouse, Steven Stenberg Hansen, Angelos Filos, Ethan Brooks, Maxime Gazeau, Himanshu Sahni, Satinder Singh, and Volodymyr Mnih. In-context reinforcement learning with algorithm distillation. In Deep Reinforcement Learning Workshop NeurIPS 2022, 2022. URL https://openreview.net/forum?id=9jsWJfk3xR. Lee et al. (2023) Jonathan Lee, Annie Xie, Aldo Pacchiano, Yash Chandak, Chelsea Finn, Ofir Nachum, and Emma Brunskill. Supervised pretraining can learn in-context reinforcement learning. ArXiv, abs/2306.14892, 2023. URL https://api.semanticscholar.org/CorpusID:259262142. Mahowald et al. (2024) Kyle Mahowald, Anna A. Ivanova, Idan A. Blank, Nancy Kanwisher, Joshua B. Tenenbaum, and Evelina Fedorenko. Dissociating language and thought in large language models. Trends in Cognitive Sciences, 28(6):517–540, 2024. ISSN 1364-6613. doi: 10.1016/j.tics.2024.01.011. Min et al. (2022) Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. Rethinking the role of demonstrations: What makes in-context learning work?, 2022. URL https://arxiv.org/abs/2202.12837. Mirzadeh et al. (2025) Seyed Iman Mirzadeh, Keivan Alizadeh, Hooman Shahrokhi, Oncel Tuzel, Samy Bengio, and Mehrdad Farajtabar. GSM-symbolic: Understanding the limitations of mathematical reasoning in large language models. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id=AjXkRZIvjB. Monea et al. (2024) Giovanni Monea, Antoine Bosselut, Kianté Brantley, and Yoav Artzi. LLMs are in-context bandit reinforcement learners. 2024. Nie et al. (2025) Allen Nie, Yi Su, Bo Chang, Jonathan N. Lee, Ed H. Chi, Quoc V. Le, and Minmin Chen. Evolve: Evaluating and optimizing llms for in-context exploration, 2025. URL https://arxiv.org/abs/2410.06238. Pan et al. (2025) Lan Pan, Hanbo Xie, and Robert Wilson. Large language models think too fast to explore effectively. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025. URL https://openreview.net/forum?id=jW8nBi6y9F. Sharma et al. (2024) Mrinank Sharma, Meg Tong, Tomasz Korbak, David Duvenaud, Amanda Askell, Samuel R. Bowman, Esin DURMUS, Zac Hatfield-Dodds, Scott R Johnston, Shauna M Kravec, Timothy Maxwell, Sam McCandlish, Kamal Ndousse, Oliver Rausch, Nicholas Schiefer, Da Yan, Miranda Zhang, and Ethan Perez. Towards understanding sycophancy in language models. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=tvhaxkMKAn. Shi et al. (2023) Freda Shi, Xinyun Chen, Kanishka Misra, Nathan Scales, David Dohan, Ed H. Chi, Nathanael Scharli, and Denny Zhou. Large language models can be easily distracted by irrelevant context. In International Conference on Machine Learning, 2023. URL https://api.semanticscholar.org/CorpusID:256459776. Shinn et al. (2023) Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: language agents with verbal reinforcement learning. In Proceedings of the 37th International Conference on Neural Information Processing Systems, NIPS ’23, Red Hook, NY, USA, 2023. Curran Associates Inc. Shojaee et al. (2025) Parshin Shojaee, Seyed Iman Mirzadeh, Keivan Alizadeh, Maxwell Horton, Samy Bengio, and Mehrdad Farajtabar. The illusion of thinking: Understanding the strengths and limitations of reasoning models via the lens of problem complexity. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025. URL https://openreview.net/forum?id=YghiOusmvw. Srivastava et al. (2023) Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R. Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, Agnieszka Kluska, Aitor Lewkowycz, Akshat Agarwal, Alethea Power, Alex Ray, Alex Warstadt, Alexander W. Kocurek, Ali Safaya, Ali Tazarv, Alice Xiang, Alicia Parrish, Allen Nie, Aman Hussain, Amanda Askell, Amanda Dsouza, Ambrose Slone, Ameet Rahane, Anantharaman S. Iyer, Anders Johan Andreassen, Andrea Madotto, Andrea Santilli, Andreas Stuhlmüller, Andrew M. Dai, Andrew La, Andrew Kyle Lampinen, Andy Zou, Angela Jiang, Angelica Chen, Anh Vuong, Animesh Gupta, Anna Gottardi, Antonio Norelli, Anu Venkatesh, Arash Gholamidavoodi, Arfa Tabassum, Arul Menezes, Arun Kirubarajan, Asher Mullokandov, Ashish Sabharwal, Austin Herrick, Avia Efrat, Aykut Erdem, Ayla Karakaş, B. Ryan Roberts, Bao Sheng Loe, Barret Zoph, Bartłomiej Bojanowski, Batuhan Özyurt, Behnam Hedayatnia, Behnam Neyshabur, Benjamin Inden, Benno Stein, Berk Ekmekci, Bill Yuchen Lin, Blake Howald, Bryan Orinion, Cameron Diao, Cameron Dour, Catherine Stinson, Cedrick Argueta, Cesar Ferri, Chandan Singh, Charles Rathkopf, Chenlin Meng, Chitta Baral, Chiyu Wu, Chris Callison-Burch, Christopher Waites, Christian Voigt, Christopher D Manning, Christopher Potts, Cindy Ramirez, Clara E. Rivera, Clemencia Siro, Colin Raffel, Courtney Ashcraft, Cristina Garbacea, Damien Sileo, Dan Garrette, Dan Hendrycks, Dan Kilman, Dan Roth, C. Daniel Freeman, Daniel Khashabi, Daniel Levy, Daniel Moseguí González, Danielle Perszyk, Danny Hernandez, Danqi Chen, Daphne Ippolito, Dar Gilboa, David Dohan, David Drakard, David Jurgens, Debajyoti Datta, Deep Ganguli, Denis Emelin, Denis Kleyko, Deniz Yuret, Derek Chen, Derek Tam, Dieuwke Hupkes, Diganta Misra, Dilyar Buzan, Dimitri Coelho Mollo, Diyi Yang, Dong-Ho Lee, Dylan Schrader, Ekaterina Shutova, Ekin Dogus Cubuk, Elad Segal, Eleanor Hagerman, Elizabeth Barnes, Elizabeth Donoway, Ellie Pavlick, Emanuele Rodolà, Emma Lam, Eric Chu, Eric Tang, Erkut Erdem, Ernie Chang, Ethan A Chi, Ethan Dyer, Ethan Jerzak, Ethan Kim, Eunice Engefu Manyasi, Evgenii Zheltonozhskii, Fanyue Xia, Fatemeh Siar, Fernando Martínez-Plumed, Francesca Happé, Francois Chollet, Frieda Rong, Gaurav Mishra, Genta Indra Winata, Gerard de Melo, Germàn Kruszewski, Giambattista Parascandolo, Giorgio Mariani, Gloria Xinyue Wang, Gonzalo Jaimovitch-Lopez, Gregor Betz, Guy Gur-Ari, Hana Galijasevic, Hannah Kim, Hannah Rashkin, Hannaneh Hajishirzi, Harsh Mehta, Hayden Bogar, Henry Francis Anthony Shevlin, Hinrich Schuetze, Hiromu Yakura, Hongming Zhang, Hugh Mee Wong, Ian Ng, Isaac Noble, Jaap Jumelet, Jack Geissinger, Jackson Kernion, Jacob Hilton, Jaehoon Lee, Jaime Fernández Fisac, James B Simon, James Koppel, James Zheng, James Zou, Jan Kocon, Jana Thompson, Janelle Wingfield, Jared Kaplan, Jarema Radom, Jascha Sohl-Dickstein, Jason Phang, Jason Wei, Jason Yosinski, Jekaterina Novikova, Jelle Bosscher, Jennifer Marsh, Jeremy Kim, Jeroen Taal, Jesse Engel, Jesujoba Alabi, Jiacheng Xu, Jiaming Song, Jillian Tang, Joan Waweru, John Burden, John Miller, John U. Balis, Jonathan Batchelder, Jonathan Berant, Jörg Frohberg, Jos Rozen, Jose Hernandez-Orallo, Joseph Boudeman, Joseph Guerr, Joseph Jones, Joshua B. Tenenbaum, Joshua S. Rule, Joyce Chua, Kamil Kanclerz, Karen Livescu, Karl Krauth, Karthik Gopalakrishnan, Katerina Ignatyeva, Katja Markert, Kaustubh Dhole, Kevin Gimpel, Kevin Omondi, Kory Wallace Mathewson, Kristen Chiafullo, Ksenia Shkaruta, Kumar Shridhar, Kyle McDonell, Kyle Richardson, Laria Reynolds, Leo Gao, Li Zhang, Liam Dugan, Lianhui Qin, Lidia Contreras-Ochando, Louis-Philippe Morency, Luca Moschella, Lucas Lam, Lucy Noble, Ludwig Schmidt, Luheng He, Luis Oliveros-Colón, Luke Metz, Lütfi Kerem Senel, Maarten Bosma, Maarten Sap, Maartje Ter Hoeve, Maheen Farooqi, Manaal Faruqui, Mantas Mazeika, Marco Baturan, Marco Marelli, Marco Maru, Maria Jose Ramirez-Quintana, Marie Tolkiehn, Mario Giulianelli, Martha Lewis, Martin Potthast, Matthew L Leavitt, Matthias Hagen, Mátyás Schubert, Medina Orduna Baitemirova, Melody Arnaud, Melvin McElrath, Michael Andrew Yee, Michael Cohen, Michael Gu, Michael Ivanitskiy, Michael Starritt, Michael Strube, Michal Swkedrowski, Michele Bevilacqua, Michihiro Yasunaga, Mihir Kale, Mike Cain, Mimee Xu, Mirac Suzgun, Mitch Walker, Mo Tiwari, Mohit Bansal, Moin Aminnaseri, Mor Geva, Mozhdeh Gheini, Mukund Varma T, Nanyun Peng, Nathan Andrew Chi, Nayeon Lee, Neta Gur-Ari Krakover, Nicholas Cameron, Nicholas Roberts, Nick Doiron, Nicole Martinez, Nikita Nangia, Niklas Deckers, Niklas Muennighoff, Nitish Shirish Keskar, Niveditha S. Iyer, Noah Constant, Noah Fiedel, Nuan Wen, Oliver Zhang, Omar Agha, Omar Elbaghdadi, Omer Levy, Owain Evans, Pablo Antonio Moreno Casares, Parth Doshi, Pascale Fung, Paul Pu Liang, Paul Vicol, Pegah Alipoormolabashi, Peiyuan Liao, Percy Liang, Peter W Chang, Peter Eckersley, Phu Mon Htut, Pinyu Hwang, Piotr Miłkowski, Piyush Patil, Pouya Pezeshkpour, Priti Oli, Qiaozhu Mei, Qing Lyu, Qinlang Chen, Rabin Banjade, Rachel Etta Rudolph, Raefer Gabriel, Rahel Habacker, Ramon Risco, Raphaël Millière, Rhythm Garg, Richard Barnes, Rif A. Saurous, Riku Arakawa, Robbe Raymaekers, Robert Frank, Rohan Sikand, Roman Novak, Roman Sitelew, Ronan Le Bras, Rosanne Liu, Rowan Jacobs, Rui Zhang, Russ Salakhutdinov, Ryan Andrew Chi, Seungjae Ryan Lee, Ryan Stovall, Ryan Teehan, Rylan Yang, Sahib Singh, Saif M. Mohammad, Sajant Anand, Sam Dillavou, Sam Shleifer, Sam Wiseman, Samuel Gruetter, Samuel R. Bowman, Samuel Stern Schoenholz, Sanghyun Han, Sanjeev Kwatra, Sarah A. Rous, Sarik Ghazarian, Sayan Ghosh, Sean Casey, Sebastian Bischoff, Sebastian Gehrmann, Sebastian Schuster, Sepideh Sadeghi, Shadi Hamdan, Sharon Zhou, Shashank Srivastava, Sherry Shi, Shikhar Singh, Shima Asaadi, Shixiang Shane Gu, Shubh Pachchigar, Shubham Toshniwal, Shyam Upadhyay, Shyamolima Shammie Debnath, Siamak Shakeri, Simon Thormeyer, Simone Melzi, Siva Reddy, Sneha Priscilla Makini, Soo-Hwan Lee, Spencer Torene, Sriharsha Hatwar, Stanislas Dehaene, Stefan Divic, Stefano Ermon, Stella Biderman, Stephanie Lin, Stephen Prasad, Steven Piantadosi, Stuart Shieber, Summer Misherghi, Svetlana Kiritchenko, Swaroop Mishra, Tal Linzen, Tal Schuster, Tao Li, Tao Yu, Tariq Ali, Tatsunori Hashimoto, Te-Lin Wu, Théo Desbordes, Theodore Rothschild, Thomas Phan, Tianle Wang, Tiberius Nkinyili, Timo Schick, Timofei Kornev, Titus Tunduny, Tobias Gerstenberg, Trenton Chang, Trishala Neeraj, Tushar Khot, Tyler Shultz, Uri Shaham, Vedant Misra, Vera Demberg, Victoria Nyamai, Vikas Raunak, Vinay Venkatesh Ramasesh, vinay uday prabhu, Vishakh Padmakumar, Vivek Srikumar, William Fedus, William Saunders, William Zhang, Wout Vossen, Xiang Ren, Xiaoyu Tong, Xinran Zhao, Xinyi Wu, Xudong Shen, Yadollah Yaghoobzadeh, Yair Lakretz, Yangqiu Song, Yasaman Bahri, Yejin Choi, Yichi Yang, Sophie Hao, Yifu Chen, Yonatan Belinkov, Yu Hou, Yufang Hou, Yuntao Bai, Zachary Seid, Zhuoye Zhao, Zijian Wang, Zijie J. Wang, Zirui Wang, and Ziyi Wu. Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models. Transactions on Machine Learning Research, 2023. ISSN 2835-8856. Sutton & Barto (2020) Richard S. Sutton and Andrew Barto. Reinforcement Learning: An Introduction. Adaptive Computation and Machine Learning. The MIT Press, Cambridge, Massachusetts London, England, second edition edition, 2020. ISBN 978-0-262-03924-6. Taboada et al. (2011) Maite Taboada, Julian Brooke, Milan Tofiloski, Kimberly Voll, and Manfred Stede. Lexicon-based methods for sentiment analysis. Computational Linguistics, 37(2):267–307, June 2011. doi: 10.1162/COLI˙a˙00049. URL https://aclanthology.org/J11-2001/. Tang et al. (2023) Xiaojuan Tang, Zilong Zheng, Jiaqi Li, Fanxu Meng, Song-Chun Zhu, Yitao Liang, and Muhan Zhang. Large language models are in-context semantic reasoners rather than symbolic reasoners. arXiv preprint arXiv:2305.14825, 2023. Wang et al. (2023) Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Voyager: An Open-Ended Embodied Agent with Large Language Models. In NeurIPS 2023 Foundation Models for Decision Making Workshop, 2023. (28) Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al. Emergent abilities of large language models. Transactions on Machine Learning Research. Zhang et al. (2023) Honghua Zhang, Liunian Harold Li, Tao Meng, Kai-Wei Chang, and Guy Van den Broeck. On the paradox of learning to reason from data. In Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence, p. 3365–3373. International Joint Conferences on Artificial Intelligence Organization, 2023. Zhou et al. (2023) Shuyan Zhou, Frank F. Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Yonatan Bisk, Daniel Fried, Uri Alon, and Graham Neubig. WebArena: A Realistic Web Environment for Building Autonomous Agents. ArXiv, abs/2307.13854, 2023. Appendix A Experimental Design The appendix is organized into two sections. The first section (this one) contains experiment implementation details and experimental results associated with design choices. This includes implementation details (A.1), prompt templates and nomenclature sets(A.2), results using alternative prompting schemes (A.4 and A.3). The second section contains more granular details of the main experimental results in the paper. This includes results using the ”low” reward scale and with added variance to the reward distributions. Results are stable across variance conditions. This section is broken down into cumulative regret results for main experiments (B.1), exploration count (B.2), scaling results with different size Qwen models (B.3), a comparison between a scenario with 3 arms and 5 arms (B.4), and sample reasoning traces (B.5). A.1 Implementation Details To reduce computational overhead, we limit thinking tokens to 150 for OLMo and Qwen; for Gemini, thinking tokens are unrestricted. This allows us to contrast standard open-source models with closed-source frontier reasoning models. All other hyperparameters are set to default HuggingFace values for open-source models and Gemini standards for Gemini. We ran initial evaluations of Llama-3.1-8B-Instruct, Qwen3-8B, and Qwen3-14B. We found that the 8B models did not display sufficient exploration behaviour, while Qwen3-14B performed similarly to Qwen3-32B. We dropped these models in favour of the larger models due to known performance gains at scale (Monea et al. 2024). A.2 Prompt Templates This section contains the prompt templates used in the main experiments. Prompt template: budget_summhist_v1 You will be assigned a task. You will receive the actions you have available, followed by the action history. Your goal is to maximize the total reward across all num_turns steps. We are on step current_turn. Task Assignment: --- task_prompt --- % if context % Current Context: --- - key: value --- % endif % Available Actions: --- - action_suffix Name: action -- This action has been selected count time(s) for an observed mean of obs_mean --- When selecting an action, include only the name of the action. Do not include the dashes used to indicate the list here. Please select an action from the list of action options in order to maximize the reward for your task. When selecting an action, put the exact name of the action between action tags like: <action> action_name </action>. Your thinking budget is thinking_budget tokens. If you exceed this limit, you will be forced to provide an immediate answer. Your overall generation limit is new_token_budget tokens. Figure 6: Prompt template used in all experiments (budget_summhist_v1). red variables are filled at runtime. The % blue % conditional block (context) appears only in the clothing (contextual bandit) condition. Table 3: Task prompts used for each domain. All other template fields are held constant. Domain Task prompt Farming You are a farmer who owns 3 fields. Each action corresponds to planting your crop in a different field. Each field is associated with a Gaussian distribution with a fixed but unknown mean. At each time step, you will select a field and receive a reward drawn from the associated distribution. Your goal is to maximize your reward across all of your turns. Clothing You are a recommendation engine that chooses clothing items to display to users when they visit your webpage. You will be presented with 3 items and a user context. Selecting an item corresponds to recommending it to the user. Each item is associated with a Gaussian distribution with a fixed but unknown mean. At each time step, you will select an item and receive a reward drawn from the associated distribution. Your goal is to maximize your reward across all of your turns. Abstract You are faced with a 3-armed bandit problem. Each action corresponds to pulling a specific arm. Each arm is associated with a Gaussian distribution with a fixed but unknown mean. At each time step, you will select an arm and receive a reward drawn from the associated distribution. Your goal is to select which arm to pull in order to maximize your reward across all of your turns. Table 4: Action name sets across nomenclature conditions. Each condition assigns a name drawn from the high, mid, and low pools to the high-, mid-, and low-reward arms respectively. In the helpful condition this assignment is congruent with arm value; in the mislead condition the high and low pools are swapped. Sentiment pools contain 20 words each; three representative examples are shown. World (Clothing) names are context-dependent: the same item is high-reward in a matching weather context and low-reward in a mismatched one (representative examples from the shirt and jacket categories shown). Type High pool Mid pool Low pool Ordinal “Highest Reward” “Intermediate Reward” “Lowest Reward” Sentiment excellent, fabulous, standard, typical, awful, terrible, outstanding functional deplorable World (Farm) Productive Grassland Rocky Hills Arid Desert World (Clothing) warm context tank top, midweight t-shirt, flannel long-sleeve, windbreaker leather jacket heavy parka World (Clothing) cold context flannel long-sleeve, midweight t-shirt, tank top, heavy parka leather jacket windbreaker A.3 Alternative History Passing Formats History Nomenclature Bandit Regret H Bandit Regret M Bandit Explor H Bandit Explor M Farm Regret H Farm Regret M Farm Explor H Farm Explor M Full Ordinal 0.01 1.00 1.20 1.00 0.01 0.95 1.10 1.10 World – – – – 0.07 0.82 1.80 1.40 Sentiment 0.27 0.62 1.30 1.30 0.29 0.35 2.10 2.40 Alphanumeric 0.23 – 2.20 – 0.38 – 2.00 – Summarized Ordinal 0.00 0.79 1.00 1.70 0.03 0.71 1.60 2.10 World – – – – 0.10 0.28 2.50 2.90 Sentiment 0.27 0.35 1.90 2.30 0.17 0.33 2.40 2.40 Alphanumeric 0.16 – 3.00 – 0.15 – 3.00 – Table 5: Full vs. summarized history comparison at turn @9 for Qwen3-32B. Summarized history is presenting the observed mean reward and number of pulls for each arm. Full history is the raw action-reward history. Overall, we found that summarized history performed better. A.4 Additional Results from Prompting with Explicit Debiasing Bandit Farm Clothing Helpful Misleading Helpful Misleading Helpful Misleading Model Warn. Instr. Nomenclature @2 @9 @2 @9 @2 @9 @2 @9 @2 @9 @2 @9 Qwen3-32B – – Ordinal 1.00 1.00 1.10 1.70 1.00 1.60 1.20 2.10 1.20 1.20 1.10 1.30 – 1.50 2.00 2.10 2.40 1.60 2.30 2.00 2.80 1.20 1.30 1.40 1.60 2.50 3.00 2.60 3.00 2.70 3.00 2.90 3.00 2.80 3.00 2.80 3.00 – – World – – – – 1.90 2.50 2.00 2.90 1.30 1.70 1.20 1.40 – – – – – 1.60 2.50 1.90 2.60 1.50 1.80 1.20 1.80 – – – – 2.80 3.00 2.40 3.00 2.40 2.90 2.10 2.50 – – Sentiment 1.90 2.20 2.40 2.80 2.10 2.40 2.60 2.70 2.00 2.20 2.30 2.40 – 1.90 2.30 2.00 2.40 2.20 2.60 2.20 2.80 2.10 2.20 2.30 2.60 2.80 2.90 2.50 2.90 2.80 2.90 2.80 3.00 2.40 2.80 2.70 2.80 – – Alphanumeric 2.80 3.00 – – 2.70 3.00 – – 2.60 3.00 – – – 2.70 2.80 – – 2.40 2.80 – – 2.20 2.80 – – 2.90 3.00 – – 2.80 3.00 – – 3.00 3.00 – – OLMo-3.1 32B – – Ordinal 1.00 1.60 1.50 1.50 1.30 1.50 1.10 1.10 1.00 1.00 1.00 1.00 – 1.70 2.50 1.50 2.00 1.60 1.90 1.70 2.30 1.50 1.60 1.20 1.20 2.90 3.00 3.00 3.00 2.50 3.00 2.60 3.00 2.50 3.00 2.50 2.90 – – World – – – – 1.00 1.10 1.00 1.10 1.10 1.20 1.00 1.20 – – – – – 1.10 1.30 1.10 1.50 1.00 1.30 1.10 1.10 – – – – 2.40 2.90 2.50 3.00 1.60 2.50 2.00 2.60 – – Sentiment 1.30 2.00 1.40 1.70 1.00 1.20 1.40 1.80 1.10 1.10 1.00 1.00 – 1.60 2.40 1.30 2.10 1.50 1.80 1.30 1.80 1.10 1.10 1.10 1.20 3.00 3.00 2.90 3.00 2.70 3.00 2.60 3.00 2.60 2.80 2.10 3.00 – – Alphanumeric 1.60 2.50 – – 1.20 1.40 – – 1.10 1.20 – – – 1.90 2.30 – – 1.40 1.80 – – 1.00 1.20 – – 2.90 3.00 – – 2.90 3.00 – – 2.10 3.00 – – Gemini 3.1 Flash Lite – – Ordinal 2.80 2.80 2.40 2.90 2.70 3.00 3.00 3.00 2.60 3.00 2.60 2.90 – 2.90 3.00 3.00 3.00 2.90 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 – – World – – – – 3.00 3.00 3.00 3.00 1.90 2.10 1.90 2.20 – – – – – 3.00 3.00 3.00 3.00 2.56 2.67 2.80 3.00 – – – – 3.00 3.00 3.00 3.00 2.90 3.00 3.00 3.00 – – Sentiment 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 – 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 – – Alphanumeric 3.00 3.00 – – 3.00 3.00 – – 3.00 3.00 – – – 3.00 3.00 – – 3.00 3.00 – – 3.00 3.00 – – 3.00 3.00 – – 3.00 3.00 – – 3.00 3.00 – – Table 6: Exploration count (number of distinct arms tried) at turns @2 and @9, by model, nomenclature, and debiasing condition. Warn. = explicit bias warning; Instr. = explicit exploration instruction. H = helpful framing; M = misleading framing. Scale: high_scale; variance: No. Bandit Farm Clothing Helpful Misleading Helpful Misleading Helpful Misleading Model Warn. Instr. Nomenclature @2 @9 @2 @9 @2 @9 @2 @9 @2 @9 @2 @9 Qwen3-32B – – Ordinal 0.00 0.00 147.50 395.00 0.00 15.00 142.50 357.50 5.00 5.00 147.50 447.50 – 17.50 32.50 112.50 235.00 22.50 45.00 117.50 180.00 12.50 15.00 132.50 390.00 55.00 82.50 92.50 122.50 65.00 105.00 80.00 120.00 67.50 82.50 85.00 95.00 – – World – – – – 32.50 50.00 107.50 137.50 7.50 22.50 140.00 432.50 – – – – – 25.00 55.00 115.00 225.00 20.00 27.50 145.00 395.00 – – – – 65.00 87.50 97.50 132.50 50.00 77.50 110.00 250.00 – – Sentiment 32.50 42.50 100.00 147.50 47.50 70.00 90.00 145.00 42.50 47.50 92.50 170.00 – 40.00 70.00 80.00 145.00 52.50 67.50 95.00 150.00 40.00 42.50 100.00 190.00 67.50 97.50 87.50 122.50 70.00 107.50 82.50 92.50 75.00 92.50 85.00 122.50 – – Alphanumeric 75.00 80.00 – – 70.00 77.50 – – 67.50 80.00 – – – 80.00 117.50 – – 62.50 72.50 – – 70.00 102.50 – – 72.50 77.50 – – 75.00 97.50 – – 75.00 80.00 – – OLMo-3.1 32B – – Ordinal 0.00 20.00 132.50 395.00 7.50 15.00 140.00 455.00 0.00 0.00 150.00 500.00 – 45.00 72.50 125.00 315.00 35.00 50.00 92.50 215.00 47.50 85.00 135.00 432.50 72.50 82.50 75.00 85.00 52.50 90.00 87.50 95.00 60.00 80.00 90.00 125.00 – – World – – – – 0.00 5.00 150.00 485.00 17.50 35.00 142.50 455.00 – – – – – 2.50 7.50 137.50 420.00 15.00 45.00 137.50 452.50 – – – – 50.00 70.00 90.00 95.00 22.50 60.00 107.50 205.00 – – Sentiment 72.50 140.00 82.50 255.00 75.00 215.00 92.50 275.00 17.50 52.50 135.00 450.00 – 72.50 147.50 82.50 222.50 60.00 167.50 92.50 260.00 62.50 185.00 117.50 362.50 75.00 85.00 72.50 90.00 70.00 82.50 72.50 92.50 80.00 120.00 92.50 127.50 – – Alphanumeric 75.00 145.00 – – 75.00 237.50 – – 80.00 237.50 – – – 77.50 157.50 – – 85.00 220.00 – – 90.00 285.00 – – 77.50 87.50 – – 72.50 77.50 – – 85.00 107.50 – – Gemini 3.1 Flash Lite – – Ordinal 65.00 65.00 95.00 132.50 60.00 75.00 75.00 82.50 60.00 75.00 87.50 110.00 – 70.00 75.00 75.00 83.33 70.00 77.50 75.00 75.00 75.00 75.00 75.00 77.50 75.00 82.50 75.00 115.00 75.00 87.50 75.00 95.00 75.00 90.00 75.00 115.00 – – World – – – – 75.00 75.00 75.00 75.00 25.00 35.00 110.00 277.50 – – – – – 75.00 75.00 75.00 75.00 52.78 58.33 80.00 90.00 – – – – 75.00 80.00 75.00 91.67 70.00 80.00 75.00 125.00 – – Sentiment 75.00 75.00 75.00 75.00 75.00 75.00 75.00 75.00 75.00 75.00 75.00 75.00 – 75.00 75.00 75.00 75.00 75.00 75.00 75.00 75.00 75.00 75.00 75.00 75.00 75.00 90.00 75.00 85.00 75.00 90.00 75.00 100.00 75.00 90.00 75.00 97.50 – – Alphanumeric 75.00 75.00 – – 75.00 75.00 – – 75.00 75.00 – – – 75.00 75.00 – – 75.00 75.00 – – 75.00 75.00 – – 75.00 80.56 – – 75.00 87.50 – – 75.00 91.67 – – Table 7: Cumulative regret at turns @2 and @9, by model, nomenclature, and debiasing condition. Warn. = explicit bias warning; Instr. = explicit exploration instruction. H = helpful framing; M = misleading framing. Scale: high_scale; variance: No. Figure 7: Impact of Explicit Debiasing on Exploration Count in helpful (top) and misleading (bottom) cases. When the prompt includes both the warning and the instruction to explore, exploration coverage is raised significantly. When the prompt includes only the warning that biases may be misleading, exploration count tends to slightly increase. These results are with no variance and high scales. Figure 8: Impact of Explicit Debiasing on Regret in helpful (top) and misleading (bottom) cases. The impact on cumulative regret depends on whether the nomenclature is helpful or misleading. When the nomenclature is helpful, both the instruction to explore and the warning usually increase cumulative regret. When the nomenclature is misleading, both usually decrease cumulative regret. The strength of this impact is highly inconsistent across domains, nomenclatures, and models. These results are with no variance and high scales. A.5 AI Disclosure CoPilot and Claude Code were used in the generation of plotting code and some minor edits to core experiment functionality. Appendix B Additional Experimental Results from Main Experiments This section contains more detailed results for the main experiments. This includes results using the ”low” reward scale and with added variance to the reward distributions. Results are stable across variance conditions. This section is broken down into cumulative regret results for main experiments (Section B.1), exploration count (Section B.2), scaling results with different size Qwen models (B.3), a comparison between a scenario with 3 arms and 5 arms (B.4), and sample reasoning traces (B.5). B.1 Cumulative Regret These tables show the average cumulative regret for all nomenclatures, domains, models, and variance levels. Bandit Farm Clothing Scale Nomenclature Helpful Misleading Helpful Misleading Helpful Misleading H+ Ordinal 0.04 0.79 0.03 0.91 0.00 1.00 World – – 0.01 0.97 0.07 0.91 Sentiment 0.28 0.51 0.43 0.55 0.10 0.90 Alphanumeric 0.29 – 0.47 – 0.47 – L+ Ordinal 0.08 0.54 0.03 0.68 0.00 0.95 World – – 0.03 0.55 0.07 0.74 Sentiment 0.20 0.35 0.14 0.30 0.15 0.73 Alphanumeric 0.24 – 0.27 – 0.30 – L- Ordinal 0.12 0.23 0.08 0.36 0.06 0.47 World – – 0.12 0.19 0.08 0.42 Sentiment 0.15 0.15 0.14 0.23 0.10 0.32 Alphanumeric 0.15 – 0.20 – 0.19 – H- Ordinal 0.14 0.28 0.10 0.42 0.06 0.38 World – – 0.14 0.27 0.12 0.38 Sentiment 0.16 0.20 0.14 0.25 0.10 0.34 Alphanumeric 0.16 – 0.14 – 0.27 – Table 8: Normalized cumulative regret at the final turn for OLMo-3.1 32B, variance condition: No, by reward scale and nomenclature type. Scale labels: H = high reward magnitude; L = low; + = positive rewards; −- = negative (e.g. H+ is the high positive scale). Bandit Farm Clothing Scale Nomenclature Helpful Misleading Helpful Misleading Helpful Misleading H+ Ordinal 0.04 0.82 0.04 0.88 0.05 0.91 World – – 0.03 0.65 0.08 0.92 Sentiment 0.23 0.44 0.44 0.56 -0.00 0.76 Alphanumeric 0.27 – 0.43 – 0.52 – L+ Ordinal 0.07 0.62 0.03 0.60 0.00 0.92 World – – 0.05 0.61 0.05 0.79 Sentiment 0.11 0.39 0.25 0.42 0.15 0.64 Alphanumeric 0.18 – 0.27 – 0.32 – L- Ordinal 0.08 0.21 0.05 0.40 0.07 0.42 World – – 0.16 0.22 0.06 0.48 Sentiment 0.16 0.19 0.13 0.25 0.11 0.39 Alphanumeric 0.16 – 0.20 – 0.28 – H- Ordinal 0.12 0.26 0.07 0.43 0.08 0.47 World – – 0.14 0.23 0.09 0.33 Sentiment 0.16 0.20 0.17 0.31 0.14 0.37 Alphanumeric 0.14 – 0.30 – 0.17 – Table 9: Normalized cumulative regret at the final turn for OLMo-3.1 32B, variance condition: Low, by reward scale and nomenclature type. Scale labels: H = high reward magnitude; L = low; + = positive rewards; −- = negative (e.g. H+ is the high positive scale). Bandit Farm Clothing Scale Nomenclature Helpful Misleading Helpful Misleading Helpful Misleading H+ Ordinal 0.02 0.89 0.01 0.94 0.05 0.94 World – – 0.04 0.98 -0.01 0.80 Sentiment 0.32 0.71 0.30 0.68 0.20 0.80 Alphanumeric 0.36 – 0.41 – 0.45 – L+ Ordinal 0.01 0.82 0.05 0.56 0.01 0.93 World – – 0.04 0.64 0.04 0.69 Sentiment 0.23 0.31 0.24 0.31 0.21 0.64 Alphanumeric 0.22 – 0.29 – 0.38 – L- Ordinal 0.09 0.36 0.14 0.38 0.09 0.44 World – – 0.13 0.17 0.11 0.40 Sentiment 0.15 0.19 0.17 0.19 0.16 0.40 Alphanumeric 0.16 – 0.25 – 0.20 – H- Ordinal 0.08 0.31 0.08 0.47 0.11 0.39 World – – 0.17 0.25 0.14 0.37 Sentiment 0.14 0.26 0.15 0.27 0.12 0.35 Alphanumeric 0.17 – 0.20 – 0.37 – Table 10: Normalized cumulative regret at the final turn for OLMo-3.1 32B, variance condition: High, by reward scale and nomenclature type. Scale labels: H = high reward magnitude; L = low; + = positive rewards; −- = negative (e.g. H+ is the high positive scale). Bandit Farm Clothing Scale Nomenclature Helpful Misleading Helpful Misleading Helpful Misleading H+ Ordinal 0.00 0.79 0.03 0.71 0.01 0.90 World – – 0.10 0.28 0.04 0.86 Sentiment 0.09 0.29 0.14 0.29 0.10 0.34 Alphanumeric 0.16 – 0.15 – 0.16 – L+ Ordinal 0.00 0.42 0.02 0.45 0.01 0.35 World – – 0.18 0.24 0.05 0.61 Sentiment 0.17 0.33 0.17 0.18 0.11 0.30 Alphanumeric 0.12 – 0.15 – 0.15 – L- Ordinal 0.15 0.16 0.15 0.20 0.14 0.16 World – – 0.15 0.18 0.12 0.48 Sentiment 0.15 0.18 0.18 0.20 0.16 0.19 Alphanumeric 0.15 – 0.17 – 0.15 – H- Ordinal 0.15 0.17 0.15 0.20 0.14 0.15 World – – 0.15 0.15 0.15 0.46 Sentiment 0.15 0.15 0.15 0.15 0.15 0.16 Alphanumeric 0.16 – 0.17 – 0.15 – Table 11: Normalized cumulative regret at the final turn for Qwen3-32B, variance condition: No, by reward scale and nomenclature type. Scale labels: H = high reward magnitude; L = low; + = positive rewards; −- = negative (e.g. H+ is the high positive scale). Bandit Farm Clothing Scale Nomenclature Helpful Misleading Helpful Misleading Helpful Misleading H+ Ordinal 0.00 0.88 0.03 0.70 -0.01 0.89 World – – 0.11 0.26 0.04 0.81 Sentiment 0.12 0.54 0.16 0.28 0.09 0.37 Alphanumeric 0.21 – 0.13 – 0.25 – L+ Ordinal 0.00 0.72 0.04 0.51 0.05 0.77 World – – 0.10 0.21 0.04 0.58 Sentiment 0.13 0.31 0.16 0.34 0.11 0.29 Alphanumeric 0.21 – 0.16 – 0.17 – L- Ordinal 0.16 0.16 0.15 0.27 0.13 0.25 World – – 0.16 0.15 0.13 0.43 Sentiment 0.13 0.16 0.16 0.17 0.13 0.18 Alphanumeric 0.16 – 0.17 – 0.14 – H- Ordinal 0.14 0.13 0.15 0.17 0.15 0.19 World – – 0.16 0.15 0.12 0.40 Sentiment 0.15 0.14 0.18 0.18 0.16 0.15 Alphanumeric 0.16 – 0.15 – 0.15 – Table 12: Normalized cumulative regret at the final turn for Qwen3-32B, variance condition: Low, by reward scale and nomenclature type. Scale labels: H = high reward magnitude; L = low; + = positive rewards; −- = negative (e.g. H+ is the high positive scale). Bandit Farm Clothing Scale Nomenclature Helpful Misleading Helpful Misleading Helpful Misleading H+ Ordinal 0.01 0.87 0.05 0.75 -0.00 0.80 World – – 0.17 0.29 0.03 0.73 Sentiment 0.19 0.37 0.14 0.32 0.13 0.40 Alphanumeric 0.19 – 0.21 – 0.16 – L+ Ordinal 0.02 0.43 0.01 0.39 0.06 0.57 World – – 0.17 0.27 0.03 0.65 Sentiment 0.09 0.25 0.18 0.25 0.13 0.32 Alphanumeric 0.13 – 0.13 – 0.14 – L- Ordinal 0.13 0.26 0.20 0.18 0.12 0.31 World – – 0.16 0.18 0.10 0.49 Sentiment 0.12 0.22 0.18 0.22 0.18 0.15 Alphanumeric 0.16 – 0.16 – 0.17 – H- Ordinal 0.12 0.16 0.15 0.21 0.16 0.18 World – – 0.16 0.14 0.11 0.45 Sentiment 0.13 0.20 0.17 0.18 0.18 0.20 Alphanumeric 0.20 – 0.17 – 0.19 – Table 13: Normalized cumulative regret at the final turn for Qwen3-32B, variance condition: High, by reward scale and nomenclature type. Scale labels: H = high reward magnitude; L = low; + = positive rewards; −- = negative (e.g. H+ is the high positive scale). Bandit Farm Clothing Scale Nomenclature Helpful Misleading Helpful Misleading Helpful Misleading H+ Ordinal 0.13 0.27 0.15 0.17 0.15 0.22 World – – 0.15 0.15 0.07 0.56 Sentiment 0.15 0.15 0.15 0.15 0.15 0.15 Alphanumeric 0.15 – 0.15 – 0.15 – L+ Ordinal 0.15 0.20 0.15 0.16 0.15 0.15 World – – 0.15 0.15 0.08 0.49 Sentiment 0.15 0.15 0.15 0.15 0.15 0.15 Alphanumeric 0.15 – 0.15 – 0.15 – L- Ordinal 0.15 0.19 0.15 0.18 0.15 0.16 World – – 0.15 0.15 0.10 0.36 Sentiment 0.15 0.16 0.15 0.15 0.15 0.16 Alphanumeric 0.15 – 0.15 – 0.16 – H- Ordinal 0.15 0.18 0.15 0.18 0.15 0.15 World – – 0.15 0.15 0.12 0.27 Sentiment 0.15 0.15 0.15 0.15 0.15 0.15 Alphanumeric 0.15 – 0.15 – 0.15 – Table 14: Normalized cumulative regret at the final turn for Gemini 3.1 Flash Lite, variance condition: No, by reward scale and nomenclature type. Scale labels: H = high reward magnitude; L = low; + = positive rewards; −- = negative (e.g. H+ is the high positive scale). Bandit Farm Clothing Scale Nomenclature Helpful Misleading Helpful Misleading Helpful Misleading H+ Ordinal – – – – 0.13 0.18 World – – – – 0.06 0.51 Sentiment 0.14 0.14 0.15 0.14 0.14 0.18 Alphanumeric – – – – 0.15 – L+ Ordinal – – – – 0.14 0.16 World – – – – 0.07 0.52 Sentiment 0.15 0.17 0.14 0.16 0.13 0.17 Alphanumeric – – – – 0.15 – L- Ordinal – – – – 0.15 0.16 World – – – – 0.11 0.24 Sentiment 0.14 0.14 0.17 0.16 0.14 0.16 Alphanumeric – – – – 0.16 – H- Ordinal – – – – 0.15 0.17 World – – – – 0.10 0.26 Sentiment 0.13 0.16 0.15 0.15 0.13 0.15 Alphanumeric – – – – 0.17 – Table 15: Normalized cumulative regret at the final turn for Gemini 3.1 Flash Lite, variance condition: Low, by reward scale and nomenclature type. Scale labels: H = high reward magnitude; L = low; + = positive rewards; −- = negative (e.g. H+ is the high positive scale). Bandit Farm Clothing Scale Nomenclature Helpful Misleading Helpful Misleading Helpful Misleading H+ Ordinal – – – – 0.12 0.23 World – – – – 0.06 0.53 Sentiment 0.17 0.11 0.11 0.18 0.20 0.22 Alphanumeric – – – – 0.16 – L+ Ordinal – – – – 0.13 0.12 World – – – – 0.08 0.50 Sentiment 0.12 0.18 0.20 0.15 0.16 0.15 Alphanumeric – – – – 0.18 – L- Ordinal – – – – 0.12 0.19 World – – – – 0.15 0.37 Sentiment 0.13 0.18 0.19 0.18 0.18 0.17 Alphanumeric – – – – 0.12 – H- Ordinal – – – – 0.18 0.16 World – – – – 0.13 0.23 Sentiment 0.18 0.16 0.13 0.23 0.21 0.15 Alphanumeric – – – – 0.14 – Table 16: Normalized cumulative regret at the final turn for Gemini 3.1 Flash Lite, variance condition: High, by reward scale and nomenclature type. Scale labels: H = high reward magnitude; L = low; + = positive rewards; −- = negative (e.g. H+ is the high positive scale). B.2 Exploration Count These tables show the average exploration count at turn 3 (the earliest step that full exploration is possible) and turn 10 for all nomenclatures, domains, models, and variance levels. Bandit Farm Clothing Helpful Misleading Helpful Misleading Helpful Misleading Scale Nomenclature @3 @10 @3 @10 @3 @10 @3 @10 @3 @10 @3 @10 H+ Ordinal 1.0 1.6 1.5 1.5 1.3 1.5 1.1 1.1 1.0 1.0 1.0 1.0 World – – – – 1.0 1.1 1.0 1.1 1.1 1.2 1.0 1.2 Sentiment 1.3 2.0 1.4 1.7 1.0 1.2 1.4 1.8 1.1 1.1 1.0 1.0 Alphanumeric 1.6 2.5 – – 1.2 1.4 – – 1.1 1.2 – – L+ Ordinal 1.8 2.1 1.6 2.2 1.2 1.6 1.2 2.0 1.0 1.0 1.1 1.2 World – – – – 1.2 1.4 1.4 2.4 1.1 1.2 1.4 1.7 Sentiment 1.9 2.5 1.8 2.5 1.2 1.9 2.2 2.6 1.4 1.8 1.3 1.7 Alphanumeric 2.1 2.8 – – 1.7 2.7 – – 1.6 2.0 – – L- Ordinal 2.3 2.7 2.8 2.8 2.0 2.2 2.4 2.5 1.7 1.9 1.9 2.1 World – – – – 2.6 2.7 2.7 3.0 1.7 2.3 2.1 2.8 Sentiment 2.7 3.0 2.9 3.0 2.5 2.9 2.2 2.9 2.1 2.3 2.3 2.7 Alphanumeric 3.0 3.0 – – 2.4 3.0 – – 2.6 2.9 – – H- Ordinal 2.4 2.9 2.4 2.8 2.3 2.5 2.2 2.4 2.0 2.0 2.3 2.5 World – – – – 2.2 2.9 2.3 2.8 2.2 2.6 2.1 2.7 Sentiment 2.6 3.0 2.6 3.0 2.5 2.8 2.3 2.8 2.3 2.3 2.1 2.7 Alphanumeric 2.8 3.0 – – 2.4 2.8 – – 2.1 2.8 – – H+ UCB1 3.0 3.0 – – 3.0 3.0 – – 3.0 3.0 – – L+ UCB1 – – – – – – – – – – – – L- UCB1 – – – – – – – – – – – – H- UCB1 3.0 3.0 – – 3.0 3.0 – – 3.0 3.0 – – Table 17: Exploration count (unique arms tried) at turns 3 and 10 for OLMo-3.1 32B, variance condition: No, by reward scale and nomenclature type. Scale labels: H = high reward magnitude; L = low; + = positive rewards; −- = negative (e.g. H+ is the high positive scale). Bandit Farm Clothing Helpful Misleading Helpful Misleading Helpful Misleading Scale Nomenclature @3 @10 @3 @10 @3 @10 @3 @10 @3 @10 @3 @10 H+ Ordinal 1.6 1.9 1.4 1.4 1.3 1.3 1.0 1.3 1.0 1.0 1.1 1.2 World – – – – 1.1 1.4 1.3 2.0 1.1 1.5 1.1 1.4 Sentiment 1.9 1.9 1.8 2.1 1.4 1.8 1.5 1.7 1.0 1.1 1.0 1.0 Alphanumeric 1.8 2.2 – – 1.3 1.8 – – 1.0 1.0 – – L+ Ordinal 1.7 2.1 1.7 2.0 1.3 1.9 1.5 2.1 1.0 1.1 1.0 1.2 World – – – – 1.3 1.5 1.5 2.3 1.3 1.7 1.2 1.8 Sentiment 1.7 2.4 1.9 2.5 1.9 2.4 1.9 2.5 1.1 1.3 1.4 1.8 Alphanumeric 2.5 3.0 – – 1.9 2.4 – – 1.3 1.6 – – L- Ordinal 2.2 2.5 2.7 2.9 1.8 2.1 2.1 2.4 1.7 2.1 1.8 2.2 World – – – – 2.3 2.9 2.5 2.9 2.0 2.3 2.0 2.6 Sentiment 2.8 3.0 2.6 3.0 2.3 2.8 2.4 3.0 2.0 2.3 2.1 2.6 Alphanumeric 2.7 3.0 – – 2.6 3.0 – – 2.3 2.8 – – H- Ordinal 2.5 2.7 2.5 2.8 2.2 2.2 1.9 2.4 2.0 2.3 1.6 2.2 World – – – – 2.4 2.8 2.5 2.8 2.2 2.5 2.4 2.8 Sentiment 2.7 2.9 2.6 2.9 2.4 2.8 2.3 2.7 2.2 2.4 2.2 2.5 Alphanumeric 2.7 3.0 – – 2.2 2.6 – – 2.4 2.9 – – H+ UCB1 3.0 3.0 – – 3.0 3.0 – – 3.0 3.0 – – L+ UCB1 – – – – – – – – – – – – L- UCB1 – – – – – – – – – – – – H- UCB1 3.0 3.0 – – 3.0 3.0 – – 3.0 3.0 – – Table 18: Exploration count (unique arms tried) at turns 3 and 10 for OLMo-3.1 32B, variance condition: Low, by reward scale and nomenclature type. Scale labels: H = high reward magnitude; L = low; + = positive rewards; −- = negative (e.g. H+ is the high positive scale). Bandit Farm Clothing Helpful Misleading Helpful Misleading Helpful Misleading Scale Nomenclature @3 @10 @3 @10 @3 @10 @3 @10 @3 @10 @3 @10 H+ Ordinal 1.6 1.7 1.3 1.4 1.2 1.2 1.0 1.2 1.0 1.1 1.1 1.3 World – – – – 1.0 1.3 1.0 1.2 1.0 1.3 1.2 1.5 Sentiment 1.3 1.8 1.4 1.5 1.4 1.9 1.3 1.6 1.0 1.0 1.2 1.5 Alphanumeric 1.8 2.6 – – 1.2 1.9 – – 1.0 1.1 – – L+ Ordinal 1.4 1.5 1.1 1.6 1.4 1.6 1.4 2.2 1.1 1.1 1.1 1.2 World – – – – 1.4 1.7 1.5 2.1 1.0 1.3 1.3 1.7 Sentiment 1.9 2.7 1.9 2.5 1.8 2.3 2.0 2.5 1.1 1.3 1.2 1.6 Alphanumeric 2.1 2.9 – – 1.8 2.4 – – 1.4 1.8 – – L- Ordinal 2.2 2.6 2.2 2.8 2.2 2.3 2.3 2.5 1.5 2.1 1.9 2.3 World – – – – 2.5 2.9 2.6 3.0 2.1 2.4 2.1 2.7 Sentiment 2.7 3.0 2.8 3.0 2.4 2.9 2.2 2.8 2.2 2.6 2.4 2.5 Alphanumeric 2.8 3.0 – – 2.2 2.9 – – 2.3 2.9 – – H- Ordinal 2.2 2.7 2.5 2.7 2.0 2.1 2.0 2.4 2.1 2.2 2.0 2.2 World – – – – 2.4 2.9 2.7 3.0 2.3 2.7 1.9 2.9 Sentiment 2.5 3.0 2.8 2.9 2.4 2.8 2.4 2.7 2.1 2.4 2.2 2.5 Alphanumeric 2.8 3.0 – – 2.7 2.9 – – 2.0 2.6 – – H+ UCB1 3.0 3.0 – – 3.0 3.0 – – 3.0 3.0 – – L+ UCB1 – – – – – – – – – – – – L- UCB1 – – – – – – – – – – – – H- UCB1 3.0 3.0 – – 3.0 3.0 – – 3.0 3.0 – – Table 19: Exploration count (unique arms tried) at turns 3 and 10 for OLMo-3.1 32B, variance condition: High, by reward scale and nomenclature type. Scale labels: H = high reward magnitude; L = low; + = positive rewards; −- = negative (e.g. H+ is the high positive scale). Bandit Farm Clothing Helpful Misleading Helpful Misleading Helpful Misleading Scale Nomenclature @3 @10 @3 @10 @3 @10 @3 @10 @3 @10 @3 @10 H+ Ordinal 1.0 1.0 1.1 1.7 1.0 1.6 1.2 2.1 1.2 1.2 1.1 1.3 World – – – – 1.9 2.5 2.0 2.9 1.3 1.7 1.2 1.4 Sentiment 1.9 2.2 2.4 2.8 2.1 2.4 2.6 2.7 2.0 2.2 2.3 2.4 Alphanumeric 2.8 3.0 – – 2.7 3.0 – – 2.6 3.0 – – L+ Ordinal 1.0 1.0 1.8 2.7 1.2 1.3 1.7 2.7 1.2 1.2 2.0 2.9 World – – – – 1.8 2.6 2.6 2.9 1.5 1.8 1.5 2.2 Sentiment 2.2 2.7 2.2 2.4 2.5 2.8 2.2 2.9 2.2 2.4 2.4 2.6 Alphanumeric 2.6 2.6 – – 2.7 3.0 – – 3.0 3.0 – – L- Ordinal 2.6 3.0 2.9 3.0 2.2 3.0 2.9 3.0 2.6 2.9 3.0 3.0 World – – – – 3.0 3.0 3.0 3.0 2.3 2.7 2.2 2.8 Sentiment 2.9 3.0 2.8 3.0 3.0 3.0 2.8 3.0 3.0 3.0 2.8 3.0 Alphanumeric 3.0 3.0 – – 3.0 3.0 – – 2.8 3.0 – – H- Ordinal 2.7 3.0 3.0 3.0 2.8 3.0 2.9 3.0 2.5 2.9 3.0 3.0 World – – – – 3.0 3.0 3.0 3.0 2.5 3.0 2.3 2.8 Sentiment 2.9 3.0 3.0 3.0 3.0 3.0 3.0 3.0 2.9 3.0 2.8 3.0 Alphanumeric 3.0 3.0 – – 3.0 3.0 – – 3.0 3.0 – – H+ UCB1 3.0 3.0 – – 3.0 3.0 – – 3.0 3.0 – – L+ UCB1 – – – – – – – – – – – – L- UCB1 – – – – – – – – – – – – H- UCB1 3.0 3.0 – – 3.0 3.0 – – 3.0 3.0 – – Table 20: Exploration count (unique arms tried) at turns 3 and 10 for Qwen3-32B, variance condition: No, by reward scale and nomenclature type. Scale labels: H = high reward magnitude; L = low; + = positive rewards; −- = negative (e.g. H+ is the high positive scale). Bandit Farm Clothing Helpful Misleading Helpful Misleading Helpful Misleading Scale Nomenclature @3 @10 @3 @10 @3 @10 @3 @10 @3 @10 @3 @10 H+ Ordinal 1.2 1.3 1.1 1.5 1.3 1.5 1.4 1.8 1.2 1.3 1.2 1.4 World – – – – 1.9 2.4 2.2 2.8 1.3 1.5 1.2 1.5 Sentiment 1.5 2.1 1.7 2.1 2.0 2.4 2.6 2.7 1.7 1.9 2.1 2.4 Alphanumeric 2.5 2.7 – – 3.0 3.0 – – 2.4 2.7 – – L+ Ordinal 1.0 1.1 1.2 2.0 1.3 1.8 1.4 2.5 1.6 1.7 1.5 1.6 World – – – – 1.9 2.5 2.3 3.0 1.7 1.8 1.7 2.2 Sentiment 2.0 2.5 2.3 2.7 1.9 2.5 2.3 2.7 2.0 2.1 2.4 2.6 Alphanumeric 2.3 2.9 – – 2.7 3.0 – – 2.7 3.0 – – L- Ordinal 2.6 2.9 3.0 3.0 2.8 3.0 2.7 3.0 2.8 2.9 2.8 3.0 World – – – – 2.9 3.0 3.0 3.0 2.0 2.8 2.5 2.9 Sentiment 2.7 2.9 2.9 3.0 3.0 3.0 2.8 3.0 2.8 2.9 2.8 3.0 Alphanumeric 3.0 3.0 – – 3.0 3.0 – – 2.9 3.0 – – H- Ordinal 2.6 2.9 3.0 3.0 2.8 2.9 3.0 3.0 2.7 2.9 2.9 2.9 World – – – – 3.0 3.0 2.9 3.0 2.5 2.9 2.7 2.8 Sentiment 2.8 3.0 2.9 3.0 2.9 3.0 2.9 3.0 2.8 3.0 3.0 3.0 Alphanumeric 3.0 3.0 – – 3.0 3.0 – – 3.0 3.0 – – H+ UCB1 3.0 3.0 – – 3.0 3.0 – – 3.0 3.0 – – L+ UCB1 – – – – – – – – – – – – L- UCB1 – – – – – – – – – – – – H- UCB1 3.0 3.0 – – 3.0 3.0 – – 3.0 3.0 – – Table 21: Exploration count (unique arms tried) at turns 3 and 10 for Qwen3-32B, variance condition: Low, by reward scale and nomenclature type. Scale labels: H = high reward magnitude; L = low; + = positive rewards; −- = negative (e.g. H+ is the high positive scale). Bandit Farm Clothing Helpful Misleading Helpful Misleading Helpful Misleading Scale Nomenclature @3 @10 @3 @10 @3 @10 @3 @10 @3 @10 @3 @10 H+ Ordinal 1.1 1.5 1.4 1.4 1.2 1.8 1.2 1.8 1.1 1.2 1.3 1.4 World – – – – 2.0 2.7 2.0 3.0 1.1 1.5 1.6 1.7 Sentiment 2.2 2.4 2.0 2.4 2.2 2.7 2.6 2.7 2.2 2.2 2.3 2.4 Alphanumeric 2.2 2.7 – – 2.7 2.9 – – 2.6 2.8 – – L+ Ordinal 1.0 1.5 1.8 2.6 1.2 1.6 1.9 2.8 1.1 1.4 1.7 2.1 World – – – – 2.1 2.7 2.2 2.9 1.4 1.5 1.5 2.1 Sentiment 2.0 2.4 2.6 2.8 2.1 2.7 2.3 2.8 2.1 2.3 2.5 2.6 Alphanumeric 2.8 3.0 – – 2.7 3.0 – – 2.9 3.0 – – L- Ordinal 2.5 2.8 2.9 2.9 2.9 3.0 2.8 3.0 2.6 2.9 2.8 2.9 World – – – – 3.0 3.0 3.0 3.0 2.3 2.8 2.4 2.8 Sentiment 3.0 3.0 2.9 3.0 2.7 3.0 3.0 3.0 2.9 3.0 2.8 3.0 Alphanumeric 3.0 3.0 – – 3.0 3.0 – – 3.0 3.0 – – H- Ordinal 2.6 2.8 3.0 3.0 2.9 3.0 2.9 3.0 2.6 2.8 2.9 3.0 World – – – – 3.0 3.0 3.0 3.0 2.2 2.8 2.6 2.9 Sentiment 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 2.9 3.0 Alphanumeric 3.0 3.0 – – 3.0 3.0 – – 3.0 3.0 – – H+ UCB1 3.0 3.0 – – 3.0 3.0 – – 3.0 3.0 – – L+ UCB1 – – – – – – – – – – – – L- UCB1 – – – – – – – – – – – – H- UCB1 3.0 3.0 – – 3.0 3.0 – – 3.0 3.0 – – Table 22: Exploration count (unique arms tried) at turns 3 and 10 for Qwen3-32B, variance condition: High, by reward scale and nomenclature type. Scale labels: H = high reward magnitude; L = low; + = positive rewards; −- = negative (e.g. H+ is the high positive scale). Bandit Farm Clothing Helpful Misleading Helpful Misleading Helpful Misleading Scale Nomenclature @3 @10 @3 @10 @3 @10 @3 @10 @3 @10 @3 @10 H+ Ordinal 2.8 2.8 2.4 2.9 2.7 3.0 3.0 3.0 2.6 3.0 2.6 2.9 World – – – – 3.0 3.0 3.0 3.0 1.9 2.1 1.9 2.2 Sentiment 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 Alphanumeric 3.0 3.0 – – 3.0 3.0 – – 3.0 3.0 – – L+ Ordinal 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 2.7 3.0 3.0 3.0 World – – – – 3.0 3.0 3.0 3.0 2.1 2.3 2.0 2.4 Sentiment 3.0 3.0 3.0 3.0 3.0 3.0 2.9 3.0 3.0 3.0 3.0 3.0 Alphanumeric 3.0 3.0 – – 3.0 3.0 – – 3.0 3.0 – – L- Ordinal 2.9 3.0 3.0 3.0 3.0 3.0 3.0 3.0 2.7 3.0 3.0 3.0 World – – – – 3.0 3.0 3.0 3.0 2.1 2.5 2.4 2.9 Sentiment 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 Alphanumeric 3.0 3.0 – – 3.0 3.0 – – 3.0 3.0 – – H- Ordinal 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 World – – – – 3.0 3.0 3.0 3.0 2.3 2.7 2.6 3.0 Sentiment 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 Alphanumeric 3.0 3.0 – – 3.0 3.0 – – 3.0 3.0 – – H+ UCB1 3.0 3.0 – – 3.0 3.0 – – 3.0 3.0 – – L+ UCB1 – – – – – – – – – – – – L- UCB1 – – – – – – – – – – – – H- UCB1 3.0 3.0 – – 3.0 3.0 – – 3.0 3.0 – – Table 23: Exploration count (unique arms tried) at turns 3 and 10 for Gemini 3.1 Flash Lite, variance condition: No, by reward scale and nomenclature type. Scale labels: H = high reward magnitude; L = low; + = positive rewards; −- = negative (e.g. H+ is the high positive scale). Bandit Farm Clothing Helpful Misleading Helpful Misleading Helpful Misleading Scale Nomenclature @3 @10 @3 @10 @3 @10 @3 @10 @3 @10 @3 @10 H+ Ordinal – – – – – – – – 2.8 3.0 2.7 3.0 World – – – – – – – – 1.9 2.0 2.0 2.2 Sentiment 3.0 3.0 2.9 3.0 3.0 3.0 3.0 3.0 2.9 3.0 2.9 3.0 Alphanumeric – – – – – – – – 3.0 3.0 – – L+ Ordinal – – – – – – – – 2.8 3.0 3.0 3.0 World – – – – – – – – 1.8 2.2 1.9 2.4 Sentiment 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 2.8 3.0 3.0 3.0 Alphanumeric – – – – – – – – 3.0 3.0 – – L- Ordinal – – – – – – – – 2.9 3.0 3.0 3.0 World – – – – – – – – 2.2 2.6 3.0 3.0 Sentiment 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 Alphanumeric – – – – – – – – 3.0 3.0 – – H- Ordinal – – – – – – – – 2.9 3.0 3.0 3.0 World – – – – – – – – 2.3 2.5 2.4 2.8 Sentiment 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 Alphanumeric – – – – – – – – 3.0 3.0 – – H+ UCB1 3.0 3.0 – – 3.0 3.0 – – 3.0 3.0 – – L+ UCB1 – – – – – – – – – – – – L- UCB1 – – – – – – – – – – – – H- UCB1 3.0 3.0 – – 3.0 3.0 – – 3.0 3.0 – – Table 24: Exploration count (unique arms tried) at turns 3 and 10 for Gemini 3.1 Flash Lite, variance condition: Low, by reward scale and nomenclature type. Scale labels: H = high reward magnitude; L = low; + = positive rewards; −- = negative (e.g. H+ is the high positive scale). Bandit Farm Clothing Helpful Misleading Helpful Misleading Helpful Misleading Scale Nomenclature @3 @10 @3 @10 @3 @10 @3 @10 @3 @10 @3 @10 H+ Ordinal – – – – – – – – 2.4 2.9 2.5 2.9 World – – – – – – – – 2.0 2.1 1.8 2.4 Sentiment 2.9 3.0 2.9 3.0 3.0 3.0 2.9 3.0 2.9 3.0 3.0 3.0 Alphanumeric – – – – – – – – 3.0 3.0 – – L+ Ordinal – – – – – – – – 3.0 3.0 2.9 3.0 World – – – – – – – – 2.1 2.2 1.9 2.4 Sentiment 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 Alphanumeric – – – – – – – – 3.0 3.0 – – L- Ordinal – – – – – – – – 3.0 3.0 3.0 3.0 World – – – – – – – – 2.1 2.9 2.5 2.9 Sentiment 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 Alphanumeric – – – – – – – – 3.0 3.0 – – H- Ordinal – – – – – – – – 3.0 3.0 3.0 3.0 World – – – – – – – – 2.2 2.8 2.4 2.9 Sentiment 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 Alphanumeric – – – – – – – – 3.0 3.0 – – H+ UCB1 3.0 3.0 – – 3.0 3.0 – – 3.0 3.0 – – L+ UCB1 – – – – – – – – – – – – L- UCB1 – – – – – – – – – – – – H- UCB1 3.0 3.0 – – 3.0 3.0 – – 3.0 3.0 – – Table 25: Exploration count (unique arms tried) at turns 3 and 10 for Gemini 3.1 Flash Lite, variance condition: High, by reward scale and nomenclature type. Scale labels: H = high reward magnitude; L = low; + = positive rewards; −- = negative (e.g. H+ is the high positive scale). Figure 9: Impact of Reward Scale on Exploration Count We observe more exploration when reward is negative (blue) than positive (brown). These results are with no variance and low scales. B.3 Qwen Scaling Results Bandit Farm Clothing Regret Exploration Regret Exploration Regret Exploration Model Nomenclature H M H M H M H M H M H M Qwen3-8B Ordinal 0.01 0.95 1.10 1.10 0.00 1.00 1.00 1.00 – – – – World – – – – 0.01 0.78 1.10 1.50 – – – – Sentiment 0.33 0.60 1.10 1.00 0.20 0.70 1.00 1.00 – – – – Alphanumeric 0.40 – 1.60 – 0.34 – 1.60 – – – – – Qwen3-14B Ordinal 0.07 0.49 1.70 2.30 0.04 0.71 1.60 2.10 0.09 0.75 1.70 1.80 World – – – – 0.10 0.32 2.20 2.60 0.07 0.83 1.70 1.50 Sentiment 0.13 0.24 2.50 2.90 0.12 0.27 2.50 2.80 0.12 0.64 1.90 1.90 Alphanumeric 0.17 – 3.00 – 0.15 – 3.00 – 0.17 – 3.00 – Qwen3-32B Ordinal 0.00 0.79 1.00 1.70 0.03 0.71 1.60 2.10 0.01 0.90 1.20 1.30 World – – – – 0.10 0.28 2.50 2.90 0.04 0.86 1.70 1.40 Sentiment 0.09 0.29 2.20 2.80 0.14 0.29 2.40 2.70 0.10 0.34 2.20 2.40 Alphanumeric 0.16 – 3.00 – 0.15 – 3.00 – 0.16 – 3.00 – Table 26: Normalised cumulative regret and exploration count at turn 9, by Qwen model and nomenclature. Columns are grouped by domain and split into regret/exploration, with H/M = helpful/misleading framing. Regret is normalised by the same turn-count scale factor used in the notebook. Overall, we found that Qwen3-8B almost never explored, while Qwen3 14B and 32B behaved fairly similarly. To us, this indicated that the 8B model was too small to adequately perform the task. This is particularly clear in its inability to explore in the alphanumeric setting. Since Qwen3 14B and 32B performed similarly, we decided to move forward with the larger model. Scale: high_scale; variance: No; history: Summarized. B.4 3 Arm and 5 Arm Comparison Arms Model Nom. Bandit Regret Bandit Explor Farm Regret Farm Explor Clothing Regret Clothing Explor 3 Arms Qwen3-32B Helpful 0.00 1.00 0.03 1.60 0.01 1.20 Mislead 0.79 1.70 0.71 2.10 0.90 1.30 Gemini 3.1 Flash Lite Helpful 0.13 2.80 0.15 3.00 0.15 3.00 Mislead 0.27 2.90 0.17 3.00 0.22 2.90 OLMo-3.1-32B Helpful 0.04 1.60 0.03 1.50 0.00 1.00 Mislead 0.79 1.50 0.91 1.10 1.00 1.00 5 Arms Qwen3-32B Helpful 0.06 1.10 0.07 1.50 0.00 1.10 Mislead 0.97 1.10 0.82 1.60 0.81 1.50 Gemini 3.1 Flash Lite Helpful 0.09 3.00 0.20 4.40 0.20 4.40 Mislead 0.60 2.67 0.29 4.86 0.34 4.33 OLMo-3.1-32B Helpful 0.02 1.50 0.05 2.10 0.02 1.60 Mislead 0.87 1.50 0.79 1.80 0.86 1.50 Table 27: Normalized cumulative regret and exploration count at turn 9 for 3-arm and 5-arm outputs, by model, nomenclature, and domain. H = helpful, M = misleading. Note that the exploration count results are not directly comparable as for the 5 arms, the maximum value is now 5. We find slightly less exploration for Qwen and slightly more exploration for OLMo. We note that Gemini no longer saturates the exploration count metric - this is particularly notable for the abstract bandit, where it does not exceed 3 (out of a maximum of 5). These results indicate that OLMo and Qwen do not necessarily explore more when there are more arms, while Gemini tends to explores more but does not explore fully. We do not observe consistent changes in regret. B.5 Sample Reasoning Trace In this section, we provide an example where Gemini follows a misleading semantic prior despite contradictory reward evidence. This example comes from Gemini on the recommendation domain with the misleading world nomenclature, where the user is looking for a shirt and is in a cold climate. The model has previously selected ”flannel long-sleeve shirt” for a reward of 25 and ”midweight t-shirt” for a reward of 50. It has not selected ”tank top” which has a true reward of 75. We observe that the reasoning traces explicitly reject exploring ”tank top” due to the negative semantic label. We also observe that the model re-explores ”flannel long-sleeve shirt” due to the positive semantic label, even after observing a low reward. Turn 6: The model selects midweight t-shirt as it is the highest observed reward. ”To maximize the reward, I need to balance exploring unknown options with exploiting the option that has demonstrated the highest satisfaction score so far. The midweight t-shirt has a solid observed mean of 50.0 after 4 trials. The flannel long-sleeve shirt has a lower observed mean (25.0) and is more appropriate for the ”cold” climate, but it has underperformed compared to the t-shirt. The tank top has never been tried and is objectively the worst choice for a ”cold” climate, making it a poor candidate for exploration. Given the goal is to maximize reward and the midweight t-shirt is currently yielding the highest return, I will continue to exploit this high-performing option. midweight t-shirt” Turn 7: The model selects the flannel again despite the lower observed reward signal. ”To maximize the reward, I need to balance exploiting the option with the highest observed mean (midweight t-shirt) and exploring the remaining options to ensure no better-performing item is missed, especially given the cold climate context. The ”midweight t-shirt” has a high observed mean, but it is not ideally suited for ”cold” weather compared to the ”flannel long-sleeve shirt.” Although the flannel shirt has a lower observed mean based on only two trials, it is theoretically a better fit for the climate, suggesting its true mean satisfaction could be higher. flannel long-sleeve shirt” Turn 8: The model returns to the midweight t-shirt ”To maximize the total reward, I need to balance exploring the items that have not been tested (like the tank top) and exploiting the items that have shown higher satisfaction scores (the midweight t-shirt). Given that the current climate is ”cold,” a tank top is highly unlikely to provide a high reward. The ”midweight t-shirt” has been tested 5 times with a consistent mean of 50.0, which is significantly higher than the ”flannel long-sleeve shirt.” Continuing to choose the best-performing option in the remaining steps is the optimal strategy. midweight t-shirt ”

Instant research discovery