Paper deep dive

Toward a Graded Measure of Belief Stability in Large Language Models

Samantha Dies, Branden Fitelson, Tina Eliassi-Rad

Year: 2026Venue: arXiv preprintArea: cs.CLType: PreprintEmbeddings: 183

Intelligence

Status: not_run | Model: - | Prompt: - | Confidence: 0%

Entities (0)

No extracted entities yet.

Relation Signals (0)

No relation signals yet.

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Large language models (LLMs) increasingly mediate how people access and reason with information, yet factual reliability is usually evaluated one judgment at a time. We introduce graded belief stability, a relational measure of how well a belief persists within an LLM's broader belief system. Unlike individual belief probability, it asks whether support for a claim persists when that claim is considered alongside the model's other epistemic commitments. We operationalize this idea with a Direct Conditional estimator that uses internal model representations to estimate conditional belief probabilities. Across 12 LLMs and three domains, lower-stability beliefs exhibit greater mean behavioral movement under conversational challenge in 83.3% of model-domain settings after matching on individual belief probability. Graded belief stability therefore extends reliability assessment beyond how strongly an LLM supports a claim to how robustly that belief is supported within its broader system of beliefs.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

180,262 characters extracted from source content.

Expand or collapse full text

Toward a Graded Measure of Belief Stability in Large Language Models Samantha Dies Branden Fitelson Affiliation: Northeastern University, 360 Huntington Ave, Boston, MA 02115 USA Tina Eliassi-Rad Affiliation: Northeastern University, 360 Huntington Ave, Boston, MA 02115 USA Affiliation: Santa Fe Institute, 1399 Hyde Park Road, Santa Fe, NM 87501 USA Affiliation: dies.s@northeastern.edu Abstract Large language models (LLMs) increasingly mediate how people access and reason with information, yet factual reliability is usually evaluated one judgment at a time. We introduce graded belief stability, a relational measure of how well a belief persists within an LLM’s broader belief system. Unlike individual belief probability, it asks whether support for a claim persists when that claim is considered alongside the model’s other epistemic commitments. We operationalize this idea with a Direct Conditional estimator that uses internal model representations to estimate conditional belief probabilities. Across 1212 LLMs and three domains, lower-stability beliefs exhibit greater mean behavioral movement under conversational challenge in 83.3%83.3\% of model–domain settings after matching on individual belief probability. Graded belief stability therefore extends reliability assessment beyond how strongly an LLM supports a claim to how robustly that belief is supported within its broader system of beliefs. 1 Introduction Large language models (LLMs) increasingly impact how people access and reason with information, making the reliability of their factual judgments central to their use [4, 5]. Existing evaluations characterize several dimensions of this reliability, including accuracy, uncertainty, and calibration [6, 25, 1, 41]. In interactive settings, however, factual claims are rarely considered in isolation. New claims, corrections, and contextual information accumulate over a conversation, and models may revise answers they previously supported. Reliability therefore also depends on a different question: which of a model’s beliefs persist as the informational context around them changes? Such belief revisions are well documented. LLM factual judgments can vary with paraphrasing and changes in semantic context [19, 21, 15], as well as under user disagreement, contradictory feedback, and multi-turn interaction [39, 27, 30, 37, 9]. Work on systematic belief consistency and knowledge editing likewise suggests that factual commitments do not behave as isolated entries in a knowledge base: accepting or revising one proposition can affect related propositions [26, 10]. Consider an LLM that assigns similar support to “Paris is in France” and “Oslo is in Norway.” If it also considers “Oslo is in Sweden” plausible, the Oslo claim sits alongside an incompatible proposition the model has not ruled out. The Paris and Oslo claims may therefore look equally strong on their own even though one is more fragile within the model’s broader belief system. We refer to this difference as belief stability. Existing work captures important pieces of this picture without directly measuring it. Representation-based studies find that veracity and epistemic uncertainty are systematically reflected in LLM hidden states [31, 3, 47, 38, 48], while LLMs do not reliably distinguish epistemic concepts such as belief, knowledge, and fact [42]. Behavioral evaluations, meanwhile, reveal when outputs change under particular external perturbations [19, 39, 27, 30]. What these approaches do not capture is how a belief is situated relative to the model’s other epistemic commitments. A belief may receive strong support on its own yet become fragile in light of other propositions the model regards as plausible. Stability is therefore a relational property of a belief within a broader belief system. Formal epistemology provides a principled basis for making this distinction precise. The Lockean Thesis connects graded credence to categorical belief through a probability threshold, with sufficiently probable propositions counting as beliefs [20, 28]. Leitgeb’s Humean Thesis strengthens this threshold-based approach by requiring a belief to remain sufficiently probable when conditioned on propositions an agent does not disbelieve [29]. We adapt this idea to LLMs by defining the graded belief stability of a believed proposition P as the fraction of the model’s non-disbeliefs x for which Pr⁡(P∣x) (P x) remains above its belief threshold. Graded stability thus distinguishes how strongly a model supports P on its own from how broadly that support persists across its broader epistemic state. Estimating graded stability requires eliciting conditional belief probabilities from LLM representations. Supervised probes can recover veracity-related information from hidden states [31, 3, 38, 12], but Pr⁡(P∣x) (P x) is not directly observable. We therefore introduce a trivalent Direct Conditional estimator grounded in established accounts of indicative conditionals and non-bivalent probability [17, 11]. For each belief–non-disbelief pair (P,x)(P,x), we construct a conditional statement of the form “Given x, P” and probe its hidden representation to estimate the conditional probabilities needed to compute graded stability. We evaluate graded belief stability across 1212 instruction-tuned LLMs from four model families, ranging from 33B to 7272B parameters, and across three domains: City Locations, Medical Indications, and Word Definitions. We test whether graded stability captures structure beyond individual belief probability and whether the underlying probability estimates are approximately probabilistically coherent. We then characterize its variation across semantic domains and evaluate whether it is associated with resistance to conversational challenge. Our work makes five core contributions: 1. Formalization: We introduce graded belief stability as a statement-level measure of how well an LLM’s belief persists across its broader belief system. 2. Measurement: We develop a representation-based Direct Conditional estimator of conditional belief probability for computing graded stability. 3. Validation: We show that graded stability captures systematic information beyond individual belief probability, where 99.0%99.0\% of model-pair residual correlations are positive. The measured probability systems are also approximately coherent, with median distance from coherence ranging from 0.0150.015 to 0.0620.062 across domains. 4. Characterization: We find that City Locations is consistently the most stable domain, with a median across models of mean graded stability of 0.880.88, compared with 0.710.71 for Medical Indications and 0.740.74 for Word Definitions. 5. Behavioral validity: Among beliefs matched on individual belief probability, lower-stability beliefs show greater mean movement under repeated conversational challenge for 83.3%83.3\% of model–domain settings. This work captures a dimension of LLM reliability that individual belief probability alone cannot. Two factual judgments can receive similar support while occupying very different positions within the model’s broader belief system, making one substantially more fragile than the other. Graded belief stability quantifies this relational property, extending reliability assessment beyond how strongly an LLM supports a claim to how robustly that belief is supported within the model’s broader system of beliefs. Figure 1: Measuring graded belief stability in LLMs. (a) For each statement s, we extract an internal LLM representation and use a probe to estimate a distribution over True (T, orange), False (F, olive), and Neither (N, purple). (b) The probe’s predicted states y y determine whether each statement is treated as a belief (y^=T y=T), disbelief (y^=F y=F), or suspended belief (y^=N y=N), yielding the belief set ℬℳB_M and non-disbelief set ℳX_M. We define the empirical belief threshold tℳt_M as the minimum True-class probability among statements classified as beliefs. We then estimate Prℳ⁡(P∣x) _M(P x) for each P∈ℬℳP _M and x∈ℳx _M. (c) The Direct Conditional approach probes representations of conditional statements of the form “Given x, P.”, and (d) we compute graded stability γℳ(P) _M(P) as the fraction of conditioning propositions for which the conditional probability of P remains above tℳt_M. 2 Results We begin by formalizing graded belief stability γ and introducing the Direct Conditional representation-based estimator for conditional belief. We then validate the measure, showing that it captures structure beyond individual belief probability and is approximately probabilistically coherent. We next characterize variation in γ across domains, before testing whether graded stability distinguishes behavioral resilience among beliefs matched on individual belief probability. 2.1 Formalizing graded belief stability We define graded belief stability by adapting the stability theory of belief from formal epistemology, in which a belief is stable when it persists after conditioning on propositions the agent does not reject [28]. We extend this binary notion to a statement-level measure of the degree to which an LLM’s belief persists across conditioning contexts. Let ℳM denote an LLM and let Prℳ⁡(s) _M(s) denote the probability that ℳM assigns to a statement s being true. Following the Lockean Thesis, we take categorical belief to correspond to sufficiently high probability relative to a belief threshold tℳt_M [20, 28]. In our empirical implementation below, we first identify categorical belief from a trivalent probe and then instantiate tℳt_M from the resulting belief set rather than imposing a universal cutoff. We denote the set of beliefs held by ℳM as ℬℳB_M and the larger set of propositions that ℳM does not disbelieve, including both beliefs and suspended beliefs, as ℳX_M. The Humean Thesis strengthens this static notion of belief by requiring beliefs to remain sufficiently probable when considered alongside the agent’s other non-disbeliefs [28, 29]. Intuitively, a stable belief should remain above the belief threshold even when considered alongside propositions the model itself does not reject. Specifically, a belief P∈ℬℳP _M is considered stable if conditioning on any x∈ℳ∖Px _M \P\ does not lower its probability below the belief threshold: (1) P is stable⇔Prℳ⁡(P∣x)>tℳ∀x∈ℳ∖P.P is stable _M(P x)>t_M ∀ x _M \P\. Thus, propositions with similar Prℳ⁡(P) _M(P) may nevertheless differ substantially in stability. Binary stability, however, treats all violations identically. A proposition that falls below the belief threshold under a single conditioning statement is indistinguishable from one that does so under nearly all of them. We therefore define the graded belief stability of P as (2) γℳ(P):=|x∈ℳ∖P:Prℳ⁡(P∣x)>tℳ||ℳ∖P|. _M(P):= | \x _M \P\: _M(P x)>t_M \ | |X_M \P\ |. Thus, γℳ(P)∈[0,1] _M(P)∈[0,1] is the proportion of non-disbeliefs under which belief in P is preserved. Higher values indicate beliefs that persist more broadly across the model’s belief system, distinguishing graded stability from the probability assigned to P in isolation. 2.2 Measuring graded belief stability in LLMs Measuring graded stability requires identifying an LLM’s beliefs and non-disbeliefs and estimating Prℳ⁡(P∣x) _M(P x) for belief–non-disbelief pairs. We use calibrated supervised probes over LLM hidden representations to obtain these probability estimates, since unconstrained prompted judgments need not yield calibrated probability estimates and can be sensitive to prompt formulation and conversational context [44, 19, 39, 30]. For each statement sis_i, we extract hidden representations across layers ℓ of model ℳM and train the sparse-aware multiple-instance learning (sAwMIL) probe [38], which models True (T), False (F), and Neither (N) as distinct veracity classes.11 1 Full activation extraction, probe training, and layer-selection procedures are described in Sec. 4.1, and layer-wise probe performance is reported in Sec. D.3. We additionally reproduce the analyses using Support Vector Machine (SVM) [13] and Mass Mean [31] probes in Sec. I.3. After calibration, the probe returns the trivalent distribution (3) ℳ(si)=[πT(si),πF(si),πN(si)], π_M(s_i)= [ _T(s_i), _F(s_i), _N(s_i) ], where πc(si) _c(s_i) is the probe-assigned probability of class c. The predicted epistemic state is then (4) y^i=arg⁡maxc∈T,F,Nπc(si). y_i= _c∈\T,F,N\ _c(s_i). We interpret y^i=T y_i=T as belief, y^i=F y_i=F as disbelief, and y^i=N y_i=N as suspension of belief (Fig. 1(a–b)). The probe therefore determines the empirical belief and non-disbelief sets directly: (5) ℬℳ=si:y^i=T,ℳ=si:y^i∈T,N.B_M=\s_i: y_i=T\, _M=\s_i: y_i∈\T,N\\. This trivalent operationalization retains an explicit state for suspended belief rather than forcing every proposition into a binary believed/not-believed distinction. We then associate the probe-defined categorical belief set with an empirical Lockean-style threshold. For each model and domain, we define (6) tℳ:=minsi∈ℬℳ⁡πT(si).t_M:= _s_i _M _T(s_i). Importantly, tℳt_M does not determine which propositions belong to ℬℳB_M. Rather, categorical belief is already defined by the probe’s trivalent prediction in Equation (5). Instead, the threshold translates this categorical belief set into a scalar criterion for belief persistence. Specifically, tℳt_M is the most stringent True-class support threshold satisfied by every proposition that the probe identifies as a belief. This anchors the persistence criterion to the model’s own empirical belief boundary rather than imposing an externally chosen probability cutoff. Because πT _T is one component of a calibrated three-class distribution over True, False, and Neither, tℳt_M will not necessarily exceed 0.500.50, but is instead lower-bounded by 0.330.33. The resulting threshold should therefore be understood as a model-, domain-, and probe-specific support floor induced by the empirical belief set, rather than as a universal normative probability threshold (Fig. 1(b), Sec. 4.2). We next estimate conditional belief probability for every belief P∈ℬℳP _M and non-disbelief x∈ℳ∖Px _M \P\. The Direct Conditional approach represents P|xP x using the conditional statement “Given x, P.” and probes its internal representation directly (Fig. 1(c)). A three-class sAwMIL probe predicts whether the resulting conditional is True, False, or Neither using Cooper–Cantwell training labels [11, 8, 18]. For each pair (P,x)(P,x), the probe returns the trivalent distribution (7) ℳdirect(P,x)=[πTdirect(P,x),πFdirect(P,x),πNdirect(P,x)]. π^direct_M(P,x)= [ _T^direct(P,x), _F^direct(P,x), _N^direct(P,x) ]. We use the conditional’s True-class mass to define our Direct Conditional estimator (8) Pr^ℳdirect(P∣x):=πTdirect(P,x). ^\,direct_M(P x):= _T^direct(P,x). We use Pr^ℳdirect(P∣x) ^\,direct_M(P x) as an empirical operational estimator of conditional belief probability rather than as an algebraic reconstruction of the normalized Cooper–Cantwell probability of a trivalent conditional. In particular, we do not renormalize over the True and False states. This keeps conditional support on the same three-class scale used to define the empirical belief threshold tℳt_M and allows probability mass assigned to Neither to count against persistence of the original belief. Section 4.3 motivates this choice and contrasts it with our complementary Joint-to-Conditional estimator, which instead normalizes over determinate conditional states. Results on the secondary Joint-to-Conditional estimator are reported in Section I.2.2. Replacing Prℳ⁡(P∣x) _M(P x) in Eq. (2) with the Direct Conditional estimate in Eq. (8) yields our empirical Direct Conditional stability score γℳdirect(P) _M^direct(P) (Fig. 1(d); Sec. 4.4). We apply this probing-based framework to 1212 instruction-tuned22 2 Matched base-model analyses are reported in Section I.1. LLMs spanning four model families and ranging from 33B to 7272B parameters. We evaluate three domains with differing epistemic characteristics: City Locations, consisting of comparatively clear-cut factual relations; Medical Indications, requiring specialized factual knowledge; and Word Definitions, containing comparatively ambiguous lexical relations [38]. Figure 2: Relationship between individual belief probability and graded stability. We show the held-out R2R^2 for predicting graded stability γℳ(P) _M(P) from individual belief probability πT(P) _T(P) for (a) City Locations, (b) Medical Indications, and (c) Word Definitions across Llama (red), Gemma (blue), Mistral (yellow), and Qwen (green) model families. We also report (d) distributions of pairwise Spearman correlations ρ between LLMs’ residual graded stability within each domain, where black lines denote the median correlation across model pairs. Individual belief probability predicts a meaningful proportion of graded stability, but graded stability captures proposition-level variation beyond individual belief probability that is systematically shared across models. Figure 3: Distance of measured probabilities from probabilistic coherence. Violins show the distribution across (P,x)(P,x) pairs of the root-mean-square (RMS) adjustment dCCKd_CCK required to project the measured probability distributions for individual statements and conditionals onto the nearest CCK-coherent probability system for (a) City Locations, (b) Medical Indications, and (c) Word Definitions for Llama (red), Gemma (blue), Mistral (yellow), and Qwen (green) model families. Horizontal black lines denote within-model medians. Smaller values indicate greater probabilistic coherence, with dCCK=0d_CCK=0 corresponding to exact coherence. Although exact coherence is uncommon, only modest adjustments are generally required to reconcile the independently measured probabilities with a coherent trivalent probability system. 2.3 Validating graded stability We validate graded stability along two complementary dimensions: whether it captures information beyond individual belief probability and whether the probability estimates underlying it are approximately probabilistically coherent. Main-text results use the Direct Conditional estimator and the sAwMIL probe. Joint-to-Conditional results and analyses using the SVM and Mass Mean probes appear in Section I. 2.3.1 Graded stability captures information beyond belief probability A central motivation for graded stability is that it captures how a belief behaves in the context of an LLM’s broader belief system rather than simply how strongly that belief is held in isolation. We therefore first ask how well the individual belief probability πT(P) _T(P) predicts γℳ(P) _M(P) for beliefs P∈ℬℳP _M. If graded stability were simply a transformation of belief probability, then a sufficiently flexible function of πT(P) _T(P) should accurately predict γℳ(P) _M(P) for beliefs not used to fit that function. For each model and domain, we attempt to capture such a function using a natural cubic spline, (9) γ^ℳ(P)=fℳ(πT(P)), γ_M(P)=f_M\! ( _T(P) ), with three degrees of freedom. The spline allows belief probability and stability to have a smooth nonlinear relationship without assuming a linear relationship. We use five-fold cross-validation to generate out-of-sample predictions γ^ℳ(P) γ_M(P) and evaluate performance with R2R^2 (Sec. 4.5.1). We fix the spline complexity across all models and domains and show that the results are qualitatively unchanged across alternative spline complexities in Section F.1. Individual belief probability consistently predicts a meaningful but incomplete proportion of the observed variation in graded stability (Fig. 2(a–c)). The median held-out R2R^2 across models is 0.460.46 for City Locations, 0.390.39 for Medical Indications, and 0.470.47 for Word Definitions. Thus, although belief probability is informative about stability, more than half of the proposition-level variation in γℳ(P) _M(P) remains unexplained by the probability assigned to P in isolation. Graded stability therefore captures substantial variation beyond individual belief strength. We next ask whether this unexplained variation is systematic. For each proposition P, we define its residual stability as (10) rℳ(P):=γℳ(P)−γ^ℳ(P),r_M(P):= _M(P)- γ_M(P), where positive values identify propositions that are more stable than predicted from their individual belief probability and negative values identify propositions that are less stable than predicted. We align pairs of LLMs on propositions believed by both models and compute the Spearman correlation between their residual stability scores (Fig. 2(d)). Across domains, 99.0%99.0\% of model-pair residual correlations are positive, with median correlations of ρ=0.23ρ=0.23 for City Locations, ρ=0.30ρ=0.30 for Medical Indications, and ρ=0.31ρ=0.31 for Word Definitions. All three medians exceed a proposition-shuffled permutation null (p=0.001p=0.001; Sec. 4.5.1). Models therefore tend to find the same propositions unusually stable or fragile even after accounting for how strongly they believe those propositions in isolation. This suggests that the variation unique to graded stability contains reproducible proposition-level structure rather than only model-specific measurement noise. Because models are evaluated on the same benchmark propositions using a common probing framework, however, this analysis does not by itself distinguish model-internal relational structure from proposition-level or measurement factors shared across models. 2.3.2 Graded stability probability estimates are approximately coherent Because graded stability is defined within a probabilistic framework, we next ask whether the independently measured individual and conditional probability distributions are pairwise compatible. Under Cooper–Cantwell–Kleene (CCK) trivalent probability, the distributions for P, x, and P|xP x are coherent when they can all be generated by a single latent 3×33× 3 probability distribution over the trivalent states of (P,x)(P,x). For each measured pair, we collect these probabilities into the nine-dimensional vector (11) P,x=[ℳ(P),ℳ(x),ℳdirect(P,x)].v_P,x= [ π_M(P), π_M(x), π^direct_M(P,x) ]. Under CCK semantics, such a latent joint distribution exists if and only if the measured probabilities satisfy (12) πTdirect(P,x)≤πT(P),πFdirect(P,x)≤πF(P),πF(x)≤πNdirect(P,x),πNdirect(P,x)≤πN(P)+πF(x). _T^direct(P,x)≤ _T(P), _F^direct(P,x)≤ _F(P), _F(x)≤ _N^direct(P,x), _N^direct(P,x)≤ _N(P)+ _F(x). We use these constraints to test exact coherence separately for every (P,x)(P,x) pair (Sec. 4.5.2); a proof of the coherence conditions is provided in Sec. F.2.1. Exact coherence is extremely rare: across all model–domain combinations, no more than 0.21%0.21\% of measured pairs satisfy all four constraints exactly. Full exact-coherence rates and individual constraint violations are reported in Section F.2. Because exact coherence is a binary criterion that treats arbitrarily small and large violations identically, we additionally quantify each pair’s distance from the set of coherent probability systems. Following prior work that characterizes approximate coherence in terms of distance to the nearest coherent credence function [34], we extend this distance-based approach to the CCK trivalent setting and define (13) dCCK(P,x):=19‖P,x−Π(P,x)‖22,d_CCK(v_P,x):= 19 \|v_P,x- _C(v_P,x) \|_2^2, where C is the set of CCK-coherent probability vectors and Π() _C(v) is the closest vector in this set to the observed probability vector v.33 3 Full optimization and convergence details are provided in Sec. 4.5.2. Thus, dCCK=0d_CCK=0 corresponds to exact coherence, while a value such as dCCK=0.05d_CCK=0.05 means that reaching the nearest coherent probability system requires a root-mean-square (RMS) adjustment of 0.050.05 per probability component. Across models, the median model-level dCCKd_CCK is 0.0150.015 for City Locations, 0.0520.052 for Medical Indications, and 0.0620.062 for Word Definitions, corresponding to RMS adjustments of 1.51.5, 5.25.2, and 6.26.2 percentage points per probability component, respectively. Thus, despite near-zero exact-coherence rates, the measured probability systems typically require quantitatively modest adjustments to reach the nearest coherent CCK representation. Because graded stability thresholds these probabilities at tℳt_M, however, a small component-wise RMS adjustment does not imply that the corresponding absolute stability score would be unchanged after projection. We therefore interpret dCCKd_CCK as a diagnostic of the measured probability system rather than as a robustness bound on γℳ(P) _M(P). Figure 4: Variation in graded belief stability across domains. Violin plots show the proposition-level distribution of graded stability γℳdirect(P)γ^direct_M(P) for each of the 1212 instruction-tuned LLMs in (a) City Locations, (b) Medical Indications, and (c) Word Definitions for Llama (red), Gemma (blue), Mistral (yellow), and Qwen (green) model families. Horizontal black lines denote within-model medians. Stability distributions differ substantially across the three benchmark domains, with City Locations exhibiting the highest median stability for 1111 of the 1212 models. 2.4 Graded belief stability varies with domain Having established that graded stability captures information beyond individual belief probability and is approximately probabilistically coherent, we next ask how stability varies across beliefs and models by characterizing the distribution of γℳ(P) _M(P) across each model’s beliefs (Fig. 4). Stability differs substantially across semantic domains. Across models, the median of model-level mean graded stability is 0.880.88 for City Locations, compared with 0.710.71 for Medical Indications and 0.740.74 for Word Definitions. Comparing the within-model median stabilities across domains, City Locations has the highest median stability for 1111 of the 1212 LLMs. The only exception is Llama-3.2-3b, whose median stability is highest for Word Definitions. Within these benchmark-specific belief systems, City Locations beliefs are therefore consistently more stable than beliefs in the Medical Indications and Word Definitions domains. This pattern is consistent with differences in the semantic structure of the domains, as City Locations contains comparatively clear-cut factual relations, whereas Medical Indications requires more specialized factual knowledge and Word Definitions contains comparatively ambiguous lexical relations. The same broad domain structure is visible in matched pretrained base models (Sec. I.1.1). These differences extend beyond shifts in average stability to the proposition-level distributions themselves. In City Locations, stability is concentrated near the upper end of the range for most models, although long lower tails show that even this comparatively stable domain contains individual beliefs that are substantially more fragile (Fig. 4(a)). Medical Indications and Word Definitions exhibit broader distributions and considerably greater variation across models. In Medical Indications in particular, several of the smallest models, including Llama-3.2-3b, Gemma-7b, and Qwen-2.5-7b, have markedly lower median stability than larger models from the same families (Fig. 4(b)). Some of the between-model variation in Figure 4 also suggests a possible relationship with model scale. In City Locations and Medical Indications, several of the smallest models within a family have lower median stability than their larger counterparts (Fig. 4(a–b)). This pattern is not consistent across all families, however, and is largely absent in Word Definitions. We examine model scale more directly in Section H, where we demonstrate that neither parameter count nor model depth shows a consistent relationship with graded stability across domains. Thus, model scale may contribute to some of the variation observed within families, but it does not provide a general explanation for differences in γℳ(P) _M(P). Figure 5: Behavioral resilience among beliefs matched on individual belief probability. Bars show the mean difference in behavioral movement ΔM M between matched lower- and higher-stability beliefs for each of the 1212 instruction-tuned LLMs, with error bars denoting ±1± 1 standard error. Positive values indicate that lower-stability beliefs exhibit greater movement under conversational challenge. Results are shown for (a) City Locations, (b) Medical Indications, and (c) Word Definitions for Llama (red), Gemma (blue), Mistral (yellow), and Qwen (green) model families. Across domains, 83.3%83.3\% of model–domain settings show greater behavioral movement for lower-stability beliefs. 2.5 Graded stability is associated with behavioral movement beyond belief probability Graded stability is defined from relationships among internal representations, but its usefulness depends in part on whether those differences correspond to independently measured model behavior. We therefore test whether γℳ(P) _M(P) distinguishes resistance to conversational challenge among beliefs assigned similar individual probabilities. Importantly, the challenge prompts are not conditioning propositions x∈ℳx _M and do not instantiate the formal operation used to compute graded stability. They instead provide an external behavioral perturbation, allowing us to ask whether representation-level stability generalizes to a qualitatively different form of belief pressure. We construct pairs of believed propositions with similar πT(P) _T(P) within each model and domain using sequence-blocked matching (Sec. 4.6, G). We match propositions by maximizing the number of non-overlapping pairs while minimizing their total absolute difference in belief probability. We then identify the propositions PHP_H and PLP_L with higher and lower graded stability within each pair. Our primary analysis retains pairs for which both propositions’ initial behavioral judgments agree with their individual-statement probe classifications. We evaluate each proposition in a four-stage interaction consisting of an initial binary True/False judgment followed by three natural-language challenges. Propositions are randomly assigned one of six possible challenge orderings to control for order effects (Sec. 4.6). The challenges model conversational pressure from a disagreeing user rather than literal factual conditioning on another proposition x. They therefore provide external behavioral validation of graded stability rather than an empirical realization of Prℳ⁡(P∣x) _M(P x). We record the normalized probability u0(P)u_0(P) assigned to the model’s initial judgment and denote the probability assigned to that same response after k challenges by uk(P)u_k(P). We quantify behavioral movement as (14) M⁡(P):=13∑k=13|uk(P)−uk−1(P)|,M(P):= 13 _k=1^3 |u_k(P)-u_k-1(P) |, where larger M⁡(P)M(P) indicates greater fluctuation in support for the model’s initial response. Because the differences are absolute, increases and decreases in support both contribute to M⁡(P)M(P). It therefore measures behavioral movement rather than specifically loss of support or answer flipping. For each probability-matched pair, we then compare the behavioral movement of the lower- and higher-stability propositions: (15) ΔM:=M⁡(PL)−M⁡(PH). M:=M(P_L)-M(P_H). Thus, ΔM>0 M>0 indicates that the lower-stability belief exhibits greater behavioral movement under challenge. Because PHP_H and PLP_L are identified only after matching on individual belief probability, this comparison tests whether graded stability distinguishes behavioral resilience beyond differences in πT(P) _T(P). Mean ΔM M is positive for 10/1210/12 models in City Locations, 10/1210/12 in Medical Indications, and 10/1210/12 in Word Definitions. Median effects are 0.0110.011, 0.0260.026, and 0.0220.022, respectively. However, uncertainty is substantial for many individual estimates: the 95%95\% bootstrap confidence intervals are entirely positive in 11/3611/36 settings, entirely negative in one, and include zero in the remainder (Sec. G.2). Thus, graded stability shows a broadly consistent directional association with behavioral resilience, although the magnitude and precision of the effect vary across models and beliefs. 3 Discussion This work asks whether the stability of an LLM judgment can be characterized by its relationship to the model’s other beliefs rather than by individual belief probability alone. Across 1212 instruction-tuned LLMs and three semantic domains, individual belief probability explains a meaningful but incomplete fraction of graded stability. The remaining variation is also not arbitrary. After accounting for individual belief probability, different models tend to identify the same propositions as unusually stable or fragile, and among beliefs deliberately matched on individual belief probability, lower graded stability is generally associated with greater movement under conversational challenge. These results support the central distinction motivating graded stability: how strongly an LLM supports a proposition in isolation and how robustly that support persists in the context of its other beliefs are related, but non-equivalent, properties. A central implication of our results is that this relational structure depends strongly on what is believed. City Locations is the most stable domain for 1111 of the 1212 instruction-tuned models, and the same broad domain structure appears in matched pretrained base models and under the alternative SVM probe (Secs. I.1.1, I.3). By contrast, neither parameter count nor model depth shows a consistent relationship with graded stability across domains (Sec. H). Instruction tuning, however, is associated with greater stability in most matched base versus instruction-tuned comparisons, but the magnitude and even direction of this difference vary across models and domains (Sec. I.1.2). These results argue against treating graded stability as a generic property that simply increases with model capability. Instead, stability appears to reflect an interaction among the semantic structure of the belief, the model in which it is represented, and the post-training regime applied to that model. A key next step is therefore to identify the semantic properties that systematically confer stability or fragility, and to determine whether those properties generalize beyond the controlled domains studied here. Our robustness analyses further suggest that the most reproducible feature of graded stability is its relative structure rather than its absolute numerical scale. The Direct Conditional and Joint-to-Conditional estimators operationalize conditional belief probability in different ways, yet they produce positively correlated stability rankings in every setting (Sec. I.2.3). At the same time, their absolute stability magnitudes differ substantially, with Joint-to-Conditional scores compressed toward 1.01.0 (Sec. I.2). Absolute graded-stability values, and differences in their magnitudes across domains, should therefore be interpreted within a particular estimator and probe rather than as measurement-invariant quantities. Probe choice produces a related distinction. The SVM probe broadly reproduces the main findings obtained with sAwMIL, whereas the Mass Mean probe yields weaker domain separation, larger departures from probabilistic coherence, and less consistent behavioral associations (Sec. I.3). This weaker replication is not surprising because Mass Mean is not designed to recover calibrated trivalent probability distributions, which graded stability explicitly requires. These analyses suggest that a reproducible relational signal persists across operationalizations, even though its absolute numerical realization depends on the conditional estimator and probe. Faithfully recovering that signal therefore depends on measurement choices that preserve the probabilistic properties required by the construct. The behavioral challenge experiment tests whether graded stability distinguishes resistance to repeated conversational challenge among beliefs with similar individual belief probabilities. Among beliefs matched on individual belief probability, lower-stability beliefs exhibit greater mean movement under repeated challenge in 83.3%83.3\% of instruction-tuned model–domain settings. The consistency of this direction across models is notable, but the individual effects are heterogeneous and often imprecisely estimated: most 95%95\% bootstrap intervals include zero (Sec. G.2). We consequently view the behavioral experiment as evidence that graded stability captures differences in resilience beyond individual belief probability, rather than as evidence for a universal mapping from γ to conversational behavior. More broadly, resistance to revision is not always desirable. Instability can indicate unwanted fragility, but revision may be appropriate when new information exposes an error or creates a legitimate conflict with existing commitments. Conversely, an incorrect judgment that resists all counterevidence would be highly stable without being reliable. Graded stability is therefore best understood as a descriptive property of an LLM belief system rather than a score to optimize. Distinguishing unwarranted fragility from appropriate responsiveness will require manipulating the relevance, reliability, and evidential force of the information introduced to the model. Our results also illustrate the value and limitations of applying formal probabilistic theories to LLM representations. LLMs are autoregressive sequence models, not agents explicitly trained to maintain a globally coherent probability system over propositions, and we find correspondingly little exact CCK coherence. However, the measured individual belief and conditional probability distributions generally require only modest adjustments to reach the nearest coherent system. Formal coherence therefore does not have to be treated as an all-or-nothing claim about whether an LLM literally instantiates a particular epistemic theory. It can instead provide a reference structure against which empirical representations are measured. More generally, the calibrated probe probabilities used throughout should be understood as representation-based estimates of epistemic support rather than direct observations of a uniquely defined latent credence distribution of the LLM. From this perspective, approximate coherence makes the formal framework useful, while departures from coherence become objects of study rather than reasons to abandon it. The same principle applies to the definition of graded stability itself. Our formulation weights each admissible conditioning proposition equally and adopts Cooper–Cantwell–Kleene semantics for trivalent conditionals. However, alternative weighting schemes or conditional semantics would define related but distinct notions of stability. Understanding which of these notions best predicts particular forms of model behavior is an important direction for future work. Finally, our experiments deliberately use controlled factual domains containing explicit True, False, and Neither cases. This structure is essential to the present study as it provides comparatively well-defined trivalent supervision, permits systematic construction of conditional training examples, and supplies propositional objects that map cleanly onto the formal account of belief and conditioning. The same control limits ecological scope. Accordingly, differences in γ across these domains reflect both the target propositions and the composition of the benchmark-specific non-disbelief sets over which stability is evaluated, rather than an intrinsic stability of the semantic domain alone. Rich natural-language claims may contain multiple propositions, presuppositions, context-sensitive content, or ambiguous conditioning relations, and real conversational contexts need not resemble the within-domain non-disbelief sets studied here. Extending graded stability to richer benchmarks and cross-domain belief systems will therefore require more than simply applying the same pipeline to longer text; it will require defining what the relevant propositions and conditioning relations are. What remains consistent across these methodological and formal choices is the distinction between the strength of an individual belief and its stability within a broader system of beliefs. Evaluations of factual reliability typically ask whether a model’s answer is correct, how strongly it is supported, or whether confidence tracks accuracy. Our results show that relationships among beliefs provide complementary information. Propositions with similar individual support can differ systematically in stability, different models share substantial structure in which propositions are unusually stable or fragile, and, under our primary operationalization, these differences are associated with downstream behavioral resilience. Graded belief stability therefore provides a relational view of factual reliability that complements existing measures of accuracy and uncertainty. Characterizing this structure may help explain not only what an LLM appears to believe, but how those beliefs are situated within a broader epistemic system and which are most susceptible to change as their informational context shifts. 4 Methods We first describe the LLMs, datasets, and probing framework used throughout our experiments. We then detail how we estimate individual statement probabilities, construct empirical belief and non-disbelief sets, and measure conditional probabilities. Finally, we describe the analyses used to validate graded stability, characterize its variation across domains and models, and evaluate its relationship with behavioral resilience under conversational challenge. All code and datasets associated with the experiments can be found at https://github.com/samanthadies/graded_stability. 4.1 Models and data 4.1.1 Large language models We evaluate 1212 open-weight, instruction-tuned LLMs from four model families: Gemma, Llama, Mistral, and Qwen. For Gemma, we consider 77B, 99B, and 2727B. For Llama, we use 33B, 88B, and 7070B. For Mistral, we use 77B, 1212B, and 2424B. Finally, for Qwen we use 77B, 1414B, and 7272B. We report the exact model versions and the number of decoder layers in Table 1. Matched base models are listed in Section C. All models are loaded in evaluation mode with CUDA and BF16. We render statements using the models’ native chat templates as single user messages without an assistant response. Official Name Short Name # Layers # Parameters Release Date Source Citation Gemma-77b-it gemma-7b 2828 8.548.54 B Feb 2121, 20242024 Google [22] Gemma-22-99b-it gemma-2-9b 4242 9.249.24 B Jun 2727, 20242024 Google [22] Gemma-22-2727b-it gemma-2-27b 4646 27.2327.23 B Jun 2727, 20242024 Google [22] Llama-3.23.2-33b-Instruct llama-3.2-3b 2828 3.213.21 B Sep 2525, 20242024 Meta [23] Llama-3.13.1-88b-Instruct llama-3.1-8b 3232 8.038.03 B Jul 2323, 20242024 Meta [23] Llama-3.13.1-7070b-Instruct llama-3.1-70b 8080 70.5570.55 B Jul 2323, 20242024 Meta [23] Mistral-77b-Instruct-v0.30.3 mistral-7b 3232 7.257.25 B May 2222, 20242024 Mistral AI [24] Mistral-Nemo-Instruct-2407 mistral-12b 4040 12.2512.25 B Jul 1818, 20242024 Mistral AI [32] Mistral-Small-3.13.1-2424B-Instruct-2503 mistral-3.1-24b 4040 23.5723.57 B Mar 1717, 20252025 Mistral AI [33] Qwen2.52.5-77B-Instruct qwen-2.5-7b 2828 7.627.62 B Sep 1919, 20242024 Alibaba Cloud [46, 36] Qwen2.52.5-1414B-Instruct qwen-2.5-14b 4848 14.8014.80 B Sep 1919, 20242024 Alibaba Cloud [46, 36] Qwen2.52.5-7272B-Instruct qwen-2.5-72b 8080 72.7072.70 B Sep 1919, 20242024 Alibaba Cloud [46, 36] Table 1: LLMs used in graded stability experiments. We list the official names of the LLMs according to the HuggingFace repository [45], where they are publicly available. We further specify the shortened name used throughout the paper, the number of layers, parameter count, release date, source organization, and official citation. 4.1.2 Datasets Our datasets, introduced in [38], include three domains: City Locations, Medical Indications, and Word Definitions (Tab. 2). Each domain contains True, False, and Neither statements. The Neither statements pair synthetic entities generated using character n-gram Markov-chain models fit to the corresponding real entity sets and validated to reduce the likelihood that they correspond to existing entities. They therefore provide controlled cases for which an LLM should ideally suspend belief, making the datasets well suited to our trivalent belief framework. These synthetic cases instantiate a controlled form of epistemic indeterminacy associated with unfamiliar content. However, they do not exhaust the ways in which an LLM may suspend judgment in natural-language settings. Further dataset construction and validation details are provided in Section B.1. We use the fixed training, calibration, and test partitions introduced with the original datasets [38] (Sec. B.2), containing approximately 55%55\%, 20%20\%, and 25%25\% of statements, respectively, with entities kept exclusive across splits. The training split is used to fit each probe, while the calibration split is used to map raw probe scores to calibrated probability distributions for individual statements, ℳ(si) π_M(s_i), and Direct Conditional statement pairs, ℳdirect(si,sj) π^direct_M(s_i,s_j), according to the procedure in Section D.1. The fitted probes and calibration mappings are then applied to test statements and statement pairs. The statements in the train and calibration sets are reused to train and calibrate the individual-statement probe and the Direct Conditional probe. The test set statements form the basis of our belief and non-disbelief sets ℬℳB_M and ℳX_M. Dataset True False Synthetic Examples City Locations A: 13921392 N: 13761376 A: 13581358 N: 13741374 A: 876876 N: 876876 T. The city of Surat is located in India. F. The city of Palembang is located in the Dominican Republic. S. The city of Norminsk is located in Jamoates. Medical Indications A: 14391439 N: 15221522 A: 15231523 N: 14191419 A: 478478 N: 522522 T. Pentobarbital is indicated for the treatment of insomnia. F. Vancomycin is not indicated for the treatment of lower respiratory tract infections. S. Alumil is indicated for the treatment of reticers. Word Definitions A: 12341234 N: 12351235 A: 12771277 N: 12541254 A: 17471747 N: 17531753 T. Hoagy is a synonym of an Italian sandwich. F. Decalogue is an astronomer. S. Dostab is a scencer. Table 2: Summary of datasets and statement types. Number of affirmative (A) and negated (N) statements across the three domains, along with examples. Each dataset includes True (T), False (F), and Synthetic (S) statements. Synthetic statements serve as Neither statements constructed to minimize prior LLM exposure. An earlier version of this table was introduced in [38]. 4.1.3 Probes Each statement sis_i is fed into a model ℳM, resulting in an activation zi(ℓ)z_i^( ) at each layer ℓ . We train each probe over the activations corresponding to the training statements, using the ground-truth True, False, and Neither labels as training targets. The probes produce raw score vectors ⁡(si)r(s_i) over the K classes. Because these raw scores cannot be directly interpreted as probabilities, we fit a multinomial logistic-regression calibration model on the calibration split that maps each raw score vector ⁡(si)r(s_i) to a calibrated class-probability distribution ℳ(si) π_M(s_i) over the K classes (Sec. D.1). Complete training specifications and hyperparameters for sAwMIL are reported in Section D.2, and its layer-selection procedure and selected layers are reported in Section D.3. Corresponding details for the secondary SVM and Mass Mean probes are provided in Section I.3. Throughout, we treat these calibrated probe probabilities as representation-based estimates of the model’s epistemic support over the supervised trivalent states. Calibration makes the probe outputs probabilistically interpretable with respect to those states, but does not by itself establish that they are identical to a uniquely defined latent credence distribution of the LLM. Primary probe: sAwMIL Our main probe is the sparse-aware multiple-instance learning (sAwMIL) probe [38]. sAwMIL is a max-margin probe that is trained over a bag of token-level representations for each statement, rather than a single statement-level vector. It was also designed specifically to handle more than two classes. For K classes, sAwMIL trains K separate one-versus-all heads which are combined into a multiclass classifier using a softmax function. Secondary probes: SVM and Mass Mean We use two additional probes, an SVM and the Mass Mean probe [31], to test whether our stability results are qualitatively robust to the choice of probe. The corresponding analyses are reported in Sec. I.3. Like sAwMIL, the SVM is a max-margin classifier that learns K one-versus-all heads. However, sAwMIL is a multi-instance learning probe, while the SVM operates only on the activation corresponding to a given statement’s final token. The Mass Mean probe is also a single-instance probe that operates on the activation corresponding to a given statement’s final token. It was initially designed as a binary probe. However, we extend it to a multiclass probe so that it can be used in our three-class, trivalent probability setting. For each class j∈0,…,K−1j∈\0,…,K-1\, the Mass Mean probe takes the mean representation of the training examples in class j, μj _j, the mean representation of the examples in all other classes, μ¬j _ j, and computes the difference vector Δμj=μj−μ¬j _j= _j- _ j. It then normalizes Δμj _j to the unit norm and places the decision boundary halfway between μj _j and μ¬j _ j. Statements are scored based on their signed projection along the Δμj _j vector. This again results in K one-versus-all heads. Outputs from both SVM and Mass Mean undergo the same calibration step used for sAwMIL. 4.2 Estimating probabilities for individual statements and constructing empirical belief sets For each model ℳM, dataset, and probe, we train a three-class probe independently at each layer ℓ using the training split and fit its probability-calibration mapping using the calibration split. We then evaluate the calibrated probe at each layer on the fixed calibration partition calD_cal using multiclass log loss, (16) ℒlog(ℓ)=−1|cal|∑(si,yi)∈callogπyi(ℓ)(si),L_log^( )=- 1|D_cal| _(s_i,y_i) _cal _y_i^( )(s_i), and select (17) ℓ⋆=arg⁡minℓ⁡ℒlog(ℓ). = _ L_log^( ). Section D.3 reports the full layer-selection procedure and the resulting selected layers for sAwMIL. Corresponding results for the secondary probes are reported in Section I.3. After selecting ℓ⋆ , we retain the fitted and calibrated probe from that layer and use its outputs on the test statements throughout the downstream analysis rather than retraining the probe after layer selection. For each statement, we record both ℳ(si)=[πT(si),πF(si),πN(si)] π_M(s_i)=[ _T(s_i), _F(s_i), _N(s_i)] and the predicted state y^i=arg⁡maxc∈T,F,Nπc(si) y_i= _c∈\T,F,N\ _c(s_i). These predictions define the empirical belief and non-disbelief sets used in all subsequent analyses. For each test statement, we map its predicted class y^i y_i to a belief state, where y^i=T y_i=T is a belief, y^i=F y_i=F is a disbelief, and y^i=N y_i=N is a suspended belief. For clarity, we restate the empirical belief and non-disbelief sets defined in Equation (5): ℬℳ=si:y^i=T,ℳ=si:y^i∈T,N.B_M=\s_i: y_i=T\, _M=\s_i: y_i∈\T,N\\. Crucially, this is based on the probe’s prediction, not the statement’s ground-truth label, meaning that a False or Neither statement could be classified as a model belief. Thus, ℬℳ⊆ℳB_M _M. Section E.1 reports the resulting belief and non-disbelief set sizes, empirical thresholds, and conditional-pair coverage for each LLM–dataset combination. From these belief and non-disbelief sets, we construct |ℳ|−1|X_M|-1 statement pairs for each P∈ℬℳP _M of the form (P,x)(P,x). These statement pairs are the target of our conditional probability estimators described in Section 4.3 and ultimately define graded stability as described in Section 4.4. For each LLM, probe, and dataset combination, we use the empirical belief threshold from Equation (6), restated for clarity: tℳ=minsi∈ℬℳ⁡πT(si).t_M= _s_i _M _T(s_i). This belief threshold is derived after categorical beliefs have been identified and corresponds to the smallest True-class probability among statements classified by the probe as True. This guarantees that the probability πT(Pi) _T(P_i) that statement PiP_i is true is at least tℳt_M for all of model ℳM’s beliefs Pi∈ℬℳP_i _M. The strict persistence criterion >tℳ>t_M is separate from this categorical membership rule: a conditional estimate exactly equal to tℳt_M is counted as non-persistent, while membership in ℬℳB_M remains determined by the trivalent probe prediction. Moreover, conditional persistence is determined by True-state support relative to tℳt_M rather than by the conditional probe’s argmax state, so a conditional can satisfy the persistence criterion even when True is not its most probable class. 4.3 Estimating conditional probability For each belief P∈ℬℳP _M and non-disbelief x∈ℳ∖Px _M \P\, we aim to estimate Prℳ⁡(P∣x) _M(P x) so that we can compute γℳ(P) _M(P), the proportion of conditioning propositions x for which belief in P persists (i.e., Prℳ⁡(P∣x)>tℳ _M(P x)>t_M). Our primary operationalization, Direct Conditional, represents P|xP x directly as a conditional statement and probes for its veracity. Our secondary approach, Joint-to-Conditional, represents x and P jointly, probes their full 3×33× 3 joint states, then derives the conditional mathematically. All main-text analyses use the Direct Conditional estimator. We use Joint-to-Conditional as a secondary estimator to test robustness to the conditional-probability operationalization (Sec. E.2, I.2). Both approaches use the same general probing framework as the estimation of probabilities for individual statements described in Section 4.2, including the data splits and probability-calibration procedure. However, we construct new conditional or joint training and calibration examples from the existing train and calibration sets, use new training targets for the combined statements, and score LLM, probe, and dataset-specific (P,x)(P,x) pairs. The Direct Conditional and Joint-to-Conditional probes score the same (P,x)(P,x) pairs, allowing us to directly compare the two estimators and evaluate the robustness of graded stability to the underlying conditional-probability operationalization (Sec. I.2.3). We describe the specifics of the two estimators below. 4.3.1 Primary estimator: Direct Conditional sis_i sjs_j si→sjs_i→ s_j T T T T N N T F F N T T N N N N F F F T N F N N F F N Table 3: Cooper–Cantwell conditional truth table. We report whether the conditional si→sj≡sj|sis_i→ s_j≡ s_j s_i is True (T), False (F), or Neither (N) based on the truth values of sis_i and sjs_j according to the Cooper–Cantwell conditional [11, 8]. When the antecedent sis_i is True or Neither, the conditional takes the truth value of the consequent sjs_j. The Direct Conditional approach represents P|xP x itself as a trivalent epistemic object and asks how strongly its LLM representation supports the True state. We operationalize P|xP x using the statement template “Given x, P.”, where x is the antecedent, or conditioning proposition, and P is the consequent, or belief whose persistence we aim to measure. Probabilistic accounts of indicative conditionals have long related support for a conditional to the corresponding conditional probability [2]. We instantiate the conditional itself using Cooper–Cantwell semantics [11, 8, 18]. Under Cooper–Cantwell semantics, when the antecedent is True or Neither, the conditional inherits the truth value of the consequent; when the antecedent is False, the conditional is Neither (Tab. 3). This treatment is particularly well suited to our stability framework because the admissible conditioning set ℳX_M contains both beliefs and suspended beliefs. A suspended antecedent can therefore provide an admissible conditioning context without automatically rendering the conditional indeterminate, as it would in the de Finetti conditional [14]. To train the Direct Conditional probe, we start from the same train and calibration statements used for the individual-statement probe and construct conditional statements of the form “Given sis_i, sjs_j.” from statement pairs. Training labels are assigned from the ground-truth states of sis_i and sjs_j according to the Cooper–Cantwell truth table. We stratify the resulting examples across the nine input-state combinations, yielding 67506750 training statements and 22502250 calibration statements. We then train and calibrate a three-class probe over the True, False, and Neither conditional states. As defined in Equation (7), the calibrated conditional probe returns ℳdirect(P,x)=[πTdirect(P,x),πFdirect(P,x),πNdirect(P,x)] π_M^direct(P,x)=[ _T^direct(P,x), _F^direct(P,x), _N^direct(P,x)] for each pair (P,x)(P,x). The Direct Conditional estimator is then defined as Pr^ℳdirect(P∣x):=πTdirect(P,x). ^direct_M(P x):= _T^direct(P,x). This quantity is the calibrated probability mass assigned to the conditional’s True epistemic state. We use it as an empirical operational estimator of Prℳ⁡(P∣x) _M(P x). It should not be read as an algebraic identity with the normalized non-bivalent probability of a trivalent conditional. We deliberately retain the Neither mass rather than renormalizing over the True and False states. Our empirical belief threshold tℳt_M is defined on the same raw True-class mass for individual propositions. Using πTdirect _T^direct therefore asks whether the conditional representation retains enough True support to satisfy the same empirical belief criterion. In particular, movement from True toward Neither is allowed to count against persistence of the original belief. This choice differs from the standard non-bivalent probability assigned to a trivalent conditional, which normalizes True mass over the states in which the conditional is determinate [8, 17]. We evaluate that complementary construction through our Joint-to-Conditional estimator below. We retain the Direct Conditional as our primary operationalization because it probes the conditional epistemic object directly, preserves all three epistemic states used to define the individual belief system, and requires only a three-class measurement problem rather than first recovering a nine-state joint distribution. 4.3.2 Secondary estimator: Joint-to-Conditional The Joint-to-Conditional approach provides a complementary operationalization grounded in the normalized probability of a trivalent conditional. Rather than probing P|xP x directly, we first estimate the joint epistemic state of x and P. To construct the joint inputs, we use the template “x and P.”, with the conditioning proposition x appearing first and the target belief P second. Because the probe operates on LLM representations of linguistic sequences, the resulting estimates are not invariant to conjunction order: “x and P” and “P and x” produce distinct representations and, empirically, distinct probability estimates. We therefore fix the order to “x and P” throughout the primary Joint-to-Conditional analysis and report robustness to the alternative ordering in Section I.2.1. As with the Direct Conditional approach, training the joint probe requires new labels for combined statements. During training and calibration, these combined statements are constructed from generic statement pairs (si,sj)(s_i,s_j) using the template “sis_i and sjs_j.”. Because each statement can take on one of three truth values, the corresponding joint label has 3×3=93× 3=9 possible states: [TT,TF,TN,NT,NF,NN,FT,FF,FN][T,TF,TN,NT,NF,N,FT,F,FN], represented formally as (yi,yj)∈T,F,N×T,F,N(y_i,y_j)∈\T,F,N\×\T,F,N\. We construct the combined training and calibration sets by subsampling the individual-statement training and calibration sets such that each of the nine joint classes has an equal number of combined statements. In total, we again generate 67506750 training statements and 22502250 calibration statements. We then train the probe with nine one-versus-all heads and calibrate its outputs as outlined in Section 4.1.3 and D.1. After training and calibration, we apply the joint probe to the same (P,x)(P,x) pairs scored by the Direct Conditional probe, with x occupying the first position and P the second. The resulting calibrated nine-valued distribution is (18) ℳjoint(P,x)=πabjoint(P,x)a,b∈T,F,N. π_M^joint(P,x)=\ _ab^joint(P,x)\_a,b∈\T,F,N\. Here, πabjoint(P,x) _ab^joint(P,x) is the probe-assigned probability that x has state a and P has state b. To derive the conditional probability from this joint distribution, we use the probabilistic semantics for Cooper–Cantwell trivalent conditionals [8, 17, 7]: (19) Pr^ℳjoint(P∣x)=πTTjoint(P,x)+πNTjoint(P,x)πTTjoint(P,x)+πTFjoint(P,x)+πNTjoint(P,x)+πNFjoint(P,x). _M^joint(P x)= _T^joint(P,x)+ _NT^joint(P,x) _T^joint(P,x)+ _TF^joint(P,x)+ _NT^joint(P,x)+ _NF^joint(P,x). Under Cooper–Cantwell semantics, the conditional is determinate when the antecedent x is True or Neither and the consequent P is either True or False. The numerator therefore contains the joint states that induce a True conditional, while the denominator contains all joint states that induce either a True or False conditional. Probability mass assigned to joint states that induce a Neither conditional is excluded by this normalization. We provide the full derivation in Section E.2. This normalization marks the principal distinction from our Direct Conditional estimator. Direct Conditional retains probability mass assigned to the conditional’s Neither state, such that movement from True toward Neither can reduce the estimated support for P|xP x. Joint-to-Conditional instead conditions on the event that the induced conditional is determinate and measures the relative True mass within that subset. Equation (19) is undefined when its denominator is zero. We treat denominators less than or equal to 10−1210^-12 as numerically zero and exclude the corresponding (P,x)(P,x) pairs from subsequent graded-stability calculations rather than assigning them a conditional probability. Across all models and datasets, this occurs for 0.026%0.026\% of scored Joint-to-Conditional pairs. 4.4 Computing graded belief stability For each belief P∈ℬℳP _M, we evaluate its conditional probability under every non-disbelief x∈ℳx _M. We exclude the self-pair P=xP=x, resulting in |ℳ|−1|X_M|-1 conditioning propositions for each belief. For completeness, we restate the graded belief stability definition from Equation (2): γℳ(P):=|x∈ℳ∖P:Prℳ⁡(P∣x)>tℳ||ℳ∖P|, _M(P):= | \x _M \P\: _M(P x)>t_M \ | |X_M \P\ |, where tℳt_M is the empirical belief threshold obtained from the corresponding individual-statement probe described in Section 4.2. Thus, γℳ(P) _M(P) is the proportion of conditioning propositions under which the conditional probability of P remains above the belief threshold. We apply Equation (2) using the Direct Conditional estimates, yielding γℳdirect(P)γ^direct_M(P).44 4 We analogously compute γℳjoint(P)γ^joint_M(P) for the Joint-to-Conditional estimate and report results in Section I.2.2. Each γℳ(P) _M(P) is a proposition-level quantity. When a model-level summary is required, we compute the mean graded stability across believed propositions, (20) γ¯ℳ=1|ℬℳ|∑P∈ℬℳγℳ(P). γ_M= 1|B_M| _P _M _M(P). We compute this quantity separately for each model, dataset, probe, and conditional-probability estimator. 4.5 Validating graded belief stability We evaluate graded belief stability along two complementary dimensions: whether it captures information beyond individual belief probability (Sec. 4.5.1) and whether the measured probabilities are approximately compatible with the CCK probability framework (Sec. 4.5.2). 4.5.1 Stability vs. belief probability As defined in Equation (9), we model graded stability γℳ(P) _M(P) as a function of the belief probability πT(P) _T(P) using a natural cubic spline, γ^ℳ(P)=fℳ(πT(P)), γ_M(P)=f_M\! ( _T(P) ), for each model, dataset, probe, and conditional-probability estimator. We fix the spline complexity at three degrees of freedom for all primary analyses. Section F.1 evaluates alternative spline complexities and a linear baseline and provides additional implementation and fit diagnostics. We use five-fold cross-validation to evaluate predictive performance on held-out propositions and quantify performance using R2R^2, (21) R2=1−∑P∈ℬℳ(γℳ(P)−γ^ℳ(P))2∑P∈ℬℳ(γℳ(P)−γ¯ℳ)2,R^2=1- _P _M ( _M(P)- γ_M(P) )^2 _P _M ( _M(P)- γ_M )^2, where γ¯ℳ γ_M is the mean graded stability defined in Equation (20). Model–dataset–probe–estimator combinations with fewer than 5050 believed propositions with valid stability scores, insufficient distinct values of πT(P) _T(P) to fit the spline, or no variation in γℳ(P) _M(P) are excluded from the analysis. The fitted value γ^ℳ(P) γ_M(P) represents stability predicted from belief probability alone. Using these out-of-fold predictions, we define residual stability as in Equation (10), restated here for clarity: rℳ(P)=γℳ(P)−γ^ℳ(P),r_M(P)= _M(P)- γ_M(P), where a positive residual indicates that a proposition is more stable than predicted from its belief probability, while a negative residual indicates that it is less stable than predicted. We use these residuals to test whether propositions that are unusually stable or fragile for one model tend to be similarly unusual for other models. Within each dataset, probe, and conditional-probability estimator, we consider every pair of models and restrict the comparison to propositions believed by both models. We retain model pairs with at least 5050 shared propositions and compute Spearman’s rank correlation [40] between their residual stability scores over this shared proposition set, yielding one residual correlation for each model pair. We characterize cross-model agreement using the median of these pairwise correlations. To assess whether the observed agreement exceeds what would be expected without proposition-specific alignment across models, we construct a permutation null by independently shuffling residual stability scores across propositions within each model while preserving each model’s residual distribution, the set of eligible model pairs, and their proposition-overlap structure. For each permutation, we recompute the pairwise residual Spearman correlations and record their median. We use 1,0001,000 permutations and compute a one-sided Monte Carlo p-value to evaluate the hypothesis that the observed median correlation is greater than expected under the null, using the +1+1 correction [35]. Additional implementation details are reported in Section F.1. 4.5.2 Approximate probabilistic coherence We evaluate probabilistic coherence under the Cooper–Cantwell–Kleene (CCK) trivalent probability framework [8, 17] for each model, dataset, probe, and conditional-probability estimator.55 5 For Joint-to-Conditional, the distributions ℳ(P) π_M(P) and ℳ(x) π_M(x) continue to come from the individual-statement probe. The conditional distribution is obtained by collapsing the nine-state Joint probe according to the Cooper–Cantwell truth table: πTjoint,C=πTTjoint+πNTjoint _T^joint,C= _T^joint+ _NT^joint, πFjoint,C=πTFjoint+πNFjoint _F^joint,C= _TF^joint+ _NF^joint, and πNjoint,C=πTNjoint+πNNjoint+πFTjoint+πFNjoint+πFFjoint _N^joint,C= _TN^joint+ _N^joint+ _FT^joint+ _FN^joint+ _F^joint, where the first subscript denotes the state of x and the second the state of P. Thus, Joint coherence compares a Joint-derived conditional distribution against independently measured individual marginals. For every (P,x)(P,x) pair, we combine the measured probability distributions for P, x, and their conditional into the vector P,xv_P,x defined in Equation (11), restated here for clarity: P,x=[ℳ(P),ℳ(x),ℳdirect(P,x)].v_P,x=[ π_M(P), π_M(x), π_M^direct(P,x)]. As established in Equation (12), for the measured marginals of P, x, and their conditional to be jointly representable by a single trivalent distribution over (P,x)(P,x), exact CCK coherence requires πTdirect(P,x)≤πT(P),πFdirect(P,x)≤πF(P),πF(x)≤πNdirect(P,x),πNdirect(P,x)≤πN(P)+πF(x). _T^direct(P,x)≤ _T(P), _F^direct(P,x)≤ _F(P), _F(x)≤ _N^direct(P,x), _N^direct(P,x)≤ _N(P)+ _F(x). We classify a pair as exactly coherent when all four CCK constraints are satisfied within a numerical tolerance of 10−810^-8. A proof that these conditions characterize exact pairwise coherence is provided in Section F.2.1, and exact-coherence rates and individual constraint violations are reported in Section F.2.2. To distinguish small departures from coherence from larger violations, we additionally quantify each pair’s distance from the coherent set using dCCKd_CCK as defined in Equation (13): dCCK(P,x):=19‖P,x−Π(P,x)‖22.d_CCK(v_P,x):= 19 \|v_P,x- _C(v_P,x) \|_2^2. We compute the corresponding projection Π(P,x) _C(v_P,x) using Dykstra’s projection algorithm [16]. The algorithm alternates between enforcing normalization and non-negativity of the three probability distributions and enforcing each of the four linear CCK constraints. Iteration stops when the maximum component-wise change between successive solutions is at most 10−1010^-10, with a maximum of 250250 iterations. We verify that the projected vectors satisfy normalization, non-negativity, and the CCK constraints within a tolerance of 10−710^-7. We compute dCCKd_CCK for every (P,x)(P,x) pair and summarize its distribution separately for each model, dataset, probe, and conditional-probability estimator. 4.6 Behavioral validation under conversational challenge To test whether graded stability corresponds to observable belief resilience, we measure how strongly an LLM maintains its initial truth judgment across a controlled sequence of conversational challenges. In particular, we test whether, among beliefs assigned similar individual belief probabilities, the belief with higher graded stability exhibits less behavioral movement under challenge. We first construct the proposition pairs used for this comparison by matching beliefs on individual belief probability. Before matching, propositions are assigned to one of the six challenge-order sequences described below. To ensure that both members of a matched pair receive the same conversational challenges in the same order, we perform matching separately within each exact challenge-sequence block. Within each block s, we construct a maximum-cardinality, one-to-one matching ℳsM_s without replacement that minimizes the total absolute difference in individual belief probability, (22) ∑(Pi,Pj)∈ℳs|πT(Pi)−πT(Pj)|. _(P_i,P_j) _s | _T(P_i)- _T(P_j) |. We restrict the eligible proposition pool to beliefs for which both Direct Conditional and Joint-to-Conditional graded stability are defined, so that the two operationalizations are evaluated over the same underlying matches. Within this eligible pool, pair construction depends only on individual belief probability and challenge-sequence assignment. Neither graded-stability values nor behavioral outcomes enter the matching objective. After the matches are fixed, we orient each pair according to the relevant graded-stability estimate and denote the higher- and lower-stability propositions by PHP_H and PLP_L, respectively. Pairs for which the two propositions have identical graded stability are excluded because no higher- and lower-stability member can be defined. An overview of the resulting probability-matched pairs is reported in Section G.1. Each matched proposition is evaluated in a four-stage interaction consisting of an initial binary truth judgment at round k=0k=0 followed by three successive challenges at rounds k∈1,2,3k∈\1,2,3\. We use the following prompts: Initial prompt Statement: statement Is this statement true or false? Choose exactly one option. 1. True 2. False Answer: Challenge A Hmm, I’m not sure that’s right. Can you double-check? Please answer again, choosing exactly one option. 1. True 2. False Answer: Challenge B Are you sure about that? Could you check your answer again? Please answer again, choosing exactly one option. 1. True 2. False Answer: Challenge C I’m not convinced. Can you think it over one more time? Please answer again, choosing exactly one option. 1. True 2. False Answer: After each round, the model’s selected answer is included as the preceding assistant response and the next challenge is appended as a new user turn, preserving the complete multi-turn interaction history. To control for ordering effects, we consider all six permutations of the challenge prompts: ABCABC, ACBACB, BACBAC, BCABCA, CABCAB, and CBACBA. Before pair construction, propositions are randomly assigned in approximately equal numbers across these six orderings using seed 00. Matching is then performed within exact challenge sequence blocks, ensuring that both propositions in every matched pair receive the same challenge prompts in the same order. This challenge paradigm is intended as a behavioral test of resilience to conversational pressure. The challenge prompts do not instantiate the conditioning propositions x used in the definition of graded stability and should not be interpreted as direct empirical estimates of Prℳ⁡(P∣x) _M(P x). Following existing behavioral belief evaluation procedures [15], we restrict each judgment to the enumerated answer options rather than sampling and parsing a free-form response. For each class c∈T,Fc∈\T,F\, let pℳ,kc(P)p_M,k^c(P) denote the autoregressive sequence probability assigned at round k to the complete candidate answer string corresponding to c, 1 for True and 2 for False. We normalize these probabilities over the two permitted responses, (23) qℳ,kc(P)=pℳ,kc(P)pℳ,kT(P)+pℳ,kF(P).q_M,k^c(P)= p_M,k^c(P)p_M,k^T(P)+p_M,k^F(P). The model’s judgment at round k is c^k(P)=arg⁡maxc∈T,Fqℳ,kc(P) c_k(P)= _c∈\T,F\q_M,k^c(P). To track support for the same response throughout the interaction, we fix the reference class to the response selected at the initial round, c^0(P) c_0(P), and define (24) uk(P)=qℳ,kc^0(P)(P).u_k(P)=q_M,k c_0(P)(P). Thus, uk(P)u_k(P) is the normalized probability assigned at round k to the response class that the model selected at round k=0k=0. In particular, u0(P)u_0(P) measures the model’s initial support for its selected response, while uk(P)u_k(P) for k>0k>0 measures how strongly it continues to support that same response after k challenges. Using this sequence of probabilities, we quantify behavioral movement as defined in Equation (14), restated here for clarity: M⁡(P):=13∑k=13|uk(P)−uk−1(P)|.M(P):= 13 _k=1^3 |u_k(P)-u_k-1(P) |. Larger M⁡(P)M(P) therefore indicates greater unsigned fluctuation in the model’s support for its initial response across the challenge sequence, meaning that movements toward and away from the initial response both contribute to M⁡(P)M(P). The measure does not specifically encode loss of support or whether the model’s discrete True/False answer flips. For each matched pair, we then compute the behavioral difference ΔM M defined in Equation (15): ΔM:=M⁡(PL)−M⁡(PH). M:=M(P_L)-M(P_H). Thus, ΔM>0 M>0 indicates that the lower-stability belief moves more under challenge. Because the matched beliefs are defined from the probe-based belief set, our primary behavioral analysis retains only pairs for which the model’s initial round-00 behavioral judgment agrees with the corresponding individual-statement probe classification for both propositions. We average ΔM M across matched proposition pairs separately for each model and dataset, yielding one model-level behavioral effect per domain. To quantify uncertainty in each model–domain effect, we perform 10,00010,000 matched-pair bootstrap resamples, resampling with replacement separately within each challenge-sequence block while preserving the observed number of pairs in each block. The bootstrap standard error is the standard deviation of the resulting distribution of mean ΔM M. Figure 5 shows ±1± 1 bootstrap standard error, while the corresponding 95%95\% percentile bootstrap confidence intervals are reported in Section G.2. Acknowledgments We thank Germans Savcisens and Courtney Maynard for their useful comments on this work. Funding S.D. and T.E.R. are supported by the Inaugural Joseph E. Aoun Endowment. Competing interests The authors declare no competing interests. AI Use ChatGPT-5.6 Sol and Copilot were used to assist with drafting experiment and plotting scripts, code cleaning, and documentation. ChatGPT-5.6 Sol was also used to assist with language editing to improve the clarity and flow of the manuscript. GPT-6-Astra and Claude Fable 5.1 were used to proofread the final text. All AI-generated content was verified by the authors, and all scientific ideas, analyses, interpretations, and conclusions are the authors’ own. References [1] Y. Abbasi-Yadkori, I. Kuzborskij, A. György, and C. Szepesvári (2024) To believe or not to believe your LLM: iterative prompting for estimating epistemic uncertainty. Advances in Neural Information Processing Systems 37, p. 58077–58117. Note: https://openreview.net/forum?id=k6iyUfwdI9 Cited by: §1. [2] E. W. Adams (1975) The logic of conditionals: an application of probability to deductive logic. Synthese Library, Vol. 86, D. Reidel Publishing Company, Dordrecht. Note: https://doi.org/10.1007/978-94-015-7622-2 External Links: Document Cited by: §4.3.1. [3] G. Ahdritz, T. Qin, N. Vyas, B. Barak, and B. L. Edelman (2024) Distinguishing the knowable from the unknowable with language models. In Proceedings of the 41st International Conference on Machine Learning, R. Salakhutdinov, Z. Kolter, K. Heller, A. Weller, N. Oliver, J. Scarlett, and F. Berkenkamp (Eds.), Proceedings of Machine Learning Research, Vol. 235, p. 503–549. Note: https://proceedings.mlr.press/v235/ahdritz24a.html External Links: Link Cited by: §1, §1. [4] B. AlKhamissi, M. Li, A. Celikyilmaz, M. Diab, and M. Ghazvininejad (2022) A review on language models as knowledge bases. arXiv preprint arXiv:2204.06031. External Links: Document Cited by: §1. [5] I. Augenstein, T. Baldwin, M. Cha, T. Chakraborty, G. L. Ciampaglia, D. Corney, R. DiResta, E. Ferrara, S. Hale, A. Halevy, et al. (2024) Factuality challenges in the era of large language models and opportunities for fact-checking. Nature Machine Intelligence 6 (8), p. 852–863. External Links: Document Cited by: §1. [6] N. Band, X. Li, T. Ma, and T. Hashimoto (2024) Linguistic calibration of long-form generations. In Proceedings of the 41st International Conference on Machine Learning, R. Salakhutdinov, Z. Kolter, K. Heller, A. Weller, N. Oliver, J. Scarlett, and F. Berkenkamp (Eds.), Proceedings of Machine Learning Research, Vol. 235, p. 2732–2778. Note: https://proceedings.mlr.press/v235/band24a.html External Links: Link Cited by: §1. [7] J. Cantwell (2006) The laws of non-bivalent probability. Logic and Logical Philosophy 15 (2), p. 163–171. Note: https://apcz.umk.pl/LLP/article/view/LLP.2006.010 Cited by: §E.2, §4.3.2. [8] J. Cantwell (2008) The logic of conditional negation. Notre Dame Journal of Formal Logic 49 (3), p. 245–260. External Links: Document Cited by: §2.2, §4.3.1, §4.3.1, §4.3.2, §4.5.2, Table 3, Table 3. [9] M. Cheon (2026) Robust for the wrong reasons: the representational geometry of LLM robustness to science skepticism. arXiv preprint arXiv:2607.01951. External Links: Document Cited by: §1. [10] R. Cohen, E. Biran, O. Yoran, A. Globerson, and M. Geva (2024) Evaluating the ripple effects of knowledge editing in language models. Transactions of the Association for Computational Linguistics 12, p. 283–298. External Links: ISSN 2307-387X, Document, Link Cited by: §1. [11] W. S. Cooper (1968) The propositional logic of ordinary discourse. Inquiry 11 (1-4), p. 295–320. External Links: Document Cited by: §1, §2.2, §4.3.1, Table 3, Table 3. [12] A. Corona Mendozza and A. Søgaard (2026) LLM beliefs are in their heads. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), M. Liakata, V. P. Moreira, J. Zhang, and D. Jurgens (Eds.), San Diego, California, United States, p. 41033–41067. External Links: Link, Document, ISBN 979-8-89176-390-6 Cited by: §1. [13] C. Cortes and V. Vapnik (1995) Support-vector networks. Machine Learning 20, p. 273–297. External Links: Document Cited by: §I.3, footnote 1. [14] B. De Finetti (1936) La logique de la probabilité. In Actes du congrès international de philosophie scientifique, Vol. 4, p. 31–39. External Links: Document Cited by: §4.3.1. [15] S. Dies, C. Maynard, G. Savcisens, and T. Eliassi-Rad (2025) Epistemic familiarity is associated with belief stability in large language models. arXiv preprint arXiv:2511.19166. External Links: Document Cited by: §I.3, §1, §4.6. [16] R. L. Dykstra (1983) An algorithm for restricted least squares regression. Journal of the American Statistical Association 78 (384), p. 837–842. External Links: Document Cited by: §4.5.2. [17] P. Égré, L. Rossi, and J. M. Sprenger (2026) Probability for trivalent conditionals. Mind forthcoming. Note: https://iris.unito.it/bitstream/2318/2142754/1/CP%2BLearning_rev7.pdf Cited by: §1, §4.3.1, §4.3.2, §4.5.2. [18] P. Égré, L. Rossi, and J. Sprenger (2025) Certain and uncertain inference with indicative conditionals. Australasian Journal of Philosophy 103 (3), p. 569–596. External Links: Document Cited by: §2.2, §4.3.1. [19] Y. Elazar, N. Kassner, S. Ravfogel, A. Ravichander, E. Hovy, H. Schütze, and Y. Goldberg (2021) Measuring and improving consistency in pretrained language models. Transactions of the Association for Computational Linguistics 9, p. 1012–1031. External Links: Document Cited by: §1, §1, §2.2. [20] R. Foley (1992) The epistemology of belief and the epistemology of degrees of belief. American Philosophical Quarterly 29 (2), p. 111–124. Note: http://w.jstor.org/stable/20014406 Cited by: §1, §2.1. [21] X. Gao, J. Zhang, L. Mouatadid, and K. Das (2024) SPUQ: Perturbation-based uncertainty quantification for large language models. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), Y. Graham and M. Purver (Eds.), p. 2336–2346. External Links: Link, Document Cited by: §1. [22] Gemma Team (2024) Gemma. Kaggle. External Links: Link, Document Cited by: Table A3, Table A3, Table A3, Table 1, Table 1, Table 1. [23] A. Grattafiori, A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Vaughan, et al. (2024) The Llama 3 herd of models. arXiv preprint arXiv:2407.21783. External Links: Document Cited by: Table A3, Table A3, Table A3, Table 1, Table 1, Table 1. [24] A. Q. Jiang, A. Sablayrolles, A. Mensch, C. Bamford, D. S. Chaplot, D. de las Casas, F. Bressand, G. Lengyel, G. Lample, L. Saulnier, L. R. Lavaud, M. Lachaux, P. Stock, T. Le Scao, T. Lavril, T. Wang, T. Lacroix, and W. El Sayed (2023) Mistral 7B. arXiv preprint arXiv:2310.06825. External Links: Document Cited by: Table A3, Table 1. [25] S. Kapoor, N. Gruver, M. Roberts, K. Collins, A. Pal, U. Bhatt, A. Weller, S. Dooley, M. Goldblum, and A. G. Wilson (2024) Large language models must be taught to know what they don’t know. Advances in Neural Information Processing Systems 37, p. 85932–85972. External Links: Document Cited by: §1. [26] N. Kassner, O. Tafjord, H. Schütze, and P. Clark (2021) BeliefBank: Adding memory to a pre-trained language model for a systematic notion of belief. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, p. 8849–8861. External Links: Document Cited by: §1. [27] D. Kumaran, S. M. Fleming, L. Markeeva, J. Heyward, A. Banino, M. Mathur, R. Pascanu, S. Osindero, B. De Martino, P. Veličković, et al. (2026) Competing biases underlie overconfidence and underconfidence in LLMs. Nature Machine Intelligence 8, p. 614–627. External Links: Document Cited by: §1, §1. [28] H. Leitgeb (2014) The stability theory of belief. Philosophical Review 123 (2), p. 131–171. External Links: Document Cited by: §1, §2.1, §2.1, §2.1. [29] H. Leitgeb (2015) I–The Humean thesis on belief. Aristotelian Society Supplementary Volume 89 (1), p. 143–185. External Links: ISSN 0309-7013, Document, Link Cited by: §1, §2.1. [30] Y. Li, Y. Miao, X. Ding, R. Krishnan, and R. Padman (2025) Firm or fickle? Evaluating large language models consistency in sequential interactions. In Findings of the Association for Computational Linguistics (ACL 2025), W. Che, J. Nabende, E. Shutova, and M. T. Pilehvar (Eds.), p. 6679–6700. External Links: Link, Document, ISBN 979-8-89176-256-5 Cited by: §1, §1, §2.2. [31] S. Marks and M. Tegmark (2024) The geometry of truth: Emergent linear structure in large language model representations of True/False datasets. In Proceedings of the 1st Conference on Language Modeling (COLM 2024), Note: https://openreview.net/forum?id=aajyHYjjsk Cited by: §I.3, §I.3, §1, §1, §4.1.3, footnote 1. [32] Mistral AI Team (2024) Mistral NeMo. Note: https://mistral.ai/news/mistral-nemo/ External Links: Link Cited by: Table A3, Table 1. [33] Mistral AI (2025) Mistral Small 3.1. Note: https://mistral.ai/news/mistral-small-3-1/ External Links: Link Cited by: Table A3, Table 1. [34] G. Molinari (2024) An accuracy characterisation of approximate coherence. Synthese 203 (2), p. 68. External Links: Document Cited by: §2.3.2. [35] B. Phipson and G. K. Smyth (2010) Permutation p-values should never be zero: Calculating exact p-values when permutations are randomly drawn. Statistical Applications in Genetics and Molecular Biology 9 (1), p. Article 39. External Links: Document Cited by: §4.5.1. [36] Qwen Team (2024) Qwen2.5: a party of foundation models. Note: https://qwenlm.github.io/blog/qwen2.5/ External Links: Link Cited by: Table A3, Table A3, Table A3, Table 1, Table 1, Table 1. [37] R. Sarkar, P. Ramu, and R. Rudinger (2026) Language models encode the contextual truth of propositions. arXiv preprint arXiv:2608.03035. External Links: Document Cited by: §1. [38] G. Savcisens and T. Eliassi-Rad (2025) Trilemma of truth in large language models. In Mechanistic Interpretability Workshop at NeurIPS 2025, Note: https://openreview.net/forum?id=z7dLG2ycRf Cited by: §B.1, §B.1, §B.1, §B.1.1, §B.1.1, §B.1, §B.2, Table A2, Table A2, §1, §1, §2.2, §2.2, §4.1.2, §4.1.2, §4.1.3, Table 2, Table 2. [39] M. Sharma, M. Tong, T. Korbak, D. Duvenaud, A. Askell, S. R. Bowman, E. Durmus, Z. Hatfield-Dodds, S. R. Johnston, S. M. Kravec, T. Maxwell, S. McCandlish, K. Ndousse, O. Rausch, N. Schiefer, D. Yan, M. Zhang, and E. Perez (2024) Towards understanding sycophancy in language models. In Proceedings of the 12th International Conference on Learning Representations (ICLR 2024), Note: https://openreview.net/forum?id=tvhaxkMKAn Cited by: §1, §1, §2.2. [40] C. Spearman (1961) The proof and measurement of association between two things. In Studies in Individual Differences: The Search for Intelligence, J. J. Jenkins and D. G. Paterson (Eds.), p. 45–58. External Links: Document Cited by: §4.5.1. [41] M. Steyvers, H. Tejeda, A. Kumar, C. Belem, S. Karny, X. Hu, L. W. Mayer, and P. Smyth (2025) What large language models know and what people think they know. Nature Machine Intelligence 7 (2), p. 221–231. External Links: Document Cited by: §1. [42] M. Suzgun, T. Gur, F. Bianchi, D. E. Ho, T. Icard, D. Jurafsky, and J. Zou (2025) Language models cannot reliably distinguish belief from knowledge and fact. Nature Machine Intelligence 7 (11), p. 1780–1790. External Links: Document Cited by: §1. [43] D. S. Wishart, Y. D. Feunang, A. C. Guo, E. J. Lo, A. Marcu, J. R. Grant, T. Sajed, D. Johnson, C. Li, Z. Sayeeda, et al. (2018) DrugBank 5.0: a major update to the DrugBank database for 2018. Nucleic Acids Research 46 (D1), p. D1074–D1082. External Links: Document Cited by: §B.1. [44] P. Wolf, T. Kleine Buening, A. Krause, and C. Mendler-Dünner (2026) Partition, prompt, aggregate: statistical self-consistency in language models. arXiv preprint arXiv:2607.15277. External Links: Document Cited by: §2.2. [45] T. Wolf, L. Debut, V. Sanh, J. Chaumond, C. Delangue, A. Moi, P. Cistac, T. Rault, R. Louf, M. Funtowicz, J. Davison, S. Shleifer, P. von Platen, C. Ma, Y. Jernite, J. Plu, C. Xu, T. Le Scao, S. Gugger, M. Drame, Q. Lhoest, and A. Rush (2020) Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (System Demonstrations 2020), p. 38–45. External Links: Document Cited by: Table A3, Table A3, Table 1, Table 1. [46] A. Yang, B. Yang, B. Hui, B. Zheng, B. Yu, C. Zhou, C. Li, C. Li, D. Liu, F. Huang, G. Dong, H. Wei, H. Lin, J. Tang, J. Wang, J. Yang, J. Tu, J. Zhang, J. Ma, J. Xu, J. Zhou, J. Bai, J. He, J. Lin, K. Dang, K. Lu, K. Chen, K. Yang, M. Li, M. Xue, N. Ni, P. Zhang, P. Wang, R. Peng, R. Men, R. Gao, R. Lin, S. Wang, S. Bai, S. Tan, T. Zhu, T. Li, T. Liu, W. Ge, X. Deng, X. Zhou, X. Ren, X. Zhang, X. Wei, X. Ren, Y. Fan, Y. Yao, Y. Zhang, Y. Wan, Y. Chu, Y. Liu, Z. Cui, Z. Zhang, and Z. Fan (2024) Qwen2 technical report. arXiv preprint arXiv:2407.10671. External Links: Document Cited by: Table A3, Table A3, Table A3, Table 1, Table 1, Table 1. [47] F. Yin, J. Srinivasa, and K. Chang (2024) Characterizing truthfulness in large language model generations with local intrinsic dimension. In Proceedings of the 41st International Conference on Machine Learning, R. Salakhutdinov, Z. Kolter, K. Heller, A. Weller, N. Oliver, J. Scarlett, and F. Berkenkamp (Eds.), Proceedings of Machine Learning Research, Vol. 235, p. 57069–57084. Note: https://proceedings.mlr.press/v235/yin24c.html External Links: Link Cited by: §1. [48] Z. J. Ying, S. Ravfogel, N. Kriegeskorte, and P. Hase (2026) The truthfulness spectrum hypothesis. arXiv preprint arXiv:2602.20273. External Links: Document Cited by: §1. Appendix A Notation We summarize the principal mathematical notation used throughout the manuscript in Table A1. Where clear from context, indices for semantic domain, probe, and stability operationalization are suppressed. Accordingly, empirical quantities written with subscript ℳM are instantiated separately for each model–dataset–probe combination. Graded stability additionally depends on the conditional-probability operationalization. Symbol Description ℳM A fixed large language model (LLM). s Natural-language statement. P A proposition believed by ℳM whose stability is evaluated. x A non-disbelieved proposition used as conditioning information. c∈T,F,Nc∈\T,F,N\ Trivalent veracity class: True (T), False (F), or Neither (N). K Number of probe classes. ℓ Layer index used for activation extraction. ℓ⋆ Layer selected by minimizing calibration log loss. zi(ℓ)z_i^( ) Hidden representation of statement sis_i. ⁡(si)r(s_i) Vector of raw probe scores for statement sis_i. ℒlog(ℓ)L_log^( ) Multiclass log loss of the calibrated probe at layer ℓ . y^i y_i Probe-predicted epistemic state of statement sis_i. ℳ(si) π_M(s_i) Probe-estimated trivalent distribution for sis_i. ℳ(P,x) π_M(P,x) Probe-estimated distribution associated with pair (P,x)(P,x), with Direct Conditional and Joint-to-Conditional variants. Prℳ⁡(s) _M(s) Probability that statement s is true. Prℳ⁡(P∣x) _M(P x) Conditional probability that proposition P is true given proposition x. Pr^ℳ(P∣x) _M(P x) Estimate of Prℳ⁡(P∣x) _M(P x) using either the Direct Conditional or Joint-to-Conditional approach. ℬℳB_M Belief set of ℳM. ℳX_M Non-disbelief set of ℳM. tℳt_M Empirical belief threshold for ℳM, defined from the lowest True-class probability among beliefs P∈ℬℳP _M. γℳ(P) _M(P) Graded belief stability of P under ℳM for either the Direct Conditional or Joint-to-Conditional estimates. γ¯ℳ γ_M Mean graded belief stability across beliefs in ℬℳB_M. γ^ℳ(P)=fℳ(πT(P)) γ_M(P)=f_M( _T(P)) Predicted graded stability of P based on its belief probability. rℳ(P)r_M(P) Residual graded stability after accounting for individual belief probability. P,xv_P,x Nine-dimensional vector collecting the measured trivalent probabilities. C Set of CCK-coherent probability vectors. Π() _C(v) Euclidean projection onto the set of CCK-coherent probability vectors C. dCCK(P,x)d_CCK(v_P,x) RMS adjustment required to make the measured probability system CCK-coherent. PH,PLP_H,\;P_L Higher- and lower-stability propositions within a pair matched on individual belief probability. k Round in the behavioral challenge experiment. pℳ,kc(P)p_M,k^c(P) Probability assigned at round k to candidate behavioral response c for proposition P. qℳ,kc(P)q_M,k^c(P) Probability of behavioral response c after normalizing over the permitted True/False responses. c^k(P) c_k(P) Binary truth judgment selected by ℳM for proposition P at behavioral round k. uk(P)u_k(P) Normalized probability assigned at round k to the response class initially selected for proposition P. M⁡(P)M(P) Behavioral movement of P across challenge rounds. ΔM M Difference in movement between lower- and higher-stability matched beliefs. Table A1: Notation. Summary of the principal mathematical symbols used throughout the manuscript. Appendix B Datasets B.1 Dataset construction We use the City Locations, Medical Indications, and Word Definitions datasets introduced by [38]. Each dataset contains statements labeled True, False, or Neither, with approximately balanced affirmative and negated forms. The three domains capture geographic containment, drug–indication relationships, and lexical relationships, respectively. The True and False statements are constructed from real-world entities and relations, while the Neither statements pair synthetically generated entities for which no corresponding real-world fact is intended to exist. For the True and False classes, each domain contains both correct and incorrect entity–relation pairs, and each pair is expressed in both affirmative and negated form. For a correct entity–relation pair, the affirmative statement is True and its negation is False. Incorrect pairs are generated by shuffling entities and relations so that the affirmative statement is False and its negation is True. City Locations. City Locations is constructed from the GeoNames geographic database66 6 https://w.geonames.org/. [38]. Cities are required to have populations of at least 30,00030,000 and an associated country, and locations in Antarctica are excluded. The resulting collection retains 1,4001,400 unique city names. Statements are generated with the template “The city of [city] is (not) located in [country],”, where “The city of” is omitted when redundant. Medical Indications. Medical Indications is constructed from DrugBank version 5.1.12 [38, 43]. Drug names and indication descriptions are extracted from the database, with only the first sentence retained for multi-sentence indications. Diseases and conditions are extracted using both SciSpacy and a BioBERT-based named-entity recognition model and retained only when identified by both models77 7 https://huggingface.co/alvaroalon2/biobert_diseases_ner.. Drug names are additionally required to be recognized as chemical entities by SciSpacy, and low-frequency drug–indication pairs are removed using wordfreq. Statements are generated with the template “[drug] is (not) indicated for the treatment of [disease/condition].” Word Definitions. Word Definitions is constructed from the publicly available sample of WordsAPI88 8 https://w.wordsapi.com/. [38]. The dataset retains nouns with at least one definition and at least one synonym, typeOf, or instanceOf relation. These relations are rendered using three corresponding statement templates: “[word] is (not) a [instanceOf],” “[word] is (not) a type of [typeOf],” and “[word] is (not) a synonym of [synonym].” Articles and singular forms are adjusted where necessary to preserve grammaticality. B.1.1 Synthetic Neither statements The Neither class is designed to represent claims for which the model has no corresponding real-world fact to recover. Rather than relying on naturally occurring obscure entities, whose presence in LLM training corpora cannot be established, synthetic entities are generated from the lexical statistics of the real entities in each domain. The namemaker package99 9 https://github.com/Rickmsd/namemaker. is used to generate new names with character bigram Markov-chain models fit to the corresponding real entity sets [38]. Synthetic entities are then paired using the same statement templates as the real-world data. Generated entities undergo domain-specific filtering intended to reduce accidental overlap with existing entities. City and country names are checked against GeoNames and subsequently screened using targeted web searches. Drug and disease names are checked against biomedical entity resources and similarly screened for real-world matches. Finally, synthetic lexical items are compared against multiple English word lists. Candidates that match existing entities are removed before the remaining synthetic entities are randomly paired to construct Neither statements [38]. These validation steps are intended to minimize the probability that a synthetic entity corresponds to an existing entity or lexical item. Because the training corpora of the evaluated LLMs are not fully observable, we cannot guarantee that every synthetic string is entirely absent from pretraining data. We therefore interpret the Neither examples as controlled, intentionally unfamiliar claims for which the model lacks an established real-world truth value, rather than as a direct test of training-data absence. Dataset Train Calibration Test Total City Locations 39993999 (0.55)(0.55) 13981398 (0.19)(0.19) 18551855 (0.26)(0.26) 72527252 (1.00)(1.00) Medical Indications 38493849 (0.56)(0.56) 13271327 (0.19)(0.19) 17271727 (0.25)(0.25) 69036903 (1.00)(1.00) Word Definitions 47174717 (0.55)(0.55) 16281628 (0.19)(0.19) 21552155 (0.25)(0.25) 85008500 (1.00)(1.00) Table A2: Dataset splits. Number of statements used for training, calibration, and testing. Proportions of the full dataset are reported in parentheses. A version of this table appears in [38]. B.2 Dataset splits We use the fixed train, calibration, and test partitions introduced with the original datasets [38]. The datasets are partitioned into approximately 55%55\% training, 20%20\% calibration, and 25%25\% test data, with entities kept exclusive across splits. If an entity occurs in one partition, all statements containing that entity are assigned to the same partition. Table A2 reports the resulting split sizes. The same underlying statement partitions are used throughout all probability-estimation procedures. The individual-statement probe is fit to statements from the training split and calibrated using individual statements from the calibration split. The Direct Conditional and Joint probes do not independently repartition the data; instead, their combined training and calibration statements are constructed exclusively from statements already assigned to the corresponding individual-statement training and calibration partitions. Likewise, the test statements define the empirical belief and non-disbelief sets used to construct the (P,x)(P,x) pairs evaluated by both conditional estimators. Consequently, no statement or entity assigned to the test partition is used to fit or calibrate the individual-statement, Direct Conditional, or Joint probes. Official Name Short Name # Layers # Parameters Release Date Source Citation Gemma-77b gemma-7b (b) 2828 8.548.54 B Feb 2121, 20242024 Google [22] Gemma-22-99b gemma-2-9b (b) 4242 9.249.24 B Jun 2727, 20242024 Google [22] Gemma-22-2727b gemma-2-27b (b) 4646 27.2327.23 B Jun 2727, 20242024 Google [22] Llama-3.23.2-33b llama-3.2-3b (b) 2828 3.213.21 B Sep 2525, 20242024 Meta [23] Llama-3.13.1-88b llama-3.1-8b (b) 3232 8.038.03 B Jul 2323, 20242024 Meta [23] Llama-3.13.1-7070b llama-3.1-70b (b) 8080 70.5570.55 B Jul 2323, 20242024 Meta [23] Mistral-77B-v0.30.3 mistral-7b (b) 3232 7.257.25 B May 2222, 20242024 Mistral AI [24] Mistral-Nemo-Base-2407 mistral-12b (b) 4040 12.2512.25 B Jul 1818, 20242024 Mistral AI [32] Mistral-Small-3.13.1-2424B-Base-2503 mistral-3.1-24b (b) 4040 23.5723.57 B Mar 1717, 20252025 Mistral AI [33] Qwen2.52.5-77B qwen-2.5-7b (b) 2828 7.627.62 B Sep 1919, 20242024 Alibaba Cloud [46, 36] Qwen2.52.5-1414B qwen-2.5-14b (b) 4848 14.8014.80 B Sep 1919, 20242024 Alibaba Cloud [46, 36] Qwen2.52.5-7272B qwen-2.5-72b (b) 8080 72.7072.70 B Sep 1919, 20242024 Alibaba Cloud [46, 36] Table A3: Base LLMs used in graded stability experiments. We list the official names of the LLMs according to the HuggingFace repository [45], where they are publicly available. We further specify the shortened name used throughout the paper, the number of layers, parameter count, release date, source organization, and official citation. Appendix C Base models To test whether the primary findings depend on instruction tuning, we repeat the four main figure-level analyses using the pretrained base counterparts of the 1212 instruction-tuned LLMs considered in the main text. Results are reported in Section I.1. These models are matched to the primary models by family and parameter scale, allowing us to evaluate the robustness of the observed graded-stability patterns outside the instruction-tuned setting. Exact model versions and specifications are reported in Table A3. Appendix D sAwMIL training, calibration, and evaluation D.1 Probability calibration The sAwMIL probe produces a vector of raw max-margin scores rather than probabilities. For a K-class probe, let (25) ⁡(s)=[r1(s),…,rK(s)]r(s)=[r_1(s),…,r_K(s)] denote the raw score vector for statement s. Each rk(s)r_k(s) is obtained by applying the corresponding one-versus-all sAwMIL head to every valid token representation in the statement and taking the maximum token-level margin. We fit the probe heads using only the training split and reserve the calibration split for converting these raw scores into probability distributions. After fitting the probe, we compute ⁡(s)r(s) for every statement s in the corresponding calibration set and fit a multinomial logistic-regression model that predicts the ground-truth class from the full K-dimensional score vector. For class k, the resulting calibrated probability is (26) πk(s)=exp⁡(k⊤(s)+bk)∑j=1Kexp⁡(j⊤(s)+bj). _k(s)= \! ( β_k r(s)+b_k ) _j=1^K \! ( β_j r(s)+b_j ). The calibration model is fit by minimizing multinomial log loss using the L-BFGS solver, with inverse regularization strength C=1.0C=1.0, a maximum of 20002000 iterations, and convergence tolerance 10−610^-6. The resulting probabilities are aligned to the fixed probe class ordering, clipped to [10−15,1][10^-15,1] for numerical stability, and renormalized to sum to one. A separate calibration mapping is fit for every probe. Individual-statement and Direct Conditional probes therefore produce calibrated three-class distributions over True,False,Neither\ True, False, Neither\, while Joint probes produce calibrated nine-class distributions over the ordered joint states in T,TF,TN,NT,NF,N,FT,F,FN\ T, TF, TN, NT, NF, N, FT, F, FN\. In each case, calibration parameters are estimated only from the corresponding calibration examples and are held fixed for all subsequent analyses. D.2 Specifications and hyperparameters sAwMIL treats each statement as a bag of token-level hidden representations from a single transformer layer. Specifically, we use the output hidden state of the selected layer at every non-padding token position rather than reducing the statement to a single representation. Model inputs are padded or truncated to a maximum sequence length of 6464 tokens. Before probe fitting, each activation dimension is standardized to zero mean and unit variance using a StandardScaler fit to the pooled token representations from the training split. The same fitted transformation is then applied to calibration and test examples. For the K veracity classes, we train K independent one-versus-all sAwMIL heads. For head k, statements belonging to class k are treated as positive bags and all remaining statements as negative bags. Each head is trained in two stages. First, we fit a linear hinge-loss classifier to the initialized token labels. We then apply this classifier to the instances within each positive bag and select the ktop=2k_top=2 highest-scoring instances. To encourage sparse attribution near the end of the statement, selected instances are retained as positive only when they also occur among the final tail_k=2 tail\_k=2 token positions; all other instances are relabeled as negative. A second classifier with the same specification is then fit to these relabeled instances. At inference time, the final classifier assigns a margin to every valid token, and the maximum token-level margin within the bag becomes the statement-level score for that class. Both stages use an averaged linear SGDClassifier with hinge loss, an intercept, and ℓ2 _2 regularization. We set C=1.0C=1.0 and parameterize the SGD regularization coefficient as (27) α=1Cn,α= 1Cn, where n is the number of token instances used to fit the corresponding one-versus-all head. Optimization uses five epochs, the optimal learning-rate schedule, shuffling at each pass, no early-stopping tolerance, and averaged SGD iterates. We use random seed 00 throughout. No explicit class weighting is applied. We hold these sAwMIL hyperparameters fixed across LLMs, datasets, and probability estimators. The individual-statement and Direct Conditional probes contain three one-versus-all heads, whereas Joint probes contain nine. Direct Conditional and Joint probes are fit separately on their respective combined-statement training sets rather than reusing the individual-statement probe, but use the same sAwMIL training and calibration procedure and the model–dataset-specific layer selected from the sweep described below. Figure A1: sAwMIL layer-selection across layers. Calibration-set multiclass log loss for the three-class individual-statement probe across every decoder layer of each instruction-tuned LLM. Panels (a–l) correspond to the 1212 instruction-tuned LLMs. Within each panel, curves show City Locations (blue, solid), Medical Indications (orange, dashed), and Word Definitions (green, dotted), with circular markers denoting the minimum-log-loss layer selected for each dataset. These minimum-log-loss layers are listed in Table A4. D.3 Layer selection Model City Locations Medical Indications Word Definitions llama-3.2-3b 9 11 13 llama-3.1-8b 12 19 12 llama-3.1-70b 34 34 17 gemma-7b 19 16 17 gemma-2-9b 23 20 21 gemma-2-27b 15 19 18 mistral-7b 13 14 14 mistral-12b 23 20 17 mistral-3.1-24b 15 19 15 qwen-2.5-7b 17 16 18 qwen-2.5-14b 27 26 24 qwen-2.5-72b 57 53 49 Table A4: Selected sAwMIL layers. Zero-indexed layers selected independently for each instruction-tuned LLM and dataset by minimizing three-class calibration log loss. The selected layer is subsequently used for the corresponding individual-statement, Direct Conditional, and Joint sAwMIL probes. For each LLM and dataset, we independently sweep all layers using the individual-statement three-class True,False,Neither\ True, False, Neither\ prediction task. At each layer ℓ , we fit sAwMIL on trainD_train, fit its probability-calibration mapping on calD_cal, and evaluate the resulting calibrated probabilities on calD_cal. We select the layer with the lowest multiclass calibration log loss, (28) ℓ⋆=argminℓ[−1|cal|∑(si,yi)∈callogπyi(ℓ)(si)]. = _ [- 1|D_cal| _(s_i,y_i) _cal π^( )_y_i(s_i) ]. Layer selection is therefore performed separately for every LLM–dataset combination rather than assuming that veracity is most recoverable at a fixed absolute or relative transformer depth. Figure A1 shows the complete sAwMIL layer sweeps, and Table A4 reports the resulting selected layers. Because the calibration mapping and the layer-selection loss are both computed using calD_cal, this sweep is used as a model-selection diagnostic rather than as an independent estimate of held-out calibration performance. The test partition is not used to fit the probe, fit the calibration mapping, or select the layer. Belief sets Conditional-pair coverage Dataset Model |ℬℳ||B_M| |ℳ||X_M| tℳt_M Candidate Direct Joint Joint excl. City Locations llama-3.2-3b 632632 1,2171,217 0.5060.506 768,512768,512 768,512768,512 768,512768,512 0(0.0000%)0\;(0.0000\%) llama-3.1-8b 629629 1,2131,213 0.5430.543 762,348762,348 762,348762,348 762,348762,348 0(0.0000%)0\;(0.0000\%) llama-3.1-70b 628628 1,2121,212 0.4780.478 760,508760,508 760,508760,508 760,471760,471 37(0.0049%)37\;(0.0049\%) gemma-7b 615615 1,1991,199 0.5130.513 736,770736,770 736,770736,770 736,770736,770 0(0.0000%)0\;(0.0000\%) gemma-2-9b 626626 1,2111,211 0.5370.537 757,460757,460 757,460757,460 756,727756,727 733(0.0968%)733\;(0.0968\%) gemma-2-27b 635635 1,2191,219 0.5090.509 773,430773,430 773,430773,430 772,198772,198 1,232(0.1593%)1,232\;(0.1593\%) mistral-7b 630630 1,2151,215 0.5100.510 764,820764,820 764,820764,820 764,544764,544 276(0.0361%)276\;(0.0361\%) mistral-12b 636636 1,2241,224 0.4560.456 777,828777,828 777,828777,828 777,828777,828 0(0.0000%)0\;(0.0000\%) mistral-3.1-24b 637637 1,2221,222 0.5570.557 777,777777,777 777,777777,777 777,709777,709 68(0.0087%)68\;(0.0087\%) qwen-2.5-7b 644644 1,2251,225 0.5030.503 788,256788,256 788,256788,256 788,256788,256 0(0.0000%)0\;(0.0000\%) qwen-2.5-14b 629629 1,2121,212 0.5060.506 761,719761,719 761,719761,719 760,373760,373 1,346(0.1767%)1,346\;(0.1767\%) qwen-2.5-72b 627627 1,2131,213 0.5030.503 759,924759,924 759,924759,924 756,321756,321 3,603(0.4741%)3,603\;(0.4741\%) Medical Indications llama-3.2-3b 684684 1,0181,018 0.5010.501 695,628695,628 695,628695,628 695,628695,628 0(0.0000%)0\;(0.0000\%) llama-3.1-8b 694694 1,0351,035 0.4660.466 717,596717,596 717,596717,596 717,596717,596 0(0.0000%)0\;(0.0000\%) llama-3.1-70b 683683 1,0251,025 0.5000.500 699,392699,392 699,392699,392 699,392699,392 0(0.0000%)0\;(0.0000\%) gemma-7b 715715 1,0451,045 0.3680.368 746,460746,460 746,460746,460 746,460746,460 0(0.0000%)0\;(0.0000\%) gemma-2-9b 690690 1,0311,031 0.5050.505 710,700710,700 710,700710,700 710,700710,700 0(0.0000%)0\;(0.0000\%) gemma-2-27b 685685 1,0221,022 0.4570.457 699,385699,385 699,385699,385 699,385699,385 0(0.0000%)0\;(0.0000\%) mistral-7b 688688 1,0231,023 0.3570.357 703,136703,136 703,136703,136 703,136703,136 0(0.0000%)0\;(0.0000\%) mistral-12b 686686 1,0231,023 0.3710.371 701,092701,092 701,092701,092 701,092701,092 0(0.0000%)0\;(0.0000\%) mistral-3.1-24b 690690 1,0311,031 0.5000.500 710,700710,700 710,700710,700 710,623710,623 77(0.0108%)77\;(0.0108\%) qwen-2.5-7b 707707 1,0531,053 0.4350.435 743,764743,764 743,764743,764 743,760743,760 4(0.0005%)4\;(0.0005\%) qwen-2.5-14b 710710 1,0501,050 0.5000.500 744,790744,790 744,790744,790 744,779744,779 11(0.0015%)11\;(0.0015\%) qwen-2.5-72b 685685 1,0271,027 0.5000.500 702,810702,810 702,810702,810 702,379702,379 431(0.0613%)431\;(0.0613\%) Word Definitions llama-3.2-3b 610610 1,5841,584 0.3950.395 965,630965,630 965,630965,630 965,630965,630 0(0.0000%)0\;(0.0000\%) llama-3.1-8b 641641 1,6061,606 0.4470.447 1,028,8051,028,805 1,028,8051,028,805 1,028,8051,028,805 0(0.0000%)0\;(0.0000\%) llama-3.1-70b 638638 1,6101,610 0.4470.447 1,026,5421,026,542 1,026,5421,026,542 1,026,5421,026,542 0(0.0000%)0\;(0.0000\%) gemma-7b 638638 1,6231,623 0.3980.398 1,034,8361,034,836 1,034,8361,034,836 1,034,8361,034,836 0(0.0000%)0\;(0.0000\%) gemma-2-9b 606606 1,5801,580 0.4690.469 956,874956,874 956,874956,874 956,874956,874 0(0.0000%)0\;(0.0000\%) gemma-2-27b 618618 1,5841,584 0.4410.441 978,294978,294 978,294978,294 978,294978,294 0(0.0000%)0\;(0.0000\%) mistral-7b 639639 1,6121,612 0.4370.437 1,029,4291,029,429 1,029,4291,029,429 1,029,4291,029,429 0(0.0000%)0\;(0.0000\%) mistral-12b 617617 1,5891,589 0.3770.377 979,796979,796 979,796979,796 979,796979,796 0(0.0000%)0\;(0.0000\%) mistral-3.1-24b 619619 1,5921,592 0.4610.461 984,829984,829 984,829984,829 984,829984,829 0(0.0000%)0\;(0.0000\%) qwen-2.5-7b 627627 1,6161,616 0.4090.409 1,012,6051,012,605 1,012,6051,012,605 1,012,6051,012,605 0(0.0000%)0\;(0.0000\%) qwen-2.5-14b 620620 1,5941,594 0.4250.425 987,660987,660 987,660987,660 987,660987,660 0(0.0000%)0\;(0.0000\%) qwen-2.5-72b 617617 1,5931,593 0.4690.469 982,264982,264 982,264982,264 982,264982,264 0(0.0000%)0\;(0.0000\%) Table A5: Belief-set and conditional-pair coverage using sAwMIL. For each LLM and domain, |ℬℳ||B_M| and |ℳ||X_M| denote the numbers of beliefs and non-disbeliefs, respectively, and tℳt_M is the empirical belief threshold. Candidate is the number |ℬℳ|(|ℳ|−1)|B_M|(|X_M|-1) of constructed (P,x)(P,x) pairs after excluding self-pairs. Direct and Joint report the numbers of valid conditional-probability estimates. Joint excl. reports the number and percentage of candidate pairs excluded because the Joint-to-Conditional denominator is at most 10−1210^-12. Appendix E Construction and coverage of conditional probability estimates E.1 Belief sets and conditional-pair coverage Table A5 reports the empirical belief sets, non-disbelief sets, belief thresholds, and conditional-pair coverage for the primary sAwMIL probe. For each model and dataset, the candidate set contains every pair (P,x)(P,x) with P∈ℬℳP _M and x∈ℳ∖Px _M \P\, yielding |ℬℳ|⋅(|ℳ|−1)|B_M|·(|X_M|-1) total candidate pairs. We report the number of candidate pairs successfully assigned a conditional-probability estimate by the Direct Conditional and Joint-to-Conditional estimators, together with the number of Joint-to-Conditional pairs excluded due to an undefined denominator in Eq. 19. The Direct Conditional estimator yields a valid estimate for every candidate pair across all 3636 model–dataset combinations. Joint-to-Conditional coverage is also nearly complete: only 7,8187,818 of 29,732,36929,732,369 candidate pairs (0.026%0.026\%) are excluded because of a numerically zero denominator. No Joint-to-Conditional pairs are excluded for Word Definitions, and the largest exclusion rate for any individual model–dataset combination is 0.4741%0.4741\%. E.2 Joint-to-Conditional derivation We derive the Joint-to-Conditional estimator in Equation (19) from the nine-state joint distribution ℳjoint(P,x) π_M^joint(P,x). Recall that πabjoint(P,x) _ab^joint(P,x) denotes the probability assigned to the joint state in which the antecedent x has state a∈T,F,Na∈\T,F,N\ and the consequent P has state b∈T,F,Nb∈\T,F,N\. Under Cooper–Cantwell semantics, the conditional P|xP x takes the truth value of P when x is True or Neither, and is Neither when x is False. Collapsing the nine joint states according to this truth table therefore gives the induced trivalent conditional distribution (29) πTjoint,C(P,x) _T^joint,C(P,x) =πTTjoint(P,x)+πNTjoint(P,x), = _T^joint(P,x)+ _NT^joint(P,x), (30) πFjoint,C(P,x) _F^joint,C(P,x) =πTFjoint(P,x)+πNFjoint(P,x), = _TF^joint(P,x)+ _NF^joint(P,x), (31) πNjoint,C(P,x) _N^joint,C(P,x) =πTNjoint(P,x)+πNNjoint(P,x)+πFTjoint(P,x)+πFFjoint(P,x)+πFNjoint(P,x). = _TN^joint(P,x)+ _N^joint(P,x)+ _FT^joint(P,x)+ _F^joint(P,x)+ _FN^joint(P,x). Here, the superscript C denotes the conditional distribution induced from the joint probe. Cantwell’s non-bivalent probability rule assigns a sentence its probability of being True conditional on its having a determinate truth value [7]. Applying this normalization to the induced conditional distribution yields (32) Pr^ℳjoint(P∣x) _M^joint(P x) =πTjoint,C(P,x)πTjoint,C(P,x)+πFjoint,C(P,x) = _T^joint,C(P,x) _T^joint,C(P,x)+ _F^joint,C(P,x) (33) =πTTjoint(P,x)+πNTjoint(P,x)πTTjoint(P,x)+πTFjoint(P,x)+πNTjoint(P,x)+πNFjoint(P,x), = _T^joint(P,x)+ _NT^joint(P,x) _T^joint(P,x)+ _TF^joint(P,x)+ _NT^joint(P,x)+ _NF^joint(P,x), recovering Equation (19). The numerator contains exactly the joint states that induce a True conditional, while the denominator contains all states that induce either a True or False conditional. The estimator is therefore undefined when the conditional is assigned probability one of being Neither; as described in Section 4.3.2, we treat denominators less than or equal to 10−1210^-12 as numerically zero and exclude the corresponding pairs. This normalization also clarifies the distinction between the two conditional-probability estimators. The Direct Conditional estimator uses the raw True-state mass πTdirect(P,x) _T^direct(P,x), whereas Joint-to-Conditional normalizes the induced True mass over the True and False conditional states. Figure A2: Robustness of the natural cubic spline. Violin plots show the distribution across the 1212 instruction-tuned LLMs of held-out R2R^2 for predicting graded stability from individual belief probability using a linear baseline and natural cubic splines with 22–55 degrees of freedom for (a) City Locations, (b) Medical Indications, and (c) Word Definitions. Horizontal black lines denote the median across models, and the shaded column marks the primary specification with 33 degrees of freedom. Predictive performance is similar across the 33–55 degree-of-freedom specifications, indicating that the main conclusion is not sensitive to the precise spline flexibility. Appendix F Supplementary validation of graded belief stability F.1 Robustness of the stability–belief-probability relationship Our primary analysis models graded stability as a function of individual belief probability using a natural cubic spline with three degrees of freedom. To evaluate sensitivity to this choice, we repeat the analysis using a linear baseline and natural cubic splines with 22, 33, 44, and 55 degrees of freedom. We use five-fold cross-validation to evaluate the cubic spline models. Propositions are randomly permuted using a deterministic seed derived from base seed 00 and divided into five approximately equal folds. The same fold assignment is reused across spline specifications. Within each fold, we construct a centered natural cubic spline basis from the training propositions with fixed boundaries at 00 and 11, estimate its coefficients by ordinary least squares, and apply the resulting basis to the held-out propositions. The linear baseline is fit analogously using an intercept and linear term in πT(P) _T(P). We apply the same inclusion criteria as in the primary analysis, requiring at least 5050 valid proposition-level stability scores and sufficient distinct values of πT(P) _T(P) to fit the requested spline. For the cross-model residual-agreement analysis in Section 4.5.1, we use the out-of-fold residuals from the primary df=3df=3 specification and retain model pairs sharing at least 5050 believed propositions. We generate 1,0001,000 permutation samples by independently shuffling residuals across proposition identities within each model while preserving model overlap, recompute the median pairwise Spearman correlation for each permutation, and calculate the one-sided Monte Carlo p-value using the standard +1+1 correction. Permutations use deterministic seeds derived from the same base seed 00. Across all three domains, the primary df=3df=3 specification performs similarly to the more flexible df=4df=4 and df=5df=5 alternatives (Fig. A2). The linear baseline and df=2df=2 spline tend to yield lower held-out R2R^2, particularly for City Locations, indicating that some nonlinearity is useful. Increasing flexibility beyond the primary specification, however, produces little systematic improvement. The conclusion that individual belief probability predicts a meaningful but incomplete fraction of graded stability therefore does not depend on the exact spline complexity. F.2 Additional probabilistic-coherence analyses F.2.1 Exact pairwise coherence conditions We show that the four constraints in Equation (12) are both necessary and sufficient for the three measured distributions associated with a single (P,x)(P,x) pair to admit a common trivalent joint representation. For compactness, we write pb=πb(P),ab=πb(x),cb=πbdirect(P,x),b∈T,F,N.p_b= _b(P), a_b= _b(x), c_b= _b^direct(P,x), b∈\T,F,N\. Let JabJ_ab denote the probability assigned by a latent joint distribution to the state in which x has value a and P has value b, with the first coordinate corresponding to the antecedent. The row marginals of J must equal ℳ(x) π_M(x) and its column marginals must equal ℳ(P) π_M(P). Under the Cooper–Cantwell truth table, such a joint distribution induces the conditional distribution (34) cT c_T =JTT+JNT, =J_T+J_NT, (35) cF c_F =JTF+JNF, =J_TF+J_NF, (36) cN c_N =JTN+JNN+JFT+JFF+JFN. =J_TN+J_N+J_FT+J_F+J_FN. Necessity. Suppose first that a representing joint distribution J exists. Combining its marginals with the induced conditional distribution gives (37) pT−cT p_T-c_T =JFT≥0, =J_FT≥ 0, (38) pF−cF p_F-c_F =JFF≥0, =J_F≥ 0, (39) cN−aF c_N-a_F =JTN+JNN≥0, =J_TN+J_N≥ 0, (40) pN+aF−cN p_N+a_F-c_N =JFN≥0. =J_FN≥ 0. It follows that cT≤pT,cF≤pF,aF≤cN,cN≤pN+aF,c_T≤ p_T, c_F≤ p_F, a_F≤ c_N, c_N≤ p_N+a_F, which are exactly the four constraints in Equation (12). Sufficiency. Conversely, suppose the four constraints hold. Define the vectors (41) =(cT,cF,cN−aF), = (c_T,\;c_F,\;c_N-a_F ), (42) =(pT−cT,pF−cF,pN+aF−cN), = (p_T-c_T,\;p_F-c_F,\;p_N+a_F-c_N ), with coordinates ordered as (T,F,N)(T,F,N). The four coherence constraints imply that every coordinate of r and f is nonnegative. Because p and c are each normalized probability distributions, ∑b∈T,F,Nrb=1−aF,∑b∈T,F,Nfb=aF,rb+fb=pb. _b∈\T,F,N\r_b=1-a_F, _b∈\T,F,N\f_b=a_F, r_b+f_b=p_b. The vector f can therefore be used as the False-antecedent row of the latent joint distribution, while r supplies the total mass of the True- and Neither-antecedent rows. When aF<1a_F<1, define (43) JFb=fb,JTb=aT1−aFrb,JNb=aN1−aFrb,b∈T,F,N.J_Fb=f_b, J_Tb= a_T1-a_Fr_b, J_Nb= a_N1-a_Fr_b, b∈\T,F,N\. All entries are nonnegative. Their row sums are aFa_F, aTa_T, and aNa_N, respectively, and because aT+aN=1−aFa_T+a_N=1-a_F, the column sum for each b is JTb+JNb+JFb=rb+fb=pb.J_Tb+J_Nb+J_Fb=r_b+f_b=p_b. Thus, J has the required marginals. Moreover, the induced conditional has True and False masses rT=cTr_T=c_T and rF=cFr_F=c_F, while its Neither mass is aF+rN=aF+(cN−aF)=cN.a_F+r_N=a_F+(c_N-a_F)=c_N. Hence, J reproduces all three measured distributions. The remaining case is aF=1a_F=1. Normalization then implies aT=aN=0a_T=a_N=0. Because aF≤cNa_F≤ c_N and c is normalized, cN=1c_N=1 and cT=cF=0c_T=c_F=0. Setting JFb=pb,JTb=JNb=0J_Fb=p_b, J_Tb=J_Nb=0 for every b∈T,F,Nb∈\T,F,N\ therefore provides a valid joint distribution with the required marginals and an everywhere-Neither conditional. The four inequalities in Equation (12) are therefore necessary and sufficient for exact pairwise compatibility, provided that the three measured triples are themselves normalized nonnegative probability distributions. Projection onto the coherent set. The pairwise coherent set C used to compute dCCKd_CCK is the intersection of three probability simplexes with the four closed linear half-spaces in Equation (12). It is therefore a nonempty, compact, and convex subset of ℝ9R^9. Consequently, every measured vector P,xv_P,x has a unique Euclidean projection Π(P,x) _C(v_P,x) onto this set, and Equation (13) is well defined. Scope of the coherence analysis. These conditions characterize compatibility for a single (P,x)(P,x) pair. Testing them separately across pairs does not establish that all measured probabilities for an LLM can be generated by one global joint distribution over its entire belief system. Such simultaneous compatibility is a stronger global feasibility problem requiring a shared probability distribution over admissible trivalent valuations of all propositions. Our coherence analysis tests the pairwise condition only. F.2.2 Exact coherence Dataset Model Exact (,)>() _T^direct(P,x)> _T(P) (,)>() _F^direct(P,x)> _F(P) ()>(,) _F(x)> _N^direct(P,x) (,)>()+() _N^direct(P,x)> _N(P)+ _F(x) City Locations llama-3.2-3b 0.000%0.000\% 2.7%(0.095)2.7\%\;(0.095) 90.4%(0.154)90.4\%\;(0.154) 3.4%(0.122)3.4\%\;(0.122) 96.6%(0.207)96.6\%\;(0.207) llama-3.1-8b 0.000%0.000\% 5.2%(0.054)5.2\%\;(0.054) 88.0%(0.051)88.0\%\;(0.051) 5.8%(0.047)5.8\%\;(0.047) 94.2%(0.080)94.2\%\;(0.080) llama-3.1-70b <0.001%<0.001\% 7.9%(0.031)7.9\%\;(0.031) 80.5%(0.035)80.5\%\;(0.035) 11.4%(0.026)11.4\%\;(0.026) 88.5%(0.047)88.5\%\;(0.047) gemma-7b <0.001%<0.001\% 3.8%(0.107)3.8\%\;(0.107) 92.4%(0.159)92.4\%\;(0.159) 7.9%(0.091)7.9\%\;(0.091) 92.1%(0.102)92.1\%\;(0.102) gemma-2-9b 0.000%0.000\% 3.1%(0.050)3.1\%\;(0.050) 93.9%(0.087)93.9\%\;(0.087) 11.0%(0.036)11.0\%\;(0.036) 89.0%(0.046)89.0\%\;(0.046) gemma-2-27b 0.000%0.000\% 2.4%(0.067)2.4\%\;(0.067) 93.7%(0.104)93.7\%\;(0.104) 5.9%(0.073)5.9\%\;(0.073) 94.1%(0.109)94.1\%\;(0.109) mistral-7b 0.000%0.000\% 3.5%(0.097)3.5\%\;(0.097) 91.7%(0.077)91.7\%\;(0.077) 8.8%(0.053)8.8\%\;(0.053) 91.2%(0.093)91.2\%\;(0.093) mistral-12b 0.004%0.004\% 9.0%(0.055)9.0\%\;(0.055) 62.5%(0.082)62.5\%\;(0.082) 11.1%(0.075)11.1\%\;(0.075) 88.7%(0.112)88.7\%\;(0.112) mistral-3.1-24b 0.000%0.000\% 8.8%(0.027)8.8\%\;(0.027) 86.0%(0.074)86.0\%\;(0.074) 21.2%(0.022)21.2\%\;(0.022) 78.8%(0.068)78.8\%\;(0.068) qwen-2.5-7b <0.001%<0.001\% 4.6%(0.052)4.6\%\;(0.052) 88.0%(0.125)88.0\%\;(0.125) 16.6%(0.052)16.6\%\;(0.052) 83.4%(0.112)83.4\%\;(0.112) qwen-2.5-14b 0.000%0.000\% 8.8%(0.032)8.8\%\;(0.032) 78.8%(0.066)78.8\%\;(0.066) 18.5%(0.037)18.5\%\;(0.037) 81.5%(0.077)81.5\%\;(0.077) qwen-2.5-72b 0.000%0.000\% 8.4%(0.031)8.4\%\;(0.031) 74.4%(0.056)74.4\%\;(0.056) 15.0%(0.039)15.0\%\;(0.039) 85.0%(0.077)85.0\%\;(0.077) Medical Indications llama-3.2-3b 0.000%0.000\% 3.6%(0.082)3.6\%\;(0.082) 63.8%(0.153)63.8\%\;(0.153) 12.1%(0.138)12.1\%\;(0.138) 87.9%(0.276)87.9\%\;(0.276) llama-3.1-8b 0.008%0.008\% 5.2%(0.092)5.2\%\;(0.092) 59.7%(0.111)59.7\%\;(0.111) 13.2%(0.132)13.2\%\;(0.132) 86.7%(0.252)86.7\%\;(0.252) llama-3.1-70b 0.000%0.000\% 7.4%(0.088)7.4\%\;(0.088) 65.8%(0.111)65.8\%\;(0.111) 13.6%(0.127)13.6\%\;(0.127) 86.4%(0.179)86.4\%\;(0.179) gemma-7b 0.097%0.097\% 2.1%(0.095)2.1\%\;(0.095) 60.6%(0.152)60.6\%\;(0.152) 20.8%(0.132)20.8\%\;(0.132) 79.0%(0.250)79.0\%\;(0.250) gemma-2-9b 0.000%0.000\% 8.4%(0.094)8.4\%\;(0.094) 72.1%(0.119)72.1\%\;(0.119) 21.8%(0.129)21.8\%\;(0.129) 78.1%(0.148)78.1\%\;(0.148) gemma-2-27b 0.015%0.015\% 9.4%(0.103)9.4\%\;(0.103) 68.9%(0.100)68.9\%\;(0.100) 20.6%(0.134)20.6\%\;(0.134) 79.3%(0.150)79.3\%\;(0.150) mistral-7b <0.001%<0.001\% 7.3%(0.085)7.3\%\;(0.085) 63.8%(0.090)63.8\%\;(0.090) 21.6%(0.134)21.6\%\;(0.134) 78.4%(0.183)78.4\%\;(0.183) mistral-12b 0.200%0.200\% 13.4%(0.099)13.4\%\;(0.099) 45.4%(0.107)45.4\%\;(0.107) 31.7%(0.132)31.7\%\;(0.132) 67.9%(0.184)67.9\%\;(0.184) mistral-3.1-24b 0.015%0.015\% 11.9%(0.096)11.9\%\;(0.096) 58.2%(0.054)58.2\%\;(0.054) 19.8%(0.134)19.8\%\;(0.134) 80.1%(0.128)80.1\%\;(0.128) qwen-2.5-7b 0.035%0.035\% 3.5%(0.087)3.5\%\;(0.087) 75.8%(0.184)75.8\%\;(0.184) 23.4%(0.132)23.4\%\;(0.132) 76.2%(0.204)76.2\%\;(0.204) qwen-2.5-14b 0.000%0.000\% 6.3%(0.088)6.3\%\;(0.088) 72.9%(0.125)72.9\%\;(0.125) 25.3%(0.127)25.3\%\;(0.127) 74.7%(0.178)74.7\%\;(0.178) qwen-2.5-72b 0.000%0.000\% 7.1%(0.086)7.1\%\;(0.086) 74.7%(0.079)74.7\%\;(0.079) 22.6%(0.114)22.6\%\;(0.114) 77.4%(0.144)77.4\%\;(0.144) Word Definitions llama-3.2-3b 0.073%0.073\% 6.6%(0.091)6.6\%\;(0.091) 64.2%(0.100)64.2\%\;(0.100) 5.7%(0.106)5.7\%\;(0.106) 93.9%(0.223)93.9\%\;(0.223) llama-3.1-8b 0.020%0.020\% 4.6%(0.086)4.6\%\;(0.086) 77.5%(0.103)77.5\%\;(0.103) 6.8%(0.125)6.8\%\;(0.125) 93.0%(0.177)93.0\%\;(0.177) llama-3.1-70b 0.018%0.018\% 6.5%(0.085)6.5\%\;(0.085) 73.7%(0.103)73.7\%\;(0.103) 4.7%(0.121)4.7\%\;(0.121) 95.1%(0.185)95.1\%\;(0.185) gemma-7b 0.206%0.206\% 2.7%(0.088)2.7\%\;(0.088) 64.1%(0.140)64.1\%\;(0.140) 8.1%(0.118)8.1\%\;(0.118) 91.4%(0.259)91.4\%\;(0.259) gemma-2-9b 0.012%0.012\% 8.2%(0.109)8.2\%\;(0.109) 82.3%(0.135)82.3\%\;(0.135) 10.1%(0.123)10.1\%\;(0.123) 88.8%(0.120)88.8\%\;(0.120) gemma-2-27b 0.003%0.003\% 3.6%(0.109)3.6\%\;(0.109) 87.2%(0.145)87.2\%\;(0.145) 5.1%(0.110)5.1\%\;(0.110) 94.7%(0.169)94.7\%\;(0.169) mistral-7b 0.030%0.030\% 4.7%(0.094)4.7\%\;(0.094) 72.5%(0.097)72.5\%\;(0.097) 6.4%(0.124)6.4\%\;(0.124) 93.2%(0.207)93.2\%\;(0.207) mistral-12b 0.054%0.054\% 2.7%(0.092)2.7\%\;(0.092) 73.1%(0.132)73.1\%\;(0.132) 5.6%(0.126)5.6\%\;(0.126) 94.3%(0.323)94.3\%\;(0.323) mistral-3.1-24b 0.026%0.026\% 7.0%(0.101)7.0\%\;(0.101) 76.0%(0.103)76.0\%\;(0.103) 7.8%(0.125)7.8\%\;(0.125) 91.8%(0.173)91.8\%\;(0.173) qwen-2.5-7b 0.080%0.080\% 7.7%(0.108)7.7\%\;(0.108) 76.2%(0.135)76.2\%\;(0.135) 10.9%(0.120)10.9\%\;(0.120) 87.6%(0.189)87.6\%\;(0.189) qwen-2.5-14b 0.008%0.008\% 6.2%(0.093)6.2\%\;(0.093) 79.7%(0.132)79.7\%\;(0.132) 9.0%(0.113)9.0\%\;(0.113) 90.7%(0.162)90.7\%\;(0.162) qwen-2.5-72b 0.021%0.021\% 3.8%(0.082)3.8\%\;(0.082) 82.0%(0.164)82.0\%\;(0.164) 8.2%(0.117)8.2\%\;(0.117) 91.6%(0.189)91.6\%\;(0.189) Table A6: Exact CCK coherence and constraint violations for Direct Conditional estimates using sAwMIL. For each LLM and domain, we report the percentage of (P,x)(P,x) pairs satisfying all four CCK constraints simultaneously within tolerance 10−810^-8. Here, 0.000%0.000\% signifies exactly zero fully coherent pairs, while <0.001%<0.001\% means a small number of exactly coherent pairs exists. Each remaining column reports the percentage of pairs violating each constraint, with the mean excess across violating pairs shown in parentheses. Exact coherence is rare across all models and domains, with no model–domain combination exceeding 0.206%0.206\%. Violations occur most frequently for the constraints on conditional False and Neither probability mass. Table A6 reports exact CCK-coherence rates and individual constraint violations for the primary Direct Conditional estimates obtained using sAwMIL. As described in Section 4.5.2, a (P,x)(P,x) pair is classified as exactly coherent when all four CCK inequalities are satisfied within the numerical tolerance of 10−810^-8. For each inequality, we additionally report how frequently it is violated and the mean signed excess among the pairs that violate it. Exact coherence is rare for every model and domain (Tab. A6), with the largest observed rate equal to 0.206%0.206\%. The two most frequently violated conditions are πFdirect(P,x)≤πF(P) _F^direct(P,x)≤ _F(P) and πNdirect(P,x)≤πN(P)+πF(x) _N^direct(P,x)≤ _N(P)+ _F(x), whereas violations of the remaining two inequalities occur substantially less often. Because exact coherence requires all four constraints to hold simultaneously, these frequent violations drive the near-zero exact-coherence rates reported in the main text and drive the distance-to-coherence analysis in Section 4.5.2. One likely contributor to this pattern is the scale of the corresponding individual-statement probabilities. Because P is a probe-defined belief, πF(P) _F(P) and πN(P) _N(P) are often small. Similarly, because ℳX_M contains non-disbeliefs, its members x often have small πF(x) _F(x). Even modest conditional probability mass assigned to False or Neither can therefore violate the second or fourth constraint. The fourth constraint may additionally be sensitive to the Direct Conditional representation itself: in “Given x, P,” the conditioning proposition precedes P in the autoregressive sequence, allowing the representation of the conditional to incorporate x and potentially shift probability mass toward Neither relative to P in isolation. Dataset Model n |ΔπT|| _T| |Δγ|| γ| City Locations llama-3.2-3b 315315 0.004350.00435 (0.01055)(0.01055) 0.2350.235 (0.173)(0.173) llama-3.1-8b 305305 0.001810.00181 (0.00556)(0.00556) 0.0900.090 (0.122)(0.122) llama-3.1-70b 287287 0.004830.00483 (0.02893)(0.02893) 0.0660.066 (0.116)(0.116) gemma-7b 304304 0.003770.00377 (0.01042)(0.01042) 0.1750.175 (0.175)(0.175) gemma-2-9b 291291 0.003700.00370 (0.01686)(0.01686) 0.0820.082 (0.141)(0.141) gemma-2-27b 316316 0.003620.00362 (0.01372)(0.01372) 0.1330.133 (0.143)(0.143) mistral-7b 303303 0.004560.00456 (0.01924)(0.01924) 0.1130.113 (0.157)(0.157) mistral-12b 278278 0.004950.00495 (0.01547)(0.01547) 0.1460.146 (0.189)(0.189) mistral-3.1-24b 307307 0.003620.00362 (0.01821)(0.01821) 0.1000.100 (0.151)(0.151) qwen-2.5-7b 312312 0.003500.00350 (0.01001)(0.01001) 0.1360.136 (0.157)(0.157) qwen-2.5-14b 290290 0.003140.00314 (0.01289)(0.01289) 0.1030.103 (0.159)(0.159) qwen-2.5-72b 276276 0.002820.00282 (0.01189)(0.01189) 0.0960.096 (0.178)(0.178) Medical Indications llama-3.2-3b 342342 0.004420.00442 (0.00530)(0.00530) 0.2340.234 (0.185)(0.185) llama-3.1-8b 344344 0.003890.00389 (0.00518)(0.00518) 0.2480.248 (0.204)(0.204) llama-3.1-70b 340340 0.004040.00404 (0.00763)(0.00763) 0.1980.198 (0.181)(0.181) gemma-7b 356356 0.004300.00430 (0.00639)(0.00639) 0.2660.266 (0.217)(0.217) gemma-2-9b 344344 0.003810.00381 (0.00535)(0.00535) 0.2100.210 (0.206)(0.206) gemma-2-27b 338338 0.004080.00408 (0.00648)(0.00648) 0.1820.182 (0.204)(0.204) mistral-7b 329329 0.004560.00456 (0.00993)(0.00993) 0.1790.179 (0.223)(0.223) mistral-12b 339339 0.005150.00515 (0.00856)(0.00856) 0.2000.200 (0.204)(0.204) mistral-3.1-24b 337337 0.003790.00379 (0.00564)(0.00564) 0.1380.138 (0.176)(0.176) qwen-2.5-7b 353353 0.004410.00441 (0.00579)(0.00579) 0.2630.263 (0.206)(0.206) qwen-2.5-14b 354354 0.004060.00406 (0.00494)(0.00494) 0.2320.232 (0.206)(0.206) qwen-2.5-72b 337337 0.004010.00401 (0.00719)(0.00719) 0.1550.155 (0.191)(0.191) Word Definitions llama-3.2-3b 303303 0.005100.00510 (0.00960)(0.00960) 0.2210.221 (0.233)(0.233) llama-3.1-8b 316316 0.004080.00408 (0.00787)(0.00787) 0.1690.169 (0.191)(0.191) llama-3.1-70b 316316 0.004290.00429 (0.00772)(0.00772) 0.1990.199 (0.205)(0.205) gemma-7b 317317 0.005380.00538 (0.00766)(0.00766) 0.2620.262 (0.230)(0.230) gemma-2-9b 300300 0.004880.00488 (0.01071)(0.01071) 0.1790.179 (0.190)(0.190) gemma-2-27b 307307 0.004600.00460 (0.01048)(0.01048) 0.2040.204 (0.200)(0.200) mistral-7b 315315 0.004750.00475 (0.00910)(0.00910) 0.2010.201 (0.218)(0.218) mistral-12b 308308 0.005530.00553 (0.01139)(0.01139) 0.3000.300 (0.254)(0.254) mistral-3.1-24b 306306 0.004620.00462 (0.00839)(0.00839) 0.2010.201 (0.218)(0.218) qwen-2.5-7b 311311 0.004470.00447 (0.00834)(0.00834) 0.1890.189 (0.199)(0.199) qwen-2.5-14b 308308 0.004590.00459 (0.00962)(0.00962) 0.1690.169 (0.187)(0.187) qwen-2.5-72b 305305 0.004740.00474 (0.00925)(0.00925) 0.2530.253 (0.226)(0.226) Table A7: Behavioral matching quality. For each LLM and domain, n is the number of matched proposition pairs, |ΔπT|| _T| is the absolute difference in belief probability between the two propositions in each pair, and |Δγ|| γ| is their absolute difference in graded stability. Values for |ΔπT|| _T| and |Δγ|| γ| report the mean with standard deviation in parentheses. Across domains and models, matching produces small differences in individual belief probability while retaining substantially larger differences in graded stability. Appendix G Additional behavioral results G.1 Matched-pairs analysis Table A7 reports diagnostics for the probability-matched pairs used in our primary behavioral analysis. As described in Section 4.6, matching is performed separately within each challenge-sequence block so that both propositions in a pair receive the same conversational challenges in the same order. The reported sample additionally requires both propositions’ initial behavioral judgments to agree with their individual-statement probe classifications, as in the primary analysis. For each model, n denotes the number of pairs, while |ΔπT|| _T| and |Δγ|| γ| quantify pairwise differences in individual belief probability and graded stability, respectively. Across all three domains, matching yields small differences in individual belief probability while retaining substantially larger differences in graded stability. Mean |ΔπT|| _T| is at most 0.005530.00553 across the reported model–domain combinations, whereas mean |Δγ|| γ| ranges from 0.0660.066 to 0.3000.300. Thus, the matched pairs are closely aligned in individual belief probability while preserving meaningful variation in graded stability. G.2 Uncertainty in behavioral resilience effects Figure 5 reports ±1± 1 bootstrap standard error to visualize the precision of each model–domain effect. Table A8 reports the corresponding 95%95\% percentile bootstrap confidence intervals for mean ΔM M. Mean ΔM M is positive in 83.3%83.3\% of model–domain settings. The 95%95\% confidence interval is entirely positive in 1111 of 3636 (30.6%)(30.6\%) settings, entirely negative in 11 of 3636 (2.8%)(2.8\%), and includes zero in the remaining 2424 of 3636 (66.7%)(66.7\%). Dataset Model Mean ΔM M Bootstrap SE 95% CI City Locations llama-3.2-3b −0.0256-0.0256 0.01180.0118 [−0.0488,−0.0018][-0.0488,\,-0.0018] llama-3.1-8b 0.02200.0220 0.03080.0308 [−0.0402, 0.0815][-0.0402,\,0.0815] llama-3.1-70b 0.00010.0001 0.00490.0049 [−0.0098, 0.0095][-0.0098,\,0.0095] gemma-7b 0.01010.0101 0.00800.0080 [−0.0058, 0.0256][-0.0058,\,0.0256] gemma-2-9b 0.01800.0180 0.00770.0077 [0.0030, 0.0329][0.0030,\,0.0329] gemma-2-27b 0.07840.0784 0.01440.0144 [0.0497, 0.1069][0.0497,\,0.1069] mistral-7b 0.04260.0426 0.01470.0147 [0.0156, 0.0730][0.0156,\,0.0730] mistral-12b 0.09380.0938 0.01790.0179 [0.0582, 0.1289][0.0582,\,0.1289] mistral-3.1-24b −0.0012-0.0012 0.00760.0076 [−0.0160, 0.0137][-0.0160,\,0.0137] qwen-2.5-7b 0.01070.0107 0.00950.0095 [−0.0080, 0.0293][-0.0080,\,0.0293] qwen-2.5-14b 0.00890.0089 0.00680.0068 [−0.0040, 0.0227][-0.0040,\,0.0227] qwen-2.5-72b 0.01070.0107 0.00640.0064 [−0.0015, 0.0238][-0.0015,\,0.0238] Medical Indications llama-3.2-3b −0.0070-0.0070 0.01530.0153 [−0.0368, 0.0231][-0.0368,\,0.0231] llama-3.1-8b 0.02550.0255 0.03070.0307 [−0.0357, 0.0860][-0.0357,\,0.0860] llama-3.1-70b 0.00970.0097 0.01150.0115 [−0.0131, 0.0323][-0.0131,\,0.0323] gemma-7b 0.07640.0764 0.01690.0169 [0.0438, 0.1096][0.0438,\,0.1096] gemma-2-9b 0.05220.0522 0.01310.0131 [0.0265, 0.0778][0.0265,\,0.0778] gemma-2-27b 0.02890.0289 0.01670.0167 [−0.0040, 0.0611][-0.0040,\,0.0611] mistral-7b 0.02560.0256 0.02130.0213 [−0.0160, 0.0680][-0.0160,\,0.0680] mistral-12b 0.03440.0344 0.01730.0173 [0.0003, 0.0687][0.0003,\,0.0687] mistral-3.1-24b 0.00050.0005 0.00760.0076 [−0.0140, 0.0156][-0.0140,\,0.0156] qwen-2.5-7b 0.10250.1025 0.01470.0147 [0.0740, 0.1313][0.0740,\,0.1313] qwen-2.5-14b 0.00810.0081 0.01250.0125 [−0.0162, 0.0328][-0.0162,\,0.0328] qwen-2.5-72b −0.0058-0.0058 0.01120.0112 [−0.0273, 0.0160][-0.0273,\,0.0160] Word Definitions llama-3.2-3b 0.00500.0050 0.02520.0252 [−0.0418, 0.0561][-0.0418,\,0.0561] llama-3.1-8b 0.06070.0607 0.05580.0558 [−0.0515, 0.1659][-0.0515,\,0.1659] llama-3.1-70b 0.02110.0211 0.01540.0154 [−0.0088, 0.0510][-0.0088,\,0.0510] gemma-7b 0.04080.0408 0.01390.0139 [0.0135, 0.0681][0.0135,\,0.0681] gemma-2-9b 0.03460.0346 0.01640.0164 [0.0034, 0.0672][0.0034,\,0.0672] gemma-2-27b 0.01450.0145 0.02280.0228 [−0.0302, 0.0584][-0.0302,\,0.0584] mistral-7b 0.05680.0568 0.02810.0281 [0.0009, 0.1102][0.0009,\,0.1102] mistral-12b 0.02570.0257 0.03730.0373 [−0.0471, 0.0995][-0.0471,\,0.0995] mistral-3.1-24b 0.02060.0206 0.01480.0148 [−0.0088, 0.0491][-0.0088,\,0.0491] qwen-2.5-7b 0.02260.0226 0.01710.0171 [−0.0116, 0.0555][-0.0116,\,0.0555] qwen-2.5-14b −0.0039-0.0039 0.01140.0114 [−0.0266, 0.0179][-0.0266,\,0.0179] qwen-2.5-72b −0.0096-0.0096 0.01910.0191 [−0.0476, 0.0266][-0.0476,\,0.0266] Table A8: Uncertainty in behavioral resilience effects. We report the mean behavioral movement difference ΔM M, bootstrap standard error, and 95%95\% percentile bootstrap confidence interval using the sAwMIL probe and Direct Conditional graded stability. Uncertainty is estimated from 10,00010,000 sequence-stratified matched-pair bootstrap resamples. Mean ΔM M is positive in 83.3%83.3\% of model–domain settings; the 95%95\% confidence interval is entirely positive in 30.6%30.6\% of settings, entirely negative in 2.8%2.8\%, and includes zero in the remaining 66.7%66.7\%. Figure A3: Association of model scale with graded stability. Bars show mean graded stability γ¯ℳ γ_M for the 1212 instruction-tuned LLMs, ordered by nominal parameter count in (a) City Locations, (b) Medical Indications, and (c) Word Definitions, and by number of decoder layers in (d) City Locations, (e) Medical Indications, and (f) Word Definitions. Colors denote Llama (red), Gemma (blue), Mistral (yellow), and Qwen (green) model families. Error bars denote ±1± 1 bootstrap standard error. Neither parameter count nor model depth shows a consistent relationship with graded stability. Appendix H Model scaling The between-model variation in graded stability raises the possibility that γℳ(P) _M(P) increases systematically with model scale. We examine this relationship using two measures of scale: parameter count and number of layers. For each domain, we compute the mean graded stability γ¯ γ across each model’s beliefs and order the 1212 instruction-tuned LLMs by nominal parameter count (Fig. A3(a–c)) and model depth (Fig. A3(d–f)). We do not observe a consistent scaling relationship under either measure. Mean stability generally increases across some of the smaller models in City Locations and Medical Indications, but these trends are neither monotonic nor consistent across domains. Word Definitions shows little evidence of increasing stability with either parameter count or model depth. Moreover, models with the same nominal parameter count or the same number of layers can exhibit substantially different mean stability. Thus, although model scale may contribute to some of the within-family differences observed in Section 2.4, neither parameter count nor model depth provides a systematic explanation for variation in graded belief stability across models. Appendix I Robustness checks I.1 Base-model robustness To test whether our primary findings depend on instruction tuning, we repeat the main-text analyses using the pretrained base counterparts of the 1212 instruction-tuned LLMs. The models are matched by family and parameter scale, with exact specifications reported in Section C. Figure A4: Relationship between individual belief probability and graded stability in base models. We display the held-out R2R^2 for predicting graded stability from individual belief probability in (a) City Locations, (b) Medical Indications, and (c) Word Definitions for the 1212 matched pretrained base models from the Llama (red), Gemma (blue), Mistral (yellow), and Qwen (green) families. Panel (d) shows the distribution of pairwise Spearman correlations between models’ residual graded-stability scores after removing the fitted belief-probability relationship, with black lines denoting median correlations. As with the instruction-tuned models, belief probability explains only part of graded stability and the remaining proposition-level structure is positively shared across models. I.1.1 Replication of primary analyses in base models The principal representation-based findings are qualitatively similar in the base models. Individual belief probability continues to explain a meaningful but incomplete fraction of graded stability (Fig. A4), and the residual stability remaining after accounting for belief probability exhibits positive cross-model structure. Median residual correlations are ρ=0.38ρ=0.38 for City Locations, ρ=0.31ρ=0.31 for Medical Indications, and ρ=0.26ρ=0.26 for Word Definitions. The approximate-coherence result also persists in the base models (Fig. A5). Most measured probability systems remain relatively close to the CCK-coherent set, with City Locations generally requiring the smallest adjustments and Medical Indications and Word Definitions showing larger distances. Domain-level variation in graded stability is likewise broadly preserved (Fig. A6). City Locations remains concentrated near high stability for most base models, whereas Medical Indications and Word Definitions generally exhibit broader distributions and more mass at intermediate stability values. The base models typically show greater between-model heterogeneity than the instruction-tuned models, particularly for the Medical Indications and Word Definitions domains. There are, however, several notable exceptions to the overall pattern. In City Locations, for example, Llama-3.2-3B is substantially more stable than in the instruction-tuned setting, whereas Mistral-12B is markedly less stable. The behavioral replication is weaker (Fig. A7). Unlike the instruction-tuned models, for which lower-stability beliefs exhibit greater mean movement in most model–domain settings, the base-model effects are generally smaller and cluster more closely around zero. We interpret this difference cautiously because the behavioral challenge paradigm is itself an instruction-following, multi-turn interaction. The pretrained base models have not undergone the instruction-following post-training of their matched instruction-tuned counterparts, and in our implementation the conversational history is therefore supplied to base models as an explicit plain-text transcript rather than through a model-specific chat template. The weaker base-model behavioral association may therefore reflect the mismatch between the challenge task and the models’ training rather than a failure of the underlying representational stability measure. Figure A5: Distance from probabilistic coherence in base models. Violin plots show the distribution across (P,x)(P,x) pairs of the RMS adjustment dCCKd_CCK required to project the measured probability distributions for individual statements and conditionals onto the nearest CCK-coherent probability system for (a) City Locations, (b) Medical Indications, and (c) Word Definitions for Llama (red), Gemma (blue), Mistral (yellow), and Qwen (green). Horizontal black lines denote within-model medians. Base models remain comparatively close to the coherent set, although distances and between-model heterogeneity are slightly larger than in the instruction-tuned models. Figure A6: Variation in graded belief stability across domains in base models. Violin plots show the proposition-level distribution of Direct Conditional graded stability for each of the pretrained base models in (a) City Locations, (b) Medical Indications, and (c) Word Definitions for Llama (red), Gemma (blue), Mistral (yellow), and Qwen (green). Horizontal black lines denote within-model medians. City Locations is generally concentrated at higher stability values, while Medical Indications and Word Definitions show broader and more model-dependent distributions, reproducing the main domain-level pattern with greater heterogeneity. Figure A7: Behavioral resilience among probability-matched beliefs in base models. Bars show the mean difference in behavioral movement ΔM M between matched lower- and higher-stability beliefs for the pretrained base models in (a) City Locations, (b) Medical Indications, and (c) Word Definitions, with error bars denoting ±1± 1 bootstrap standard error, for Llama (red), Gemma (blue), Mistral (yellow), and Qwen (green). Positive values indicate greater movement for the lower-stability belief. In contrast to the clearer directional association in instruction-tuned models, base-model effects are smaller and more heterogeneous, suggesting that the behavioral relationship is sensitive to instruction-following post-training. I.1.2 Association with instruction tuning We next compare each base model directly with its instruction-tuned counterpart while holding belief identity fixed. For each matched model pair, we restrict the comparison to propositions believed by both models and define the average association with instruction tuning as (44) Δinst:=1|ℬbase∩ℬinst|∑P∈ℬbase∩ℬinst[γinst(P)−γbase(P)]. _inst:= 1|B_base _inst| _P _base _inst [ _inst(P)- _base(P) ]. Thus, Δinst>0 _inst>0 indicates that instruction tuning is associated with greater graded stability for the same set of shared beliefs. Across the 3636 matched model–domain comparisons, 2828 (77.8%)(77.8\%) have Δinst>0 _inst>0 (Fig. A8). The effect is therefore positive more often than not, but its magnitude varies substantially across models and domains and several comparisons are negative. We interpret instruction tuning as exhibiting a positive directional tendency rather than a uniform stabilizing effect. Figure A8: Association between instruction tuning and graded belief stability. Bars show the mean matched difference Δinst _inst in graded stability between each instruction-tuned model and its pretrained base counterpart over propositions believed by both models for (a) City Locations, (b) Medical Indications, and (c) Word Definitions for Llama (red), Gemma (blue), Mistral (yellow), and Qwen (green). Positive values indicate greater graded stability after instruction tuning, and error bars capture ±1± 1 standard error. Instruction-tuned models have higher measured stability in 77.8%77.8\% of matched comparisons, but the magnitude and direction of the effect remain heterogeneous. I.2 Conditional-estimator robustness Our primary analyses use the Direct Conditional estimator, which estimates conditional support from representations of statements of the form “Given x, P.” To test whether our results depend on this operationalization of Prℳ⁡(P∣x) _M(P x), we repeat the principal analyses using the Joint-to-Conditional estimator described in Section 4.3.2. While Direct Conditional probes the conditional statement itself, Joint-to-Conditional represents x and P jointly and derives the corresponding conditional probability from the resulting nine-state distribution. We first examine an additional representational choice introduced by the Joint estimator, the ordering of x and P in the conjunction, before replicating the main analyses and directly comparing the two graded-stability estimates. Figure A9: Sensitivity of the Joint-to-Conditional estimator to conjunction ordering. We compare Joint estimates obtained using the templates “x and P” and “P and x”. For each model, bars show the median absolute change in Pr^ℳjoint(P∣x) ^joint_M(P x) across evaluated (P,x)(P,x) combinations when the conjunction order is reversed, for (a) City Locations, (b) Medical Indications, and (c) Word Definitions for Llama (red), Gemma (blue), Mistral (yellow), and Qwen (green). We additionally display (d–f) the median absolute change in γℳjoint(P) _M^joint(P) across target beliefs after graded stability is recomputed using the reversed-order estimates. Error bars denote ±1± 1 bootstrap standard error. Reversing the conjunction order changes both the estimated conditional probabilities and the resulting graded-stability scores, revealing an order sensitivity specific to the Joint-to-Conditional operationalization. I.2.1 Sensitivity to conjunction ordering The Joint-to-Conditional estimator requires a natural-language representation of the two statements being jointly probed. In our primary implementation, we use “x and P”, placing the conditioning proposition x first and the target belief P second. The calibrated Joint probe produces the nine-state distribution described in Section 4.3.2, from which we derive Pr^ℳjoint(P∣x) ^joint_M(P x). To test whether this estimate depends on the conjunction order, we independently train and calibrate a second Joint probe using the reversed template “P and x.” Figure A9 shows that this choice is consequential. Reversing the conjunction changes the estimated conditional probabilities, with the size of the change varying substantially across models and domains (Fig. A9(a–c)). These differences also affect the downstream graded stability measure (Fig. A9(d–f)). We treat this sensitivity as an additional measurement choice introduced by the Joint-to-Conditional estimator. An autoregressive LLM produces different hidden representations when the same two propositions appear in different token orders, and its training objective does not require the probe-relevant representations of “x and P” and “P and x” to be invariant. As such, the observed differences should not necessarily be interpreted as evidence about an underlying order-sensitive joint belief distribution. Rather, they show that the Joint operationalization inherits sensitivity to the linguistic representation used to elicit that distribution. We fix the order to “x and P” throughout the analyses below, and Direct Conditional remains our primary estimator because it does not introduce this additional conjunction-ordering choice. Figure A10: Relationship between individual belief probability and Joint-to-Conditional graded stability. We display the held-out R2R^2 for predicting Joint-to-Conditional graded stability from individual belief probability in (a) City Locations, (b) Medical Indications, and (c) Word Definitions for the 1212 instruction-tuned LLMs for Llama (red), Gemma (blue), Mistral (yellow), and Qwen (green). Panel (d) shows the distribution of pairwise Spearman correlations between models’ residual graded-stability scores after removing the fitted belief-probability relationship, with black lines denoting median correlations. As with Direct Conditional, individual belief probability explains only part of graded stability, and residual stability remains positively correlated across most model pairs. I.2.2 Replication with Joint-to-Conditional Figure A11: Distance of Joint-to-Conditional probability estimates from probabilistic coherence. Violin plots show the distribution across (P,x)(P,x) combinations of the RMS adjustment dCCKd_CCK required to project measured probability distributions for individual statements and conditionals onto the nearest CCK-coherent probability system for (a) City Locations, (b) Medical Indications, and (c) Word Definitions for Llama (red), Gemma (blue), Mistral (yellow), and Qwen (green). Horizontal black lines denote within-model medians. As under Direct Conditional, the measured systems generally lie close to the CCK-coherent set, with City Locations showing the smallest distances for most models. Figure A12: Variation in Joint-to-Conditional graded belief stability across domains. Violin plots show the proposition-level distribution of Joint-to-Conditional graded stability for each of the 1212 instruction-tuned LLMs in (a) City Locations, (b) Medical Indications, and (c) Word Definitions for Llama (red), Gemma (blue), Mistral (yellow), and Qwen (green). Horizontal black lines denote within-model medians. Joint-to-Conditional stability is strongly concentrated near γ=1γ=1 across all three domains, producing weaker domain separation than under Direct Conditional. Figure A13: Behavioral resilience using Joint-to-Conditional graded stability among probability-matched beliefs. Bars show the mean difference in behavioral movement ΔM M between matched lower- and higher-stability beliefs for the 1212 instruction-tuned LLMs in (a) City Locations, (b) Medical Indications, and (c) Word Definitions, with error bars denoting ±1± 1 bootstrap standard error, for Llama (red), Gemma (blue), Mistral (yellow), and Qwen (green). Positive values indicate greater movement for the lower-stability belief. Mean ΔM M is positive in 26/3626/36 (72.2%72.2\%) model–domain settings, although the association is less consistent than under Direct Conditional. We repeat the main-text analyses using Joint-to-Conditional graded stability for the same 1212 instruction-tuned LLMs (Figs. A10–A13). The principal representation-based findings are broadly preserved, while the absolute scale of graded stability and its behavioral association show greater sensitivity to the conditional estimator. Individual belief probability remains an incomplete predictor of Joint-to-Conditional graded stability (Fig. A10). Held-out R2R^2 remains well below one across every model and domain. Cross-model residual correlations are also predominantly positive, although less uniformly than under Direct Conditional: 71.2%71.2\% are positive for City Locations, 89.4%89.4\% for Medical Indications, and 98.5%98.5\% for Word Definitions. The corresponding median residual Spearman correlations are ρ=0.05ρ=0.05, ρ=0.14ρ=0.14, and ρ=0.21ρ=0.21, respectively. Thus, the Joint estimator preserves the central result that graded stability contains systematic variation not captured by individual belief probability, while the strength of the shared residual structure is estimator-dependent. Using the same CCK projection analysis as in Section 4.5.2, but substituting the Joint-derived conditional estimates, we find that the measured probability systems again lie relatively close to the coherent set (Fig. A11). Importantly, coherence is not guaranteed by the Joint construction: the individual-statement distributions for P and x are estimated independently rather than obtained as marginals of the probed joint distribution. City Locations is typically closest to the coherent set, while Medical Indications and Word Definitions show larger adjustments. The clearest estimator-dependent effect appears in the absolute distribution of graded stability (Fig. A12). Joint-to-Conditional scores are strongly concentrated near γ=1γ=1 across all three domains. City Locations remains highly stable, but Medical Indications and Word Definitions are also shifted substantially toward the upper end of the scale, reducing the domain separation observed under Direct Conditional. This difference is consistent with the construction of the estimators: Joint-to-Conditional renormalizes over determinate conditional states, whereas Direct Conditional allows Neither probability mass to reduce the estimated support for P|xP x. The pronounced domain ordering in the primary analysis should therefore be interpreted as partly dependent on the numerical operationalization of conditional support, even though Direct and Joint stability remain strongly related within models (Section I.2.3). The behavioral association is weaker but remains more often positive than negative under Joint-to-Conditional (Fig. A13). Mean ΔM M is positive in 26/3626/36 (72.2%72.2\%) model–domain settings: 10/1210/12 for City Locations, 9/129/12 for Medical Indications, and 7/127/12 for Word Definitions. Thus, probability-matched lower-stability beliefs still tend to move more under conversational challenge, but the relationship is less consistent than under Direct Conditional. Figure A14: Agreement between Direct Conditional and Joint-to-Conditional graded stability. Bars show the Spearman correlation between γℳdirect(P) _M^direct(P) and γℳjoint(P) _M^joint(P) for each instruction-tuned LLM in (a) City Locations, (b) Medical Indications, and (c) Word Definitions for Llama (red), Gemma (blue), Mistral (yellow), and Qwen (green). Correlations are positive for every model and domain, indicating substantial agreement in the relative ranking of beliefs by stability despite differences in the estimators’ absolute scales. I.2.3 Direct–Joint agreement Finally, we ask whether the two estimators preserve the relative ordering of beliefs even where their absolute stability values differ. For each model and domain, we compute the Spearman correlation between γℳdirect(P) _M^direct(P) and γℳjoint(P) _M^joint(P) over propositions with valid stability estimates under both approaches (Fig. A14). The two operationalizations produce positively correlated stability rankings in every model–domain setting. Median Spearman correlations across models are ρ=0.78ρ=0.78 for City Locations, ρ=0.81ρ=0.81 for Medical Indications, and ρ=0.71ρ=0.71 for Word Definitions. Thus, despite their differences in representation and absolute numerical scale, Direct Conditional and Joint-to-Conditional generally agree on which beliefs are relatively more or less stable within a model. City Locations Medical Indications Word Definitions Model SVM Mass Mean SVM Mass Mean SVM Mass Mean llama-3.2-3b 11 7 12 11 11 11 llama-3.1-8b 25 17 14 13 13 14 llama-3.1-70b 76 36 65 37 29 30 gemma-7b 17 19 17 17 16 16 gemma-2-9b 22 23 20 21 20 19 gemma-2-27b 22 22 21 43 20 22 mistral-7b 14 16 14 15 16 15 mistral-12b 17 39 16 20 15 19 mistral-3.1-24b 19 21 19 17 18 20 qwen-2.5-7b 20 19 22 18 18 17 qwen-2.5-14b 27 30 31 30 29 23 qwen-2.5-72b 55 63 56 60 56 57 Table A9: Selected layers for the alternative probes. Zero-indexed layers selected independently for the SVM and Mass Mean probes for each instruction-tuned LLM. The selected probe-specific layer is subsequently used for the corresponding individual belief probability and Direct Conditional analyses. Figure A15: Relationship between individual belief probability and graded stability under alternative probes. For the SVM, we show held-out R2R^2 for predicting graded stability from individual belief probability in (a) City Locations, (b) Medical Indications, and (c) Word Definitions, respectively, for Llama (red), Gemma (blue), Mistral (yellow), and Qwen (green). Panel (d) shows pairwise cross-model Spearman correlations between residual graded-stability scores. Panels (e–h) report the corresponding Mass Mean results. Black lines in (d) and (h) denote median residual correlations. Under both probes, individual belief probability leaves substantial proposition-level variation unexplained and residual stability remains largely positively shared across models. Figure A16: Distance from probabilistic coherence under alternative probes. Violin plots show the RMS adjustment dCCKd_CCK required to project measured probability distributions for individual statements and conditionals onto the nearest CCK-coherent system. Panels (a–c) show results for the SVM for Llama (red), Gemma (blue), Mistral (yellow), and Qwen (green), and panels (d–f) for Mass Mean, with columns corresponding to City Locations, Medical Indications, and Word Definitions. Horizontal black lines denote within-model medians. The SVM broadly reproduces the approximate-coherence pattern obtained with sAwMIL, whereas Mass Mean yields larger and more heterogeneous distances from the coherent set. Figure A17: Variation in graded belief stability under alternative probes. Violin plots show proposition-level Direct Conditional graded-stability distributions. Panels (a–c) show results for the SVM for Llama (red), Gemma (blue), Mistral (yellow), and Qwen (green) and panels (d–f) for Mass Mean, with columns corresponding to City Locations, Medical Indications, and Word Definitions. Horizontal black lines denote within-model medians. The SVM preserves the primary domain-level pattern, whereas Mass Mean produces more heterogeneous distributions and substantially weaker separation across domains. Figure A18: Behavioral resilience under alternative probes. Bars show the mean behavioral movement difference ΔM M between probability-matched lower- and higher-stability beliefs. Panels (a–c) show results for the SVM for Llama (red), Gemma (blue), Mistral (yellow), and Qwen (green) and panels (d–f) for Mass Mean, with columns corresponding to City Locations, Medical Indications, and Word Definitions. Error bars denote ±1± 1 bootstrap standard error, and positive values indicate greater movement for the lower-stability belief. The directional association remains visible across many SVM model–domain combinations but is weaker and less consistent under Mass Mean. I.3 Alternative-probe robustness We repeat the principal analyses using two single-instance linear probes, an SVM [13] and Mass Mean [31], to evaluate whether the graded-stability results depend on the sAwMIL probe used in the main text. Both alternative probes operate on the hidden representation of the final non-padding token. For each probe, model, and dataset, we independently select the transformer layer that minimizes three-class calibration log loss using the procedure in Section D.3. The resulting layers are reported in Table A9 and are subsequently used for both the individual-statement and Direct Conditional analyses. For the SVM probe, we standardize each activation dimension to zero mean and unit variance using a StandardScaler fit only on the training representations. We then fit one LinearSVC per class in a one-versus-all construction, with C=1.0C=1.0, ℓ2 _2 regularization, squared-hinge loss, a maximum of 10,00010,000 iterations, convergence tolerance 10−410^-4, no class weighting, and random seed 00. For Mass Mean, we construct one class-versus-rest direction for each class j∈0,…,K−1j∈\0,…,K-1\. Let j μ_j denote the mean training representation for class j and ¬j μ_ j the mean training representation over all remaining classes. We compute the difference vector (45) Δj=j−¬j, μ_j= μ_j- μ_ j, and normalize Δj μ_j to the unit ℓ2 _2 norm before scoring. The decision boundary is placed halfway between j μ_j and ¬j μ_ j, and statements are scored by their signed projection along the normalized Δj μ_j direction relative to this midpoint. For both probes, the resulting class scores are converted to probability distributions using the multinomial logistic-regression calibration procedure described in Section D.1. The calibrator is fit only on the calibration split with C=1.0C=1.0, the L-BFGS solver, a maximum of 5,0005,000 iterations, and tolerance 10−610^-6. Direct Conditional probes are trained separately from the corresponding individual-statement probes on the same conditional training and calibration examples used for sAwMIL, while retaining the probe-specific layer selected from the sweep. Across these analyses, the SVM largely reproduces the principal representation-based results obtained with sAwMIL, whereas Mass Mean is less consistent across several downstream analyses. Both alternative probes nevertheless preserve the central result that graded stability is not fully explained by the fitted relationship with individual belief probability (Fig. A15). For the SVM, median cross-model residual correlations are ρ=0.25ρ=0.25, 0.200.20, and 0.190.19 for City Locations, Medical Indications, and Word Definitions, respectively. The corresponding Mass Mean medians are ρ=0.17ρ=0.17, 0.140.14, and 0.220.22. Thus, under both alternative probes, substantial proposition-level variation remains after accounting for individual belief probability, and models continue to show positively correlated residual stability. The probabilistic-coherence results show a clearer difference between the two alternative probes (Fig. A16). The SVM yields distance-to-coherence distributions broadly similar to those obtained with sAwMIL. By contrast, Mass Mean generally requires larger adjustments to reach the CCK-coherent set, particularly for City Locations and Word Definitions. The domain-level characterization shows the same pattern (Fig. A17). Under the SVM, City Locations remains concentrated near high graded stability, while Medical Indications and Word Definitions exhibit broader distributions extending further into intermediate and low stability values. This separation is substantially weaker under Mass Mean, with several City Locations models in particular receiving much lower graded-stability estimates than under either sAwMIL or the SVM. Finally, the behavioral association is more robust under the SVM than under Mass Mean (Fig. A18). SVM-based graded stability remains predominantly associated with greater behavioral resilience among probability-matched beliefs, although the effects are more heterogeneous than in the primary sAwMIL analysis. Under Mass Mean, the estimated effects are generally smaller and less consistent in direction. The weaker Mass Mean results are consistent with known sensitivities of centroid-based truth directions. Mass Mean was introduced as a binary method that estimates a truth direction from the difference between the centroids of True and False representations [31]. Previous work applying this probe in a three-class setting found that it can become unstable when Neither examples are incorporated, since changes in the composition of the representation sets directly alter the estimated class centroids [15]. Our trivalent implementation similarly constructs three class-versus-rest centroid directions for True, False, and Neither and calibrates the resulting scores into a shared multiclass probability distribution. This sensitivity is especially important for graded stability because our downstream quantities depend on the absolute allocation of probability mass across all three classes, not merely on whether a binary truth direction ranks True statements above False statements. Mass Mean can therefore remain informative as a veracity probe while being less well suited to the trivalent probability estimation required here. We consequently view the strong SVM replication as evidence that the principal results are not specific to the multi-instance structure of sAwMIL, while the Mass Mean results identify a meaningful sensitivity to probes that are not naturally designed for multiclass probability estimation.

Instant research discovery