Paper deep dive
Response Variability and Stability in Human Reasoning
Clemens Bombach, Rajmadan Lakshmanan, Marco Ragni
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 94%
Last extracted: 10/6/2026, 10:20:24 AM
Summary
This paper introduces a geometry-based framework to quantify variability and stability in human syllogistic reasoning. By encoding reasoning tasks and responses into binary feature vectors based on heuristic theories (Atmosphere, PHM, TransSet), the authors define metrics to measure distances between reasoning patterns. Using experimental data from 100 participants tested twice, they demonstrate that response variability (measured via local energy) interacts with correctness to predict performance gains (learning effects) and that individual reasoning patterns are stable over time.
Entities (12)
Relation Signals (10)
Rajmadan Lakshmanan → affiliatedwith → TU Chemnitz
confidence 99% · Affiliation: Predictive Analytics, TU Chemnitz
Marco Ragni → affiliatedwith → TU Chemnitz
confidence 99% · Affiliation: Predictive Analytics, TU Chemnitz
Clemens Bombach → affiliatedwith → TU Chemnitz
confidence 99% · Affiliation: Predictive Analytics, TU Chemnitz
TransSet → proposedby → Brand et al.
confidence 95% · TransSet (Brand et al., 2019)
Atmosphere Theory → proposedby → Woodworth and Sells
confidence 95% · atmosphere theory (Woodworth and Sells, 1935)
Probability Heuristics Model → proposedby → Chater and Oaksford
confidence 95% · Probability Heuristics Model (PHM) (Chater and Oaksford, 1999)
Dames et al. (2022) → usedin → Response Variability and Stability in Human Reasoning
confidence 95% · re-analyzing a dataset collected in a study on the stability of syllogistic reasoning over time (Dames et al., 2022).
reasoning patterns → exhibit → Stability Over Time
confidence 90% · reasoning patterns are stable over time within the same reasoner.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Understanding how humans reason -- and how reasoning responses vary across tasks and individuals -- remains a core challenge for modeling and explanation in cognitive science. We investigate the stability of response patterns within reasoners and whether variation in these patterns can be used to predict learning effects. We introduce a formal, geometry-based method to quantify distances between individual reasoning patterns and their internal variability, grounded in heuristic theories. The proposed framework is tested against experimental data via generalized linear mixed-effects models and clustering, where we find that our proposed variation measure interacts with correctness to predict performance gains. Moreover, we find that reasoning patterns are stable over time within the same reasoner. The method is general enough to be applied to other reasoning domains.
Tags
Links
- Source: https://arxiv.org/abs/2610.03008v1
- Canonical: https://arxiv.org/abs/2610.03008v1
Trouble viewing inline? Open PDF directly →
Full Text
39,811 characters extracted from source content.
Expand or collapse full text
Response Variability and Stability in Human Reasoning Clemens Bombach (clemens.bombach@hsw.tu-chemnitz.de) Rajmadan Lakshmanan Marco Ragni Affiliation: Predictive Analytics, TU Chemnitz Abstract Understanding how humans reason – and how reasoning responses vary across tasks and individuals – remains a core challenge for modeling and explanation in cognitive science. We investigate the stability of response patterns within reasoners and whether variation in these patterns can be used to predict learning effects. We introduce a formal, geometry-based method to quantify distances between individual reasoning patterns and their internal variability, grounded in heuristic theories. The proposed framework is tested against experimental data via generalized linear mixed-effects models and clustering, where we find that our proposed variation measure interacts with correctness to predict performance gains. Moreover, we find that reasoning patterns are stable over time within the same reasoner. The method is general enough to be applied to other reasoning domains. Keywords: Reasoning; Similarity Analysis; Cognitive Modeling; Syllogisms; GLMM; Cluster Analysis Introduction Many (classical) cognitive models implicitly assume an almost deterministic mapping from stimulus to response: holding the task and context constant, repeated presentations of the same input are expected to yield the same or at least similar response. This assumption simplifies model identification and parameter estimation, because variability can be treated as negligible. However, substantial trial-to-trial and inter-individual variability has been shown in reasoning and decision making, arising from factors such as lack of focus, attention, and context or sequence effects, often conceptualized as unwanted noise in decision making (e.g., Kahneman et al., 2021). Consider the domain of human reasoning, more precisely the domain of syllogistic reasoning. A syllogism consists of two quantified premises over three terms, for example All Architects are Bakers, All Chemists are Bakers, and a conclusion such as “All Architects are Chemists” or “Some Chemists are not Architects”. Normative and observed human responses in syllogistic reasoning differ substantially. For example, in a meta-analysis using data from 6 studies of all 6464 categorical syllogisms, it was reported that 47%47\% of all participants responded “All Architects are Chemists” (Khemlani and Johnson-Laird, 2012). However, the logically correct response is that no valid conclusion can be derived from the premises, chosen by only 31%31\% of participants. None of the theories evaluated in the meta-analysis of Khemlani and Johnson-Laird (2012) adequately explain all gaps between normative and empirically observed responses, so the development of a general psychological theory remains an open problem. Interestingly, most of the 9 theories reported assume a static cognitive process, i.e., for the same input of the task they predict in most cases the same response. Previous studies indicate a shift of the responses towards more logically correct responses if human reasoners are tested a week apart (Ragni et al., 2018; Dames et al., 2022). Responses in reasoning tasks do not vary at random, but instead reveal systematic patterns. Individual patterns are consistent across tasks (Stanovich and West, 1998). Consistency judgments have been shown to be systematically influenced by underlying representations and processing constraints (Ragni et al., 2014). However, with the exception of one theory (mReasoner), that change has not been explained by other theories and it has rarely been investigated on the individual level. A related open question is to what degree small variations in the task change the response. For instance, the syllogism above is obtained from the syllogism All Architects are Bakers, All Bakers are Chemists, by swapping the terms in the second premise. Superficially, this is a small change, but it changes the normative response from “All Architects are Chemists” to “No valid conclusion”. However, the results of Khemlani and Johnson-Laird (2012) indicates that most reasoners are insensitive to this change. A major barrier in this regard is that there is no generally accepted formalism for determining whether a response pattern is more or less variable, or for defining the distance between two response patterns. Hence, this paper is concerned with developing such a formally sound framework. The principal idea is to identify individual reasoning patterns with functions acting on spaces of binary vectors, thereby transforming questions about reasoning patterns into questions about functions on discrete spaces. Such functions are well studied in the mathematical literature, allowing us to draw on suitable notions of distance and variability. This method of identification also incorporates findings from heuristic theories of syllogistic reasoning — theories that predict responses from surface features of the premises alone, without evaluating logical validity — including the atmosphere theory (Woodworth and Sells, 1935), Probability Heuristics Model (PHM) (Chater and Oaksford, 1999), and TransSet (Brand et al., 2019). More precisely, the atmosphere theory predicts conclusions from the universal/particular and positive/negative character of the premises. The PHM ranks conclusions by informativeness derived from a probabilistic reading of the quantifiers, predicting that reasoner avoid uninformative conclusions (max-heuristic). TransSet models the reasoner’s attempt to construct a transitive inference path across the figure of the syllogism. These three theories together motivate our encoding of both tasks and responses into binary feature vectors, as detailed in the upcoming discussion. We show the value of this framework by re-analyzing a dataset collected in a study on the stability of syllogistic reasoning over time (Dames et al., 2022). "Participants solved all 64 syllogisms twice, one week apart. Performance often improved in the retest, however the degree of improvement was highly dependent on the particular task and reasoner. Based on this result and the discussion above, we formulate the following research questions: • Q0 What is a useful distance measure on syllogistic reasoning patterns? How can we measure response variability? • Q1 To what extent does response variability explain individual differences in performance improvement? In other words, are highly variable response patterns a predictor of retest performance? • Q2 Related to the result of Stanovich and West (1998): Are reasoning patterns stable within individuals? That is, is the distance between a reasoner’s initial response pattern and their retest pattern smaller than the distance between their initial pattern and another reasoner’s retest pattern? Materials and Methods We begin by briefly describing the dataset used for our analysis (Dames et al., 2022). This dataset was chosen because of its large sample size and because it includes repeated measures for all 6464 syllogisms. Session 0 assessed personality traits and general cognitive abilities. In Session 1, participants selected one response (out of the 99 canonical responses) for each of the 6464 syllogistic problems; no feedback was provided, and task order was randomized. This procedure was repeated in Session 2, which took place one week after Session 1. The final sample consists of N=100N=100 individuals. For the data analysis, we employ generalized linear mixed models (GLMMs) which were fit in R using the lme4 package (Bates et al., 2015). GLMMs a are well-known tool for hierarchical modeling. For model diagnostics, we employed the DHARMa package (Hartig, 2024). For the calculation of p-values in Q2, we employed the lmerTest package (Kuznetsova et al., 2017). All models were fit using the BOBYQA optimizer. Goodness of fit was assessed with the performance package using pseudo-R2R^2 (Lüdecke et al., 2021; Nakagawa and Schielzeth, 2013). The data and the scripts used to run the analyses are openly available on GitHub.11 1 https://github.com/c-bombach/cogsci-response-variability The new ingredient of our approach is the systematic use of response variability and distance features, which allow a finer characterization of individual responses than correctness alone and are defined in the next section. Q0: Framework for Feature Sensitivity Feature sensitivity is operationalized here as local response variability with respect to a psychologically motivated task distance. A syllogism, as above, consists of two quantified premises about three terms a, b, c. Quantifiers are A=All A=All, E=No E=No, I=Some I=Some and O=Some …not O=Some … not. There are 4 possible orderings of the terms and these are called figures: Figure 1 Figure 2 Figure 3 Figure 4 a-b b-a a-b b-a b-c c-b c-b b-c A syllogistic problem is uniquely determined by the quantifiers of the two premises and the figure (e.g., the syllogism from the introduction is A3 A A3), which we refer to as its surface features. Analogously, responses can be encoded in terms of surface features, with NVC (No Valid Conclusion) as the logically correct response in 3737 of the 6464 syllogisms. The remaining 2727 syllogisms each license a valid conclusion, which must be one of 88 possible response types consisting of a quantifier from A,E,I,O\ A, E, I, O\ combined with a direction ac,ca\ ac, ca\ (e.g., Eac for “No a are c”). In the following, we denote by X=(q1,q2,r):q1,q2∈A,E,I,O,r∈1,2,3,4X=\(q_1,q_2,r):q_1,q_2∈\ A, E, I, O\,r∈\1,2,3,4\\ the set of syllogistic tasks. The set of responses is defined as Y=(q,d):q∈A,E,I,O,d∈ac,ca∪NVCY=\(q,d):q∈\ A, E, I, O\,d∈\ ac, ca\\∪\ NVC\. As outlined in the introduction, we identify reasoning patterns with functions f:X→Yf:X→ Y that take a task x∈Xx∈ X as an input and produce an output f(x)∈Yf(x)∈ Y. To obtain notions of distance and variability for maps from X to Y, we begin by defining distance functions (also called metrics) between elements of the underlying spaces X and Y respectively. Formally, a metric on a set Z is a function dZ:Z×Z→[0,∞)d_Z:Z× Z→[0,∞) such that dZf(z,w)=0d_Zf(z,w)=0 if and only if z=wz=w, and for all z,w,u∈Zz,w,u∈ Z it holds that dZ(z,w)=dZ(w,z)d_Z(z,w)=d_Z(w,z) and dZ(z,u)≤dZ(z,w)+dZ(w,u)d_Z(z,u)≤ d_Z(z,w)+d_Z(w,u). The pair (Z,dZ)(Z,d_Z) is called a metric space. Metrics on task and response sets A simple metric can be obtained by one-hot encoding. This means that each syllogism is encoded as a unit vector in ℝ64R^64. Then, we could take the Euclidean distance on ℝ64R^64. This, however, would make all syllogisms equidistant from each other. Moreover, in applications such as clustering, such an embedding suffers from the curse of dimensionality. By contrast, our goal is to obtain a metric to express response similarities and which lives on a low-dimensional space. If we wish to stick with a binary encoding, we need at least 66 bits since 26=642^6=64. To motivate our definition of a metric on X, we recall that the syllogistic quantifiers can be divided into universal quantifiers A,E A, E versus particular quantifiers I,O I, O and positive quantifiers A,I A, I versus negative quantifiers E,O E, O. This classification is the basis of the atmosphere theory of syllogistic reasoning (Woodworth and Sells, 1935), which postulates that reasoners select conclusions according to the atmosphere created by the premises. Specifically, the theory predicts that the universality and positivity of the premises jointly determine the predicted conclusion: a single particular premise (I or O) is sufficient to bias reasoners toward a particular conclusion, and a single negative premise (E or O) toward a negative one. Because conclusion selection is thus driven by these two binary features of each premise quantifier, encoding each quantifier along precisely the universality and positivity dimensions embeds the theoretically relevant structure directly into our metric. We identify each quantifier q with a tuple (uq,pq)(u_q,p_q) of bits uq,pqu_q,p_q that encode the universality and the positivity of q, respectively. More precisely, we have uq=1u_q=1 if q is universal and 00 if it is particular. Analogously, we set pq=1p_q=1 if q is positive and 00 otherwise. In a similar fashion, we encode the figure r using two bits tr,srt_r,s_r. In particular, we set tr=1t_r=1 when r∈1,2r∈\1,2\. The definition of trt_r is motivated by the following considerations: In figure 11 syllogisms, the ordering of terms allow for a simple transitive inference from a to c (even when this inference may not be logically warranted). A figure 22 syllogism can be reduced to a figure 11 syllogism by swapping the ordering of the premises. For the figures 3,43,4, it is not immediately possible to obtain such a transitive path. We therefore see figure 11 and 22 syllogisms as transitive. This distinction between transitive and intransitive figures plays an important role in TransSet (Brand et al., 2019; Brand et al., 2020), which is an algorithmic model of syllogistic reasoning. It is based on the empirically supported observation that reasoners attempt to create a transitive path between inferences. If such a path is not immediately available, they resort to a (possibly illicit) reordering of the terms. If we wish to obtain a 66-dimensional embedding, we are now left with one free bit srs_r, that we set equal to 11 if a appears in the first premise. Thus, we set sr=1s_r=1 if r∈1,3r∈\1,3\. We can therefore encode a syllogism x=(q1,q2,r)x=(q_1,q_2,r) as the 66-dimensional binary vector j(x)=(uq1,pq1,uq2,pq2,tr,sr)∈0,16.j(x)=(u_q_1,p_q_1,u_q_2,p_q_2,t_r,s_r)∈\0,1\^6\,. We obtain a one-to-one correspondence j:X→H6j:X→ H_6 between X and the 6−6-dimensional Hamming cube H6=0,16H_6=\0,1\^6. On H6H_6, we have a natural metric given by the Hamming distance dH6(v,w)=∑m=16|vm−wm|.d_H_6(v,w)= _m=1^6|v_m-w_m|\,. This counts the number of bits on which v and w disagree. We pull back this distance to a metric on X using j by setting dX(x,y)=dH6(j(x),j(y)).d_X(x,y)=d_H_6(j(x),j(y)). To obtain a metric on Y, we proceed in a similar fashion: Given y∈Yy∈ Y we set k(y)=(0,0,0,0)k(y)=(0,0,0,0) if y=NVCy= NVC. Otherwise, if y=(q,d)y=(q,d) where q∈A,E,I,Oq∈\ A, E, I, O\ and d∈ac,cad∈\ ac, ca\, we set k(y)=(uq,pq,1,0)k(y)=(u_q,p_q,1,0) if d=acd= ac and k(y)=(uq,pq,0,1)k(y)=(u_q,p_q,0,1) if d=cad= ca. Finally, we set dY(y,y~)=∑j=14|k(y)j−k(y~)j|.d_Y(y, y)= _j=1^4|k(y)_j-k( y)_j|\,. Geometrically, we can think of k as mapping Y injectively into a proper subset of the four-dimensional Hamming cube H4H_4. Note that the NVC-response is mapped to the origin. Our choice of k implies that O-responses lie closest to NVC. This choice is inspired by the O-heuristic of the PHM theory of syllogistic reasoning (Chater and Oaksford, 1999). PHM postulates that reasoners use heuristics that are based on the informativeness of the premises. This informativeness is itself derived from a probabilistic interpretation of the quantifiers: A is maximally informative as it rules out the most possibilities, while O is least informative. The O-heuristic states that reasoners generally avoid drawing O-conclusions due to their low informativity, suggesting a psychological similarity between NVC and O-conclusions. By contrast, the most informative conclusions Aac Aac and Aca Aca have high distance to NVC in our embedding. Distance and variability of response patterns Having defined metrics dX,dYd_X,d_Y on X and Y, we may now view reasoning patterns as maps f:(X,dX)→(Y,dY)f:(X,d_X)→(Y,d_Y) between metric spaces. On the set of these maps we consider, for p≥1p≥ 1, the ℓp(X,Y) ^p(X;Y)-metrics, dℓp(X,Y)(f,g)=(∑x∈XdY(f(x),g(x))p)1/p.d_ ^p(X;Y)(f,g)= ( _x∈ Xd_Y(f(x),g(x))^p )^1/p\,. This is well-defined since X is finite and generalizes the familiar Euclidean distance for p=2p=2. Since the Hamming cubes carry a natural graph structure, we may also see reasoning patterns as maps between graphs. This means we view X,YX,Y as graphs where the vertices x,yx,y are elements of X,YX,Y and two vertices are connected with an edge (written as x∼yx y) if and only if the distance between them is 11. Thus, the degree of a vertex in X is given by deg(x)=|y:y∼x|=|y:d(y,x)=1| (x)=|\y:y x\|=|\y:d(y,x)=1\|. We use this graph structure to define the desired similarity measure f:X→Yf:X→ Y. It is a generalization of the graph energy for real-valued functions on graphs (see, e.g., Keller et al., 2021) to the setting where the target space Y is a metric space: Definition 1 Let p≥1p≥ 1, (X,dX),(Y,dY)(X,d_X),(Y,d_Y) be metric spaces such that X is the vertex set of a finite, connected graph. We define the local p-energy of f:X→Yf:X→ Y at x∈Xx∈ X as ℰp(f,x)=1deg(x)∑y∼xdY(f(x),f(y))p.E_p(f;x)= 1 (x) _y xd_Y(f(x),f(y))^p\,. The total energy of f is defined as ℰp(f)=∑x∈Xℰp(f,x).E_p(f)= _x∈ XE_p(f;x)\,. The respective normalizations are defined as Ep(f,x)=ℰp(f,x)1/p,Ep(f)=ℰp(f)1/p.E_p(f;x)=E_p(f;x)^1/p\,, E_p(f)=E_p(f)^1/p\,. Note that the local energy of f at x is 00 if f is constant on a neighborhood of x. Thus, the energy measures the degree to which f is not (locally) constant and can therefore be seen as a measure of variability. Our definition is itself a special case of the notion of p-energy of maps between metric measure spaces as defined in the work by Jost (1994). Such energies have been used extensively in machine learning for the purpose of dimensionality reduction, for example in the Laplacian eigenmaps algorithm (Belkin and Niyogi, 2003). Meta-data inference In preparation for the predictive modeling, we perform a meta-data inference using k-means clustering. The dataset consists of repeated task–response trials collected from a fixed set of participants. Let I=1,…,MI=\1,…,M\ denote the set of participants, where each participant i∈Ii∈ I completes multiple trials indexed by t=1,…,Tt=1,…,T. Each trial involves a syllogistic task xi,t∈Xx_i,t∈ X, a response fi(xi,t)∈Yf_i(x_i,t)∈ Y, and an associated local energy value derived from the response pattern. The present analysis focuses exclusively on participant-level information. This aggregation allows us to infer general response tendencies and stability in reasoning behavior. For each trial t of participant i, we define a feature vector i,t=(i,t,E2(fi,xi,t))v_i,t=(r_i,t,E_2(f_i;x_i,t)) where i,t∈0,14r_i,t∈\0,1\^4 denotes the binary response-feature encoding. E2(fi,xi,t)E_2(f_i;x_i,t) denotes the normalized local 22-energy introduced in Definition 1, evaluated at task xi,tx_i,t for participant i. To obtain a participant-level representation, we compute the empirical mean ¯i=1T∑t=1Ti,t v_i= 1T _t=1^Tv_i,t. This vector summarizes the average response tendencies and energetic profile of participant i. Before clustering, we standardize each feature dimension across participants to zero mean and unit variance. This step ensures that task features, response features, and energy contribute comparably to the distance computations. We perform participant-level clustering using the k-means algorithm. Let ~i v_i denote the standardized participant-level feature vector. For a fixed number of clusters k, the algorithm identifies cluster centroids 1,…,k∈ℝn μ_1,…, μ_k ^n by minimizing the objective function min∑i=1Nci,j‖~i−ci‖2, _\c_i\,\,\ μ_j\ _i=1^N \| v_i- μ_c_i \|^2, where ci∈1,…,kc_i∈\1,…,k\ denotes the cluster assignment of participant i. Each centroid j μ_j represents a prototypical participant profile. This profile captures a characteristic combination of task-feature exposure, response-feature usage, and energy measure. To find the optimal number of clusters, we employ the elbow criterion and silhouette score. The elbow criterion measures how much the inertia, which measures within-cluster variability, decreases as the number of clusters increases. On the other hand, the silhouette score quantifies how well-separated and coherent the resulting clusters are. From Fig. 1, both criteria indicate a clear balance at k=3k=3, which we therefore select as the optimal number of participant clusters. Figure 1: Participant-level parameter (k) selection using the elbow and silhouette criteria. The shaded region marks the selected number of clusters k. Figure 2: Principal component analysis of participant-level feature vectors. Points represent participants projected onto the first two principal components; colors and marker shapes indicate k-means cluster membership. Axis labels show the variance explained by each component. Participants in Cluster 1 were found to have substantially higher mean correctness (67.2%67.2\%) than participants in Clusters 2 (37.9%37.9\%) and 3 (30.3%30.3\%). By contrast mean energy was highest for Cluster 2 (0.4710.471), followed by Cluster 3 (0.3890.389) and Cluster 1 (0.3070.307). We apply Principal Component Analysis (PCA) after clustering to obtain a low-dimensional projection of participant-level feature vectors. Each participant is represented by averaged task features, response features, and local energy, so principal components reflect joint variation in strategy choice and response smoothness. Fig. 2 shows the participant-level clustering projected onto the first two principal components. The PCA projection reveals clear structure across the three participant clusters. Principal components are influenced by both response tendency and energy, with the PC1 showing a stronger positive influence of energy, and PC2 being more influenced by response tendency, in particular with respect to direction (ac vs. ca). Participants in Cluster 1 occupy low values along the first principal component and form a relatively compact group along the second, indicating a coherent, correct response strategy with low flexibility. Participants in Cluster 2 show high values along the first principal component and a broader spread along the second. Participants in Cluster 3 align with Cluster 2 along the first principal component but separate strongly along the second, occupying markedly lower values. This indicates differences in the preferred response direction between the two clusters. Mixed-effects modeling Q1: Prediction of retest performance For this research question, we studied the effect of the energy in the initial test on correctness in the retest. This can be reformulated as follows: When reasoners give a response that is exceptional in the sense that it has a large distance from responses to neighboring syllogisms (in the sense of the metric defined above), does this indicate a performance improvement when they are tested again? We fit a sequence of GLMMs with logit link function with the aim of predicting retest correctness CretestC_retest from the following variables: • initial correctness C, encoded such that C=1C=1 for correct responses and C=0C=0 for incorrect responses • task validity V, encoded such that V=1V=1 for valid tasks and V=−1V=-1 for invalid tasks • participant-centered and scaled, normalized local 22-energy, E2E_2 • (standardized) correctness per task averaged over all participants C¯ C In particular, we were interested in the effect of the energy variable and its interactions. Continuous variables were standardized to aid convergence and to make results easily interpretable. We decided to use the normalized 22-energy since it corresponds to the Dirichlet energy. The normalization reduces skewness and ensures that it is in the same “length unit” as the underlying spaces. We opted for participant-centering since we are interested in predicting the effect of local deviations in energy on learning. Mean correctness C¯ C was included as a predictor since models that didn’t include it showed substantial underdispersion during model diagnostics. We presume that this is caused by the large variation in task success rate since some syllogisms (such as A1 A A1) are solved correctly by almost all participants. To find a suitable model, our aim was to find a convergent model with a maximally complex random effects structure and minimal AIC. The best model we were able to fit under these criteria is given by the following formula in Wilkinson-Rogers notation (Wilkinson and Rogers, 1973): Cretest∼ C_retest C∗V∗E2−C:V:E2+C¯ C*V*E_2-C:V:E_2+ C +(C+V|participant)+(C+E2|task) +(C+V|\,participant)+(C+E_2|\,task) (M1M_1) In order to have a baseline for assessing the effect of energy, we fitted the reduced model Cretest∼C∗V+C¯+(C+V|participant)+(C|task).C_retest C*V+ C+(C+V|\,participant)+(C|\,task)\,. (M0M_0) A model comparison showed a significant improvement in fit of ( M 1 ) over ( M 0 ) (ΔAIC=39,G2(6)=50.82,p=3.2⋅10−9 =39,G^2(6)=50.82,p=3.2· 10^-9). This strongly indicates that energy has substantial predictive value for the problem under consideration. Overall, model fit was acceptable with Rconditial2=0.644,Rmarginal2=0.382R^2_conditial=0.644,R^2_marginal=0.382. The results for the fixed effects are listed in Table 1. As expected, correctness and mean correctness have the largest effects on retest correctness with coefficients βC=1.560 _C=1.560 and βC¯=1.307 _ C=1.307 respectively. The effects of validity and energy are smaller (βV=0.242,βE2=0.225 _V=0.242, _E_2=0.225) but the low standard error for energy in particular (σE2=0.076)( _E_2=0.076) suggests a robust finding. That improvement in correctness is higher for valid syllogisms is consistent with the findings of Dames et al. (2022). We note the significant correctness-energy interaction βC:E2=−0.438 _C:E_2=-0.438. Table 1: Fixed effects for models M1M_1 and M0M_0. N=6400N=6400 observations; 100 participants; 64 tasks. M1M_1 M0M_0 Intercept −0.467∗∗(0.152)-0.467^**\,(0.152) −0.360∗(0.141)-0.360^*\,(0.141) Correctness C 1.560∗(0.139)1.560^***\,(0.139) 1.446∗(0.133)1.446^***\,(0.133) Validity V 0.242∗(0.121)0.242^*\,(0.121) 0.227∗(0.109)0.227^*\,(0.109) Energy E2E_2 0.225∗∗(0.076)0.225^**\,(0.076) Av. Corr. C¯ C 1.307∗(0.074)1.307^***\,(0.074) 1.281∗(0.073)1.281^***\,(0.073) C:VC:V 0.037(0.121)0.037\,(0.121) −0.018(0.111)-0.018\,(0.111) C:E2C:E_2 −0.438∗(0.100)-0.438^***\,(0.100) V:E2V:E_2 −0.122(0.064)-0.122\,(0.064) AIC 5442.2225442.222 5481.0515481.051 BIC 5577.5035577.503 5575.7485575.748 Log Likelihood −2701.111-2701.111 −2726.526-2726.526 ∗p<0.001^***p<0.001; p∗∗<0.01^**p<0.01; ∗p<0.05^*p<0.05 To get a better sense of the effect of energy on performance improvement, it is interesting to compute effective odds ratios dependent on the value of the correctness and validity variables for energy in ( M 1 ): OR(E2)OR(E_2) valid invalid correct 0.72 0.91 incorrect 1.11 1.41 In particular, we have OR<1OR<1 for correct and OR>1OR>1 for incorrect responses. One possible interpretation for these effects is as follows: When a correct response has high energy, it is different from responses to related syllogisms and thus it is less likely that it can be associated with a consistent strategy of the participant. On the other hand, low energy can be taken as a sign of a systematic response pattern. Thus, syllogisms with correct but unsystematic responses are less likely to be answered correctly on retest. By contrast, for incorrect responses, high energy raises the probability of improvement. In particular, this effect is stronger for incorrect responses to invalid syllogisms. As a sanity check, we replaced the theory-driven encoding based on task features by a simple one-hot encoding. As expected, this led to a substantial reduction in fit over the best model with theory-driven encoding (OPENΔAIC=−25) =-25) and made the main effect of energy non-significant (β=0.070β=0.070). However, we note the observation of a significant validity-energy interaction (β=−0.25,p<0.0001β=-0.25,p<0.0001). A possible explanation is that a preference towards NVC-responses leads to a large subset on which the reasoning pattern is constant. This lowers the energy in an encoding-independent fashion. Such an effect can therefore be detected by a coarse encoding. Q2: Stability of reasoning patterns Formally, the question is: Suppose that i,ji,j are reasoners, f0i,f0jf^i_0,f^j_0 their response patterns on the initial test and f1i,f1jf^i_1,f^j_1 their response patterns on the retest. Moreover, let us write dij=dℓ2(X,Y)(f0i,f1j)d_ij=d_ ^2(X;Y)(f^i_0,f^j_1) in the following. If response patterns are stable within reasoners, then we should have (dij|i=j)<(dij|i≠j)E(d_ij|i=j)<E(d_ij|i≠ j) (1) with E denoting the (conditional) expectation. In other words, the expected distance of a reasoner between a test and retest pattern is smaller when both patterns come from the same reasoner. A plot of the empirical conditional distribution functions suggests that this is indeed the case. In Figure 3, the blue curve (same participant) is shifted to the left of the orange curve (different participants), indicating that within-reasoner distances are stochastically smaller than between-reasoner distances. Formally, our sample fulfills the stronger condition ℙ(dij<t|i=j)>ℙ(dij<t|i≠j)P(d_ij<t|i=j)>P(d_ij<t|i≠ j) for all t>0t>0. Figure 3: Empirical cumulative distribution functions (cdfs) for test-retest distance. In order to estimate the effect size, we fit a LMM of the form dij∼i=j+(1|i)+(1|j),d_ij 1_i=j+(1|i)+(1|j)\,, where i=j1_i=j denotes a binary variable that is 11 if and only if i=ji=j and 00 otherwise. The terms (1|i),(1|j)(1|i),(1|j) are random intercepts for the participants i,ji,j. We obtained a (fixed) intercept of α=11.48α=11.48 and a fixed effect of β=−2.44β=-2.44 distance units with a standard deviation σ=0.13σ=0.13, resulting in the 95% confidence interval [−2.69,−2.19][-2.69,-2.19], indicating a very significant and strong effect. We note that Rconditional2=0.709R^2_conditional=0.709 whereas Rmarginal2=0.011R^2_marginal=0.011, suggesting that while the effect itself is strong, it only explains a small part of the total variation in distance. This is to be expected since reasoning patterns differ greatly between individuals. Discussion We developed a method to define psychological distances between individual reasoning patterns, incorporating findings from heuristic theories of reasoning. We employed a notion from metric geometry, the so-called p-energy (Definition 1), as a measure of response variability. The framework supports fine-grained analyses of response variability at the level of individual reasoner–task combinations. The clustering analysis suggests that reasoners differ systematically both in response strategies and variability. Our findings from the mixed-effects modeling are twofold: First, we showed that high energy predicts an increase in correctness on retest, but only for initially incorrect responses. For initially correct responses, higher energy predicts a lower probability of a correct retest response. Energy can thus be seen as a measure of psychological consistency in response patterns: If reasoners are stably incorrect, an improvement is less likely than when they are unstably incorrect. Second, we demonstrated that individual differences in reasoning patterns are stable: reasoners were closer to their own retest patterns than to other reasoner’s retest patterns. This effect seems to be particularly robust. We note that our framework abstracts away from content effects such as belief bias (Khemlani and Johnson-Laird, 2012; Chater and Oaksford, 1999). However, the task formulations in the dataset under consideration here are specifically designed to minimize content effects. Thus, this limits the impact on the present results. More broadly, extra-logical factors such as content effects and belief bias represent additional sources of response variability that the present framework does not capture; extending the task encoding to incorporate them would be a natural refinement. Regarding generalizability, some aspects of our construction depend on the domain of syllogistic reasoning and the psychological theories adopted; alternative distance functions could be derived from mental model or logic-based approaches. However, the mathematical framework itself is domain- and theory-independent: defining distances on task and response spaces directly induces a distance and energy functional on response patterns. Recent Bayesian approaches to syllogistic reasoning predict probability distributions over responses (Tessler et al., 2022). Extending our framework from deterministic response functions to non-deterministic response distributions would therefore be a natural next step and would accommodate experimental designs allowing multiple responses, which are natural for valid syllogisms licensing several conclusions. Acknowledgements The authors thank the German Research Foundation (DFG) for financial support via grants awarded to MR within the projects 283135041 and 427257555. CB was funded by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) -– Project number 283135041. RL was funded by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) -– Project numbers 283135041; 427257555. References Bates et al. (2015) D. Bates, M. Mächler, B. Bolker, and S. Walker Fitting linear mixed-effects models using lme4. Journal of statistical software 67, p. 1–48. External Links: Document Cited by: Materials and Methods. Belkin and Niyogi (2003) M. Belkin and P. Niyogi Laplacian eigenmaps for dimensionality reduction and data representation. Neural Computation 15 (6), p. 1373–1396. External Links: Document Cited by: Distance and variability of response patterns. Brand et al. (2019) D. Brand, N. Riesterer, and M. Ragni On the matter of aggregate models for syllogistic reasoning: a transitive set-based account for predicting the population. In Proceedings of the 17th International Conference on Cognitive Modeling, p. 5–10. Cited by: Introduction, Metrics on task and response sets. Brand et al. (2020) D. Brand, N. Riesterer, and M. Ragni Extending TransSet: An individualized model for human syllogistic reasoning. In Proceedings of the 18th International Conference on Cognitive Modeling, p. 17–22. Cited by: Metrics on task and response sets. Chater and Oaksford (1999) N. Chater and M. Oaksford The probability heuristics model of syllogistic reasoning. Cognitive Psychology 38 (2), p. 191–258. External Links: ISSN 0010-0285, Document, Link Cited by: Introduction, Metrics on task and response sets, Discussion. Dames et al. (2022) H. Dames, K. C. Klauer, and M. Ragni The stability of syllogistic reasoning performance over time. Thinking & Reasoning 28 (4), p. 529–568. External Links: Document Cited by: Introduction, Introduction, Materials and Methods, Q1: Prediction of retest performance. Hartig (2024) F. Hartig DHARMa: residual diagnostics for hierarchical (multi-level / mixed) regression models. Note: R package version 0.4.7 External Links: Link Cited by: Materials and Methods. Jost (1994) J. Jost Equilibrium maps between metric spaces. Calculus of Variations and Partial Differential Equations 2 (2), p. 173–204. External Links: Document Cited by: Distance and variability of response patterns. Kahneman et al. (2021) D. Kahneman, O. Sibony, and C. R. Sunstein Noise: a flaw in human judgment. Little, Brown Spark, New York. External Links: ISBN 978-0316451390 Cited by: Introduction. Keller et al. (2021) M. Keller, D. Lenz, and R. K. Wojciechowski Graphs and discrete Dirichlet spaces. Vol. 358, Springer. External Links: Document Cited by: Distance and variability of response patterns. Khemlani and Johnson-Laird (2012) S. S. Khemlani and P. N. Johnson-Laird Theories of the syllogism: A meta-analysis. Psychological Bulletin 138 (3), p. 427–457. External Links: Document Cited by: Introduction, Introduction, Discussion. Kuznetsova et al. (2017) A. Kuznetsova, P. B. Brockhoff, and R. H. Christensen LmerTest package: tests in linear mixed effects models. Journal of statistical software 82, p. 1–26. External Links: Document Cited by: Materials and Methods. Lüdecke et al. (2021) D. Lüdecke, M. S. Ben-Shachar, I. Patil, P. Waggoner, and D. Makowski performance: an R package for assessment, comparison and testing of statistical models. Journal of Open Source Software 6 (60), p. 3139. External Links: Document Cited by: Materials and Methods. Nakagawa and Schielzeth (2013) S. Nakagawa and H. Schielzeth A general and simple method for obtaining R2R^2 from generalized linear mixed-effects models. Methods in Ecology and Evolution 4 (2), p. 133–142. External Links: Document Cited by: Materials and Methods. Ragni et al. (2014) M. Ragni, S. Khemlani, and P. N. Johnson-Laird The evaluation of the consistency of quantified assertions. Memory & Cognition 42 (1), p. 53–66. External Links: Document Cited by: Introduction. Ragni et al. (2018) M. Ragni, N. Riesterer, S. Khemlani, and P. Johnson-Laird Individuals become more logical without feedback. In Proceedings of the Annual Meeting of the Cognitive Science Society, Vol. 40. External Links: Link Cited by: Introduction. Stanovich and West (1998) K. E. Stanovich and R. F. West Individual differences in rational thought. Journal of Experimental Psychology: General 127 (2), p. 161–188. External Links: Document Cited by: 3rd item, Introduction. Tessler et al. (2022) M. H. Tessler, J. B. Tenenbaum, and N. D. Goodman Logic, probability, and pragmatics in syllogistic reasoning. Topics in Cognitive Science 14 (3), p. 574–601. External Links: Document Cited by: Discussion. Wilkinson and Rogers (1973) G. Wilkinson and C. Rogers Symbolic description of factorial models for analysis of variance. Journal of the Royal Statistical Society Series C: Applied Statistics 22 (3), p. 392–399. External Links: Document Cited by: Q1: Prediction of retest performance. Woodworth and Sells (1935) R. S. Woodworth and S. B. Sells An atmosphere effect in formal syllogistic reasoning.. Journal of Experimental Psychology 18 (4), p. 451–460. External Links: Document Cited by: Introduction, Metrics on task and response sets.