Paper deep dive
Matching Object or Relation? Tracing Abstract Reasoning Inside VLMs
Gouki Minegishi, Hiroki Furuta, Takeshi Kojima, Yusuke Iwasawa, Yutaka Matsuo
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 92%
Last extracted: 10/13/2026, 3:56:09 AM
Summary
This paper investigates the failure of Vision Language Models (VLMs) on abstract reasoning tasks by employing the Relational Match-to-Sample (RMTS) paradigm. The study evaluates frontier API models (GPT, Claude, Gemini) and open-source families (Qwen3.5, Gemma-4, InternVL3) to identify levers that shift models toward relational matching, including capability tier, model scale, number of objects, and stimulus noise. Mechanistic analysis via Representational Similarity Analysis (RSA) and Causal Mediation Analysis (CMA) reveals that VLM abstract reasoning is implemented by two competing circuits: an early circuit organizing images by surface object features and a late circuit organizing them by abstract relations. Ablation studies on ARC-AGI-1 confirm the importance of these relational heads.
Entities (13)
Relation Signals (14)
Relational Circuit → implements → Abstract Reasoning
confidence 95% · VLM abstract reasoning is implemented by two competing circuits... a late circuit that organises them by their abstract relation.
Object Circuit → implements → Surface Feature Matching
confidence 95% · an early circuit that organises images by their surface object features
Model scale → influences → Relational Match-to-Sample Performance
confidence 95% · model scale... shift VLMs toward the relational match
Relational Match-to-Sample → usedtoevaluate → VLMs
confidence 95% · We close this gap by adopting the Relational Match-to-Sample (RMTS) paradigm... evaluated across frontier API models... and three open-source families
Stimulus Noise → degrades → Relational Match-to-Sample Performance
confidence 90% · the absence of per-object stimulus noise... shift VLMs toward the relational match
Gemma 4 → evaluatedin → Relational Match-to-Sample
confidence 90% · three open-source families (Qwen3.5, Gemma-4, InternVL3)
InternVL3 → evaluatedin → Relational Match-to-Sample
confidence 90% · three open-source families (Qwen3.5, Gemma-4, InternVL3)
Qwen3.5 → evaluatedin → Relational Match-to-Sample
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Vision Language Models (VLMs) excel on visual benchmarks but fail systematically on tasks requiring abstract reasoning. Existing benchmarks document this failure but cannot say \emph{why} it happens or which cognitive capability is missing. We close this gap by adopting the Relational Match-to-Sample (RMTS) paradigm from comparative and developmental psychology and pairing it with a mechanistic analysis of the model's internals. On a parametrically controlled stimulus set evaluated across frontier API models (GPT, Claude, Gemini) and three open-source families (Qwen3.5, Gemma-4, InternVL3), we identify four levers that shift VLMs toward the relational match---capability tier, model scale, the number of objects per scene, and the absence of per-object stimulus noise---together producing a developmental-like trajectory that mirrors the human \emph{relational shift}. Opening up the model, a per-layer representational similarity analysis and a causal mediation analysis reveal that VLM abstract reasoning is implemented by two competing circuits: an early circuit that organises images by their surface object features, and a late circuit that organises them by their abstract relation. Extending the analysis to ARC-AGI-1, we find that ablating the relational heads identified on RMTS degrades performance more than ablating random heads, indicating that the relational circuit is recruited beyond our controlled stimuli. We hope this mechanism-level view serves as a step toward understanding how abstract reasoning is implemented in VLMs.
Tags
Links
- Source: https://arxiv.org/abs/2610.07646v1
- Canonical: https://arxiv.org/abs/2610.07646v1
Trouble viewing inline? Open PDF directly →
Full Text
90,238 characters extracted from source content.
Expand or collapse full text
Matching Object or Relation? Tracing Abstract Reasoning Inside VLMs Gouki Minegishi Affiliation: The University of Tokyo Email: minegishi@weblab.t.u-tokyo.ac.jp Hiroki Furuta †thanks: Work done as an advisory role only. Affiliation: Google DeepMind Takeshi Kojima Affiliation: The University of Tokyo Yusuke Iwasawa Affiliation: The University of Tokyo Yutaka Matsuo Affiliation: The University of Tokyo Abstract Vision Language Models (VLMs) excel on visual benchmarks but fail systematically on tasks requiring abstract reasoning. Existing benchmarks document this failure but cannot say why it happens or which cognitive capability is missing. We close this gap by adopting the Relational Match-to-Sample (RMTS) paradigm from comparative and developmental psychology and pairing it with a mechanistic analysis of the model’s internals. On a parametrically controlled stimulus set evaluated across frontier API models (GPT, Claude, Gemini) and three open-source families (Qwen3.5, Gemma-4, InternVL3), we identify four levers that shift VLMs toward the relational match—capability tier, model scale, the number of objects per scene, and the absence of per-object stimulus noise—together producing a developmental-like trajectory that mirrors the human relational shift. Opening up the model, a per-layer representational similarity analysis and a causal mediation analysis reveal that VLM abstract reasoning is implemented by two competing circuits: an early circuit that organises images by their surface object features, and a late circuit that organises them by their abstract relation. Extending the analysis to ARC-AGI-1, we find that ablating the relational heads identified on RMTS degrades performance more than ablating random heads, indicating that the relational circuit is recruited beyond our controlled stimuli. We hope this mechanism-level view serves as a step toward understanding how abstract reasoning is implemented in VLMs. 1 Introduction Vision Language Models (VLMs) have advanced rapidly on visual tasks (Yue and others, 2024b; Yue and others, 2024a; Li et al., 2025c; Zhang et al., 2024; Li et al., 2025b), yet they continue to fail systematically on a kind of abstract reasoning (Mitchell et al., 2023; Campbell et al., 2024; Yiu et al., 2024). Existing benchmarks document this failure but conflate it with object recognition, attention, and configuration tracking; the resulting picture is descriptive rather than mechanistic, leaving open both why VLMs underperform and which cognitive capability they lack relative to mature human intelligence. A diagnostic that ties behaviour to internal mechanism has been missing. To close this gap, we adopt the Relational Match-to-Sample (RMTS) paradigm from comparative and developmental psychology (Christie and Gentner, 2013; Shivaram et al., 2023; Gentner and Hoyos, 2017), illustrated in Figure 1. Two sample images are introduced with a novel nonsense label (e.g., “these are both X”), and the model is asked which of two test images is also a X. The relational match preserves the abstract pattern shared by the samples—here, a small--large--small size pattern—with entirely new objects; the object match reuses the samples’ own objects (a blue circle and a red triangle) but breaks the pattern. The question is not which option is objectively correct, but which kind of generalisation the system reaches for. Adult humans gradually shift toward the relational match—a developmental trajectory known as the relational shift (Gentner et al., 2011; Christie and Gentner, 2013; Goswami, 1991)—while young children and non-human apes begin near the object-match end (Christie et al., 2016; Premack, 1983). RMTS thus offers a calibrated yardstick for relational abstraction in any cognitive system, with established human-developmental and cross-species reference points. Figure 1: The Relational Match-to-Sample (RMTS) task as we present it to a VLM. Left: the prompt as the model sees it. The VLM is taught a novel nonsense label X from two example images (Sample 1 and Sample 2), and is then asked which of two test candidates—Image A or Image B—is also a X. Right: the two competing solutions the model can latch onto. Image A (top) preserves the small--large--small relation with brand-new objects (green squares); the label X therefore generalises to A only if the model has bound it to the shared abstract relation between the samples—the relational match. Image B (bottom) reuses the very objects that appeared in the samples (a blue circle and red triangles) but breaks the size pattern; the label generalises to B only if the model has bound it to shared surface features—the object match. The two readings are mutually exclusive by construction: a relational reasoner picks A, while a system that keys on surface object similarity picks B. We bring this paradigm to VLMs. To eliminate confounds from naturalistic scene content, we design a formally controlled stimulus set whose scenes consist of geometric objects parameterised by (size, color, shape), and evaluate frontier API models (GPT, Claude, Gemini) alongside three open-source families (Qwen3.5, Gemma-4, InternVL3). On this task we make two contributions. (1) A behavioural map of the object–relational axis in VLMs. (Section 3) We identify four levers that systematically shift VLMs toward the relational match: capability tier (a 5858 p gap inside a single API family between Claude-Opus-4.6 and Claude-Haiku-4.5), within-family scale, the number of objects per scene n, and the absence of per-object stimulus noise. Chain-of-thought prompting modulates the first two by +20+20 to +57+57 p on non-saturated models. The resulting picture is a developmental-like trajectory across model families and conditions, mirroring the human relational shift. (2) A mechanistic account of how the choice is made. (Section 4) Behaviour alone does not explain why models choose as they do. Using a per-layer representational similarity analysis (Kriegeskorte et al., 2008, RSA;), we show that early layers organise the four images in line with the object-match solution, while late layers swing toward the relational-match solution, with the crossover arriving earlier and sharper in larger and more relationally-biased models. A causal mediation analysis (Meng et al., 2022; Wang et al., 2023, CMA;) isolates a single dominant relational head and a single dominant object head per checkpoint, with logit-margin swings of order one unit when patched. Together, RSA and CMA reveal that VLM abstract reasoning is implemented by two competing circuits: an early circuit that organises images by their surface object features, and a late circuit that organises them by their abstract relation. When the model answers directly, its choice reflects which of the two circuits dominates at the output; CoT, by contrast, reaches the relational answer through a route that does not depend on the relational heads. We hope that bringing controlled, mechanism-aware experimental designs of this kind to VLMs can serve as a step toward a research programme on abstract reasoning that goes beyond aggregate benchmark scores and engages directly with how that capability is implemented inside the model. 2 The Relational Match-to-Sample Setup Figure 2: Typical RMTS stimuli in cognitive science. In cognitive science, RMTS is typically conducted with naturalistic scenes containing multiple objects such as animals (Gentner and Hoyos, 2017). For instance, as illustrated in Figure 2, stimuli are constructed by treating entities such as rabbits or penguins as the units over which relational and object matches are defined. In contrast, to probe how VLMs perform abstract reasoning under minimal confounds, we adopt a tightly controlled stimulus design in which the attributes of each object are explicitly parameterised. This design follows a line of work on controlled task construction in deep-learning vision research (Assouel et al., 2025; Park et al., 2024; Okawa et al., 2023). Concrete examples of our stimuli are shown in Figure 1 and, more comprehensively, in Figure 9 in Appendix D. 2.1 Trial structure: samples, relational match, object match We make the structure of an RMTS trial precise. Each scene is built from objects parameterised by three attributes drawn from finite alphabets: =red,blue,green,=small,medium,large,ℋ=circle,triangle,square,C=\red,blue,green\\,, =\small,medium,large\\,, =\circle,triangle,square\\,, the colour, size, and shape sets respectively. An object is a triple o=(c,s,h)∈×ℋo=(c,s,h) ×S×H. Definition 1 (Samples). A sample of size n is an ordered tuple of n objects that share a common shape11 1 Fixing the shape within a sample preserves the setting illustrated in Figure 2: it lets shape act as a perceptual grouping cue that signals which objects belong to the same scene, while keeping the relation carried entirely by colour and size. Allowing shape to vary within a sample would make the task substantially harder, as the model would also need to identify the grouping itself.: =(o1,…,on),oi=(ci,si,h)∀i.x=(o_1,…,o_n)\,, o_i=(c_i,s_i,h)\;\;∀ i\,. The within-scene patterns are the equivalence classes of the colour and size sequences: P()=pat(c1,…,cn)P_C(x)=pat(c_1,…,c_n) and P()=pat(s1,…,sn)P_S(x)=pat(s_1,…,s_n), where for n=3n=3 we use the standard alphabet pat∈A,AAB,ABA,ABB,ABCpat∈\ A, AAB, ABA, ABB, ABC\ (e.g. pat(small,large,small)=ABApat(small,large,small)= ABA). The relation of a scene is the pair R()=(P(),P())R(x)= (P_C(x),\,P_S(x) ). A trial is a tuple T=((1),…,(K),rel,obj,ℓ)T= (x^(1),…,x^(K);y_rel,y_obj;\, ) consisting of K≥2K\!≥\!2 sample scenes (k)x^(k) that share the same relation R∗=R((k))R^*=R(x^(k)) but use distinct shapes and distinct attribute-value assignments, two test scenes rely_rel and objy_obj, and a label ℓ (a novel three-letter nonce word) attached to each sample. The model is asked which of rel,objy_rel,y_obj is also an ℓ . Definition 2 (Relational match). A scene rely_rel is a relational match for samples sharing relation R∗R^* if (i) its relation matches: R(rel)=R∗R(y_rel)=R^*; (i) its shape is not used by any sample, h(rel)∉h((k))kh(y_rel)∉\h(x^(k))\_k; and (i) its concrete colour and size values are disjoint from those used by the samples (when the alphabet permits). Definition 3 (Object match). A scene objy_obj is an object match if (i) its relation differs from the samples’, R(obj)≠R∗R(y_obj)≠ R^*; (i) every object in objy_obj is drawn from the union of objects appearing in the samples, o:o∈obj⊆⋃ko:o∈(k)\o:o _obj\ _k\o:o ^(k)\; and (i) the shapes used in objy_obj are mixed across ≥2≥ 2 of the sample shapes. The two definitions are mutually antagonistic by construction. Definition 2 guarantees that picking rely_rel requires inferring the abstract relation from the samples and applying it to genuinely new objects (new shape, new colour/size values); Definition 3 guarantees that picking objy_obj requires only matching surface features back to the samples (because every object in objy_obj literally appears in some sample). A system that keys on object similarity therefore deterministically picks objy_obj; a system that keys on the relation R∗R^* deterministically picks rely_rel. Following the cognitive-science convention (Gentner and Hoyos, 2017; Christie and Gentner, 2013) we report the rate at which the model picks rely_rel (the relational rate, Rel%) and treat the contrast as a bias rather than an accuracy measurement. 2.2 Prompt and evaluation protocol Each trial is presented in a single turn containing all four images followed by a short instruction (see concrete prompt in Appendix C). The label is a novel three-letter nonce word drawn from a fixed pool to avoid priming any pre-existing concept; the question is phrased neutrally (“Which one is a X?”) and constrained with “Final answer: Image ” so that the next token is forced to be either A or B. We sweep three open-source families covering 0.8B–38B parameters—Qwen3.5 (Team, 2026) (0.8B/2B/4B/9B/27B), Gemma-4 (Gemma Team, Google DeepMind, 2026) (E2B/E4B/26B/31B), and InternVL3 (Zhu et al., 2025) (1B/2B/8B/14B/38B)—together with frontier VLM models from OpenAI (GPT-5.4, GPT-5.4-mini) (OpenAI, 2026), Anthropic (Anthropic, 2026) (Claude-Opus-4.6, Claude-Haiku-4.5), and Google (Google DeepMind, 2026) (Gemini-3.1-Pro). For open-source models we read the raw logits of the tokens A and B directly from the next-token distribution, avoiding free-text parsing; for closed-source API models we parse the generated response. Two sources of bias must be removed for a relational rate to be interpretable. First, the choice of letter labels (A/B) and of nonce word can each induce a constant preference; we cycle the nonce word across a fixed pool of five three-letter strings—X, Y, Z, Q, and W—assigned uniformly at random per trial, and average the relational rate over labels, integrating out any nonce-specific preference. The pool is restricted to repeated-character strings so that no element carries lexical content that could prime a real concept. Second, the model’s prior over the answer position can favour one letter—a hard “A-bias” is observable in some open-source checkpoints. To remove this, every trial is presented in both orderings (relational match as A and as B) and the relational rate is averaged over the two; any constant letter preference therefore cancels. Unless stated otherwise, the relational rate (Rel%) is reported over all trials per model. By default, we use the n=3,k=2n=3,k=2 configuration shown in Figure 1, which yields 729729 distinct trials per model; together with the two label orderings used to remove A-bias, this gives 729×2729× 2 evaluations per model. Full configuration and model details are deferred to Appendices A and B. 3 Behaviour: When VLMs Pick the Relational Match We characterise relational match bias along four levers—(1) capability tier with chain-of-thought (CoT) in Section 3.1, (2) model-size scaling in Section 3.2, (3) number of objects n in Section 3.3, and (4) stimulus noise in Section 3.4. The levers are instances of two factors: the capability of the model (tier and scale, with CoT as an inference-time counterpart), the analogue of the maturation that drives the human relational shift; and the strength of the relational evidence in the stimulus (n and noise).22 2 More objects make a shared property overwhelming and each object less individuating; per-object noise degrades exactly the colour/size evidence that defines the relation. (a) Frontier VLMs. (b) Model scale. (c) Number of objects n. Figure 3: Capability tier, model scale, and number of objects each promote the relational match. (a) Capability tier drives a 5858 p spread within Claude (Opus 100100% vs. Haiku 42.542.5%); CoT lifts every non-saturated model (+51+51 to +57+57 p). (b) Within each open-source family, larger checkpoints pick the relational match more reliably (solid lines, filled circles: CoT; dashed lines, open squares: no-CoT). (c) Raising the number of objects per scene from n=2n=2 to n=20n=20 shifts the relational rate upward. 3.1 Frontier VLMs show a wide object–relational spread We begin by measuring RMTS performance across five frontier VLMs—GPT-5.4-mini, Claude-Haiku-4.5, Claude-Opus-4.6, Gemini-3.1-Pro, and GPT-5.4—as shown in 3(a). Strikingly, even on this set of frontier models, the no-CoT condition reveals a pronounced spread in relational rate, ranging from 32% for GPT-5.4-mini to 100% for GPT-5.4 (a 68-p gap). Adding chain-of-thought prompting consistently shifts models toward the relational match: the two weakest models, GPT-5.4-mini and Claude-Haiku-4.5, gain 51-p (32 → 83) and 57-p (43 → 100) respectively, while the three already-saturated models remain at 100%.33 3 The set is designed as an instrument for analysing how a model answers rather than as a benchmark; for closed-source models it does act as one and is at ceiling, but a harder relation (numerical equality) brings them off it (GPT-5.4: 82.582.5%; Section M.1). Following Kojima et al. (2022), we elicit CoT with a “Let’s think step by step” prompt; the exact prompt template is given in Appendix C. Inspecting the generated reasoning traces, we find that the models explicitly extract the abstract relation shared between the two samples and apply it to the test scenes—example traces are provided in Appendix E. 3.2 Model scaling drives the relational match Within each open-source family, larger models pick the relational match more reliably (3(b); per-model rates in Table 5). Gemma-4 rises monotonically from 26.326.3% (E2B) to 100100% (31B), and Qwen3.5 from 50.050.0% (0.8B) to 89.389.3% (9B), with a −3.4-3.4 p dip at 27B that is at the noise floor. InternVL3 reaches 100100% at 38B; its apparent dip at 8B (28.928.9%) arises because 1B and 2B sit at chance, committing to neither candidate, whereas 8B commits to the object match before scale flips it.44 4 The rank correlation between parameter count and relational rate is positive within every family (Spearman ρ=0.80ρ=0.80, 1.001.00, 0.600.60 for Qwen3.5, Gemma-4, InternVL3; ρ=0.78ρ=0.78 over all fourteen checkpoints), and the scale effect also holds for numerical equality (Section M.1). The no-CoT rate correlates with log -model size in Gemma-4 (r=+0.92r=+0.92), Qwen3.5 (+0.80+0.80) and, more weakly, InternVL3 (+0.58+0.58), and CoT trajectories sit above the no-CoT ones in every family. The aggregate picture is a developmental-like trajectory mirroring the human relational shift (Christie and Gentner, 2013; Gentner and Hoyos, 2017): surface-dominant at small scale, relation-dominant at large scale. 3.3 More objects per scene promote the relational match Raising the number of objects per scene from n=2n=2 to n=20n=20 shifts the relational rate upward in the object-biased middle tiers of each family (3(c), per-model Pearson r(n,Rel)r(n,Rel) in the legend; example stimuli in Figure 10). Gemma-4-E4B climbs from 0.530.53 to 0.800.80 (r=+0.71r\!=\!+0.71), InternVL3-14B from 0.100.10 to 0.830.83 (r=+0.58r\!=\!+0.58) and Qwen3.5-4B from 0.750.75 to 0.820.82 (r=+0.72r\!=\!+0.72); InternVL3-8B shows a weaker trend (r=+0.36r\!=\!+0.36), and the near-ceiling Gemma-4-26B and Qwen3.5-27B show small positive correlations around saturation. As n grows, the abstract relation becomes visually salient, and the surface heuristic that succeeds at n=2n=2 (“the candidate that reuses the samples’ objects looks more familiar”) no longer suffices. The same trend holds for GPT-5.4-mini, the only non-saturated frontier model in our suite (Figure 13). 3.4 Stimulus noise weakens the relational match The controlled stimuli used so far are sharp idealisations: every object has a categorical colour (red/blue/green) and a categorical size (small/medium/large), rendered at fixed pixel values. Real-world visual tasks (e.g., natural-image analogues of RMTS) are messier—two “red” objects rarely have identical pixel values, and two “small” objects rarely have identical pixel sizes. To probe how robust the relational behaviour observed above is to this kind of low-level perturbation, we add per-object Gaussian noise to colour and size while leaving the categorical identity of each object unchanged. Concretely, for each object o in a scene we apply rgbo←rgbo+(,σc23)andsizeo←sizeo+(0,σs2),rgb_o _o+N(0, _c^2I_3) _o _o+N(0, _s^2), (1) clipped to valid pixel and size ranges. We sweep σc∈0,5,10,20,40,80 _c\!∈\!\0,5,10,20,40,80\ (RGB units) and σs∈0,1,2,4,8 _s\!∈\!\0,1,2,4,8\ (px) with three independent noise seeds per cell. Examples of the resulting stimuli across the noise sweep are shown in Figure 11. Figure 4: Per-object stimulus noise reduces the relational rate. Relational rate as a function of colour-noise scale σc _c (RGB units; x-axis) for four open-source checkpoints, with size-noise scale σs _s (px) encoded by line colour. Both noise channels degrade the relational rate, and within each family the smaller checkpoint degrades more steeply. Shaded bands: ±1± 1 s.e. over three noise seeds per cell. Figure 4shows the results for four open-source checkpoints (Gemma-4-26B/E4B, InternVL3-38B/8B). Two patterns are immediately visible. First, both noise channels degrade the relational rate: every panel slopes downward as σc _c grows, and within each panel the curves separate by σs _s (with the largest σs=8 _s=8 uniformly the lowest). Second, smaller models within each family are markedly more sensitive to noise. Gemma-4-26B drops only from ∼100 \!100% (clean) to ∼92 \!92% under the most extreme perturbation, whereas Gemma-4-E4B falls from ∼75 \!75% to ∼50 \!50% over the same sweep—a 2525-p degradation. The InternVL3 family shows the same asymmetry: InternVL3-38B loses only 88–99 p end-to-end, while InternVL3-8B falls from ∼25 \!25% to ∼10 \!10%. Taken together, these results show that relational abstraction is sensitive to low-level visual perturbations even when the categorical identity of every object (red/blue/green; small/medium/large) is preserved. Whether this is because relations are computed over the rendered pixels or because the model’s internal categorical representations become noisier remains open; internally, the same perturbation lowers the relational selectivity of late layers (Section M.4). See Appendix I for per-attribute differences and results on additional models. 4 Internal Mechanism: Two Competing Circuits The behavioural results establish that VLMs sit at varying positions on the object–relational axis and that several levers shift them, but not how the choice is made internally. Our central finding is that it is not made by a single, monolithic ‘relational module’ that grows stronger with scale. Rather, two distinct internal circuits55 5 We use circuit in a functional sense: layers and heads with a distinguishable representational signature (organising the images by object features or by relation) and a measurable causal effect on the choice under patching and ablation—not a complete, reverse-engineered computational subgraph (Elhage et al., 2021; Wang et al., 2023).—one keying on object identity, one on the abstract relation—are simultaneously active, and the model’s direct answer reflects which circuit dominates. We trace the layer-wise dynamics of the two circuits (Section 4.1), show that the levers that shifted behaviour reshape them in the predicted direction (Section 4.2), and confirm causally that they drive the choice (Section 4.3). 4.1 The two circuits compete across layer depth For every layer ℓ , we extract the residual-stream activation at the last token of each image, yielding four hidden vectors per trial: hs1(ℓ),hs2(ℓ),hrel(ℓ),hobj(ℓ)∈ℝdh_s_1^( ),h_s_2^( ),h_rel^( ),h_obj^( ) ^d. We assemble them into a ×44\!×\!4 image–image cosine similarity matrix (Kriegeskorte et al., 2008; Assouel et al., 2025), RSMij(ℓ)=cos(hi(ℓ),hj(ℓ))(i,j∈s1,s2,rel,obj),RSM^( )_ij= \! (h_i^( ),\,h_j^( ) )\ (i,j∈\s_1,s_2,rel,obj\), and compare it against two binary ×44\!×\!4 target matrices encoding the two competing solutions. TrelT_rel pulls the relational match close to both samples (it instantiates the same abstract pattern); TobjT_obj pulls the object match close to both samples (it reuses their concrete objects): Trel=(s1s2relobjs11010s20110rel1110obj0001),Tobj=(s1s2relobjs11001s20101rel0010obj1101).T_rel= ( array[]c|c&s_1&s_2&rel&obj\\ s_1&1&0&1&0\\ s_2&0&1&1&0\\ rel&1&1&1&0\\ obj&0&0&0&1 array ), T_obj= ( array[]c|c&s_1&s_2&rel&obj\\ s_1&1&0&0&1\\ s_2&0&1&0&1\\ rel&0&0&1&0\\ obj&1&1&0&1 array ). Note that TrelT_rel leaves (s1,s2)(s_1,s_2) at 00: our hypothesis concerns whether each sample sits close to the relational candidate, not how similar the two samples are to each other, and asserting the latter would reward generic image similarity unrelated to the relation.66 6 A variant target that also clusters the two samples together yields layer-wise Rel−-Obj curves that correlate with ours at r=0.91r=0.91–1.001.00 for all fourteen open checkpoints (Section M.5). We score each layer by the Pearson correlation between the upper triangle of RSM(ℓ)RSM^( ) and that of each target: Rel-RSA(ℓ)=r(RSM(ℓ),Trel),Obj-RSA(ℓ)=r(RSM(ℓ),Tobj).Rel-RSA( )=r\! (RSM^( ),T_rel ), -RSA( )=r\! (RSM^( ),T_obj ). High Rel-RSA at layer ℓ means the model is organising the four images by their abstract pattern at that depth; High Obj-RSA means it is keying on surface features. The two scores are not redundant—a layer can score positively on neither, one, or both—and the layer-wise trajectory of (Rel-RSA(ℓ),Obj-RSA(ℓ))(Rel-RSA( ),Obj-RSA( )) reveals which solution dominates at which depth. Figure 5: Layer-wise emergence of the relational circuit. Per-layer Rel-RSA(ℓ)Rel-RSA( ) (blue) and Obj-RSA(ℓ)Obj-RSA( ) (red) for three Qwen3.5 checkpoints, plotted against relative layer depth. The smallest model (0.8B; Rel% =50.0=50.0) never produces a relational circuit. The two larger models (4B, 9B; Rel% =77.3,89.3=77.3,89.3) exhibit an early object circuit followed by a relational circuit that overtakes it at intermediate depth. The crossover shifts earlier with scale. Shaded bands: ±1± 1 s.e. over trials. Figure 5shows the layer-wise trajectories of Rel-RSA(ℓ)Rel-RSA( ) and Obj-RSA(ℓ)Obj-RSA( ) for three Qwen3.5 checkpoints spanning the behavioural object–relational axis: Qwen3.5-0.8B (Rel% =50.0=50.0), Qwen3.5-4B (77.377.3), and Qwen3.5-9B (89.389.3). For Qwen3.5-0.8B, Obj-RSA(ℓ)Obj-RSA( ) stays high (∼0.6 \!0.6–0.80.8) and Rel-RSA(ℓ)Rel-RSA( ) negative (∼−0.6 \!-0.6) at every depth, matching its chance-level Rel%. For Qwen3.5-4B and Qwen3.5-9B, the picture is qualitatively different: Obj-RSA(ℓ)Obj-RSA( ) dominates the early layers (relative depth ≲0.3 0.3–0.40.4), and Rel-RSA(ℓ)Rel-RSA( ) rises to overtake it at intermediate layers and remains the dominant signal through the rest of the network. We refer to these two regimes as the object circuit (early, red shading in Figure 5) and the relational circuit (late, blue shading). The crossover depth shifts earlier with scale (∼0.45 \!0.45 in 4B, ∼0.30 \!0.30 in 9B), and the asymptotic Rel-RSA grows with scale, mirroring the behavioural ordering (50.0→77.3→89.350.0→ 77.3→ 89.3). Surface-feature matching is thus not absent in models that answer relationally; it is overwritten by the relational circuit at intermediate depth. Figure 6: The relational circuit strengthens as the number of objects per scene grows. Peak-layer gap Δ-RSA=maxℓ[Rel-RSA(ℓ)−Obj-RSA(ℓ)] -RSA= _ [Rel-RSA( )-Obj-RSA( )] as a function of n, for six open-source checkpoints. Pearson correlations with n are reported in the legend. Shaded bands: ±1± 1 s.e. over trials. Figure 7: Per-head activation patching effects in Gemma-4-26B. Δℓ=logitA−logitB =logit_A-logit_B for each (layer, head) pair: circle size encodes |Δℓ|| |, colour encodes its sign (blue: relational-supporting; red: object-supporting). Two outliers stand out—L28 H12 (deep, relational) and L12 H11 (mid, object). 4.2 The relational circuit is amplified by the number of objects In Section 3.3 we showed that increasing the number of objects per scene n shifts the behavioral relational rate upward. We now ask whether this behavioral shift is mirrored in the internal RSA dynamics and whether the two are quantitatively linked. Figure 7 plots, for each of six open-source checkpoints, the peak-layer gap Δ-RSA=maxℓ[Rel-RSA(ℓ)−Obj-RSA(ℓ)] -RSA= _ [Rel-RSA( )-Obj-RSA( ) ] as a function of n. The trend is consistent across models: Δ-RSA -RSA increases with n, with five of the six checkpoints showing a strong positive Pearson correlation with n (Gemma-4-E4B r=+0.81r\!=\!+0.81, Gemma-4-26B r=+0.76r\!=\!+0.76, InternVL3-8B r=+0.81r\!=\!+0.81, Qwen3.5-4B r=+0.75r\!=\!+0.75, Qwen3.5-27B r=+0.80r\!=\!+0.80); InternVL3-14B is the only weak case (r=+0.15r\!=\!+0.15). The lever that pushed behaviour toward the relational match thus also strengthens the relational circuit in intermediate representations, rather than merely tipping the final answer. Stimulus noise, the lever that weakened behaviour, acts on the same quantity in the opposite direction (Section M.4). 4.3 Causal mediation localises the circuits to specific heads The evidence so far is correlational: Rel-RSARel-RSA and Obj-RSAObj-RSA identify layers whose representations align with the two competing solutions, but they do not establish that those layers are causally responsible for the model’s choice. To make a causal claim, we follow the mechanistic-interpretability (Bereska and Gavves, 2024; Liu et al., April 28, 2025) convention of intervening on activations and measuring the resulting change in behaviour (Meng et al., 2022; Wang et al., 2023; Assouel et al., 2025; Heimersheim and Nanda, 2024; Dumas et al., 2025; Li et al., 2026). Concretely, we run per-head activation patching on Gemma-4-26B. We form a clean context c1c_1 (relational match in position A) and a corrupted context c2c_2 (relational match in position B). For each (layer, head) pair (ℓ,h)( ,h) we patch the post-oprojo_proj activation at the prediction position from c2c_2 into c1c_1 and record the resulting shift in the logit margin Δℓ=logitA−logitB =logit_A-logit_B. A negative Δℓ identifies a head that normally carries the relational signal (patching it from a corrupted run flips the model away from the relational answer); a positive Δℓ identifies a head that carries the object signal. Figure 7shows the resulting (layer, head) effect map. The effect is heavy-tailed: most heads have negligible |Δℓ|| |, while a few carry essentially all of the causal signal. The two clear outliers, L28 H12 (deep, relational-supporting) and L12 H11 (mid-depth, object-supporting), sit at distinct depths on opposite sides of the sign axis—consistent with the layer-wise RSA in Figure 5, where the object circuit dominates early and the relational circuit takes over late. The two-stage organisation observed in representation space is thus reflected at the level of individual heads. The same organisation appears on Gemma-4-31B (Appendix J) and on five further checkpoints from Gemma-4, InternVL3 and Qwen2.5-VL, where in every case a few heads carry most of the relational effect and the strongest sits in the second half of the network (Section M.2). CoT bypasses the relational heads. Zeroing the three strongest relational heads of Gemma-4-26B drops its direct answers to chance, yet with CoT the ablated model is back at 100100% (Section M.3): the competition governs direct answers, while the CoT gains of Section 3.1 run through a separate, language-mediated route. Figure 8: Ablating relational heads degrades ARC-AGI-1 performance more than random ablation. Cell-wise accuracy on ARC-AGI-1 for Gemma-4-26B and -31B under clean, rel-3 zero (top-3 relational heads from CMA), and random-3 (three random heads) conditions. Targeted ablation drops accuracy more than random in both checkpoints. Error bars on random-3: s.d. across seeds. 5 Discussion and Related Work Summary. Using the RMTS paradigm as a controlled probe of abstract reasoning in VLMs, Section 3 showed how model capability and the strength of relational evidence in the stimulus determine where a model sits on the object–relational axis, and Section 4 showed that the underlying mechanism is not a single “relational module” but two competing circuits—one keying on surface object features, one on the abstract relation—whose relative strength is reflected layer-wise (RSA), tracked by the same levers that shifted behaviour (n and noise), and causally localised to a small number of late-layer attention heads (CMA) across three model families; CoT, in contrast, reaches the relational answer without these heads. Extended related work is in Appendix O. Mechanistic interpretability of VLMs. Mechanistic interpretability (Bereska and Gavves, 2024; Sharkey et al., 2025; Zhang et al., 2026; Nanda et al., 2023; Olah et al., 2020; Elhage et al., 2021) seeks to reverse-engineer large neural networks at the level of weights, activations, and circuits, originally for LLMs and increasingly for VLMs (Ben Melech Stan et al., 2024; Liu et al., April 28, 2025; Liu et al., 2026; Fu et al., 2025; Yang et al., 2025). Recent work has localised several VLM-specific phenomena: information loss already at the vision encoder (Li et al., 2025a), performance gains from strengthened image attention (Chen et al., 2025), and the emergence of binding mechanisms between visual tokens and concept slots (Assouel et al., 2025). What has remained largely uncharted is the mechanism by which a VLM extracts the abstract relation between objects, as opposed to the identity of individual objects. Our RSA + CMA pipeline addresses this gap: we show that relation extraction is not a diffuse property of the network but is implemented by a small number of identifiable late-layer heads, sitting downstream of—and competing with—an earlier object-feature circuit. Closest to our work, Lepori et al. (2024) find a perceptual-then-relational decomposition of same/different judgements in vision transformers, mirroring our early object and late relational stages (Appendix O compares the two in detail). Toward AI capable of abstract reasoning. Recent work has repeatedly identified abstract reasoning as a core weakness of contemporary AI systems, with benchmarks such as ARC-AGI (Chollet, 2019; Chollet et al., 2025) highlighting the gap between human and machine performance. The gap is especially pronounced for VLMs: they underperform on ARC-style tasks (Mitchell et al., 2023), fail tasks that young children solve (Yiu et al., 2024), and struggle with analogical reasoning more broadly (Campbell et al., 2024). By bringing the controlled RMTS paradigm to VLMs and pairing behavioural measurements with internal-state and causal analyses, we provide a more mechanistic account of which circuits succeed and fail at relation extraction, and when. This account is directly actionable. As a first step, we ablated the top relational heads identified by CMA (Section 4.3) on ARC-style tasks and found that this targeted ablation degrades performance more than random head ablation of the same size (Figure 8; details in Appendix L). The relational circuit identified on RMTS is therefore not specific to our controlled stimuli—it is recruited by the model on naturalistic abstract-reasoning benchmarks as well. We hope this kind of mechanistically informed analysis can serve as a foundation for building VLMs that genuinely abstract over objects rather than merely match their surfaces. Ethics statement This work studies publicly released pretrained VLMs and commercial APIs on synthetic geometric stimuli; it involves no human subjects, personal data, or new data collection from people. The work is methodological—a diagnostic for relational reasoning in VLMs—and we see no direct dual-use implications. Identifying specific heads that carry the relational answer could, in principle, be used to induce or suppress particular generalisation patterns at deployment; any such edit should first be evaluated for downstream effects on unrelated tasks. Limitations are discussed in Appendix N. Reproducibility statement The stimulus design is fully specified in Section 2 and Appendix A; exact model identifiers, decoding settings, and compute are listed in Appendix B; verbatim prompts are given in Appendix C; the RSA and CMA procedures are defined in Section 4; and the ARC-AGI-1 evaluation protocol, including the head-edit recipe, is described in Appendix L; the protocols of the additional analyses are given in Appendix M. All open-source checkpoints are publicly available, and every experiment uses greedy decoding. AI use statement In this work, we used generative AI tools (general-purpose chat LLMs and coding assistants) to help write and edit experiment and analysis code. Additionally, we used them to draft and edit text for readability and to format the manuscript and references. Generative AI tools were not used as a component of the core methodology, and the models evaluated in this paper are the objects of study rather than research assistants. All AI-assisted code was reviewed and tested by the authors, and all AI-assisted text was checked by the authors. We take responsibility for the final content of this work, including text, claims, and artifacts produced with the aid of generative AI. Acknowledgments [To be added.] References Anthropic (2026) Anthropic Introducing Claude Opus 4.6. Note: https://w.anthropic.com/news/claude-opus-4-6Accessed: 2026-05-07 Cited by: §2.2. Assouel et al. (2025) R. Assouel, D. Campbell, Y. Bengio, and T. Webb Visual symbolic mechanisms: emergent symbol processing in vision language models. arXiv preprint arXiv:2506.15871. Cited by: §2, §4.1, §4.3, §5. Ben Melech Stan et al. (2024) G. Ben Melech Stan, E. Aflalo, R. Y. Rohekar, A. Bhiwandiwalla, S. Tseng, M. L. Olson, Y. Gurwicz, C. Wu, N. Duan, and V. Lal LVLM-Interpret: an interpretability tool for large vision-language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 8182–8187. Cited by: §5. Bereska and Gavves (2024) L. Bereska and S. Gavves Mechanistic interpretability for AI safety - a review. Transactions on Machine Learning Research. Note: Survey Certification, Expert Certification External Links: ISSN 2835-8856, Link Cited by: §4.3, §5. Campbell et al. (2024) D. Campbell, S. Rane, T. Giallanza, N. De Sabbata, K. Ghods, A. Joshi, A. Ku, S. M. Frankland, T. L. Griffiths, J. D. Cohen, et al. Understanding the limits of vision language models through the lens of the binding problem. Advances in Neural Information Processing Systems 37, p. 113436–113460. Cited by: Appendix O, §1, §5. Chen et al. (2025) S. Chen, T. Zhu, R. Zhou, J. Zhang, S. Gao, J. C. Niebles, M. Geva, J. He, J. Wu, and M. Li Why is spatial reasoning hard for VLMs? an attention mechanism perspective on focus areas. External Links: 2503.01773, Link Cited by: §5. Chollet et al. (2025) F. Chollet, M. Knoop, G. Kamradt, B. Landers, and H. Pinkard ARC-AGI-2: a new challenge for frontier AI reasoning systems. arXiv preprint arXiv:2505.11831. Cited by: §5. Chollet (2019) F. Chollet On the measure of intelligence. arXiv preprint arXiv:1911.01547. Cited by: §L.1, §5. Christie et al. (2016) S. Christie, D. Gentner, J. Call, and D. B. M. Haun Sensitivity to relational similarity and object similarity in apes and children. Current Biology 26 (4), p. 531–535. External Links: ISSN 0960-9822, Document, Link Cited by: §1. Christie and Gentner (2013) S. Christie and D. Gentner Language helps children succeed on a classic analogy task. Cognitive science 38, p. . External Links: Document Cited by: Appendix O, §1, §2.1, §3.2. Dumas et al. (2025) C. Dumas, C. Wendler, V. Veselovsky, G. Monea, and R. West Separating tongue from thought: activation patching reveals language-agnostic concept representations in Transformers. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), W. Che, J. Nabende, E. Shutova, and M. T. Pilehvar (Eds.), Vienna, Austria, p. 31822–31841. External Links: Link, Document, ISBN 979-8-89176-251-0 Cited by: §4.3. Elhage et al. (2021) N. Elhage, N. Nanda, C. Olsson, T. Henighan, N. Joseph, B. Mann, A. Askell, Y. Bai, A. Chen, T. Conerly, N. DasSarma, D. Drain, D. Ganguli, Z. Hatfield-Dodds, D. Hernandez, A. Jones, J. Kernion, L. Lovitt, K. Ndousse, D. Amodei, T. Brown, J. Clark, J. Kaplan, S. McCandlish, and C. Olah A mathematical framework for transformer circuits. Transformer Circuits Thread. Note: https://transformer-circuits.pub/2021/framework/index.html Cited by: §5, footnote 5. Fu et al. (2025) S. Fu, tyler bonnen, D. Guillory, and T. Darrell Hidden in plain sight: VLMs overlook their visual representations. In Second Conference on Language Modeling, External Links: Link Cited by: §5. Gemma Team, Google DeepMind (2026) Gemma Team, Google DeepMind Gemma 4. Note: https://deepmind.google/models/gemma/gemma-4/Accessed: 2026-05-07 Cited by: §2.2. Gentner et al. (2011) D. Gentner, F. K. Anggoro, and R. S. Klibanoff Structure mapping and relational language support children’s learning of relational categories. Child development 82 4, p. 1173–88. External Links: Link Cited by: §1. Gentner et al. (1997) D. Gentner, S. Brem, R. Ferguson, and P. Wolff Analogy and creativity in the works of Johannes Kepler. Creative thought: An investigation of conceptual structures and processes, p. . External Links: ISBN 1-55798-404-2, Document Cited by: Appendix O. Gentner and Hoyos (2017) D. Gentner and C. Hoyos Analogy and abstraction. Topics in Cognitive Science 9 (3), p. 672–693. External Links: Document, Link, https://onlinelibrary.wiley.com/doi/pdf/10.1111/tops.12278 Cited by: Appendix O, §1, §2.1, §2, §3.2. Goel (1997) A.K. Goel Design, analogy, and creativity. IEEE Expert 12 (3), p. 62–70. External Links: Document Cited by: Appendix O. Google DeepMind (2026) Google DeepMind Gemini 3.1 Pro. Note: https://deepmind.google/models/gemini/pro/Accessed: 2026-05-07 Cited by: §2.2. Goswami (1991) U. Goswami Analogical reasoning: what develops? a review of research and theory. Child Development 62 (1), p. 1–22. External Links: Document, Link, https://onlinelibrary.wiley.com/doi/pdf/10.1111/j.1467-8624.1991.tb01511.x Cited by: §1. Heimersheim and Nanda (2024) S. Heimersheim and N. Nanda How to use and interpret activation patching. arXiv preprint arXiv:2404.15255. Cited by: §4.3. Kojima et al. (2022) T. Kojima, S. S. Gu, M. Reid, Y. Matsuo, and Y. Iwasawa Large language models are zero-shot reasoners. Advances in neural information processing systems 35, p. 22199–22213. Cited by: §3.1. Kriegeskorte et al. (2008) N. Kriegeskorte, M. Mur, and P. Bandettini Representational similarity analysis – connecting the branches of systems neuroscience. Frontiers in Systems Neuroscience 2, p. 4. External Links: Document Cited by: §1, §4.1. Leatherdale (1974) W. H. Leatherdale The role of analogy, model, and metaphor in science. American Elsevier Pub. Co., New York. External Links: Link Cited by: Appendix O. Lepori et al. (2024) M. A. Lepori, A. R. Tartaglini, W. K. Vong, T. Serre, B. M. Lake, and E. Pavlick Beyond the doors of perception: vision transformers represent relations between objects. In Advances in Neural Information Processing Systems, Vol. 37, p. 131503–131544. Cited by: Appendix O, §5. Li et al. (2026) Q. Li, Z. Ye, X. Feng, W. Zhong, W. Ma, and X. Feng Causal tracing of object representations in large vision language models: mechanistic interpretability and hallucination mitigation. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 40, p. 31645–31653. Cited by: §4.3. Li et al. (2025a) W. Li, R. Tang, C. Li, C. Zhang, I. Vulic, and A. Søgaard Lost in embeddings: information loss in vision-language models. arXiv preprint arXiv:2509.11986 2. Cited by: §5. Li et al. (2025b) Y. Li, Z. Lai, W. Bao, Z. Tan, A. Dao, K. Sui, J. Shen, D. Liu, H. Liu, and Y. Kong Visual large language models for generalized and specialized applications. External Links: 2501.02765, Link Cited by: §1. Li et al. (2025c) Z. Li, X. Wu, H. Du, F. Liu, H. Nghiem, and G. Shi A survey of state of the art large vision language models: benchmark evaluations and challenges. In Proceedings of the Computer Vision and Pattern Recognition Conference, p. 1587–1606. Cited by: §1. Liu et al. (April 28, 2025) Y. Liu, Y. Zhang, and S. Yeung-Levy Mechanistic interpretability meets vision language models: insights and limitations. In ICLR Blogposts 2025, Note: https://d2jud02ci9yv69.cloudfront.net/2025-04-28-vlm-understanding-29/blog/vlm-understanding/ External Links: Link Cited by: §4.3, §5. Liu et al. (2026) Z. Liu, Z. Chen, H. Liu, C. Luo, X. Tang, S. Wang, J. Zeng, Z. Dai, Z. Shi, T. Wei, H. Lu, B. Dumoulin, and H. Tong Seeing but not believing: probing the disconnect between visual attention and answer correctness in VLMs. In The Fourteenth International Conference on Learning Representations, External Links: Link Cited by: §5. Liu et al. (2022) Z. Liu, O. Kitouni, N. Nolte, E. J. Michaud, M. Tegmark, and M. Williams Towards understanding grokking: an effective theory of representation learning. In Advances in Neural Information Processing Systems, A. H. Oh, A. Agarwal, D. Belgrave, and K. Cho (Eds.), External Links: Link Cited by: Appendix O. Liu et al. (2023) Z. Liu, E. J. Michaud, and M. Tegmark Omnigrok: grokking beyond algorithmic data. In The Eleventh International Conference on Learning Representations, External Links: Link Cited by: Appendix O. Meng et al. (2022) K. Meng, D. Bau, A. Andonian, and Y. Belinkov Locating and editing factual associations in GPT. Advances in neural information processing systems 35, p. 17359–17372. Cited by: §1, §4.3. Merrill et al. (2023) W. Merrill, N. Tsilivis, and A. Shukla A tale of two circuits: grokking as competition of sparse and dense subnetworks. arXiv preprint arXiv:2303.11873. Cited by: Appendix O. Minegishi et al. (2025) G. Minegishi, J. Feng, H. Furuta, T. Kojima, Y. Iwasawa, and Y. Matsuo Emergent analogical reasoning in Transformers. arXiv preprint. Cited by: Appendix O. Minegishi et al. (2023) G. Minegishi, Y. Iwasawa, and Y. Matsuo Bridging lottery ticket and grokking: is weight norm sufficient to explain delayed generalization?. arXiv preprint arXiv:2310.19470. Cited by: Appendix O. Mitchell et al. (2023) M. Mitchell, A. B. Palmarini, and A. Moskvichev Comparing humans, GPT-4, and GPT-4V on abstraction and reasoning tasks. arXiv preprint arXiv:2311.09247. Cited by: §1, §5. Mondal et al. (2024) S. S. Mondal, J. D. Cohen, and T. W. Webb Slot abstractors: toward scalable abstract visual reasoning. External Links: 2403.03458, Link Cited by: Appendix O. Nanda et al. (2023) N. Nanda, L. Chan, T. Lieberum, J. Smith, and J. Steinhardt Progress measures for grokking via mechanistic interpretability. Cited by: Appendix O, §5. Okawa et al. (2023) M. Okawa, E. S. Lubana, R. Dick, and H. Tanaka Compositional abilities emerge multiplicatively: exploring diffusion models on a synthetic task. In Advances in Neural Information Processing Systems, A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (Eds.), Vol. 36, p. 50173–50195. External Links: Link Cited by: §2. Olah et al. (2020) C. Olah, N. Cammarata, L. Schubert, G. Goh, M. Petrov, and S. Carter Zoom in: an introduction to circuits. Distill. Note: https://distill.pub/2020/circuits/zoom-in External Links: Document Cited by: §5. OpenAI (2026) OpenAI GPT-5.4 Thinking System Card. Technical report OpenAI. External Links: Link Cited by: §2.2. Park et al. (2024) C. F. Park, M. Okawa, A. Lee, H. Tanaka, and E. S. Lubana Emergence of hidden capabilities: exploring learning dynamics in concept space. In Advances in Neural Information Processing Systems, A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang (Eds.), Vol. 37, p. 84698–84729. External Links: Document, Link Cited by: §2. Power et al. (2022) A. Power, Y. Burda, H. Edwards, I. Babuschkin, and V. Misra Grokking: generalization beyond overfitting on small algorithmic datasets. External Links: 2201.02177, Link Cited by: Appendix O. Premack (1983) D. Premack The codes of man and beasts. Behavioral and Brain Sciences 6 (1), p. 125–136. External Links: Document Cited by: §1. Sharkey et al. (2025) L. Sharkey, B. Chughtai, J. Batson, J. Lindsey, J. Wu, L. Bushnaq, N. Goldowsky-Dill, S. Heimersheim, A. Ortega, J. I. Bloom, S. Biderman, A. Garriga-Alonso, A. Conmy, N. Nanda, J. M. Rumbelow, M. Wattenberg, N. Schoots, J. Miller, W. Saunders, E. J. Michaud, S. Casper, M. Tegmark, D. Bau, E. Todd, A. Geiger, M. Geva, J. Hoogland, D. Murfet, and T. McGrath Open problems in mechanistic interpretability. Transactions on Machine Learning Research. Note: Survey Certification External Links: ISSN 2835-8856, Link Cited by: §5. Shivaram et al. (2023) A. Shivaram, R. Shao, N. Simms, S. Hespos, and D. Gentner When do children pass the Relational-Match-To-Sample task?. p. . Cited by: §1. Team (2026) Q. Team Qwen3.5-Omni technical report. arXiv preprint arXiv:2604.15804. Cited by: §2.2. Thagard (1992) P. Thagard Analogy, explanation, and education. Journal of Research in Science Teaching 29 (6), p. 537–544. External Links: Document Cited by: Appendix O. Wang et al. (2023) K. R. Wang, A. Variengien, A. Conmy, B. Shlegeris, and J. Steinhardt Interpretability in the wild: a circuit for indirect object identification in GPT-2 small. External Links: Link Cited by: §1, §4.3, footnote 5. Webb et al. (2023a) T. Webb, K. J. Holyoak, and H. Lu Emergent analogical reasoning in large language models. Nature Human Behaviour 7 (9), p. 1526–1541. Cited by: Appendix O. Webb et al. (2024) T. W. Webb, S. M. Frankland, A. Altabaa, S. Segert, K. Krishnamurthy, D. Campbell, J. Russin, T. Giallanza, Z. Dulberg, R. O’Reilly, J. Lafferty, and J. D. Cohen The relational bottleneck as an inductive bias for efficient abstraction. External Links: 2309.06629, Link Cited by: Appendix O. Webb et al. (2023b) T. W. Webb, S. S. Mondal, and J. D. Cohen Systematic visual reasoning through object-centric relational abstraction. External Links: 2306.02500, Link Cited by: Appendix O. Yang et al. (2025) J. Yang, C. Cui, Y. Zhou, Y. Chen, P. Xia, Y. Wei, T. Yu, Y. Huang, and L. Wang IKOD: mitigating visual attention degradation in large vision-language models. External Links: 2508.03469, Link Cited by: §5. Yiu et al. (2024) E. Yiu, M. Qraitem, A. N. Majhi, C. Wong, Y. Bai, S. Ginosar, A. Gopnik, and K. Saenko KiVA: kid-inspired visual analogies for testing large multimodal models. arXiv preprint arXiv:2407.17773. Cited by: §1, §5. Yue et al. (2024a) X. Yue et al. MMMU-Pro: a more robust multi-discipline multimodal understanding benchmark. In EMNLP, Cited by: §1. Yue et al. (2024b) X. Yue et al. MMMU: a massive multi-discipline multimodal understanding and reasoning benchmark for expert AGI. In CVPR, Cited by: §1. Zhang et al. (2026) H. Zhang, Z. Zhang, M. Wang, Z. Su, Y. Wang, Q. Wang, S. Yuan, E. Nie, X. Duan, Q. Xue, et al. Locate, steer, and improve: a practical survey of actionable mechanistic interpretability in large language models. arXiv preprint arXiv:2601.14004. Cited by: §5. Zhang et al. (2024) Z. Zhang, A. Zhang, M. Li, H. Zhao, G. Karypis, and A. Smola Multimodal chain-of-thought reasoning in language models. External Links: 2302.00923, Link Cited by: §1. Zhu et al. (2025) J. Zhu, W. Wang, Z. Chen, Z. Liu, S. Ye, L. Gu, H. Tian, Y. Duan, W. Su, J. Shao, et al. InternVL3: exploring advanced training and test-time recipes for open-source multimodal models. arXiv preprint arXiv:2504.10479. Cited by: §2.2. Appendix A Trial counts and relation coverage A trial is one RMTS instance: two sample images, a relational candidate, and an object candidate. Each image contains n objects. Within an image, the n objects vary along two attributes—colour and size—and each attribute follows one of five abstract patterns: A (all the same), AAB (last different), ABA (outer pair shared, middle different), ABB (first different), or ABC (all different). A relation is therefore one pair of patterns (Pcolor,Psize)(P_color,P_size), drawn from the resulting 5×5=255× 5=25 relation types. n=3n=3. This is the canonical setting and the basis of every headline number in the paper. Three positions are enough to instantiate all five patterns unambiguously, so a single trial fixes one of the 25 relations. We then enumerate every distinct value assignment that realises each relation— e.g. ABA colour with A=A= red, B=B= blue is a different trial from A=A= blue, B=B= green—and obtain 729 unique trials in total. The trials grade naturally by complexity: 9 of them have Pcolor=Psize=AAAP_color=P_size= A (call this cpx=0cpx=0, both patterns trivial), 144 have one non-trivial attribute (cpx=1cpx=1), and the remaining 576 have both attributes non-trivial (cpx=2cpx=2). Each trial is run twice (relational match in position A and in position B) so the per-model evaluation totals 729×2=1458729× 2=1458 datapoints. Larger n. To probe the effect of visual saliency we re-render the same 25-relation design at n∈2,3,4,5,6,8,10,12,16,20n∈\2,3,4,5,6,8,10,12,16,20\, drawing two value assignments per relation at every n—i.e. 25×2=5025× 2=50 trials per n, and 500 trials in total across the ten n values. At n=3n=3 a trial in this sweep is structurally identical to a canonical cpx=2cpx=2 trial; at n≠3n≠ 3 each pattern is extended deterministically (all-same to n positions, ABA to “outer pair shared, interior different,” and so on). The relation pool and the per-relation trial count are held fixed across n, so the only thing that varies along the n-axis is the number of objects per scene—any change in the relational rate therefore reflects scene composition rather than a change in sample size. Appendix B Model details Decoding. All evaluations use greedy decoding (temperature=0.0, do_sample=False) with max_tokens=4096 (no-CoT and CoT). All Hugging Face IDs below are loaded with trust_remote_code=True and run in bfloat16 (vLLM where supported, HF Transformers otherwise). Table 1: Open-source checkpoints. HF identifiers as released by the model authors at the time of evaluation. Tag (used in figures) Hugging Face identifier Qwen3.5 (5 sizes) qwen3.5-0.8b Qwen/Qwen3.5-0.8B qwen3.5-2b Qwen/Qwen3.5-2B qwen3.5-4b Qwen/Qwen3.5-4B qwen3.5-9b Qwen/Qwen3.5-9B qwen3.5-27b Qwen/Qwen3.5-27B Gemma-4 (4 sizes) gemma-4-e2b google/gemma-4-E2B-it gemma-4-e4b google/gemma-4-E4B-it gemma-4-26b google/gemma-4-26B-A4B-it gemma-4-31b google/gemma-4-31B-it InternVL3 (5 sizes) internvl3-1b OpenGVLab/InternVL3-1B-Instruct internvl3-2b OpenGVLab/InternVL3-2B-Instruct internvl3-8b OpenGVLab/InternVL3-8B-Instruct internvl3-14b OpenGVLab/InternVL3-14B-Instruct internvl3-38b OpenGVLab/InternVL3-38B-Instruct Qwen2.5-VL (used only in Section M.2) qwen2.5-vl-7b Qwen/Qwen2.5-VL-7B-Instruct API checkpoints. The API model identifiers below are used verbatim with the official provider SDKs. Table 2: API checkpoints. Tag Provider Model identifier gpt-5.4 OpenAI gpt-5.4 gpt-5.4-mini OpenAI gpt-5.4-mini claude-opus-4.6 Anthropic claude-opus-4-6 claude-sonnet-4.6 Anthropic claude-sonnet-4-6 claude-haiku-4.5 Anthropic claude-haiku-4-5-20251001 gemini-3.1-pro Google gemini-3.1-pro-preview gemini-3.1-flash Google gemini-3.1-flash-lite-preview Compute. All open-source experiments were run with NVIDIA H200 nodes (one GPU per job, 80–320 GB host RAM), with the runtime stack containerised in a Singularity image (PyTorch 2.4, Transformers 4.45, vLLM 0.19.1). API-model experiments require no local GPU. Appendix C Prompt templates No-CoT prompt. The model is explicitly instructed to suppress reasoning and emit only the final answer line. ⬇ I will teach you a new concept called ’X’ using examples. The first image is a X. The second image is also a X. Now look at the third image (Image A) and the fourth image (Image B). Which one is a X? The answer is equally likely to be Image A or Image B. Do NOT default to A - examine both images carefully. Respond with ONLY one of these two lines (no reasoning, no explanation): Final answer: Image A Final answer: Image B CoT prompt. The model is invited to reason step by step before committing to an answer. ⬇ I will teach you a new concept called ’X’ using examples. The first image is a X. The second image is also a X. Now look at the third image (Image A) and the fourth image (Image B). Which one is a X? The answer is equally likely to be Image A or Image B. Do NOT default to A - examine both images carefully. Let’s think step by step. After your reasoning, conclude with exactly one of these two lines: Final answer: Image A Final answer: Image B Appendix D Examples of RMTS stimuli Figure 9: Canonical stimulus examples (n=3n=3). Figure 10: n-axis stimulus examples. Same column layout as Figure 9, with the number of objects per scene rising from n=2n=2 (top) to n=5n=5 (bottom); n=3n=3 is omitted because the canonical Figure 9 already shows it. Figure 11: Per-object noise on a single cpx-2 trial (sample 1 only). (a) color noise sweep at σs=0 _s=0: at σc≥40 _c\!≥\!40 the categorical color identity becomes visibly ambiguous; at σc=80 _c\!=\!80 the middle “green” object is closer to lime/yellow. (b) size noise sweep at σc=0 _c=0: even at σs=8 _s\!=\!8 the small/medium/large categorical ordering survives, because the categorical sizes are 30/55/85 px and thus 25 px apart. This geometric difference is the simplest explanation for the color-vs-size asymmetry reported in Section 3.4. Appendix E Concrete CoT outputs Figure 12: Example RMTS trial used to elicit the chain-of-thought trace shown below. The two top images are the samples sharing the relation R∗R^*; the bottom two are the candidate test scenes (relational match and object match). Figure 12shows an example RMTS trial together with the corresponding chain-of-thought trace produced by Claude-Opus-4.6 on this trial. Inspecting the trace, we observe that the model explicitly extracts the pattern shared between the two sample scenes before evaluating the candidates—confirming the qualitative claim made in the main text that CoT prompting surfaces the abstract relation rather than surface-level object features. “I need to figure out what ‘X’ means from the examples. Example 1: Three red circles arranged in a horizontal row. Example 2: Three blue triangles arranged in a horizontal row. The common pattern: both examples show three identical shapes arranged in a row. The shapes are all the same…” [selects relational match] Appendix F Color-vs-size attribute asymmetry The relations in our set are defined over two attributes (color and size). Stratifying by which attribute carries the relation (Table 3) reveals a non-trivial asymmetry under no-CoT. Most open-source families lean color: Gemma-4 (+14.6+14.6 p), InternVL3 (+5.6+5.6 p), Qwen3.5 (+2.9+2.9 p). The distilled tiers from Anthropic and OpenAI lean the opposite way: Frontier (Anthropic) −36.1-36.1 p, Frontier (OpenAI distilled) −17.4-17.4 p. The asymmetry is large enough that benchmarks dominated by a single attribute could systematically over- or under-state relational competence; we recommend evaluating on at least two orthogonal attributes whenever possible. Chain-of-thought largely erases this asymmetry (right half of Table 3): the Anthropic distilled tier swings from −36.1-36.1 p (no-CoT) to 0.00.0 p under CoT, OpenAI distilled from −17.4-17.4 to +4.5+4.5, and Gemma-4 from +14.6+14.6 to +0.8+0.8. We read this as a side-effect of the CoT lift documented in Section 3: once CoT pushes every non-saturated model close to ceiling on both attribute strata, the residual asymmetry collapses. A practical implication: benchmarks that report CoT-only relational rates substantially understate the attribute-bias effects we identify under no-CoT. Table 3: Color-only vs. size-only relational rate (%), aggregated by family. “Color-only” = trials with non-trivial color pattern but uniform size; “size-only” is symmetric. For each family the rate is averaged across non-saturated members only (models at 100100% on both attributes carry no information about the asymmetry); Frontier (Google) is at ceiling on both for every member under no-CoT, so its Δ collapses to “—”. Under no-CoT, families differ in the direction of the asymmetry (Anthropic / OpenAI distilled tiers lean toward size; Gemma-4 leans toward color); under CoT every family’s |Δ|| | collapses to ≤7≤ 7 p, indicating that explicit reasoning closes the attribute asymmetry. No-CoT CoT Family Color (%) Size (%) Δ Color (%) Size (%) Δ Frontier (OpenAI) 21.5 38.9 −17.4-17.4 92.7 88.2 +4.5+4.5 Frontier (Anthropic) 63.2 99.3 −36.1-36.1 98.6 98.6 +0.0+0.0 Frontier (Google) 100.0 100.0 — 83.1 76.4 +6.8+6.8 Qwen3.5 72.9 70.0 +2.9+2.9 82.8 84.1 −1.4-1.4 Gemma-4 68.5 53.9 +14.6+14.6 75.7 74.9 +0.8+0.8 InternVL3 51.0 45.5 +5.6+5.6 65.3 66.2 −0.9-0.9 Appendix G n-axis on frontier API models (companion to 3(c)) The main-text n-axis figure (3(c)) shows the open-source families to keep the panel readable. The corresponding curves for the frontier API models we evaluated on the n-axis are shown in Figure 13: GPT-5.4 saturates at ceiling for every n, while GPT-5.4-mini exhibits the same competition the rest of the paper documents—chance at n=2n=2 (≈0.48≈ 0.48), object-biased at n=3n=3 (≈0.23≈ 0.23, matching its 3232% rate on the canonical 729-trial set), and rising back across the chance line as n grows (≈0.67≈ 0.67 at n=20n=20). The two API curves and the OSS curves in 3(c) are now evaluated on the same v2-aligned harder n-axis stimuli (variable shape per scene + size variation + sample-pool object match), so the n=3n=3 point reproduces the canonical 729-trial rate up to relation-pool sampling. The Anthropic and full Gemini families are omitted from this companion because of API-credit interruptions in the n-axis sweep window; the canonical 729-trial set in Section 3 and the noise sweep in Section 3.4 cover them. Figure 13: n-axis on frontier API models (no-CoT, v2v2-aligned hard n-axis stimuli). GPT-5.4 saturates at 1.01.0 at every n. GPT-5.4-mini drops to ≈0.23≈ 0.23 at n=3n=3—reproducing the 3232% canonical rate up to relation-pool sampling—and rises back as n grows. 95%95\% CI bands. Appendix H Shape-coherence ablation The confound. The trial structure introduced in Section 2.1 contains a subtle shape-level asymmetry. By definition (Definition 3), the object match mixes shapes from ≥2≥\!2 different samples, while the relational match uses a single fresh shape. The two samples are themselves single-shape (Definition 2). Concretely: the samples are shape-coherent within themselves, the relational match is shape-coherent within itself, but the object match is shape-mixed. A model could in principle exploit this asymmetry alone—“the candidate that is shape-coherent within itself, like the samples, is the answer”—and arrive at the relational match without ever inferring the colour/size relation that defines the trial. The ablation. To test whether the relational behaviour reported in Section 3 is driven by this shape-coherence cue or by genuine relation extraction, we generated an ablated stimulus set in which the object match is also rebuilt as a single-shape scene, drawn at random per trial from one of the two sample shapes. The relational match is unchanged (it still uses the brand-new third shape). Under this ablation, both candidates are shape-coherent within themselves, so shape coherence no longer disambiguates them; only the colour/size relation does. We refer to this set as shape-controlled and to the original set used throughout Section 3 as canonical. Reading the result. If the relational rate on the shape-controlled set matches the rate on the canonical set, the original effect cannot be driven by shape coherence. If instead the rate drops markedly, shape coherence was doing meaningful work and the relational claim should be tempered. Result. Table 4 summarises the per-model relational rate on the canonical and shape-controlled sets (no-CoT). All ceiling-bound models are essentially unaffected: GPT-5.4 (100.0→100.0100.0→ 100.0), Gemini-3.1-Pro (100.0→100.0100.0→ 100.0), Gemma-4-31B (100.0→100.0100.0→ 100.0), Gemma-4-26B (99.7→99.799.7→ 99.7, Δ=+0.1 =+0.1 p), and InternVL3-38B (100.0→98.1100.0→ 98.1, Δ=−1.9 =-1.9 p) reproduce their canonical-set rates within noise. Qwen3.5-9B, the only non-ceiling model in this comparison, is similarly unaffected (89.3→89.689.3→ 89.6, Δ=+0.3 =+0.3 p). All deltas are within ±2± 2 p. The relational claim is therefore robust to the shape-coherence confound across the six tested checkpoints. Table 4: Shape-coherence ablation: relational rate (no-CoT) on the canonical vs. shape-controlled stimulus set. On the canonical set, the object match mixes sample shapes; on the shape-controlled set, both candidates are single-shape, so shape coherence cannot disambiguate them. A small |Δ|| | means the relational claim is robust to the confound. All six tested checkpoints fall within |Δ|≤2| |\!≤\!2 p. Model Canonical Shape-controlled Δ GPT-5.4 100.0 100.0 +0.0+0.0 Gemini-3.1-Pro 100.0 100.0 +0.0+0.0 Gemma-4-31B 100.0 100.0 +0.0+0.0 InternVL3-38B 100.0 98.1 −1.9-1.9 Gemma-4-26B 99.7 99.7 +0.1+0.1 Qwen3.5-9B 89.3 89.6 +0.3+0.3 Appendix I Noise sweep: additional models (API and small Qwen) The main-text Figure 4 restricts to the four open-source checkpoints that span the clean-rate range (Gemma-4-26B/E4B, InternVL3-38B/8B) so the panel reads cleanly. For completeness we also ran the sweep on two frontier API models (Gemini-3.1-Flash, GPT-5.4-mini) and the smallest non-A-biased Qwen3.5 (2B); these companion curves are shown in Figure 14. Their qualitative behaviour matches the main figure: clean rate predicts noise robustness, and color perturbation drives a substantially larger drop than size perturbation. Which attribute matters. Beyond the overall magnitude, which attribute the noise sensitivity attaches to also varies across the models in Figure 4. InternVL3-38B, for instance, is almost entirely indifferent to colour noise and to size noise up to σs=4 _s\!=\!4, only dropping appreciably at σs=8 _s\!=\!8; its relational rate stays above 9898% across nearly the entire sweep. Other models display the opposite asymmetry, where colour perturbations cost more than comparable size perturbations (cf. Gemma-4-E4B, where the σc _c slope is sharper than the σs _s separation). Figure 14: Noise sweep companions: API frontier models and Qwen3.5-2B. Top: relational rate vs. σc _c (lines = σs _s). Bottom: same data plotted vs. σs _s (lines = σc _c). SE bands across three seeds. Same σ-grid as Figure 4. Appendix J CMA result on Gemma-4-31B Figure 15: Per-head ablation effect on Gemma-4-31B (60 layers, 32/64 heads alternating). Same encoding as Figure 7: bubble size =|Δℓ|=| | capped at the 95th percentile; color == signed Δℓ (blue rel, red object). The dominant rel head sits at L58 H16 (Δℓ=−11.7 =-11.7, an order of magnitude larger than the 26B counterpart and the largest single-head effect in our experiments); dominant object heads cluster at L30–L31. The depth at which the rel head appears scales with total depth (∼95% 95\% in both 26B and 31B), so the two-stage organisation is preserved across the family. The wide gap at heads 32–63 on alternating rows again reflects Gemma-4’s local/global attention pattern. Appendix K Per-model relational rates Table 5reports the per-model no-CoT and CoT rates underlying Figures 3(a) and 3(b). Notable rows: Claude-Haiku-4.5 (+57.4+57.4 p; →99.942.5\!→\!99.9%) and GPT-5.4-mini (+51.2+51.2 p; →83.432.2\!→\!83.4%) are the two largest distilled-tier gains; Gemma-4-E2B (+57.3+57.3 p) and Gemma-4-E4B (+23.6+23.6 p) show consistent within-family gains for Gemma; Qwen3.5-9B (−4.0-4.0 p), Qwen3.5-4B (−2.4-2.4 p), and InternVL3-2B (−3.3-3.3 p) show a small negative CoT effect at thin capability. Table 5: No-CoT vs. CoT relational rate, full per-model breakdown. Numbers in percent over parsed (relational++object) trials, excluding parse-failures and unclear responses; sign of Δ tells whether CoT helps (positive) or hurts (negative). Several Anthropic CoT entries are computed over the parsed subset that survived API failures during the run. Model no-CoT (%) CoT (%) Δ GPT-5.4 100.0 99.8 −0.2-0.2 GPT-5.4-mini 32.2 83.4 +51.2+51.2 Claude-Opus-4.6 100.0 100.0 +0.0+0.0 Claude-Sonnet-4.6 100.0 100.0 +0.0+0.0 Claude-Haiku-4.5 42.5 99.9 +57.4+57.4 Gemini-3.1-Pro 100.0 100.0 +0.0+0.0 Gemini-3.1-Flash 99.7 99.9 +0.2+0.2 Qwen3.5-0.8B 50.0 49.5 −0.5-0.5 Qwen3.5-2B 47.4 52.2 +4.8+4.8 Qwen3.5-4B 77.3 74.9 −2.4-2.4 Qwen3.5-9B 89.3 85.3 −4.0-4.0 Qwen3.5-27B 85.9 86.6 +0.7+0.7 Gemma-4-E2B 26.3 83.6 +57.3+57.3 Gemma-4-E4B 74.3 97.9 +23.6+23.6 Gemma-4-26B 99.7 100.0 +0.3+0.3 Gemma-4-31B 100.0 100.0 +0.0+0.0 InternVL3-1B 50.1 48.5 −1.6-1.6 InternVL3-2B 49.5 46.1 −3.3-3.3 InternVL3-8B 28.9 51.7 +22.8+22.8 InternVL3-14B 54.1 90.6 +36.5+36.5 InternVL3-38B 100.0 98.8 −1.2-1.2 Appendix L ARC-AGI-1 task detail Figure 16: One ARC-AGI-1 training task as the model sees it. Two of the four train pairs (input → output) and the test input are shown to the model as rendered colour images; the rule here is grid transposition (rows become columns). Figure 8tests whether the relational heads we identified on RMTS make a measurable contribution on a fully out-of-domain abstract-reasoning benchmark. This appendix gives the exact stimulus, prompt, evaluator, and head-edit recipe. L.1 Dataset and task subset We use the public ARC-AGI-1 corpus (Chollet, 2019) (training split; 400400 tasks). Each task contains 22–55 train input/output pairs and one test input; the model must emit the test output grid. Two filters are applied before evaluation: • Image budget: the prompt contains 2K+12K+1 images for K train pairs. We cap at 1414 images per prompt (--limit-m-per-prompt 14), which covers K≤6K≤6 pairs. • Task cap: after the image filter we take the first LIMIT tasks in canonical filename order (6666 for Gemma-4-26B, 8080 for Gemma-4-31B; the difference is purely a job-budget choice and has no effect on the comparison since each condition uses the same n within a model). Identical task subsets are evaluated under every head-edit condition for a given model, so all condition deltas are paired. L.2 Stimulus rendering Each H×WH×W grid is rendered as an RGB image with one 32×3232×32\,px cell per integer (00–99) using the canonical ARC palette (=0\!=\!black, =1\!=\!blue, =2\!=\!red, =3\!=\!green, =4\!=\!yellow, =5\!=\!grey, =6\!=\!pink, =7\!=\!orange, =8\!=\!light-blue, =9\!=\!maroon), with a 11\,px dark border around each cell. The model never sees the integer matrix—it sees the rendered colour image, the same modality used throughout the rest of the paper. L.3 Prompt format A single user turn carries an instruction header, the palette key, and an interleaved sequence of (image, shape annotation) blocks for each train pair followed by the test input. Verbatim: You are given input/output examples that follow a hidden rule. Each example is shown as a coloured grid; each cell is one integer 0–9. [palette line]. Infer the rule from the examples and apply it to the final test input. Be concise and decisive: do NOT second-guess yourself or restate the examples. After your reasoning, output ONLY the final grid for the test input between the literal markers <grid> and </grid>: one row per line, integers 0–9 separated by spaces, no row numbers, no colour names, no extra text. Example 1 input: [image] (H1×W1H_1×W_1 grid) Example 1 output: [image] (H1′×W1′H _1×W _1 grid) … Test input: [image] (Ht×WtH_t×W_t grid). Produce the test output grid …between <grid>…</grid>. We do not use chain-of-thought (--cot flag off) for the intervention experiments, because (a) the rel-3 ablation we want to detect is a representational effect that should not require verbal scratchpad to manifest, and (b) CoT lengthens the response, increasing the chance that any head edit becomes visible only via reasoning errors rather than directly via the predicted grid. L.4 Decoding and parsing We run the unquantised checkpoint via HuggingFace Transformers (no vLLM in the intervention path, since vLLM does not expose pre-o_proj hooks). Decoding is greedy (do_sample=False, no temperature) with max_new_tokens=4096. The parser: • First tries the explicit <grid>…</grid> block. • Falls back to the last Output: / Answer: block. • For each line, extracts integer tokens; if none are found, tries the colour-name fallback (e.g., “black blue red” → [0,1,2][0,1,2], including the two-word phrases “light blue” and “dark red”). L.5 Metric: cell-wise accuracy ARC-AGI-1’s official metric is exact-match (whole grid correct or incorrect), but exact-match is too lossy at the capability tier of Gemma-4-26B (clean exact-match ≈0.23≈ 0.23): every condition crashes to near zero and the rel-vs-random comparison becomes statistically uninformative. We therefore report cell-wise accuracy—the fraction of cells in the ground-truth grid matched by the parsed grid, counted as 00 if the parsed shape does not match. This is a natural relaxation that preserves ordering on hard tasks (a one-cell-wrong answer scores HW−1HW HW-1HW, not 00) while still penalising shape errors. L.6 Head-edit recipe We use the same per-head intervention machinery as our RMTS CMA scan (Section 4.3): a forward-pre-hook on each layer’s o_proj reshapes its input to (B,T,nh,dh)(B,T,n_h,d_h) and multiplies the selected head slots by the chosen scale. In every condition reported in the main text the scale is 00 (full ablation). • Gemma-4-26B (30 layers, 16/32 heads): rel-3 zero: L28 H12, L28 H7, L19 H2. obj-3 zero: L12 H11, L14 H1, L15 H5. Random-3 (three seeds): each is three (layer, head) pairs sampled reproducibly from the layer-by-head grid excluding the top-5 rel and top-5 object heads, so each random condition zeros out the same number of head slots as the rel/obj conditions. • Gemma-4-31B (60 layers, 32/64 heads): rel-3 zero: L58 H16, L58 H19, L49 H16 (L58 H16 has Δℓ=−11.7 =-11.7 in the CMA scan, four-to-five times the runner-up). obj-3 zero: L30 H9, L30 H14, L31 H28. Random-3 sampled analogously. The random-3 baseline controls for the trivial “editing any head hurts” explanation: if rel-3 ablation drops cell-wise accuracy beyond what random-3 ablation does, the rel circuit is contributing causal capacity to ARC, not just disturbing the residual stream. Appendix M Additional analyses This appendix collects analyses that probe the generality of the behavioural and mechanistic findings: a structurally different relation (Section M.1), causal mediation on further model families (Section M.2), the interaction between CoT and the relational heads (Section M.3), the internal effect of stimulus noise (Section M.4), and the robustness of the RSA target (Section M.5). M.1 A structurally different relation: numerical equality Design. All relations in the main text are ordinal patterns over colour and size. To test whether the findings depend on this relation type, we construct trials in which the two samples share their object count and nothing else. With three objects, for example, Sample 1 shows three red squares and Sample 2 three blue triangles, so that count is their only commonality. The relational match shows three green circles—the right count, in a colour and shape used by neither sample—while the object match shows five red squares, Sample 1’s objects copied verbatim with the wrong count. A model that matches appearance picks the red squares; only counting reaches the green circles. As in the main experiments, every trial is presented in both answer orderings, so any positional preference cancels (chance =50=50%), and no CoT is used. Scale effect on open-source models. We generate 6464 trials (128128 judgements per model) and evaluate two checkpoints from each open-source family (Table 7). In every family the relational rate rises with model size, by 4646, 3030 and 7373 p respectively, so the scale effect of Section 3.2 is robust to changing the relation itself. Frontier models come off the ceiling. The three frontier models that saturate the ordinal relations are no longer saturated on numerical equality (Table 7). Together with the noise sweep, in which checkpoints that saturate the clean stimuli also fall off the ceiling (Gemma-4-26B 100→92100→ 92%, InternVL3-38B 100→92100→ 92%, Gemini-3.1-Flash 99→8699→ 86% under the strongest colour and size jitter; Figure 4 and Appendix I), this indicates that the ceiling on the canonical set partly reflects its simplicity for the strongest models, and that harder relations discriminate among them. Table 6: Numerical equality, open-source models. Relational rate (%, no-CoT), small → large checkpoint per family. Family (small → large) Small Large Qwen3.5 (2B → 9B) 47.7 93.8 Gemma-4 (E4B → 26B) 26.6 57.0 InternVL3 (8B → 38B) 25.8 98.4 Table 7: Numerical equality, frontier models. Relational rate (%), averaged over n=2n=2–55 and both orderings. Model Rel. rate Gemini-3.1-Pro 99.0 Claude-Opus-4.6 93.0 GPT-5.4 82.5 M.2 Causal mediation on further model families To test whether the head-level picture of Section 4.3 is specific to Gemma-4-26B, we run the same per-head activation patching, unchanged, on five further checkpoints covering Gemma-4, InternVL3 and Qwen2.5-VL (3030 trials each, both answer orderings). Table 8 reports the strongest relational head per checkpoint. In every family a small number of heads carry most of the relational effect, and in every case the strongest one sits in the second half of the network, as in the Gemma-4-26B result of the main text. Table 8: Strongest relational head per checkpoint. Effect: change in the log-odds of the relational answer when that head alone is patched (negative = relational-supporting, as in Figure 7). Depth: layer index divided by the total number of layers. Model Top relational (layer, head) Effect Depth Gemma-4-26B (main text) (28, 12) −1.02-1.02 0.97 Gemma-4-E4B (21, 5) −1.27-1.27 0.51 InternVL3-38B (55, 23) −1.36-1.36 0.87 InternVL3-14B (28, 29) −0.82-0.82 0.60 InternVL3-8B (17, 27) −0.44-0.44 0.63 Qwen2.5-VL-7B (20, 24) −0.32-0.32 0.74 M.3 CoT bypasses the relational heads CoT lifts weaker models toward the relational match (Section 3.1). Does it do so by strengthening the relational circuit, or through a different route? On Gemma-4-26B we zero out the relational heads identified by CMA (the single strongest, L28 H12, or the top three, L28 H12, L28 H7 and L19 H2; Appendix L) and run the identical trials twice, once answering directly and once with the CoT prompt of Appendix C. We use 3030 trials from the most demanding relation subset in both answer orderings (6060 judgements per condition). Table 9 shows that removing the three strongest relational heads reduces direct answers to chance, which also shows that these heads are necessary rather than merely correlated with the behaviour. With CoT, however, the same ablated model is back at 100100%. The CoT gain therefore does not run through the circuit we identified. Table 9: Relational rate (%) of Gemma-4-26B under head ablation, without and with CoT. Heads zeroed Direct answer With CoT None (intact model) 100.0 100.0 Strongest relational head 86.7 100.0 Three strongest relational heads 50.0 100.0 M.4 Stimulus noise and internal representations We ask whether stimulus noise, which lowers the relational rate (Section 3.4), acts on the same internal quantity as the number of objects (Section 4.2). We use Gemma-4-E4B, the checkpoint with the largest behavioural noise effect, and apply independent Gaussian jitter to every object’s radius, keeping each object inside its original size category so that the categorical description of the scene is unchanged and only the rendered pixels move. We use three noise levels, from no jitter to the strongest jitter of the behavioural sweep, with three noise seeds and 4040 trials per seed (120120 trials per condition), and measure the relational selectivity Δ-RSA=Rel-RSA−Obj-RSA -RSA=Rel-RSA-Obj-RSA used throughout Section 4. Averaged over the late third of the layers (layers 29–42), Δ-RSA -RSA falls from +0.24+0.24 without jitter to +0.15+0.15 at the strongest jitter (s.e. 0.060.06), while the relational rate on the same trials falls from 7878% to 6969%. Noise thus acts on the same internal quantity as n, in the opposite direction, and the internal decline tracks the behavioural one. M.5 Robustness of the relational RSA target The relational target TrelT_rel (Section 4.1) leaves the sample–sample entry at 00. We also evaluate the alternative in which all relation-sharing images, including the two samples, are clustered together. For each model and each target we compute the layer-wise curve Rel-RSA(ℓ)−Obj-RSA(ℓ)Rel-RSA( )-Obj-RSA( ) and take the Pearson correlation across layers between the two curves. The two definitions agree at r=0.91r=0.91–1.001.00 for all fourteen open checkpoints across the three families, so every layer-wise conclusion in the paper holds under either target. Appendix N Limitations Shape coherence as a residual cue. Within a trial, every sample scene uses one shape, the relational match introduces a fresh shape, and the object match by construction (Definition 3) mixes ≥2≥\!2 sample shapes; this enforces shape-disjointness with the samples but also means the relational candidate is shape-coherent while the object candidate is shape-mixed. A model could therefore exploit a “which candidate is shape-coherent like the samples” heuristic without inferring the colour/size relation. We address this with a candidate-level ablation in Appendix H: the object match is rebuilt as single-shape (drawn from a sample shape) so both candidates are shape-coherent within themselves, and only the colour/size relation can disambiguate them. Across the six tested checkpoints the relational rate moves by at most |Δ|≤2| |\!≤\!2 p, indicating that the headline effect is not driven by shape coherence. Stimulus distribution. Our formal geometric stimuli isolate the relational step at the cost of a narrow distribution; we mitigate this by sweeping n=2n\!=\!2–2020 across the full ×55\!×\!5 pattern grid (3(c)), by adding per-object Gaussian noise (Figure 4), and by testing a structurally different relation, numerical equality (Section M.1). These relations still do not exhaust those VLMs will ultimately need; spatial, causal and semantic relations, as well as naturalistic scenes, are an immediate next step. Ceiling effects. The strongest frontier models are at ceiling on the ordinal colour/size relations. For these closed-source models the stimulus set functions only as a benchmark, and at ceiling it cannot discriminate genuine relational abstraction from simpler strategies; the numerical-equality relation partly addresses this by bringing them off the ceiling (Section M.1). Mechanistic scope. Per-head CMA evidence covers two Gemma-4 checkpoints in full (Sections 4.3 and J) and five further checkpoints from Gemma-4, InternVL3 and Qwen2.5-VL on a smaller trial budget (Section M.2). Our use of circuit is functional (Section 4); we do not characterise how the identified heads compose. Finally, the circuit account covers direct answers only: under CoT the model reaches the relational answer without the relational heads (Section M.3), and the mechanism of that route is left open. Appendix O Extended Related Work Analogical reasoning. The RMTS task is treated in cognitive science as an instance of analogical reasoning (Christie and Gentner, 2013; Gentner and Hoyos, 2017), in which a learner transfers an abstract structural correspondence from a source to a target despite surface-level dissimilarity. Analogy has been studied as a candidate explanation for the remarkable sample efficiency of human learning (Thagard, 1992; Gentner and Hoyos, 2017) and for its role in scientific discovery (Leatherdale, 1974; Goel, 1997; Gentner et al., 1997). Studies of analogical reasoning in deep neural networks remain comparatively scarce, but several lines of work are directly relevant. Webb et al. (2023a) compare large language models to humans on a battery of analogical-reasoning tasks and report broadly human-level behavioural performance. Campbell et al. (2024) document VLM failures on visual analogy benchmarks, in line with the broader picture that vision-grounded analogical inference remains a weakness of current systems. From a more mechanistic angle, Minegishi et al. (2025) formalise analogy in a category-theoretic framework, design a controlled toy task, and analyse how a Transformer trained from scratch on this task implements analogical mappings. The closest mechanistic study is Lepori et al. (2024), who show that vision transformers decompose a same/different judgement into a perceptual stage followed by a relational stage, close to the early object stage and late relational stage we describe. We differ in three respects: we study generative VLMs on a task where the relational answer must beat a specific object-matching alternative rather than give a binary report; we track that competition across capability tier, scale, CoT, scene size and stimulus noise; and we test the relational stage causally, including on ARC-AGI-1. These prior efforts are largely either purely behavioural (assessing whether a model gets analogy problems right or wrong) or focused on toy networks trained from scratch on synthetic tasks. To our knowledge, the present work is the first to combine (i) a parametrically controlled analogical-reasoning paradigm imported from comparative and developmental psychology, (i) an evaluation across the current frontier of pretrained VLMs (API and open-source families), and (i) a mechanistic dissection that traces how the analogical computation is implemented inside the network at the level of layers and individual attention heads. Relational inductive biases and architectures. A complementary line of work builds architectures with explicit relational inductive biases, aiming for systematic generalisation beyond surface features. Webb et al. (2023b) propose Object-Centric Relational Abstraction (OCRA), combining slot-based object representations with an explicit relational module for abstract visual reasoning. Webb et al. (2024) synthesise and extend this theme via the relational bottleneck, an architectural constraint that encourages abstraction by forcing computation to pass through a relation-focused representation. Mondal et al. (2024) scale these ideas to more complex multi-object settings by combining slot-based perception with Transformer-style abstraction. Competing internal circuits. The two-circuit picture we observe is reminiscent of similar phenomena documented elsewhere in the mechanistic-interpretability literature. Most strikingly, the grokking phenomenon (Liu et al., 2023; Power et al., 2022) has been explained as a competition between a memorisation circuit and a generalisation circuit, with the latter eventually dominating after extended training (Nanda et al., 2023; Liu et al., 2022; Minegishi et al., 2023; Merrill et al., 2023). Our findings suggest that the same competitive logic surfaces at inference time in pretrained VLMs: a surface-feature (object) circuit and an abstract (relational) circuit are both present, and the model’s behavioural output reflects which circuit dominates at the prediction position. We conjecture that large pretrained models harbour many such circuits implementing alternative solutions to the same task, and that the model’s response is governed by whichever circuit exerts the largest causal influence on the output logits.