Paper deep dive
The Geometry of Contextual Relations: Language Models Address Facts by Order of Mention
Yufa Zhou
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 95%
Last extracted: 10/3/2026, 1:44:41 AM
Summary
This paper investigates how Large Language Models (LLMs) represent contextual facts, proposing the 'ordinal addressing hypothesis.' The authors demonstrate that LLMs address facts based on their order of mention in the context rather than the specific names or entities involved. By analyzing hidden states, they identify an 'ordinal vector' that steers the model's attention to specific facts by their position in the list. This mechanism is found to be low-rank, steerable, and emergent across various models (Qwen, Gemma, Llama) and scales.
Entities (7)
Relation Signals (6)
Ordinal Addressing Hypothesis → proposesmechanism → Order of Mention
confidence 95% · We find that LLMs represent the fact a question asks about by its order of mention, not by the name the question contains.
Ordinal Vector → steersattentionto → Fact Address
confidence 94% · Averaged over many lists, this change is a steering vector, which we call the ordinal vector. It points to a fact by its order of mention...
Ordinal Vector → islowrank → True
confidence 93% · they span a low-rank subspace in which the first-mentioned fact is the easiest to reach
Qwen2.5-7B → exhibitsproperty → Ordinal Addressing
confidence 92% · Across Qwen, Gemma, and Llama, fact addresses are (1) ordered by mention... (2) steerable...
Gemma 3 4B → exhibitsproperty → Ordinal Addressing
confidence 92% · Across Qwen, Gemma, and Llama, fact addresses are... emergent... hold from 1.5B to 32B parameters
LLaMA-3.1-8B → exhibitsproperty → Ordinal Addressing
confidence 92% · Across Qwen, Gemma, and Llama, fact addresses are... low-rank... shared in late-middle layers
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Human reasoning depends on how objects are related within propositions. \textit{How do relations organize the language representations of contextual contents?} We give an LLM a list of facts in its context (e.g., \emph{Alice eats an apple. Bob eats a pear.}) and measure how its hidden state changes when the question switches from what Alice eats to what Bob eats. Averaged over many lists, this change is a steering vector, which we call the \emph{ordinal vector}. It points to a fact by its \emph{order of mention}, the order in which the facts were stated in the context. We find that LLMs represent the fact a question asks about by its order of mention, not by the name the question contains. We state this as the \textit{ordinal addressing hypothesis}: each order of mention has a \emph{fact address} in the model's state, shared by all contexts, and a question moves the state to the fact address of the fact it asks about, while the context supplies what that fact says. Across Qwen, Gemma, and Llama, fact addresses are (1) \emph{ordered by mention}: query states are organized by the order of facts, not of names, even when one fact has multiple subjects; (2) \emph{steerable}: added to a question about the first fact of a new list, the ordinal vector makes the model answer with the second fact of that list; (3) \emph{low-rank}: they span a low-rank subspace in which the first-mentioned fact is the easiest to reach, surprisingly similar to human recall; and (4) \emph{emergent}: they are shared in late-middle layers, hold from 1.5B to 32B parameters, and form early in pretraining. Language models reach a stated fact by where it was mentioned, deepening our understanding of LLM reasoning.
Tags
Links
- Source: https://arxiv.org/abs/2610.00910v1
- Canonical: https://arxiv.org/abs/2610.00910v1
Trouble viewing inline? Open PDF directly →
Full Text
111,319 characters extracted from source content.
Expand or collapse full text
The Geometry of Contextual Relations: Language Models Address Facts by Order of Mention Yufa Zhou Duke University yufa.zhou@duke.edu Human reasoning depends on how objects are related within propositions.How do relations organize the language representations of contextual contents?We give an LLM a list of facts in its context (e.g.,Alice eats an apple. Bob eats a pear.) and measure how its hidden state changes when the question switches from what Alice eats to what Bob eats. Averaged over many lists, this change is a steering vector, which we call theordinal vector. It points to a fact by itsorder of mention, the order in which the facts were stated in the context. We find that LLMs represent the fact a question asks about by its order of mention, not by the name the question contains. We state this as theordinal addressing hypothesis: each order of mention has afact addressin the model’s state, shared by all contexts, and a question moves the state to the fact address of the fact it asks about, while the context supplies what that fact says. Across Qwen, Gemma, and Llama, fact addresses are (1)ordered by mention: query states are organized by the order of facts, not of names, even when one fact has multiple subjects; (2)steerable: added to a question about the first fact of a new list, the ordinal vector makes the model answer with the second fact of that list; (3)low-rank: they span a low-rank subspace in which the first-mentioned fact is the easiest to reach, surprisingly similar to human recall; and (4)emergent: they are shared in late-middle layers, hold from 1.5B to 32B parameters, and form early in pretraining. Language models reach a stated fact by where it was mentioned, deepening our understanding of LLM reasoning. Code is available athttps://github.com/MasterZhou1/order-of-mention. one context a Latently, the model first finds which fact the question is about, then reads that fact real prompt (facts and question) Alice eats an apple. Bob eats a pear. Carol eats a plum. What does Bob eat? LLM latent, not a written chain of thought step 1the 2nd fact step 2read it: “Bob eats a pear.” find “Bob”not by the name answers pear(fact 2) 1 2 3 123 calibration contexts a completely new context b (1) Measure the change between two questions calibration context (eats, other names) 1Helen Rhodes eats the chana masala. 2Gavin Spencer eats the roast pork. (+2 more facts) + What does Helen Rhodes eat? asks about fact 1 h 1 + What does Gavin Spencer eat? asks about fact 2 h 2 LLM, block 20 Δ = h 2 − h 1 ordinal vector v = average Δ over contexts (2) Add v once in a new context new relation, new contents, new names 1Duncan Hardy owns a sloth. 2Catherine Eaton owns a penguin. (+2 more facts) Which animal does Duncan Hardy own? with v answers penguin (fact 2) without v answers sloth(fact 1) add v at block 20 v was measured on eats with other names; here the relation, contents, and names are all new. Figure1:Language models address stated facts by order of mention.(a) Asked what Bob eats, the model’s state after the question points to the second fact, not to the name, and the answer is then read from that fact. (b) A real Qwen2.5-7B run: (1) the block-20 query change∆between questions about fact 1 and fact 2, averaged over calibration contexts, is the ordinal vector푣; (2) added once in a new context,푣moves the answer to fact 2’s animal. arXiv:2610.00910v1 [cs.CL] 1 Oct 2026 The Geometry of Contextual Relations: Language Models Address Facts by Order of Mention 1. Introduction Human reasoning draws conclusions from relations among objects and events [13]. The same inference works for any names, yet its conclusion depends on who plays which role: if Alice gives Bob a book, Bob has it; if Bob gives Alice a book, Alice has it. Combining facts therefore requires preserving each element’s semantic role, who does what to whom, even as the contents change [31]. Large language models (LLMs) face this problem when reasoning from contextual facts [7]. InAlice loves Bob. Carol loves David., the relationloveshas twosemantic roles, the person who loves and the person loved [11], and abindingspecifies who fills each role: Bob and David share a role, but whom Alice loves depends on the binding. Semantic structure shapes language-model representations: word-vector analogies and concept geometry reveal directions shared across related meanings [22,27,28], and contexts trace recurring trajectories as logically structured information accumulates [35]. Relations add a particular challenge: the same relational form recurs across examples, while particular facts depend on which objects fill each role. This leads to our central question: How do relations organize the language representations of contextual contents? We study it where a question meets the facts. Given the factsAlice eats the appleandBob eats the pear, a listener asked what Bob eats can find the fact by its description, the one about Bob, or by its order of mention, the second one, asthe formerandthe latterdo. People track the order in which facts are mentioned, and the first-mentioned participant is the easiest to retrieve [10,14]. Which does a language model use? We change the question from what Alice eats to what Bob eats and measure how the hidden state changes before the model answers. Averaged over contexts, this change is a steering vector, which we call theordinal vector(Figure 1). We find that language models address stated facts by order of mention. We state this as a hypothesis (Section 3): each order of mention has afact address, shared by all contexts; a question moves the state to the address of the fact it asks about, and the context supplies what is stated there, the fact’sbinding identity. Its consequences hold in Qwen, Gemma, and Llama: (1) Questions are organized by order of mention.Query changes keep their direction on new names and reverse when the facts are reversed, in fourteen LLMs with base as well as instruction-tuned weights. Query states group by the queried fact’s order of mention, not by the named person, and follow the order of facts, not of names, when one fact has two subjects (Sections 3 and 4). (2) The fact address transfers across contexts.Added once to a context with new names, contents, and relation, the ordinal vector selects the designated fact among four, and that context supplies the content, also in short paragraphs and from 1.5B to 32B parameters (Section 5; Table 5). The vector that transfers follows order of mention, not names, relation, or token distance (Section 5). (3) Fact addresses are low-rank.They span a low-rank subspace where ordinal vectors compose; later addresses crowd together and the first is easiest to reach (Section 6). (4) The fact address is shared in a depth window and learned early in pretraining.In late-middle blocks, ordinal vectors and states copied from another context select the receiving context’s facts; later states carry the answer’s content. In OLMo-2-7B checkpoints, questions are first organized as if by the named person and switch to order of mention by 42B tokens, before the model answers reliably, and the full depth profile forms late (Section 7). A question thus says where to look, and the context says what is there, a division learned from next-token prediction alone. 2. Related Work Mechanistic interpretability.Circuits, features, and causal interventions explain behavior through internal components and the variables they carry [24,6,34,2,21,9], and activation addition and function vectors show that an activation estimated across examples can alter behavior on new inputs [30,33]. Closest to our question, 2 The Geometry of Contextual Relations: Language Models Address Facts by Order of Mention studies of contextual binding find binding identifiers, propositional probes, and positional and lexical contributions to bound-entity retrieval [7,8,29,12], and editing low-rank subspaces of bound-token states changes bindings [4,5]. Prakash et al.[29]find ordering IDs that a lookback mechanism uses to retrieve bound information, and Gur-Arieh et al.[12]find that this positional mechanism grows noisy in the middle of long lists, where lexical and reflexive mechanisms take over. These works locate order in the states of the bound tokens within one context; we study the question’s state across contexts, find that the address it carries counts facts rather than mentioned entities, state its geometry as a hypothesis, and show that this address transfers to new contexts and crowds in the middle of a list, a geometric account of that noise. Representation analysis and geometry.Linear and categorical representation theories tie semantic contrasts and concept hierarchies to directions [22,1,27,28], language models transform concepts with shared geometry [16], and statistical accounts derive such geometry from co-occurrence and its symmetries [18,19,23,17]. Geometry formed while reading structured input has been studied for in-context graphs, sentence trajectories, and logical reasoning [26,15,35], and steering can depend on how a context is presented [25]; the geometry we study is set by the order of the stated facts rather than by word meaning. 3. Order of Mention and Fact Addresses Given the factsAlice eats the appleandBob eats the pear, the questionWhat does Bob eat?is answered in two steps: find the fact the question is about, then read it. We formalize the first step. Definition 3.1(Stated facts, order of mention, and binding identity).A context퐶states facts푓 1 ,...,푓 푛 in this order, where푓 푖 binds one or more entities푒 푖 and a content푥 푖 under a relation푟 푖 . The index푖is the fact’sorder of mention, and thebinding identityat order of mention푖is훽 퐶 (푖) = (푒 푖 ,푟 푖 ,푥 푖 ), what퐶states there. A question푞 names part of one fact and asks for another part;푝 퐶 (푞)is that fact’s order of mention, and the answer is read from 훽 퐶 (푝 퐶 (푞)). For a fixed relation,푞 퐶 (푝)is the question that names the entity of fact푓 푝 and asks for its content. A question names an entity, but the model could reach the fact by its order of mention instead, as ordinal reference and human serial memory do [14]. Contexts and states.Aword setis one set of names and contents; a context states one fact per name in fixed sentence frames, and itsarrangementorders the facts while itscontent assignmentpairs names with contents. For a context퐶and question푞, thequery stateℎ ℓ (퐶,푞)∈R 푑 is the residual stream at the last prompt token after Transformer blockℓ. Definition 3.2(Query change and ordinal vector).For orders of mention푝̸=푝 ′ and calibration contexts풞 cal from other word sets, define ∆ ℓ (퐶;푞→푞 ′ ) =ℎ ℓ (퐶,푞 ′ )−ℎ ℓ (퐶,푞), 푣 푝→푝 ′ ℓ =E 퐶∈풞 cal ∆ ℓ (︀ 퐶;푞 퐶 (푝)→푞 퐶 (푝 ′ ) )︀ .(1) The mean푣is a steering vector, theordinal vector. In areceiver퐶 * , intervene once: ℎ ℓ (퐶 * ,푞 퐶 * (푝))←ℎ ℓ (퐶 * ,푞 퐶 * (푝)) +푣 푝→푝 ′ ℓ ,(2) keep the question, and ask whether the complete answer is thedesignated answer, the content of thedesignated factat푝 ′ in퐶 * . Replacing푣by the receiver’s own∆ ℓ (퐶 * ;푞 퐶 * (푝)→푞 퐶 * (푝 ′ ))gives theceiling. Hypothesis 3.3(Ordinal addressing).At an intermediate blockℓthere are context offsets푐 퐶 andfact addresses 푎 1 ,푎 2 ,..., one per order of mention and shared by all contexts, such that ℎ ℓ (퐶,푞) =푐 퐶 +푎 푝 퐶 (푞) +휀 ℓ (퐶,푞)(3) with small휀 ℓ (measured by held-out푅 2 ), and, beyond blockℓ(thereadout clause), the answer is the content of 훽 퐶 (푝)when the last-token state is푐 퐶 +푎 푝 : a question moves the state to the address of the fact it asks about, and the context supplies the binding identity found there. 3 The Geometry of Contextual Relations: Language Models Address Facts by Order of Mention a Stated facts, order of mention, and binding identity (Definition 3.1) context C binding identity β C (i) Alice loves Bob.(Alice, loves, Bob) Carol loves David.(Carol, loves, David) question q: Whom does Carol love? p C (q) = 2answer: David, from β C (2) swap Bob and David β C changes p C same reorder the facts loves loves same facts p C changes passive paraphrase is loved by is loved by β C same p C same AliceBob CarolDavid CarolDavid AliceBob BobAlice DavidCarol loves 1 2 1 2 1 2 1 2 fact 1 (a 1 ) fact 2 (a 2 ) −0.5 0.0 0.5 ordinal vector ( v ) −0.3 0.0 0.3 which people are named −0.1 0 +0.1 context offset ( c C ) fact 1 fact 2 −1 0 1 cosine with v new names reversed order b Hypothesis plane: one context, raised by its offset c C dot: question about person A arrow: switch to person B (purple: facts reversed) same arrow a 2 − a 1 everywhere; flipped if reversed c Measured: Qwen2.5-7B, block 20 each plane: two test contexts, one offset level dark: calibration order; purple: facts reversed d All models Qwen Gemma Llama prediction other instruction-tuned base Figure2:The problem, the hypothesis, and its first test.(a) Definition 3.1 on the introduction’s example: each stated fact has an order of mention and a binding identity, and a question is answered from the binding identity at its order of mention. Swapping Bob and David changes the binding identities; reordering keeps the facts and changes푝 퐶 ; the passive paraphrase changes neither. (b) Hypothesis 3.3 as a displacement field: every context shares the fact addresses푎 1 ,푎 2 , so the question moves each context by the same푎 2 −푎 1 , or by−(푎 2 −푎 1 )when the two facts are reversed. (c) The same form measured in Qwen for eight test contexts, two arrangements at each of four offset levels (percentiles of the context-offset coordinate), on three axes fit on calibration word sets: along푣, which people are named, and the context offset; units of‖푣‖, with the offset axis stretched for display (Appendix B). (d) Cosines with푣for new names (dark; mean of 64 name pairs) and between the ordinal vectors of reversed arrangements (purple), one dot per model, against the predictions of Proposition 3.4; small dots: eleven further models (Appendix G; open: base weights). Let푦(퐶,푞) =ℎ ℓ (퐶,푞)− 1 푛 ∑︀ 푝 ℎ ℓ (퐶,푞 퐶 (푝))be thequery-dependent state: the query state centered over the context’s푛questions, one per fact. With휀= 0, the hypothesis has four consequences (Propositions 3.4 to 3.6, Corollary 3.7), each tested below; Proposition 3.9 relaxes휀= 0. Proposition 3.4(Invariance and reversal).For any contexts퐶,퐶 * and orders of mention푝̸=푝 ′ ,∆ ℓ (퐶;푞 퐶 (푝)→ 푞 퐶 (푝 ′ )) = ∆ ℓ (퐶 * ;푞 퐶 * (푝)→푞 퐶 * (푝 ′ )) =푎 푝 ′ −푎 푝 , for any names and contents they state. If퐶 ′ exchanges the facts at 푝and푝 ′ of퐶, the same pair of questions gives−(푎 푝 ′ −푎 푝 ). Proposition 3.5(Organization by order of mention).푦(퐶,푞) =푎 푝 퐶 (푞) − ̄푎with ̄푎= 1 푛 ∑︀ 푝 푎 푝 : the query-dependent state depends on푞only through푝 퐶 (푞), not on the entities or relation it names or on token distances. If퐶 ′ moves the fact at푝in퐶to휋(푝), then푦(퐶 ′ ,푞) =푎 휋(푝 퐶 (푞)) − ̄푎. Proposition 3.6(Transfer).For any receiver퐶 * , Equation(2)givesℎ ℓ (퐶 * ,푞 퐶 * (푝)) +푣 푝→푝 ′ ℓ =푐 퐶 * +푎 푝 ′ , so by the 4 The Geometry of Contextual Relations: Language Models Address Facts by Order of Mention readout clause the answer is the content of훽 퐶 * (푝 ′ ). Corollary 3.7(Linear structure).Every change∆ ℓ (퐶;푞→푞 ′ )lies in the address subspace퐴 푛 = span푎 푝 −푎 1 :푝≤푛, of dimension at most푛−1, shared by all contexts of푛facts, and ordinal vectors compose:푣 푝→푝 ′ =푣 푝→푝 ′ +푣 푝 ′ →푝 ′ . Proof of Propositions 3.4 to 3.6 and Corollary 3.7.Substitute Equation (3):푐 퐶 cancels in every within-context difference, so every∆ ℓ (퐶;푞 퐶 (푝)→푞 퐶 (푝 ′ ))and every mean푣 푝→푝 ′ equals푎 푝 ′ −푎 푝 ; exchanging or moving facts changes only which address each question reaches; the readout clause gives the answer; and differences of addresses lie in퐴 푛 and telescope. Remark3.8.A relation component, shared by a context’s questions, cancels in∆ ℓ ; coding the question by the named entity or the answer’s content would contradict Propositions 3.4 to 3.6. Proposition 3.9(Spacing bounds transfer).Let instead휀 ℓ ∼ 풩(0,휎 2 퐼), let푣be exact,푣=푎 푝 ′ −푎 푝 , and let the readout return the fact whose address is nearest toℎ ℓ −푐 퐶 * . The edit of Proposition 3.6 then selects훽 퐶 * (푝 ′ )with probability between1−(푛−1) ̄ Φ(푑 푝 ′ /2휎) and1− ̄ Φ(푑 푝 ′ /2휎) , both rising with푑 푝 ′ = min 푝 ′ ̸=푝 ′ ‖푎 푝 ′ −푎 푝 ′ ‖ , where ̄ Φ is the standard normal tail: isolated addresses are reached more reliably. This extends the readout clause, to which it reduces at휀= 0. Proof.The edited state is푐 퐶 * +푎 푝 ′ +휀 ℓ . It is read as fact푝 ′ only if the component of휀 ℓ toward푎 푝 ′ exceeds ‖푎 푝 ′ −푎 푝 ′ ‖/2, with probability ̄ Φ(‖푎 푝 ′ −푎 푝 ′ ‖/2휎)≤ ̄ Φ(푑 푝 ′ /2휎); a union bound over the푛−1other addresses gives the lower bound. A component toward the nearest address beyond푑 푝 ′ /2alone causes an error, which gives the upper bound. Models and measurement.We use Qwen2.5-7B-Instruct, Gemma-3-4B-IT, and Llama-3.1-8B-Instruct. Themain blocksare 20, 22, and 15, fixed on separate word sets before testing. Each edit occurs once, before the first answer token, in native chat templates (Appendix A). Tests use 64 new word sets per model, and calibration uses others. We score complete greedy answers, counting invalid and truncated outputs as failures. Invariance and reversal hold across models.In two-fact contexts (Appendix B), the query change behaves as Proposition 3.4 predicts (Figure 2c,d). Over 64 new name pairs, its mean cosine with the ordinal vector is0.93, 0.88, and0.92in Qwen, Gemma, and Llama, and the ordinal vectors of the two arrangements have cosines of −0.93,−0.82, and−0.85. Eleven further models show the same signs; the weakest reversal cosine is−0.44, in base Gemma-3-4B (Figure 2d). 4. Questions Are Organized by Order of Mention We next test Proposition 3.5: the query-dependent state is set by the queried fact’s order of mention. Query states group by the queried fact.Afact-address template, the mean query-dependent state over calibration contexts whose queried fact has a given order of mention, explains66%,29%, and49%of the query-dependent variance of held-out word sets in Qwen, Gemma, and Llama, and aperson template, by the queried person (the 푘-th name of each word set), explains none (Figure 3). Inside one word set, where the names are fixed, the named person sets the state in early blocks but adds at most0.10to the fact-address template’s푅 2 at the main blocks (Appendix C.1, Figure 7). 5 The Geometry of Contextual Relations: Language Models Address Facts by Order of Mention −1.0−0.50.00.5 PC 1 (39% of variance) −0.5 0.0 0.5 PC 2 (28%) predicted fact 1→2 1 2 3 4 −1.0−0.50.00.5 PC 1 (22% of variance) −0.5 0.0 0.5 PC 2 (10%) 1 2 3 4 −1.0−0.50.00.5 PC 1 (34% of variance) −0.5 0.0 0.5 PC 2 (19%) 1 2 3 4 0.00.51.0 cosine: measured vs predicted person×relation interaction shuffled −0.02 Qwen Gemma Llama original pair new order of mention re- calibrated 0 50 100 Designated answers (%) owns visitedworks as plays instr. plays sport 0 50 100 Designated answers (%) a Qwen2.5-7B, block 20 fact-address R 2 0.66 person R 2 −0.01 b Gemma-3-4B, block 22 fact-address R 2 0.29 person R 2 −0.01 c Llama-3.1-8B, block 15 fact-address R 2 0.49 person R 2 −0.01 Predicted interaction b Rearranged facts c ‘eats’ vector, other relations QwenGemmaLlamaperson changerelation changeown-relation vector Figure3:Query states are organized by order of mention, not by the queried person.(a–c) Query-dependent states of held-out word sets at each model’s main block, projected onto the top two principal directions of the four fact-address templates fit on eight other word sets; axis labels give the share of held-out query-dependent variance on each direction, and the unit is the fact-1 to fact-2 template distance. Each dot is one context under one question, colored by the queried fact’s order of mention; gray lines join a context’s fact-1 and fact-2 questions, the dark arrow is the predicted change푎 2 −푎 1 , and contours enclose 50% and 85% of each order’s density. Panel headers: held-out푅 2 of fact-address and person templates. The address belongs to the fact, not to the name.When one fact has two subjects, the order of facts and the order of names come apart: Carol is the third name but the subject of the second fact inAlice and Bob eat the apple. Carol eats the pear.Such query states lie closer to the fact-address template of their fact than to the template at their name’s order in99.2%,96.2%, and96.2%of cases in Qwen, Gemma, and Llama (Appendix C.2). In two-place facts such asAlice loves Bob, query states are organized by fact andargument slot, which argument is asked for, not by person: a fact-by-slot template has held-out푅 2 of0.82,0.65, and0.70(Appendix C.3, Figure 8). 5. The Fact Address Transfers across Contexts If the address is shared, it should also move between contexts: by Proposition 3.6, an ordinal vector calibrated on other contexts should select the designated fact in a new context, which supplies its content. With two facts any change reaches the only other content, so we use four and the twelve directed changes between their orders of mention, each averaged over calibration contexts (Appendix A). In a list that beginsPetra eats the buckwheatand Freya eats the amaranth,푣 1→2 added to the question about Petra should giveamaranth. The answer follows the receiver’s facts.Across 64 new word sets, the ordinal vector selects the designated fact in84.1%of cases in Qwen,53.9%in Gemma, and36.2%in Llama (Table 1A), and equally long random directions in at most0.01%(Appendix D). It still works in short paragraphs that state the facts with varied verbs among unrelated sentences (44.5%,42.4%, and14.0%; Appendix G.3). Facts define the vector.With every direction scaled to one length, the ordinal vector calibrated with the facts selects the designated answer in71.3%,41.7%, and10.3%of cases. Directions computed from inputs without the facts select none from the question alone and at most3.9%from a list of the names (Table 1B): the vector arises from reading the question against the facts. A whole state from another context.A stronger test replaces the receiver’s query state for question푠with a donorcontext’s state. If the donor asked question푡about another fact, receivers answer with their own fact at the order of mention of question푡in80.9%,72.0%, and51.3%of cases, against1.7%,13.9%, and6.1%when the 6 The Geometry of Contextual Relations: Language Models Address Facts by Order of Mention Table 1:An ordinal vector from other con- texts selects the designated fact in a new context.Designated answers (%), question unchanged;±is half the 95% word-set boot- strap interval; shading is proportional to the rate. (A) 64 new word sets, 12,288 cases per model; Random: at most0.01%. (B) 64 fur- ther word sets, 1,024 cases per row, all direc- tions scaled to one length; the question and name rows are computed without the facts (Appendix D). Ceiling: the receiver’s own state for the designated question; it differs between panels because the word sets differ. (C) The re- ceiver’s state replaced by a donor’s; 16 donors ×64 receivers; designated: the receiver’s fact at the order of mention of the other question. Qwen Gemma Llama A. Transfer to new contexts Ordinal vector푣 84.1±2.053.9±1.636.2±1.6 Random0.00.00.0 Ceiling96.9±1.396.4±1.291.9±1.9 B. Where the vector comes from With facts (푣) 71.3±4.041.7±2.410.3±2.0 Question alone0.00.00.0 Names listed, same order2.5±1.11.3±0.83.9±1.4 Names listed, shuffled 0.00.1±0.10.2±0.2 Random0.00.00.0 Ceiling94.4±2.497.9±1.193.2±2.4 C. Whole state from another context Donor asked another fact 80.9±3.172.0±2.751.3±3.1 Donor asked the receiver’s question1.7±0.913.9±1.66.1±1.1 donor asked the receiver’s own question (Table 1C). It rarely yields the donor’s complete answer (Appendix F.1): the state carries which fact to read, not what the donor read. Which ordinal vector transfers.It is the one chosen by the receiver’s orders of mention, not by its names, relation, or token distances. When one fact has two subjects, a vector chosen by the order of facts selects the designated answer in65.3%of Qwen cases, and one chosen by the order of names in none (Appendix C.2). When the facts are rearranged, the vector of the original question pair mostly fails, and the one chosen by the new orders of mention raises Qwen’s transfer for person changes from9.0%to69.7%(Appendix D.2). Across six relations, a vector calibrated only oneatsand rescaled to the receiving relation’s change length comes within five percentage points of each relation’s own (Appendix D.3). When filler separates order from token distance, the vector selects the fact at the calibrated order over the one at the calibrated absolute distance by72percentage points in Qwen, and by less in Gemma and Llama (Appendix D.4). 6. Fact Addresses Are Low-Rank 6.1. Fact addresses span a shared low-rank subspace If one address set serves every context, the addresses should span a small shared subspace. As Corollary 3.7 predicts, the three-dimensional address subspace spanned by fact-address templates from other contexts holds 69%,33%, and52%of the squared norm of held-out changes in Qwen, Gemma, and Llama, within 1–8 points of the best three-dimensional subspace, against0.1%(about3/푑) for a random one (Figure 4a; Appendix E.1). The subspaces of six relations nearly coincide (mean cosine of the largest principal angle0.78–0.89), and ordinal vectors estimated on disjoint word sets compose along a path:푣 푝→푝 ′ +푣 푝 ′ →푝 ′ has cosine0.99,0.97, and0.98with 푣 푝→푝 ′ . Fact addresses also predict how person and relation changes interact (Appendix E.4). 6.2. Later addresses crowd together, and the first is easiest to reach In lists of up to eight facts, the fact addresses form an ordered chain whose later links shorten (Figure 4b,c). Each address is fixed once its fact is read: the vector from fact 1 to fact푝keeps its direction when later facts are removed (median cosine0.99with the eight-fact vector; Appendix E.3), so the direction of푎 푝 −푎 1 does not depend on the list length. Transfer follows this spacing, as Proposition 3.9 predicts: the farther a fact address lies from its nearest neighbor, the better an ordinal vector reaches it (푅 2 = 0.82within models; Figure 4d). Accordingly, transfer is 7 The Geometry of Contextual Relations: Language Models Address Facts by Order of Mention −0.5 0 0.5 dir. 1 −0.5 0 0.5 dir. 2 −0.5 0 0.5 dir. 3 fact 1 fact 2 fact 3 fact 4 −0.50.00.5 PC 1 (unit: fact 1–2 distance) −0.5 0.0 0.5 PC 2 1–23–45–67–8 neighboring facts 0.0 0.5 1.0 next-address distance (rel. to facts 1–2) 0.00.51.0 distance to nearest address 0 50 100 Designated answers (%) R 2 = 0.82 a Changes lie in a three-dimensional subspace (Qwen)b Eight addressesc Addresses crowdd Spacing → transfer QwenGemmaLlama Figure4:Fact addresses are low-rank and ordered by mention.(a) Qwen’s held-out contexts in the address subspace fit on eight other word sets: query states (small dots, colored by the queried fact) and changes (gray); large dots: the four fact addresses, joined as a tetrahedron. (b) The eight fact addresses of eight-fact lists in their first two principal directions (unit: fact-1 to fact-2 distance); large marker: fact 1. (c) Distance between neighboring fact addresses, relative to facts 1–2, for four, six, and eight facts (light to dark). (d) Each address’s distance to its nearest neighbor (unit: fact-1 to fact-2 distance) against transfer to that fact (Proposition 3.9), for 54 addresses;푅 2 after removing each model’s mean. highest to the first fact (82–95%in Qwen), falls toward the middle, and in most lists rises slightly at the last: a serial-position curve with strong primacy, as in human recall (14; Appendix E.2, Figure 11). 7. Where and When the Fact Address Is Shared Addressing is an intermediate step: inside one word set, the query state carries first the named person, then the fact address, then the requested content (Appendix C.1). Hypothesis 3.3 should therefore fail near the output, where the state carries the retrieved content푥 푝 of훽 퐶 (푝): what it selects then depends on the context it came from. To see where the address is shared, we apply the two interventions of Section 5, the ordinal vector and the donor state, at every block and record three outcomes: the receiver’s fact selected by the ordinal vector, the receiver’s fact selected by the donor state, and the donor’s own first answer token (Figure 5a; Appendix F.1). A state carrying only an address selects the receiver’s content; one carrying retrieved content, the donor’s. 7.1. The shared address occupies a depth window In early blocks even the ceiling has no effect, though order of mention is already readable (Figure 13); in a late-middledepth window, ordinal vectors and donor states both select receiver contents (Figure 5b–d). The ordinal vector fails first: Gemma’s falls from44.3%at block 22 to zero at block 23 (Figure 5c), while the ceiling stays at 96%. In Llama, from block 17 to 21, the ordinal vector stays below1%while donor states select30–36%: each whole state holds the address, which is no longer a shared difference. The donor’s own first token then takes over, reaching67.0%at block 23 in Gemma (baked apple crumblejoins the donor’s first word to the receiver’s content). The geometry agrees: fact-address푅 2 collapses where the shared address ends, from0.29at block 22 to0.05at block 23 in Gemma, and each state aligns with its answer’s output direction (Figure 13). 7.2. The fact address forms during pretraining We repeat the tests on ten checkpoints of OLMo-2-7B [32], from 5B training tokens to the released model (Figure 6; Appendix F.2). Questions switch from the named person to order of mention: at 5B tokens the query change is the 8 The Geometry of Contextual Relations: Language Models Address Facts by Order of Mention Receiver, asks about fact 1:What does Irene Sutton eat? fact 1Irene Sutton eats the apple crumble. fact 2Graham Terry eats the pear tart.(+2 more) Donor, asks about fact 2:What does Austin Gray eat? fact 1Angela Fletcher eats the grilled salmon. fact 2Austin Gray eats the baked trout.(+2 more) ordinal vector add v from other contexts → pear tart, its fact 2 (block 22) donor → receiver's fact copy the donor's state→ pear tart, its own fact 2 (block 22) donor → donor's first token copy the donor's state→ baked apple crumble (block 33) stage by blockshared addresswhole-state addresscontent Qwen2.5-7B Gemma-3-4B Llama-3.1-8B Qwen2.5-1.5B Qwen2.5-3B Qwen2.5-14B Qwen2.5-32B Qwen3-8B inputrelative depth (block / last block)output 081624 Transformer block 0 50 100 Answers (%) 081624 Transformer block 0.0 0.4 0.8 R 2 / alignment 08162432 Transformer block 0 50 100 08162432 Transformer block 0.0 0.4 0.8 08162431 Transformer block 0 50 100 08162431 Transformer block 0.0 0.4 0.8 a Three outcomes through depth: one example, and where each succeeds b Qwen2.5-7B: answers a Qwen2.5-7B c Gemma-3-4B: answers b Gemma-3-4B d Llama-3.1-8B: answers c Llama-3.1-8B own state (ceiling)ordinal vectordonor → receiver's factdonor → donor's first token fact-address R 2 answer alignment person R 2 Figure5:Through depth, the fact address is first shared across contexts, then held only in whole states, then replaced by the answer’s content.(a) The two interventions and three outcomes on one Gemma example, and each block classified over relative depth: shared address (the ordinal vector selects the receiver’s fact in at least10%of cases), whole-state address (only donor states do, at least10%), and content (the donor’s first token reaches50%); blocks are classified one by one, so the first two can interleave; lower rows: other Qwen models (Appendix G.1). (b–d) The three outcomes through depth, with the ceiling as reference, on the prompts of Table 1C (one arrangement, two content assignments, four person changes; 16 donors×64 receivers), where the ordinal vector selects55.5%,44.3%, and25.8%at the main blocks, below Table 1A over all arrangements; bands: 95% intervals resampling donor and receiver word sets. Shading: the depth window, blocks where the ordinal vector selects at least10%. same in both arrangements (cosine+0.98), as if questions were represented by the named person, and it reverses by 42B (−0.61), as Proposition 3.4 predicts, while the model answers only half the questions; fact-address푅 2 rises from0.00to0.52over the same span. The depth profile forms later and settles only at the end of training (Figure 6a): shared-address blocks first appear at 42B but are absent at 105B and again at 3.9T, content blocks appear from 416B, all three stages first co-occur at 839B, and only the released model shows them as one contiguous profile. Transfer at block 16, the released model’s main block, peaks at 210B (50.1%against a56.8%ceiling) and then declines as the depth profile moves away from it. 8. Discussion Order of mention is learned from pretraining.Resolving a question by order of mention separates the structure of a context from its contents: one address, which counts facts rather than tokens, serves any names and contents, and the context supplies them. No objective asks for this separation, yet OLMo-2-7B learns it early in pretraining, after first organizing questions as if by the named person. It also shares two signatures of human discourse 9 The Geometry of Contextual Relations: Language Models Address Facts by Order of Mention shared addresswhole-state addresscontent 5B 17B 42B 105B 210B 416B 839B 1.7T 3.9T final tokens inputrelative depthoutput 10B100B1Tfinal training tokens −1.0 −0.5 0.0 0.5 1.0 fact-address R 2 cos, new names cos, reversed order 10B100B1Tfinal training tokens 0 50 100 designated answers (%) ceiling ordinal vector random unedited a Stage by block through training b From person to order of mention c Transfer at block 16 Figure6:In pretraining, questions switch from the named person to order of mention before the model answers reliably, and the full depth profile forms late.Ten checkpoints of OLMo-2-7B, from 5B training tokens to the released model (final, after mid-training). (a) Stages as in Figure 5a, with each threshold applied to the gain over the unedited answer; all three stages first co-occur at 839B. (b) Fact-address푅 2 (peak over blocks) and the cosines of Proposition 3.4 at block 16. (c) Designated answers at block 16; dashed: unedited accuracy; dotted: random directions. processing: the first-mentioned fact is the easiest to reach, and later ones crowd together, as in serial recall [10,14]. Unlike linear concept representations [27], a fact address is a place in the context, not a meaning. Fact retrieval as addressing.Retrieving a stated fact may then depend on where it stands: crowded addresses in the middle of a list transfer least, which may help explain weaker retrieval in the middle of long contexts [20] and reasoning that changes when premises are reordered [3]. We study templated lists and short paragraphs and edit one token position at one block. In this setting the address is clearest in Qwen; fact-address templates explain29% of Gemma’s query-dependent variance, and the ordinal vector moves Llama’s answers less often in paragraphs and with two-subject facts (12–14%). Long lists also draw on lexical and reflexive retrieval [12]. The natural next steps are long documents and multi-step reasoning, where a fact reached by its address must be combined with others. 9. Conclusion We asked how relations organize the language representations of contextual contents, and answered by measuring and editing the state a question leaves before the model answers. Language models address stated facts by order of mention: a question moves the state to a fact address shared across contexts, and the context supplies what is stated there. These addresses steer answers in new contexts, span a low-rank subspace in which later facts crowd together, are shared in a late-middle depth window, and form early in pretraining. References [1]Sanjeev Arora, Yuanzhi Li, Yingyu Liang, Tengyu Ma, and Andrej Risteski. Linear algebraic structure of word senses, with applications to polysemy.Transactions of the Association for Computational Linguistics, 6: 483–495, 2018. [2] Trenton Bricken, Adly Templeton, Joshua Batson, Brian Chen, Adam Jermyn, Tom Conerly, Nick Turner, Cem Anil, Carson Denison, Amanda Askell, Robert Lasenby, Yifan Wu, Shauna Kravec, Nicholas Schiefer, Tim Maxwell, Nicholas Joseph, Zac Hatfield-Dodds, Alex Tamkin, Karina Nguyen, Brayden McLean, Josiah E 10 The Geometry of Contextual Relations: Language Models Address Facts by Order of Mention Burke, Tristan Hume, Shan Carter, Tom Henighan, and Christopher Olah. Towards monosemanticity: Decom- posing language models with dictionary learning.Transformer Circuits Thread, 2023. https://transformer- circuits.pub/2023/monosemantic-features/index.html. [3] Xinyun Chen, Ryan Andrew Chi, Xuezhi Wang, and Denny Zhou. Premise order matters in reasoning with large language models. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp, editors,Proceedings of the 41st International Conference on Machine Learning, volume 235 ofProceedings of Machine Learning Research, pages 6596–6620. PMLR, 21–27 Jul 2024. URLhttps://proceedings.mlr.press/v235/chen24i.html. [4] Qin Dai, Benjamin Heinzerling, and Kentaro Inui. Representational analysis of binding in language models. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen, editors,Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 17468–17493, Miami, Florida, USA, November 2024. Association for Computational Linguistics. doi: 10.18653/v1/2024.emnlp-main.967. URLhttps: //aclanthology.org/2024.emnlp-main.967/. [5]Qin Dai, Benjamin Heinzerling, and Kentaro Inui. Cell-based representation of relational binding in language models.arXiv preprint arXiv:2604.19052, 2026. URLhttps://arxiv.org/abs/2604.19052. [6] Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Nova DasSarma, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds, Danny Hernandez, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, and Chris Olah. A mathematical framework for transformer circuits.Transformer Circuits Thread, 2021. https://transformer-circuits.pub/2021/framework/index.html. [7]Jiahai Feng and Jacob Steinhardt. How do language models bind entities in context? InInternational Conference on Learning Representations, 2024. URLhttps://proceedings.iclr.c/paper_files/pa per/2024/hash/9d1b7fc578c0d2d6431fc26d736ecaf3-Abstract-Conference.html. [8]Jiahai Feng, Stuart Russell, and Jacob Steinhardt. Monitoring latent world states in language models with propositional probes. InInternational Conference on Learning Representations, 2025. URLhttps: //proceedings.iclr.c/paper_files/paper/2025/hash/3132d0405fabe24b2a7b6cd7ba9de6 b5-Abstract-Conference.html. [9]Atticus Geiger, Hanson Lu, Thomas Icard, and Christopher Potts. Causal abstractions of neural networks. In Advances in Neural Information Processing Systems, volume 34, 2021. URLhttps://papers.nips.c/p aper/2021/hash/4f5c422f4d49a5a807eda27434231040-Abstract.html. [10]Morton Ann Gernsbacher and David J. Hargreaves. Accessing sentence participants: The advantage of first mention.Journal of Memory and Language, 27(6):699–717, 1988. doi: 10.1016/0749-596X(88)90016-2. [11]Daniel Gildea and Daniel Jurafsky. Automatic labeling of semantic roles.Computational Linguistics, 28(3): 245–288, 2002. doi: 10.1162/089120102760275983. URLhttps://aclanthology.org/J02-3001/. [12]Yoav Gur-Arieh, Mor Geva, and Atticus Geiger. Mixing mechanisms: How language models retrieve bound entities in-context. InInternational Conference on Learning Representations, 2026. URLhttps://proceedi ngs.iclr.c/paper_files/paper/2026/hash/2eeff35664016c7f0f8a704f0d9a83e-Abstrac t-Conference.html. [13]Graeme S. Halford, William H. Wilson, and Steven Phillips. Relational knowledge: the foundation of higher cognition.Trends in Cognitive Sciences, 14(11):497–505, November 2010. ISSN 1364-6613. doi: 10.1016/j.tics.2010.08.005. URLhttp://dx.doi.org/10.1016/j.tics.2010.08.005. [14] Richard N. A. Henson. Short-term memory for serial order: The Start-End model.Cognitive Psychology, 36 (2):73–137, 1998. doi: 10.1006/cogp.1998.0685. 11 The Geometry of Contextual Relations: Language Models Address Facts by Order of Mention [15]Eghbal A. Hosseini and Evelina Fedorenko. Large language models implicitly learn to straighten neural sentence trajectories to construct a predictive representation of natural language. InAdvances in Neural Information Processing Systems, volume 36, 2023. URLhttps://proceedings.neurips.c/paper_f iles/paper/2023/hash/88dddaf430b5bc38ab8228902b61821-Abstract-Conference.html. [16]Zhimin Hu, Lanhao Niu, and Sashank Varma. Language models represent and transform concepts with shared geometry.arXiv preprint arXiv:2607.04525, 2026. URLhttps://arxiv.org/abs/2607.04525. [17]Minyoung Huh, Brian Cheung, Tongzhou Wang, and Phillip Isola. Position: The platonic representation hypothesis. InProceedings of the 41st International Conference on Machine Learning, volume 235 ofProceedings of Machine Learning Research, pages 20617–20642, 2024. URLhttps://proceedings.mlr.press/v2 35/huh24a.html. [18] Yibo Jiang, Goutham Rajendran, Pradeep Kumar Ravikumar, Bryon Aragam, and Victor Veitch. On the origins of linear representations in large language models. InProceedings of the 41st International Conference on Machine Learning, volume 235 ofProceedings of Machine Learning Research, pages 21879–21911, 2024. URL https://proceedings.mlr.press/v235/jiang24d.html. [19] Dhruva Karkada, Daniel J. Korchinski, Andres Nava, Matthieu Wyart, and Yasaman Bahri. Symmetries in language statistics shape the geometry of model representations. InProceedings of the 43rd International Conference on Machine Learning, 2026. URLhttps://icml.c/virtual/2026/poster/63405. [20]Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. Lost in the middle: How language models use long contexts.Transactions of the Association for Computational Linguistics, 12:157–173, 2024. doi: 10.1162/tacl_a_00638. [21] Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual associations in GPT. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors,Advances in Neural Information Processing Systems, volume 35, pages 17359–17372. Curran Associates, Inc., 2022. doi: 10.52202/068431-1262. URLhttps://proceedings.neurips.c/paper_files/paper/2022/fi le/6f1d43d5a82a37e89b0665b33bf3a182-Paper-Conference.pdf. [22]Tomas Mikolov, Wen-tau Yih, and Geoffrey Zweig. Linguistic regularities in continuous space word representa- tions. InProceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 746–751. Association for Computational Linguistics, 2013. URLhttps://aclanthology.org/N13-1090/. [23]Alexander Modell, Patrick Rubin-Delanchy, and Nick Whiteley. The origins of representation manifolds in large language models.arXiv preprint arXiv:2505.18235, 2025. URLhttps://arxiv.org/abs/2505.18235. [24]Chris Olah, Nick Cammarata, Ludwig Schubert, Gabriel Goh, Michael Petrov, and Shan Carter. Zoom in: An in- troduction to circuits.Distill, 2020. doi: 10.23915/distill.00024.001. https://distill.pub/2020/circuits/zoom- in. [25]Gustaw Opiełka, Hannes Rosenbusch, and Claire E. Stevenson. Causality̸=invariance: Function and concept vectors in LLMs. InInternational Conference on Learning Representations, 2026. URLhttps: //proceedings.iclr.c/paper_files/paper/2026/hash/86b3697c4eb7792c951831636bfdac d5-Abstract-Conference.html. [26] Core Francisco Park, Andrew Lee, Ekdeep Singh Lubana, Yongyi Yang, Maya Okawa, Kento Nishi, Martin Wattenberg, and Hidenori Tanaka. ICLR: In-context learning of representations. InInternational Conference on Learning Representations, 2025. URLhttps://arxiv.org/abs/2501.00070v2. [27] Kiho Park, Yo Joong Choe, and Victor Veitch. The linear representation hypothesis and the geometry of large language models. InProceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 39643–39666, 2024. URLhttps://proceedings.mlr. press/v235/park24c.html. 12 The Geometry of Contextual Relations: Language Models Address Facts by Order of Mention [28]Kiho Park, Yo Joong Choe, Yibo Jiang, and Victor Veitch. The geometry of categorical and hierarchical concepts in large language models. InInternational Conference on Learning Representations, 2025. URL https://arxiv.org/abs/2406.01506. [29] Nikhil Prakash, Natalie Shapira, Arnab Sen Sharma, Christoph Riedl, Yonatan Belinkov, Tamar Rott Shaham, David Bau, and Atticus Geiger. Language models use lookbacks to track beliefs. InInternational Conference on Learning Representations, 2026. URLhttps://proceedings.iclr.c/paper_files/paper/2026 /hash/0d1941f73833eb107939f62d323d6c0-Abstract-Conference.html. [30]Nina Rimsky, Nick Gabrieli, Julian Schulz, Meg Tong, Evan Hubinger, and Alexander Turner. Steering Llama 2 via contrastive activation addition. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors, Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15504–15522, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10.18653/v1/2024.acl-long.828. URLhttps://aclanthology.org/2024.acl-long.828/. [31] Paul Smolensky. Tensor product variable binding and the representation of symbolic structures in connectionist systems.Artificial Intelligence, 46(1–2):159–216, 1990. doi: 10.1016/0004-3702(90)90007-M. URL https://w.sciencedirect.com/science/article/pii/000437029090007M. [32] Team OLMo, Pete Walsh, Luca Soldaini, Dirk Groeneveld, Kyle Lo, Shane Arora, et al. 2 OLMo 2 furious. arXiv preprint arXiv:2501.00656, 2024. [33] Eric Todd, Millicent Li, Arnab Sen Sharma, Aaron Mueller, Byron Wallace, and David Bau. Function vectors in large language models. In B. Kim, Y. Yue, S. Chaudhuri, K. Fragkiadaki, M. Khan, and Y. Sun, editors, International Conference on Learning Representations, volume 2024, pages 17282–17333, 2024. URLhttps: //proceedings.iclr.c/paper_files/paper/2024/file/4ae163cb8788970e53b4fd95781411 39-Paper-Conference.pdf. [34] Kevin Ro Wang, Alexandre Variengien, Arthur Conmy, Buck Shlegeris, and Jacob Steinhardt. Interpretability in the wild: a circuit for indirect object identification in GPT-2 small. InThe Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023. URL https://openreview.net/pdf?id=NpsVSN6o4ul. [35]Yufa Zhou, Yixiao Wang, Xunjian Yin, Shuyan Zhou, and Anru R. Zhang. The geometry of reasoning: Flowing logics in representation space. InThe Fourteenth International Conference on Learning Representations, 2026. URLhttps://openreview.net/forum?id=ixr5Pcabq7. 13 The Geometry of Contextual Relations: Language Models Address Facts by Order of Mention Appendix Contents The appendix follows the main text. Appendix A gives the shared protocol, and each later part opens with the main-text claim it supports. A Setup15 B Invariance and Reversal in Two-Fact Contexts16 C Questions Are Organized by Order of Mention16 C.1 Query-State Geometry . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16 C.2 Coordinated Subjects: The Order of Facts, Not of Names . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17 C.3 Two-Place Relations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18 D The Fact Address Transfers across Contexts19 D.1 What the Facts and Whole Donor States Contribute . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19 D.2 Rearranged Facts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20 D.3 Six Relations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20 D.4 Order of Mention versus Token Distance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21 E Fact Addresses Are Low-Rank22 E.1 The Address Subspace . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22 E.2 Longer Lists . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23 E.3 Prefixes of a List . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24 E.4 Person and Relation Do Not Add . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24 F Where and When the Fact Address Is Shared25 F.1 The Depth Window . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25 F.2 Pretraining Checkpoints . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26 G Generality: Model Scale, Base Weights, and Prose28 G.1 Model Scale . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28 G.2 Base and Instruction-Tuned Weights . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28 G.3 Facts Stated in Prose . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29 14 The Geometry of Contextual Relations: Language Models Address Facts by Order of Mention A. Setup Every intervention uses one protocol: a change of the query state is calibrated on some word sets, added once to contexts built from new word sets, and judged by the complete answer. Models and main blocks.We use Qwen2.5-7B-Instruct, Gemma-3-4B-IT, and Llama-3.1-8B-Instruct; block indices are zero-based Transformer block outputs. Qwen’s main block 20 and Gemma’s block 22 were chosen before the experiments reported here, as the blocks where an earlier, related edit of the query state changed the most complete answers on separate word sets, and both were confirmed on the block-selection word sets of this protocol before any test. Llama’s block 15, and the main block of every model in Appendix G, come from the following rule on those block-selection word sets. For each tested block, the rule scores the ordinal vector’s gain over the unedited answer, minus the mean gain of random directions, minus any rise in invalid answers; it finds the best block and chooses the earliest block within two points of it. On the blocks profiled for Qwen it also gives block 20; on Gemma’s plain-prompt profile it ties blocks 20 and 22 (51.6points each) and would take block 20. The models of Appendix G test every second block, and where no block reaches two points, the block nearest Qwen2.5-7B’s relative depth is used. No main block was chosen on test word sets. Word sets.Aword setis a set of names and contents: four people and four contents in the four-person experiments. Except where noted, each experiment draws new names and new contents, mostly multiword dishes (ackee and saltfish,pad see ew). Calibration and test word sets share no complete name or content string, although subwords can recur. Within every word set, the four answers have pairwise distinct first tokens under the model’s tokenizer. Every experiment tests on 64 word sets not used for calibration, except the depth curves of Appendix G and Appendix F.2, which use 8 donor and 32 receiver word sets. Unless stated otherwise, calibration uses 16 word sets; Table 1A–B, the two-fact contexts of Appendix B, and the rearranged facts of Appendix D.2 use 8. A context states one fact per person, such asPetra eats the buckwheat, followed by a question,What does Petra eat?, under the instructionRead the sentences. Answer the question with only the food, in lowercase, with no explanation.Acontent assignmentassigns the four contents to the four people; anarrangementorders the four facts. Both rotate cyclically, so every person and every content occupies every order of mention. For Table 1A, the ordinal vector푣 푝→푝 ′ for a directed pair of orders of mention averages Equation (1) over all calibration contexts, content assignments, and arrangements that realize it. Each test word set crosses four arrangements, four content assignments, and twelve directed changes: 192 cases per word set, 12,288 per model and condition. Intervention.We edit the query state, the state of the last prompt token, at the output of one block, once, while the prompt is processed. All later blocks and positions are recomputed on the receiving context, and generation continues from there. Prompts use each model’s native chat template (Appendix G.2 adds a plain prompt), and computation is in FP32. Random controls are two fixed random directions with the length of the ordinal vector they replace. Theceilingreplaces only the receiver’s last-token state at the edited block with its own state for the designated question푞 ′ ; the question tokens still read푞, so it measures how far replacing this one state moves the answer. Decoding and scoring.Answers are greedy, with at most 16 new tokens. We score the complete normalized answer string: thedesignatedanswer is the content that the designated question푞 ′ asks for in the receiving context. An answer isinvalidwhen its normalized string is not exactly one content of the context. Invalid and truncated answers remain in every denominator, as do cases whose unedited answer is wrong. Intervals.Intervals are 95% percentile intervals from 5,000 bootstrap draws over test word sets, keeping all content assignments, arrangements, questions, and paired conditions of a word set together. For donor states, donor and receiver word sets are resampled independently. All intervals are pointwise, one per reported rate. 15 The Geometry of Contextual Relations: Language Models Address Facts by Order of Mention In the text, intervals are given in brackets; in tables,±gives half the interval. Differences between rates are in percentage points (“points”). B. Invariance and Reversal in Two-Fact Contexts Section 3 claims that in two-fact contexts the query change between the two people’s questions keeps its direction under new names and reverses with the order of mention (Proposition 3.4); this section gives the materials, the coordinates of Figure 2c, and the cosines of Figure 2d. Materials.Each context states two facts, such asRuby eats the tunaandSimon eats the guava, and asksWhat does Ruby eat?There are 72 word sets, each with two names and two pairs of contents, and a context uses one pair: 8 word sets calibrate, and 64 are test word sets. All 144 names are distinct, calibration and test contents disjoint, and a word set’s contents start with different letters. Each word set is realized in the active form and in the passive form (The tuna is eaten by Ruby.), with both content assignments and both arrangements, and the question follows the facts. Every model answers every calibration question correctly. Coordinates.Label the two people퐴and퐵within each word set; the labels stay with the people when their sentences are reversed. For arrangement표∈0,1(표= 0:퐴’s fact first), let푣 표 be the mean ofℎ(퐶,푞 퐵 )−ℎ(퐶,푞 퐴 ) over both content assignments and the eight calibration word sets, so the ordinal vector is푣=푣 0 . Figure 2c uses three axes fit on calibration word sets only: the arrangement contrast푢 1 ∝푣 0 −푣 1 , close to푣 0 because푣 1 ≈−푣 0 ; the component푢 2 of(푣 0 +푣 1 )/2orthogonal to푢 1 (“which people are named”); and the top principal direction of the calibration context offsets 1 2 (ℎ(퐶,푞 퐴 ) +ℎ(퐶,푞 퐵 ))after projecting out푢 1 ,푢 2 . States are centered by the mean calibration offset and expressed in units of‖푣‖, the root-mean-square norm of푣 0 ,푣 1 . Figure 2c draws the offset axis stretched for display; its tick labels give measured values. The eight displayed test contexts are the active contexts with the first content assignment at the 10th, 37th, 63rd, and 90th percentiles of the offset coordinate, four in each arrangement. Along푢 2 , test pairs separate by whether a woman or a man is named first, because every calibration pair names a woman first:푢 2 carries which two people are named, while푢 1 , the arrangement contrast within the same pairs, carries the order of mention. The plane of푢 1 ,푢 2 holds91.7%,88.8%, and94.8%of the squared norm of the test changes in Qwen, Gemma, and Llama. Cosines.Figure 2d uses full-dimensional states. For each of the 64 test word sets, the change of that word set in a given arrangement has mean cosine0.93,0.88, and0.92with the calibration푣 표 of the same arrangement (minimum 0.85,0.71, and0.78). The changes calibrated in the two arrangements are nearly opposite:cos(푣 0 ,푣 1 ) =−0.93, −0.82, and−0.85. The query change keeps its direction when the names change and flips when the order of mention flips, as a difference of two fact addresses should. C. Questions Are Organized by Order of Mention Section 4 claims that the query-dependent state is set by the queried fact’s order of mention, not by the named person, also when one fact has two subjects and when facts have two arguments; this section gives the query-state geometry and the tests behind that claim. C.1. Query-State Geometry Query states are organized by the order of mention of the queried fact (Figure 3a–c): fact-address templates predict held-out states, and person templates predict nothing. 16 The Geometry of Contextual Relations: Language Models Address Facts by Order of Mention Data.This geometry and that of Figure 13 use last-token states saved at every block for the four-person contexts behind Table 1C. Calibration contexts are 16 word sets×4 content assignments×4 arrangements, 256 contexts, each with all four questions; because the arrangement rotates, every person occupies every order of mention. Test contexts are the 64 test word sets×2 content assignments in one arrangement, 128 contexts. For every block, the query-dependent state is푦(퐶,푞) =ℎ(퐶,푞)− 1 4 ∑︀ 푞 ′ ℎ(퐶,푞 ′ )at the last prompt token. Templates and held-out variance.A fact-address template푎 푝 averages푦(퐶,푞)over calibration contexts by the order of mention푝of the queried fact; a person template푏 푒 averages by the queried person푒. Held-out variance explained is푅 2 = 1− ∑︀ ‖푦−푎 푝(푞) ‖ 2 / ∑︀ ‖푦‖ 2 , and likewise for푏 푒 ; a negative value means the template predicts worse than zero. We report the leave-one-word-set-out estimate on calibration contexts, here and at every block of Figure 13: templates fit on 15 word sets predict the held-out word set, and sums are pooled over the 16 held-out word sets. Only calibration contexts can separate the two templates: test contexts keep one arrangement, in which person and order of mention coincide. At the main blocks the estimate is0.66,0.29, and0.49for the fact-address template and−0.01for person in all three models. Projection.The plane of Figure 3a–c is spanned by the top two principal directions of the four fact-address templates fit on eight calibration word sets. The points are the query-dependent states of the other eight calibration word sets, unseen by the fit, in all four arrangements and content assignments. Each model contributes 512 points, and the axis labels give the share of their variance on each displayed direction. Within one word set.Across word sets the names change, so a person template can only carry what the푘-th names of different word sets have in common. Inside one word set the four names and contents stay fixed while its 16 contexts put every person at every order of mention. For each calibration word set and each held-out context, we fit templates on the other 15 contexts of that word set and score the held-out context’s four query-dependent states, pooling over the 16 word sets (Figure 7). At the first block the state is set by the person the question names (푅 2 0.93,0.99, and0.98in Qwen, Gemma, and Llama); this share falls to zero as the share of the fact address rises. The fact address peaks inside the window of Figure 5 in Qwen and Gemma (blocks 18 and 19) and earlier in Llama (block 9), and the requested content, a template that averages by the content the question asks for, takes over in later blocks. At the main blocks the queried person explains−0.04,−0.03, and0.02of the variance and adds 0.07,0.02, and0.10to a fact-address template, which alone explains0.64,0.22, and0.44. The requested content is already present there in Gemma (0.43), whose main block ends its window. At the main blocks the fact address, not the name, sets the state. C.2. Coordinated Subjects: The Order of Facts, Not of Names The address belongs to the fact, not to the name (Sections 4 and 5). In every other experiment each fact has one subject, so the푘-th fact and the푘-th name coincide. Here one of four facts has two coordinated subjects,Alice and Bob eat the apple. Carol eats the pear. Dave eats the plum. Emma eats the fig., so five names are mentioned in four facts and, after the coordinated fact, a person’s name order exceeds the order of their fact by one. Calibration uses ordinary four-fact lists of 16 word sets in four arrangements and four content assignments: fact-address templates 푎 푝 of the query-dependent state by the queried fact’s order of mention, and the twelve ordinal vectors푣 푝→푝 ′ . Test contexts come from 64 new word sets of five people and four multiword contents, with the coordinated fact at each of the four orders of mention and two content assignments, and every single-person question is asked: 2,560 questions, of which 1,280 have a fact order푓that differs from their name order푚. Geometry.For each question with푓̸=푚and푚≤4(768 questions), the query-dependent state푦is compared with the template of its fact order,푎 푓 , and with the template of its name order,푎 푚 , in the address subspace. It is closer to푎 푓 in99.2% [98.6,99.7]of questions in Qwen,96.2% [94.9,97.5]in Gemma, and96.2% [94.8,97.5] 17 The Geometry of Contextual Relations: Language Models Address Facts by Order of Mention 081624 Transformer block 0.0 0.5 1.0 Within-word-set R 2 08162432 Transformer block 0.0 0.5 1.0 08162431 Transformer block 0.0 0.5 1.0 a Qwen2.5-7B b Gemma-3-4B c Llama-3.1-8B queried personfact addressrequested content Figure7:Inside one word set, the query state passes from the named person to the fact address to the requested content.Held-out푅 2 of query-dependent states under templates for the queried person, the queried fact’s order of mention, and the requested content, each fit on the other 15 contexts of the same word set and pooled over 16 calibration word sets. Shading: first to last block where the ordinal vector selects at least10%of designated answers (Figure 5). in Llama, and the mean advantagecos(푦,푎 푓 )−cos(푦,푎 푚 )is0.94,0.61, and0.68. The full space gives the same fractions. Held-out calibration questions are assigned to their own template in100%,100%, and99.4%of cases. Behavior.For receivers whose fact and name orders differ and each target order of mention푡, an ordinal vector chosen by fact order,푣 푓→푡 , is compared with one chosen by name order,푣 푚→푡 . Only targets for which the two orders predict different contents, and for which the name-order prediction is not the original answer, enter the contrast. The fact-order vector selects the content of fact푡in65.3% [63.0,67.7]of cases in Qwen,34.0% [32.4,35.6] in Gemma, and12.1% [10.5,13.7]in Llama. The name-order vector selects the content its order predicts in0.0%of cases in all three models and mostly leaves the original answer. On receivers whose two orders agree, the same vectors select their target in78.2%,57.7%, and21.9%of cases, and the ceiling in97.9%,97.1%, and84.7%; random directions keep the original answer in at least99.9%of cases, and unedited answers are correct in at least99.9%. An ordinal vector acts only when it starts at the state’s own fact address, so both measures place each question at the address of its fact, not of its name. C.3. Two-Place Relations Queries over two-place facts are organized by fact and argument slot, not by person (Section 4). Each context states three facts of one two-place relation among six people, such asAlice loves Bob. Carol loves David. Emma loves Frank., with five relations (loves,trusts,helps,visits,pays). A question asks for one argument given the other,Whom does Alice love?orWho loves Bob?Every answerable question is a pair of an order of mention and an argument slot, so a query change can move to another fact (same slot), to the other argument of the same fact, or to both. Calibration uses 16 word sets with all arrangements and all content assignments, whose contents are people. The 64 test word sets, with new names whose first tokens are distinct, are edited once at the main block with each calibration mean, the means of the other four relations, two random directions, and the ceiling. Templates are fit on calibration word sets and scored on the test word sets. All three changes transfer to new names: fact, slot, and joint changes select the designated person in82%, 52%, and79%of Qwen cases,49%,65%, and54%of Gemma cases, and22%,34%, and34%of Llama cases, while random directions stay at or below1.7%(Figure 8a). For fact changes, a change calibrated on the other four relations comes within four points of the receiving relation’s own. Query states are organized by the two 18 The Geometry of Contextual Relations: Language Models Address Facts by Order of Mention factslotboth 0 50 100 Designated answers (%) ceilingrandom fact ×slot fact +slot factslot person 0.0 0.4 0.8 QwenGemmaLlama factslot 0.0 0.5 1.0 Cosine a Change fact, slot, or both b Held-out R 2 by template c Shared across relations Figure8:Queries over two-place facts address a fact and an argument slot.(a) Designated answers for changes of fact, of argument slot, and of both, against the ceiling (gray ticks) and random directions (×), on 64 new active contexts. (b) Held-out푅 2 under fact-by-slot, additive (fact+slot), fact-only, slot-only, and person templates. (c) Cosines of fact and of slot changes across relations. In (b) and (c), small markers: individual relations; large markers: their mean. coordinates: a fact-by-slot template explains0.82,0.65, and0.70of held-out variance, an additive template0.77, 0.58, and0.63, and the queried person nothing (Figure 8b). Fact changes estimated on different relations point the same way (mean cosine0.94–0.96), and so do slot changes (0.77–0.88). A question about a two-place fact addresses the fact by its order of mention and, within it, the argument slot it asks for. D. The Fact Address Transfers across Contexts Section 5 claims that an ordinal vector from other contexts selects the designated fact in a new context, that the facts define this vector, that a whole donor state carries only which fact to read, and that the vector that transfers is chosen by the receiver’s orders of mention, not by its names, relation, or token distances. Random directions of each ordinal vector’s length select the designated fact in at most0.01%of the 12,288 cases per model (Table 1A). We give the controls of Table 1B–C, then which vector transfers under rearranged facts, six relations, and token distance. D.1. What the Facts and Whole Donor States Contribute No-facts directions.Table 1B compares directions from different inputs at one length. The ordinal vector for a directed pair of orders of mention is the calibration mean of Equation (1) over eight calibration word sets, four arrangements, and four content assignments (128 query changes per pair). Each no-facts direction is the difference between the states for the two questions of the case, computed on an input without the facts: the question alone; the receiving names listed in the order of their facts; or the same names in a shuffled order. Every direction, including the random ones, is scaled to one common length, the root-mean-square length of the twelve factual calibration directions. Each of 64 test word sets contributes two arrangements, two content assignments, and four directed changes, 1,024 cases per row. These word sets and this subset of arrangements and content assignments differ from those of Table 1A, so the two panels’ rates are not directly comparable. The no-facts inputs contain the question words and the names but no facts, and their directions select at most3.9%of designated answers. Whole donor states.In Table 1C, the donors are the 16 calibration word sets and the receivers the 64 test word sets, all in one fixed arrangement. For each donor, receiver, content assignment, and directed person change푠→푡, the receiver asks question푠, and its state for푠is replaced by the donor’s state for question푡(another fact) or 19 The Geometry of Contextual Relations: Language Models Address Facts by Order of Mention for question푠(the receiver’s own question); in both rows the designated answer is the receiver’s fact at푡, which gives 8,192 cases per condition. The paired gain of the first replacement over the second is79.1 [75.4,82.6],58.1 [54.4,61.9], and45.2 [42.0,48.2]percentage points, and it is positive for every donor: the smallest per-donor gain is 68.0,50.2, and38.5points. On the same prompts, which are also those of the depth curves of Figure 5, the ordinal vector selects55.5%,44.3%, and25.8%, below the whole donor state and below its rates in Table 1A (84.1%,53.9%, and36.2%), which average over all arrangements. Complete donor answers are rare at the main blocks (Figure 5), and at the final block the same replacement gives them in only2.1%,8.2%, and3.6%of cases, with91.0%,87.1%, and79.5%invalid (Appendix F.1). A donor state carries which fact to read, not the donor’s content. D.2. Rearranged Facts When facts move, the ordinal vector must follow their new orders of mention (Section 5). Each context states four facts about two people and two relations, (Alice eats the apple. Alice carries the pear. Bob eats the plum. Bob carries the peach.), and a question asks about one of the fourcells, a pair of a person and a relation; a person change keeps the relation, and a relation change keeps the person. Calibration uses eight word sets in the person-major arrangement above (grouped by person). The test contexts, on 64 word sets, state the facts in two held-out arrangements,(1,4,2,3)and(3,2,4,1), numbering the facts of the person-major arrangement, with four content assignments and four starting questions: 2,048 cases per condition and change type. Three edits target the same designated answer. Theoriginal-pairedit keeps the ordinal vector for the same pair of questions, which in the new arrangement connects different orders of mention. Thenew-order-of-mentionedit takes the ordinal vector for the pair of orders of mention that the two questions occupy in the new arrangement. Therecalibratededit is estimated on calibration word sets in the held-out arrangement. Table 2:When facts move, the ordinal vector must follow their orders of mention.Designated answers (%) under two held-out arrangements, 64 test word sets and 2,048 cases per entry;±is half the 95% word-set bootstrap interval; bold: the new-order-of-mention edit; ceiling: the receiver’s own state for the designated question; no facts: the direction computed from the question alone. Model Change Original pair New order of mention Recalibrated Ceiling Random No facts Qwen Person9.0±1.169.7±2.380.9±2.590.2±1.70.00.0 Relation22.6±1.661.6±2.168.5±1.977.3±1.95.77.0 Gemma Person17.6±0.940.8±1.461.1±2.179.2±1.70.10.1 Relation5.5±1.319.9±2.941.5±2.873.6±1.60.80.7 Llama Person6.2±1.415.7±2.816.4±2.829.8±3.70.00.0 Relation0.7±0.50.6±0.32.8±1.111.8±2.10.30.3 Choosing the ordinal vector by the new orders of mention recovers most of the recalibrated rate in Qwen and nearly all of it for Llama’s person changes, and it more than doubles Gemma’s rates while leaving a larger gap to recalibration (Table 2). Llama’s relation changes do not act in these contexts even when recalibrated, and even the ceiling reaches only11.8%. In Qwen, unedited answers toeatsquestions sometimes name the same person’s carried item, so6.4%of relation-change cases already give the designated answer before any edit; random and no-facts directions leave that rate unchanged (gains of−0.7and+0.5points), which is why their relation row is not zero. D.3. Six Relations The same ordinal vector acts under every relation (Section 5). Each context lists four facts under one of six relations, each with its own contents:eatsa dish,ownsan animal,visiteda city,works asa job,playsan instrument, orplays a sport. The question asks for one person’s content under the same relation, and the model answers with only that content. Each relation has 16 calibration and 64 test word sets, with new names and contents in the test word sets. Calibration follows Section 5: for each of the twelve directed changes between orders of mention, the 20 The Geometry of Contextual Relations: Language Models Address Facts by Order of Mention mean query change over a relation’s calibration contexts at the main block. Each test receiver is edited once with the ordinal vector of every relation, with theeatsvector rescaled to the receiving relation’s own change length, with two random directions of that length, and with the ceiling. Other cross-relation changes keep the source relation’s length, so their gap to the receiving relation’s own change mixes direction and length. Geometry uses the query-dependent states at this block, with templates fit on calibration and tested on test word sets. Unedited answers are correct in at least97.5%of cases for every relation and model, and the ceiling reaches 88–99%. Random directions select the designated fact in at most0.03%of cases. Averaged over relations, the receiving relation’s own vector selects the designated fact in72%,47%, and18%of Qwen, Gemma, and Llama cases, and a vector from another relation in70%,46%, and16%(Figure 9a). Rescaled to the receiving relation’s length, theeatsvector comes within five points of each relation’s own vector in every model (paired differences from−4.0to+4.6points). Fact-address templates give held-out푅 2 of0.61–0.71,0.26–0.35, and0.41–0.52, while person templates stay within0.01of zero (Figure 9b). The ordinal vectors of two relations point the same way, with cosines of0.79–0.96over relation pairs and models, against0.18–0.38between mismatched ordinal vectors (Figure 9c); the address subspaces of the six relations nearly coincide as well (Appendix E.1). The relation changes the contents at an order of mention, not the ordinal vector between them. eatsownsvisited works as plays instr. plays sport 0 50 100 Designated answers (%) same relation other relation random Qwen Gemma Llama QwenGemmaLlama 0.0 0.4 0.8 filled: fact address open: person QwenGemmaLlama 0.0 0.5 1.0 gray: mismatched orders of mention a Transfer by receiving relation b Held-out R 2 c Ordinal vectors across relations Figure9:The same ordinal vector acts under every relation.(a) Designated answers by receiving relation: an ordinal vector calibrated on the same relation (filled), on each of the five other relations at its own length (open), and random directions (×); 64 test word sets, 3,072 cases per point. Not plotted: theeatsvector rescaled to the receiving relation’s change length comes within five points of the filled marker for every relation (paired differences −4.0to+4.6points). (b) Held-out푅 2 of query states under fact-address (filled) and person (open) templates, one point per relation. (c) Cosine between ordinal vectors fit separately on two relations, one point per relation pair, averaged over pairs of orders of mention; gray bars are the cosine between mismatched ordinal vectors. D.4. Order of Mention versus Token Distance The ordinal vector follows order of mention, not token distance (Section 5). When all facts have the same length, the푘-th fact is also a fixed number of tokens before the question, so an address by order of mention cannot be told apart from one by distance. Each context of 64 new four-person word sets is rendered under eight length conditions with the same arrangement and content assignments: uniform facts; one fact lengthened by a neutral opening clause (four conditions); the second and fourth facts lengthened; filler lines after facts 1 and 3; and a filler paragraph before the question. Ordinal vectors are calibrated on uniform contexts from 16 other word sets, or on mixed-length contexts, and applied unchanged. For an edit from order of mention푝to푝 ′ on a question about fact푠, let푑(푠)be the token distance of푠from the question and푑 ref [푝]the distance of order푝in uniform contexts. Theabsolutedistance address selects the fact whose distance is nearest푑 ref [푝 ′ ]; therelativeone selects the fact whose distance is nearest푑(푠) +푑 ref [푝 ′ ]−푑 ref [푝]; ties go to the fact nearer the question. Both rules and the contrast were fixed before testing: answers naming the 21 The Geometry of Contextual Relations: Language Models Address Facts by Order of Mention uniform long fact 1 long fact 2 long fact 3 long fact 4 long facts 2, 4 filler lines after 1, 3 filler paragraph 0 50 100 Designated answers (%) QwenGemmaLlamauniform calib.mixed calib.ceilingrandom absolute distance relative distance 0 50 100 Paired difference (points) a Designated answers by length conditionb Order minus distance Figure10:The ordinal vector follows order of mention, not token distance.(a) Designated answers by length condition for ordinal vectors calibrated on uniform (filled) or mixed-length (open) contexts, with the ceiling (gray ticks) and random directions (×). (b) Answers naming the fact at the calibrated order minus those naming the fact at the calibrated absolute or relative distance, on edits where the two facts differ from each other and from the unedited answer; markers as in (a), with 95% intervals over 64 word sets. fact at the calibrated order minus answers naming the distance-predicted fact, on edits where the two facts differ from each other and from the fact the unedited question asks about. Transfer survives every length condition, from48%to71%in Qwen, from47%to55%in Gemma, and from 12%to28%in Llama, with random directions at zero (Figure 10a). On edits where order and distance predict different facts, the ordinal vector selects the fact at the calibrated order far more often than the fact at the calibrated distance: by72.1 [70.0,74.0]points in Qwen,58.9 [57.0,60.9]in Gemma, and28.4 [26.6,30.1]in Llama against absolute distance, and by60.7,65.2, and17.6points against relative distance. Mixed-length calibration gives the same verdict. Held-out query states are fit by fact-address templates (푅 2 0.52,0.26, and0.41) and not by a token-distance template, the fact-address template of the order of mention whose mean token distance from the question in uniform-length contexts is nearest the queried fact’s distance (0.02,−0.11,−0.15); both templates are fit on calibration word sets and scored on the test word sets. Transfer counts facts, not tokens. E. Fact Addresses Are Low-Rank Section 6 claims that fact addresses span a shared low-rank subspace, crowd together in longer lists, are fixed once their fact is read, and predict how person and relation changes combine; this section gives the analyses behind each claim. E.1. The Address Subspace Fact addresses span a shared low-rank subspace (Section 6): the three-dimensional span of the fact-address templates holds nearly as much of each held-out query change as the best three-dimensional subspace. The numbers of Section 6 use the main-block states of Appendix C.1, with templates fit on all 16 calibration word sets; Figure 4a, like Figure 3, fits the templates on eight of them and shows the states of the other eight. Share of held-out changes.With the fact-address templates푎 푝 of Appendix C.1 fit on the 16 calibration word sets, the address subspace is퐴= span푎 푝 −푎 1 , three-dimensional for four facts. For each of the 128 test contexts and each of its six pairs of orders of mention, the per-context change is∆ 퐶 (푝→푝 ′ ) =ℎ(퐶,푝 ′ )−ℎ(퐶,푝), where ℎ(퐶,푝)is the state for the question about the fact at order of mention푝(the reverse change has the same share). Its share inside퐴is‖푃 퐴 ∆ 퐶 ‖ 2 /‖∆ 퐶 ‖ 2 , averaged over the 768 changes:69%,33%, and52%in Qwen, Gemma, and 22 The Geometry of Contextual Relations: Language Models Address Facts by Order of Mention Llama. Two references bound this share. The best three-dimensional subspace for these changes, the top three principal directions of the held-out changes themselves, holds70%,41%, and60%; this is an in-sample upper bound. One hundred random three-dimensional subspaces hold0.1%on average, as expected for dimensions 3,584, 2,560, and 4,096. Relative to the gap between random and best,퐴reaches99%,81%, and87%. Across relations.For each relation푟of the six in Appendix D.3, a separate subspace퐴 푟 is fit on that relation’s 16 calibration word sets at the main block. For two relations푠and푡, the principal angles between퐴 푠 and퐴 푡 give three cosines. The smallest of them, the cosine of the largest principal angle, averaged over the 15 relation pairs, is0.89,0.78, and0.88(ranges over pairs0.83–0.94,0.66–0.90, and0.80–0.94); for random three-dimensional subspaces, even the largest of the three cosines is only about0.05. Composition on disjoint word sets.Write푣 푆 (푝→푝 ′ )for the mean of∆ 퐶 (푝→푝 ′ )over the calibration contexts of a subset푆of word sets. The 16 calibration word sets are split into two halves, and one half into two disjoint quarters푆 1 ,푆 2 . For each of the 24 ordered triples푝→푝 ′ →푝 ′ of distinct orders of mention, we compare the target푣 푇 (푝→푝 ′ )on the other half with the sum푣 푆 1 (푝→푝 ′ ) +푣 푆 2 (푝 ′ →푝 ′ ), so the three vectors come from disjoint word sets and no identity links the sum to the target. Averaged over triples and over the assignment of halves and quarters, the cosine is0.99,0.97, and0.98, and the relative error‖sum−target‖/‖target‖is0.15,0.23, and0.18. The null relabels the summands over the 23 non-identity relabelings of the orders of mention1,2,3,4: cosine−0.09and relative error1.45–1.56in all three models. The target estimated directly on푆 1 ∪푆 2 agrees with 푣 푇 (푝→푝 ′ )at cosine0.995,0.989, and0.995, the noise ceiling. Ordinal vectors add along paths, as Corollary 3.7 predicts. E.2. Longer Lists In longer lists, fact addresses form an ordered chain whose later links shorten, and transfer is highest to the first fact (Section 6). The transfer test of Section 5 is repeated with lists of four, six, and eight single-relation facts, one person per fact, in all three models at their main blocks. Each test word set has eight people and eight contents, and a list of length푁uses the first푁of each, so the three lengths share names and contents; within a word set every person and every content appears once at every order of mention. Calibration uses 16 word sets and every combination of content assignment and arrangement; all푁(푁−1)directed pairs of orders of mention are calibrated and tested (12, 30, and 56) on 64 test word sets, each in its푁cyclic arrangements. This experiment draws its names and contents from a separate pool, on which four-fact transfer is lower than in Table 1A (Qwen 75.6%versus84.1%). Figure 11 shows two regularities. First, the ordinal vector transfers at every length, far above random directions: overall75.6,50.4, and33.7%in Qwen for four, six, and eight facts,48.4,30.2, and21.1%in Gemma, and20.1, 11.0, and7.2%in Llama, while the ceiling stays high at every length (Figure 11). Second, transfer depends on the designated fact’s order of mention: it is highest for the first fact (Qwen82–95%), falls toward the middle, and rises again at the last fact in eight of the nine lists, a serial-position curve with a strong primacy effect [14]. The chain of addresses.For lists of푁= 4,6,8facts, the templates푎 1 ,...,푎 푁 average the query-dependent states of the 16 calibration word sets, over all푁arrangements and푁content assignments, by the order of mention of the queried fact; the templates from two halves of these word sets have mean cosine at least0.98. Lengths are in units of‖푎 2 −푎 1 ‖at each푁. Figure 4b projects the eight templates for푁= 8on their top two principal directions, which hold58%,84%, and77%of their variance. Figure 4c plots‖푎 푝+1 −푎 푝 ‖; for푁= 8the last link is 0.44,0.13, and0.25of the first. Figure 4d plots, for each model, length푁, and order of mention푝(54 points), the distancemin 푝 ′ ̸=푝 ‖푎 푝 −푎 푝 ′ ‖from푎 푝 to its nearest other address against the transfer rate to the fact at푝on the test word sets (Figure 11). Subtracting each model’s mean distance and mean rate before the least-squares fit gives 푅 2 = 0.82(Spearman0.89); without centering,푅 2 = 0.78. 23 The Geometry of Contextual Relations: Language Models Address Facts by Order of Mention 12345678 designated fact's order of mention 0 50 100 Designated answers (%) a Qwen2.5-7B 12345678 designated fact's order of mention 0 50 100 b Gemma-3-4B 12345678 designated fact's order of mention 0 50 100 c Llama-3.1-8B 4 facts6 facts8 factsceiling Figure11:Transfer falls with the designated fact’s order of mention and is highest for the first fact.Designated answers for the ordinal vector by the designated fact’s order of mention, in lists of four, six, and eight facts, with word-set bootstrap 95% intervals (bands, mostly narrower than the markers); dashed lines are the ceiling. Random directions give at most0.1%at every length. E.3. Prefixes of a List Each fact address is fixed once its fact is read: removing later facts leaves earlier addresses in place (Section 6). Every eight-fact context of Appendix E.2 (the 16 calibration and 64 test word sets, eight contexts each: the eight cyclic arrangements, each with one content assignment that varies across word sets) is cut after its first푘facts, 푘= 2,...,8, and asked about each of its푘facts, with states at the main blocks. Templates푎 (푘) 푝 are computed within each prefix length from all 80 word sets as in Appendix C.1 and compared only through their differences, which do not depend on how the query-dependent states are centered. For facts before the prefix’s last, the vector from fact 1 to fact푝has median cosine0.99with its eight-fact counterpart in every model (minimum0.95), and its length changes by at most8%. The gap between neighboring addresses, relative to the first gap, depends on the fact’s order of mention and not on the prefix length: in Gemma, gaps 2 to 4 are0.62–0.66,0.41–0.43, and0.18–0.20at every푘, and the first gap is constant in absolute size. Gap sizes indexed by order of mention, fit on the other prefix lengths, predict the gaps of a held-out prefix length with 푅 2 = 0.97,0.99, and0.91in Qwen, Gemma, and Llama; indexed by distance from the list’s end,푅 2 <0. The last fact of a prefix deviates slightly (median cosine0.91,0.97, and0.94), and in Llama its gap is larger by0.075of the first gap, a recency effect. E.4. Person and Relation Do Not Add Fact addresses predict that a person change and a relation change need not add up to the joint change, and they predict the direction of the gap (Section 6). Contexts state four facts about two people and two relations,Alice eats the apple. Alice carries the pear. Bob eats the plum. Bob carries the peach.Index the question cells (Appendix D.2) by person and relation,(푒,푟)∈0,1 2 , and let푝 표 (푒,푟)be their fact’s order of mention under arrangement표. Lemma E.1(Fact addresses induce a person–relation interaction).Under Hypothesis 3.3 with휀= 0, the query state of the question about cell(푒,푟)isℎ 푒푟 =ℎ ℓ (퐶,푞 푒푟 ) =푐 퐶 +푎 푝 표 (푒,푟) , and the four-cell contrast of these states is 퐼=ℎ 11 −ℎ 10 −ℎ 01 +ℎ 00 =푎 푝 표 (11) −푎 푝 표 (10) −푎 푝 표 (01) +푎 푝 표 (00) .(4) The right-hand side need not vanish and can change with the arrangement. Proof. Substituteℎ 푒푟 =푐 퐶 +푎 푝 표 (푒,푟) into the four-cell contrast. The context term has coefficients+1−1−1 + 1 = 0, leaving Equation (4). For a scalar example, take푎 1 = 0,푎 2 = 1,푎 3 = 2,푎 4 = 4, and place the cells00,01,10,11at 24 The Geometry of Contextual Relations: Language Models Address Facts by Order of Mention orders of mention1,2,3,4. The contrast is4−2−1 + 0 = 1. Exchanging the orders of mention of the last two cells changes it to2−4−1 + 0 =−3, although neither case has a separate person-by-relation term. Joint versus summed changes.The contexts above appear in four arrangements: grouped by person, grouped by relation, and their reversals. Calibration uses 16 word sets, and each change is the calibration mean for its pair of question cells within the same arrangement. From a starting cell,푣 퐸 changes the person,푣 푅 the relation, and 푣 퐽 both; thesame-start sumis푣 퐸 +푣 푅 , with both terms measured from the starting cell. Each of 64 test word sets contributes four arrangements, content assignments, and starting cells each: 4,096 cases per condition, all scored against the joint target. Table 3:The joint change selects the joint target more often than the sum of its parts.Designated answers (%), 64 test word sets, 4,096 cases per edit;±: half the 95% paired word-set bootstrap interval, in points; bold: the joint change. EditQwen GemmaLlama Joint change푣 퐽 68.557.126.0 Same-start sum푣 퐸 +푣 푅 43.844.018.8 Sum rescaled to‖푣 퐽 ‖35.331.714.1 Ceiling (own state for the joint question)87.990.756.7 Joint minus sum24.7±1.313.1±1.47.2±0.9 Joint minus rescaled sum33.2±1.525.4±1.411.9±1.0 The sum is longer than the joint change, so its shortfall is not a lack of length: rescaling it to‖푣 퐽 ‖lowers its rate further (Table 3). The person and relation changes are nearly orthogonal (cosine0.11,0.13, and0.12), and their sum has cosine0.88,0.86, and0.89with푣 퐽 ; the interaction퐼=푣 퐽 −푣 퐸 −푣 푅 , the four-cell contrast of Lemma E.1, has54%,59%, and51%of the norm of푣 퐽 . The interaction predicted by fact addresses.For the prediction of Lemma E.1, fact-address templates are fit on person-by-relation contexts from 16 calibration word sets, rendered in the twelve even permutations of the four facts with four content assignments. Held-out contexts are 64 test word sets rendered in the twelve odd permutations, which calibration never used, and in the original arrangement (an even permutation, also used in calibration; its word sets are held out), with four content assignments: 3,328 contexts. In each, the measured interaction 퐼=푦 11 −푦 10 −푦 01 +푦 00 of the query-dependent states is compared with푎 푝 표 (11) −푎 푝 표 (10) −푎 푝 표 (01) +푎 푝 표 (00) computed from the templates and the orders of mention of the four cells in that context’s arrangement. The mean cosine is0.47,0.38, and0.46at the main blocks (Figure 12). Relabeling the templates by each of the 23 non-identity relabelings of the orders of mention gives−0.02in every model, and at the final block the cosine falls to0.07,0.04, and0.07. The measured interaction points toward the one fact addresses predict, far above relabeled orders, and the agreement vanishes at the output. F. Where and When the Fact Address Is Shared Section 7 claims that the fact address is shared as a difference only within a depth window, and that it forms during pretraining; this section gives the every-block design and the checkpoint tests. F.1. The Depth Window The fact address is shared as a difference only within a depth window, after which whole states still carry it and then the donor’s content takes over (Section 7). The depth curves in Figure 5b–d repeat the donor design of Appendix D.1 at every block of all three models, on the prompts of Table 1C: the 16 calibration word sets as donors, 25 The Geometry of Contextual Relations: Language Models Address Facts by Order of Mention −1.0−0.50.00.5 PC 1 (39% of variance) −0.5 0.0 0.5 PC 2 (28%) predicted fact 1→2 1 2 3 4 −1.0−0.50.00.5 PC 1 (22% of variance) −0.5 0.0 0.5 PC 2 (10%) 1 2 3 4 −1.0−0.50.00.5 PC 1 (34% of variance) −0.5 0.0 0.5 PC 2 (19%) 1 2 3 4 0.00.51.0 cosine: measured vs predicted person×relation interaction shuffled −0.02 Qwen Gemma Llama original pair new order of mention re- calibrated 0 50 100 Designated answers (%) owns visitedworks as plays instr. plays sport 0 50 100 Designated answers (%) a Qwen2.5-7B, block 20 fact-address R 2 0.66 person R 2 −0.01 b Gemma-3-4B, block 22 fact-address R 2 0.29 person R 2 −0.01 c Llama-3.1-8B, block 15 fact-address R 2 0.49 person R 2 −0.01 Predicted interaction b Rearranged facts c ‘eats’ vector, other relations QwenGemmaLlamaperson changerelation changeown-relation vector Figure12:Fact addresses predict the person–relation interaction.Distribution, over 3,328 held-out contexts per model, of the cosine between the measured interaction퐼of the query-dependent states (Equation (4)) and its prediction from fact-address templates and the context’s orders of mention; means0.47,0.38, and0.46in Qwen, Gemma, and Llama. Dashed line: the mean cosine when the order labels are shuffled (−0.02). crossed with the 64 test word sets as receivers in one fixed arrangement, two content assignments, and four directed person changes. At every block the conditions are the unedited question, the ordinal vector re-estimated at that block from the calibration word sets, the ceiling, and the donor’s state for the other question; the main and final blocks add the donor’s state for the same question (Table 1C). Intervals for donor conditions resample donors and receivers independently; intervals for the other conditions resample receivers. The example.Figure 5a shows the first case of Gemma’s block-selection word sets: a receiver asked about its first fact (apple crumble) takes the state of a donor asked about its second (baked trout), and answerspear tart, its own second fact, at block 22, butbaked apple crumble, the donor’s first word followed by its own content, at block 33. Scoring donor outputs.Each donor output is scored three ways: it names the receiver’s designated content, it is the donor’s own complete answer, or it begins with the first token of the donor’s answer. Past the window most outputs begin with the donor’s first word and continue with receiver content, as inbaked apple crumble; such outputs name no content of either context and count as invalid. First-token agreement therefore measures when the donor’s content takes over, and Figure 5 reports it alongside complete answers. Complete donor answers remain rare at every block, at most2.1%in Qwen,9.8%in Gemma, and4.1%in Llama. Answer alignment.For the alignment curves of Figure 13, the output direction of an answer in a context is the final-norm scale times the unembedding row of the answer’s first token, and the four directions of a context are centered. Alignment is the cosine between each query-dependent state and its own answer’s direction, averaged over test contexts; it uses only the unembedding and final-norm weights. It is near zero at the main blocks (0.001, 0.017, and0.004) and reaches0.26,0.12, and0.25at the final block. Past the window, states align with their own answer’s output direction. F.2. Pretraining Checkpoints In pretraining, questions switch from the named person to order of mention before the model can use the address, and the depth profile settles last (Section 7). Figure 6 repeats the tests on ten checkpoints of OLMo-2-7B [32]: stage-1 steps 1k, 4k, 10k, 25k, 50k, 99k, 200k, 400k, and 929k (5B to 3.9T training tokens) and the released model after stage-2 mid-training. The prompt is the plain format of Appendix G.2 with the tokenizer’s<|endoftext|> as the first token, which marks a document start in OLMo-2’s packed pretraining data. Each checkpoint runs 26 The Geometry of Contextual Relations: Language Models Address Facts by Order of Mention Receiver, asks about fact 1:What does Irene Sutton eat? fact 1Irene Sutton eats the apple crumble. fact 2Graham Terry eats the pear tart.(+2 more) Donor, asks about fact 2:What does Austin Gray eat? fact 1Angela Fletcher eats the grilled salmon. fact 2Austin Gray eats the baked trout.(+2 more) ordinal vector add v from other contexts → pear tart, its fact 2 (block 22) donor → receiver's fact copy the donor's state→ pear tart, its own fact 2 (block 22) donor → donor's first token copy the donor's state→ baked apple crumble (block 33) stage by blockshared addresswhole-state addresscontent Qwen2.5-7B Gemma-3-4B Llama-3.1-8B Qwen2.5-1.5B Qwen2.5-3B Qwen2.5-14B Qwen2.5-32B Qwen3-8B inputrelative depth (block / last block)output 081624 Transformer block 0 50 100 Answers (%) 081624 Transformer block 0.0 0.4 0.8 R 2 / alignment 08162432 Transformer block 0 50 100 08162432 Transformer block 0.0 0.4 0.8 08162431 Transformer block 0 50 100 08162431 Transformer block 0.0 0.4 0.8 a Three outcomes through depth: one example, and where each succeeds b Qwen2.5-7B: answers a Qwen2.5-7B c Gemma-3-4B: answers b Gemma-3-4B d Llama-3.1-8B: answers c Llama-3.1-8B own state (ceiling)ordinal vectordonor → receiver's factdonor → donor's first token fact-address R 2 answer alignment person R 2 Figure13:The fact-address geometry follows the stages of Figure 5.Query-state geometry at every block:푅 2 of fact-address and person templates (leave-one-word-set-out over the 16 calibration word sets, 256 contexts), and each query state’s alignment with its answer’s first-token output direction (128 test contexts). Shading: the depth window of Figure 5. the protocol of Appendix G.1 with the depth curves: its own rule-chosen block, the transfer test, the cosines, the query-state geometry at every block, and the depth curves at every second block (8 donors×32 receivers). Transfer and cosines are read at block 16, the released model’s rule-chosen block; checkpoints whose rule-chosen block differs are also evaluated at block 16. Early checkpoints answer in sentences, so answers are scored by the single stated content they name. A base checkpoint can name the designated answer without any edit (up to14%of cases), so the stages of Figure 6a apply the thresholds of Figure 5a to the gain over the unedited answer. Table 4:OLMo-2-7B checkpoints.Rule-chosen block; at block 16: unedited accuracy and designated answers (%) for the ordinal vector (bold), a random direction and the ceiling, scored by the single stated content an answer names, 64 test word sets and 12,288 cases per edit; new-name and reversed-order cosines of Proposition 3.4; and the peak leave-one-word-set-out fact-address푅 2 over blocks (its block).final: the released model after stage-2 mid-training. At block 16 Scored by stated content (%)Cosine Tokens Rule block Unedited푣Random Ceiling New names Reversed Peak푅 2 (block) 5B221.31.21.21.20.090.980.00 (31) 17B2224.916.316.016.70.370.250.29 (11) 42B1450.025.114.631.50.70−0.610.52 (13) 105B2263.923.711.327.60.76−0.740.61 (13) 210B1667.450.19.356.80.79−0.740.74 (13) 416B1679.946.06.355.90.80−0.710.71 (13) 839B1686.121.74.629.70.75−0.560.62 (13) 1.7T1691.523.52.035.90.80−0.650.69 (14) 3.9T2289.214.62.524.30.77−0.590.66 (13) final1692.433.11.861.80.85−0.780.71 (14) Table 4 gives the numbers behind Figure 6b,c. At 5B tokens the reversed-order cosine is+0.98: the change for a pair of people does not depend on their order, as expected if questions were represented by the named people. It falls to0.25at 17B and−0.61at 42B. Fact-address geometry precedes the ability to answer: at 17B tokens the model names the right content in24.9%of cases, the rate of guessing among four, yet fact-address templates already explain0.29of the variance. The depth profile moves throughout training (Figure 6a). Shared-address blocks appear at 42B, vanish at 105B, 27 The Geometry of Contextual Relations: Language Models Address Facts by Order of Mention and are present from 210B through 1.7T; content blocks appear from 416B. All three stages first co-occur at 839B and again at 1.7T, each time separated by blocks that reach no threshold. At 3.9T whole-state and content blocks remain, but no block reaches the shared-address threshold, and only the released model shows all three stages as one contiguous profile. Read at the fixed block 16, transfer therefore peaks at 210B (50.1%against a ceiling of56.8%) and falls later as the profile moves: the ceiling at block 16 drops from55.9%at 416B to24.3–35.9% from 839B to 3.9T and returns to61.8%after mid-training. Questions are organized by order of mention early in pretraining; the depth profile settles only late. G. Generality: Model Scale, Base Weights, and Prose The main text reports its tests on three instruction-tuned models and templated lists, and states that they hold from 1.5B to 32B parameters (Table 5), in base weights (Figure 2d), and in short paragraphs (Section 5); this section gives those tests on eleven further models and on prose: the instruction-tuned Qwen2.5-1.5B, Qwen2.5-3B, Qwen2.5-14B, Qwen2.5-32B, Qwen3-8B, and Llama-3.2-3B, and the base weights of Qwen2.5-1.5B, Qwen2.5-3B, Qwen3-8B, Gemma-3-4B, and Llama-3.2-3B. Materials, calibration and test word sets, and scoring follow Appendix A except where noted, and every rate is a share of complete greedy answers. G.1. Model Scale Ordinal addressing holds at every model size tested (Sections 5 and 7). We run Qwen2.5-Instruct at 1.5B, 3B, 14B, and 32B parameters and Qwen3-8B with its thinking mode switched off. Each model’s main block follows the rule of Appendix A, and the 7B model keeps its block 20. Each model then runs the transfer test of Table 1A, the cosines of Figure 2d, the templates of Appendix C.1, and the depth curves of Figure 5 with 8 donors and 32 receivers. Every model addresses facts by order of mention (Table 5). New names keep the direction of the ordinal vector and reversed orders reverse it, with cosines of0.92to0.97and−0.85to−0.96; at the main block, the fact-address template explains0.46–0.76of the query-dependent variance (leave-one-word-set-out) and person none. The ordinal vector selects the designated fact in34.6–84.1%of cases and random directions almost never. Its rate does not rise with size, and neither does the ceiling: the ceiling selects65.2–96.9%of designated facts, lowest in the 3B and 14B models, and in the 3B model the chat template lowers it (Appendix G.2). Relative to the ceiling, the ordinal vector reaches68–87%in Qwen2.5 and46%in Qwen3-8B. In every model the ordinal vector acts within a window of a few blocks at63–86%of relative depth (block divided by the last block; the window is where it selects at least10%, as in Figure 5), and whole donor states outlast it: two blocks past the window they still select7–30% of receiver facts while the ordinal vector selects under5%, as in Section 7. G.2. Base and Instruction-Tuned Weights Instruction tuning is not needed for ordinal addressing (Section 3). We compare five base models with their instruction-tuned partners on one plain prompt, the instruction, facts, and question followed byAnswer:, and add each instruct model in its native chat template. Each row chooses its own main block by the rule above. In Qwen the base weights carry the address about as strongly as the tuned ones; in Llama the base model is weaker (Table 6). On the identical prompt, the ordinal vector selects67.3%against69.6%in Qwen2.5-1.5B,80.0% against72.2%in Qwen2.5-3B,31.6%against33.3%in Qwen3-8B, and41.9%against54.8%in Llama-3.2-3B. The four Qwen and Llama base models keep a new-name cosine of at least0.86and a reversed-order cosine of at most −0.74, with fact-address푅 2 between0.48and0.78. Gemma-3-4B’s base model answers in full sentences, so we score the single stated content each answer names: it answers92.9%of unedited questions correctly, and the ordinal vector selects the designated fact in21.9%of cases against0.6%for random directions, below its instruct partner; its new-name and reversed-order cosines,0.76and−0.44, are also the weakest, while its fact-address푅 2 , 0.49, equals that partner’s on the same prompt. Within Qwen, the one large gap comes from the chat template, not the tuning: Qwen2.5-3B-Instruct selects 28 The Geometry of Contextual Relations: Language Models Address Facts by Order of Mention Table 5:Ordinal addressing holds from 1.5B to 32B parameters.Every model runs the transfer test of Table 1A (64 new word sets, 12,288 cases), the cosines of Figure 2d, the fact-address and person templates of Appendix C.1, and the depth curves of Figure 5 at every second block. Main block: block index / number of blocks, chosen on block-selection word sets by the rule of Appendix A. Unedited: correct answers without an edit. Window: first to last block at which the ordinal vector selects at least10%, as in Figure 5 (8 donors×32 receivers). † The main-text model; its block was chosen before this sweep (Appendix A), and its window comes from the every-block curves of Figure 5. Designated answers (%)Change cosine푅 2 Model Main block Unedited푣Random Ceiling New names Reversed Fact address Person Window Qwen2.5-1.5B 18/2898.672.00.1 95.70.97−0.960.76−0.02 18–22 Qwen2.5-3B 26/3696.648.00.0 66.10.92−0.850.65−0.02 26–30 Qwen2.5-7B † 20/2899.984.10.0 96.90.93−0.930.66−0.01 18–22 Qwen2.5-14B 34/4899.944.40.0 65.20.92−0.920.46−0.01 32–34 Qwen2.5-32B 48/64100.055.10.0 74.70.94−0.940.51−0.01 46–50 Qwen3-8B22/3699.634.60.0 74.80.95−0.930.71−0.01 22–28 72.2%with the plain prompt and48.0%in its chat template, where the ceiling also falls from90.3%to66.1%. The prompt format, not the tuning, changes how strongly the ordinal vector acts. G.3. Facts Stated in Prose Fact addresses carry over from lists to prose (Section 5). Each prose context is one paragraph stating the four facts with four different verb frames drawn from sixteen, such asFor lunch, Petra had the buckwheatorThe teff ended up in front of Sabrina; none useseats, and four frames put the content first. Two or three unrelated sentences, such as It had rained earlier that morning, and the streets were still wet, sit between facts, and an opening or closing narrative sentence is added with probability one half. A fact’s order of mention is its place in the paragraph. Paragraphs run 37–87 words, against 20–30 for the list, and the instruction and question are those of the main text. Test word sets are 64 new word sets whose names and contents appear in no main-text word set; each is also rendered as a list with the same people, contents, and questions. The ordinal vector calibrated on the main-text lists, unchanged, selects the designated fact in paragraphs (Table 7):44.5%,42.4%, and14.0%of cases in Qwen, Gemma, and Llama, against80.5%,52.2%, and29.3%on the matched lists, while random directions select none. It stays above random at every designated order of mention (lowest22.9%,15.0%, and3.6%), and the ceiling still reaches92.8%,96.3%, and83.1%. To separate format from word sets, 16 new word sets are rendered both as paragraphs and as lists for calibration. On prose receivers, the ordinal vector calibrated on their paragraphs selects fewer designated facts than the one calibrated on their lists (40.2%against44.0%in Qwen,32.3%against43.4%in Gemma,8.5%against10.2%in Llama), and on list receivers the paragraph-calibrated vector still selects41.1%,24.1%, and12.2%: the drop on prose comes from the receivers, not from a vector specific to lists. The geometry agrees. In paragraphs, the fact-address template explains0.45,0.23, and0.33of the query- dependent variance at the main block (leave-one-word-set-out) and person none. Fact-address templates from prose and from lists point the same way (cosines0.76,0.81, and0.84), as do their ordinal vectors (0.75,0.80, and0.81), and35%,17%, and34%of the squared norm of query-dependent prose states lies in the three-dimensional span of the main-text ordinal vectors, against0.1%for random subspaces. Paragraphs largely reuse the fact addresses of lists, though less of each prose state lies in them. 29 The Geometry of Contextual Relations: Language Models Address Facts by Order of Mention Table 6:Base models address facts the same way.Plain: the instruction, facts, and question followed byAnswer:, with no chat template, identical for base and instruct weights; chat: the instruct model’s native template. Each row chooses its own main block by the rule of Appendix A and runs the tests of Table 5. * Gemma-3-4B base answers in sentences, so its answers are rescored by the single stated content they name. ‡ Main-text Gemma run (Table 1A, Figure 2d, Appendix C.1). Designated answers (%)Change cosine ModelWeights, prompt Block Unedited푣Random Ceiling New names Reversed Fact- address푅 2 Qwen2.5-1.5B base, plain1887.767.31.7 82.00.97−0.950.78 instruct, plain2098.769.60.1 95.30.95−0.940.63 instruct, chat1898.672.00.1 95.70.97−0.960.76 Qwen2.5-3B base, plain2695.580.00.1 94.10.94−0.870.69 instruct, plain2695.772.20.0 90.30.93−0.880.65 instruct, chat2696.648.00.0 66.10.92−0.850.65 Qwen3-8Bbase, plain2296.931.60.0 71.70.95−0.930.64 instruct, plain2298.533.30.0 74.60.92−0.890.64 instruct, chat2299.634.60.0 74.80.95−0.930.71 Gemma-3-4B base, plain226.714.00.1 27.60.76−0.440.49 by content * 92.921.90.6 52.6 instruct, plain2094.447.70.0 85.70.93−0.880.49 instruct, chat ‡ 2299.953.90.0 96.40.88−0.820.29 Llama-3.2-3B base, plain1488.141.90.4 85.40.86−0.740.48 instruct, plain1495.554.80.0 94.00.89−0.820.48 instruct, chat1497.451.50.0 93.90.91−0.850.51 Table 7:Ordinal addressing carries over to prose.Top: designated answers (%) on 64 new word sets, 12,288 cases per entry, each edit added once at the main block. Receivers state the four facts either as a paragraph (prose) or as the list of the main text, with the same people, contents, arrangements, and questions. The ordinal vectors are calibrated on the main-text lists (the푣of Table 1A), on lists of 16 new word sets, or on paragraphs of the same 16 word sets. Bottom: at the main block, leave-one-word-set-out푅 2 of fact-address and person templates on paragraphs and on the matched lists (the list푅 2 uses this experiment’s word sets, not those of Appendix C.1); mean cosine between matching prose and list fact-address templates and between matching prose and list ordinal vectors; fraction of the squared norm of query-dependent prose states inside the span of the main-text ordinal vectors, against random subspaces of the same dimension. Ordinal vector calibrated on Model Receiver format Unedited Lists (main) Lists (new word sets)Prose Random Ceiling Qwenprose99.744.5±1.844.0±1.840.2±1.50.0 92.8±1.6 list99.980.5±2.178.1±2.241.1±1.70.0 95.4±1.5 Gemma prose99.242.4±1.243.4±1.232.3±1.20.0 96.3±0.9 list99.752.2±1.452.8±1.424.1±1.20.0 96.5±0.9 Llama prose99.614.0±1.110.2±0.98.5±0.90.0 83.1±1.5 list99.829.3±1.922.4±1.712.2±1.10.0 85.6±1.7 Prose푅 2 List푅 2 Prose vs. listSquared norm in list span (%) Model Fact address Person Fact address Template cos. Vector cos. Prose statesRandom Qwen0.45−0.010.710.760.75350.1 Gemma0.23−0.010.390.810.80170.1 Llama0.33−0.010.530.840.81340.1 30