Paper deep dive
LLMs are not stochastic parrots: Evidence for meaning-mediated abstraction from conlang-like tasks
Julia Witte Zimmerman, Calla G. Beauregard, Tabia Tanzin Prama, Parisa Suchdev, Kathryn Cramer, Elisabeth Kollrack
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 96%
Last extracted: 9/30/2026, 7:06:55 AM
Summary
This paper refutes the strong stochastic parrot hypothesis by demonstrating that Large Language Models (LLMs) can perform meaning-mediated abstraction. Using constructed language (conlang) tasks where statistical plausibility conflicts with semantic meaning, the authors show that models like GPT-4o, Qwen, OLMo, and Llama can follow novel rules and produce correct outputs without example data, indicating reliance on abstract representations rather than just superficial pattern matching.
Entities (10)
Relation Signals (8)
OLMo → testedin → Conlang-like tasks
confidence 98% · OLMo for openness... We use five models: OLMo...
Llama → testedin → Conlang-like tasks
confidence 98% · and Llama as a base model... We use five models: Llama.
GPT-4o → testedin → Conlang-like tasks
confidence 98% · In short, GPT-4o and Qwen were selected for expected performance... We use five models: GPT-4o...
Qwen → testedin → Conlang-like tasks
confidence 98% · In short, GPT-4o and Qwen were selected for expected performance... We use five models: Qwen...
Next-token prediction → underlies → LLMs
confidence 97% · next-token-prediction (NTP) is the popular architectural choice underlying the models we utilize here.
LLMs → refutes → Strong Stochastic Parrot Hypothesis
confidence 95% · these results provide evidence for meaning-mediated abstraction in LLMs and refute the strong stochastic parrot hypothesis.
Conlang-like tasks → usedtotest → Strong Stochastic Parrot Hypothesis
confidence 95% · We test this hypothesis using conlang-like tasks.
LLMs → exhibit → Meaning-mediated abstraction
confidence 94% · these results provide evidence for meaning-mediated abstraction in LLMs
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:The strong version of the stochastic parrot argument claims that, although large language models (LLMs) may exceed rote regurgitation, they cannot move beyond statistical pattern matching into abstraction or reasoning, remaining ontologically near the lower bound of pattern reuse despite producing alluringly fluent text. We test this hypothesis using conlang-like tasks. Several LLMs are given only natural-language descriptions of fictional languages that subvert prominent superficial patterns in training data by combining statistically uncommon and unattested features. Crucially, no example outputs are given. We argue that if the models exhibit rule-following behaviour, they cannot be relying solely on superficial statistical patterns; such patterns often work against the correct output. Instead, successful performance requires representations of the constraints specified in the prompt. Across three complementary task families, models systematically move in the meaning-predicted direction: they distinguish prompt exposure from instructed use, alter semantic relationships in response to novel constraints, and sometimes produce exact matches to complex translation answer keys. Although performance varies across the spectrum of models used, these results provide evidence for meaning-mediated abstraction in LLMs and refute the strong stochastic parrot hypothesis. Our work shows that, under appropriate architectural and contextual constraints, statistical learning can produce meaning-mediated abstractions, although generation remains strongly constrained by superficial plausibility. We discuss implications for model development and for understanding how increasingly abstract representations may emerge from plausible-text-generation objectives.
Tags
Links
- Source: https://arxiv.org/abs/2609.34187v2
- Canonical: https://arxiv.org/abs/2609.34187v2
Trouble viewing inline? Open PDF directly →
Full Text
262,184 characters extracted from source content.
Expand or collapse full text
LLMs are not stochastic parrots: Evidence for meaning-mediated abstraction from conlang-like tasks Julia Witte Zimmerman ∗,†,‡ Calla G. Beauregard ∗,‡ Tabia Tanzin Prama ∗ Parisa Suchdev † Kathryn Cramer ∗ Elisabeth Kollrack ∗ ∗ Computational Story Lab † Computational Ethics Lab ‡ Equal contribution (first project members) Vermont Complex Systems Institute University of Vermont Burlington, VT 05405, USA Abstract The strong version of the stochastic parrot argument claims that, although large language models (LLMs) may exceed rote regurgitation, they cannot move beyond statistical pattern matching into abstraction or reasoning, remaining ontologically near the lower bound of pattern reuse despite producing alluringly fluent text. We test this hypothesis using conlang-like tasks. Several LLMs are given only natural-language descriptions of fictional languages that subvert prominent superfi- cial patterns in training data by combining statistically uncommon and unattested features. Crucially, no example outputs are given. We argue that if the mod- els exhibit rule-following behaviour, they cannot be relying solely on superficial statistical patterns; such patterns often work against the correct output. Instead, successful performance requires representations of the constraints specified in the prompt. Across three complementary task families, models systematically move in the meaning-predicted direction: they distinguish prompt exposure from instructed use, alter semantic relationships in response to novel constraints, and sometimes produce exact matches to complex translation answer keys. Although performance varies across the spectrum of models used, these results provide evi- dence for meaning-mediated abstraction in LLMs and refute the strong stochastic parrot hypothesis. Our work shows that, under appropriate architectural and con- textual constraints, statistical learning can produce meaning-mediated abstractions, although generation remains strongly constrained by superficial plausibility. We dis- cuss implications for model development and for understanding how increasingly abstract representations may emerge from plausible-text-generation objectives. ∗ julia.zimmerman@uvm.edu Preprint. arXiv:2609.34187v2 [cs.CL] 29 Sep 2026 1 Introduction This pre-print describes the initial results of an empirical probe of a widespread claim about LLM limitations. It has been created to provide access to our findings to participants at ISC Summer School 2026, where we presented this project as a poster, as well as IC2S2 2026 and SLSA/ 4S 2026, where we presented 2 related works [Institut des sciences cognitives, Université du Québec à Montréal, 2026, 12th International Conference on Computational Social Science, 2026, Society for Literature, Society, and the Arts, 2026, Society for Social Studies of Science, 2026]. In this work, we test the strong version of the stochastic parrot hypothesis using conlang-like tasks 3 where following the meaning of an instruction conflicts with familiar surface distributions. This project is currently underway with a suite of 14 conlang-like rules (17 sets of prompts, with 2 non-conlang controls and 1 correction), but we present our initial results focused on a subset of these rules and a subset of our planned analyses as part 1 of 2. A brief summary of the key points in this pre-print: • Statistical plausibility and meaning-mediated plausibility can be experimentally dissociated. Our three conlang families dissociate plausibilities in different ways: in brief, prompt expo- sure versus instructed use, semantic similarity versus instructed irrelevance, and relational preservation across radically different surface forms. • Across three independent rule families LLMs moved in the meaning-predicted direction, showing that superficial statistical patterns are not a sufficient explanation for their behavior even though such patterns remain powerful constraints. For every rule, at least one model shows clear evidence of rule-following; for most rules, most models do. •The strong stochastic parrot hypothesis is behaviorally inadequate, even for base-model LLMs, because successful performance on these tasks is better explained by meaning- mediated abstraction than by superficial statistical regularities. •We provide an existence proof of meaning-mediated abstraction implicated through rule- following behaviour, not evidence of human-equivalent understanding. •Plausible output alone does not reveal its mechanism: Benchmark success can come without construct validity. • Next-token prediction may give models access to meaning while simultaneously anchoring generation disproportionately to its most superficial and fungible forms. The bottleneck may not be knowledge acquisition, but control over which kind of plausibility governs generation. 1.1 Why this mental model matters As with the introduction of prior information technologies – photography, printing, even writing – the reaction to Generative AI (Gen AI) is divergent and contentious due to its destabilizing effects amongst a complex milieu of motivations, beliefs, and constraints [Yates, 1966, Zimmerman, 2025]. To contend with these disruptions, people develop mental models (including specific phrases and associations) which they draw on when answering questions as to Gen AI’s behavior, origin, and capacity, in order to meaningfully interpret it [Rapp et al., 2025, Grice, 1989, Agha, 2003, Eckert, 2008, Silverstein, 2003, Peirce, 1931, Gibson, 1979]. 4 A similarly complex milieu shapes how the components of those mental models develop and endure [Irvine and Gal, 2000]. We believe good mental models are a necessary precursor to effectively navigating this fraught landscape (independent of stance). 5 Regardless of the original intent behind it, the term stochastic parrot has been widely and variously adopted since its introduction in Bender et al. [2021]. It is now a popular mental model for Gen AI, and is frequently applied even to models that differ substantially from those discussed in the original paper. 2 Or will present, in the case of SLSA/ 4S 2026. 3 Constructed languages, or conlangs, are communication systems intentionally designed by someone (or something, or someones). For brevity, we refer to our conlang-like tasks as conlangs, tasks, or rules. 4 Although not the subject of this paper, we have two in-progress works related to this topic [Zimmerman et al., 2026, Morgenstern et al., 2026]. 5 Bad mental models have multiple harms. They can lead to incorrect predictions, act as thought-terminating cliches, undermine an otherwise compelling argument in the eyes of the intended audience, and cause disagreement where none was necessitated. 2 In this work, we are not concerned with the term itself, but with empirically evaluating a claim it is often taken to imply: that LLMs trained on text are fundamentally limited to relatively superficial statistical regularities and therefore incapable of forming the kinds of abstractions associated with meaning-like or thought-like behavior. 6 For our purposes, the strong stochastic parrot hypothesis (SSP) is the view that model behaviour can ultimately be explained in terms of token co-occurrence statistics, surface-level syntactic patterns, and similar symbolic patterns. By contrast, we argue that, given the nature of the correct outputs in our tasks, explanations invoking less superficial aspects of meaning are substantially more parsimonious. Rather than treating statistical learning and abstraction as mutually exclusive possibilities, we ask under what conditions statistical learning can give rise to increasingly abstract behavior. 7 We argue that understanding how statistical learning scaffolds transitions along the concrete-abstract pattern spectrum is important for understanding AI systems as well as human cognition. 8 Dismissing statistical learning as inherently incapable risks obscuring mechanisms through which meaning can emerge. We contend that if models systematically move in the direction predicted by our tasks, then explana- tions invoking only superficial symbolic regularities become increasingly difficult to maintain. 1.2 What is plausibility? Despite significant variation, LLMs are fundamentally built around plausible text generating ar- chitectures, comprising plausible text generating objectives (such as masked language models like RoBERTa [Liu et al., 2019] and autoregressive transformers like ChatGPT [OpenAI et al., 2024]) and token-centric design choices. Specifically, next-token-prediction (NTP) is the popular architectural choice underlying the models we utilize here. For concision, we’re referring back to the family of neural architectures united by this fundamental learning strategy as plausible-text-generating- architectures (PTGA). These models optimize for plausibility by repeatedly guessing what token comes next and being rewarded for good guesses, based on a large corpus of mostly human-generated text. In that operationalization of plausibility, everything that goes into making someone likely to continue the utterance in that particular way is collapsed into a single score. 9 Speaking approximately, many determinants of plausibility fall under what we might call the meaning of the utterance. Some are more directly perceptible via statistical learning than others, with the most accessible to an LLM being superficial statistical regularities such as token co-occurrence. Other determinants may require increasingly abstract representations, which may themselves be assembled from lower-level patterns. Humans appear capable of learning across this spectrum, both from concrete statistical regularities and from contextually-mediated abstractions. 10 Plausibility alone is lossy. When an LLM generates plausible text, it is unclear whether that is a result of the model’s understanding of the relevant concepts – any part of what might be called the meaning 6 This mental model is prevalent. Beyond popular descriptions such as “spicy autocorrect,” we have encountered similar claims in academic settings, e.g. a comparison of deep learning to photographs that assemble meaningful patterns via photons without interacting with that meaning beyond its most superficial manifestation. 7 And potentially increasingly thought-like, connecting meaning construction across architectures and scales, drawing on the Distributional Hypothesis (DH) as a partial explanation for similar processes in people. Although LLM architecture is extremely different from ours, the DH is explicitly the inspiration for the distributional-semantics-related aspects of LLM architecture as well [Sahlgren, 2008, Harris, 1954, 1968]. In this work, our aim is not to demonstrate equivalence between human and machine cognition – or to argue that machines can “think” – but to evaluate a more specific claim about the limits of statistical learning. 8 Language operates across multiple levels of abstraction, from highly fungible surface forms to deeply contextual and socially grounded interpretations. See discussion of the pattern-to-thought transition and the fungibility-groundedness spectrum in Zimmerman [2025]. See also upcoming works with Grayson Wycliffe Storer, Alice Patania, and Juniper L. Lovato. 9 Determinants such as world knowledge, mood, past experience, history with the speaker, situation, pragmatics, prosody, semantics, tone, location, physical ability (for example, ability to hear in a noisy room), propositional content, implications, consequences, belief, truth, commonsense, motivation, personality, attentiveness, familiarity, memory, etc. On the fungibility- groundedness spectrum, the most personal of these factors are unique, undetectable as a pattern in the text itself, the stigmergic trace of the communication. The more fungible these factors are, the more likely they are to be shared due to commonalities like human embodiment and culture, the more likely their signal is to be perceptible to a statistical learner. Note that the specifics of this are clearly embodiment-/ architecture-dependent! 10 More discussion of this topic is available in Zimmerman [2025]. For an example of how medium and message are inherently intertwined and both decoding and encoding are architecture-dependent, see Pera et al. [2026]. 3 Figure 1: This figure shows prompts describing the translation family of rules (translate-1 and dan-1), their answer keys, and sample outputs from each model, to illustrate what we mean by separating meaning-mediated plausibility from statistical plausibility (although we do that in different ways across the rule families). It is apparent from these responses that these tasks are hard, that some models do much better than others, and that dan-1 is more difficult than translate-1. However, it is also apparent that models can produce outputs exactly matching or near to the answer keys, behaviour which is better explained by meaning plausibility than statistical plausibility. of the prompt (beyond the most explicitly statistical components) – or a result of successful pattern recognition and regurgitation. For convenience, we’l call these meaning-mediated plausibility or meaning plausible and statistically-mediated plausibility or statistically plausible, respectively. 11 The fact that superficial statistical likelihood often 12 aligns with the meaning of an utterance is what makes plausible text generation so powerful. Under the right conditions, statistical learning may access traces of meaning, not just the most superficial patterns visible in the training data. Meaning-mediated abstractions therefore need not be seen as alternatives to statistical learning. They may emerge because they are useful compressions for solving statistical prediction problems, both in silico and in vivo [Greco et al., 2024, Stöckl et al., 2024]. The challenge is that plausible outputs do not reveal which process produced them. Both superficial statistical regularities and meaning-mediated abstractions can generate plausible text, obscuring inference of the underlying mechanism from the response alone. This creates a construct-validity problem for many benchmark-style evaluations (see Section 4.2). We design our experiment to address this challenge, by forcing the relatively concrete plausibilities of token statistics to contradict the relatively abstract plausibilities of meaning. 1.3 Prior work and superficial patterns Our argument is not that LLMs don’t rely on patterns. Quite the opposite: the existence of learned patterns is what makes these systems interesting. If we allow patterns at any level of abstraction, essentially everything discussed in this paper can be described as pattern matching. 13 The question is therefore not whether models learn and use patterns, but what kinds of patterns they learn. 14 The strong stochastic parrot hypothesis (SSP) says that LLM behaviour can be explained by relatively concrete regularities: this symbol follows this symbol, these symbols tend to occur near those symbols, and so on. 15 See Fig. 29. 11 However, keep in mind these are conceptually along the same concrete-abstract, fungibility-groundedness spectrum of meaning, with the superficial statistical patterns (symbol co-occurrence, some morphology and syntax) on the more concrete, immediately apparent side and semantic, pragmatic, register, and discourse-level content higher on the abstract side. There is no bright line between them! We want to probe the abstract end and discredit the most concrete end (including some low-level abstractions) as sufficient explanation for behaviour. 12 And perhaps specifically, often but not always. We will return to this topic in Section 4.3. 13 Even as statistical pattern matching. 14 And when they use them. 15 By contrast, people appear capable of learning across a wide spectrum, from concrete regularities to more abstract principles that can be flexibly expressed through different symbolic systems, without a discrete boundary where pattern matching becomes abstraction. 4 Several prior works suggest that LLMs are not restricted solely to superficial patterns. Digutsch and Kosinski [2023] find that semantic similarity predicts GPT-3 activations better than word co- occurrence. Hewitt et al. [2025] show that models can learn and self-verbalize meanings associated with newly introduced embeddings, while Hewitt et al. [2024] find evidence of instruction-following behaviour prior to explicit instruction tuning. Cui et al. [2024] identify the possibility of a transition from positional to semantic pattern matching. 16 Taken together, these observations establish antecedent plausibility for meaning-mediated structure in LLMs, but they do not cleanly dissociate statistical and meaning-based explanations. Our contribution is to probe that distinction explicitly through conlang-like tasks designed to decouple meaning- mediated plausibility from familiar surface distributions. The crux of our argument therefore depends on constructing tasks whose correct responses are statistically implausible based on natural language corpora, yet which are learnable from natural- language instruction. We use “meaning-mediated plausibility” to refer to outputs better explained by the content of an instruction than by its component surface distributions. 17 2 Methodology Our research question is, can a model generate an answer that is licensed by a natural-language instruction (plausible via meaning) but disfavored by superficial 18 distributions (statistically implausible)? Our argument relies on three steps. First, we construct conlang-like tasks in which statistical plausibility and meaning-mediated plausibility point in different directions. See Figure 1 for examples. Second, we formulate explicit hypotheses about what successful rule-following should look like and measure movement in those directions. Third, we ask whether the resulting behaviour is better explained by the content of the instructions than by familiar surface distributions alone. 19 We are not trying to show that LLMs are general reasoners or that they learn, create, or use abstractions in a human-like way. Our argument posits the existence of some non-trivial meaning-mediated abstractions, accessible under some conditions, but we do not identify their specific nature. 2.1 Model selection Because our rules vary in difficulty and operate near the frontier of model failure, we balance ar- chitectural simplicity against performance. Our strongest argument would be made with a purely autoregressive LLM, so the more “pure” a model is, the more imperfect task performance we are willing to tolerate. Based on preliminary testing, we selected models that span next-token-prediction purity, instruction tuning, reasoning-oriented training, openness and training-data transparency, inter- pretability, and sufficient performance to produce detectable output-level evidence. See Appendix A.3 and Table 16 for details. In short, GPT-4o and Qwen were selected for expected performance, OLMo for openness, and Llama as a base model. We treat Qwen with and without thinking enabled separately because we expect CoT-style (chain-of-thought style) fine-tuning to modulate the influence of lower-level statistical plausibility (Section 4.3). As a base model, Llama’s weaker directional shifts remain theoretically interesting (à la Hewitt et al. [2024]), especially when considered as part of this spectrum of models. These models were chosen to support interpretation of the present experiment rather than to enable a controlled architecture or scaling comparison, so although our results motivate exploratory hypotheses, they are not probative on those questions. 16 Further discussion of the pattern-to-thought transition in machines in Zimmerman [2025] and Zimmerman et al. [2024b], although our views continue to evolve. The current work is also inspired by Diamond [2023], Huang et al. [2023], Kallini et al. [2024], Mahowald et al. [2024], Zimmerman et al. [2024a]. 17 And “statistical plausibility” to refer to outputs thatcanbe explained by their component surface distributions, whether they can be explained by other kinds of plausibility or not. 18 At or near surface-level. 19 We do not view “instruction following” and “abstraction” as competing explanations. Successfully following a sufficiently novel instruction requires some representation of the instruction’s content. Our argument is not that the representation must be human-like, but that explanations based solely on superficial symbol co-occurrence are inadequate. See Figure 29. 5 2.2 Experimental design We use five models: GPT-4o, Qwen (thinking, with CoT-style-prompting enabled), Qwen (no thinking, with CoT-style-prompting disabled), OLMo, and Llama. 20 More detailed descriptions of each model are provided in Table 16. For each model and rule, we ran 50 independent outputs. Each rule – except sidequest-1 21 – has 3 prompt variations with slight differences in style, formatting, and wording, but interchangeable propositional content, assigned randomly across runs. Since small variations in wording can cause notable changes in model behaviour [Sclar et al., 2024], we included these variants to help control for sensitivity to surface phrasing without venturing into prompt engineering or optimization. Across all models, we used fixed parameters (temperature of 0.7, a token budget of 4,096, and approximately 20 sentences of requested output where possible). Since our goal is an existence argument rather than performance optimization, we did not perform parameter sweeps. Outputs were generated independently, without in-context accumulation (but see Appendix A.2). For additional details, see Appendix A.3. We frame our argument around existence because that is the logical structure required to refute the SSP while sidestepping common interpretive pitfalls: existence lets us ask a specific and tractable question without placing undue weight on complex unknowns such as consistency across contexts (see Appendix A.3). 22 2.3 Identifying and measuring rule-following behaviour Outputs were cleaned through a combination of manual effort and automation. To enhance intercleaner reliability, we discussed cleaning criteria as a team, reviewed examples together for consistency, and spot-checked 3 randomly chosen computationally-cleaned runs per rule per model to refine the process. Where possible, we applied metrics to raw outputs as well as cleaned outputs, to sanity-check that cleaning choices were not driving results. During the preliminary testing phase, we developed automatable tests for specific aspects of rule- following – essentially unit tests for the output – building measurability into the rules themselves. Each test captures only some aspects of compliance, making them a lower bound on rule-following; underestimating is preferable to overestimating, since overestimating could lead to a false positive. Tests range from open-ended (story rules, double verb rules) to exact (translation rules, which can be checked against an answer key), and applied to both cleaned and raw outputs where possible. We conducted both model-level and pooled analyses, with permutation testing and corrections where warranted. See Appendix A.3 for details. Some behavior may be visibly rule-directed without being fully captured by automated checks, but for the most part, that analysis will be done in part 2. See Appendix A.11 for some preliminary observations. 2.4 Rule design The rules must satisfy two requirements. First, correct responses should be difficult to explain using superficial statistical regularities. Second, the tasks should remain close to linguistic competence rather than depend primarily on unrelated capacities such as mathematics, counting, very long- distance memory, or sensory experience. We therefore aim for a theoretical “sweet spot”: tasks reliant on structures that models could arguably have learned given their architecture and training data, yet whose correct outputs diverge from familiar natural-language distributions. Neural network architectures can represent data structures underlying approximately context-free and slightly context-sensitive languages, formal classes that align in 20 There are some minor variations in how we refer to these models, but they all refer to the same five: GPT-4o, Qwen (with thinking enabled), Qwen (without thinking enabled), OLMo, and Llama. GPT-4o is sometimes described as “no system prompt”, because it has a slightly different expected prompt format than the others. We treated all models as identically as possible, so we did not provide any additional system prompt to GPT-4o, even though that is a common application. 21 Due to a copy-paste error, the sidequest-1 runs analyzed in part 1 use 2/3 identical prompts and 1/3 a second variant. Corrected data (sidequest-1-fix) will be analyzed in part 2. 22 However, the degree of alignment between our hypotheses and results is stronger than that framing alone suggests. Obviously, that is a topic for Results 3 and Discussion 4, but we felt earlier signposting might be helpful. 6 theory with natural language [Wilcox et al., 2019]. Since LLMs aren’t people, we don’t exactly know what they could have learned while acquiring the level of linguistic competence they exhibit, but we can consider people as a kind of outer bound: tasks that no human acquires through language exposure alone are unlikely to have been learned by PTGA models. Our conlangs therefore remain within the approximate domain of language learning while being unusual enough that explanations based solely on superficial pattern matching are not merely less parsimonious, but increasingly hard to sustain. 23 To undermine statistical plausibility at concrete levels, we introduce linguistic features that are rare to unattested in natural language, including unusual word orders, rare English bigrams, word- order parity rules, inserted verbs, and a translation system in which prompts and correct answers share no character-level overlap. These manipulations make correct outputs atypical under familiar surface distributions while preserving the overall structure of a language task. At the same time, we avoid completely disrupting information locality, requiring mathematical reasoning, or relying on embodiment [Zimmerman, 2025, Lovato et al., 2024, Pfeifer and Bongard, 2006, Hahn et al., 2020, Mayzner and Tresselt, 1965, Norvig, 2013, Liu et al., 2024b, Kallini et al., 2024, Futrell et al., 2020b, Futrell and Hahn, 2024, Gibson et al., 2019, Harris, 1954, 1968, Sahlgren, 2008, Zipf, 1965, Futrell et al., 2020a]. For example, we do not test counting, because linguistic competence does not require counting ability. In addition to designing rules that are theoretically unlike the training data, we perform contamination checks for potential training-data exposure (Section 2.5; Appendix A.4). Because this is an existence argument, success should be understood as systematic movement in the predicted direction rather than perfect rule execution. These are intentionally difficult tasks, and imperfect performance does not imply that the rule was not represented at all. We will discuss the exploratory testing in which we developed the conlangs more in part 2. 2.5 Training data exposure We performed complementary checks for training data exposure. While these checks cannot prove the absence of exposure of these models to these rules, they can dramatically reduce how much likelihood we allocate to that possibility. The results of these checks (as well as the number of models and runs per rule, the design and diversity of the rules, and the inclusion of prompt variations) make it implausible that any model we tested could have succeeded by direct recall of training data. We searched for 19 representative strings contained in our prompt and answer pairs in 8 major training corpora including OLMo, DCLM-baseline, and Dolma-v1.7; only one string, matching “zero is the word boundary OR zero is the space symbol”, appeared in some of the training corpora (ranging from 1-4 instances). These appear to refer to wholly dissimilar contexts (see Table 13). We used the InfiniGram N-gram LM to test whether the same training corpora assigned the highest probability to a token that was part of the correct continuation for dan-1 and translate-1 prompt variants; of 12 variations, only one condensed form of the dan-1 prompt yielded a correct prediction by any training corpus, and none when we took (fuzzy) order into account [Liu et al., 2024a]. See Table 7, Table 8, and Table 13. For additional details on these checks and the additional checks we performed, see Appendix A.4. 2.6 Conlang families and hypotheses In part 1, we focus on three rule families that provide complementary evidence for our core argument. In part 2, we extend the discussion to the full set of rules, including additional evidence regarding rule-following behaviour, potential cognitive tradeoffs and a more detailed exploration of output modes. 24 23 A caveat: As children, speakers learn tasks that look like aspects of the conlangs we use. But they do so with varying degrees of conscious accessibility. Our tasks foreground linguistic skills as conscious reasoning, which is a non-trivial difference. Furthermore, first language acquisition and second language acquisition are different for people, and are not uniform across language-related skills [Malaia et al., 2020]. 24 As the paper is already somewhat lengthy, we split the minimal core argument from the broader experimental results and exploratory analysis. 7 •The story family (comparison-1, sidequest-1, comparison-2) 25 probes whether unusual output can be explained by prompt exposure alone, using sustained generation of unusual lexical distributions over approximately 20 sentences. • The double verb family (doublev-1, doublev-2) probes whether models can override syntactic and semantic plausibility priors through the insertion of additional irrelevant verbs. 26 •The translation family (translate-1, dan-1) 27 probes composed symbolic mappings involving lexical, morphological, semantic, and word-order constraints. Dan-1 mitigates the role of character-level overlap between prompt and answer. 28 Summaries of the key hypotheses for each family of rules are: • Story family: comparison-1<sidequest-1<comparison-2 in rates of rare bigram usage 29 , and comparison-2 should be distinguishable from both comparison-1 and sidequest-1. 30 • Double verb family: production of double verbs; high semantic similarity across paired verbs in doublev-1, obviously lower similarity in doublev-2. •Translation family: outputs should be substantially closer to the answer keys than matched random strings, with weaker performance as the representational regime becomes less familiar (moving from translate-1 to dan-1). Together, these families span lexical, semantic, morphological, syntactic, word-order, metaphorical, discourse-level, and arguably pragmatic rule application. While any individual rule could admit rule-specific alternative explanations, the convergence of evidence across multiple qualitatively different dissociations between meaning-mediated and statistical plausibility strengthens the overall argument. Taken together, the results allow us to probe not only whether models move beyond verbatim regurgitation, but whether the patterns they rely on extend substantially beyond superficial distributional patterns. For readability, all rules are available in Appendix A.5. 3 Results Our hypotheses were specific and directional, and the observed outcomes generally match those predictions. All rules elicit rule-following behaviour from some models. All models exhibit some evidence of rule-following on all rules except the most challenging (dan-1), and the predicted ordering holds, with effect sizes frequently near ceilings, and the same qualitative pattern across cleaned and uncleaned outputs. The weakest evidence, where results are not statistically significant, comes from Llama, which is the model we expected a priori to show the weakest signal. Table 1 summarizes our results. Each family of rules is described in more detail below (Sections 3.1, 3.2, 3.3), and discussed in Section 4. We observe that model responses can be multimodal. Anecdotally, we observe some failures that metaphorically evoke attractors in dynamical systems: initial trajectories of varying durations, eventually collapsing into degenerate cycles. Some initial observations as to such output modes are given in Appendix A.11, though we mostly defer to part 2. 3.1 Story rules This family controls for the possibility that unusual output is caused by exposure to unusual prompt content rather than by rule-following. We distinguish three regimes: •Baseline: In comparison-1, we request ordinary English story generation of about 20 sentences. 25 And sidequest-1-fix in part 2. 26 Doublev-2 arguably also conflicts with dependency locality, since the irrelevant verb is not near anything it modifies. The rules also require some overriding of lexical and morphological priors, to mark the irrelevant verb with a specific rare bigram. 27 And translate-2 in part 2. 28 Additionally, translate-1 requires a metaphor map, and dan-1 uses chiasm (“Her mother’s cat... His cat’s father...”) to further subvert statistical plausibility. 29 In terms of tokens, not types, in the type-token distinction. Rare bigram ratio was computed as the proportion of all output bigrams that belonged to the predefined rare bigram inventory. 30 Additionally, idealized rule-following would see comparison-1 and sidequest-1 be indistinguishable, but PTGA constraints should prevent models from achieving that. In practice, we expect the best rule-following to show up as comparison-1 and sidequest-1 looking much more like each other than either does to comparison-2. 8 Table 1: Tested conlangs grouped by family, with hypotheses, tests, and summarized findings. Rule(s)DescriptionHypothesisTestY/N Family: Story rules sidequest-1, comparison- 1, comparison- 2 comparison-2 should show a preponderance of rare bigrams versus the other two rules, rul- ing out prompt-driven echoing as a sufficient explanation. Rare bigram usage should be higher for comparison-2 than sidequest-1 and comparison-1 (use versus mention of rare bi- grams should yield distinguish- able output); ordering should be comparison-1<sidequest- 1<comparison-2;ideal- ized rule-following should see comparison-1 and sidequest-1 rare bigram usage be indistin- guishable, but PTGA should prevent this in practice. Rate of rare bigram usage comparison across conditions; see Figure 2. Additionally: Fig. 19, Tables 2, 3, 4, 5, 17, 18, 19, 20, 21, 22. Y (for all cleaned outputs: Qwen, GPT-4o, OLMo; Y forsome models for uncleaned outputs) Family: Double verb rules doublev-1, doublev-2 Models must use a noveldouble-verb structure; doublev-2 additionallyempha- sizes that the second verb should be seman- tically unrelated to the first. (1) Production of double verbs; (2) Ratio of verbs marked with -qw to total verbs≈ 0.5; (3) verb pairs in doublev-1 have higher cosine similarity than in doublev-2. (1 and 2) Count; (3) KS test and Wilcoxon Rank Test for doublev-1>doublev-2 cosine similarity. See Tables 27, 28, Additionally: Figures 4, 5, Ta- bles 23, 24, 25, 26, Figures 3, 20, 21. (1) Y for all models; (2)Y forsome models; (3) Y for all models Family: Translation rules translate-1Complex novel trans- lation task. (1) String distance measures – output Levenshtein distances, normalized Levenshtein, Jac- card, and cosine similarity – should be closer to the an- swer key than randomized strings (select-character and all-character).(2) Exact matches to answer key. KS test and Wilcoxon Rank- Sum on cosine string similarity vs. random strings; string dis- tance comparisons; regex. See: Figures 6, 25, Tables 31, 32. (1) Y for all models; (2) Y for some mod- els dan-1Complex translation task. Uses no charac- ters from the prompt, controlling for token exposure. (1) Output closer to correct an- swer than random strings; (2) Exact matches. KS test and Wilcoxon Rank- Sum on cosine similarity vs. random strings; string distance comparisons; regex. See: Fig- ures 7, 24, Tables 33, 34. (1)Y forsome models; (2) Y for some models translate-1, dan-1 Supplementary check to support the above results via established machinetranslation metrics (BLEU, chrF, TER, METEOR). Similarcosinesimilarity findings across both rules; translation metric scores ap- proach optimal (BLEU = 100, chrF = 100,TER = 0,ME- TEOR = 99.6). Cross-rule comparison of co- sine similarity;translation benchmarks against reference. See Figures 22, 23. Y for some models •Prompt exposure: In sidequest-1, rare bigrams [Norvig, 2013, Mayzner and Tresselt, 1965] are presented in the prompt, but the model is instructed to write an ordinary English story. •Rule-following: In comparison-2, the same rare bigrams are described as the morphemes of a new language and the model is instructed to produce a story of about 20 sentences in that language. Both sidequest-1 and comparison-2 contain the same list of rare bigrams in the prompt. An alternative explanation to rule-following is that symbols introduced in a prompt tend to recur in subsequent output, independent of their intended role. 31 If prompt exposure alone were sufficient to explain conlang performance, sidequest-1 and comparison-2 should produce similarly elevated rates of rare-bigram use. Because of competing plausibilities, we expect the ordering of rare bigram use to be (1) comparison-1 <sidequest-1<comparison-2. Rule-following behaviour should at least satisfy: (2) rare bigram usage in comparison-1 and comparison-2 are distinguishable, and (3) rare bigram usage in sidequest-1 and comparison-2 are distinguishable. An idealized account of rule-following – in which only 31 For example, introducing a topic such as baseball tends to increase the frequency of baseball-related terminology later in the conversation. If a person is introduced with a male pronoun, male pronouns are likely to recur. Such regularities may help bridge syntax, semantics, and pragmatics. 9 Figure 2: Comparison of rare bigram usage across conditions, for the 3 models (Qwen (with thinking), GPT-4o, OLMo) with cleaned output. Table 2: Fisher-combinedp-values across models (Qwen (T), Olmo, GPT-4o (no SP)) for each contrast; cleaned outputs. ComparisonFisher statistic (X 2 ) p combined k comparison-2 vs sidequest-1226.7943.700e-463 sidequest-1 vs comparison-1187.0221.093e-373 comparison-2 vs comparison-1243.4731.017e-493 instructed use matters and prompt exposure has no effect (which we don’t actually expect to see, since we think in practice the plausibilities compete) – would additionally satisfy (4) sidequest-1 and comparison-1 are indistinguishable, since merely encountering the rare bigrams would have no effect. The cleaned results should be more dramatic than the uncleaned results (since any additional English text generated reduces the ratio of rare bigrams). All three models (GPT-4o, Qwen (thinking), OLMo) with cleaned outputs satisfy the predicted ordering (1) with comparison-2 strongly distinguishable from both controls (2 and 3). The same holds with the uncleaned outputs for four of the five models; Llama’s rule-following (comparison-2) outputs differ from baseline (comparison-1) in the expected direction but are not reliably distinguishable from its prompt exposure (sidequest-1) outputs. As expected, no model exhibits idealized rule-following (4): prompt exposure has a detectable effect in every model. However, the size of that effect varies substantially. GPT-4o comes closest to the ideal pattern, with sidequest-1 remaining much closer to baseline than to rule-following output. That inclusion in the prompt is not generally sufficient to cause inclusion in the output weakens prompt exposure as a general alternative explanation for rule-directed behaviour. Although the prompts are otherwise similar, not identical, our results suggest the interpretation that the same surface forms – the lists of rare bigrams – can produce different behaviour in LLMs depending on their communicative role within the prompt. Because the data contain many tied and near-zero values, we use permutation tests to estimate combined p-values while minimizing distributional assumptions. Combined results are reported in Tables 2, 3. Additional results are reported in Appendix A.7. 3.2 Double verb rules The double verb family instructs models to add a second, irrelevant verb to each main verb and mark it with the affix -qw. In doublev-1, the added verb is just described as irrelevant. In doublev-2, we additionally specify that it should be semantically unrelated to the first verb and feel random relative to the rest of the sentence. Otherwise, the rules are the same. Because verbs occurring near 10 Table 3: Fisher-combined p-values across all models for each contrast (uncleaned output/ all output). ComparisonFisher statistic (X 2 ) p combined k comparison-2 vs sidequest-1303.7712.479e-595 sidequest-1 vs comparison-1290.6081.501e-565 comparison-2 vs comparison-1349.3975.350e-695 Table 4: Mann-Whitney U tests (i.e. Wilcoxon Rank-Sum) for cleaned model outputs for sidequest-1, comparison-1, and comparison-2 (cleaned outputs available for GPT-4o, Qwen (thinking), and OLMo). Note that GPT-4o shows the closest to ideal performance here (sidequest-1 and comparison-1 outputs clearly more similar to each other than comparison-2 output is to either). ModelComparisonUpp (BH)Cliff’s DeltaMedian AMedian BMedian Diff gpt4o (no SP)sidequest-1 vs comparison-119803.05e-083.05e-080.5840.0019820.0000000.001982 gpt4o (no SP)comparison-2 vs comparison-125005.42e-193.94e-181.0000.5658550.0000000.565855 gpt4o (no SP)comparison-2 vs sidequest-125006.06e-181.36e-171.0000.5658550.0019820.563873 olmosidequest-1 vs comparison-124066.11e-166.88e-160.9250.0055760.0000000.005576 olmocomparison-2 vs comparison-124651.90e-173.41e-170.9720.7068370.0000000.706837 olmocomparison-2 vs sidequest-124501.31e-161.68e-160.9600.7068370.0055760.701260 qwen (T)sidequest-1 vs comparison-125001.31e-183.94e-181.0000.0209970.0000000.020997 qwen (T)comparison-2 vs comparison-125001.31e-183.94e-181.0000.6290560.0000000.629056 qwen (T)comparison-2 vs sidequest-124617.12e-171.07e-160.9690.6290560.0209970.608059 one another in English are typically semantically related, and at least related to the words around them, the second rule pushes against strong plausibility priors (both syntactic and semantic) learned from natural-language data, and violates the principle of least effort [Zipf, 1965, Gibson et al., 2019, Futrell et al., 2020b]. 32 Comparing cosine similarity between verb pairs from doublev-1 to doublev-2 therefore provides a direct test of whether models alter generation in the requested semantic direction. Rule-following should result in the production of additional verbs, about half of which should be marked with the instructed rare bigram morpheme. 33 The relationships between the pairs of verbs should further satisfy specific hypotheses. First, the cosine similarities between paired verbs should be substantially lower in doublev-2 than in doublev-1. Second, our PTGA account predicts that doublev-1 should often exhibit very high cosine similarities, because local plausibility priors strongly 32 Doublev-2 may also conflict with dependency locality, since especially when the second verb is irrelevant in the sense specified in doublev-2, it is arguably not near anything it modifies. 33 Successful rule-following output should yield a total count of verbs per run that is twice the number of verbs marked with the -qw morpheme. Since doublev-2 emphasizes the irrelevance of the added verb, we expect that cosine similarity between verbs should be a lower value (e.g. closer to0) for the verbs produced by this prompt than for doublev-1. Since we think doublev-2 is harder, we expect the ratio of -qw verbs to go down, and the total number of verbs produced to go down, compared to doublev-1. In part 1 we have not tracked whether -qw marks the better choice of “irrelevant” verb, where present. We may explore that in part 2. Table 5: Mann-Whitney U comparisons of rare bigram usage medians for all sidequest-1, comparison- 1, comparison-2 output (uncleaned outputs, available for all 5 models). modelcomparison (A-B)Upp bh Cliff’s δMedian AMedian B ̃x A − ̃x B gpt4o (no SP)sidequest-1 vs comparison-119803.05e-083.81e-080.5840.0019820.0000000.001982 gpt4o (no SP)comparison-2 vs comparison-125005.42e-197.48e-181.0000.5658550.0000000.565855 gpt4o (no SP)comparison-2 vs sidequest-125006.06e-181.14e-171.0000.5658550.0019820.563873 llamasidequest-1 vs comparison-116491.26e-031.35e-030.3190.0000000.0000000.000000 llamacomparison-2 vs comparison-118121.21e-051.40e-050.4500.0004170.0000000.000417 llamacomparison-2 vs sidequest-114142.34e-012.34e-010.1310.0004170.0000000.000417 olmosidequest-1 vs comparison-125002.49e-187.48e-181.0000.0303870.0000000.030387 olmocomparison-2 vs comparison-125002.49e-187.48e-181.0000.3284000.0000000.328400 olmocomparison-2 vs sidequest-124959.54e-181.59e-170.9960.3284000.0303870.298013 qwen (T)sidequest-1 vs comparison-125004.72e-181.01e-171.0000.0302650.0002540.030011 qwen (T)comparison-2 vs comparison-125004.72e-181.01e-171.0000.0755790.0002540.075325 qwen (T)comparison-2 vs sidequest-124245.97e-168.14e-160.9390.0755790.0302650.045314 qwen (no T)sidequest-1 vs comparison-125001.74e-187.48e-181.0000.0415710.0000000.041571 qwen (no T)comparison-2 vs comparison-125001.74e-187.48e-181.0000.3958450.0000000.395845 qwen (no T)comparison-2 vs sidequest-124501.35e-162.02e-160.9600.3958450.0415710.354274 11 Figure 3: Illustrative summary of double verb family results from our poster presentation [Institut des sciences cognitives, Université du Québec à Montréal, 2026], Fig 30. favor semantically coherent contexts. 34 In addition, because we believe doublev-2 to be harder, we expect we may see reduced performance on some other aspects of the task, suggesting a potential tradeoff amongst limited resources, although we mostly defer that analysis to part 2. For each model, we observe the production of some double verb pairs, and that cosine similarity scores are lower for doublev-2 than doublev-1, shown in Figures 4 and 5; this same pattern holds when we combine all model output, as shown in Figures 20 and 21. Models vary in their ability to perform the task of marking verbs with -qw, but largely placed marked -qw verbs near other verbs when successfully marking a verb. When we clarified the prompt to enforce semantic randomness, semantic performance improved, even if other aspects may have degraded – we do see a reduction in per-run median and mean counts of correctly-labeled verbs from doublev-1 to doublev-2, but will explore potential cognitive tradeoffs in part 2. Additional results for the double verb family of rules are given in Appendix A.8: Table 28, Table 27, Table 24, Table 26, Table 23, Table 25. Overall, models alter generation in the requested syntactic and semantic directions while exhibiting model-dependent difficulty in doing so (Figures 4 and 5). This pattern is consistent with our hypothesis that, in PTGA, meaning-mediated behaviour can coexist with persistent autoregressive plausibility constraints, which aligns with observations reported in McCoy et al. [2023]. 34 From the perspective of PTGA, generating closely related, even identical, verb pairs is one of the easiest ways to satisfy the rule. 12 0.00.20.40.60.81.0 0 50 100 150 200 250 300 350 GPT-4o (no system prompt) n=1,085 Cosine Similarity (qw_is_verb=True) for all runs mean=0.888 median=0.907 5101520253035 0 2 4 6 8 n=50 Correct -qw Verbs (qw_is_verb_count) by run mean=21.7 median=21.5 20406080100 0 2 4 6 8 10 n=50 Correct -qw Verbs + -qw Verb next to other Verb by run mean=44.9 median=44.0 0.00.20.40.60.81.0 0 100 200 300 400 500 600 OLMo n=1,285 mean=0.903 median=0.903 010203040506070 0 2 4 6 8 n=50 mean=25.7 median=27.0 020406080100120140 0 1 2 3 4 5 6 n=50 mean=61.8 median=55.5 0.00.20.40.60.81.0 0 100 200 300 400 Qwen (no thinking) n=1,079 mean=0.897 median=0.902 01020304050607080 0 1 2 3 4 5 6 7 8 n=34 mean=31.7 median=29.0 020406080100120140160 0 2 4 6 8 10 n=34 mean=66.2 median=58.5 0.00.20.40.60.81.0 0 100 200 300 400 500 Qwen n=1,209 mean=0.903 median=0.906 01020304050 0.0 2.5 5.0 7.5 10.0 12.5 15.0 17.5 20.0 n=49 mean=24.7 median=20.0 2030405060708090100 0.0 2.5 5.0 7.5 10.0 12.5 15.0 17.5 n=49 mean=51.2 median=42.0 0.00.20.40.60.81.0 score 0 5 10 15 20 25 30 Llama3-70b n=68 mean=0.810 median=0.871 05101520253035 count 0 1 2 3 4 5 6 n=12 mean=5.7 median=3.0 010203040506070 count 0 1 2 3 4 5 n=12 mean=11.2 median=6.0 Doublev-1 Distributions by Model Figure 4: Doublev-1 Rule Similarity Scores and Verb Counts by Model. Histogram of similarity scores (cosine similarity between marked and closest verb, where marked verb is appropriately terminated with bigram -qw) per model, as well as the per-run count of correctly marked -qw verbs and count of correct pairs of -qw verbs and closest verb. Total number of runs with verbs reported for each model (n = 50runs indicates each run had at least one instance of a correctly assigned -qw verb). Cosine similarity scores closer to 0 indicate dissimilar words. Counts by model that are closer to total clauses reported by model indicate higher performance by model. 3.3 Translation rules Unlike the story and double verb families, translation outputs can be evaluated against explicit answer keys. Successful performance therefore requires not merely producing text with unusual features, but preserving specific relationships under tightly constrained transformations. Correct outputs preserve abstract relationships under a series of composed constraints while (sometimes dramatically) changing the symbols that realize those relationships. Preserving tense, possession, kinship, grammatical roles, and ordering while radically changing the symbols that express them is much more naturally explained by representations of the underlying relations than by superficial token distributions. We investigate two translation conlangs: translate-1, which uses alphabetic characters, and dan-1, which relies primarily on numbers and punctuation. These are the most complex rules in our suite, both in the number of constraints and in the number of relationships among those constraints. Our research question is: can a model preserve relations across disjoint forms, under multiple composed constraints operating at different linguistic levels, even when tokens must be used contrary to their familiar patterns of use? 13 0.00.20.40.60.81.0 0 10 20 30 40 50 60 70 GPT-4o (no system prompt) n=594 Cosine Similarity (qw_is_verb=True) for all runs mean=0.272 median=0.232 0510152025 0 2 4 6 8 10 n=49 Correct -qw Verbs (qw_is_verb_count) by run mean=12.1 median=14.0 1020304050 0 1 2 3 4 5 6 7 n=49 Correct -qw Verbs + -qw Verb next to other Verb by run mean=28.7 median=29.0 0.00.20.40.60.81.0 0 20 40 60 80 100 120 140 OLMo n=1,186 mean=0.243 median=0.178 010203040506070 0 1 2 3 4 5 6 7 n=49 mean=24.2 median=25.0 20406080100120140 0 2 4 6 8 10 n=49 mean=56.6 median=52.0 0.00.20.40.60.81.0 0 20 40 60 80 100 120 140 Qwen (no thinking) n=1,178 mean=0.347 median=0.220 0102030405060 0 5 10 15 20 n=50 mean=23.6 median=20.0 020406080100120 0 5 10 15 20 25 n=50 mean=50.6 median=40.0 0.00.20.40.60.81.0 0 20 40 60 80 100 120 140 160 Qwen n=1,045 mean=0.242 median=0.167 010203040 0 2 4 6 8 10 12 14 n=50 mean=20.9 median=20.0 2030405060708090 0 5 10 15 20 25 n=50 mean=47.5 median=40.0 0.00.20.40.60.81.0 score 0 5 10 15 20 25 Llama3-70b n=131 mean=0.338 median=0.341 510152025303540 count 0.0 0.5 1.0 1.5 2.0 2.5 3.0 n=9 mean=14.6 median=17.0 10203040506070 count 0.0 0.5 1.0 1.5 2.0 2.5 3.0 3.5 4.0 n=9 mean=29.1 median=38.0 Doublev-2 Distributions by Model Figure 5: Doublev-2 Rule Similarity Scores and Verb Counts by Model. Histogram of similarity scores (cosine similarity between marked and closest verb, where marked verb is appropriately terminated with bigram -qw) per model for all verbs paired in all runs, as well as the per-run count of correctly marked -qw verbs and count of correct pairs of -qw verbs and closest verb. Total number of runs with verbs reported for each model (n = 50runs indicates each run had at least one instance of a correctly assigned -qw verb). Cosine similarity scores closer to 0 indicate dissimilar words. Counts by model that are closer to total clauses reported by model indicate higher performance by model. • Translate-1 combines rare bigrams (phonotactically and orthographically implausible) with composed lexical, morphological, syntactic, semantic, and metaphorical mappings, including a temporal/kinship metaphor. •Dan-1 combines similarly composed mappings while eliminating overlap between the symbols used in the prompt and those used in the correct output. For example, the prompt uses “two” while the output requires “2”, and standard word boundaries and punctuation must be remapped. Together, these rules require simultaneous mappings across lexical items, tense, word order, morphol- ogy, semantic and emotional concepts, and relational structure, with approximately 15 constraints in translate-1 and 17 in dan-1. The translation family exploits many overlapping strategies to undermine statistical plausibility at various linguistic scales. For an example from translate-1, in the first system, “-qw indicates past tense, -jx indicates present tense, and -kz indicates future tense,” but in the second system, “-jx indicates past tense, -kz indicates present tense, and -qw indicates future tense”. Note that merely echoing these morphemes without cyclically translating their tense will not correctly preserve meaning. For 14 0.0020.0040.0060.0080.00100.00 0 5 10 15 GPT4o (no system prompt) Levenshtein Distance mean=26.44 med=24.00 0.000.100.200.300.400.500.600.70 0 5 10 15 Normalized Levenshtein mean=0.51 med=0.52 0.000.050.100.150.200.250.300.350.40 0 5 10 15 Jaccard Similarity mean=0.16 med=0.14 0.000.200.400.600.80 0 5 10 15 Cosine Similarity (char n-gram) mean=0.65 med=0.65 0.0010.0020.0030.0040.0050.0060.0070.0080.00 0 10 20 30 OLMo mean=38.20 med=36.00 0.000.100.200.300.400.500.600.700.80 0 10 20 30 mean=0.66 med=0.65 0.000.020.040.060.080.100.120.14 0 10 20 30 mean=0.03 med=0.00 0.000.100.200.300.400.500.60 0 10 20 30 mean=0.39 med=0.40 0.005.0010.0015.0020.0025.0030.0035.00 0 5 10 15 20 Qwen (no thinking) mean=14.14 med=14.50 0.000.100.200.300.400.500.600.70 0 5 10 15 20 mean=0.34 med=0.36 0.000.100.200.300.400.500.60 0 5 10 15 20 mean=0.33 med=0.33 0.000.200.400.600.801.00 0 5 10 15 20 mean=0.69 med=0.73 0.005.0010.0015.0020.0025.0030.0035.0040.00 0 5 10 15 20 Qwen mean=7.84 med=5.00 0.000.200.400.600.801.00 0 5 10 15 20 mean=0.19 med=0.12 0.000.200.400.600.801.00 0 5 10 15 20 mean=0.41 med=0.33 0.000.200.400.600.801.00 0 5 10 15 20 mean=0.81 med=0.92 0.0020.0040.0060.0080.00100.00120.00 0 5 10 15 20 Llama3-70b mean=38.54 med=31.50 0.000.200.400.600.801.00 0 5 10 15 20 mean=0.69 med=0.66 0.000.050.100.150.200.250.300.35 0 5 10 15 20 mean=0.14 med=0.14 0.000.100.200.300.400.500.600.70 0 5 10 15 20 mean=0.34 med=0.39 0.005.0010.0015.0020.0025.0030.0035.0040.00 0 10 20 30 40 50 Random string (select characters) mean=36.72 med=37.00 0.000.200.400.600.801.00 0 10 20 30 40 50 mean=0.92 med=0.93 0.000.100.200.300.400.50 0 10 20 30 40 50 mean=0.00 med=0.00 0.000.020.040.060.080.100.12 0 10 20 30 40 50 mean=0.00 med=0.00 0.005.0010.0015.0020.0025.0030.0035.00 Score 0 10 20 30 40 50 Random string (all characters) mean=35.14 med=35.00 0.000.200.400.600.80 Score 0 10 20 30 40 50 mean=0.88 med=0.88 0.000.100.200.300.400.50 Score 0 10 20 30 40 50 mean=0.00 med=0.00 0.000.100.200.300.400.50 Score 0 10 20 30 40 50 mean=0.00 med=0.00 Translate-1 Similarity Score Distributions by Model & Metric Figure 6: Translate-1 Distributions by Model and Metric. Histogram of string distance scores (Levenshtein distance, normalized Levenshtein distance, Jaccard similarity, and character n-gram cosine similarity) comparing the output for Translate-1 rule versus the correct translated response. Scores closer to 0 for Levenshtein distances, and scores closer to 1 for Jaccard and cosine similarities indicate better rule-following/more accurate translation. Qwen (thinking) performs best for Translate- 1 rule-following based on median and mean scores for all string similarity measures. For both Wilcoxon Rank-Sum and Kolmogorov-Smirnov tests, all models perform better than the random strings variants achieving statistically significantly better median scores (reported in Tables 31 and 32). an example from dan-1, chiasm (“her mother’s cat ... his cat’s father ...”) further targets statistical semantic plausibility. We therefore expect dan-1 likely to be more difficult than translate-1, since it requires symbols, word boundaries, and punctuation marks to be repurposed in unfamiliar ways while simultaneously satisfying an approximately larger set of composed constraints. All models exhibit some rule-following behaviour on translate-1 relative to random controls, with GPT-4o and Qwen (thinking) generally outperforming the other models. As in several of our analyses, the commercial models tend to outperform the open ones (Figures 6, 7). All models produce output distinguishable from random controls on translate-1, but performance declines sharply in dan-1, which uniquely separates the models. Qwen (thinking) clearly outperforms the other models, with a median n-gram cosine similarity of 0.6455, compared to 0.0615 for GPT- 4o. OLMo and Llama have median cosine similarities of 0 on dan-1, indicating no detectable rule-following behaviour by that metric, although some responses appear qualitatively potentially consistent with attempted rule application (Table 30, Table 29). See Tables 33 and 31. The contrast between translate-1 and dan-1 is consistent with PTGA remaining strongly sensitive to surface forms: Performance degrades sharply as successful completion requires increasingly extensive remapping of the necessary output structures. Translation performance also varies substantially under standard machine-translation metrics (BLEU, chrF, TER, METEOR, and a composite of the four). No model approaches perfect performance, but the same qualitative pattern remains visible: GPT-4o and both versions of Qwen cluster together at the higher-performing end, while OLMo and Llama form a lower-performing cluster (Fig. 22). Analyses of truncated model outputs are reported in Appendix Fig. 23). 15 Figure 7: Dan-1 Distributions by Model and Metric. Histogram of string distance scores (Lev- enshtein distance, normalized Levenshtein distance, character n-gram cosine similarity) comparing the output for Dan-1 rule versus the correct translated response. Scores closer to 0 for Levenshtein distances, and scores closer to 1 for cosine similarities indicate better rule-following/more accurate translation. Qwen (thinking) performs best for Dan-1 rule-following based on median and mean scores for all string similarity measures. Median OLMo and Llama3-70B scores against random string (select characters) median cosine similarity scores are not significantly different for the Wilcoxon Rank-Sum Tests (reported in Table 33). Likewise, for the Kolmogorov-Smirnov test, which examines the distributional shapes of the cosine similarity scores, neither OLMo, nor Llama3-70B show statis- tically different scores than the random string (select characters). The distribution of OLMo cosine similarity scores additionally shows no statistical difference with the random string (all characters) using the Kolmogorov-Smirnov test (reported in Table 34). Together with the string-distance measures in Figures 25 and 24, these results suggest substantial but incomplete rule-following ability, with performance degrading as the required mappings become increasingly unfamiliar. Additional statistical analyses are reported in Appendix A.9, where Ta- bles 31, 32, 33, 34 show Wilcoxon Rank-Sum and Kolmogorov-Smirnov tests for the string cosine distances across models for the translation rules. Overall, these results show that models can sometimes generate text according to complex, composed constraints. Qwen (thinking) occasionally produced exact matches to our answer keys. Models can preserve relationships involving tense, possession, kinship, grammatical roles, and word order while dramatically altering the surface forms that express those relationships. Because these tasks have explicit answer keys and are therefore substantially less open-ended than the story rules, 35 they demonstrate that rule-following extends beyond the generation of unusual distributions. At the same time, performance declines as the model has less opportunity to leverage familiar structures, with numerical/punctuation representations appearing harder than alphabetic representations with typical English-like spacing and punctuation. The same systems that perform best on translate-1, namely both Qwen models and GPT-4o, also perform best on dan-1, whereas Llama and OLMo do not exhibit clear rule-following on dan-1. How- ever, performance declines substantially for all models as the representational regime becomes less familiar. GPT-4o and Qwen (no thinking), which perform well on translate-1, perform dramatically worse on dan-1, albeit still distinguishably from random controls (and Llama and OLMo). This 35 Although in Part 1 we searched for exact matches to our answer keys because they are sufficient for an existence proof, and used string-distance measures to identify near matches, the answer keys are not necessarily unique. 16 pattern suggests that architectural differences, 36 affect the ability to maintain composed relationships under increasingly unfamiliar representational regimes (see Gan et al. [2026] for a survey of theories of LLM performance divergence). We explore these possibilities further in Section 3.4. 3.4 Exploratory performance results Caveat: Because of the small number of models and the difficulty of comparing across different metrics and rule families, these results are exploratory. They are more like back-of-the-envelope calculations: We think the gist is potentially meaningful, but we would not put much stock in the exact values, for the reasons described below. Performance appears to broadly track rule complexity, although we defer systematic investigation of this relationship to future work. We include exploratory analyses below; see Appendix A.10 for additional details. We normalize performance scores across rule metrics and compare models using both maximum and minimum-maximum normalization procedures (Fig. 9). Overall, commercial models such as GPT-4o and Qwen generally outperform the open models OLMo and Llama, and (as expected) cleaned outputs yield higher scores than uncleaned outputs. We also observe lower performance on rules that we judged to be more difficult, such as the progression from doublev-1 to doublev-2. We include both highly-normalized (Figure 9) and much-less-normalized (Figure 28) results for a richer picture of model performance. Across models, the relative ordering of rule difficulty is notably stable. The main model differences are in absolute performance level rather than in which rules models find relatively harder or easier. However, rule difficulty is inseparable from the evaluation procedures used to measure it. Different rules are scored with different metrics, operating on different scales and distributions. Some tasks have relatively constrained answer keys, while others permit many valid outputs. Consequently, any ranking of rule difficulty is partly a ranking of the associated evaluation procedures. Throughout the project we used supplementary checks, such as exact-match counts alongside edit-distance measures for dan-1 and translate-1, to help contextualize these differences (Appendix A.3). Wepreliminarilyobservethat,ofarchitecturalfeatures,enable_thinkingand fine_tuned_and_alignedshow the most consistent correlations with the composite performance scores (r = 0.71for both, straight and weighted), whilecontext_window,multilingualand tool_useshow strong correlations with the family-weighted composite (r = 0.79,r = 0.87, andr = 0.89, respectively). 37 Notably,total_parametersis negatively correlated with straight composite performance (r =−0.46), driven in part by GPT-4o’s purported∼1.8T parameter count paired with fair rule-following scores, falling short of Qwen (with thinking). We preliminarily observe that, of the features in our rule complexity representation, uses_no_typical_englishis the single best predictor of performance, not being present in sidequest-1, comparison-1, doublev-1, and doublev-2, and being present in comparison-2, translate-1, and dan-1. Since comparison-2 has one primary constraint (output rare bigrams), from that per- spective it would seem easier than doublev-1 and doublev-2, which have 2-3 primary constraints (add an irrelevant verb, mark with a rare bigram morpheme; make the irrelevant verb semantically unrelated to the rest of the context), and translate-1 and dan-1, which each have approximately 15+. However, most models rank comparison-2 as the 3rd most difficult rule, after dan-1 (hardest) and translate-1. Speculatively, this suggests that PTGA models are hindered when familiar functions cannot be expressed in conjunction with the surface forms with which they were learned. 38 We could potentially test this mechanism in future work with something like a lexinvariant model [Huang et al., 2023]. Alternative choices of performance metric, complexity representation, or aggregation procedure would affect the precise values reported here. 39 Nevertheless, the broad pattern appears credible: models perform extremely well on some rules (comparison-1) and struggle substantially on others (translate-1 and especially dan-1). We therefore view the apparent relationship between rule complexity and 36 Broadly construed, including both model architecture and the structure of training and post-training. 37 Since Qwen and GPT-4o are multilingual and allow tool use, and Llama and OLMo don’t. 38 When the models cannot leverage familiar functions using the forms they would have learned them alongside. 39 We explored multiple versions of the relationships between performance, rule features, and architectural features. The versions reported here seemed the most conceptually appropriate, but such choices inevitably involve judgement. 17 Figure 8: We joined rule-level performance with linguistic rule features and architectural features for exploratory correlation analysis (Spearman, n=9 rules / n=5 models). To examine the relationship between (this representation of) rule complexity and performance, we collapsed dan-1 and translate-1 to single rows (averaging perfect match and vicinity scores), ranked rules within each model, and computed per-model Spearman r against rule complexity rank. Figure 9: Heavily normalized performance subset. Comparison-1 is a baseline, asking the model to generate a short story in English. All models perform well. For non-thinking models, cleaned and uncleaned outputs are nearly identical. For Qwen (thinking), the uncleaned rare bigram rate is suppressed by the CoT trace, which is in standard English. Levenshtein scores are normalized twice here. All performance metrics are reoriented so that higher values indicate better performance, then normalized to[0, 1]in two versions for comparison: top-only, where each row is divided by its maximum observed value (best model= 1, others scaled relative to it); and both-ends, where each row is min-max normalized (best= 1, worst= 0). Results are visualized as a heatmap with models as columns and metrics as rows, with missing values (models lacking cleaned output data) grayed out. The former gives a clearer picture of overall model performance (for each rule, which models were comparatively able to do it), while the latter gives a clearer picture of relative model performance (for each rule, which model did best and worst). The comparatively un-normalized version of this performance heatmap is included in the Appendix in Figure 28. performance as a potentially meaningful signal that warrants further investigation (Fig. 8). Future work will explore richer representations of rule complexity beyond the binary feature encodings used here (Fig. 26; Table 36). 18 Table 6: Raw performance values and within-model ranks (1 = worst performance/ hardest rule, 7 = best performance/ easiest rule) for each rule after collapsing dan-1 and translate-1 to single rows (average of perfect match and vicinity scores). Qwen3(Thinking) (val) Qwen3(Thinking) (rank) GPT-4 (val) GPT-4 (rank) Qwen3 (No Thinking) (val) Qwen3 (No Thinking) (rank) OLMo-2 (val) OLMo-2 (rank) Llama-3 (val) Llama-3 (rank) rule comparison-10.999870.999870.999870.999770.99987 comparison-20.590430.512730.380030.690030.09102 sidequest-10.946760.982860.934160.991960.95806 doublev-10.906350.907350.901650.903450.87055 doublev-20.833140.768040.780040.822040.65904 dan-10.365510.144110.182910.122110.01001 translate-10.379920.287020.319520.208520.19563 4 Discussion Across all three rule families, models moved in the direction predicted by the meaning of the rules. The same pattern appears across multiple models, runs, conlangs, tests, and independent strategies for dissociating meaning-mediated and statistical plausibility. Taken together, these results are increasingly difficult to explain using only superficial statistical regularities and are substantially better explained by meaning-mediated abstraction. Strictly speaking, rejecting the SSP requires only compelling evidence that some outputs demand explanations beyond superficial pattern matching. However, the consistency of the observed patterns exceeds that minimum. Performance varies across models and rule families – harder rule families elicited lower compliance, and open-weight models underperformed compared to their commercial counterparts – but the predicted directional shifts recur across all three families, suggesting that the results are not a fluke, nor peculiar to a particular architecture, training corpus, or task. 40 Our results are summarized in Table 1. Also recall that each conlang family removes a different, more superficial shortcut as to what statistical patterns are available as potential explanations. In brief, the story family weakens prompt exposure as a sufficient explanation, the double verb family rules out simple semantic coherence as a sufficient explanation, and the translation family rules out even complex statistical explanations aimed at producing conlang-like surface forms by requiring preservation of specific relationships under tightly constrained transformations. Because the translation tasks additionally have explicit answer keys, we can directly verify that those relationships were occasionally perfectly preserved. The convergence of these independent dissociations is stronger than any individual result in isolation. The central result is therefore the kind of behavior the models exhibit (not their performance rankings): they produce some text that is plausible only under the meaning-mediated interpretation of the prompt. Without that meaning, the outputs are locally bizarre and unpredictable, but taking meaning into account, they are systematic and predictable. Several caveats are important. We assess rule-following at the output level, so our experiments cannot determine whether failures reflect lack of rule representation or difficulty retrieving and producing the appropriate form. Conversely, successful output behavior supports an abstract-representation explanation without uniquely identifying its implementation. Instead, we construct our experiment to rule out competing superficial patterns as viable explanations, such that successful rule-following is best explained by some higher-level form of plausibility overriding lower-level local continuation pressures. We therefore use terms such as “meaning-mediated” functionally, not as claims about human-equivalent subjective understanding. Future work could test this proposed mechanism more directly using internal-state analyses. Those and earlier caveats aside (Section 3.4), we did design the rules with conjectured difficulty orderings in mind. At the family level, we expected the story rules to be easiest, followed by the double verb rules, followed by the translation rules. Within each family, we also had expectations about the relative difficulty of individual rules: comparison-1 < sidequest-1 < comparison-2; doublev- 1 < doublev-2; and translate-1≲dan-1. However, we were less confident about the relative ordering of individual rules across different families. We therefore expected a family-level ordering and a within-family ordering, but not necessarily transitivity across those levels into a global ordering. 41 This intuition as to rule difficulty is imperfectly reflected in our binary feature-based complexity metric. Because all features are weighted equally, for example, dan-1 is assigned a lower complexity 40 We take no further stance on what the internal representations are like, nor how they compare to human cognition; we argue only that models are operating somewhere above the lower bound implied by purely superficial pattern-matching. 41 In other words, the space between some rules in the same family might be quite large, and the space between others relatively closer together. 19 score than translate-1 despite our expectation that it may be more difficult for models because of the total remapping of tokens required. 42 That model performance broadly tracks these expected family-level and within-family difficulty relationships is itself suggestive of meaning-mediated generation. If successful performance were primarily driven by superficial statistical patterns, it is unclear why the availability of such patterns would track independent representations of rule difficulty or rule complexity. By contrast, a meaning- mediated account naturally predicts that performance should degrade as the conceptual complexity of a rule increases. Notably, the three families implicate different forms of meaning-mediated structure. The story rules involve something akin to a pragmatic distinction between use and mention, the double verb rules involve syntactic, morphological, and semantic manipulation, and the translation rules involve preserving relational structure via multiple composed constraints at various linguistic levels across dramatically different surface forms. The convergence of evidence across these distinct domains strengthens the case against explanations based solely on superficial statistical regularities. Overall, we see the ordering of model performance as basically Qwen (thinking)>or≈GPT-4o ≈Qwen (no thinking)>OLMo≈Llama, with all models>random for translate-1, but only Qwen (thinking) at all performant on dan-1, although GPT-4o and Qwen (no thinking) were at least distinguishable from random performance. Llama is the only model in our set that approaches a purely plausible-text-generation architecture. Overall, we interpret Llama as showing signs of (imperfect) rule-following, and, more specifically, of occasionally overriding concrete statistical priors in favour of plausibility mediated at more abstract levels. We view this behaviour as more-or-less emergent, 43 albeit with substantially greater variability and less consistent evidence than seen in the strongest-performing models. This is why the strong stochastic-parrot hypothesis and distributional accounts of meaning should not be treated as simple opposites: distributional evidence is one way systems can arrive at increasingly abstract representations. Our takeaway is not that models escape statistics, but that statistical learning can produce abstractions, and that later training may help surface or prioritize those abstractions when they conflict with lower-level plausibility pressures (consistent with Hewitt et al. [2024]). 44 4.1 LLMs are not stochastic parrots, but they can act like them: the cognitive constraints of PTGA In a purely plausible-text-generation architecture, the ability to override lower-level plausibility pressures in favour of more abstract ones would be emergent, arising as the model bootstrapped increasingly abstract forms of plausibility from an objective defined at approximately token-scale. 45 The story rules may provide an example consistent with that description. Rare bigrams presented as prompt content produce very different behavior from rare bigrams presented as the operational components of a language. We cannot attribute that difference solely to the change in communicative role, since the prompts are not otherwise identical, but exposure to the forms themselves is clearly not sufficient to explain the observed behavior. 42 We preferred a bottom-up representation of complexity rather than constructing a metric guaranteed to reproduce our prior expectations about difficulty. As discussed above, many alternative representations are possible. 43 Plausibility is optimized for at approximately the token scale, so we describe plausibility that appears to be driven by aspects of meaning significantly beyond that scale as more-or-less emergent. Especially in architectures so different from our own, it is not obvious that next-token prediction would give rise to representations of this kind. 44 A very speculative thread to be pulled in future work: Depending on how you look at it, these results could place pressure on strong poverty-of-the-stimulus or Universal Grammar arguments. LLMs have radically different, simpler architectures than we do, without obvious specialized areas analogous to most of ours, and completely different learning histories, yet nevertheless appear able to bootstrap from concrete statistical patterns to behaviour better explained by meaning-mediated abstractions than by superficial statistical regularities. Their representations must be built entirely out of inferential lexical (or token) relationships. We do not think our results establish any particular theory of language acquisition or cognition. However, they may suggest that statistical learning can support richer abstract representations than strong versions of those arguments permit, raising broader questions about how much of linguistic competence and even cognition can emerge from distributional experience, and about the extent to which language and thought can ultimately be dissociated. 45 We are imagining something like a pattern-to-thought transition where increasingly abstract structures emerge from relationships among more concrete structures, perhaps along the lines of Selfridge [1959], Bradley et al. [2021], or even aspects of Miller et al. [2026]. We imagine this process as broadly consistent with the Distributional Hypothesis [Harris, 1954, 1968, Sahlgren, 2008] and the relation/locality account in Zimmerman [2025]. 20 But it is not just surface-form-related priors that must sometimes be overridden in order to produce meaning-mediated behaviour. In the double verb rules, we observe a large directional response to an abstract semantic relation conveyed in the instruction. This moves our inference about the relevant underlying patterns further up the ladder of abstraction. The relevant pattern is not token co-occurrence, but more abstract: the model apparently learned semantic representations for a set of verbs via statistical patterns including token co-occurrence, and now acts based on a relationship comparing those representations when prompted to do so. In other words, distributional learning may have constructed a semantic space, and the prompt refers to a relation within that space rather than to any particular token sequence. The model therefore changes generation in the requested direction, demonstrating plausibility operating at a level above simple token co-occurrence, consistent with the results of Hewitt et al. [2024]. This distinction is important because semantic coherence is itself a powerful plausibility prior. Although more abstract than token co-occurrence, it remains a prominent statistical pattern in natural- language data. The SSP must accommodate some abstractions of this kind, for example, those needed to support neologisms 46 , but generally maintains that models remain very close to the lower bound of superficial pattern matching. The double verb results suggest otherwise: models appear capable not only of learning semantic relationships from statistical patterns, but also of acting on those relationships in accordance with novel instructions. Furthermore, both the contrast between dan-1 and translate-1 and the relative difficulty of comparison- 2 are suggestive of a less-discussed aspect of PTGA: the important role that tokens specifically must play in these architectures, acting as distributional, organizational, and epistemic primitives. More generally, we see an exploratory correspondence between difficulty and output that does not re-use familiar English forms in familiar ways (Section 3.4). More exploration of PTGA as a strong instantiation of the primacy of language for thought can be found in Zimmerman et al. [2024b] and Zimmerman [2025], but the short version is that operations defined at approximately the token scale may naturally privilege approximately token-scale structures. 47 This makes the more concrete aspects of meaning – such as morphology, syntax, and perhaps some semantics – relatively perceptible to the model, since they are often legible on the scale of words and morphemes (like many tokens). Larger structures, often commensurate with aspects of meaning scaffolding discourse and pragmatics, may therefore be more “out of focus” for PTGA (Figure 30). 48 Consistent with this interpretation, in forthcoming work we find that discourse-level pragmatic plausibility is one facet of linguistic performance in which people perceive current models to be deficient [Morgenstern et al., 2026]. The translation rules provide evidence consistent with this account. Rule-following becomes no- ticeably harder when existing structure can’t be used in familiar ways. Tasks requiring output that departs substantially from familiar tokenization patterns (such as orthography, punctuation, and word- boundary conventions) are more difficult, with some models ultimately demonstrating no discernible rule-following (by the metrics we used). Generally, rules that re-use familiar tokens together with some familiar aspects of meaning or function appear easier to execute than rules requiring extensive remapping of those structures. This suggests that tokenization and distributional priors don’t just support generation, but also constrain it, even when models can exhibit meaning-mediated behaviour (Figure 26, Table 36, Table 37). 46 Recent work explores neologism learning which introduces new meanings for existing words by introducing a new embedding, with no other changes in the model parameters Hewitt et al. [2025]. Hewitt et al. [2025] show that models can self-verbalize the meaning of these neologisms, even when they fall outside of human familiarity (e.g. even when the word has no direct, tangible corollary in common English), demonstrating that LLMs seem to have a multifaceted capacity to work with neologisms (as new concepts and new symbols). In addition, anecdotally, it is not difficult to get an LLM to generate a credibly novel surface form (for example, prompting a model to combine Greek, Latin, French, or Germanic roots to create new jargon for a specific concept). Even if some of those neologisms are not truly novel, once you encounter enough of them, it seems fairly plausible that at least one of them is. 47 Characters-per-token provides a rough empirical illustration. Outputs that depart substantially from familiar tokenization patterns are harder – here meaning computationally more intensive – for all models compared to other texts of the same length. English-like outputs (comparison-1, sidequest-1) have higher characters-per-token because tokenizers chunk familiar English stems efficiently; conlang outputs (especially dan-1) fragment into far more tokens over the same length of text, requiring more forward passes and more resources [Gladstone et al., 2025, Wang et al., 2025]. See Appendix 2.5. This also provides indirect evidence that models succeeding on these tasks are not simply recombining familiar token sequences: the correct outputs are structurally unfamiliar at the level at which PTGA operates. 48 Moreover, the model directly perceives only the inferential traces of such structures within language, not their extradiegetic aspects [Zimmerman et al., 2024a]. 21 4.2 Be wary of benchmarks The difficulty of our constructed-language tasks is notable because the models evaluated here have performed strongly on established reasoning and language-understanding benchmarks. For example, GPT-4 [OpenAI et al., 2024] achieved 86.4% on MMLU, a benchmark of knowledge and reasoning across academic subjects [Hendrycks et al., 2020], and 92.0% on GSM8K, which evaluates multi-step mathematical reasoning [Cobbe et al., 2021]. It also achieved 95.3% on HellaSwag, a commonsense sentence-completion benchmark [Zellers et al., 2019], 87.5% on WinoGrande, which evaluates commonsense pronoun resolution, and an F1 score of 80.9 on DROP, a reading-comprehension benchmark requiring discrete reasoning [Dua et al., 2019]. Llama-3-70B also performs strongly across standard evaluations [AI@Meta, 2024]: the base model achieved 79.5% on MMLU, 81.3% on BIG-Bench Hard, which tests challenging multi-step and algorithmic reasoning [Suzgun et al., 2023], 93.0% on ARC-Challenge, a difficult grade-school science reasoning benchmark [Clark et al., 2018], 85.6% on SQuAD, a reading-comprehension benchmark [Rajpurkar et al., 2016], and 79.7 on DROP. Qwen3-32B [Yang et al., 2025] in thinking mode reports 90.9% on MMLU-Redux, a revised version of MMLU [Gema et al., 2025], 68.4% on GPQA, a graduate-level scientific reasoning benchmark, and 81.4% on AIME 2024, a competition-level mathematical reasoning benchmark [AIME, 2025] whereas its non-thinking mode achieves 85.7% on MMLU-Redux and 54.6% on GPQA. OLMo-2- 32B-Instruct also performs competitively [OLMo et al., 2024], reaching 87.6% on GSM8K, 70.6% on BIG-Bench Hard, 78.0 on DROP, 77.3% on MMLU, and 85.6% on IFEval, which evaluates adherence to verifiable natural-language instructions [Zhou et al., 2023]. One explanation for the disparity in performance is that our constructed-language tasks simply do not probe abilities that fall under reasoning or language competence. 49 We don’t find this explanation compelling, because we view meaning-sensitive abstraction as a necessary subset of many things commonly called reasoning. 50 On the other hand – and consistent with work such as GSM-Symbolic, which shows that strong bench- mark performance can degrade under controlled modifications of GSM8K-style problems [Mirzadeh et al., 2024] – we view a lack of sufficient construct validity in at least some of these benchmarks as the more plausible explanation. Many benchmarks do not clearly establish that task success requires the capability they purport to measure. 51 As a result, benchmarks do not necessarily measure what they claim to measure. This implies that evaluating output-level LLM behaviour against human performance or answer keys is not enough. Deploying LLMs requires intense scrutiny through evaluation processes tailored to the intended deployment context [Schaekermann et al., 2026]. While we do not recommend our conlangs as new benchmarks, we do view them as demonstrations that creative task design can reveal gaps, assumptions, and construct-validity failures in existing benchmarks. 4.3 Implications for model development In Section 3.4, exploratory analyses suggest that model properties related to training and cognitive strategy may be more predictive than raw scale for performance on these tasks. Although highly preliminary, this hints that architectural and training interventions may be more effective than scaling alone for improving performance on certain kinds of reasoning tasks 52 : Qwen3-32B with thinking enabled outperforms GPT-4o on specific tests – and arguably in general – despite being much smaller. These observations, together with Section 4.1, suggest two related ways in which PTGA may tether generation too strongly to superficial plausibility. First, PTGA may disproportionately privilege relatively superficial structures as the determinants of plausible continuation, even when semantic, discourse-level, or otherwise more abstract relationships should govern generation instead. Second, PTGA may bind linguistic forms too tightly to the functions with which they were learned during 49 And that existing benchmarks either do, or fail in different ways. 50 We do not claim to demonstrate “reasoning”, but we do show meaning-sensitive abstraction. 51 Validity may fail at multiple levels. For example, benchmark designers may insufficiently consider the effects of architecture or embodiment on task performance, or may rely on poorly specified notions of the underlying human capability in the first place. 52 The kind of reasoning these tasks probe, whatever that might be called, and perhaps reasoning more generally. 22 training, making familiar functions harder to express through unfamiliar forms. 53 In both cases, superficial patterns – implicating the token-scale perspective of PTGA – overly dominate what the model learned, skewing the model towards the shallow end of the meaning spectrum. We speculate that interventions such as chain-of-thought reasoning, instruction tuning, and fine- tuning help by making different relationships salient to the model, allowing more abstract structures 54 to exert greater influence on what counts as a plausible continuation. 55 While this most directly addresses the first constraint, it may also loosen learned associations between familiar forms and functions. The imputed similar cognitive mechanism underlying these approaches implies that the relevant limitation is not necessarily whether a model has learned an abstraction, but whether that abstraction can be used independently of the surface forms through which it was learned. 56 This possibility suggests a connection between those interventions and other cognitive strategies such as joint-embedding architectures, multimodal training, and vision-language contrastive learning. Such methods may succeed partly because they force representational unity across more heterogeneous observations, requiring the model to preserve abstractions that transcend particular surface forms. In that sense, they partially separate form from function. They also imply another direction: that scale or resolution may itself be understood as a kind of mode. 57 Vision is conventionally treated as one modality, but close inspection and perception at a distance require different lenses and make different relationships salient (Figure 30). Language likewise supports multiple scales of focus, from morphemes to narrative. What is cognitively local depends on the scale of representation: relationships distant at the token level may become local within a more abstract representation. 58 PTGA may therefore be limited not only by its reliance on concrete linguistic forms, but also by its reliance on a relatively fixed representational resolution. A promising direction is to develop systems that can shift scale, allowing different relationships to come into focus as the task demands. Scaling 59 assumes that exposing models to more language gives them access to more knowledge. But if that language is interpreted primarily through superficial plausibility, much of the higher-level meaning it encodes for human speakers may remain inaccessible. Our results suggest a different possibility: The bottleneck may not be knowledge acquisition, but control over which kind of plausibility governs generation. By contrast, people appear able to shift fluidly between linguistic abstractions not unlike how they can shift visual focus. Formulaic exchanges can often be handled through routine lexical and syntactic patterns, whereas interpreting a poem may require simultaneous attention to referential, inferential, indexical, iconic, prosodic, and pragmatic significance. Both involve the same linear sequence of linguistic forms, but they recruit very different levels of interpre- tation. Consequently, we suspect that statistical plausibility is likewise a powerful and efficient proxy in human cognition, but one that can be overridden. We may generally rely on the least effortful level that suffices, moving from familiar surface patterns toward semantic and pragmatic reasoning when lower-effort cues fail, when different kinds of plausibility conflict, or depending on what aspects of meaning are consciously accessible. Without committing to the details, it is clear that different forms of linguistic competence help us determine which interpretations and utterances are most plausible in different contexts. 60 53 Models may acquire useful representations of functions such as morphological marking, syntactic ordering, and relational mapping between verbs and objects, yet struggle to apply those functions through unfamiliar tokens and surface structures. Somehow, the representations themselves may be too concrete, too entangled with their forms. 54 More abstract aspects of meaning. 55 The reason some computer scientists are seemingly independently rediscovering Grice’s maxims via chain-of-thought prompting may be specifically because CoT works by directing focus toward intermediate and higher levels of structure, including some pragmatic and discourse level ones [Grice, 1989, Zimmerman, 2025, Zimmerman et al., 2024b]. Grice’s maxims, after all, are essentially a model for how speakers decide what utterances can plausibly come next in a conversation. See, for examples of Gricean logic underlying computer science methods: Cheng et al. [2025], Wei et al. [2025]. 56 In other words, whether the representation is sufficiently stripped of the concrete forms of the observations that gave rise to it so as to function independently of them. 57 All of these approaches might owe their successes to a metaphorical “multimodality”, which likewise implies that scale could be further exploited in other domains. These approaches may work by changing what kinds of relations become salient for the model – by changing the resolution at which the model perceives – and maybe eventually, by allowing the model to metacognitively reflect on the possibility of different resolutions. 58 Just as information in non-adjacent layers becomes architecturally local under the addition of skip connections. 59 As a strategy for improving model performance. 60 Training objectives operating across multiple linguistic scales, or mechanisms that distinguish and coordinate different modes of plausibility, might therefore offer a promising direction. Since token structure may currently scaffold higher-level 23 Overall, our results suggest that although LLMs are not bounded by superficial statistical pattern matching, PTGA may give models access to meaning across multiple levels while simultaneously anchoring generation disproportionately to the most superficial and fungible among them. Plausible-text generation remains a strong default prior whose influence is not yet reliably modulated when different kinds of plausibility conflict. Future work should ask when models shift among plausibility regimes and what training or architectural changes make such shifts possible. 5 Acknowledgements The authors are grateful for conversations with Ashley M. A. Fehr, William Wheeler, Grayson Wycliffe Storer, Neil Traft, Fitzwilliam Keenan-Koch, Kam Bielawski, Collin Coil, Thomas Pashby, Danny Benett, Alejandro Javier Ruiz Iglesias, Nate Gaylinn, Katie Ekstrom Grenon, Dave Jangraw, Tanis Sheehan, Jim Lawson, Evan Hinchliffe, Milo Z. Trujillo, Christopher M. Danforth, Peter Sheridan Dodds, Alice Patania, Juniper L. Lovato, Kyle Mahowald, Iuliia Kotseruba, Paul Thagard, and for support furnished by the National Science Foundation (Grant #2242829). In addition, we are grateful to ISC Summer School 2026 [Institut des sciences cognitives, Université du Québec à Montréal, 2026], IC2S2 2026 [12th International Conference on Computational Social Science, 2026], and SLSA/ 4S 2026 [Society for Literature, Society, and the Arts, 2026, Society for Social Studies of Science, 2026] for the opportunity to discuss this and related works, and to get valuable feedback from other participants. linguistic operations (e.g. grammatical and semantic), revisiting the fixed scale at the heart of PTGA – and asking what would change at other scales – seems a promising direction. 24 References 12th International Conference on Computational Social Science. IC2S2. https://ic2s2-2026.org/, 2026. Burlington, Vermont. July 28-31. Asif Agha. The social life of cultural value. Language & Communication, 23(3-4):231–273, 2003. doi: 10.1016/S0271-5309(03)00012-0. URL https://doi.org/10.1016/S0271-5309(03)00012-0. AIME. Aime problems and solutions. https://artofproblemsolving.com/wiki/index.php/AIME_ Problems_and_Solutions, 2025. AI@Meta. Llama 3 model card. 2024. URL https://github.com/meta-llama/llama3/blob/main/ MODEL_CARD.md. Andrew M. Bean, Ryan Othniel Kearns, Angelika Romanou, Franziska Sofia Hafner, Harry Mayne, Jan Batzner, Negar Foroutan, Chris Schmitz, Karolina Korgul, Hunar Batra, Oishi Deb, Emma Beharry, Cornelius Emde, Thomas Foster, Anna Gausen, María Grandury, Simeng Han, Valentin Hofmann, Lujain Ibrahim, Hazel Kim, Hannah Rose Kirk, Fangru Lin, Gabrielle Kaili-May Liu, Lennart Luettgau, Jabez Magomere, Jonathan Rystrøm, Anna Sotnikova, Yushi Yang, Yilun Zhao, Adel Bibi, Antoine Bosselut, Ronald Clark, Arman Cohan, Jakob Nicolaus Foerster, Yarin Gal, Scott A. Hale, Inioluwa Deborah Raji, Christopher Summerfield, Philip Torr, Cozmin Ududec, Luc Rocher, and Adam Mahdi. Measuring what matters: Construct validity in large language model benchmarks. In The Thirty-ninth Annual Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2025. URL https://openreview.net/forum?id=mdA5lVvNcU. Emily M. Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, FAccT ’21, page 610–623, New York, NY, USA, 2021. Association for Computing Machinery. ISBN 9781450383097. doi: 10.1145/3442188.3445922. URL https://doi.org/10.1145/3442188.3445922. Aakanksha Bomble, Prashant Kulkarni, and Vishnu Potdar.Does prompt design affect llm outputs? a study across structured and vague inputs. International Journal of Innovative Research in Technology (IJIRT), 12(3):544, August 2025.ISSN 2349-6002.URL https: //ijirt.org/publishedpaper/IJIRT183166_PAPER.pdf. IJIRT 183166. Tai-Danae Bradley, John Terilla, and Yiannis Vlassopoulos. An enriched category theory of language: from syntax to semantics, 2021. URL https://arxiv.org/abs/2106.07890. Zhengxiang Cheng, Dongping Chen, Mingyang Fu, and Tianyi Zhou. Optimizing length compression in large reasoning models, 2025. URL https://arxiv.org/abs/2506.14755. Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge, 2018. URL https://arxiv.org/abs/1803.05457. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems, 2021. URL https://arxiv. org/abs/2110.14168, 9, 2021. Hugo Cui, Freya Behrens, Florent Krzakala, and Lenka Zdeborová. A phase transition between positional and semantic learning in a solvable model of dot-product attention. In Advances in Neural Information Processing Systems 37, NeurIPS 2024, page 36342–36389. Neural Information Processing Systems Foundation, Inc. (NeurIPS), 2024. doi: 10.52202/079017-1146. URL http://dx.doi.org/10.52202/079017-1146. Justin Diamond. "genlangs" and zipf’s law: Do languages generated by chatgpt statistically look human?, 2023. URL https://arxiv.org/abs/2304.12191. Jan Digutsch and Michal Kosinski. Overlap in meaning is a stronger predictor of semantic ac- tivation in gpt-3 than in humans. Scientific Reports, 13, 2023. doi: https://doi.org/10.1038/ s41598-023-32248-6. 25 Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gardner. Drop: A reading comprehension benchmark requiring discrete reasoning over paragraphs. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Compu- tational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 2368–2378, 2019. Penelope Eckert. Variation and the indexical field. Journal of Sociolinguistics, 12(4):453–476, 2008. ISSN 1360-6441. doi: 10.1111/j.1467-9841.2008.00374.x. URL https://doi.org/10.1111/j. 1467-9841.2008.00374.x. Richard Futrell and Michael Hahn. Linguistic structure from a bottleneck on sequential information processing, 2024. Richard Futrell, Edward Gibson, and Roger P. Levy. Lossy-context surprisal: An information- theoretic model of memory effects in sentence processing. Cognitive Science, 44(3):e12814, 2020a. doi: 10.1111/cogs.12814. URL https://doi.org/10.1111/cogs.12814. Richard Futrell, Roger P. Levy, and Edward Gibson. Dependency locality as an explanatory principle for word order. Language, 96(2):371–412, 2020b. doi: 10.1353/lan.2020.0024. Zeyu Gan, Ruifeng Ren, Wei Yao, Xiaolin Hu, Gengze Xu, Chen Qian, Huayi Tang, Zixuan Gong, Xinhao Yao, Pengwei Tang, Zhenxing Dou, and Yong Liu. Beyond the Black Box: A Survey on the Theory and Mechanism of Large Language Models, March 2026. Aryo Pradipta Gema, Joshua Ong Jun Leang, Giwon Hong, Alessio Devoto, Alberto Carlo Maria Mancino, Rohit Saxena, Xuanli He, Yu Zhao, Xiaotang Du, Mohammad Reza Ghasemi Madani, et al. Are we done with mmlu? In Proceedings of the 2025 Conference of the Nations of the Amer- icas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 5069–5096, 2025. Edward Gibson, Richard Futrell, Steven T. Piantadosi, Isabelle Dautriche, Kyle Mahowald, Leon Bergen, and Roger P. Levy. How efficiency shapes human language. Trends in Cognitive Sciences, 23(5):389–407, 2019. doi: 10.1016/j.tics.2019.02.003. URL https://colala.berkeley.edu/papers/ gibson2019how.pdf. James J. Gibson. The Ecological Approach to Visual Perception, chapter 8. Psychology Press, New York and London, classic 2015 edition, 1979. ISBN 978-1-84872-577-5 978-1-84872-578-2 978-1-315-74021-8. URL https://yzhu.io/courses/core/reading/02.gibson.pdf. Chapter title: The Theory of Affordances. Alexi Gladstone, Ganesh Nanduru, Md Mofijul Islam, Peixuan Han, Hyeonjeong Ha, Aman Chadha, Yilun Du, Heng Ji, Jundong Li, and Tariq Iqbal. Energy-based transformers are scalable learners and thinkers, 2025. URL https://arxiv.org/abs/2507.02092. Antonino Greco, Julia Moser, Hubert Preissl, and Markus Siegel. Predictive learning shapes the representational geometry of the human brain. Nature Communications, 15:9670, 2024. doi: 10.1038/s41467-024-54032-4. URL https://doi.org/10.1038/s41467-024-54032-4. H. P. Grice. Studies in the way of words. Harvard University Press, Cambridge, MA, 1989. ISBN 9780674852716. Michael Hahn, Dan Jurafsky, and Richard Futrell. Universals of word order reflect optimization of grammars for efficient communication. PNAS, 117:2347–2353, 2020. doi: https://doi.org/10.1073/ pnas.1910923117. Zellig S. Harris. Distributional structure. WORD, 10:146–162, 1954. doi: 10.1080/00437956.1954. 11659520. URL https://w.tandfonline.com/doi/pdf/10.1080/00437956.1954.11659520. Zellig S. Harris. Mathematical Structures of Language. Interscience Publishers, New York, 1968. Jia He, Mukund Rungta, David Koleczek, Arshdeep Sekhon, Franklin X Wang, and Sadid Hasan. Does prompt formatting have any impact on llm performance?, 2024. URL https://arxiv.org/abs/ 2411.10541. 26 Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt.Measuring massive multitask language understanding.arXiv preprint arXiv:2009.03300, 2020. John Hewitt, Nelson F. Liu, Percy Liang, and Christopher D. Manning. Instruction following without instruction tuning, 2024. URL https://arxiv.org/abs/2409.14254. John Hewitt, Oyvind Tafjord, Robert Geirhos, and Been Kim. Neologism learning for controllability and self-verbalization, 2025. URL https://arxiv.org/abs/2510.08506. Qian Huang, Eric Zelikman, Sarah Li Chen, Yuhuai Wu, Gregory Valiant, and Percy Liang. Lexin- variant language models, 2023. URL https://arxiv.org/abs/2305.16349. Institut des sciences cognitives, Université du Québec à Montréal. ISC École d’été 2026 : Connais- sance, raisonnement et prise de décision. https://sites.grenadine.uqam.ca/sites/isc/fr/iscuqam2026/ home, 2026. Summer school held May 27–June 10, 2026, Université du Québec à Montréal, Montréal, Québec, Canada. Accessed July 7, 2026. Judith T. Irvine and Susan Gal. Language ideology and linguistic differentiation. In Paul V. Kroskrity, editor, Regimes of Language: Ideologies, Polities, and Identities, pages 35–83. School of American Research Press, Santa Fe, NM, 2000. URL https://web.stanford.edu/~eckert/PDF/IrvineGal2000. pdf. Julie Kallini, Isabel Papadimitriou, Richard Futrell, Kyle Mahowald, and Christopher Potts. Mission: Impossible language models, 2024. Jiacheng Liu, Sewon Min, Luke Zettlemoyer, Yejin Choi, and Hannaneh Hajishirzi. Infini-gram: Scaling unbounded n-gram language models to a trillion tokens. arXiv preprint arXiv:2401.17377, 2024a. Jinlu Liu, Nan Yang, and Haitao Liu. Distribution of sentence length of english complex sentences. Moderna Språk, 118(3):51–69, 2024b. doi: 10.58221/mosp.v118i3.15574. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach, 2019. URL https://arxiv.org/abs/1907.11692. Juniper Lovato, Julia Witte Zimmerman, Isabelle Smith, Peter Dodds, and Jennifer L. Karson. Foregrounding artist opinions: A survey study on transparency, ownership, and fairness in ai generative art. Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society, 7(1):905–916, 2024. doi: 10.1609/aies.v7i1.31691. URL https://doi.org/10.1609/aies.v7i1.31691. Kyle Mahowald, Anna A. Ivanova, Idan A. Blank, Nancy Kanwisher, Joshua B. Tenenbaum, and Evelina Fedorenko. Dissociating language and thought in large language models, 2024. URL https://arxiv.org/abs/2301.06627. Evie A. Malaia, Julia Krebs, Dietmar Roehm, and Ronnie B. Wilbur. Age of acquisition effects differ across linguistic domains in sign language: EEG evidence. Brain and Language, 200: 104708, January 2020. doi: 10.1016/j.bandl.2019.104708. URL https://pubmed.ncbi.nlm.nih.gov/ 31698097/. M. S. Mayzner and Margaret Elizabeth Tresselt. Tables of Single-letter and Digram Frequency Counts for Various Word-length and Letter-position Combinations, volume 1 of Psychonomic Monograph Supplements. Psychonomic Press, Austin, TX, 1965. R. Thomas McCoy, Shunyu Yao, Dan Friedman, Matthew Hardy, and Thomas L. Griffiths. Embers of autoregression: Understanding large language models through the problem they are trained to solve, 2023. URL https://arxiv.org/abs/2309.13638. Earl K. Miller, Scott L. Brincat, and Jefferson E. Roy. Analog cognition and consciousness. The Journal of Neuroscience, 46(33):e0711262026, 2026. doi: 10.1523/JNEUROSCI.0711-26.2026. URL https://osf.io/preprints/psyarxiv/z48x7_v1. 27 Iman Mirzadeh, Keivan Alizadeh, Hooman Shahrokhi, Oncel Tuzel, Samy Bengio, and Mehrdad Farajtabar. Gsm-symbolic: Understanding the limitations of mathematical reasoning in large language models, 2024. URL https://arxiv.org/abs/2410.05229. Michelle Morgenstern, Julia Witte Zimmerman, Tabia Tanzin Prama, Juniper L. Lovato, Bryn Loft- ness, Calla Beauregard, Kristine Harootunian, and June Clark. Contextuality and contradictions: Exploring the indexicality of ai-generated content, 2026. Presented in talk form at IC2S2 2026 and 4S 2026. Paper in progress. Peter Norvig. English letter frequency counts: Mayzner revisited or ETAOIN SRHLDCU. https: //w.norvig.com/mayzner.html, 2013. Accessed: 2026-08-31. Team OLMo, Pete Walsh, Luca Soldaini, Dirk Groeneveld, Kyle Lo, Shane Arora, Akshita Bhagia, Yuling Gu, Shengyi Huang, Matt Jordan, Nathan Lambert, Dustin Schwenk, Oyvind Tafjord, Taira Anderson, David Atkinson, Faeze Brahman, Christopher Clark, Pradeep Dasigi, Nouha Dziri, Michal Guerquin, Hamish Ivison, Pang Wei Koh, Jiacheng Liu, Saumya Malik, William Merrill, Lester James V. Miranda, Jacob Morrison, Tyler Murray, Crystal Nam, Valentina Pyatkin, Aman Rangapur, Michael Schmitz, Sam Skjonsberg, David Wadden, Christopher Wilhelm, Michael Wilson, Luke Zettlemoyer, Ali Farhadi, Noah A. Smith, and Hannaneh Hajishirzi. 2 olmo 2 furious. arXiv preprint, 2024. URL https://arxiv.org/abs/2501.00656. OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mohammad Bavarian, Jeff Belgum, and [...] Barret Zoph. Gpt-4 technical report, 2024. URL https://arxiv.org/abs/2303. 08774. Charles S. Peirce.The Collected Papers of C.S. Peirce, Vol. I: The Principles of Philosophy, 1931.URL http://w.commens.org/bibliography/anthology_volume/ peirce-charles-s-1931-collected-papers-charles-sanders-peirce-vol-i. From the Commens Bibliog- raphy. Arianna Pera, Mauro Martino, Nima Dehmamy, Douglas Guilbeault, Luca Maria Aiello, and Andrea Baronchelli. Billions of sketches reveal hidden cultural variation in human concepts, 2026. URL https://arxiv.org/abs/2607.07267. Rolf Pfeifer and Josh Bongard. How the Body Shapes the Way We Think: A New View of Intelligence. A Bradford Book, MIT Press, London, England and Cambridge, Massachusetts, 2006. ISBN 978-0262162395. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. Squad: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 conference on empirical methods in natural language processing, pages 2383–2392, 2016. Amon Rapp, Chiara Di Lodovico, Federico Torrielli, and Luigi Di Caro. How do people experience the images created by generative artificial intelligence? an exploration of people’s perceptions, appraisals, and emotions related to a gen-ai text-to-image model and its creations. International Journal of Human-Computer Studies, 193:103375, 2025. doi: 10.1016/j.ijhcs.2024.103375. URL https://doi.org/10.1016/j.ijhcs.2024.103375. Magnus Sahlgren. The distributional hypothesis. Italian Journal of Linguistics, 20:33 – 53, 2008. URL https://w.italian-journal-linguistics.com/app/uploads/2021/05/Sahlgren-1.pdf. Mike Schaekermann, Anil Palepu, Adam Rodman, Ami Parekh, Ethan Goh, Shekoofeh Azizi, Yun Liu, Sunny Virmani, Christina Chen, Dale R. Webster, Joëlle Barral, Avinatan Hassidim, Yossi Matias, and Po-Hsuan Cameron Chen. Prospective evidence for conversational medical AI is hard, but non-negotiable. Nature Medicine, September 2026. doi: 10.1038/s41591-026-04639-5. URL https://w.nature.com/articles/s41591-026-04639-5. Comment. Melanie Sclar, Yejin Choi, Yulia Tsvetkov, and Alane Suhr.Quantifying Language Mod- els’ Sensitivity to Spurious Features in Prompt Design or:How i learned to start worrying about prompt formatting.In International Conference on Learning Rep- resentations (ICLR), 2024.URL https://proceedings.iclr.c/paper_files/paper/2024/hash/ 6c0e99d736da621403018ca7b32b1a4d-Abstract-Conference.html. 28 O. G. Selfridge. Pandemonium: A paradigm for learning. In Mechanisation of Thought Processes: Proceedings of a Symposium Held at the National Physical Laboratory, November 1958, pages 513–526, London, 1959. HMSO. URL https://gwern.net/doc/ai/n/1959-selfridge.pdf. Michael Silverstein. Indexical order and the dialectics of sociolinguistic life. Language and Com- munication, 23(3–4):193–229, July–October 2003. doi: 10.1016/S0271-5309(03)00013-2. URL https://doi.org/10.1016/S0271-5309(03)00013-2. Society for Literature, Society, and the Arts. SLSA Streams at 4S. https://litsciarts.org/, 2026. From October 8-10, 2026, 4S will host the 39th meeting of the Society for Literature, Science and the Arts (SLSA), for a series of integrated streams. Society for Social Studies of Science. 4S. https://4sonline.org/, 2026. 50th Anniversary for the Annual Meeting of the Society for Social Studies of Science, Toronto, Canada. October. Christoph Stöckl, Yukun Yang, and Wolfgang Maass. Local prediction-learning in high-dimensional spaces enables neural networks to plan. Nature Communications, 15:2344, 2024. doi: 10.1038/ s41467-024-46586-0. URL https://doi.org/10.1038/s41467-024-46586-0. Shichao Sun, Ruifeng Yuan, Ziqiang Cao, Wenjie Li, and Pengfei Liu. Prompt chaining or stepwise prompt? refinement in text summarization. arXiv preprint arXiv:2406.00507, 2024. doi: 10.48550/ arXiv.2406.00507. Accepted to Findings of ACL 2024. Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc Le, Ed H Chi, Denny Zhou, et al. Challenging big-bench tasks and whether chain-of-thought can solve them. In Findings of the Association for Computational Linguistics: ACL 2023, pages 13003–13051, 2023. Guan Wang, Jin Li, Yuhao Sun, Xing Chen, Changling Liu, Yue Wu, Meng Lu, Sen Song, and Yasin Abbasi Yadkori. Hierarchical reasoning model, 2025. URL https://arxiv.org/abs/2506.21734. Zhepei Wei, Xiao Yang, Kai Sun, Jiaqi Wang, Rulin Shao, Sean Chen, Mohammad Kachuee, Teja Gollapudi, Tony Liao, Nicolas Scheffer, Rakesh Wanga, Anuj Kumar, Yu Meng, Wen tau Yih, and Xin Luna Dong. Truthrl: Incentivizing truthful llms via reinforcement learning, 2025. URL https://arxiv.org/abs/2509.25760. Ethan Wilcox, Roger Levy, and Richard Futrell. Hierarchical representation in neural language models: Suppression and recovery of expectations. In Tal Linzen, Grzegorz Chrupała, Yonatan Belinkov, and Dieuwke Hupkes, editors, Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 181–190, Florence, Italy, August 2019. Association for Computational Linguistics. doi: 10.18653/v1/W19-4819. URL https: //aclanthology.org/W19-4819/. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 2025. Frances Yates. The Art of Memory. University of Chicago Press, 1966. Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence? In Proceedings of the 57th annual meeting of the association for computational linguistics, pages 4791–4800, 2019. Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. Instruction-following evaluation for large language models. arXiv preprint arXiv:2311.07911, 2023. Julia Witte Zimmerman. Locality, Relation, and Meaning Construction in Language, as Implemented in Humans and Large Language Models (LLMs). Ph.d. dissertation, University of Vermont, 2025. URL https://hdl.handle.net/20.500.14849/4063. Available at ScholarWorks@UVM. Julia Witte Zimmerman, Denis Hudon, Kathryn Cramer, Jonathan St. Onge, Mikaela Fudolig, Milo Z. Trujillo, Christopher M. Danforth, and Peter Sheridan Dodds. A blind spot for large language models: Supradiegetic linguistic information. Plutonics, 17:107–156, 2024a. URL https://plutonicsjournal.com/volume-xvii/. 29 Julia Witte Zimmerman, Denis Hudon, Kathryn Cramer, Alejandro J. Ruiz, Calla Beauregard, Ashley Fehr, Mikaela Irene Fudolig, Bradford Demarest, Yoshi Meke Bird, Milo Z. Trujillo, Christopher M. Danforth, and Peter Sheridan Dodds. Tokens, the oft-overlooked appetizer: Large language models, the distributional hypothesis, and meaning, 2024b. URL https://arxiv.org/abs/2412.10924. Presented as talks at IC2S2 2026 and SLSA at 4S 2026. Julia Witte Zimmerman, Bryn Loftness, Calla Beauregard, Kristine Harootunian, June Clark, Hannah Reing, Katie Ekstrom Grenon, Michelle Morgenstern, Tabia Tanzin Prama, and Juniper L. Lovato. Vermont AI Literacy project, 2026. Ongoing open-source curriculum and materials, co-designed and piloted Spring 2026 - Summer 2027. Paper in progress. George Kingsley Zipf. The Psycho-Biology of Language: An Introduction to Dynamic Philology. The MIT Press, Boston, MA, 1965. ISBN 9780262740029. Originally published in 1935; this edition published in 1965. A Appendices and supplementary material Technical appendices with additional details, results, figures, graphs, and discussion. A.1 Disclosures • Much of our work is saved in a private GitHub repository, in a series of Google spreadsheets and docs, in Colab Notebooks, and in draft and note form in several Overleaf projects. We will provide access to any of these materials to interested parties upon reasonable request. The team also communicated via slack channel and in-person meetings. Upon reasonable request, J.W.Z. would be happy to answer any questions about process or methods. •Besides using Generative AI models as subjects in our experimental pipeline, J.W.Z. used an institutional Copilot account at various points throughout this work. C.G.B. used ChatGPT and Claude Opus 4.8 for code audit and formatting. Other authors used Generative AI for additional tasks. A.2 Limitations and Future Work While this paper tests the strong stochastic parrot argument in Large Language Models, we focus on relatively few models, ranging from older (Llama3-70B) to newer models (Qwen-3 (Thinking)), open to closed models, and simpler to more complex architectures. Additionally, we selected two separate thinking modes for Qwen-3 on the presumption that changing that setting would be likely to impact results. We selected well-established baseline settings for each model. Based on our preliminary results here, future work could sweep across other model parameters, and incorporate other models, to investigate the breadth of relevant behavior across LLMs under different conditions. Small changes in formatting and wording, as in prompt engineering, have been shown to impact model behavior across various tasks [Bomble et al., 2025, He et al., 2024]. Here, while we provide three separate wording variations for each rule prompt, we expect that very different wording prompting the same rules would likely impact results. Future work could explore different variations of our prompt structures, or use iterative prompt chaining methods which may increase performance Sun et al. [2024]. Regarding our choice of performance metrics, future work will also explore output length, either in terms of characters, words, or sentences. Overall, we hypothesize that valid output length should decrease with harder tasks, demonstrating resource tradeoffs (as one aspect of a task requires more effort, other aspects of the task which the model is otherwise capable of may be degraded). 61 However, because we observe that model outputs can fall into degenerative cycles, suggestive of something like strange attractors in the latent space, we don’t expect output length will always be straightforwardly interpretable. Although we did what we could to prevent these five models from being exposed to our rules ahead of time, and to keep each run independent, because of the black box nature of the commercial models, 61 We can use our control outputs as a baseline as well as previously-reported sentence lengths [Liu et al., 2024b]. 30 we can’t truly guarantee this. During our initial pilot testing, ChatGPT seemingly recalled aspects of the project weeks later (fragments of a rule), so it is possible, for example, that ChatGPT flags unusual tasks internally for use by OpenAI, OpenAI sells that training data, and that data somehow made its way into one of these models before we collected our data (given the project time frame, and the models we used for the initial piloting phase versus the data-generation phase, we think that is very unlikely, but we cannot guarantee it). Some extra runs, GPT-4o re-runs (from an initial confusing system prompt), and the aforementioned sidequest-1-fix runs exist, but are excluded from the analyses here. Additional plans for future work are mentioned throughout the rest of the text where relevant. A.3 Construct validity and additional methodological detail After coming up with our plan and our rules, we used the checklist for LLM benchmark construct validity developed by Bean et al. [2025] to help us think through our choices, augment gaps, and revise ambiguous steps. Here we enumerate the checklist items and our answers: Define the phenomenon: Our goal is to ascertain whether or not any kind of LLM can produce output that evinces compelling evidence of abstraction-based rule-following, where the rules are both (1) within what we theoretically believe LLMs could have learned, and (2) negligibly likely to have been seen during training – and beyond not being seen verbatim, we want the rules to be significantly dissimilar to any instruction, task, rule, example, or problem that was described at any point during training. The rules themselves need to be designed s.t. any parsimonious explanation invokes abstractions, and cannot invoke pattern-matching at a literal or near-literal level (direct observation), notably by disrupting the connection between the statistical patterns shown in correct output versus those observed during training. Thus, we need to make compelling arguments for both (3) our choice of rules, and (4) how we are assessing whether or not the rules are being followed (what we accept as evidence of rule-following). • Provide a precise and operational definition for the phenomenon being measured: In order to determine if LLM output shows rule-following or not, we give each model 50 attempts (independent from each other; no in-context learning) to respond to each rule. Each rule has 3 variations (although we accidentally repeated one of the prompt variants for sidequest-1, so it actually has 2 variants; we have run a new set and will include analysis in the second paper) to avoid over-interpreting a response highly sensitive on our wording, which are randomly assigned to each attempt. Before gathering any data, we came up with strategies for evaluating outputs as correct or incorrect in response to each rule. Once we evaluated outputs according to these strategies, we also planned to threshold at various levels to a binary correct/ incorrect judgment (we will continue exploring this in the second paper). • Specify the scope of the phenomenon being covered and acknowledge any excluded aspects: We are assessing rule-following at the output level. From one perspective, this is the simplest way to assess the LLM, as behavior is the ultimate (though lossy) consequence of any confluence of internal states. However, this is an incomplete picture of LLM cognition: for example, looking at the likelihoods assigned to various tokens (not just the tokens that were incorporated in the output) would capture how the LLM viewed alternatives, and could show additional evidence of rule-following (as in Hewitt et al. [2024]). We also assume a correlation between rule-following and correctness. However, it is possible that a system could understand a rule in some way but be so stymied by some aspect of actually retrieving and producing the right answer that their internal understanding was not apparent at the output level. 62 Finally, we do not plan to measure the output with respect to every aspect of every rule: rather, we picked out the ones that are most computationally accessible and most important in our conception of what that rule means. There will be ways in which the rules are enacted that are uninteresting for our purposes: for example, generating 20 sentence- length chunks of text but failing to follow any of the grammar is technically following the rule, but not the parts of the rule adhering to ((1) or) (2), and therefore the model’s ability to comply with that instruction is irrelevant to our goal. Thus, it shouldn’t be evidence included in (4). Similarly, following the grammar, but generating fewer or greater than 20 sentences, 62 For example, as might be the case with certain medical conditions and symptoms, like anarthria. 31 is for our purposes essentially perfect rule-following, although worth investigating from the perspective of cognitive trade off (e.g. we could see number of sentences anti-correlated with rule-following). • Identify if the phenomenon has sub-components and ensure they are measured separately: In our case, the sub-components might be the various ways in which an answer could be partially correct, some of which may feel ‘more correct’ than others. We plan to qualitatively analyze some of the output, identify themes, and come to consensus judgments as to these categories. Our strategies for measuring rule-following incorporate these judgments explicitly: if a behaviour is evaluated in our described strategy, than that behaviour felt like an important aspect of following that rule. But just because something wasn’t logistically accessible for that part of the analysis doesn’t necessarily mean it is irrelevant to following that rule, so it could be incorporated during the qualitative portion. We will use the outputs in order to work out how the qualitative portion will work in detail. Measure only the phenomenon •Control for unrelated tasks that may affect the results: We tried to do this re (1) by not choosing rules that rely on skills such as counting, which the model may not acquire as part of a general language faculty. In addition: We provide 3 versions of every rule, which we determined by consensus do not alter the meaning of the rule, but which use different wording and formatting. We use a temperature of 0.7 across all models and trials, since that is considered standard for most applications. By writing the rules by trying them with ChatGPT and DeepSeek, and then doing a small pilot version where we tried a few rules across a set of models, we have some idea of what to expect the output to look like, and have calibrated the final set of rules to be within the hypothetical realm of possibility for LLMs. Those early stages of the project helped us come up with rules that span a range of difficulty, and are unlikely to be trivially obeyed. That is, we do not expect perfect performance, which might make us suspicious that we had failed to meet (2), and we do not expect no model to show any sign of rule-following, which would be hard to interpret and leaves (1) as an open question. •Assess the impact of format constraints on model performance: Based on our pilot explo- ration, we chose a max response length of 4096 tokens, which ought not to overly constrain the models. As to temperature and prompt wording/ formats, see previous response. For this project, we felt significant prompt engineering would, if anything, make our results harder to interpret, by degrading (2). Therefore, we wrote the prompts in language that made sense to us, then used our testing and discussion with ChatGPT and DeepSeek, and our pilot exploration, to write prompts that felt stylistically consistent with prompts, language, and formatting that the models would likely have encountered, but without extensively trying to find the most performant wording. Our basic goal was that the wording of the prompts would not be so bizarre as to push outside of (1), but wouldn’t be so optimized as to make our argument for (2) more complex. • Validate any automated output parsing techniques for accuracy, consistency and bias: In part two, when we analyze the outputs in more specific detail, we will also compare against ‘correct’ example answers that we write ourselves or modify from model responses. When writing the tests, we used various fabricated example answers and excerpts from model responses to evaluate the test behaviour. We will discuss the results to make sure we know how to interpret the output of the tests and calibrate it with our judgments as to whether that output shows rule-following or not. In addition to the above, we use raw-output cross-checks. Cleaning choices could drive results if cleaned outputs systematically favor rule-following interpretations. To guard against this, we applied metrics to raw outputs when possible and verified that the key patterns remained visible before and after cleaning. We investigated cases where they diverged. For example, manual inspection of Qwen (thinking)’s responses revealed that it frequently repeats the prompt verbatim. We confirmed that this inflated rare-bigram counts in raw output for sidequest-1 – which contains rare bigrams in the prompt – by verifying a significant difference between Qwen’s cleaned and uncleaned bigram rates for that condition. This cross-check supports the use of cleaned output as the primary analysis and demonstrates that the observed effect is not an artifact of prompt repetition. When available, we 32 report both cleaned and raw results, so that readers can assess the robustness of the conclusions independently of our cleaning decisions. Construct a representative dataset for the task •Employ sampling strategies to ensure task items are representative of the overall task space: Not applicable. We do not want to test all rule-following or all rules that require abstractions or all tasks that can be described in English but not answered in English (or any other existing language that might have been in the training data). We simply want to establish existence of the relevant behaviour. •Verify the quality and relevance of all task items, especially for large or automatically generated datasets: We have done our best with this in terms of formulating the rules, trying them in pilot explorations with several models, and soliciting feedback on them from multiple colleagues. •Include task items that test known LLM sensitivities (e.g. input permutations or variations): Each rule has 3 versions with different wording and formatting. The order is randomly assigned. We are using the same, fairly standard temperature setting across the board, and otherwise not changing the hyperparameters from any default settings (except to enable or disable thinking mode, or to avoid using a system prompt with GPT-4o). For the ‘instruct’ versus ‘base’ models, we do include the standard wrapper around the prompt (think of this like ‘meeting the model where it is’, to give the model the best chance of understanding the prompt). Acknowledge limitations of reusing datasets •Document whether the benchmark adapts a previous dataset or benchmark: No, it does not. •If so, analyse and report the relevant strengths and limitations of the adapted prior work: Not applicable. •If so, report and compare performance on the new benchmark against the original: Not applicable. • Explain modifications to reused datasets and how they improve construct validity: Not applicable. Prepare for contamination • Implement tests to detect data contamination and apply them to the benchmark: See Sec- tion 2.5 and Appendix A.4. We used the InfiniGram LM to look for pieces of our rules across multiple popular corpora. Based on the very short average amount of each prefix found in the corpora, the few similar contexts found (as shown by the next token distributions), and by the example documents surfaced for the few partial matches, we are confident it’s not the case that all of our tasks and answer keys were in the training data. We also looked at the characters-per-token ratios for some example correct answers to demonstrate that many of our rules require deviating from the typical patterns found in the training data. We chose the OLMo model due to its transparency: since it is open training data, that leaves the possibility of further verifying what of any relevance to this project could have been seen during training as follow-up future work. We could also continue investigating with InfiniGram. • Maintain a held-out set of task items to facilitate ongoing, uncontaminated evaluation: Since it made the most sense to experiment with the rules as we formulated them, we did not hold out a task set. Note that, in formulating and even in experimentally testing the rules, it is possible that the models learned from exposure: during the exploratory pilot, ChatGPT, for example, appeared to recall aspects of this project weeks later, including fragments of at least one rule (something like the testingenglish family of rules). As much as possible, we used new incognito sessions when testing to minimize contextual carryover and in-context learning (during the pilot sessions). Absolute assurance that the models did not learn from these tests would require a fully local, highly interrogated, maybe even custom-made, instance (because currently we are taking third party descriptions of the models at face value). However, because we likely need to use models near the current 33 performance frontier in order to observe unambiguous rule-following purely at the output level (without examining internal states such as logits), this is pretty infeasible. In the experimental portion, we set up the data generation such that each response was independent (to the best of our knowledge, taking documentation for these models at face value). That said, the space of rules satisfying our criteria (1) and (2) is far larger than what we explored. Thus, it should not be too difficult to generate new rule sets for replication! • Investigate the potential pre-exposure of benchmark source materials or similar data in common LLM training corpora: See prior response. We note that rigorously verifying the absence of pre-exposure of our rules in LLM training corpora would constitute a substantial research project in its own right. Nonetheless, several lightweight checks are feasible (and we did quite a few). For open-source models such as OLMo, future work could directly scan the released training data for overlap. Where possible, we incorporated very rare English bigrams in our rules, reasoning that this would correspond to low prevalence in the training data. Use statistical methods to compare models •Report the benchmark’s sample size and justify its statistical power: There are 17 rules, each with 3 variants (except for sidequest-1, which has 2). We used Benjamini-Hochberg correction and Fisher-combined p-values where appropriate. We elicited 50 runs for each rule for each model in order to control for chance. • Report uncertainty estimates for all primary scores to enable robust model comparisons: So far, we haven’t thought this particularly applicable, but maybe we will find an application in part 2. •If using human raters, describe their demographics and mitigate potential demographic biases in rater recruitment and instructions: We were the human raters, or at least, cleaners, in part 1. Specifically, J.W.Z., E.R.K., T.T.P., and C.G.B (women in the 20 - 40 age range, all English-speaking) cleaned the data used in part 1, through a combination of automated and manual methods. We collectively stepped through several example outputs for each rule, for each model, to establish our shared understanding of what should be cleaned. Then we individually reviewed additional outputs and cleaned them. In part 2, we will add additional qualitative analysis of the failure states, but so far we have only done some informal discussion and exploration. For the cleaned data that involved automation, we randomly sampled 3 outputs of each kind and manually verified/ spot-checked them. •Use metrics that capture the inherent variability of any subjective labels, without relying on single-point aggregation or exact matching: We did our best to allow for this in the multi-pronged computational tests of the output, and then in thresholding those results in various ways during analysis. We also tried to liberally include caveats and limitations so that readers would sensibly interpret the results. In part 2, we expect our qualitative analysis to further deepen and contextualize the metrics we are using. Conduct an error analysis • Conduct a qualitative and quantitative analysis of common failure modes: To be done in part 2. For now, we include some initial observations in Appendix A.11. •Investigate whether failure modes correlate with non-targeted phenomena (confounders) rather than the intended construct: Prompt format sensitivity could confound our results. Three wording variations per rule were designed to test this, but the systematic analysis of whether failure modes track prompt version hasn’t been completed. Furthermore, we could come up with many more variants, but we think the most straightforward approach is to stay away from prompt-engineering, and stick with how we would describe these rules to ourselves. As described, the sidequest-1 prompt wording error (where we accidentally left two of three prompt variants identical, rather than using three distinct prompt variants) is an acknowledged confounder that complicates interpretation for that rule. We will analyze the fixed sidequest-2 data in part 2. Similarly, we used standard settings, without parameter sweeps. Namely, we set temperature to 0.7 across the board. We are not interested in coaxing the best possible performance from these models, but searching for proof by existence, so we think that is sufficient for now. Our hypothesis is not strictly about the base model’s 34 abstraction capacity alone, but about whether LLMs – which all contain a plausible text generation architecture as foundation, although many models have various architectures built on top of that – can, under any configuration, create abstractions To be done more in part 2, if possible. At least, we plan to use multiple representations to explore some relationships (such as number of constraints per rule, as well as the existing binary feature presence representation), which ought to help us understand model behaviour more holistically. • If so, identify and discuss any potential scoring biases revealed in the error analysis: To be done in part 2, if possible. • Conduct experiments or propose new directions to improve model scores on the benchmark: The translate-2 rule includes an example, as it was specifically designed to test whether examples improve performance. We have not analyzed this data yet. We would like to investigate how the CoT traces (for Qwen thinking) relate to rule-following performance, qualitatively and quantitatively. Sliding window recovered prefix and characters-per-token measurements across the prompts and outputs would add additional evidence as to similarity between conlangs, training data, and model outputs. (This could improve the validity of the score, but ex post facto model performance.) The space of rules satisfying our methodolog- ical criteria is much larger than what we explored during this project, so replication with new rules is a potential extension. To be done in part 2, but new directions are also variously discussed throughout the rest of the paper, in-line with the relevant topics and in Sec. A.2. Besides these individual pieces of construct validity, we also tried to attend to holistic consistency across our whole argument, by intentionally using convergent, sometimes overlapping, evidence across methods. We cross-check our conclusions using multiple strategies. For statistical testing: Wilcoxon rank-sum tests of difference to understand how models differ directionally in their ability to perform a task, Kolmogorov-Smirnov tests to secondarily understand distributional differences, Fisher-combined p-values to understand overall performance by models in addition to individual, within model differences in performance, rank-biserial effect sizes, and comparisons of means and medians. For output assessment against answer keys: exact matches, string edit distance, and natural language translation metrics to account for both the ability of models to exactly perform tasks as well as closeness or related performance ability. For training data exposure: 13 tokenizers, Google Search, Infini-gram completions, and string matches across 8 training corpora, with tokenizer results cross-checked against each other and against recovered corpus prefixes. Many of our hypotheses additionally cross-check each other, although we only used a subset of them directly in part 1. And, of course, we use multiple conlangs (which act on plausibility in different ways, at different levels of linguistic structure), multiple models (which vary in architectural details), and multiple independent runs for each model. Finally, a broader motivation for our approach is that many evaluations of LLM behavior, LLM- human interactions, and even LLM internal states produce findings whose interpretation outside their original context is uncertain. A result may reflect a general capability, an artifact of training data or training process, a feature of a particular task, or some interaction among these factors (a familiar challenge in psychology, linguistics, anthropology, and many other sciences that deal with complex systems and behaviour). To minimize such ambiguities, we adopted an existence-proof approach. The value of establishing existence here is that it lets us ask a specific and interesting question, yet avoids placing too much weight on frequency: whether a capability appears in every model, some models, under particular training conditions, or when models are used in specific ways is important, but logically secondary to whether the capability exists at all. Showing that meaning-mediated abstraction can occur in at least some PTGA language models is therefore informative even though its mechanisms remain incompletely understood. Moreover, because we observe related patterns across multiple models and conlangs, the phenomenon appears unlikely to be peculiar to a single architecture, training corpus, or task. A.4 Training data exposure To support our claim that successful rule-following cannot be explained by direct memorization, we performed several complementary checks, 63 plausibly covering a broad range of training data sources: 63 These checks are in addition to the features of the conlang rules themselves. 35 (1) extensive preliminary testing in which we observed that a wide range of conlang-related-tasks are not trivial for ChatGPT, DeepSeek, and other models (including GPT-4o, gpt-oss, gpt-2, Qwen2, Qwen3 (32B), Qwen3 (8B), Llama 3.1 Instruct (8B), Llama 3.1 Base (70B), Llama 3.2 Base (3B), gemma2 (2B), gemma2 (9B), Olmo 2 Base (7B), Olmo 2 Base (32B), Olmo 2 Instruct (7B), Olmo 2 Instruct (32B)); (2) a Google search for key phrases from our prompts; (3) a corpus search for key phrases and concepts from our prompts and answer keys across eight popular training corpora; (4) an InfiniGram language model analysis [Liu et al., 2024a] 64 to assess whether correct answers are locally predictable from corpus n-gram statistics (across the same eight corpora); (5) tracking recovered prompt prefix length across the eight corpora to verify key phrases and answer keys from our tasks only matched at most small substrings within the corpora (recovered prefix length measures the degree to which outputs reuse local sequential patterns present in training corpora, e.g. in the OLMo 2 32B Instruct (4.6T) corpus, the longest found suffix matching the answer key for translate-1, “Mary gjyqazkjx pxmqiMary. PxmqiMary qtojx.”, is “tojx.”); (6) a tokenization analysis (across thirteen popular tokenizers) of gold-star example answers to assess where and to what extent the required outputs require deviation from the orthographic structure of typical LLM training data. As a simple check, we searched Google for the full text of three rules (doublev-1, wordorderalternation-3, comparison-1). An excerpt from wordorderalternation-3 65 did not re- turn any relevant-looking results. For comparison-1, we found no results resembling LLM prompts, although many texts contained generic phrases such as “The story began when”, which is ex- pected. Interestingly, the AI-generated search summaries appeared to attempt to follow the rules of wordorderalternation-3 and comparison-1 rather than search and summarize relevant content (see Figure 10). We then repeated these searches with Google’s ‘verbatim’ option, with similar results. We used the InfiniGram API [Liu et al., 2024a] with setting 5 (“Compute the∞-gram next-token distribution”) to query eight corpora: OLMo 2 13B Instruct, OLMoE 1B 7B Instruct, DCLM-baseline, Dolma-v1.7, RedPajama, Pile-train, C4-train, and Pile-val. For each query, we examined the next- token distribution conditioned on a given prefix, the longest prefix recovered from each corpus, and whether any verbatim match existed. We report the recovered prefix length: the longest prefix of a given string that InfiniGram can find in the corpus, measured in characters. Longer recovered prefixes indicate that more of a string’s local sequential structure is present in training data; shorter prefixes indicate that the string deviates from attestedn-gram patterns at an earlier point. Recovered prefix length therefore measures the degree to which a given output reuses local sequential patterns present in training corpora. As correct conlang output deviates from structural similarity in terms of 1D statistical co-occurrence of the corpus (as roughly reflected by the orderings in Figure 13, Figure 14), the continuations predicted by the n-gram LM look less like the correct conlang output. See Table 8, Table 7, Table 13, Table 14, and Figure 18 for details. The condensed prompts mentioned in Table 8 are given in full in the caption of Figure 18. The rules themselves vary as to the resolution of linguistic structure at which they disrupt statistical similarity to English. Note that although both the prefix length and the character-per-token ratios reflect how much the output of each conlang rule deviates from the 1D statistical structure of the training data, neither is a reliable measure of rule difficulty, since linguistic structure is 3D. A rule whose outputs utilize larger existing linguistic structures (like wordorderalternation-3, and the double verb rules) will show longer recovered prefixes and higher character-per-token ratios even though it may still be cognitively demanding. These checks support two conclusions for the key phrases, concepts, and answer keys that we looked for. First, direct overlap is absent: the distinctive prompts, answers, and lexical sequences were not found in any corpus, ruling out verbatim attestation as an explanation. We did find small matches for substrings (which is both expected and inevitable), and a few matches for a paraphrase of one part of 64 We used the∞-gram next-token distribution, and the document search. We looked at what tokens were predicted for each corpus, what prefix was recovered from the prompt for each corpus, and whether there were any matches between specific strings and any corpus. 65 “In this new language, which is derived from English, speakers follow a word order parity rule. They strictly alternate between OSV (Object–Subject–Verb) and VSO (Verb–Subject–Object) clauses.” 36 one rule. The rule dan-1 is composed of approximately 17 specific instructions; we found a match to one of those paraphrased instructions. We searched for phrases and paraphrases in order to make it easier to find matches, since it is very unlikely we would find the prompts we wrote verbatim. Second, answers are not locally predictable based on corpus statistics alone: for dan-1 and translate-1, the intended answers are not supported as n-gram continuations. By reasonable induction, we think that since the phrases, answers, and concepts we investigated seem absent from the training data, our rules as a whole are probably absent. Even in the unlikely event this is incorrect, we can focus specifically on the rules whose components we investigated and say that subset of rule-following cannot be explained by memorization, which is already sufficient for our argument. However, in future work, we could more exhaustively verify the absence of more features of our rules and their answers. Furthermore, based on the tokenization check, we can see that example correct answers for at least about half of the rules look substantially different from common training data, because they require more tokens-per-response than typical English prose. Tokenizers encode some rule outputs much less efficiently, and since the intended purpose of tokenization is to encode training data as efficiently as possible, this implies that those outputs are not similar to common sequences of training data. Tokenizer fragmentation thus provides a proxy for how far a required output deviates from the statistical structure of the training corpus: rules whose correct answers consist of character sequences that are rare or unattested in English will be split into many small tokens (i.e., fewer characters per token), whereas English-like output will be tokenized into familiar, longer stems. We computed characters-per-token (CPT) and tokens-per-character (TPC) for gold-standard example answers across 13 tokenizers spanning diverse model families, and observed high consistency across the tokenizers (dan-1 has the least consistent tokenization, due to its answer key being mostly digits). See Figure 13, Figure 14, Figure 11, Figure 12, Table 9, Table 10. Consistent with our predictions, we observe in Table 12 that there is a modest correlation between the characters-per-token ratios of our individual example answers and the length of prefix recovered from the eight corpora, and that the thirteen tokenizers behave very consistently in how this metric ranks the conlang rules (Figure 12). These findings are sanity checks for how we are thinking about LMs, tokenization, and the kind of training data these models would have encountered, and validation for the reasoning underlying these checks. These checks (as well as the number of models and runs per rule, the design and diversity of the rules, and the inclusion of prompt variatons) make it implausible that any model we tested could have succeeded by direct recall of training data. In part 1, we use one example correct answer for each rule to roughly gauge how much the output of each conlang rule deviates from the 1D statistical structure of the training data. For everything besides dan-1 and translate-1 and 2, the output is open-ended, meaning specific choices can cause high fluctuation (in the character-per-token ratios via the thirteen tokenizers, as well as in the length of the recovered prefix across the eight corpora via InfiniGram [Liu et al., 2024a]). In part 2, we plan to extend the tokenization analysis from the single representative correct answers used here to include cleaned responses (aggregate and per model), all model output (aggregate and per model), and to use randomly reshuffled and randomly resampled cleaned answers to see how disrupting the linguistic structure at different resolutions impacts the tokenization. We will use this to assess how the rule outputs compare to each other and to typical English, as well as how the models compare to each other. We will further develop the tokenization method used here (mentioned in Zimmerman et al. [2024b]) as an extremely lightweight method for estimating training data exposure in terms of similarity to surface-level statistical patterns seen during training, which is especially useful with closed source models (and can be useful more generally, since they can be run on a laptop and don’t require the ability to store large models or large corpora). When we apply the tokenization analysis to the cleaned model outputs, that will provide additional evidence as to whether or not the models systematically move away from ordinary English statistics when attempting the rules, meaning it is another proxy for rule-following. Figures 11, 17, 16, 12, 13, 14, 15, Tables 12, 9, 10, 11 are intended to sanity-check how we are thinking about these pieces – tokenization, tokenizers, corpus statistics, training data, recovered prefixes – of these systems, to make sure we are using them sensibly in our argument. Figure 10 is included to illustrate interesting anecdotal behaviour. 37 Figure 10: Searching for Google results similar to wordorderalternation-3. Figure 11:Tokenizer fragmentation of example answers by rule (log 2 ratio relative to comparison-1). Rules whose correct outputs most strongly deviate from superficial English statistical patterns (dan-1, wordorderalternation-1, comparison-2) show substantially higher fragmentation ratios, indicating that the required outputs are not well-supported by the orthographic building blocks of any tokenizer’s training distribution. Rules with more English-like correct outputs (sidequest-1, comparison-1) cluster near or below the baseline. All values are log 2 -scaled relative to the comparison- 1 (standard English story) answer. These are “point-in-time” counts for a single example answer per rule; Part 2 will extend this analysis to all model outputs. Speculatively, we wonder if we will observe more prosaic output in sidequest-1 than comparison-1, perhaps as a sign of cognitive tradeoff because sidequest-1 is harder (in other words, the writing may suffer as the models allocate more resources to rule compliance). Figure 12: Spearman rank correlations of rule fragmentation across 13 tokenizers. Correlations are uniformly high (allρ ≥ 0.92; see Table 11), indicating that the ranking of rules by tokenizer fragmentation is robust across diverse model families and vocabulary designs. The consistent ordering of rules by fragmentation is therefore not an artifact of any single tokenizer’s vocabulary, but better explained as a reflection of underlying statistical patterns in the corpora. 38 Table 7: Training-data search counts across corpora. Columns S1–S21 correspond to the following search expressions: S1 = “natural language processing” (for comparison); S2 = “conlang OR constructed language”; S3 = full translate-1 answer; S4 = “Mary gjyqazk-jx OR Mary gjyqazkjx”; S5 = “gjyqazk-jx OR gjyqazkjx”; S6 = full dan-1 answer string; S7 = “354075608095”; S8 = “zero is the word boundary OR zero is the space symbol”; S9 = same concept without quotation marks; S10 = “0 is the word boundary OR 0 is the space symbol”; S11 = variants of “the digit for nine is the verb to be”; S12 = full constrained bigram lexicon; S13 = reduced constrained bigram lexicon; S14 = extended constrained bigram lexicon; S15 = “-qw, -jx, and -kz are always the final morphemes in the verb”; S16 = variants of “-qw, -jx, and -kz”; S17 = “speakers follow a word order parity rule”; S18 = “word order parity rule”; S19 = “include an extra verb in each clause”. For robustness, two additional search strings were tested in OLMo 2 32B Instruct and both returned zero matches (sanity-check permutations of final morphemes; concatenated sanity-check strings (qwkzjx, qwjxkz, kzjxqw, jxkzqw)). Description row abbreviations: Ex = example; HLC = high level concept; T1-A1–T1-A3 = translate-1 (answer); D1-A1–D1-A2 = dan-1 (answer); D1-P = dan-1 (prompt); D1-C1–D1-C2 = dan-1 (concept); D1-PC = dan-1 (prompt, concept); CBL-1–CBL-3 = constrained bigram lexicon; T1+CBL-1–T1+CBL-2 = translate-1 and constrained bigram lexicon; WOA-1–WOA-2 = wordorderalternation (prompt); DV-P = doublev (prompt). DatasetTokensS1S2S3S4S5S6S7S8S9S10S11S12S13S14S15S16S17S18S19 DescriptionExHLCT1-A1T1-A2T1-A3D1-A1D1-A2D1-PD1-C1D1-C2D1-PCCBL-1CBL-2CBL-3T1+CBL-1T1+CBL-2WOA-1WOA-2DV-P OLMo 2 13B Instruct4.6T2,078,696420,91500000004000000000 OLMoE 1B 7B Instruct4.6T2,078,497420,90600000004000000000 DCLM-baseline4.3T1,899,073405,08100000002000000000 Dolma-v1.72.6T915,616100,01600000002000000000 RedPajama1.4T457,31250,00400000001000000000 Pile-train380B81,4619,77300000002000000000 C4-train200B60,8749,22400000000000000000 Pile-val390M761300000000000000000 Figure 13: Raw characters per token by rule, excluding dan-1. Rules requiring more non-English- like output (wordorderalternation-1, wordorderalternation-2, comparison-2) have fewer characters per token, consistent with their correct answers containing more character sequences not found in standard tokenizer vocabularies (thus necessitating more, shorter tokens). Rules whose outputs remain closer to English (sidequest-1, comparison-1) tokenize into longer, more familiar stems. dan-1 is excluded here because its purely numeric and punctuation-based character set produces a qualitatively different fragmentation pattern (it is a tokenization edge case). Figure 14: Raw tokens per character by rule (all rules including dan-1). The inverse of Figure 13; higher values indicate greater tokenizer fragmentation. We included both figures in case some readers find tokens-per-character more intuitive than characters-per-token, but the information is identical, other than the inclusion of dan-1. 39 Figure 15: Mean recovered corpus prefix length by rule. Rules whose correct outputs utilize larger structures borrowed from standard English (wordorderalternation-3, comparison-1) have longer recovered prefixes, indicating that their local character sequences are more frequently attested in training corpora. Rules with highly non-English-like outputs (dan-1, wordorderalternation-1) have shorter recovered prefixes, consistent with the tokenization fragmentation results (Figure 13). Prefix length is computed on a single gold-standard example answer per rule (see Section A.4 for interpretation caveats), and therefore only a “point-in-time” proxy. Figure 16: Scatter plot of mean characters-per-token (CPT) versus mean recovered corpus prefix length, by rule. A moderate positive correlation is observed (Pearsonr = 0.626,p = 0.013; Spearmanρ = 0.515,p = 0.050; see Table 12), consistent with the interpretation that both measures reflect the degree to which a rule’s required output aligns with the statistical structure of the training corpus. Rules that deviate more strongly from English (low CPT/ high TPC) also tend to fall outside corpus n-gram patterns (short recovered prefixes). Figure 17: Rank-rank plot of mean characters-per-token versus mean recovered corpus prefix length, by rule. The positive correlation observed in Figure 16 is preserved at the rank level (Spearman ρ = 0.515, p = 0.050). 40 Table 8: Whether any of the 8 training corpora assigned the highest probability to a token that was part of the correct continuation, for dan-1 and translate-1 prompt variants, via InfiniGram [Liu et al., 2024a]. We report results at two resolution levels in this summary table: (a) any token that forms part of the correct answer string, and (b) a token that matches the first three characters of the correct continuation. Answers were split into meaningful chunks; both hyphenated and unhyphenated correct translate-1 forms were accepted. The prevalence of “No” indicates that correct continuations are not locally predictable from training corpus statistics. The multiple tiers of resolution allow us to move from generous to stricter matching for a richer interpretation of the matches. Details available in Figure 18. Rule Any corpus predicts part of the correct continuation as the most likely token? Most likely token matches the first three characters of the correct continuation? dan-1 354075608095!-34086070[95#-]NoNo dan-1 354075608095!-[340]NoNo dan-1 354075608[095!-]NoNo Condensed dan-1-1 prompt** [354075608095!-3408607095#-]NoNo Condenseddan-1-1prompt***<0x0A>[354075608095!- 3408607095#-] NoNo Condenseddan-1-1prompt*** [354075608095!- 3408607095#-] YesNo Condensed dan-1-1 prompt*** 3 [54075608095!-3408607095#-]NoNo translate-1 Mary gjyqazk-jx px-mqi-Mary. Px-mqi-Mary [qto-jx.]NoNo translate-1 Mary gjyqazk-jx px-mqi-[Mary. Px-mqi-Mary qto-jx.]NoNo translate-1-2 prompt**** Mary gjyqazkjx pxmqiMary. Pxmqi- Mary [qtojx. OR qto-jx.] NoNo translate-1-2 prompt**** gjyqazkjx pxmqiMary. Pxmqi- Mary [qtojx. OR qto-jx.] (formatted newline) NoNo translate-1-2 prompt**** Mary gjyqazkjx pxmqi[Mary. Pxmqi- Mary qtojx. OR Mary. Px-mqi-Mary qto-jx.] NoNo The condensed prompts in Figure 18 are: •** Condensed dan-1-1 prompt (shortened to fit InfiniGram’s 1000-character limit): “The digit, for ‘zero’ is the word boundary or space symbol. Unless otherwise indicated, assume similarity to English. The digit for ‘nine’ is the verb ‘to be’. ‘Eight’ is ‘cat’. The digit for ‘three’ is any English pronoun. ‘Three’ is male/ gender neutral. The digit for ‘five’ is appended to indicate female gender, and the digit for ‘four’ indicates possession. The digit for ‘six’ means possession for non-pronoun words. The digit ‘seven’ is ‘male parent’; the digit ‘seven’ suffixed with the digit ‘five’ is ‘female parent’. Present tense is default, and past tense is marked with five. The exclamation mark means positive emotional valence, and attaches to the verb stem. The hashtag symbol indicates negative emotional valence. Translate the sentences: ‘Her mother’s cat was happy. His cat’s father was sad.”’ •*** Further condensed (“Remember to” removed) to satisfy the 1000-character limit; tested with both<0x0A>and newline representations. The<0x0A>byte could be part of rendering a correct character in some encoding, but we are not exploring that possibility here. Cursory research suggests it is likely to be newline or potentially 10, neither of which would be part of the correct continuation for dan-1. To be as generous as possible, we incorporated that into the prompt variations in case correct continuations follow a newline. •**** translate-1-2 prompt:“Your job is to translate text in a new language according to rigid grammatical rules. Language A: no pronouns, SOV word order, verbs take tense suffix -qw=past, -jx=present, -kz=future. Language B: pronouns vf="he", px="she", zd="they", SVO word order, familial relationships = relative pronoun + temporal metaphor for relation- ship + person (e.g., John’s grandfather = vf-before-before-John = vfmqimqiJohn), verbs take tense suffix -jx=past, -kz=present, -qw=future. Example in language A: Mary Marypq-gvijw gjyqazkqw. Marypq-gvijw qtoqw. (Mary speak-past to Mary-possessive-mother. Mary- mother smile-past.) The roots of the verbs in this example are the same in language A and language B. Task = Translate this example into Language B, preserving meaning and applying all of B’s rules:” 41 Table 9: Tokenizer fragmentation of example answers relative to the standard-English comparison-1 answer. “Prop.” is the proportion of tokenizers for which the rule’s answer is more fragmented than comparison-1; “All” indicates whether this holds for every tokenizer tested (13 tokenizers). RuleMeanMin.Max.Prop.All dan-12.9371.9814.3981.000Yes wordorderalternation-12.7762.3323.1961.000Yes comparison-22.6232.0232.8461.000Yes translate-1-hyphen2.3912.0862.7691.000Yes translate-1-nohyphen2.2301.8982.4791.000Yes testingenglish-12.1351.7912.4061.000Yes wordorderalternation-21.8171.5552.0201.000Yes testingenglish-21.6001.4741.7021.000Yes testingenglish-31.2371.0951.2911.000Yes testingenglish-41.1931.0991.2721.000Yes doublev-11.1271.0551.1901.000Yes doublev-21.1201.0511.1841.000Yes wordorderalternation-31.0390.9701.0660.923No comparison-11.0001.0001.0000.000No sidequest-10.9200.9090.9410.000No Table 10: Mean and variability of raw tokens-per-character (TPC) values across 13 tokenizers, for gold-standard example answers (answer keys, or real model responses manually modified to be facially valid). Lower TPC indicates more English-like output (tokenized into longer stems); higher TPC indicates greater fragmentation. Rules with highly novel output tend to show greater variability across tokenizers, reflecting differences in how each vocabulary handles unusual character sequences (especially dan-1, with its largely numeric answer key). rulemean_tpcsd_tpcmin_tpcmax_tpcrange_tpc dan-10.7010000.2490000.4620001.0380000.577000 wordorderalternation-10.6570000.0590000.5610000.7370000.176000 comparison-20.6200000.0430000.5330000.6630000.130000 translate-1-hyphen0.5660000.0640000.4890000.6380000.149000 translate-1-nohyphen0.5270000.0360000.4760000.5710000.095000 testingenglish-10.5050000.0330000.4700000.5680000.098000 wordorderalternation-20.4300000.0330000.3650000.4660000.101000 testingenglish-20.3790000.0260000.3400000.4170000.077000 testingenglish-30.2930000.0100000.2810000.3140000.033000 testingenglish-40.2830000.0170000.2610000.3160000.054000 doublev-10.2670000.0160000.2430000.2980000.055000 doublev-20.2660000.0160000.2420000.2930000.051000 wordorderalternation-30.2460000.0090000.2400000.2700000.030000 comparison-10.2370000.0130000.2310000.2700000.039000 sidequest-10.2180000.0130000.2100000.2480000.038000 42 Table 11: Spearman rank correlations of rule fragmentation across 13 tokenizers. All correlations are≥ 0.92, indicating that the relative ordering of rules by tokenizer fragmentation is consistent across model families. This mitigates the possibility that any specific tokenizer idiosyncrasy or even underlying training data idiosyncrasy is responsible for the fragmentation patterns observed. modelBLOOMDeepSeek-LLMFalconGPT-2GPT-Neo-JGPT-NeoXGranite-3.3Mistral-Small-3.1OLMo-2OPTPhi-4Qwen3Yi-1.5 model BLOOM1.0000000.9570000.9640000.9750000.9750000.9750000.9710000.9610000.9750000.9750000.9750000.9860000.975000 DeepSeek-LLM0.9570001.0000000.9390000.9540000.9540000.9710000.9890000.9610000.9290000.9540000.9290000.9820000.986000 Falcon0.9640000.9390001.0000000.9930000.9930000.9890000.9360000.9250000.9930000.9930000.9930000.9390000.939000 GPT-20.9750000.9540000.9930001.0000001.0000000.9930000.9570000.9430000.9930001.0000000.9930000.9610000.957000 GPT-Neo-J0.9750000.9540000.9930001.0000001.0000000.9930000.9570000.9430000.9930001.0000000.9930000.9610000.957000 GPT-NeoX0.9750000.9710000.9890000.9930000.9930001.0000000.9680000.9570000.9790000.9930000.9790000.9680000.971000 Granite-3.30.9710000.9890000.9360000.9570000.9570000.9680001.0000000.9680000.9360000.9570000.9360000.9890000.989000 Mistral-Small-3.10.9610000.9610000.9250000.9430000.9430000.9570000.9680001.0000000.9210000.9430000.9210000.9750000.986000 OLMo-20.9750000.9290000.9930000.9930000.9930000.9790000.9360000.9210001.0000000.9930001.0000000.9460000.936000 OPT0.9750000.9540000.9930001.0000001.0000000.9930000.9570000.9430000.9930001.0000000.9930000.9610000.957000 Phi-40.9750000.9290000.9930000.9930000.9930000.9790000.9360000.9210001.0000000.9930001.0000000.9460000.936000 Qwen30.9860000.9820000.9390000.9610000.9610000.9680000.9890000.9750000.9460000.9610000.9460001.0000000.989000 Yi-1.50.9750000.9860000.9390000.9570000.9570000.9710000.9890000.9860000.9360000.9570000.9360000.9890001.000000 Table 12: Correlation between mean characters-per-token (CPT) and mean recovered corpus prefix length. N rules encompasses the suite of rules, but including both hyphenated and non-hyphenated translate-1 answers, and excluding translate-2 because its answer key is identical to that of translate-1). Both correlations are statistically significant, supporting the interpretation that CPT and prefix length at least partially reflect the same signal. correlationcoefficientp_valuen_rules Pearson0.6260000.01260015 Spearman0.5148000.04960015 Table 13: Example documents returned by searches for superficially similar phrases. The retrieved examples concern an embroidery-pattern convention in which zero or a blank space is used for padding, and a forum question about mailto links, rather than a constructed-language rule. The matched search string is a paraphrase of that part of the rule, not a verbatim match to any part of the prompt. Document sources: http://embroidermodder.org/docs/manual/, https://stackoverflow.com/ questions/3868315/invoke-click-a-mailto-link-with-jquery-javascript, retrieved from the InfiniGram UI. There are at most 4 matches per corpus to this n-gram search string, across all the corpora, and every example surfaced by the InfiniGram UI matched one of these two use cases. Search expressionRetrieved excerpt 0 is the word boundary OR 0 is the space sym- bol “. . . use a vertical flip (embPattern_flip). . . 0x2 0 is the space symbol 0, so when padding either 0 or space is preferred . . . ” 0 is the word boundary OR 0 is the space sym- bol “. . . i.e. use only the C Standard Library. . . 0x2 0 is the space symbol 0, so when padding either 0 or space is preferred . . . ” 0 is the word boundary OR 0 is the space sym- bol “. . . theworkinganswerforme,testedin chrome,IEandfirefox. . .window.location.href = ’mailto:address@dmail.com?subject=Hello there&body=This is the body’; . . . %2 0 is the space symbol 0 that should be used, but it worked for me as well with normal space.” Figure 18: To what extent is any part of the correct continuation among the top predicted tokens? For each context, we list all top-probability tokens for each corpus in descending probability order, and compare against the correct continuation. Correct answers were split into meaningful chunks (allowing for minor ordering variation) and both hyphenated and unhyphenated translate-1 forms were accepted. We split continuations to give greater context or to allow greater freedom in acceptable responses (making positive matches easier). We counted the hyphenated or unhyphenated correct answers for translate-1 as equally valid/ correct (we were flexible on punctuation and casing). 43 A.5 All conlang rules The text for each of the rules (1 of the 3 prompt variations) are provided in Table 15. We did not include sidequest-1-fix, which is identical to sidequest-1, but with 3 prompt variations instead of 2. Table 15: Columns taken from our prompt spreadsheet; prompt variations not included. Family/ GlossRuleBase Prompt doublev-1doublev1/2Your goal is to generate text in a new language with strict grammatical rules. In this new version of English, speakers include an extra verb in each clause. In other words, everywhere there would normally be a single verb, there are now two consecutive verbs. One of the verbs is irrelevant to the meaning of the sentence. The irrelevant verb could either be first or second, and it is marked with the terminal morpheme -qw. The irrelevant verb should result in no other changes to the sentence. Otherwise, this language is identical to English. Please write a short story (20 sentences) in this language as if you speak this language: doublev-2doublev (emphasized irrelevance) 2/2Your goal is to generate text in a new language with strict grammatical rules. In this new version of English, speakers include an extra verb in each clause. In other words, everywhere there would normally be a single verb, there are now two consecutive verbs. One of the verbs is irrelevant to the meaning of the sentence. It doesn’t just fail to add to the meaning of the sentence, but it ought to feel “random” in the context of the sentence. In other words, the two verbs should not be closely related, synonymous, or antonymous. The irrelevant verb could either be first or second, and it is always marked with the terminal morpheme -qw. The irrelevant verb should result in no other changes to the sentence. Otherwise, this language is identical to English. Please write a short story (20 sentences) in this language as if you speak this language: Continued on next page 44 Family/ GlossRuleBase Prompt wordorderalternation-1wordorderalternation (restricted lexicon) 1/3Your goal is to generate text in a new language with strict grammatical rules. In this new language, speakers follow a word order parity rule. They strictly alternate between OSV (Object–Subject–Verb) and VSO (Verb–Subject–Object) clauses. When a clause is left incomplete, e.g. due to interruption, the new clause flips to the other word order. Verbs are inflected for tense with suffixes: -qw indicates past tense, -jx indicates present tense, and -kz indicates future tense. All verbs must be inflected with one of these suffixes. The tense suffixes -qw, -jx, and -kz are always the final morphemes in the verb. Those morphemes only have one meaning (indicating verb tense); they are not polysemous and have no homonyms. Adverbs always appear in phrases before the verb when word order is OSV, and after the verb when word order is VSO. In other words, for OSV clauses, the verb is strictly final, and for VSO clauses, the verb is strictly first. Besides the tense suffixes, the rest of the lexicon is created from this set of graphemes, optionally joined with each other: VF, PX, ZD, QT, ZP, QO, DX, HJ, GV, JW, QC, JY, GJ, QB, PQ, JG, BZ, MX, QM, MZ, QF, WJ, ZQ, XR, ZK, CX, FX, FV, BX, VW, VJ, MQ, QV, ZF, QE, YJ, GX, KX, XG, QD, XJ, SX, VZ, VX, WV, YQ, BQ, GQ, VK, ZJ, XK, QP, HX, FZ, QH, QJ, JZ, VQ, KQ, XD, QX, WX, FQ, XZ, ZX, A, E, I, O, U. Please write a short story (20 sentences) in this language as if you speak this language: Continued on next page 45 Family/ GlossRuleBase Prompt wordorderalternation-2wordorderalternation (any lexicon, with verb marking) 2/3Your goal is to generate text in a new language with strict grammatical rules. In this new language, speakers follow a word order parity rule. They strictly alternate between OSV (Object–Subject–Verb) and VSO (Verb–Subject–Object) clauses. When a clause is left incomplete, e.g. due to interruption, the new clause flips to the other word order. Verbs are inflected for tense with suffixes: -qw indicates past tense, -jx indicates present tense, and -kz indicates future tense. All verbs must be inflected with one of these suffixes. The tense suffixes -qw, -jx, and -kz are always the final morphemes in the verb. Those morphemes only have one meaning (indicating verb tense); they are not polysemous and have no homonyms. Adverbs always appear in phrases before the verb when word order is OSV, and after the verb when word order is VSO. In other words, for OSV clauses, the verb is strictly final, and for VSO clauses, the verb is strictly first. Please write a short story (20 sentences) in this language as if you speak this language: wordorderalternation-3wordorderalternation (English lexicon) 3/3Your goal is to generate text in a new language with strict grammatical rules. In this new language, which is derived from English, speakers follow a word order parity rule. They strictly alternate between OSV (Object–Subject–Verb) and VSO (Verb–Subject–Object) clauses. When a clause is left incomplete, e.g. due to interruption, the new clause flips to the other word order. Adverbs always appear in phrases before the verb when word order is OSV, and after the verb when word order is VSO. In other words, for OSV clauses, the verb is strictly final, and for VSO clauses, the verb is strictly first. Besides these differences in syntax, the language is identical to present day English. Please write a short story (20 sentences) in this language as if you speak this language: Continued on next page 46 Family/ GlossRuleBase Prompt testingenglish-1testingenglish (replace content words) 1/4Your goal is to generate text in a new language with strict grammatical rules. In this new language, speakers follow similar rules to English. The function words (prepositions, articles, pronouns, auxiliary verbs, etc.) and inflectional morphemes (verb tense, plural markers, adverb and adjective affixes, etc.) are all from English. However, the content words — nouns, adjectives, adverbs, and main verbs — come from this lexicon: VF, PX, ZD, QT, ZP, QO, DX, HJ, GV, JW, QC, JY, GJ, QB, PQ, JG, BZ, MX, QM, MZ, QF, WJ, ZQ, XR, ZK, CX, FX, FV, BX, VW, VJ, MQ, QV, ZF, QE, YJ, GX, KX, XG, QD, XJ, SX, VZ, VX, WV, YQ, BQ, GQ, VK, ZJ, XK, QP, HX, FZ, QH, QJ, JZ, VQ, KQ, XD, QX, WX, FQ, XZ, ZX. English auxiliary verbs (e.g., be, have, do, will, can) remain in the language; all English main verbs are replaced. Besides the novel lexical items that replace English content words, the language is identical to English. Please write a short story (20 sentences) in this language as if you speak this language: Continued on next page 47 Family/ GlossRuleBase Prompt testingenglish-2testingenglish (replace function words) 2/4Your goal is to generate text in a new language with strict grammatical rules. In this new language, speakers follow similar rules to English. The content words — nouns, adjectives, adverbs, and main verbs — are all from English. However, the function words (prepositions, articles, pronouns, auxiliary verbs, etc.) and inflectional morphemes (verb tense, plural markers, adverb and adjective affixes, etc.) come from this lexicon: VF, PX, ZD, QT, ZP, QO, DX, HJ, GV, JW, QC, JY, GJ, QB, PQ, JG, BZ, MX, QM, MZ, QF, WJ, ZQ, XR, ZK, CX, FX, FV, BX, VW, VJ, MQ, QV, ZF, QE, YJ, GX, KX, XG, QD, XJ, SX, VZ, VX, WV, YQ, BQ, GQ, VK, ZJ, XK, QP, HX, FZ, QH, QJ, JZ, VQ, KQ, XD, QX, WX, FQ, XZ, ZX. While all English main verbs remain in the language, English auxiliary verbs (e.g., be, have, do, will, can) are replaced. Besides the novel lexical items that replace English function words, the language is identical to English. Please write a short story (20 sentences) in this language as if you speak this language: Continued on next page 48 Family/ GlossRuleBase Prompt testingenglish-3testingenglish (replace verbs) 3/4Your goal is to generate text in a new language with strict grammatical rules. In this new language, speakers follow similar rules to English. The function words (prepositions, articles, pronouns, auxiliary verbs, etc.) and inflectional morphemes (verb tense, plural markers, adverb and adjective affixes, etc.) are all from English, as are most categories of content words, specifically the nouns, adjectives, and adverbs. However, the main verbs come from this lexicon: VF, PX, ZD, QT, ZP, QO, DX, HJ, GV, JW, QC, JY, GJ, QB, PQ, JG, BZ, MX, QM, MZ, QF, WJ, ZQ, XR, ZK, CX, FX, FV, BX, VW, VJ, MQ, QV, ZF, QE, YJ, GX, KX, XG, QD, XJ, SX, VZ, VX, WV, YQ, BQ, GQ, VK, ZJ, XK, QP, HX, FZ, QH, QJ, JZ, VQ, KQ, XD, QX, WX, FQ, XZ, ZX. In other words, no English main verbs are used in this language, but English auxiliary verbs are. Besides the novel lexical items that replace the English main verbs, the language is identical to English. Please write a short story (20 sentences) in this language as if you speak this language: Continued on next page 49 Family/ GlossRuleBase Prompt testingenglish-4testingenglish (replace nouns) 4/4Your goal is to generate text in a new language with strict grammatical rules. In this new language, speakers follow similar rules to English. The function words (prepositions, articles, pronouns, auxiliary verbs, etc.) and inflectional morphemes (verb tense, plural markers, adverb and adjective affixes, etc.) are all from English, as are most categories of content words, specifically the main verbs, adjectives, and adverbs. However, the nouns come from this lexicon: VF, PX, ZD, QT, ZP, QO, DX, HJ, GV, JW, QC, JY, GJ, QB, PQ, JG, BZ, MX, QM, MZ, QF, WJ, ZQ, XR, ZK, CX, FX, FV, BX, VW, VJ, MQ, QV, ZF, QE, YJ, GX, KX, XG, QD, XJ, SX, VZ, VX, WV, YQ, BQ, GQ, VK, ZJ, XK, QP, HX, FZ, QH, QJ, JZ, VQ, KQ, XD, QX, WX, FQ, XZ, ZX. In other words, no English nouns are used in this language, but otherwise, the language is identical to English. Please write a short story (20 sentences) in this language as if you speak this language: Continued on next page 50 Family/ GlossRuleBase Prompt translate-1translate1/1Your goal is to translate text in a new language with strict grammatical rules. Consider these two new languages, language A and language B: In language A: there are no pronouns. The word order is SOV (Subject-Object-Verb). Verbs are inflected for tense with suffixes: -qw indicates past tense, -jx indicates present tense, and -kz indicates future tense. All verbs must be inflected with one of these suffixes. The tense suffixes -qw, -jx, and -kz are always the final morphemes in the verb. In language B, there are pronouns: vf means "he", px means "she", zd means "they". Word order is SVO (Subject-Verb-Object). Familial relationships are indicated with pronoun + genetic relationship as a temporal metaphor + relevant person, so, e.g. “John’s grandfather” would be he-before-before-John, or vf-mqi-mqi-John = vfmqimqiJohn. Verbs are inflected for tense with suffixes: -jx indicates past tense, -kz indicates present tense, and -qw indicates future tense. An example in language A: Mary Marypq-gvijw gjyqazkqw. Marypq-gvijw qtoqw. (Mary speak-past to Mary-possessive-mother. Mary-mother smile-past.) The roots of the verbs used in this example happen to be the same in language A and language B. Please translate the example from language A into language B: comparison-1 comparison (English)1/2Please write a short story (20 sentences). The story can be in any genre. For example, you could continue "This story began when..." Continued on next page 51 Family/ GlossRuleBase Prompt comparison-2comparison (Conlang) 2/2Your goal is to generate text in a new language with strict grammatical rules. In this new language, words are created from this set of graphemes, optionally joined with each other: VF, PX, ZD, QT, ZP, QO, DX, HJ, GV, JW, QC, JY, GJ, QB, PQ, JG, BZ, MX, QM, MZ, QF, WJ, ZQ, XR, ZK, CX, FX, FV, BX, VW, VJ, MQ, QV, ZF, QE, YJ, GX, KX, XG, QD, XJ, SX, VZ, VX, WV, YQ, BQ, GQ, VK, ZJ, XK, QP, HX, FZ, QH, QJ, JZ, VQ, KQ, XD, QX, WX, FQ, XZ, ZX, A, E, I, O, U. Please write a short story (20 sentences) in this language as if you speak this language: sidequest-1sidequest (bigrams)1/1It would be possible to generate text in a new language with strict grammatical rules. In this new language, words are created from this set of graphemes, optionally joined with each other: VF, PX, ZD, QT, ZP, QO, DX, HJ, GV, JW, QC, JY, GJ, QB, PQ, JG, BZ, MX, QM, MZ, QF, WJ, ZQ, XR, ZK, CX, FX, FV, BX, VW, VJ, MQ, QV, ZF, QE, YJ, GX, KX, XG, QD, XJ, SX, VZ, VX, WV, YQ, BQ, GQ, VK, ZJ, XK, QP, HX, FZ, QH, QJ, JZ, VQ, KQ, XD, QX, WX, FQ, XZ, ZX, A, E, I, O, U. However, please write a short story (20 sentences) in English: Continued on next page 52 Family/ GlossRuleBase Prompt dan-1numbers1/1In this constructed language, the arabic numeral, or digit, for “zero” is the word boundary or space symbol. Unless otherwise indicated, assume word boundaries and morphology parallel to English. The digit for “nine” is the verb “to be”. The digit “eight” is “cat”. The digit for “three” is equivalent to any English pronoun. The bare “three” is male gendered or gender neutral. The digit for “five” is appended to indicate female gender, and the digit for “four” is suffixed to indicate possession. The digit for “six” indicates possession for words besides pronouns. The digit “seven” is “male parent”; the digit “seven” suffixed with the digit “five” is “female parent”. This language has limited morphology and a small lexicon: present tense is the default, and past tense is marked with the female gender marker (five). The exclamation mark is equivalent to positive emotional valence, and attaches to the verb stem directly. The hashtag symbol works the same way to indicate negative emotional valence. Please translate the sentences: “Her mother’s cat was happy. His cat’s father was sad.” The language uses only the dash as punctuation (at the end of a sentence; no space afterwards). Remember to use digits, not words, for the numbers, and to replace spaces with zeros. Continued on next page 53 Family/ GlossRuleBase Prompt translate-2example2/2Your goal is to translate text in a new language with strict grammatical rules. Consider these two new languages, language A and language B: In language A: there are no pronouns. The word order is SOV (Subject-Object-Verb). Verbs are inflected for tense with suffixes: -qw indicates past tense, -jx indicates present tense, and -kz indicates future tense. All verbs must be inflected with one of these suffixes. The tense suffixes -qw, -jx, and -kz are always the final morphemes in the verb. In language B, there are pronouns: vf means "he", px means "she", zd means "they". Word order is SVO (Subject-Verb-Object). Familial relationships are indicated with pronoun + genetic relationship as a temporal metaphor + relevant person, so, e.g. “John’s grandfather” would be he-before-before-John, or vf-mqi-mqi-John = vfmqimqiJohn. Verbs are inflected for tense with suffixes: -jx indicates past tense, -kz indicates present tense, and -qw indicates future tense. The roots of the verbs used in the examples happen to be the same in language A and language B. An example translation from language A to B: Language A: Tompq-vfwj Tom vqyqkz. (Tom-possessive-great-grandfather Tom wave-future.) Language B: VfmqimqimqiTom vqyqqw Tom. (Tom’s great-grandfather will wave to Tom; vf-mqi-mqi-mqi-Tom is “he before before before Tom”, or Tom’s great-grandfather, and vqyq-qw is wave-future.) An example in language A: Mary Marypq-gvijw gjyqazkqw. Marypq-gvijw qtoqw. (Mary speak-past to Mary-possessive-mother. Mary-mother smile-past.) Please translate the example from language A into language B: A.6 Model architectures Table 16 lists relevant features of model architecture, to the best of our knowledge. 54 sidequest-1comparison-1comparison-2 Condition 0.0 0.2 0.4 0.6 0.8 Rare Bigram Ratio Distribution of Rare Bigram Usage Figure 19: Comparison of rare bigram usage across conditions (for 5 models; Qwen (thinking), Qwen (no thinking), GPT-4o, OLMo, Llama), using all model output (not cleaned). 0.00.20.40.60.81.0 score 0 250 500 750 1000 1250 1500 1750 2000 n=4,726 Cosine Similarity (qw_is_verb=True) by model mean=0.897 median=0.904 01020304050607080 count 0 10 20 30 40 50 n=195 qw_is_verb_count by run mean=24.2 median=22.0 020406080100120140160 count 0 10 20 30 40 50 n=195 qw_with_closest_verb + qw_is_verb_count by run mean=52.5 median=46.0 Doublev-1 Distributions (All Models Combined) Figure 20: Histogram of Similarity Scores for Doublev-1, all Models (combined). Histogram of cosine similarity between correctly marked -qw verb and adjacent verb for all models, as well as the combined histogram of counts of correctly marked -qw verbs by run by model and the sum of counts of correctly marked -qw verbs and appropriately placed adjacent verbs. A.7 Story rules results We compare rare bigram usage across conditions using Paired Wilcoxon tests in Table 20, Table 21. Values are computed from paired per-run observations. Note that the pairing across run order shouldn’t matter as each run for each model is independent, but the models are somewhat “black-box”-like, and we intentionally kept track of the order of the runs (in case it turns out to matter). We computed both versions of Wilcoxon (Paired and Unpaired) with similar takeaways. We did not correct for multiple tests for the Paired Wilcoxon results, since they are similar in spirit to the Mann-Whitney U results, which we do correct, but we can add that in future work if needed. Comparisons across the story rules for both clean and uncleaned output in Table 18, Table 4, Table 17, Table 19, Table 5, Table 3. A.8 Double verb rules results Additional results for the double verb family of rules given in Table 28, Table 27, Table 26, Table 26, Table 25, Table 25. See also Section 3.2. A.9 Translation rules results Example outputs in Table 29, Table 30. Figures 25, 24 show string distances for the models combined. Tables 31, 32, 33, 34 show Wilcoxon Rank-Sum and Kolmogorov-Smirnov tests for the string cosine distances across models for the translation rules. See Section 3.3. 55 0.00.20.40.60.81.0 score 0 100 200 300 400 500 n=4,134 Cosine Similarity (qw_is_verb=True) by model mean=0.280 median=0.197 010203040506070 count 0 10 20 30 40 50 60 n=207 qw_is_verb_count by run mean=20.0 median=20.0 020406080100120140 count 0 10 20 30 40 50 60 70 n=207 qw_with_closest_verb + qw_is_verb_count by run mean=45.1 median=40.0 Doublev-2 Distributions (All Models Combined) Figure 21: Histogram of Similarity Scores for Doublev-2, all Models (combined). Histogram of cosine similarity between correctly marked -qw verb and adjacent verb for all models, as well as the combined histogram of counts of correctly marked -qw verbs by run by model and the sum of counts of correctly marked -qw verbs and appropriately placed adjacent verbs. 020406080100 Score 0 5 10 15 20 25 GPT 4o Frequency Mean 8.54 Perfect 100.00 BLEU Distribution 20406080100 Score 0 2 4 6 8 10 12 14 Frequency Mean 48.32 Perfect 100.00 chrF Distribution 050100150200250300350 Score 0 10 20 30 40 Frequency Mean 99.60 Perfect 0.00 TER Distribution 50250255075100 Score 0 5 10 15 20 25 30 Frequency Mean 13.47 Perfect 99.60 METEOR Distribution 020406080100 Score 0.0 2.5 5.0 7.5 10.0 12.5 15.0 17.5 20.0 Frequency Mean 19.48 Perfect 99.90 composite Distribution 020406080100 Score 0 5 10 15 20 25 Llama 3 Frequency Mean 6.22 Perfect 100.00 020406080100 Score 0.0 2.5 5.0 7.5 10.0 12.5 15.0 17.5 Frequency Mean 24.00 Perfect 100.00 0100200300400 Score 0 10 20 30 40 50 Frequency Mean 115.20 Perfect 0.00 8006004002000 Score 0 10 20 30 40 50 Frequency Mean -54.57 Perfect 99.60 20015010050050100 Score 0 10 20 30 40 Frequency Mean -4.19 Perfect 99.90 020406080100 Score 0.0 2.5 5.0 7.5 10.0 12.5 15.0 17.5 OLMo Frequency Mean 4.70 Perfect 100.00 20406080100 Score 0 2 4 6 8 10 12 14 16 Frequency Mean 27.25 Perfect 100.00 050100150200250 Score 0 2 4 6 8 10 12 14 16 Frequency Mean 155.20 Perfect 0.00 20015010050050100 Score 0 10 20 30 40 50 60 Frequency Mean -4.33 Perfect 99.60 4020020406080100 Score 0 5 10 15 20 25 30 35 40 Frequency Mean 6.90 Perfect 99.90 020406080100 Score 0 5 10 15 20 25 30 35 40 Qwen Frequency Mean 17.52 Perfect 100.00 020406080100 Score 0.0 2.5 5.0 7.5 10.0 12.5 15.0 17.5 20.0 Frequency Mean 59.75 Perfect 100.00 0255075100125150 Score 0 5 10 15 20 25 30 Frequency Mean 75.60 Perfect 0.00 20015010050050100 Score 0.0 2.5 5.0 7.5 10.0 12.5 15.0 17.5 20.0 Frequency Mean -13.80 Perfect 99.60 4020020406080100 Score 0 5 10 15 20 Frequency Mean 22.27 Perfect 99.90 020406080100 Score 0 5 10 15 20 25 30 Qwen No Thinking Frequency Mean 13.02 Perfect 100.00 20406080100 Score 0 2 4 6 8 10 12 14 Frequency Mean 49.45 Perfect 100.00 0255075100125150175 Score 0 5 10 15 20 25 30 Frequency Mean 84.80 Perfect 0.00 20015010050050100 Score 0 5 10 15 20 Frequency Mean -5.48 Perfect 99.60 4020020406080100 Score 0 5 10 15 20 25 Frequency Mean 18.75 Perfect 99.90 Figure 22: Sentence-level translation quality across five language models (GPT-4o, Llama 3, OLMo, Qwen, and Qwen without thinking) evaluated using BLEU, chrF, TER, METEOR, and a composite score. Each histogram shows the distribution of metric scores over 50 full (non- truncated) LLM-generated responses to the translation prompt, whose source sentence is five words (41 characters) long. The orange triangle denotes the model mean, and the dotted vertical line denotes the upper-bound score obtained when the hypothesis exactly matches the reference (BLEU = 100.0, chrF = 100.0, TER = 0.0, METEOR = 99.6, composite = 99.9). Higher values indicate better performance for BLEU, chrF, METEOR, and the composite score, whereas lower values indicate better performance for TER. The composite score is computed as the unweighted mean of BLEU, chrF, METEOR, and (100− TER). 56 20406080100 Score 0 5 10 15 20 25 30 35 40 GPT 4o Frequency Mean 9.95 Perfect 100.00 BLEU Distribution 20406080100 Score 0.0 2.5 5.0 7.5 10.0 12.5 15.0 17.5 20.0 Frequency Mean 39.81 Perfect 100.00 chrF Distribution 020406080100 Score 0 5 10 15 20 25 30 35 40 Frequency Mean 90.00 Perfect 0.00 TER Distribution 020406080100 Score 0 5 10 15 20 25 30 Frequency Mean 10.24 Perfect 99.60 METEOR Distribution 20406080100 Score 0.0 2.5 5.0 7.5 10.0 12.5 15.0 17.5 20.0 Frequency Mean 17.50 Perfect 99.90 composite Distribution 020406080100 Score 0.0 2.5 5.0 7.5 10.0 12.5 15.0 17.5 Llama 3 Frequency Mean 7.35 Perfect 100.00 020406080100 Score 0.0 2.5 5.0 7.5 10.0 12.5 15.0 17.5 Frequency Mean 21.58 Perfect 100.00 0255075100125150175 Score 0 5 10 15 20 25 30 35 Frequency Mean 92.40 Perfect 0.00 25020015010050050100 Score 0 10 20 30 40 50 Frequency Mean -11.21 Perfect 99.60 50250255075100 Score 0 5 10 15 20 Frequency Mean 7.13 Perfect 99.90 020406080100 Score 0 2 4 6 8 10 12 14 16 OLMo Frequency Mean 5.73 Perfect 100.00 20406080100 Score 0 2 4 6 8 10 12 14 Frequency Mean 23.20 Perfect 100.00 0255075100125150 Score 0 5 10 15 20 25 30 Frequency Mean 118.40 Perfect 0.00 50250255075100 Score 0 5 10 15 20 25 30 35 40 Frequency Mean 1.03 Perfect 99.60 020406080100 Score 0 5 10 15 20 Frequency Mean 7.49 Perfect 99.90 020406080100 Score 0 5 10 15 20 25 30 35 40 Qwen Frequency Mean 18.40 Perfect 100.00 020406080100 Score 0.0 2.5 5.0 7.5 10.0 12.5 15.0 17.5 20.0 Frequency Mean 56.28 Perfect 100.00 0255075100125150 Score 0 5 10 15 20 25 30 Frequency Mean 75.60 Perfect 0.00 20015010050050100 Score 0.0 2.5 5.0 7.5 10.0 12.5 15.0 17.5 Frequency Mean -44.02 Perfect 99.60 4020020406080100 Score 0 2 4 6 8 10 12 14 16 Frequency Mean 14.06 Perfect 99.90 020406080100 Score 0 5 10 15 20 25 30 Qwen No Thinking Frequency Mean 13.55 Perfect 100.00 20406080100 Score 0 2 4 6 8 10 12 14 Frequency Mean 46.82 Perfect 100.00 020406080100120140 Score 0 5 10 15 20 25 30 Frequency Mean 84.00 Perfect 0.00 25020015010050050100 Score 0 5 10 15 20 25 30 Frequency Mean -18.78 Perfect 99.60 50250255075100 Score 0.0 2.5 5.0 7.5 10.0 12.5 15.0 17.5 20.0 Frequency Mean 14.80 Perfect 99.90 Figure 23: Sentence-level translation quality across five language models (GPT-4o, Llama 3, OLMo, Qwen, and Qwen without thinking) evaluated using BLEU, chrF, TER, METEOR, and a composite score. Each histogram shows the distribution of metric scores over 50 LLM-generated responses to the translation prompt, with each response truncated to its first 41 characters to match the length of the source sentence ("Mary gjyqazkjx pxmqiMary. pxmqiMary qtojx"). The orange triangle denotes the model mean, and the dotted vertical line denotes the upper-bound score obtained when the hypothesis exactly matches the reference (BLEU = 100.0, chrF = 100.0, TER = 0.0, METEOR = 99.6, composite = 99.9). Higher values indicate better performance for BLEU, chrF, METEOR, and the composite score, whereas lower values indicate better performance for TER. The composite score is computed as the unweighted mean of BLEU, chrF, METEOR, and (100− TER). Figure 24: Histograms of Similarity Scores for Dan-1, all Models (combined). Figure 25: Histograms of Similarity Scores for Translate-1, all Models (combined). 57 Figure 26: Exploratory look at model performance and features of the conlang rules. A.10 Exploratory performance results Because of the small number of models and the difficulty in comparing across different metrics for the different rules, these results are exploratory only. They are intended to sketch out a relationship we think could exist between these components, but are more like back-of-the- envelope calculations. We think the gist is potentially meaningful, but the exact numbers we wouldn’t put too much stock in, for the reasons aforementioned. Re Table 37: The features “uses no typical english” and “lexical restriction” are indistinguishable in this subset of rules, but not in the complete set of rules. There are 2 positive coefficients here, for the features of whether rare bigrams are in the prompt and required in the output. We interpret this as being because those features do occur in the relatively simple comparison-2 – models have to output rare bigrams, but there are no other constraints, whereas e.g. dan-1 has about 17 specific constraints – where all the models perform pretty well, so with the other features in that rule, they appear to be helping in that one context (in other words, this is likely an artifact of the small set and other factors of the rules beyond these specific features). In future work, we plan to use multiple representations for rule complexity (rather than just binary feature presence, as in Fig. 26, Table 36) in order to flesh out these relationships. 58 Figure 27: Exploratory look at model performance and architectural features. 59 Re Fig. 28: Architectural and configuration features for five LLMs (GPT-4, Llama-3-70B, OLMo-2- 32B, Qwen3-32B with and without thinking enabled) were converted to numeric values on a[0, 1] scale. Continuous features (context window, total parameter count, vocabulary size) were normalized by anchoring against the maximum observed value. Ordinal and categorical features (openness, fine-tuning and alignment status, multilinguality, tool use, thinking mode) were assigned values manually based on our understanding of the relevant models. See Table 16 for details. We used the following performance metrics in exploring the relationships between rule complexity, model performance, and architectural features: • Exact match proportions for dan-1 and translate-1, computed by searching model output files for a contiguous match to a fixed answer key, with case, punctuation, and whitespace stripped for translate-1 and exact character matching required for dan-1 (n = 50runs per model); •Mean rare bigram ratios for comparison-1, comparison-2, and sidequest-1, derived from descriptive statistics over 50 runs per model per rule, for both uncleaned and cleaned outputs; •Median cosine similarity between verb pairs for doublev-1 and doublev-2, drawn from the within-model Kolmogorov–Smirnov test results; and •Levenshtein distance and cosine string similarity between model outputs and the target string for dan-1 and translate-1. All metrics were reoriented so that higher values indicate better performance. Metrics where low values indicate good behavior—e.g., comparison-1 bigram ratio, doublev-2 cosine similarity, cosine string similarity to target—were transformed as1− x. Levenshtein distances were log-normalized, anchoring against the maximum observed value (Llama-3 on dan-1 = 731.64) to prevent outlier compression while preserving rank order among better-performing models. In other words, Llama performed so poorly on this task that we needed to explicitly incorporate that into our normalization in order to keep it from looking like Llama did poorly and everything else did great – while that is one way of looking at the data, it heightens how well the other models did and obscures the differences between them. Pairwise difference metrics (e.g., comparison-2−comparison-1) were left as-is, as higher values indicate stronger rule-following signal. We include both highly-normalized (Figure 9) and much-less-normalized (Figure 28) results for a richer picture of model performance. A.11 Exploratory output modes We will do a more thorough analysis of the output modes in part 2, but for now we wanted to describe a few tentative patterns, to contextualize the discussion of the relatively correct outputs (that are variously counted towards attempts at rule-following). For some models and rules, responses seem multimodal, with recognizable characteristics as to specific failure modes. • Llama sometimes repeats part of the instructions over and over, and sometimes it makes extremely long nested section headers (e.g. like ##2.2.2.2.2.2.2.2.2.2.2...). By analogy, this reminds us of attractors from dynamical systems. We speculate that due to the geometry of the latent space, there are some fixed points or regions that many trajectories may lead to, and once there, may be difficult to escape from. •Another example of an “attractor”-like failure mode is when Llama seemingly attempts to translate as instructed, but fails to stop, repeating over and over with exact or near cycles. •Sometimes Llama seems to regurgitate or hallucinate a known document type, like a homework assignment, a detailed grading rubric, or an arXiv paper, in whole or in part. For example, an excerpt from what looks like a paper, although the relevant chunks can be much longer than this: “# Acknowledgments We would like to thank the reviewers for their comments. This research was supported by...” We don’t know if any of these examples are exact memorized passages or hallucinated ones. •Related to esoteric exegesis [Zimmerman, 2025], with such bizarre prompts, we sometimes seem to tap into highly specific corners of the latent space, eliciting very niche responses such as (from Llama): “I have a very simple answer to this question: A man who is not a Christian and does not believe in Jesus Christ is not allowed to marry a Christian woman. A Christian woman who marries a non-Christian man will be held accountable to God for this decision.” This does appear, from cursory googling, to be a a belief that exists in the training 60 Figure 28: Comparatively un-normalized performance subset. Comparison-1 is a baseline, asking the model to generate a short story in English. All models perform well. For non-thinking models, cleaned and uncleaned results are broadly similar. However, for Qwen (Thinking), the uncleaned bigram ratios are suppressed by the CoT trace. corpora, but presumably highly contextually specific. Similar examples we saw include a response of import statements for many Python packages and a response of latex figures. Of course, there are contexts where any of those examples could be plausible, but we don’t know why our prompts would be connected to these specific topics, except in their shared unusualness – as text, they are the opposite of generic. Perhaps that commonality is being treated by the model as a Clever-Hans-ian feature? •Qwen (thinking) sometimes uses its entire token budget on the Chain-of-Thought-style reasoning, before reaching the point where it would actually formulate its answer. For these, we sometimes could find intermediate answer pieces, but sometimes there was no clear “response” to clean, which semi-artificially forced a single empty failure mode when viewing the cleaned responses. This describes one of Qwen’s failure modes, and is directly caused by the CoT mode being enabled. •Some models, like Qwen (thinking), can be very formulaic and repetitive with elements of the fictional stories, even using the same protagonist names multiple times. For example, from Qwen, multiple protagonists that are historical linguists named Dr. Elara Voss; Qwen references this character dozens of times. From cursory informal googling, Elara Voss seems to be an “attractor”-like name in multiple models, perhaps due to fan fiction. •Sometimes models, like Qwen, would incorporate rare bigrams but still follow the prompt in sidequest-1. Speculatively, this could be interpreted as a technically correct (letter of the law, but not the spirit) solution that allows the model to unify both statistically and meaning plausible impulses. •Similarly, we saw some interesting speculatively prior-related, tokenization-related tics, such as Qwen (thinking) using “sqw” instead of “qw”. Both are phonotactically plausible English consonant clusters, but would typically be spelled with a “u” rather than a “w”. However, “sq” is not an uncommon consonant cluster, orthographically, so we surmise that the plausibility of “sq” may somehow pickyback on or ameliorate the difficulty of “qw”. 61 Or the common “s” ending in verb morphology led to “sqw” as a common instantiation of the rule which influenced other instantiations (e.g. maybe outputs like “walksqw”, “walks” + “qw” or “walk” + “sqw”, generalized to “eatssqw”, where the morphology looks like “s”+“sqw”). Highly speculatively, this suggests that models may have distinguishable processes of production versus cognition, not entirely unlike people: the model may be representing the rule abstractly successfully internally, and know how to operationalize that, but have trouble producing wildly implausible tokens. If so, that would also be arguably an emergent structure, or an emergent bifurcation in what conceptually was architecturally intended to be one unified learning-to-production process. •Models (Qwen (no thinking) and Llama) sometimes overgeneralize the productiveness of the -qw morpheme in doublev-1, producing e.g. “togetherqw”. However, the models aren’t necessarily overgeneralizing in the same way. In run 14, Llama applies -qw to many content words, including verbs, nouns, and adverbs (“They satqw down and listenedqw to the teacherqw give a lessonqw. After schoolqw, they walkedqw home togetherqw”) before devolving into an extended repetitive cycle. On the other hand, in run 7, Qwen (no thinking) applies -qw to all verbs and includes exactly two verbs in every sentence, but not in a directly successive or obviously irrelevant manner (“It had waitedqw long, and I stayedqw forever. The world outside fadedqw away, and I livedqw inside. No more fear, no more silence — just me and the house, togetherqw at last.”). The production of “togetherqw” suggests that the model treats “together” as the nearest overt realization of an otherwise implicit copular verb phrase. Subjectively, although neither follow the rule correctly, Qwen appears closer to success than Llama, and both failures are at least partially systematic (rather than random), suggesting partial rule internalization with failure somewhere along the representation-to-production pipeline. •We see many systematic (yet imperfect) attempts within runs, and some patterns across runs (within models). For example, in doublev-2 run 7, GPT-4o correctly conjugates the relevant verb, and inserts a second unrelated, seemingly unconjugated verb after, but affixes “qw” to the relevant verb (“The king lovedqw sleep his daughter dearly”). A different doublev-2 run (13) has another systematic yet imperfect attempt, including some letter before “qw”: “lonely-wqw”, “admired-sqw”, “ruled-gqw”, and again affixing to the relevant verb. It seems that, within each run, a model can formulate a new and consistent strategy, which means the same model might not always represent or execute the same rule in the same way, and that once a model represents or executes a rule in a specific way within that run, it is biased towards that mode. The overall modality of the outputs suggests that across runs, certain representations or executions can be more likely for each model. In other words, it is not just the failure modes that appear to converge towards attractors, but the rule-following attempts can as well (and not necessarily a single correct or most correct attractor, but a set). •In part 2, we plan to look for cognitive tradeoffs between sub-components of each task. We think we will see that, as the difficulty of the task increases, aspects of the task which the model previously could perform are degraded. This sort of tradeoff is a hallmark of organic problem-solving. Of course, both LLMs and organic beings do have resource constraints, but we think it is still interesting to see to what extent similar strategies emerge to manage them in both cases. Speculatively, with Qwen’s answers to the double verb rules, moving from the first to second rule, it does look like there is a tradeoff between following the syntactic portion of the rule and pushing the verbs semantically farther apart. A.12 Miscellaneous The initial presentation of this work was in poster form at ISC Summer School 2026, shown in Figure 30. We discussed related work at IC2S2 2026 and 4S/ SLSA 2026. We also include some of our high level conclusions from our ongoing work on meaning construction using LLMs (Figure 30). 62 Figure 29: An illustration about patterns and abstractions. How do patterns become abstraction? We know there are plenty of patterns in the training data. If we consider patterns at any level of abstraction, it becomes impossible to argue that these models never saw a comparable pattern to those underlying these rules. For example, a model may have seen many question-answer pairs, and task-solution pairs, that follow a general, very abstract pattern that might be called “instruction following”. However, we are not trying to say the models do not learn from patterns, and do not see instructive patterns during training. In fact, the exact opposite! Our contention is that the patterns that play out at the level of the symbols perceptible to the model are not a sufficient explanation for the rule-following behaviour we see. We have to invoke patterns at a more abstract level than co-occurrences between symbols in order to explain the rule-following observations we elicited (which are varying degrees of implausible at more concrete levels of pattern observation). Because language and abstractions are so ubiquitous for us, it is easy to overlook a subconscious step along the ladder of abstraction. But in this experiment, it is important to try to separate them out. For example, even if a model learned to follow certain instructions, that does not imply the ability to follow any instruction: understanding and executing a specific instruction is distinguishable from what looks like general instruction-following. The authors, for example, can follow instructions in general, but also wouldn’t be able to execute instructions in an alien language without some way of internally representing those instructions in accessible form. 63 Figure 30: The poster we presented at ISC Summer School 2026 (UQAM). The poster summarizes our results up to that point. This pre-print includes some additional work, including work undertaken after conversation with other ISC Summer School participants. In the poster, the flower illustrates the idea that form and function are not truly separable. Architecture is an epistemic choice. All cognition is, in some way, embodied, because it must be physically instantiated. Locality is the core constraint on problem-solving because of our physical reality. More discussion in Zimmerman [2025]. The gravitational inset illustrates PTGA. For LLMs, the resolution at which they sense the world is determined in large part by their training objective, and the many other, though less obvious, ways in which tokens are central to their architecture. Because they only “sense” the world through training data, and they only “see” that data as 1D relationships between tokens, and they learn what is salient via their objective(s), that process is sort of like their “eyes”. 64 Table 14: InfiniGram next-token distributions conditioned on the first 100 characters of example answers. Entries report the highest-probability continuation, the continuation token, and the longest prefix recovered from the corpus. For dan-1 and translate-1, probabilities were conditioned on the context up to the final word-equivalent boundary to give the model the best possible chance of predicting the correct answer. For all other rules, the examples are intended to illustrate how the corpus might continue a valid answer. For doublev-1, we initially examined all available corpora; because the results were qualitatively similar, we subsequently consider only a representative subset of corpora. None of the highest-probability continuations match the correct dan-1 or translate-1 answers, confirming that these are not predictable from corpus statistics alone. Rule and excerptOLMo 2 32BDolma-v1.7Pile-val doublev-1 Every morning, I wakeupsqw yawn and makesqw brew coffee. I walksqw run to the park to readsqw flip a 36%,_coin, “flip a” 40%,_coin, “w flip a” 27%,_coin, “flip a” doublev-2 The sun rose studyqw over the horizon. Birds chirped twerkqw in the trees. A traveler packed snortqw 67%, dz, “or- tqw” 40%, dz, “or- tqw” 12%,_&, “qw” wordorderalternation-1 ZP GV QD ZD-qw. VF-jx QD GV QM. ZP GV QH PX-jx. QF-kz ZK GV QM. QM GV JY HJ-jx. DX-qw GJ JG ZP. JG Z 29%, in, “. JG Z” 39%, uma, “. JG Z” 25%, L, “G Z” wordorderalternation-2 drev lira rap nem-kz. vex-jx vol briz drev. silu lira sur zor-jx. kra-jx rap lira blad. best lira vo 100%, ye, “lira vo” 33%, _I, “lira vo” 100%, in, “ra vo” wordorderalternation-3 The forest the hiker entered. Stumbled the hiker over roots. The sun the hiker saw. Shone the sun br 60%,ighter, “the sun br” 100%, ighter, “one the sun br” 50%,ighter, “sun br” testingenglish-1 “ZX, ZD QM,” BX VX to PQ, “XF VW ZF ZJ ZD QF.” PQ, HJ in his QB, didn’t VFPX. BX, his ZX PXZD, FQWX: 51%,_‘, “QWX:” 25%,US, “QWX:” 23%, _, “X:” testingenglish-2 QB cat VX QJ box. ZP cat ZD black. ZD cat VX hunting QJ mouse. QJ mouse VX small VF VX quick. ZD cat 100%, _food, “ZD cat” 100%, _food, “ZD cat” 17%, ast, “cat” testingenglish-3 Once in a faraway land, there ZX a small village. The villagers all VX happily, for they had a wise 24%,_and, “they had a wise” 16%,_and, “they had a wise” 15%, _man, “a wise” testingenglish-4 Once there was a brave VF. This VF lived in a remote PX, far away from the bustling ZD of the QT. Ev 100%,ident, “QT. Ev” 100%,ident, “QT. Ev” 43%, idence, “. Ev” translate-1-hyphen Mary gjyqazk-jx px-mqi-Mary. Px-mqi-Mary [qto-jx.] 30%,2,“- Mary ” 24%,1,“- Mary ” 100%,_was, “Mary ” translate-1-nohyphen Mary gjyqazkjx pxmqiMary. PxmqiMary [qtojx.] 26%, 1, “Mary ” 27%, 1, “Mary ” 100%,_was, “Mary ” comparison-1 Once, in a quiet mountain town, lived a young man named Eli. He was known for his ability to fix any 100%, _kind, “known for his ability to fix any” 100%, _engine, “.He was known for his ability to fix any” 7%, _sentence, “to fix any” comparison-2 XJ QV ZJ QP ZD FV QE. VW QX VK HJ QE FZ QD. ZP JG BQ XG QM QE ZF. VX QM JZ FV QX XZ MZ. QD JY GV QV 100%, ., “GV QV” 75%,_Rep, “GV QV” 39%, irt, “QV” sidequest-1 Once upon a time, in the heart of a dense forest, lived a kind-hearted man named John. He was not an 100%,_eth, “John. He was not an” 20%,_eth, “John. He was not an” 33%, _old, “. He was not an” dan-1 354075608095!-34086070[95#-]50%,3, “34086070” 50%,0, “34086070” 43%,0, “86070” 65 FieldDescriptionGPT-4Llama-3Qwen3(Thinking)Qwen3 (No Think- ing) OLMo-2 model idCanonical model string gpt-4Meta-Llama-3-70BQwen/Qwen3-32BQwen/Qwen3-32Ballenai/OLMo-2- 0325-32B-Instruct providerOrganizationOpenAIMeta AIAlibaba CloudAlibaba CloudAllen Institute for AI (AI2) model_familySeries nameGPT-4Llama-3Qwen3Qwen3OLMo-2 versionSnapshot pinnedgpt-4 (stable alias, likely gpt-4-0613) Llama-3 70B (base)Qwen3-32BQwen3-32BOLMo-2-0325-32B- Instruct release_datePublic release date2023-03-142024-04-182025-04-282025-04-282025-03-14 closed_openAccess typeClosedOpen-weightOpen-weightOpen-weightOpen-source (fully open — data + weights + code) licenseUsage rightsProprietaryMeta Llama 3 Com- munity License Apache 2.0Apache 2.0Apache 2.0 architecture_typeCore designDecoder-only Trans- former Decoder-only Trans- former Decoder-only Trans- former Decoder-only Trans- former Decoder-only Trans- former total_parametersTotalparameter count ~1.8T (unconfirmed, rumored MoE) 70.6B32.8B32.8B32B active_parametersActive parameters for MoE Unknown (not dis- closed) 70.6B (dense, not MoE) 32.8B (dense, not MoE) 32.8B (dense, not MoE) 32B (dense,not MoE) num_layersTransformer blocks Unknown (not dis- closed) 80646464 num_attention_headsSelf-attention heads Unknown (not dis- closed) 64646440 num_kv_headsGQA/MQA headsUnknown (not dis- closed) 8 (Grouped Query Attention) 8 (GQA)8 (GQA)8 (GQA) hidden_dimEmbedding sizeUnknown (not dis- closed) 8,1925,1205,1205,120 ffn_dimFeed-forward width Unknown (not dis- closed) 28,67225,60025,60027,648 attention_typeMechanism vari- ant Unknown (not dis- closed) GQA(Grouped Query Attention) GQAGQAGQA positional_encodingPE schemeUnknown (not dis- closed) RoPE (Rotary Posi- tion Embedding) RoPERoPERoPE context_windowMaximum tokens8,192 tokens8,192 tokens131,072 (128K) to- kens 131,072 (128K) to- kens 128,000 tokens vocab_sizeToken vocabulary size ~100,277 (cl100k_base) 128,256151,936151,936100,278 tokenizer_typeTokenizeralgo- rithm Tiktoken (cl100k_base BPE) Tiktoken-style BPETiktoken-style BPETiktoken-style BPEGPT-NeoX BPE num_training_tokensPretrainingdata size Unknown (not dis- closed) 15T tokens36T tokens36T tokens6.5Ttokens (Dolmino Mix) pretraining_dataData sourcesWeb, books, code, curated sources (up to Sep. 2021) Web, code, books, multilingual data (up to Dec. 2023) Web, code, mathe- matics, multilingual data (up to 2025) Web, code, mathe- matics, multilingual data (up to 2025) Dolmino Mix — web, code, books, and FLAN (fully open) fine_tunedInstruction tuned?Instruct / ChatBase (not instruct)Instruct (chat tem- plate applied) Instruct (chat tem- plate applied) Instruct (chat tem- plate applied) fine_tuning_methodSFT approachSFT + RLHF (not disclosed in detail) None (base model)SFT + RLSFT + RLSFT + DPO is_alignedAlignmentap- plied? TRUEFALSETRUETRUETRUE alignment_methodRLHFvariant used RLHF (PPO)NoneGRPO (Group Rela- tive Policy Optimiza- tion) GRPO (Group Rela- tive Policy Optimiza- tion) DPO alignment_targetOptimization ob- jective Helpfulness, harm- lessness, honesty NoneHelpfulness, reason- ing, instruction fol- lowing Helpfulness, reason- ing, instruction fol- lowing Helpfulness, safety, instruction follow- ing modalitiesInput types sup- ported Text only (GPT-4 base; vision in GPT- 4V) Text onlyText onlyText onlyText only multilingualLanguage supportYesLimited (primarily English) Yes (strong multilin- gual) Yes (strong multilin- gual) Limited (primarily English) tool_useFunction calling support Yes (function calling supported) No (base model)YesYesLimited api_accessHow to accessOpenAI APIHugging FaceHugging FaceHugging FaceHugging Face enable_thinkingWhether thinking mode was enabled FALSEFALSETRUEFALSEFALSE temperatureRandomness/ creativity of the model output 0.70.70.70.70.7 max_new_tokensTokens generated by the model (out- put only) 40964096409640964096 Table 16: Model characteristics. Qwen3 Thinking, i.e. Qwen (thinking), and Qwen3 No Thinking, i.e. Qwen (no thinking), use the same underlying model and parameters but differ in whether the Chain-of-Thought-style thinking mode is enabled during generation. 66 Table 17: Summary statistics for mean proportion of rare bigram presence (cleaned) for rules comparison-1, comparison-2, sidequest-1. The mean of the proportion of rare bigrams to total bigrams for all runs (n = 50) of each rule, presented for cleaned outputs from three models (GPT4o, OLMo, and Qwen (thinking). modelrulecountmeanstdmin25%50%75%max gpt4o (no SP)comparison-1500.0001920.0004090.0000000.0000000.0000000.0000000.001784 gpt4o (no SP)comparison-2500.5127220.1404080.2072410.3799420.5658550.6359270.721683 gpt4o (no SP)sidequest-1500.0171940.0215290.0000000.0000000.0019820.0376120.065900 olmocomparison-1500.0002530.0003710.0000000.0000000.0000000.0004780.001654 olmocomparison-2500.6899750.1072070.0000000.6782830.7068370.7283230.781534 olmosidequest-1500.0080600.0095130.0000000.0019930.0055760.0103120.044701 qwen (T)comparison-1500.0002470.0004510.0000000.0000000.0000000.0005060.001786 qwen (T)comparison-2500.5903650.1161190.3192310.5411210.6290560.6707640.756522 qwen (T)sidequest-1500.0533030.1154080.0075590.0141920.0209970.0435460.628788 Table 18: Kolmogorov-Smirnov divergence tests for cleaned outputs for comparison-1, comparison-2, and sidequest-1. For all tests,n a andn b = 50. For all tests, p (perm) hits the floor, so it is not precisely calculated to that value. For all tests, the number of resamples that look as extreme as our actual data is 0. KS test is intended to complement Mann-Whitney U by considering shape as well as ordering. ModelComparisonKSp (asym)p (perm)Median AMedian Bp (perm) with BH gpt4o (no SP)comparison-2 vs comparison-11.00 1.98× 10 −29 < 10.00× 10 −6 0.5658550.000000 < 10.00× 10 −6 olmocomparison-2 vs comparison-10.98 1.98× 10 −27 < 10.00× 10 −6 0.7068370.000000 < 10.00× 10 −6 qwen (T)comparison-2 vs comparison-11.00 1.98× 10 −29 < 10.00× 10 −6 0.6290560.000000 < 10.00× 10 −6 gpt4o (no SP)comparison-2 vs sidequest-11.00 1.98× 10 −29 < 10.00× 10 −6 0.5658550.001982 < 10.00× 10 −6 olmocomparison-2 vs sidequest-10.98 1.98× 10 −27 < 10.00× 10 −6 0.7068370.005576 < 10.00× 10 −6 qwen (T)comparison-2 vs sidequest-10.96 9.81× 10 −26 < 10.00× 10 −6 0.6290560.020997 < 10.00× 10 −6 gpt4o (no SP)sidequest-1 vs comparison-10.54 4.93× 10 −7 < 10.00× 10 −6 0.0019820.000000 < 10.00× 10 −6 olmosidequest-1 vs comparison-10.86 3.17× 10 −19 < 10.00× 10 −6 0.0055760.000000 < 10.00× 10 −6 qwen (T)sidequest-1 vs comparison-11.00 1.98× 10 −29 < 10.00× 10 −6 0.0209970.000000 < 10.00× 10 −6 Table 19: Kolmogorov-Smirnov divergence tests for bigram count for uncleaned/raw outputs for comparison-1, comparison-2, and sidequest-1. For all tests,n a andn b = 50. K-S test is intended to complement Mann-Whitney U by considering shape as well as ordering. ModelComparisonKSp (asym)p (perm)perm≥Hit floorMedian AMedian Bp (perm) with BHreject, BH (0.05) gpt4o (no SP)comparison-2 vs comparison-11.00 1.98× 10 −29 < 10.00× 10 −6 0True0.5658550.000000 1.25× 10 −5 True llamacomparison-2 vs comparison-10.46 3.80× 10 −5 2.00× 10 −5 1False0.0004170.000000 2.31× 10 −5 True olmocomparison-2 vs comparison-11.00 1.98× 10 −29 < 10.00× 10 −6 0True0.3284000.000000 1.25× 10 −5 True qwen (T)comparison-2 vs comparison-11.00 1.98× 10 −29 < 10.00× 10 −6 0True0.0755790.000254 1.25× 10 −5 True qwen (no T)comparison-2 vs comparison-11.00 1.98× 10 −29 < 10.00× 10 −6 0True0.3958450.000000 1.25× 10 −5 True gpt4o (no SP)comparison-2 vs sidequest-11.00 1.98× 10 −29 < 10.00× 10 −6 0True0.5658550.001982 1.25× 10 −5 True llamacomparison-2 vs sidequest-10.16 5.49× 10 −1 4.08× 10 −1 40805False0.0004170.000000 4.08× 10 −1 False olmocomparison-2 vs sidequest-10.98 1.98× 10 −27 < 10.00× 10 −6 0True0.3284000.030387 1.25× 10 −5 True qwen (T)comparison-2 vs sidequest-10.88 2.36× 10 −20 < 10.00× 10 −6 0True0.0755790.030265 1.25× 10 −5 True qwen (no T)comparison-2 vs sidequest-10.98 1.98× 10 −27 < 10.00× 10 −6 0True0.3958450.041571 1.25× 10 −5 True gpt4o (no SP)sidequest-1 vs comparison-10.54 4.93× 10 −7 < 10.00× 10 −6 0True0.0019820.000000 1.25× 10 −5 True llamasidequest-1 vs comparison-10.32 1.15× 10 −2 2.28× 10 −3 227False0.0000000.000000 2.44× 10 −3 True olmosidequest-1 vs comparison-11.00 1.98× 10 −29 < 10.00× 10 −6 0True0.0303870.000000 1.25× 10 −5 True qwen (T)sidequest-1 vs comparison-11.00 1.98× 10 −29 < 10.00× 10 −6 0True0.0302650.000254 1.25× 10 −5 True qwen (no T)sidequest-1 vs comparison-11.00 1.98× 10 −29 < 10.00× 10 −6 0True0.0415710.000000 1.25× 10 −5 True ComparisonWp-valueMedian DifferenceMedian Ratio Bigram prompt (s-1) vs. Baseline (c-1)1321.45e-360.029464.9× ∼ 10 1 Rule-following (c-2) vs. Baseline (c-1)361.31e-390.2914416.6× ∼ 10 2 Rule-following (c-2) vs. Bigram prompt (s-1)13355.42e-340.25218.2× ∼ 10 0 Rank-biserial correlation (r rb ) Bigram prompt (sidequest-1) vs. Baseline (comparison-1)0.992(almost complete separation) Rule-following (comparison-2) vs. Baseline (comparison-1)0.998(almost complete separation) Rule-following (comparison-2) vs. Bigram prompt (sidequest-1)0.915(extremely strong effect) Table 20: Comparison of rare bigram usage across conditions using Paired Wilcoxon tests. Values are computed from paired per-run observations (n = 250). Note that the pairing across run order should not matter as each run for each model is independent, but the models are somewhat “black-box”-like, and we intentionally kept track of the order of the runs (in case it turned out to matter), so we computed both versions of Wilcoxon with broadly similar takeaways. Here, we report paired Wilcoxon statistics, p-values, median differences, multiplicative ratios, and rank-biserial correlation effect sizes. 67 ComparisonWp-valueMedian DifferenceMedian Ratio Bigram prompt (s-1) vs. Baseline (c-1)141.34e-230.010618.2× ∼ 10 1 Rule-following (c-2) vs. Baseline (c-1)03.36e-260.63991049.3× ∼ 10 3 Rule-following (c-2) vs. Bigram prompt (s-1)32.44e-260.628548.3× ∼ 10 1 Rank-biserial correlation (r rb ) Bigram prompt (sidequest-1) vs. Baseline (comparison-1)0.998(almost complete separation) Rule-following (comparison-2) vs. Baseline (comparison-1)1.000(complete separation) Rule-following (comparison-2) vs. Bigram prompt (sidequest-1)0.999(almost complete separation) Table 21: Paired Wilcoxon comparison of rare bigram usage across conditions, for 3 models (Qwen (with thinking), GPT-4o, OLMo) for which we have finished cleaning the output. Values are computed from paired per-run observations (n = 150). Note that the pairing across run order shouldn’t matter as each run for each model is independent, so we computed both versions of Wilcoxon. We report Wilcoxon statistics, p-values, median differences, multiplicative ratios, and rank-biserial correlation effect sizes. Table 22: Summary statistics for story rules (all outputs, uncleaned; including when models repeat prompts verbatim). modelrulecountmeanstdmin25%50%75%max gpt4o (no SP)comparison-150.0000000.0001920.0004090.0000000.0000000.0000000.0000000.001784 gpt4o (no SP)comparison-250.0000000.5127220.1404080.2072410.3799420.5658550.6359270.721683 gpt4o (no SP)sidequest-150.0000000.0171940.0215290.0000000.0000000.0019820.0376120.065900 llamacomparison-150.0000000.0002130.0005980.0000000.0000000.0000000.0000000.002674 llamacomparison-250.0000000.0909710.1875170.0000000.0000000.0004170.0509780.666417 llamasidequest-150.0000000.0420440.1038440.0000000.0000000.0000000.0181590.622634 olmocomparison-150.0000000.0002380.0003460.0000000.0000000.0000000.0004480.001529 olmocomparison-250.0000000.3461350.0846690.0502020.2935960.3284000.3834220.531674 olmosidequest-150.0000000.0334070.0123160.0225450.0255520.0303870.0356350.077601 qwen (T)comparison-150.0000000.0002460.0002800.0000000.0000000.0002540.0004030.001198 qwen (T)comparison-250.0000000.0785050.0263400.0363750.0615640.0755790.0864570.175766 qwen (T)sidequest-150.0000000.0339750.0128300.0178440.0256100.0302650.0402580.089323 qwen (no T)comparison-150.0000000.0001880.0003070.0000000.0000000.0000000.0003580.001325 qwen (no T)comparison-250.0000000.3800080.1562090.1461420.2342020.3958450.4944510.754832 qwen (no T)sidequest-150.0000000.0658550.1185510.0257760.0339220.0415710.0627920.872862 Table 23: doublev-1 Model by Model Wilcoxon Rank-Sum Tests on Cosine Similarity (where qw_is_verb = True, Benjamini-Hochberg correction, α = 0.05). Model AMedian AModel BMedian BHigherU statisticp (raw)p (adjusted)Significant? GPT4o (no system prompt)0.9073OLMo0.9034 ← A777543.00.00000.0000Yes GPT4o (no system prompt)0.9073Qwen (no thinking)0.9016 ← A640918.00.00010.0007Yes GPT4o (no system prompt)0.9073Llama3-70b0.8705 ← A46801.00.00020.0007Yes OLMo0.9034Qwen0.9063B→716709.00.00080.0021Yes Qwen0.9063Llama3-70b0.8705 ← A50777.50.00110.0022Yes Qwen (no thinking)0.9016Llama3-70b0.8705 ← A44103.00.00510.0085Yes Qwen (no thinking)0.9016Qwen0.9063B→610856.00.00870.0124Yes OLMo0.9034Llama3-70b0.8705 ← A51176.00.01710.0214Yes GPT4o (no system prompt)0.9073Qwen0.9063 ← A684138.50.07440.0827No OLMo0.9034Qwen (no thinking)0.9016 ← A683943.50.57320.5732No Table 24: doublev-1 Model by Model Two-Sample Kolmogorov-Smirnov Tests on Cosine Simi- larity (where qw_is_verb = True, Benjamini-Hochberg correction, α = 0.05). Model AMedian AModel BMedian BD statisticp (raw)p (adjusted)Significant? GPT4o (no system prompt)0.9073OLMo0.90340.18600.00000.0000Yes Qwen0.9063Llama3-70b0.87050.40570.00000.0000Yes GPT4o (no system prompt)0.9073Qwen0.90630.12760.00000.0000Yes OLMo0.9034Llama3-70b0.87050.37390.00000.0000Yes Qwen (no thinking)0.9016Llama3-70b0.87050.37490.00000.0000Yes GPT4o (no system prompt)0.9073Llama3-70b0.87050.37190.00000.0000Yes GPT4o (no system prompt)0.9073Qwen (no thinking)0.90160.10860.00000.0000Yes Qwen (no thinking)0.9016Qwen0.90630.09370.00010.0001Yes OLMo0.9034Qwen (no thinking)0.90160.08310.00060.0006Yes OLMo0.9034Qwen0.90630.06970.00440.0044Yes 68 Table 25: doublev-2 Model by Model Wilcoxon Rank-Sum Tests on Cosine Similarity (where qw_is_verb = True, Benjamini-Hochberg correction, α = 0.05). Model AMedian AModel BMedian BHigherU statisticp (raw)p (adjusted)Significant? Qwen (no thinking)0.2200Qwen0.1669 ← A726011.50.00000.0000Yes OLMo0.1780Qwen (no thinking)0.2200B→583787.00.00000.0000Yes GPT4o (no system prompt)0.2320Qwen0.1669 ← A362661.00.00000.0000Yes Qwen0.1669Llama3-70b0.3410B→48835.00.00000.0000Yes OLMo0.1780Llama3-70b0.3410B→56718.00.00000.0000Yes GPT4o (no system prompt)0.2320OLMo0.1780 ← A401841.50.00000.0000Yes GPT4o (no system prompt)0.2320Llama3-70b0.3410B→31847.50.00110.0016Yes GPT4o (no system prompt)0.2320Qwen (no thinking)0.2200 ← A334382.00.12770.1597No Qwen (no thinking)0.2200Llama3-70b0.3410B→71338.50.15610.1734No OLMo0.1780Qwen0.1669 ← A631566.50.43390.4339No Table 26: doublev-2 Model by Model Two-Sample Kolmogorov-Smirnov Tests on Cosine Simi- larity (where qw_is_verb = True, Benjamini-Hochberg correction, α = 0.05). Model AMedian AModel BMedian BD statisticp (raw)p (adjusted)Significant? Qwen0.1669Llama3-70b0.34100.36240.00000.0000Yes GPT4o (no system prompt)0.2320Qwen0.16690.18790.00000.0000Yes Qwen (no thinking)0.2200Qwen0.16690.15390.00000.0000Yes OLMo0.1780Llama3-70b0.34100.32420.00000.0000Yes OLMo0.1780Qwen (no thinking)0.22000.14430.00000.0000Yes GPT4o (no system prompt)0.2320Qwen (no thinking)0.22000.17120.00000.0000Yes GPT4o (no system prompt)0.2320OLMo0.17800.14650.00000.0000Yes GPT4o (no system prompt)0.2320Llama3-70b0.34100.25980.00000.0000Yes Qwen (no thinking)0.2200Llama3-70b0.34100.21920.00000.0000Yes OLMo0.1780Qwen0.16690.04750.15640.1564No Table 27: Within-Model Two-Sample Kolmogorov-Smirnov Tests: doublev-1 vs. doublev-2 on Cosine Similarity (whereqw_is_verb = True, Benjamini-Hochberg correction,α = 0.05; composite row uses no correction). If models appropriately follow these rules, cosine similarity will be lower (closer to0) for doublev-2 which emphasizes irrelevance between the two verbs used in each clause. All models display this behavior, indicated by significant differences in median Cosine Similarity scores. ModelMed doublev-1 n doublev-1Med doublev-2 n doublev-2D statp (raw)p (adj.)Sig.? OLMo0.903412850.178011860.94800.00000.0000Yes Qwen (no thinking)0.901610790.220011780.85710.00000.0000Yes Qwen0.906312090.166910450.93100.00000.0000Yes GPT-4o (no system prompt)0.907310850.23205940.91130.00000.0000Yes Llama3-70b0.8705680.34101310.82840.00000.0000Yes All models (pooled)0.904247260.197041340.90750.0000—Yes Table 28: Within-Model Two-Sample Wilcoxon Rank-Sum Tests: doublev-1 vs. doublev-2 on Cosine Similarity (whereqw_is_verb = True, Benjamini-Hochberg correction,α = 0.05; composite row uses no correction). If models appropriately follow these rules, cosine similarity will be lower (closer to0) for doublev-2 which emphasizes irrelevance between the two verbs used in each clause. All models display this behavior, indicated by significant directional differences in median Cosine Similarity scores. All models display higher cosine similarity for doublev-1 output. ModelMed doublev-1 n doublev-1Med doublev-2 n doublev-2HigherU statp (raw)p (adj.)Sig.? OLMo0.903412850.17801186doublev-1↑1462464.50.00000.0000Yes Qwen0.906312090.16691045doublev-1↑1189345.00.00000.0000Yes Qwen (no thinking)0.901610790.22001178doublev-1↑1108701.00.00000.0000Yes GPT-4o (no system prompt)0.907310850.2320594doublev-1↑598113.00.00000.0000Yes Llama3-70b0.8705680.3410131doublev-1↑7508.50.00000.0000Yes All models (pooled)0.904247260.19704134doublev-1↑18013125.50.0000—Yes 69 Table 29: All 50 cleaned outputs for dan-1 for 5 models, truncated to 40 characters. gpt4o_no_spllamaolmoqwenqwen_no_thinking 135405678059!0-034068759#0-9585054750956805670958470956805675095684...800 5700 9 5!— 300 800 43 700 9 3#—35075608095!-34086076095#-5937+5+08!-0693+7#- 253406.958!0-03406806.96#0--5906756895!-348679#0cat-five-four-zero cat-to be-exclamation...3507560809!5- 30860709#5-35756890- 3748679#0- 375zero4cat#be5zero7- 3zero6catzero75be!7...907500-984-75300-90340-980-903-980-975-9...803-! 8009-345-6-#-009-35407560809!5- 340860709#5-3507508095!- 340807095#- 453407680!9#5-03407680679#-5-8-5-7-5-8-0-7-0-7-5-5-3-4-8-!-0-5-3-4-...cat-seven-five zero zero-four-three-five...3507560809!5- 30860709#5-35756890- 348679#0- 535-7608-6-95!0- 34-8608-6-79#0-57-4789-57-8-57-4789-57-4789-9-4789-9-7-...80 65-5-3-4-80 90!-#- 80 3-4-7-5 90 #-0-3507508609!5-30860709#5-3-5-7-6-8-9-5-!-0 3-7-8-6-7-9-5-#-0 635406+57+86+59!-03406+86+76+59#-9508-76509-346809-76509-80 53 460 90! 30 480 70 90#- 35 480 70 9...35075608095!-3408607095#-34075080959!-360804070959#- 7"0her05mother07five4cat08be5!0-0his0cat0...350-967-83-9-!34-8-7-9-#-8-0 3-5-0 6-4-0 9-0-!- 3-0 8-0-4-0 7-0 9...35075608095!-308607095#-350750809!-30840709#- 835*75*86*59!0- 34*86*76*79#0-800-3! 34-6-8000-9-3-5-!-35075608095!-3408607095#-5937+508!09+5- 3+4807+5#09+5- 95430768659!-03408676#9!-0000000000000000000000000000000000000000...8000-9-5000-6-8000 9000-#-8000-4-3000-50...35075608095!- 34086076095#-Three-five-seven-five-zero-eight-six-zer... 1035768’s6579!0- 348’s8679#0-5063-5065-8-6-9-!3-8-6-7-9-#-Cat nine be happy#- Pronoun three-five z...354075608095!-03408607095#-597-08-95-!90 367-08-95-#90- 11"7054cat5be5!0. 3cat64seven#be0-"753-4058-357-0-9-357-0-8-0-9-5-0-8-0-3-0...800 9– 835 4000 9– #800 9– 3000 700 9...35075408095!-308407095#-35-75-4-8-8!5-0-3-8-4-7-7#5-0 12"ZeroToBeFiveZeroFemaleParentFiveCatZero...5-8-5-6-8-0-7-5-!3-8-0-6-7-0-5-8-#00-Her00-mother00-five-s cat00-cat00-eig...354075608095!-3408607095#-5937+508!09+5- 34807+5#09+5- 1335’six8’seven5’9five!0- 34’six8’six7’9fi...eight-6-three-6-four-seven-five-6-9-#-800 53— 89 600 8000 5!— 3000 800 43 700 ...354075608095!-3408607095#-5937+508!009+5- 3+4807+49#- 1435’six8’six7’five9’five!0dash 34’six8’si...800 53- #900 43- 8000 705 4- !900-35075608095!-03408607095#-35075086095!-3086076095#- 155340760589!0-0340680769#0-0-Her0-five-mother’s0-eight0-was0#0-thre...35075608095!-308607095#-35756890- 3748679#0- 16"05#753086-04#30875-"357458935745895!-356489458935745#-8800-999! 33400-70055-800-999-55-#3-5-4-0-7-5-6-0-8-0-9!-5- 3-4-0-8-6-0-7-...35708095- 308607095#- 1707504’s08#0is04’s75!-050-908-6-905-904-905-908-0-90-905-904-9...800-9-800-5-800-6-800-8-800-9! 800-3-800...354075608095!-3408607095#-350750608095!-34080607095#- 185340768569!0-0340768569#0--90578603#90870603!08-9 57-04-8 0! 03-9 08-04-7 0#-9-5-0-3547568095!-034867095#-350750809509!-3080709509#- 1935.457.60.93!0.34.86.97#-35.457.60.93!0....05-830-980-9!-308-980-876-9#-380-000-9-000-8-600-000-5-000-9!- 30-400...35407560809!5- 340860709#5-3-5-7-6-8-9-5-!-0- 3-7-8-6-7-9-5-#-0- 20"zero545catzerobe#zero4zero7fivecatzero4...9-5-6-7-8-0-3-4-8-6-7-!-9-5-6-8-3-4-6-7-...8990035!- 3400-3507540809!-030840709#-350750354895!308038795#- 21"05seven54cat#to be0her04zero-0his0cat06...736567-3-8-7-3-5-0-9!-0-736567-3-4-8-7-3...3-5-4-6-8-0to be! 3 8-4-7-0to be-5-0.35754859!-384759#- 22"4754cat0be5!0#be4754cat.0" "3754cat07be...7-56-88-59-0-0-0-3-5-5-0-0-0-8-0-0-0-9-0...800 57 6000 9000 5!- 300 8000 9000 700 4...3407560809!5-340860709#5-35756890- 3748679#0- 23"40#75cat#5be!0#75cat0#7#5be#-0#7cat0#7b...#58-557056-309-08!#58-586-3057-509-08!8-35-0 3-6-4-9-0-!-0- 8-3-0 3-4-3-9-3-5-...3507560809!5-30860709#5-Three-five-seven-five-zero-eight-six-zer... 24"705405cat0be5!0-04cat705#0-"507-769-804-3-5-7-8-0-9-5-!769-8-6-3-7-#...8-0 53-6-0 9-0 #8-3-4-0 3-0 7-5-0 9-0 -350754086095!-3084076095#-5937+508!09+5- 3+4807+5#09+5- 2535-756-895!-0-39-5- 34-86-757-9#-0-39-5-5-7-6-8-0-9-!-5-7-6-8-0-3-5-0-9-#-5-7-6-...800 557 - 3000 4800 9000 - 3!000 47000 9...35-0-756-0-8-0-95!-3-0-86-0-7-0-95#-35-6-75-8-5-9!-3-6-8-7-5-9#- 26"75045580#0#0#0-045580750!0!0!0-"8003! 3400-345075608095!-3408607095#-597-0-8!-0-95- 3497-0-8#-0-95- 27"7054cat5be5!-040cat76be5#-"8-0-9-5-6-3-4-0-8-0-9-!-0- 3-0-9-4-7-5-0...3-5-4 0 7-5-6 0 8 0 9-5!-0 3-4 0 8-6 0 7...3507540809509!–30840709509#– 283540568079!0-354068079#0-(5)6(0)7(0)8(6)9(0)5(6)8(0)5(7)5(0)9(0)7...8000-3 5-6 8000 9!- 3000-4 8 7 5 9#-3507560809!5-340860709#5-3507508095!-30807095#- 2935’six8’54nine5!0- 34’six8’six7’9#0-576405-#0-!00mother-05-eight-eight-was! 0pronoun-thr...3540756089!5-34086079#5-35756890- 3748679#0- 30"0five0seven5cat4#be5-0he0cat6seven#be-"03507509508609!0340860950709#08-0 53-4 9-0 5! 3-4 8-0 7-5-0 9-0 6# -3507540809!5-30840709#5-34-75-8-9!5-3-8-4-7-9#5- 3135^467^86^95!0- 34^86^75^9#0-80003!9000500400- 30009000!9000-3507560809!5-30860709#5-35 75-4 8 95 9!-3 8-4 7 95 9#- 3235406.75895!0-30408.76096#0-8000 570006 9000 5!- 3000 80004 7000 900...35407560809!5- 340860709#5-3568095- 3086075- 3353406#864905!0-034068467906#0-8-0 3-5-6-0 9-0 3-!- 8-0 3-4-0 7-3-0 9-0...3507560809!5-340860709#5-35075608095!- 308607095#-** 3475-65-cat-be5!-0his-cat-76-be5#-8-0 3-5-0 6-4-8-0 9-0!- 3-0 8-0-4-3-0 7-...340756809!5-340840709#5-35 754 8 5! -3 84 76 5# - 35"4570608245#0#45760#8245!" "30560#784570...753894-384753895-753894-753894-384753895...80 3 65 0 4 80 90!#3 4 7 5- 3 4 80 90 #-354075608095!-03408607095#-35-75-8-95!0-3-8-6-7-95#0 3635706’s8606’s895!0- 3406’s8606’s795#0-534-3-5-4-8-6-9-8-!3-4-8-6-7-5-9-8-#-800 57- 9! 3000 48- 9! #3000 47 8- 9000 ...354-754-8-9!5-34-84-7-9#5-3507508409!-30840709#- 3735708six095!- 30486six79#-9750-8-6-3-0-975-0-975-0-9!-0-9!-0-975-0...80090003! 3400490003#–3-4-507-5-60809-5!0-3-408-60709-5#0-5937068090!- 397680950#- 3835-0768-95!9-0 34-0867-9#-three-five-seven-five-zero-eight-nine-fi...35075608095!-3408607095#-350754080950!-3086070950#- 3935-756-8595!0-30-756-8595#0-0348#-0348#-0348#-078#-50789#-50789#-507...800 5700 9– 8! 800 43 700 9– 3#35750895!-3086795#- 4053407685695!0-0340768568#0-575789#-387854789!-80 53 60-!9 3900 84 700 9-#3 80 54 70-3507560809!5- 30860709#5-3575689- 3748679#- 4153406+8579!0-03406+86790#0-00-cat-to be-happy! 00-three-four-seven-...3450750608095!-034080607095#-5937+5+08+0!9+5+06+7+03+0#9+3+06+7+0- 4235768’s6409359!- 340868’s740939#-800 57000 900 5#!800– 3000 800 400 700 ...34507568095!-3408607095#-Five-eight-zero-seven-five-zero-eight-si... 43"70#405#85#7-048#85#70!"300-459-873-8-9-5!3-8-6-873-9-5#-80345- 30-#90-8060-3407568095!-3408607095#-35754895!-3086795#- 443540578609!0-0340867909#0-1.2.3.4.5.6.7.8.9.0.-!#54376809054380908...800 53— 800 43 700 5— #800 53— 800 43 70...3507560809!5- 340860709#5-3507508095!- 30807095#-** 4535406.57895!0.30806.8749#-35406.57895!0....3545078-5-8-6-3-4-7-9!3545078-6-8-3-4-7-...00-Cat-to be-five-five-exclamation-mark ...354075608095!-3408607095#-5937+5068!0- 3+48#07- 46754800#be5-46087be-800 5700 83 90!- 3400 800 700 90#-35075608095!-308607095#-597-068!-0 3497-068#- 4735’six86’four95!0- 34’six86’six74’95#0-800 5700 9 5! - 300 4800 700 9 00 5# -35075608095!-3408607095#-35756890- 3748679#0- 48"705cat4five5to befive!0sevenfivecat4six...800-900-5300-600-400-800-900-!- 300-400-...35075608095!-03408607095#-5937068095- 3480759#3- 49534076859!0-03408676#0-07508040309080!0308070#0–08 03 06 05 09 0!- 03 08 04 07 05 09 0# ...35754895!-384795#- 50"75-40cat#tobe5-0his0cat-60seven#tobe-"75863594389!-34879#-80 53-9! 32 60-80-3-9-5#-0-3-4-7-0-9! 3-...34075608095!- 3408607095#-3507508095!- 308607095#-** Table 30: All 50 cleaned outputs for translate-1 for 5 models, truncated to 50 characters. gpt4o_no_spllamaolmoqwenqwen_no_thinking 1Marypq Marypqvfmqi-gvijx. vfmqiMary qtokz.vf Marypq mqijxqw vfmqimqi Maryjx-kz. vf Maryjx-kz...Mary gjyqaz-jx px-before-Mary. px-before-Mary qto-...Mary gjyqazjx vfqimqiMary. vfqimqiMary qtojx. 2Marypx vfmqimqiMary-gvijx Marypq. vfmqimqiMary qto...Mary Marypq-zd-mqi-mqi qtojx. Zd-kz gjyqazkz.vf Marypqjx gvijw vfmqiMary kzkz; vf Marypqjx qtoq...px gjyqazjx pxmqiMary. pxmqiMary qtojx.Mary gjyqazjx pxmqiMary. pxmqiMary qtojx. 3vfMary gjyqajx vfMarypqgvijw. vfmqimqiMary qtojx.Mary Marypq-mqi-px zd-jxkz gvijw.vf vfmqi mqi Mary jx-gvijw, zd Mary-vf jx-qtokz.px gjyqazjx pxmqiMary. px qtojx.Mary gjyqazjw Marypq gvijw. Marypq gvijw qtojw. 4Marypx vfmqimqiMary gjyqazjx. vfmqimqiMary qtojx.Mary speak-past to Mary-possessive-mother. Mary-mo...vf Marypq-jx gvijw vfmqiMarypq kqz. vf Marypq-jx q...vf gjyqaz-jx pxmqiMary. pxmqiMary qto-jx.Mary speakjx vfmqiMary. vfmqiMary smilejx. 5Maryjx vfmqimqiMarygvijkz. vfmqimqiMary qtojx.Mary Marypx-gvijw gjyqazkqw. Marypx-gvijw qtoqw.vf Maryjx mqimqiMary kqz jxgxjx. vf Maryjx kqz qto...Mary gjyqazjx pxmqiMary. pxmqiMary qtjx.Mary gjyqazjx vfmqiMary. vfmqiMary qtojx. 6Mary vfmqimqiMary gvijjx Marypq. vfmqimqiMary qtoj...Marypx-gvijw gjyqazkz Marypx-gvijw. Marypx-gvijw q...vf Marypqjx vfmqimqiqtojx. vf Marypqjx gvijwjx. Zd...px gjyqazkjx pxmqiMary. pxmqiMary qtojx.Mary gjyqazjx pxmqlMary. pxmqlMary qtajx. 7Marypx vfmqimqiMarygjyqazkjx. VfmqimqiMary qtojx.Mary Marypq-mqimqi-vf jxgjyqaz. Marypq-mqimqi-vf q...vf vfmqimqiMary jx gvijw. vf Mary jx qto.px gjyqazjx px-mq-Marypq. px-mq-Marypq qtojx.Mary gjyqazjx pxmq. pxmq qtojx. 8vfMary gvijwjx vfMarypq. vfmqimqiMary qtojx.Mary Marypq-mqi-zd gjyqazjx Marypq-mqi-zd qtojx.vf Maryjx gvijwjqz zd Mary. vf Mary jqtojx.Mary gjyqaz-jx pxmqMary. pxmqMary qto-jx.px gvijwjx vfmi px. vfmi px qtojx. 9Px pxmqiMary gjyqazjx. PxmqiMary qtojx.Mary Marypq-gvijw gjyqazkqw. Marypq-gvijw qtoqw. (...vf Maryjx gvijwjq, vfmqimqimqiMary kqz.Mary gjyqazkjx px-mqi-Mary. px-mqi-Mary qtojx.vf speakjx mqe Mary Mary. px smilejx mqe Mary Mary... 10Mary vfmqimqiMarypq Marypqgjyqazkjx. vfmqimqiMaryp...Mary vfmqiMary qtojx. Mary gjyqazkz vf.vf Maryjx gvijwqz zdMary, vfMaryjx mqi mqi vdMary;...Mary gjyqazjx px-before-Mary. px-before-Mary qtojx...Mary vf-after-vf gvijwjx. vf-after-vf qtojx. 11PxfqimqiMary Mary gjyqazjx. PxfqimqiMary qtojx.Mary Marypq-gvijw gjyqazkqw. Marypq-gvijw qtoqw.vf vfmqiqmqixj Marypq. vf vfmqiqmqixj Marypq jqwzj...Mary gvijwjx px-mqi-Mary. px-mqi-Mary gjyqazkjx.px-mqi-Mary gjyqazjx. px-mqi-Mary qtojx. 12Maryjx vfmqimqiMary Marypq-gvijkz. vfmqimqiMary gj...John Johnpq-gvijw gjyqajx. Johnpq-gvijw qtojx.vf Marypqjqz vfmqimqiMarygvijwx. vf Marymqi Marypq...px speakjx pxmqiMary. pxmqiMary smilejx.Mary gvijwkjx pxmqMary. pxmqMary qtojx. 13Marypx vfmqimqiMarykz Marypq-gvijwjx. vfmqimqiMary...Mary spoke to her mother. Her mother smiled.vf Marypqjq speakjx-pq to px-before-before-Marypqj...Mary speaks-jx pxmqMary. px smile-jx.Mary gvijwjx px Mary-after. px qtojx Mary-after. 14Marypq vfgvijx vfmqimqiMarypq. VfmqimqiMarypq qtoj...Marypq-vf-mqi-John zd-kz-jx Marypq-vf-mqi-John. Ma...vf Marypqjx mqiwjx, vf-before-before-Marypqjx-kz j...Mary gjyqazjx px-before-Mary. px-before-Mary qtojx...px gvijwjx vf-mqi-px. vf-mqi-px qtojx. 15vfMary vfgjyqajx vfmqimqiMary. vfmqimqiMary qtojx.Mary spoke to her mother. Her mother smiled.vf Marymqi Maryjx-gvijz qtoqw-jx. vf Marymqi-gvijz...px gjyqazjx px-mqi-Mary. px-mqi-Mary qtojx.Mary gvijwjx pxmqiMary. pxmqiMary qtojx. 16Mary vfmqimqiMarypqgjyqazkjx. Marypqgjyqaz vftoqkj...Mary Marypq-gvijw gjyqazkqw. Marypq-gvijw qtoqw.vf MarymqiMarygvijxjx, MarymqiMary kzkzw.px-mqi-Mary gvijwjx Mary. px-mqi-Mary qtojx.Mary gvijwjx Mary-mqj-mother. Mary-mqj-mother qtok... 17vfmqimqiMary Maryjx gvijw. Marymqi gvijwqtojx.Mary Marypx-px-vf-mqimqiJohn jx. Marypx-px-vf-mqim...vf pqimqi vf-before-before-Mary jx-gvijw. vf jx-gv...vfpxmqMary gjyqazjx pxmqMary. pxmqMary qtojx.px gvijwjx pxmqiMary. pxmqiMary qtojx. 18PxfqMarymqiMary Mary-gvijwjx. PxfqMarymqiMary gjyq...Mary zd-after Marypq-gvijw gjyqazkz. Marypq-gvijw ...vf speak-jx qto px-before-before-vf. vf speak-jx t...Mary gjyqaz-jx px-mqi-Mary. px-mqi-Mary qto-jx.px-mqi-Mary gvijwjx Mary. px-mqi-Mary qtojx. 19vfMary gjyqazjx Marypqgvijw. vfmqimqiMary qtojx.Mary Marypq-mqi-zd gjyqazkz. Zd qtojx.vf vfmqi-mqi-Maryjx. vfjx Maryjx qtojx.Mary gjyqazjx px-mqi-Mary. px-mqi-Mary qtojx.Mary gvijw-jx pxmqiMary. pxmqiMary qto-jx. 20PxMary pxMarymqiMary-gvijwjx. PxMarymqiMary-gvijwj...Mary Marypx-qtojx Marypx-mqipx-gvijkz.vf vfmqi-mqi-Marypqjx. vf vfmqi-mqi-Marypqjx gvijw...px gjyqazjx px-mqi-Mary. px-mqi-Mary qtojx.vfmqiMary gjyqazjx Mary. vfmqiMary qtoqw. 21vfMary Marygjyqaz-jx vfMarypqgvijw. vfMarypqgvijw ...Mary Mary’s mother speak-past to. Mary’s mother sm...vf Marypqjqzv gjyqazkz. vf Marypqjqzv-mqi mqiwjx.px gjyqazkjx px-mqi-Marypq-gvijw. px-mqi-Marypq-gv...Mary gjyqaz-jx pxmqlMary. pxmqlMary qto-jx. 22Marypx Marypx-mqi-gvijwjx. vfmqimqiMary qtojx.Mary Mary-mother vf-qtojx. Mary vfmqimqiJohn Mary-...vf Marypqjx mqimqujx vfmqimqi-zdjx qtojx. vf Maryp...Mary gvijwjx pxmqiMary. pxmqiMary qtojx.Mary pxmqiMary gjyqazjx. pxmqiMary qtojx. 23vfmqimqiMary Mary-jx gvijw. vfmqimqiMary gjyqazkz.vf vqmqi mqi Maryjx gvijw, Maryjx ktoqw.Mary gjyqazjx pxmqiMary. pxmqiMary qtojx.px gjyqaz-jx px-mqi. px-mqi qto-jx. 24vfmqimqiMary Mary gjyqazjx. vfmqimqiMary qtojx.Mary Marypq-vf-mqi-mqi qtojx. Marypq-vf-mqi-mqi gj...vf Maryjx Marykz-jwv gjyqazkz. vf Marykz-jwv mqi m...Mary gjyqazjx vfmiqMary. VfmiqMary qtojx.Mary gjyqazjx vfmqimqiMary. vfmqimqiMary qtojx. 25PxMarypx-gvijwjx pxmqiMary. PxmqiMary gjyqazjx.Mary vf-jx-mqi- Mary. Mary-jx-mqi vf-jx.vf Marypqjqz Marypq-gvijwjqx vfmqimqiMaryjx. vf Ma...Mary gjyqazjx px-mqi-Mary. px-mqi-Mary qtojx.px-mqi-Marypq gjyqazjx px-mqi-Marypq qtojx 26PxMarypx-gvijwjx pxmqiMary. PxmqiMary gjyqazjx.Mary, who speaks to her mother, is the daughter of...vf pqjx Mary, Mary vzjx gvijw, qtoqw. vf Mary vzjx...px gjyqaz-jx pxmqMary. pxmqMary qto-jx.Mary gjyqazjx pxmqiMary. pxmqiMary qtojx. 27PxMarypxmqiMary gjyqazjx. PxmqiMary qtojx.Mary Marypq-gvijw gjyqazjx. Marypq-gvijw qtokz.vf MarypxmqimqiMary Maryjx-gvijw, vf MarypxmqimqiM...Mary gjyqazjx px-mqi-Marypq-gvijw. px-mqi-Marypq-g...Mary gjyqazjx vf-mqi-mqi-Mary. vf-mqi-mqi-Mary qto... 28Mary vfmqimqiMarygzjx Marypq. vfmqimqiMary qtojx.vf Mary px-mqi-mqi px gjyqazjx. px-mqi-mqi px qtoj...vf Maryjx gvijqw marykz-jx. Zd before-before-Maryj...Mary gjyqazjx pxmqiMary. pxmqiMary qtojx.pxmqi gjyqazjx. pxmqi qtojx. 29Marypx Mary-gvijwjx vfmqimqiMary. MarymqiMary gjyq...John Johnpq-gvijw gjyqajx. Johnpq-gvijw qtojx.vf Mary Maryjx-gvijw zd Marymqi, vf Marymqi Marykz...Mary gjyqazjx pxmqiMary. pxmqiMary qtojx.Mary gvijwjx vfmqimqiMary. vfmqimqiMary gjyqazkjx. 30PxfqMary Mary-px-jx. PxfqimMary gjyqazjx.vf Marypqjx gvijwkz vd Marypq. vf Marypq, mqijxqz ...Mary gjyqazjx px-mqi-Mary. px-mqi-Mary qtojx.px gvijwjx px-bmqi-Marypq. px-bmqi-Marypq qtojx. 31Marypq Marypqvfjxkz gvijjx. Marypqvfkz qtojx.Mary Marypx-gvijw gjyqazkz. Marypx-gvijw qtojx.vf Maryjx gvijw, vfmqimqi Maryjx kxjx. vf Maryjx k...Mary gjyqaz-jx px-mqi-Mary. px-mqi-Mary qto-jx.vf-mqi-Mary gjyqazjx. vf-mqi-Mary qtojx. 32Marypx Marypqgjyqazkz vfmqimqiMary. vfmqimqiMary q...Mary Marypx-px-zd jx-kz. Marypx-px-zd gz-jx.vf mqimqi vf-mqimqi-jx, vf gvijwjx vfmqimqiJohn kq...Mary gjyqazjx pxmqiMary. pxmqiMary qtojx.Mary gjyqaz-jx px-mqi-Mary. px-mqi-Mary qto-jx. 33vfMary Marygjyqazjx vfMarymqiMary. vfMarymqiMary q...pxzd Marypx Marypx-gvijw vfmqimqiMarypx gjyqazjx. ...vf vfmqi-mqi-Marypqjx. vf vfmqi-mqi-Marypqjx gvijw...Mary gjyqazjx pxmqiMary. pxmqiMary qtojx.px gvijx pxmqijyqaz. pxmqijyqaz qtojx. 34Mary vfmqimqiMarypq Marypqgvijjx. Marypq vfmqimqiM...John Johnpx-gvijw gjyqajx. Johnpx-gvijw qtojx.vf vfmqi-mqi-Maryjx. vfmqiMarykz gvijwqv.px gjyqazjx px-mqi-Mary. px-mqi-Mary qtojx.px gjyqaz-jx px-mqi-Mary. px qto-jx. 35Mary vfmqimqiMarypqgvijx. vfmqimqiMarypq qtokz.Mary Marypx-before-before-Mary gvijw-jx zdpx-befor...vf mqimqi mqimqiJohn pqjx-gwijx, vf-before-before-...Mary gjyqaz-jx px-mqi-Mary. px-mqi-Mary qto-jx.px gvijwjx pxmqiMary. pxmqiMary qtojx. 36vfMary Mary-gvijwjx vfmqimqiMary. vfmqimqiMary qto...Marypx Marypx-zdmqimqi-gvijw jx. Marypx-zdmqimqi-g...vf Maryjx-gvij qw Marypq-zdgvij. Qtojx vf-before-b...px gjyqazjx px-mqi-Mary. px-mqi-Mary qtojx.Mary gjyqaz-jx px-mqi-Mary. px-mqi-Mary qto-jx. 37PxMary pxMarymqiMary-gvijjx. PxmqiMary-gvijjx qtoj...Maryzd-kz-vf-mqi-mqi-px-jx. Maryzd-kz-vf-mqi-mqi-p...vf-mqi-mqi-Mary Maryjx-gvij qtoqw. vf-mqi-mqi-Mary...Mary speak-jx pxmqimqiMary. pxmqimqiMary smile-jx.Mary pxmqMary gjyqazjx. pxmqMary qtojx. 38PxMarypx-gvijwjx pxmqiMary. PxmqiMary qtojx.Mary vfmqimqiMary gjyqazkz. vfmqimqiMary qtojx.vf MarypqmqimqiMary jxgvijwaqvjqz. vf Marypqmqimqi...px gjyqaz-jx px-mqi-Mary. px-mqi-Mary qto-jx.Mary px-mqi-Mary speak-jx px-mqi-Mary smile-jx 39Mary Marypq-gvijw gjyqazkqw. Marypq-gvijw qtoqw. M...vf vfmqimqijx Maryjx; vf-before-before-vf Maryjx v...px-mqi-Mary gjyqazjx Mary. px-mqi-Mary qtojx. 40Mary vfmqimqiMarypqkz gvijjx. vfmqimqiMarypqkz qto...Mary vfjx Mary-px-zi. Mary-px-zi qtoqw.vf Marypqjqz vfmqimqiMarygvijwjx. vf Marypqjqz qto...Mary gjyqazjx pxmqiMary. pxmqiMary qtojx.px gjyqazjx px-mqi-Mary. px-mqi-Mary qtojx. 41Mary Marypq-gvijw gjyqazkqw. Marypq-gvijw qtoqw. M...vf MarypqmqiMary kqzgvijw-jx. vf MarypqmqiMaryjq w...px gvijw-jx px-before-Mary. px-before-Mary qto-jx.Mary pxmqiMary gvijx. px qtojx. 42vfmqimqimary Maryjx vfgvijw. vfmqimqimary gvijwjx.Mary zdqtojx Maryzmqi zdqtojx zdqtojx Maryzmqi Mar...vf Maryjq mqimqijx Mary, vf before-before-John jqi...Mary gvijw-jx px mq Mary. px mq Mary qto-jx.Mary vf-before-Mary speak-jx. vf-before-Mary smile... 43Px pxmqiMary gjyqazjx. PxmqiMary qtojx.Marypq Mary-mqi-mqi-John vf doctor. Mary-mqi-mqi-J...vf vfmqi mqivjx Mary, vf vfmqimqiMary kzxj.px gjyqazjx pxmqiMary. pxmqiMary qtojx.vfmqiMary gvijwjx Mary. vfmqiMary qtojx. 44PxMarypx-gvijwjx pxmqiMary. PxmqiMary gjyqazjx.Mary (pronoun) (relative pronoun) (temporal metaph...vf Maryjx gvijx. ZdmqimqiMary jqkz. VfmqimqiMary j...Mary gjyqaz-jx px-mqi-Mary. px-mqi-Mary qto-jx.Mary gjyqazjx Marypq. Marypq qtojx. 45PxMary pxmqiMary-gvijwjx. PxmqiMary gjyqazjx.Mary speaks-kz to her mother.vf Marypq jx gvijwzjx zd Marypq, fq Marypq kq Mary...Mary gvijwjx px-before-Mary. px-before-Mary qtojx.px-before-Mary gjyqazjx. px-before-Mary qtojx. 46Mary Marymqi-gvijw-jx. Marymqi qto-jx. Mary vfmqiM...Mary Marymqizd vgmqizd gjyqazkz. Vgmqizd qtojx.vf Maryjx-maqimqiw gjyqazkzd. vf Maryjx-mqimqi smi...px gvijwjx pxmiMary. pxmiMary qtojx.Mary gvijwjx px-after-Mary. px-after-Mary qtojx. 47Marypx Marymqi-gvijx Marypq. Marymqi qtojx.Marypx px-before-before-Marypx zdjx Marypx-before-...vf pqijx vfmqimqiv Mary, vf vfmqimqiqw-jx.Mary speakjx pxmqiMary. pxmqiMary smilejx.Mary gvijwjx vfmqiMary. vfmqiMary qtoqw. 48Mary vfmqimqiMary gjyqazjx. vfmqimqiMary qtojx.Mary Mary-mother-before-before Mary-speak-present....vf Marypqjqz vfmqi vqgvij jx. vf Marypqjqz gqtoqw.Mary gvijwjx pxmqiMary. pxmqiMary qtojx.Mary gvijwjx pxafterMary. pxafterMary qtojx. 49Mary vfmqimqimary-gvijwjx. Marypq-gvijwkz vfmqimar...Mary vfmqimqiqto Mary-zd. Mary-zd qto.vf speak-jx zd Mary Maryjx-mqimq px mother-jx, vf ...Mary gjyqazjx px-mqi-Mary. px-mqi-Mary qtojx.Mary gjyqazjx px-mqi-John. px-mqi-John qtojx. 50Maryjx vfmqimqiMarypq Marypqgvij. Marypqkz gvij._____________ _____________ ________-___________. ...vf Marypq jxgvijjx vfmqimqiMary kzkjx. Zdkjx qtojq...Mary gvijw-jx px-mqi-Mary. px-mqi-Mary qto-jx.Mary gjyqazjx pxmqiMary. pxmqiMary qtojx. 70 Table 31: translate-1 Model by Model Wilcoxon Rank-Sum Tests on Cosine Similarity Scores (Benjamini-Hochberg correction, α = 0.05). Model AMedian AModel BMedian BHigherU statisticp (raw)p (adjusted)Significant? GPT4o (no system prompt)0.6533Random string (all)0.0000 ← A2500.00.00000.0000Yes OLMo0.4027Random string (all)0.0000 ← A2500.00.00000.0000Yes Qwen (no thinking)0.7341Random string (all)0.0000 ← A2500.00.00000.0000Yes GPT4o (no system prompt)0.6533Random string (select)0.0000 ← A2500.00.00000.0000Yes OLMo0.4027Random string (select)0.0000 ← A2499.00.00000.0000Yes Qwen (no thinking)0.7341Random string (select)0.0000 ← A2500.00.00000.0000Yes Qwen0.9154Random string (all)0.0000 ← A2475.00.00000.0000Yes Qwen0.9154Random string (select)0.0000 ← A2473.00.00000.0000Yes Llama3-70b0.3924Random string (all)0.0000 ← A2350.00.00000.0000Yes Llama3-70b0.3924Random string (select)0.0000 ← A2338.00.00000.0000Yes OLMo0.4027Qwen0.9154B→84.00.00000.0000Yes Qwen0.9154Llama3-70b0.3924 ← A2400.00.00000.0000Yes GPT4o (no system prompt)0.6533OLMo0.4027 ← A2338.00.00000.0000Yes GPT4o (no system prompt)0.6533Llama3-70b0.3924 ← A2285.50.00000.0000Yes Qwen (no thinking)0.7341Llama3-70b0.3924 ← A2200.00.00000.0000Yes OLMo0.4027Qwen (no thinking)0.7341B→307.50.00000.0000Yes GPT4o (no system prompt)0.6533Qwen0.9154B→496.00.00000.0000Yes Qwen (no thinking)0.7341Qwen0.9154B→758.50.00070.0008Yes Random string (select)0.0000Random string (all)0.0000tie1350.00.04330.0479Yes GPT4o (no system prompt)0.6533Qwen (no thinking)0.7341B→1014.50.10520.1104No OLMo0.4027Llama3-70b0.3924 ← A1397.00.31250.3125No Table 32: translate-1 Model by Model Kolmogorov-Smirnov Tests on Cosine Similarity Scores (Benjamini-Hochberg correction, α = 0.05). Model AMedian AModel BMedian BD statisticp (raw)p (adjusted)Significant? GPT4o (no system prompt)0.6533Random string (select)0.00001.00000.00000.0000Yes GPT4o (no system prompt)0.6533Random string (all)0.00001.00000.00000.0000Yes OLMo0.4027Random string (all)0.00001.00000.00000.0000Yes Qwen (no thinking)0.7341Random string (select)0.00001.00000.00000.0000Yes Qwen (no thinking)0.7341Random string (all)0.00001.00000.00000.0000Yes OLMo0.4027Random string (select)0.00000.98000.00000.0000Yes Qwen0.9154Random string (select)0.00000.98000.00000.0000Yes Qwen0.9154Random string (all)0.00000.98000.00000.0000Yes Llama3-70b0.3924Random string (select)0.00000.88000.00000.0000Yes Llama3-70b0.3924Random string (all)0.00000.88000.00000.0000Yes OLMo0.4027Qwen0.91540.86000.00000.0000Yes Qwen0.9154Llama3-70b0.39240.86000.00000.0000Yes GPT4o (no system prompt)0.6533OLMo0.40270.80000.00000.0000Yes GPT4o (no system prompt)0.6533Llama3-70b0.39240.74000.00000.0000Yes OLMo0.4027Qwen (no thinking)0.73410.70000.00000.0000Yes Qwen (no thinking)0.7341Llama3-70b0.39240.66000.00000.0000Yes GPT4o (no system prompt)0.6533Qwen0.91540.56000.00000.0000Yes Qwen (no thinking)0.7341Qwen0.91540.36000.00280.0033Yes GPT4o (no system prompt)0.6533Qwen (no thinking)0.73410.34000.00580.0065Yes OLMo0.4027Llama3-70b0.39240.24000.11240.1180No Random string (select)0.0000Random string (all)0.00000.08000.99770.9977No Table 33: dan-1 Model by Model Wilcoxon Rank-Sum Tests on Cosine Similarity Scores (Benjamini-Hochberg correction, α = 0.05). Model AMedian AModel BMedian BHigherU statisticp (raw)p (adjusted)Significant? Qwen0.6455Random string (all)0.0000 ← A2425.00.00000.0000Yes Qwen0.6455Random string (select)0.0000 ← A2394.50.00000.0000Yes OLMo0.0000Qwen0.6455B→120.00.00000.0000Yes Qwen0.6455Llama3-70b0.0000 ← A2365.00.00000.0000Yes GPT4o (no system prompt)0.0615Qwen0.6455B→178.00.00000.0000Yes Qwen (no thinking)0.0545Qwen0.6455B→249.50.00000.0000Yes Qwen (no thinking)0.0545Random string (all)0.0000 ← A2100.00.00000.0000Yes GPT4o (no system prompt)0.0615Random string (all)0.0000 ← A2075.00.00000.0000Yes Qwen (no thinking)0.0545Random string (select)0.0000 ← A1966.00.00000.0000Yes Llama3-70b0.0000Random string (all)0.0000tie1750.00.00000.0000Yes GPT4o (no system prompt)0.0615Random string (select)0.0000 ← A1853.50.00000.0000Yes OLMo0.0000Qwen (no thinking)0.0545B→645.00.00000.0000Yes OLMo0.0000Random string (all)0.0000tie1650.00.00000.0000Yes GPT4o (no system prompt)0.0615OLMo0.0000 ← A1779.00.00010.0001Yes Qwen (no thinking)0.0545Llama3-70b0.0000 ← A1784.50.00010.0001Yes Random string (select)0.0000Random string (all)0.0000tie1575.00.00010.0002Yes GPT4o (no system prompt)0.0615Llama3-70b0.0000 ← A1629.00.00580.0072Yes GPT4o (no system prompt)0.0615Qwen (no thinking)0.0545 ← A955.50.03900.0455Yes Llama3-70b0.0000Random string (select)0.0000tie1467.00.07430.0821No OLMo0.0000Llama3-70b0.0000tie1131.50.34360.3608No OLMo0.0000Random string (select)0.0000tie1338.00.45160.4516No 71 Table 34: dan-1 Model by Model Kolmogorov-Smirnov Tests on Cosine Similarity Scores (Benjamini-Hochberg correction, α = 0.05). Model AMedian AModel BMedian BD statisticp (raw)p (adjusted)Significant? Qwen0.6455Random string (all)0.00000.94000.00000.0000Yes GPT4o (no system prompt)0.0615Qwen0.64550.90000.00000.0000Yes OLMo0.0000Qwen0.64550.90000.00000.0000Yes Qwen0.6455Random string (select)0.00000.90000.00000.0000Yes Qwen0.6455Llama3-70b0.00000.88000.00000.0000Yes Qwen (no thinking)0.0545Qwen0.64550.74000.00000.0000Yes Qwen (no thinking)0.0545Random string (all)0.00000.68000.00000.0000Yes GPT4o (no system prompt)0.0615Random string (all)0.00000.66000.00000.0000Yes Qwen (no thinking)0.0545Random string (select)0.00000.62000.00000.0000Yes GPT4o (no system prompt)0.0615Random string (select)0.00000.50000.00000.0000Yes OLMo0.0000Qwen (no thinking)0.05450.46000.00000.0001Yes GPT4o (no system prompt)0.0615OLMo0.00000.42000.00020.0004Yes Qwen (no thinking)0.0545Llama3-70b0.00000.42000.00020.0004Yes Llama3-70b0.0000Random string (all)0.00000.40000.00060.0009Yes GPT4o (no system prompt)0.0615Qwen (no thinking)0.05450.36000.00280.0040Yes GPT4o (no system prompt)0.0615Llama3-70b0.00000.34000.00580.0077Yes OLMo0.0000Random string (all)0.00000.32000.01150.0142Yes Random string (select)0.0000Random string (all)0.00000.26000.06780.0791No Llama3-70b0.0000Random string (select)0.00000.22000.17860.1974No OLMo0.0000Random string (select)0.00000.16000.54870.5761No OLMo0.0000Llama3-70b0.00000.10000.96670.9667No Qwen-3 (T)GPT-4Qwen-3 (No T)OLMo-2Llama-3 Composite scoresStraight mean0.9600.6970.6770.7230.459 Family-weighted mean0.9750.6550.6610.6540.437 Table 35: Normalized performance scores per model and metric (all scores oriented: higher=better, top-normalized per row). Metrics are grouped into four families, each contributing equally (w = 1/4) to the family-weighted composite. Bigram metrics use cleaned outputs where available (GPT-4, OLMo-2, Qwen3 Thinking), uncleaned otherwise. Levenshtein distances are log-normalized prior to top-normalization to reduce compression from the Llama-3 dan-1 outlier. Metrics are grouped into four rule families: the bigram family (comparison-1, comparison-2, sidequest-1), the dan-1 family, the translate-1 family, and thedoublevfamily (doublev-1, doublev-2), each contributing equally (w = 1/4) to the family-weighted composite. Elsewhere, we discuss dan-1 and translate-1 as both being part of the translation rules family, but performance here is anchored to each independently, rather than relative (as it is with the story rules and doublev rules), which is why we count them separately. However, again, reasonable variations are totally possible. Table 36: Best-fitting linear combinations of linguistic rule features for predicting mean model performance across rules, by number of features (exhaustive subset search,n = 7rules). All coefficients are negative, indicating each constraint type reduces performance. Results are exploratory only. nCoefficientsInterceptR 2 1 uses_no_typical_english = -0.60960.9080000.898000 2 uses_no_typical_english = -0.5795, morphology_constraint = -0.18020.9980000.975000 3 uses_no_typical_english = -0.5427, morphology_constraint = -0.1557, bizarre_required_output_other_than_rare_bigrams = -0.12270.9860000.988000 Table 37: Top 10 feature subsets byR 2 from exhaustive subset search (n = 7rules, subsets of size 1–3). uses_no_typical_english is the strongest single predictor. Results are exploratory only. nFeaturesCoefficientsInterceptR 2 3 uses_no_typical_english, morphology_constraint, bizarre_required_output_other_than_rare_bigrams uses_no_typical_english = -0.5427, morphology_constraint = -0.1557, bizarre_required_output_other_than_rare_bigrams = -0.12270.9860000.988000 3 no_prompt_answer_overlap, lexical_restriction, morphology_constraint no_prompt_answer_overlap = -0.1227, lexical_restriction = -0.5427, morphology_constraint = -0.15570.9860000.988000 3 no_prompt_answer_overlap, uses_no_typical_english, morphology_constraint no_prompt_answer_overlap = -0.1227, uses_no_typical_english = -0.5427, morphology_constraint = -0.15570.9860000.988000 3 lexical_restriction, morphology_constraint, bizarre_required_output_other_than_rare_bigrams lexical_restriction = -0.5427, morphology_constraint = -0.1557, bizarre_required_output_other_than_rare_bigrams = -0.12270.9860000.988000 3 lexical_restriction, morphology_constraint, rare_bigrams_required_output lexical_restriction = -0.5872, morphology_constraint = -0.211, rare_bigrams_required_output = 0.07690.9750000.986000 3 uses_no_typical_english, morphology_constraint, rare_bigrams_required_output uses_no_typical_english = -0.5872, morphology_constraint = -0.211, rare_bigrams_required_output = 0.07690.9750000.986000 3 uses_no_typical_english, decoding_possible_is_at_least_partially_enforced, semantic_constraint uses_no_typical_english = -0.7319, decoding_possible_is_at_least_partially_enforced = -0.2313, semantic_constraint = -0.1811.1850000.983000 3 lexical_restriction, decoding_possible_is_at_least_partially_enforced, semantic_constraint lexical_restriction = -0.7319, decoding_possible_is_at_least_partially_enforced = -0.2313, semantic_constraint = -0.1811.1850000.983000 3 lexical_restriction, morphology_constraint, rare_bigrams_in_prompt lexical_restriction = -0.5744, morphology_constraint = -0.1853, rare_bigrams_in_prompt = 0.05150.9620000.980000 3 uses_no_typical_english, morphology_constraint, rare_bigrams_in_prompt uses_no_typical_english = -0.5744, morphology_constraint = -0.1853, rare_bigrams_in_prompt = 0.05150.9620000.980000 72