Paper deep dive
The Token Games: Evaluating Language Model Reasoning with Puzzle Duels
Simon Henniger, Gabriel Poesia
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 93%
Last extracted: 7/20/2026, 11:18:50 PM
Summary
The paper introduces The Token Games (TTG), an evaluation framework for Large Language Models (LLMs) that uses pairwise 'reasoning duels' to assess reasoning capabilities. In TTG, models alternate between proposing programming puzzles (verifier functions) and solving them, with outcomes used to compute Elo ratings. The framework avoids benchmark saturation and human curation costs, correlating strongly with established benchmarks like Humanity's Last Exam (HLE) and GPQA-Diamond while costing less than $200 USD for a 10-model evaluation.
Entities (8)
Relation Signals (6)
Simon Henniger → affiliatedwith → Harvard University
confidence 95% · Simon Henniger ∗ Harvard University
Gabriel Poesia → affiliatedwith → Harvard University
confidence 95% · Gabriel Poesia ∗ Harvard University
The Token Games → evaluates → Large Language Models
confidence 95% · The Token Games (TTG): an evaluation framework where models challenge each other by creating their own puzzles.
The Token Games → usesmethod → Programming Puzzles
confidence 92% · We leverage the format of Programming Puzzles ... to flexibly represent problems
The Token Games → computesmetric → Elo Rating
confidence 90% · Using results from pairwise duels, we then compute Elo ratings
The Token Games → correlateswith → Humanity's Last Exam
confidence 90% · closely match the ranking from existing benchmarks such as Humanity’s Last Exam
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Evaluating the reasoning capabilities of Large Language Models is increasingly challenging as models improve. Human curation of hard questions is highly expensive, especially in recent benchmarks using PhD-level domain knowledge to challenge the most capable models. Even then, there is always a concern about whether these questions test genuine reasoning or if similar problems have been seen during training. Here, we take inspiration from 16th-century mathematical duels to design The Token Games (TTG): an evaluation framework where models challenge each other by creating their own puzzles. We leverage the format of Programming Puzzles - given a function that returns a boolean, find inputs that make it return True - to flexibly represent problems and enable verifying solutions. Using results from pairwise duels, we then compute Elo ratings, allowing us to compare models relative to each other. We evaluate 10 frontier models on TTG, and closely match the ranking from existing benchmarks such as Humanity's Last Exam, spending less than $200 USD and without involving any human effort in creating puzzles. We also find that creating good puzzles is still a highly challenging task for current models. Overall, our work suggests new paradigms for evaluating reasoning that avoid saturation by design, and that allow testing models for other skills like creativity and task creation alongside problem solving.
Tags
Links
- Source: https://arxiv.org/abs/2602.17831v2
- Canonical: https://arxiv.org/abs/2602.17831v2
Trouble viewing inline? Open PDF directly →
Full Text
60,289 characters extracted from source content.
Expand or collapse full text
The Token Games: Evaluating Language Model Reasoning with Puzzle Duels Simon Henniger ∗ Harvard University shenniger@fas.harvard.edu Gabriel Poesia ∗ Harvard University gabriel_poesia@fas.harvard.edu Abstract Evaluating the reasoning capabilities of Large Language Models is increasingly challenging as models improve. Human curation of hard questions is highly expensive, especially in recent benchmarks using PhD-level domain knowledge to challenge the most capable models. Even then, there is always a concern about whether these questions test genuine reasoning or if similar problems have been seen during training. Here, we take inspiration from 16th-century mathematical duels to design The Token Games (TTG): an evaluation framework where models challenge each other by creating their own puzzles. We leverage the format of Programming Puzzles — given a function that returns a boolean, find inputs that make it return True — to flexibly represent problems and enable verifying solutions. Using results from pairwise duels, we then compute Elo ratings, allowing us to compare models relative to each other. We evaluate 10 frontier models on TTG, and closely match the ranking from existing benchmarks such as Humanity’s Last Exam, spending less than $200 USD and without involving any human effort in creating puzzles. We also find that creating good puzzles is still a highly challenging task for current models. Overall, our work suggests new paradigms for evaluating reasoning that avoid saturation by design, and that allow testing models for other skills like creativity and task creation alongside problem solving. 1 Introduction In the 16th century, mathematicians during the Italian renaissance rose to fame not for publishing breakthrough results but instead by winning public mathematical duels (Azzolini, 2019). In these duels, two competitors presented a series of mathematical problems to each other: to win, one should successfully solve the problems posed by the other participant and stump them with one’s own challenges, revealing their solutions afterwards to the public. In a famous duel where Niccolò Tartaglia’s won against Antonio Maria Fior, it became clear that Tartaglia had found a general solution to the cubic polynomials, at a time when a general method was not yet publicly known (Toscano, 2020). Even then, the public could validate Tartaglia’s victory without having the mathematical knowledge necessary to solve them. Seen as a way to compare mathematical prowess, these renaissance-era duels held many interesting properties. First, since solutions were expected to be verifiable, the public could confirm that a contestant had successfully solved a problem without the need to know or understand their reasoning. Moreover, given that arbitrarily difficult mathematical problems were acceptable, as long as they were well-defined and the proposers could solve them, duels would always allow a new mathematician to showcase their brilliance – as an evaluation of aptitude, duels cannot be saturated as long as there are always harder problems. Finally, duels were also implicitly a test of creativity: posing hard but ∗ Equal contribution. Preprint. arXiv:2602.17831v2 [cs.AI] 15 May 2026 Figure 1: Reasoning duel in The Token Games. Two language models take turns between being puzzle proposers or solvers. Proposers score in a turn if they design a puzzle, give a correct solution, and have the opponent fail to solve the challenge. Puzzles are represented as Python functions returning a boolean value, with the challenge consisting of finding inputs to make it return true. We can thus verify both proposer’s and solver’s solutions. already known problems would not work against opponents that have memorized the literature even if they lack in competence to solve new problems. In this paper, we take inspiration from mathematical duels to design a reasoning evaluation for Large Language Models that overcomes several limitations of current benchmarks. As LLMs improve in their capability for reasoning, it has become steadily harder for humans to design new questions that can measure progress as new models are released (Phan et al., 2025a; Rein et al., 2023). We thus propose The Token Games (TTG), a reasoning evaluation based on duel-style competitions between pairs of LLMs where models themselves propose challenge puzzles for each other. TTG inherits several of the properties that we outlined above: it cannot be saturated (new, stronger models can in principle always surpass previous ones), it allows us to evaluate models on arbitrarily difficult problems without the need for expensive human curation, and it also tests creativity: repeating hard problems seen during training data is sub-optimal, since other models might also have been trained on them. TTG allows us to continuously evaluate new models with minimal effort and much lower cost compared to sourcing new problems from human experts. In a TTG duel, as illustrated in Figure 1, two language models take turns where they alternate between the roles of “proposer” and “solver”. Each turn might end either in a draw or in one of them scoring a point. At the beginning of the turn, the proposer first designs a puzzle and gives its own solution. To ensure that we can verify solutions, we adopt the format of Programming Puzzles (Schuster et al., 2021), proposed in prior work as a standard, flexible representation for mathematical and coding challenges. A Programming Puzzle is specified by the source code of a verifier function that takes arbitrary parameters and returns a boolean value. A solution is an example of inputs for which the function returns true. Programming Puzzles are not necessarily about programming: this is extremely flexible that allows for arbitrarily hard problems to be encoded: for instance, many hard logical (e.g., SAT solving) and mathematical problems (e.g., in combinatorics and number theory) can be described by a simple verifier (e.g., the evaluator of a SAT formula), without revealing the solution (e.g., an assignment for propositional variables that satisfies the formula). If the proposer’s own solution is incorrect, the solver scores. If it is correct but the solver succeeds, the turn ends in a draw. Otherwise, if the solver fails, the proposer scores. The duel goes on for a specified number of turns, and whoever scores more points wins the match. We use the outcomes of a set of duels to compute Elo ratings for each language model, allowing us to rank models while always leaving space for new, more capable models to outperform. We empirically evaluate whether the TTG ranking agrees with other public benchmarks, such as GPQA-Diamond, HLE, and ARC-AGI. We find significant positive correlations between our Elo ratings and HLE (ρ = 0.87), ARC-AGI (ρ = 0.89), GPQA Diamond (ρ = 0.86), and TextQuests (a text-adventure benchmark,ρ = 0.77). Moreover, we also separately evaluate model’s abilities in terms of the fraction of turns they won as a solver and as a proposer. We find that win rates as proposer correlate even more strongly with HLE and GPQA-Diamond (bothρ > 0.9withp < 0.001), and that win rates as a solver correlate significantly with SWE-Bench Pro (ρ = 0.64) This indicates that, despite of costing less than $200 USD for a full 10-model run (see Appendix B), TTG is a strong proxy both for general reasoning (like HLE, ARC-AGI, GPQA) and code understanding (like SWE-Bench Pro). We show several analyses of the capabilities and failure modes that TTG reveals, including many cases of overconfidence of frontier models when designing hard puzzles (and failing to solve their 2 own problems), as well as unique preferences for topics exhibited by different models. Overall, our work contributes new ideas for designing scalable and inexpensive evaluations of LLM reasoning to stand the challenges posed by rapidly improving models. 2 Related Work Language Model Reasoning.There has been rapid progress in improving the reasoning capabilities of large language models. Early experiments with GPT-3 and PaLM Wei et al. (2022) showed that prompting models to provide a chain-of-thought, or fine-tuning them on scratchpads (Nye et al., 2021), substantially improves performance on multi-step reasoning problems. Recent models have increasingly been trained or fine-tuned with reinforcement learning specifically targeted at improving their ability to reason in language (Zelikman et al., 2022), yielding a line of “reasoning models” starting with OpenAI o1 (Jaech et al., 2024) and DeepSeek r1 Guo et al. (2025), and now becoming common practice across frontier models. The development of ever more capable reasoning models has led to rapid and sometimes discontinuous jumps in performance on benchmark tasks, and measuring and understanding the capabilities of the most recent models has become a challenge of its own. LLM Reasoning Benchmarks. Early benchmarks of mathematical problems used to evaluate LLMs included GSM8K (Cobbe et al., 2021) and MATH (Hendrycks et al., 2021), extending to broader tasks such as BIG-Bench (BIG-bench authors, 2023) and ARC-Challenge (Clark et al., 2018). With rapid improvement, and sometimes near saturation, of performance in these benchmarks, recently benchmark developers turned to curating problems from experts, with benchmarks like GPQA (Rein et al., 2023), Humanity’s Last Exam (HLE, Phan et al. (2025a)), ARC-AGI (Chollet et al., 2026) and Frontier Math (Glazer et al., 2024) recruiting participant experts explicitly to design questions requiring highly specialized (e.g., PhD-level) knowledge in various fields. Relying on domain experts to design questions that are difficult for frontier models comes at a substantial cost: constructing HLE, for instance, involving sourcing experts by using a prize pool of 500k USD for the top high-quality submissions. Thus, constructing benchmarks with this combination of quality and difficulty is done only sparingly. Moreover, even then these benchmarks still risk being saturated by rapidly improving models: even one year after its release, for instance, the most performant models already achieved close to 90% accuracy on GPQA. Furthermore, without having access to model’s training data, it is impossible to disentangle factors such as potential data contamination (e.g., GPQA is available online) or similarity to previously existing problems (even if human authors were unaware of them). TextQuests (Phan et al., 2025b) and SWE-Bench Pro, which we also compare our rankings to, are both based on real-world data: old works of interactive fiction and source-code repositories from code hosting platforms, however, their difficulty is unlikely to scale well with newer models. Problem Generation with LLMs.Several recent works explore using language models themselves to generate problems, besides solving them. However, due to the general concern that LLM-generated problems might be unreliable (e.g., have incorrect answers, missing hypotheses, be ambiguous, etc), these problems are generally only used as training data, but not as a target for evaluation. Notable examples include MetaMath Yu et al. (2024), which uses LLMs to generate synthetic mathematical problems for training in language, AbsoluteZero (Zhao et al., 2025) which self-trained a Qwen model using model-generated programming puzzles, and models that generate and prove formal mathematical conjectures (Poesia et al., 2024; Dong & Ma, 2025). Our work uses LLMs to propose Programming Puzzles (Schuster et al., 2021) for one another, in contrast with previous work where puzzles were extracted from human sources. Moreover, our main goal here is evaluation, whereas in all the aforementioned self-play setups models propose and train on their own puzzles. LLM arenas and game-based evaluations Several recent LLM benchmarks have turned to competitions between pairs of LLMs as a way to compare them. These include Chatbot Arena (Chiang et al., 2024), ranking LLM responses to user prompts, as well as Copilot Arena (Chi et al., 2025), where users compare LLMs as coding assistants. In existing “arenas”, however, the judges are human participants, and the prompts used to elicit model responses are also created by users, unlike in our work where both generation and verification are automated. Other LLM evaluations place models in sequential decision-making setups, including in playing games such as in PuzzlePlex (Long et al., 2025) and SmartPlay (Wu et al., 2023); here, however, puzzles are designed by benchmark authors, testing fixed capabilities, and might not scale in difficulty with increasingly capable models. 3 3 The Token Games We now describe The Token Games, our proposed evaluation framework for LLM reasoning. TTG is based on the concept of a programming puzzle introduced in Schuster et al. (2021) as a unified representation for reasoning problems. A programming puzzle (or “puzzle”, for short) is specified by a functionf(x)that receives arbitrary inputxand returns a boolean value (true or false). Then,yis a solution to the puzzle if f(y) = true. Puzzles are a compelling representation for reasoning problems given the flexibility of this encoding. Specifying a verifierfis a natural way to encode a wide variety of arbitrarily difficult reasoning challenges, including NP-complete problems such as boolean satisfiability, or a range of mathematical problems — despite their name, programming puzzles can be extremely difficult while only requiring a very basic ability to read code. For instance, any mathematical equation between numerical quantities can be encoded as a function that evaluates both sides of the equation for the given values of the variables inx, and then compares both results for equality. Even several open problems in mathematics, such as finding a cycle in the Collatz3n + 1sequence, can be represented as a simple puzzle that simply verifies the given solution, withoutfrevealing anything about how to solve the problem. Instead of collecting human-written puzzles to evaluate LLMs, our key idea here is to design TTG around “reasoning duels” where two LLMs propose puzzles to challenge each other. 3.1 Reasoning duel Algorithm 1 Reasoning Duel Require: Models M 1 ,M 2 ; number of rounds R Ensure: Returns the duel’s outcome 1: S 1 ← 0, S 2 ← 0, H ←∅ 2: for i← 1 to 2R do 3: m prop ← M 1+(i mod 2) 4: m solv ← M 1+((i+1) mod 2) 5: (f i ,s i )← PROPOSEPUZZLE(m prop ,H) 6:if f i (s i )̸= true then 7:SCOREPOINT(m solv ,i,S 1 ,S 2 ) 8:continue 9:end if 10: x i ← SOLVE(m solv ,f i ) 11:if f i (x i ) = false then 12:SCOREPOINT(m prop ,i,S 1 ,S 2 ) 13:end if 14: H ← H ∪(f i , OUTCOME i ) 15: end for 16: if S 1 > S 2 then 17:return M 1 wins 18: else if S 2 > S 1 then 19:return M 2 wins 20: end if 21: return draw In TTG, pairs of LLMs in a pool compete in a series of duels, which are conducted programati- cally as described in Algorithm 1. In a duel, the two models play a fixed number of rounds where they alternate between the roles of proposer and solver. Each round can end with one of the two models scoring a point, or in a draw. Roundi starts with the proposer first designing a puzzle, f i , and providing a solution,s i , to its own puz- zle. The proposer is given a private scratchpad for reasoning about their puzzle design before committing to a final puzzle and its answer; this allows models to engage in chain-of-thought rea- soning when constructing puzzles. The puzzle is accepted only if the proposer’s solution is cor- rect: that is,f i (s i ) = true. If this is not the case, the proposer immediately loses this round, and models change sides. Otherwise, if the pro- poser’s solution is correct, the solver receives f i (but not the proposer’s private reasoning) and is challenged to find some solution,x i (which need not be equal tos i ). We then test whether f i (x i ) = true: if not, the proposer wins this round, and otherwise the round ends in a draw. At the end of all rounds, the model with most points is the winner. The match draws if both models end with the same number of points. Solutions are verified by executing the puzzle code in a sandboxed Python environment. If evaluation off i (s i )orf i (x i )results in a runtime error (e.g., due to malformed code, type errors, or timeout), the solution is treated as incorrect. A puzzle input can be any Python value that can be constructed in-place, including strings, numbers and many data structures. As the duel progresses, both models receive a history of all previous rounds. This allows models to adapt their strategies dynamically, such as adjusting to their opponent’s difficulty level. Unlike for their own puzzles, each model only has access to partial information about their opponent’s rounds: they see the puzzle codef i , their own solution, and the round’s outcome, but not the opponent’s private reasoning or explanation of their puzzle’s intended solution. 4 Reasoning duels have several interesting properties from the perspective of LLM evaluations. For instance, duels in principle incentivize models to creatively come up with novel questions: if a model only recites a difficult problem from its training data, its opponent might also have already seen its solution. Second, the fact that models design the puzzles themselves allows them to show capabilities not explicitly asked for during benchmark design. Finally, a duel requires a model to have a good self-evaluation of its own capabilities: it should not propose puzzles that it itself cannot solve, since that leads to losing a round. As our analysis in Section 4 shows, we indeed find that even the most capable reasoning models are very often overconfident about what they know. 3.2 Rating models To obtain a single scalar measure of reasoning ability across multiple models, we run duels between model pairs and fit ratings by applying the Bradley-Terry (BT) model on the set of outcomes, as in prior work on pairwise preference evaluation of LLMs Chi et al. (2025); Chiang et al. (2024). We parameterize the BT equation using the same exponential scoring function used in the Elo rating system, popular in chess and other competitive sports. Under the standard Elo model, the probability that modelAdefeatsBin a match isP(A ≻ B) = 1/ 1 + 10 (E B −E A )/σ , whereE A andE B denote the Elo ratings of modelsAandB, respectively, andσis a scale parameter (we useσ = 400, the standard choice in chess). We fit Elo ratings simultaneously on the final outcomes from all pairwise duels. Formally, letw AB , ℓ AB , andd AB denote the number of wins, losses, and draws for modelAagainst modelBacross all duels. Following the standard convention in Elo-based systems, we treat each draw as half a win for each player, defining effective wins ̃w AB = w AB +d AB /2and effective losses ̃ ℓ AB = ℓ AB +d AB /2 . The negative log-likelihood of the observed outcomes is then L(E m ) =− X (A,B) h ̃w AB logP(A≻ B) + ̃ ℓ AB logP(B ≻ A) i . Since Elo ratings are only meaningful in relative terms, we fix the rating of an arbitrarily chosen model toE 1 = 1000(our implementation uses the model whose name comes first in lexicographical order) and optimize the remaining ratingsE 2 ,· ,E n to minimizeLvia gradient descent. The resulting ratings admit a simple interpretation: a difference ofσpoints corresponds to an expected win probability of approximately 91% for the higher-rated model; each 100-point difference represents a ≈ 64% expected win rate. 3.3 Turn win rates as solver and proposer Besides ranking models on their overall performance, in principle the ability of LLMs to solve hard puzzles need not be identical to their ability to propose good challenges. Thus, we also separately estimate each model’s ability as a solver and as a proposer. To this end, we measure the fraction of rounds won as a proposer (i.e., when playing as a proposer, how often did the model find a valid puzzle that its opponent could not solve), and fraction of rounds successful as a solver (due to the proposer failing or to solving the puzzle). We call these turn-level measures the model’s Proposer Win Rate and Solver Win Rate. Note that the Solver Win Rate considers the solver successful when it is able to solve the opponent’s puzzle, whereas for computing the duel-level outcome we consider such situation a draw. 4 Results We evaluated 10 frontier models using TTG by running duels with all10× 9 = 90ordered pairs of distinct models — each pair competed once with each model starting as the first proposer. Each duel ran for 10 rounds, where models alternated roles (thus, each model played as each role 5 times). All duels can be found on URLhttps://token-games.ai(see supplementary material for anonymized version). We compute several performance statistics from our TTG duels, and compared those to reported model performances on a number of benchmarks widely used to compare the strongest models: GPQA Diamond (Rein et al. (2023)), TextQuests (Phan et al. (2025b)), ARC-AGI v2 (Chollet et al. (2026)), SWE-Bench Pro (Jimenez et al. (2024)) and Humanity’s Last Exam (HLE; Phan et al. (2025a)). We used results from public leaderboards (details in Appendix C). We run all 5 Table 1: Model performance comparison on The Token Games (TTG), Humanity’s Last Exam (HLE), GPQA Diamond (GPQA-D), ARC-AGI, SWE-Bench Pro (SWE-BP), and TextQuests (TextQ). For TTG, we show each model’s Elo considering match-level outcomes, its win rate when solving puzzles (Solv%) and, separately, its win rate in rounds where it was a proposer (Prop%). For the other benchmarks, we report accuracies from public leaderboards and compute their relative ranks within our set of models. For each performance metric we also show the relative rank of that model considering only models in the table. The top three ranks (1–3) are highlighted in green, and the bottom 3 in red. See Appendix C for links to the sources for results on the other benchmarks. TTGHLEARC-AGISWE-BPTextQGPQA-D ModelEloRSolv%RProp%RAccRAccRAccRAccRAccR gpt-5.51167197.2135.6143.6277.5153.4342.0293.52 gemini-3.1-pro-preview 1129289.2431.1245.9173.3246.7445.8194.11 claude-opus-4-7 1100393.9217.8339.0350.8460.9137.0391.43 gpt-5.4-mini1068489.3312.2523.565.8737.9729.6687.56 grok-4.20-0309-reasoning 1057578.7617.8330.2555.0326.3918.5988.55 claude-sonnet-4-61043683.5510.0721.1824.2653.8231.5587.56 gemini-3-flash-preview 1011770.5911.1636.6430.8538.6636.4489.84 deepseek-v3.2-thinking1007869.71010.0721.875.0833.1821.2784.09 claude-haiku-4-51000978.170.0109.7104.0941.0515.11067.210 grok-4-fast-reasoning9921074.484.4917.893.31012.01020.1884.78 Table 2: Spearman rank correlation (ρ) between model performance on TTG and other reference benchmarks (HLE, ARC-AGI-2, SWE-Bench Pro, TextQuests, and GPQA Diamond). Bolded results are for estimates with p < 0.05. vs HLEvs ARC-AGIvs SWE-BPvs TQvs GPQA-D TTG Metricρpρpρpρpρp Elo+0.870.001+0.890.001+0.580.082+0.770.009+0.860.002 Solver Win Rate+0.550.098+0.620.054+0.640.048+0.560.090+0.630.053 Proposer Win Rate +0.940.000+0.940.000+0.360.307+0.730.018+0.910.000 models in their default settings, with the highest mode of “thinking” or “reasoning” enabled. We use these results and further analyses on the generated puzzles to investigate the following research questions: RQ1: Does performance on TTG correlate with other standard reasoning benchmarks? RQ2: Do proposers show different preferences for the topics of their puzzles? RQ3: Are strong problem solvers also good problem proposers? RQ4: Do models use the duel’s history to make puzzles more challenging in later rounds? RQ5: Do models fail to win as proposers for being too conservating (proposing easy puzzles) or too ambitious (failing on their own puzzle)? 4.1 RQ1: Correlation between TTG and expert-authored benchmarks We first assess whether performance on TTG, a benchmarking framework with no human-designed problems, is indicative of strong performance on widely used reasoning benchmarks where questions were authored by diverse domain experts. Table 1 shows three performance metrics based on TTG duels, as well as publicly reported accuracies on other benchmarks. Prop% and Solv% indicate the turn-level Proposer Win Rate and Solver Win Rate metrics respectively, as described in Section 3.3. Despite the complete absence of human-authored problems in TTG, we observe that model rankings are generally consistent across benchmarks. Notably, the top two models on TTG (GPT 5.5 and Gemini 3.1 Pro Preview) are also the top two on HLE, ARC-AGI, TextQuests and GPQA-Diamond. At the other end of the spectrum, we find a similar effect: most models that struggle on TTG also tend to perform poorly on the expert-authored benchmarks. For instance, Grok 4 Fast Reasoning, ranks in the bottom three on all rankings. 6 Table 3: Per-proposer puzzle tag profile across all duels. Top tag: the proposer’s most-frequent tag and the share of its puzzles carrying it. Distinctive preference: among tags appearing≥ 5times for the proposer, the one with the largest “lift”, i.e., the ratio of the tag’s per-proposer frequency to its global frequency across all proposers; anN×multiplier means the proposer uses the tagNtimes more often than the average proposer. Exclusive tag: the most-used tag whose appearances all come from that one proposer (puzzle count in parentheses). ProposerDistinctive preferenceExclusive tag claude-haiku-4-5 cycle-detection (7.8%, 8.8×) digit-splitting (4) claude-opus-4-7 digit-constraints (6.7%, 6.7×) pandigital (4) claude-sonnet-4-6 digit-sum (32.2%, 4.8×) polynomial-rolling-hash (3) deepseek-v3.2-thinking palindrome (22.2%, 3.6×) lambda-functions (4) gemini-3-flash-preview bit-rotation (12.2%, 8.5×) fixed-width-integers (4) gemini-3.1-pro-preview quine (8.9%, 10.0×) quine (8) gpt-5.4-mini affine-transformation (5.6%, 6.2×) hex (2) gpt-5.5 rot13 (6.7%, 10.0×) hmac (20) grok-4-fast-reasoning perfect-cubes (7.8%, 4.1×) math-series (2) grok-4.20-0309-reasoning floating-point (5.6%, 7.1×) linear-congruential-generator (3) Table 2 presents Spearman rank correlations between TTG metrics and the two benchmarks. TTG Elo correlates positively with HLE, ARC-AGI, TextQuests, and GPQA-Diamond (ρ = 0.87for HLE, ρ = 0.89for ARC-AGI,ρ = 0.77for TextQuests,ρ = 0.86for GPQA-Diamond). Proposer Win Rates correlate even more strongly with these benchmarks (as high asρ = 0.94atp < 0.001for both HLE and ARC-AGI). Even with a sample size of justn = 10models, these relationships well surpass the threshold for statistical significance (p < 0.01for all of them). We also find a statistically significant correlation of TTG Solver Win Rate with SWE-Bench Pro. This may indicate that solving a puzzle can often require specific coding skills, while proposing a new puzzle requires more of the general creative reasoning that other benchmarks like ARC-AGI test for. In summary, we find a positive answer to our first research question: we find a statistically significant correlation between problem-solving performance in TTG, as measured by both solver and proposer win rates, and strong performance on expert-authored reasoning benchmarks. This is striking due to the minimal cost in designing TTG, with no human-authored problems. Furthermore, we find that problem generation is a qualitatively different capability with weaker correlation to previous evaluations, highlighting TTG’s ability to separately assess these reasoning skills. 4.2 RQ2: What are the puzzles about for different proposers? To understand what TTG puzzles are about and whether different models exhibit diverse prefer- ences of topics for their puzzles, we usedgpt-5.4-nanowith high reasoning to tag each puzzle in our annotated corpus with short topic tags (e.g.,modular-arithmetic,bit-manipulation, diophantine-equations), given 5 manually constructed examples, and reviewed the outputs for accuracy. Tags represent both broad (e.g.,number-theory) and skill-specific categories (e.g., chinese-remainder-theorem), allowing us to understand topics at different levels of granularity, much like competitive programming problems are tagged in online platforms like Codeforces. Here, the LLM was given both the puzzle as well as the proposer’s private description of the puzzle, so it did not need to solve the puzzle to determine what skills it required. This allowed us to obtain both a global understanding of puzzle topics, as well as a topic profile per proposer, which is summarized in Table 3. On average, each puzzle had 5.28 tags associated with it — puzzles frequently combine multiple topics by implementing a verifier that tests multiple, sometimes widely different, constraints. We make the following observations: Top topics differ widely across models. Although a few topics are globally common across proposers (the three most frequent tags globally aremodular-arithmetic(53.2% of puzzles), strings(39.8%), andnumber-theory(29.4%)), the most frequent tag for each individual model still varies markedly: Gemini 3 Flash heavily usesbit-manipulation(78.9% of its puzzles), Gemini 3.1 Pro frequently usesstrings(24.4%) GPT-5.5’s most common tag iscryptography (46.7%), and the Claude family notably uses number-theory as its most frequent tag. 7 Each model has its own signature topic. In Table 3, the “Distinctive preference” column reports the tag from each proposer that had the highest “lift” (ratio between frequency in that proposer’s puzzles and global frequency). Remarkably, these tags have no overlap across the proposers, with lifts between3.6×and10.0×over the global rate. We see many striking examples of strong preferences not shared by other models: only Gemini 3.1 Pro writesquinepuzzles (8 in our corpus; no other proposer writes any), and only GPT-5.5 writesrot13puzzles. GPT-5.5 is also responsible for all 20 hmac-tagged puzzles, an unusually large exclusive count consistent with its top tag (cryptography, 47%). Other distinctive preferences include cycle-detection puzzles for Claude Haiku, digit-sum puzzles for Claude Sonnet, palindromes for DeepSeek, bit-rotation for Gemini 3 Flash, and floating- point quirks for Grok-4.20. Puzzle topics have a long tail. Aside from the globally common topics, we also find a long tail of topics that different proposers contribute to TTG. Globally, most topics are rare: we find 543 tags (81% of all tags; including date-time, unicode, lexicographic-ordering) that only appear in 5 or fewer puzzles. The most frequent topicc for each model that is only used by that pro- poser is reported on the last column of Table 3. We find several niches that are filled by ex- actly one model, such asdigit-splittingfrom Claude Haiku,pandigitalnumbers (those containing all digits from0to9) from Claude Opus,lambda-functionsfrom DeepSeek and linear-congruential-generatorfrom Grok-4.20. Overall, we both observe (1) a wide range of topics arising from the simple structure of TTG, and (2) that proposers strongly show distinct preferences for particular topics besides common ones. We list diverse interesting examples in Appendix D, and release our puzzle labels together with the data. 4.3 RQ3: Are strong solvers also good proposers? Unlike existing evaluations, TTG allows us to measure both the aptitude for solving puzzles as well as for creating challenging ones. Here, we investigate how these abilities are related: do models that excel at solving puzzles also excel at proposing challenging problems? We find a strong positive correlation between solver and proposer win rates (Spearmanρ = 0.855, p = 0.0016). As shown in Table 1, models that rank highly as solvers tend to also rank highly as proposers: for instance, GPT-5.5 leads in both roles. This suggests that the reasoning capabilities required to solve difficult puzzles overlap with those needed to construct them. However, we also observe a notable asymmetry when examining absolute performance levels. Across all rounds, proposers either fail to generate a valid puzzle, or to stump their opponents, in 82.1% of the time. Even the strongest proposer, GPT-5.5, wins only 35.6% of rounds as a proposer, compared to a near-perfect 97.2% as solver. At the other extreme, Claide Haiku 4.5 fails to win a single round as proposer (0%), despite still Deepseek v3.2 Thinking (the worst solver) solving 69.7% of valid puzzles proposed by its opponents (see Table 5). This disparity reveals that creating good puzzles is substantially harder than solving them. Models frequently either propose puzzles that their opponents can solve, or fail to solve their own problems. The proposer role thus also represents a highly discriminating test of reasoning capability, with a wider spread in performance across models. 4.4 RQ4: Can proposers increase difficulty using the duel’s history? In TTG duels, proposers can see the whole history of interactions. This design decision was based on the hypothesis that proposers should adapt the difficulty of their problems as they observe either their own failures in proposing valid puzzles with correct solutions (likely indicating they should be less ambitious) or seeing the opponent’s successes as solvers (pressuring them to attempt to propose harder puzzles). To assess whether such an adaptation is taking place, we use the performance of Claude Sonnet 4.6 and GPT-5.4 Mini on all puzzles extracted from all duels as a uniform assessment of puzzle difficulty beyond the two competing models. When solving a puzzle, as also happens during duels, models are only given general instructions on format and the puzzle itself, but no other previous context. We evaluate both models’ success rates for puzzles proposed across turns from 1 to 10 in all duels. Figure 4 shows the results. Solve rates decrease by a total of up to 8.2% (for Claude Sonnet 4.6) across the ten turns. GPT 5.4 Mini, a stronger solver, shows smaller sensitivity (2.1% decrease 8 Figure 2: Solve rates of GPT 5.4-mini and Claude Sonnet 4.6 by turn number on puzzles from all duels, by turn when puzzle was proposed. Regression fit shows puzzles becoming slightly harder each turn. Figure 3: Failure modes for proposers. Did the model’s own sample solution fail (red) or did the solver succeed in solving the puzzle (blue)? Each model had 90 puzzle proposals. overall). Overall, we find that proposers measurably increase difficulty across turns, and this is especially demonstrated for weaker solvers. 4.5 RQ5: Are failing proposers too conservative or too ambitious? When playing as a proposer, a model can fail to win the turn in one of two ways: (1) its own solution is wrong (giving a point to its opponent), or (2) the opponent’s solution is correct (ending the turn in a draw). In the former case, the proposer was arguably overly confident in its ability to solve the puzzle it proposed. In the latter case, the puzzle was not difficult enough to stump the opponent. How do models behave with respect to these two failure modes as proposers? Figure 3 shows the number of failures as proposers of each of these two categories and Table 5 shows overall performance as a proposer. Across all models, we see that failing at providing hard-enough puzzles is slightly more common (68.6% of all puzzle attempts) than providing a too hard puzzle with an incorrect sample solution (16.3%). Data, however, differs a lot by model. Gemini 3 Flash Preview exhibits high overconfidence (44.4% penalties for an incorrect solution to its own puzzle). Claude Sonnet 4.6, on the other hand, usually writes puzzles that are too easy – its competitors can solve Sonnet’s puzzles 88.8% of the time. 5 Limitations and Conclusion We propose a competition-style evaluation of LLM reasoning with no human-authored problems. For 10 frontier models, performance on TTG correlates significantly with other standard reasoning benchmarks that were highly expensive to design. With 10 models, we were able to run duels between all pairs of models. This is in principle not necessary, since BT ratings can be inferred from sparse subsets of all duels. Our current evaluation did not comprise this partial observability scenario, and this will become important as we add new models to TTG and dueling against all previous models becomes infeasible. TTG’s main limitation is also what allows us to prevent saturation: unlike in prior work, here we do not fix a capability to measure by design, but instead allow models to challenge each other in an open-ended fashion. This can in principle lead to obscure problems requiring skills that are uninteresting or uninterpretable to humans. For now, however, we see that our evaluation is still highly aligned with existing expert-authored benchmarks across varied topics of human importance, though we might observe a divergence in the future. Overall, we believe TTG to be a flexible paradigm to measure and understand model reasoning that will stand the challenge of evaluating increasingly more capable models, avoiding saturation by 9 design. We will continuously evaluate more models as they are released. Moreover, we hope TTG’s design contributes ideas for evaluating important capabilities beyond verifiable reasoning. Broader Impacts Statement As machine learning and LLMs improve at a fast pace, benchmarks need to keep up to help us understand limitations and capabilities of modern models. Quick saturation has been a longstanding issue with benchmarks based on sourcing new, ever more challenging human-authored problems, which are also expensive to obtain. Our work presents a benchmark that can in principle stand this challenge, as well as assess other orthogonal skills not present in previous evaluations, such as the ability of models to create problems and correctly estimate their own capabilities. Our evaluation does not intend to replace, but rather complement, existing benchmarks that test models for more specific skills. Although currently TTG correlates with existing reasoning benchmarks, it is possible that they could diverge in the future with more capable models exploiting puzzles that deviate more and more from problems that humans create. Also, evaluating models on TTG involves executing AI-written code, which has to always be done with caution. References Azzolini, M. There were no medals to be won: Scientific duels in the italian renaissance. Nuncius, 34(2):258–283, 2019. BIG-bench authors. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. Transactions on Machine Learning Research, 2023. ISSN 2835-8856. URL https://openreview.net/forum?id=uyTL5Bvosj. Chi, W., Chen, V., Angelopoulos, A. N., Chiang, W.-L., Mittal, A., Jain, N., Zhang, T., Stoica, I., Donahue, C., and Talwalkar, A. Copilot arena: A platform for code llm evaluation in the wild. arXiv preprint arXiv:2502.09328, 2025. Chiang, W.-L., Zheng, L., Sheng, Y., Angelopoulos, A. N., Li, T., Li, D., Zhu, B., Zhang, H., Jordan, M., Gonzalez, J. E., et al. Chatbot arena: An open platform for evaluating llms by human preference. In Forty-first International Conference on Machine Learning, 2024. Chollet, F., Knoop, M., Kamradt, G., Landers, B., and Pinkard, H. Arc-agi-2: A new challenge for frontier ai reasoning systems, 2026. URL https://arxiv.org/abs/2505.11831. Clark, P., Cowhey, I., Etzioni, O., Khot, T., Sabharwal, A., Schoenick, C., and Tafjord, O. Think you have solved question answering? try ARC, the AI2 reasoning challenge. arXiv preprint arXiv:1803.05457, 2018. Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., Hesse, C., and Schulman, J. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021. Dong, K. and Ma, T. Stp: Self-play llm theorem provers with iterative conjecturing and proving. arXiv preprint arXiv:2502.00212, 2025. Glazer, E., Erdil, E., Besiroglu, T., Chicharro, D., Chen, E., Gunning, A., Olsson, C. F., Denain, J.-S., Ho, A., Santos, E. d. O., et al. Frontiermath: A benchmark for evaluating advanced mathematical reasoning in ai. arXiv preprint arXiv:2411.04872, 2024. Guo, D., Yang, D., Zhang, H., Song, J., Wang, P., Zhu, Q., Xu, R., Zhang, R., Ma, S., Bi, X., et al. Deepseek-r1 incentivizes reasoning in llms through reinforcement learning. Nature, 645(8081): 633–638, 2025. Hendrycks, D., Burns, C., Kadavath, S., Arora, A., Basart, S., Tang, E., Song, D., and Steinhardt, J. Measuring mathematical problem solving with the MATH dataset. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2021. Jaech, A., Kalai, A., Lerer, A., Richardson, A., El-Kishky, A., Low, A., Helyar, A., Madry, A., Beutel, A., Carney, A., et al. Openai o1 system card. arXiv preprint arXiv:2412.16720, 2024. 10 Jimenez, C. E., Yang, J., Wettig, A., Yao, S., Pei, K., Press, O., and Narasimhan, K. R. SWE-bench: Can language models resolve real-world github issues? In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=VTF8yNQM66. Long, Y., Jiang, Y., Liu, H., Zhao, Y., Sun, J., Shen, Y., Zhao, C., Cohan, A., and Shasha, D. Puzzleplex: Benchmarking foundation models on reasoning and planning with puzzles. arXiv preprint arXiv:2510.06475, 2025. Nye, M., Andreassen, A. J., Gur-Ari, G., Michalewski, H., Austin, J., Bieber, D., Dohan, D., Lewkowycz, A., Bosma, M., Luan, D., Sutton, C., and Odena, A. Show your work: Scratchpads for intermediate computation with language models. arXiv preprint arXiv:2112.00114, 2021. Phan, L., Gatti, A., Han, Z., Li, N., Hu, J., Zhang, H., Scale AI, and Center for AI Safety. Humanity’s last exam. arXiv preprint arXiv:2501.14249, 2025a. Phan, L., Mazeika, M., Zou, A., and Hendrycks, D. Textquests: How good are llms at text-based video games?, 2025b. URL https://arxiv.org/abs/2507.23701. Poesia, G., Broman, D., Haber, N., and Goodman, N. Learning formal mathematics from intrinsic motivation. Advances in Neural Information Processing Systems, 37:43032–43057, 2024. Rein, D., Hou, B. L., Stickland, A. C., Petty, J., Pang, R. Y., Dirani, J., Michael, J., and Bowman, S. R. GPQA: A graduate-level google-proof q&a benchmark. arXiv preprint arXiv:2311.12022, 2023. Schuster, T., Kalyan, A., Polozov, O., and Kalai, A. T. Programming puzzles. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2021. Toscano, F. The Secret Formula: How a Mathematical Duel Inflamed Renaissance Italy and Uncovered the Cubic Equation. Princeton University Press, 2020. Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E. H., Le, Q. V., and Zhou, D. Chain-of-thought prompting elicits reasoning in large language models. In Advances in Neural Information Processing Systems, volume 35, 2022. Wu, Y., Tang, X., Mitchell, T. M., and Li, Y. Smartplay: A benchmark for llms as intelligent agents. arXiv preprint arXiv:2310.01557, 2023. Yu, L., Jiang, W., Shi, H., Yu, J., Liu, Z., Zhang, Y., Kwok, J. T., Li, Z., Weller, A., and Liu, W. MetaMath: Bootstrap your own mathematical questions for large language models. In International Conference on Learning Representations, 2024. Zelikman, E., Wu, Y., Mu, J., and Goodman, N. STaR: Bootstrapping reasoning with reasoning. In Advances in Neural Information Processing Systems, volume 35, 2022. Zhao, A., Wu, Y., Yue, Y., Wu, T., Xu, Q., Lin, M., Wang, S., Wu, Q., Zheng, Z., and Huang, G. Absolute zero: Reinforced self-play reasoning with zero data. arXiv preprint arXiv:2505.03335, 2025. 11 A Additional Results Table 4 shows puzzle difficulty by each model as a proposer, as evaluated by the ability of GPT-5.4 Mini and Claude Sonnet 4.6 to solve them. Table 4: Claude Sonnet 4.6 and GPT-5.4 Mini Solving Puzzles By Proposer Model. After completing all duels, we mined puzzles from their logs and had Claude Sonnet 4.6 and GPT-5.4 Mini try the puzzles. We only used valid puzzles, i.e. those that had correct sample solutions, and we show each model’s solve rate for puzzles by their original proposer. Proposer Modelclaude-sonnet-4-6gpt-5.4-miniTotal claude-haiku-4-576/86 (88%)76/86 (88%)152/172 (88%) grok-4-fast-reasoning56/67 (84%)57/67 (85%)113/134 (84%) claude-sonnet-4-680/89 (90%)68/89 (76%)148/178 (83%) deepseek-v3.2-thinking44/61 (72%)49/61 (80%)93/122 (76%) gpt-5.4-mini53/70 (76%)53/70 (76%)106/140 (76%) claude-opus-4-765/85 (76%)58/85 (68%)123/170 (72%) grok-4.20-0309-reasoning55/80 (69%)57/80 (71%)112/160 (70%) gpt-5.562/85 (73%)56/85 (66%)118/170 (69%) gemini-3-flash-preview36/50 (72%)32/50 (64%)68/100 (68%) gemini-3.1-pro-preview58/80 (72%)48/80 (60%)106/160 (66%) Total585/753 (78%)554/753 (74%)1139/1506 (76%) Table 5: Puzzle outcomes by model and role. The left columns show performance in the solver role (number of puzzles solved and failed to solve). The middle columns shows performance in the proposer role (number of invented puzzles solved, unsolved (% of all correct (non-penalized) puzzles), puzzles with incorrect sample solutions which the proposer was penalized for). In a second run, we had every model try to solve its own puzzle without the prior context (i.e. without seeing its notes). "Self S" shows the proportion of the model’s proposed puzzles that it could itself solve in the separate context, and "Either S" shows the proportion of puzzles that could be solved by either the model itself in a separate context or the model’s opponent in the duel. This is to ensure that puzzles are solvable. ModelSolv S (%)Solv FProp SProp U (%)Prop P (%)Self S (%)Either S (%) gpt-5.569 (97.2%)25332 (37.6%)5 (5.6%)84 (98.8%)84 (98.8%) claude-opus-4-762 (93.9%)46916 (18.8%)5 (5.6%)78 (91.8%)80 (94.1%) gpt-5.4-mini 67 (89.3%)85911 (15.7%)20 (22.2%)61 (87.1%)67 (95.7%) gemini-3.1-pro-...66 (89.2%)85228 (35.0%)10 (11.1%)76 (95.0%)77 (96.2%) claude-sonnet-4-666 (83.5%)13809 (10.1%)1 (1.1%)78 (87.6%)82 (92.1%) grok-4.20-...59 (78.7%)166416 (20.0%)10 (11.1%)63 (78.8%)70 (87.5%) claude-haiku-4-5 57 (78.1%)16860 (0.0%)4 (4.4%)84 (97.7%)86 (100.0%) grok-4-fast-r...64 (74.4%)22634 (6.0%)23 (25.6%)64 (95.5%)65 (97.0%) gemini-3-flash-...55 (70.5%)234010 (20.0%)40 (44.4%)38 (76.0%)44 (88.0%) deepseek-v3.2-t...53 (69.7%)23529 (14.8%)29 (32.2%)51 (83.6%)53 (86.9%) Total618 (82.1%)135618135 (17.9%)147 (16.3%)677 (89.9%)708 (94.0%) Table 6: Puzzle Outcomes By Turn Number Turn NumberSolvedUnsolvedPenalty (%) 166915 (16.7%) 2671013 (14.4%) 3631116 (17.8%) 4641610 (11.1%) 5691011 (12.2%) 6541818 (20.0%) 7581616 (17.8%) 8581319 (21.1%) 9571716 (17.8%) 10621513 (14.4%) 12 B Cost Table 7 shows the cost of our tournament (all 90 duels). Table 7: Cost of running our 90 duels. ModelIn $/MOut $/MIn tokensOut tokensCallsCost (USD) claude-haiku-4-51.005.00191 3261 970 086163$10.04 claude-opus-4-75.0025.00212 9631 157 530156$30.00 claude-sonnet-4-63.0015.00203 8172 750 483169$41.87 deepseek-v3.2-thinking0.270.40159 3192 006 757166$0.85 gemini-3-flash-preview 0.503.00213 0243 719 913168$11.27 gemini-3.1-pro-preview2.0012.00221 6653 405 222164$41.31 gpt-5.4-mini0.754.50169 4031 781 796165$8.15 gpt-5.55.0030.00260 0291 358 059161$42.04 grok-4-fast-reasoning 0.200.50190 9341 447 295176$0.76 grok-4.20-0309-reasoning1.252.50157 2731 769 149165$4.62 Total1 979 75321 366 290$190.90 C Sources for public results on existing benchmarks For HLE, TextQuests, SWE-Bench Pro, and ARC-AGI, we used numbers from the Center for AI Safety’s AI leaderboard (https://dashboard.safe.ai/). For GPQA Diamond, we used the leaderboard provided by Artificial Analysis (https://artificialanalysis.ai/evaluations/ gpqa-diamond). D Puzzle Examples The project’s website (https://token-games.ai) allows one to navigate all duels generated in our evaluation. Here we show several interesting examples of puzzles designed and solved by various models. D.1 Brainfuck Interpreter d e f m y s t e r y ( x ) : i f n o t i s i n s t a n c e ( x , s t r ) o r l e n ( x ) > 3 8 : r e t u r n F a l s e t a p e = [ 0 ] * 1 0 0 p t r = 0 o u t = " " l o o p s = s t a c k = [ ] f o r i , c i n e n u m e r a t e ( x ) : i f c == ’ [ ’ : s t a c k . a p p e n d ( i ) e l i f c == ’ ] ’ : i f n o t s t a c k : r e t u r n F a l s e s t a r t = s t a c k . p o p ( ) l o o p s [ s t a r t ] = i l o o p s [ i ] = s t a r t i f s t a c k : r e t u r n F a l s e p c = 0 t i c k s = 0 w h i l e p c < l e n ( x ) and t i c k s < 5 0 0 0 : 13 c = x [ p c ] i f c == ’ > ’ : p t r = ( p t r + 1 ) % 1 0 0 e l i f c == ’ < ’ : p t r = ( p t r − 1 ) % 1 0 0 e l i f c == ’ + ’ : t a p e [ p t r ] = ( t a p e [ p t r ] + 1 ) % 2 5 6 e l i f c == ’ − ’ : t a p e [ p t r ] = ( t a p e [ p t r ] − 1 ) % 2 5 6 e l i f c == ’ . ’ : o u t += c h r ( t a p e [ p t r ] ) e l i f c == ’ [ ’ and t a p e [ p t r ] == 0 : p c = l o o p s [ p c ] e l i f c == ’ ] ’ and t a p e [ p t r ] ! = 0 : p c = l o o p s [ p c ] p c += 1 t i c k s += 1 r e t u r n o u t == " Z3 " Proposed by: gemini-3.1-pro-previewSolved by: gemini-3-flash-preview Outcome:SolvedAttempt:"++++++++++[>+++++++++>+++++«-]>.>+."Solution: "++++++++++[>+++++++++>+++++«-]>.>+." D.2 Multi-Phase Bit-Rotation Cipher d e f m y s t e r y ( x ) : i f n o t i s i n s t a n c e ( x , s t r ) o r l e n ( x ) ! = 8 : r e t u r n F a l s e # C h e c k f o r p r i n t a b l e A S C I I c h a r a c t e r s v = [ o r d ( c ) f o r c i n x ] i f any ( c < 32 o r c > 1 2 6 f o r c i n v ) : r e t u r n F a l s e # P h a s e 1 : M o d u l a r a f f i n e t r a n s f o r m a t i o n f o r i i n r a n g e ( 8 ) : v [ i ] = ( v [ i ] * 13 + 3 6 ) % 2 5 6 # P h a s e 2 : XOR d e p e n d e n c y c h a i n f o r i i n r a n g e ( 7 , 0 , − 1 ) : v [ i ] ^= v [ i − 1 ] v [ 0 ] ^= v [ 7 ] # P h a s e 3 : B i t w i s e r o t a t i o n ( l e f ts h i f t b y 5 ) f o r i i n r a n g e ( 8 ) : v [ i ] = ( ( v [ i ] << 5 ) | ( v [ i ] >> 3 ) ) & 0 xFF # P h a s e 4 : F i x e d p e r m u t a t i o n p = [ 5 , 2 , 0 , 7 , 4 , 1 , 3 , 6 ] v = [ v [ p [ i ] ] f o r i i n r a n g e ( 8 ) ] # P h a s e 5 : F i n a l XOR w i t h c o n s t a n t f o r i i n r a n g e ( 8 ) : v [ i ] ^= 66 r e t u r n v == [ 2 2 7 , 1 0 6 , 9 , 1 4 3 , 5 5 , 2 3 7 , 2 0 6 , 9 8 ] Proposed by: gemini-3-flash-previewSolved by: gpt-5.4-mini Outcome: Sample Solution IncorrectAttempt: –-Solution: "Python3!" D.3 Layered Digit Constraints d e f m y s t e r y ( x ) : i f n o t i s i n s t a n c e ( x , i n t ) o r i s i n s t a n c e ( x , b o o l ) : r e t u r n F a l s e i f n o t ( 1 0 * * 9 <= x < 1 0 * * 1 0 ) : r e t u r n F a l s e 14 d i g i t s = [ i n t ( c ) f o r c i n s t r ( x ) ] i f sum ( d i g i t s ) ! = 4 7 : r e t u r n F a l s e i f sum ( d * d f o r d i n d i g i t s ) ! = 2 6 1 : r e t u r n F a l s e i f d i g i t s [ 0 ] * d i g i t s [ 9 ] ! = 3 6 : r e t u r n F a l s e i f d i g i t s [ 3 ] − d i g i t s [ 5 ] ! = 4 : r e t u r n F a l s e s = 0 f o r i , d i n e n u m e r a t e ( d i g i t s ) : s = ( s * 10 + d ) ( i * 7 + 3 ) i f s % 1 0 0 0 0 0 3 ! = 2 0 1 6 6 : r e t u r n F a l s e r e t u r n T r u e Proposed by: claude-opus-4-7Solved by: gemini-3-flash-preview Outcome: Solver FailedAttempt: 4348345439Solution: 4657434329 D.4 Quine with Restricted Characters d e f m y s t e r y ( x ) : i f n o t i s i n s t a n c e ( x , s t r ) : r e t u r n F a l s e # R e s t r i c t l e n g t h t o p r e v e n t v e r b o s e w o r k a r o u n d s i f l e n ( x ) > 7 5 : r e t u r n F a l s e # B l o c k f o r m a t t i n g , m e t h o d c a l l s , a n d i n t r o s p e c t i o n e s c a p e s b a n n e d = [ ’ f ’ , ’ F ’ , ’%’ , ’ . ’ , ’ e v a l ’ , ’ e x e c ’ , ’ i m p o r t ’ , ’ __ ’ , ’ g e t a t t r ’ ] i f any ( b i n x f o r b i n b a n n e d ) : r e t u r n F a l s e t r y : f u n c = e v a l ( x ) i f n o t c a l l a b l e ( f u n c ) : r e t u r n F a l s e # T h e f u n c t i o n m u s t b e a q u i n e ( r e t u r n s i t s own s o u r c e c o d e ! ) r e t u r n f u n c ( ) == x e x c e p t E x c e p t i o n : r e t u r n F a l s e Proposed by: gemini-3.1-pro-previewSolved by: deepseek-v3.2-thinking Outcome: Solver FailedAttempt:"lambda: locals()[’x’]"Solution:"lambda a=’lambda a=: a[:9]+repr(a)+a[9:]’: a[:9]+repr(a)+a[9:]" D.5 Collatz Sequence as a Lambda d e f m y s t e r y ( x ) : i f n o t i s i n s t a n c e ( x , s t r ) : r e t u r n F a l s e # R e s t r i c t l e n g t h t o p r e v e n t h a r d c o d i n g d i c t i o n a r y l o o k u p s o r v e r b o s e w o r k a r o u n d s i f l e n ( x ) > 8 5 : r e t u r n F a l s e # Ban i t e r a t i o n , c o n d i t i o n a l s , s t a n d a r d e s c a p e s , a n d c o m p a r i s o n o p e r a t o r s b a n n e d = [ ’ i f ’ , ’ e l s e ’ , ’ e l i f ’ , ’ f o r ’ , ’ w h i l e ’ , ’ d e f ’ , ’ i m p o r t ’ , ’ e v a l ’ , ’ e x e c ’ , 15 ’ c o m p i l e ’ , ’ __ ’ , ’ < ’ , ’ > ’ , ’ == ’ , ’ ! = ’ , ’ i n ’ , ’ max ’ , ’ min ’ , ’ a b s ’ , ’ sum ’ , ’ a n y ’ , ’ a l l ’ ] i f any ( b i n x f o r b i n b a n n e d ) : r e t u r n F a l s e t r y : f = e v a l ( x ) i f n o t c a l l a b l e ( f ) : r e t u r n F a l s e # T h e f u n c t i o n m u s t c o r r e c t l y c o m p u t e t h e l e n g t h o f t h e C o l l a t z s e q u e n c e f o r n . # T h e s e q u e n c e t e r m i n a t e s when n = 1 . d e f c o l l a t z ( n ) : c = 0 w h i l e n > 1 : n = n / / 2 i f n % 2 == 0 e l s e 3 * n + 1 c += 1 r e t u r n c # T e s t e x t e n s i v e l y t o p r e v e n t m a p p i n g a s m a l l s e t o f i n p u t s f o r n i n r a n g e ( 1 , 1 0 0 ) : i f f ( n ) ! = c o l l a t z ( n ) : r e t u r n F a l s e r e t u r n T r u e e x c e p t E x c e p t i o n : r e t u r n F a l s e Proposed by: gemini-3.1-pro-previewSolved by: deepseek-v3.2-thinking Outcome:Solver FailedAttempt:"(lambda f:f(f))(lambda f:lambda n:n-1 and(1+f(f)((n&1)*(3*n+1)+(1-(n&1))*(n//2)))or0)" Solution:"lambda n,c=lambda n,c:n//2 and 1+c(n%2 and 3*n+1 or n//2,c) or 0:c(n,c)" D.6 ROT13 + Base64 + Hash Verification i m p o r t b a s e 6 4 i m p o r t c o d e c s i m p o r t h a s h l i b d e f m y s t e r y ( x ) : i f t y p e ( x ) i s n o t s t r o r l e n ( x ) ! = 6 0 : r e t u r n F a l s e t r y : t = c o d e c s . d e c o d e ( x [ : : − 1 ] , " r o t _ 1 3 " ) p a y l o a d = b a s e 6 4 . b 6 4 d e c o d e ( t , v a l i d a t e = T r u e ) e x c e p t E x c e p t i o n : r e t u r n F a l s e i f b a s e 6 4 . b 6 4 e n c o d e ( p a y l o a d ) . d e c o d e ( " a s c i i " ) ! = t : r e t u r n F a l s e r e t u r n ( h a s h l i b . md5 ( p a y l o a d ) . h e x d i g e s t ( ) == " 9 e 1 0 7 d 9 d 3 7 2 b b 6 8 2 6 b d 8 1 d 3 5 4 2 a 4 1 9 d 6 " and h a s h l i b . s h a 1 ( p a y l o a d ) . h e x d i g e s t ( ) == " 2 f d 4 e 1 c 6 7 a 2 d 2 8 f c e d 8 4 9 e e 1 b b 7 6 e 7 3 9 1 b 9 3 e b 1 2 " and h a s h l i b . s h a 2 5 6 ( p a y l o a d ) . h e x d i g e s t ( ) == " d 7 a 8 f b b 3 0 7 d 7 8 0 9 4 6 9 c a 9 a b c b 0 0 8 2 e 4 f 8 d 5 6 5 1 e 4 6 d 3 c d b 7 6 2 d 0 2 d 0 b f 3 7 c 9 e 5 9 2 " ) Proposed by: gpt-5.5Solved by: claude-haiku-4-5 16 Outcome: Solver FailedAttempt:"==jMiETV5cKLfOfMbEUVlITqiqOlpj1HqdOPriMTVhq3olWTVeAJn1SUVyutI" Solution: "==jMiETV5cKLfOFMbEUVlIzqiOlpj1JqdOPriMTVhq3olWTVeAJn1SUVyuTI" D.7 PBKDF2-HMAC with Null Byte i m p o r t h a s h l i b d e f m y s t e r y ( x ) : i f t y p e ( x ) i s n o t s t r : r e t u r n F a l s e t r y : p a s s w o r d = x . e n c o d e ( " l a t i n −1 " ) e x c e p t U n i c o d e E n c o d e E r r o r : r e t u r n F a l s e i f n o t ( 1 <= l e n ( p a s s w o r d ) <= 3 2 ) : r e t u r n F a l s e d e r i v e d = h a s h l i b . p b k d f 2 _ h m a c ( " s h a 1 " , p a s s w o r d , b " s a \ x 0 0 l t " , 4 0 9 6 , 1 6 , ) r e t u r n d e r i v e d . h e x ( ) == " 56 f a 6 a a 7 5 5 4 8 0 9 9 d c c 3 7 d 7 f 0 3 4 2 5 e 0 c 3 " Proposed by: gpt-5.5Solved by: claude-haiku-4-5 Outcome: Solver FailedAttempt: "password"Solution: "pass 00word" D.8 Unicode Palindrome Case Trick d e f m y s t e r y ( x ) : i f n o t i s i n s t a n c e ( x , s t r ) o r l e n ( x ) ! = 3 : r e t u r n F a l s e # M u s t b e a p a l i n d r o m e i f x ! = x [ : : − 1 ] : r e t u r n F a l s e # U p p e r c a s e m u s t NOT b e a p a l i n d r o m e i f x . u p p e r ( ) == x . u p p e r ( ) [ : : − 1 ] : r e t u r n F a l s e # T h e c h a r a c t e r sum m u s t b e e x a c t l y 6 9 0 i f sum ( o r d ( c ) f o r c i n x ) ! = 6 9 0 : r e t u r n F a l s e r e t u r n T r u e Proposed by: gemini-3.1-pro-previewSolved by: claude-haiku-4-5 Outcome: Solver FailedAttempt: "èéè"Solution: "a 01F0a" D.9 IEEE 754 Negative Zero d e f m y s t e r y ( x ) : i f t y p e ( x ) i s n o t f l o a t : r e t u r n F a l s e 17 i f x ! = 0 : r e t u r n F a l s e r e t u r n ( 1 / x ) < 0 Proposed by: grok-4.20-0309-reasoningSolved by: gemini-3.1-pro-preview Outcome: Sample Solution IncorrectAttempt: –-Solution: -0.0 D.10 Dual Decimal-Binary Palindrome d e f m y s t e r y ( x ) : # C h e c k i n p u t t y p e a n d p o s i t i v i t y i f n o t i s i n s t a n c e ( x , i n t ) o r x <= 0 : r e t u r n F a l s e # D e c i m a l p a l i n d r o m e s = s t r ( x ) i f s ! = s [ : : − 1 ] : r e t u r n F a l s e # B i n a r y p a l i n d r o m e b = b i n ( x ) [ 2 : ] i f b ! = b [ : : − 1 ] : r e t u r n F a l s e # D i v i s i b i l i t y b y 3 a n d 11 i f x % 3 ! = 0 o r x % 11 ! = 0 : r e t u r n F a l s e # N o t p r i m e ( c o m p o s i t e ) i f x < 2 : r e t u r n F a l s e i s _ p r i m e = T r u e f o r i i n r a n g e ( 2 , i n t ( x * * 0 . 5 ) + 1 ) : i f x % i == 0 : i s _ p r i m e = F a l s e b r e a k i f i s _ p r i m e : r e t u r n F a l s e # P r o d u c t o f d i g i t s i s a p e r f e c t s q u a r e p r o d = 1 f o r c h i n s : p r o d * = i n t ( c h ) i f p r o d <= 0 : r e t u r n F a l s e r o o t = i n t ( p r o d ** 0 . 5 ) i f r o o t * r o o t ! = p r o d : r e t u r n F a l s e # Sum o f d i g i t s i s a t r i a n g u l a r n u m b e r t o t a l = sum ( i n t ( d ) f o r d i n s ) n = i n t ( ( 2 * t o t a l ) ** 0 . 5 ) i f n * ( n + 1 ) ! = 2 * t o t a l : r e t u r n F a l s e # C o n g r u e n c e m o d u l o 13 i f x % 13 ! = 7 : r e t u r n F a l s e r e t u r n T r u e Proposed by: deepseek-v3.2-thinkingSolved by: claude-haiku-4-5 Outcome: SolvedAttempt: 33Solution: 33 18 E Prompts The following is the prompt template we show the proposer model: Let’s play a game. # Rules We take turns. One player, the proposer, gives the other a code puzzle with a Python function called ‘mystery‘ that takes a value x (either string or number) and returns a boolean. The other player, the solver, have to find a value x for which the function returns True. The proposer’s job is to make the puzzle as hard as they can, i.e. it should be very difficult to find out which values the function returns True for. To prove that the puzzle is solvable, the proposer must give a sample solution. If the sample solution turns out to be wrong, the proposer is penalized. # Instructions You are the proposer. I am the solver. Respond with a Python code block first, then explain your puzzle. Everything outside the code block is private -- I cannot see it. The last line of your response should contain only: ‘SOLUTION: x‘ where x is the value of a sample solution for your puzzle. Ensure that ‘mystery(x)‘ is valid Python code given your x. For example, it is very important to use quotes if you believe ‘x‘ is a string. Please make sure that your sample solution works -- you will be penalized if not. Do not give multiple code blocks or puzzles. # History Here are the puzzles from past turns of the game. You should NOT repeat them, but you can learn from them to ensure your puzzle is difficult enough. past # Reminder Please ensure your response ends with this line: ‘SOLUTION: x‘ where x is the value of a sample solution for your puzzle. The following is the much shorter prompt for the solver: Here’s a Python function that takes a value x and returns a boolean. Please give me a value for x such that mystery(x) is True. The last line of your response should be: ‘SOLUTION: x‘ where x is the value. Ensure that ‘mystery(x)‘ is valid Python code given your x. For example, if you believe ‘x‘ is a string, use quotes. Example: ‘SOLUTION: \"Hello, world!\"‘ ‘python puzzle ‘ 19