Paper deep dive
EurekaBench: Measuring Agentic Ability to Discover New Scientific Insights
Jiayi Geng, Zhengxuan Wu, Kevin S. Chen, Seungone Kim, Joseph Janssen, Zora Zhiruo Wang, Bhupalee Kalita, Runtian Gao, Aaron Ho, Andrew Oakleigh Nelson, Olexandr Isayev, Francisco Villaescusa-Navarro, Ching-Yao Lai, Howard Chen, Graham Neubig
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 91%
Last extracted: 10/2/2026, 9:10:17 AM
Summary
The paper introduces EurekaBench, a cross-domain benchmark designed to measure AI agents' ability to conduct long-horizon experiments and discover new scientific insights. It evaluates 26 tasks across six scientific domains (neuroscience, computer science, chemistry, astrophysics, geophysics, and plasma physics) using 306 expert-verified scientific insights. The evaluation framework assesses three axes: adherence to scientific constraints, predictive accuracy, and the derivation of scientific insights. Results indicate that while current AI agents (such as Claude Fable 5.1 and GPT 6 Astra) can achieve predictive accuracy comparable to human scientists, they significantly underperform in deriving meaningful scientific insights, often treating discovery as mere optimization rather than interpretability.
Entities (26)
Relation Signals (17)
Claude Fable 5.1 → achievespredictiveaccuracy → 47.4%
confidence 95% · Claude Fable 5.1 and GPT 6 Astra both achieve a predictive accuracy of 47.4%
Human Scientist → achievespredictiveaccuracy → 48.8%
confidence 95% · compared with 48.8% for human solutions.
GPT-6 Astra → achievespredictiveaccuracy → 47.4%
confidence 95% · Claude Fable 5.1 and GPT 6 Astra both achieve a predictive accuracy of 47.4%
GPT-6 Astra → achievesscientificinsights → 29.4%
confidence 95% · However, their SI scores are only 42.4% and 29.4%, compared with 69.7% for human scientists.
Human Scientist → achievesscientificinsights → 69.7%
confidence 95% · However, their SI scores are only 42.4% and 29.4%, compared with 69.7% for human scientists.
Claude Fable 5.1 → achievesscientificinsights → 42.4%
confidence 95% · However, their SI scores are only 42.4% and 29.4%, compared with 69.7% for human scientists.
EurekaBench → coversdomains → Chemistry
confidence 95% · EurekaBench contains an expert-verified set of 26 long-horizon tasks across neuroscience, computer science, chemistry, astrophysics, geophysics, and plasma physics
EurekaBench → coversdomains → Neuroscience
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:When Isaac Newton discovered the law of gravitation, he did so through an iterative process of analyzing observed data such as planetary patterns, finding the underlying mechanisms by describing patterns in mathematical equations, and refining his theory against the Moon's orbit, revealing the startling insight that the same force governs both falling apples and orbiting planets. Would it be possible for AI agents to make similar discoveries? To measure this ability, we introduce EurekaBench, a cross-domain benchmark that tests AI agents' ability to conduct long-horizon experiments and discover mechanisms that explain observations. We evaluate these mechanisms by the scientific insights that can be derived from them. EurekaBench contains an expert-verified set of 26 long-horizon tasks across neuroscience, computer science, chemistry, astrophysics, geophysics, and plasma physics, with a total of 306 scientific insights that the discovered mechanisms are expected to support. Our evaluation framework tests three axes of scientific discovery: agents' ability to follow known scientific constraints, the predictive accuracy of the discovered mechanisms, and whether these mechanisms yield scientific insights or inform future research. Our results show that current AI agents often overly fixate on predictive accuracy optimization, surpassing human scientists, while falling substantially short in deriving scientific insights.
Tags
Links
- Source: https://arxiv.org/abs/2610.00492v1
- Canonical: https://arxiv.org/abs/2610.00492v1
Trouble viewing inline? Open PDF directly →
Full Text
150,468 characters extracted from source content.
Expand or collapse full text
EurekaBench: Measuring Agentic Ability to Discover New Scientific Insights Jiayi Geng 1 Zhengxuan Wu 2 * Kevin S. Chen 3 Seungone Kim 1 Joseph Janssen 5 Zora Zhiruo Wang 1 Bhupalee Kalita 6 Runtian Gao 6 Aaron Ho 7 Andrew Oakleigh Nelson 8 Olexandr Isayev 6 Francisco Villaescusa-Navarro 910 Ching-Yao Lai 5 Howard Chen 11 12† Graham Neubig 1 1 Carnegie Mellon University, Language Technologies Institute 2 Stanford University, Department of Computer Science 3 Yale University, Quantitative Biology Institute 4 Yale University, Department of Molecular, Cellular, and Developmental Biology 5 Stanford University, Department of Geophysics 6 Carnegie Mellon University, Department of Chemistry 7 Massachusetts Institute of Technology, Plasma Science and Fusion Center 8 Columbia University, Department of Applied Physics and Applied Mathematics 9 Princeton University, Department of Astrophysical Sciences 10 Flatiron Institute, Center for Computational Astrophysics 11 Engram 12 Princeton University, Department of Computer Science ogeng, gneubig@cs.cmu.edu EurekaBench GithubEurekaBench Website Abstract When Isaac Newton discovered the law of gravitation, he did so through an iterative process of analyzing observed data such as planetary patterns, finding the underlying mechanisms by describing patterns in mathematical equations, and refining his theory against the Moon’s orbit, revealing the startling insight that the same force governs both falling apples and orbiting planets. Would it be possible for AI agents to make similar discoveries? To measure this ability, we introduce EUREKABENCH, a cross-domain benchmark that tests AI agents’ ability to conduct long-horizon experiments and discover mechanisms that explain observations. We evaluate these mechanisms by the scientific insights that can be derived from them. EUREKABENCH contains an expert-verified set of 26 long-horizon tasks across neuroscience, computer science, chemistry, astrophysics, geophysics, and plasma physics, with a total of 306 scientific insights that the discovered mechanisms are expected to support. Our evaluation framework tests three axes of scientific discovery: agents’ ability to follow known scientific constraints, the predictive accuracy of the discovered mechanisms, and whether these mechanisms yield scientific insights or inform future research. Our results show that current AI agents often overly fixate on predictive accuracy optimization, surpassing human scientists, while falling substantially short in deriving scientific insights. Figure 1: The scientific discovery process in EUREKABENCH. In this example from plasma physics, scientists observe that different transport models predict different fusion power outputs for the same reactor under the same operating conditions(observation). Through iterative experiments, they find that differences in turbulent transport change temperature and density distributions, explaining the different power predictions(mechanism). This helps them understand how turbulent transport may affect fusion power in larger reactors or under stronger magnetic fields(insight) . Discovering such insights is a “eureka moment” in science. * This project started while Zhengxuan Wu was a PhD student at Stanford University (now at Google DeepMind). † This project started while Howard Chen was a PhD student at Princeton University. 1 arXiv:2610.00492v1 [cs.CL] 30 Sep 2026 Initial observation: Evaluation metrics: Predictive Accuracy: Scientific Constraints: Interpretable Insights: Does the new SSA equation explain why the viscosity of a real ice shelf cannot be one scalar field? Classical SSA equations fit compression zones but struggle to explain ice flow in extension zones. Experimentation: Ice resists horizontal and vertical deformation differently, with distinct viscosities and . μ h μ v Discovered mechanism: a Neuroscience Cortical object maps reflect properties such as animacy and size. b Geophysics Classical SSA equations do not fully describe real ice shelves. c Plasma Physics Triangularity reversal improves plasma confinement. d Astrophysics A single galaxy encodes its universe’s matter density. e Computer Science Reading changes the LM’s stated beliefs. f Chemistry A given reaction proceeds from reactants to products. Example: Ice Shelf Flow Laws You are a geophysicist studying the flow of Antarctic ice shelves. Iterative experiments More observations Relative equation residuals on Amery satellite data. [0, 1] Incompressible ice, positive viscosities, and ocean-pressure boundary conditions. Insight - Predictive Accuracy Pareto Curve Ice-shelf simulator Figure 2: Initial observations, evaluation, and results in EUREKABENCH. Left: Examples of initial observations from six domains: (a) neuroscience, (b) geophysics, (c) plasma physics, (d) astrophysics, (e) computer science, and (f) chemistry. Top right: An ice shelf flow-law example illustrates how an agent uses iterative experiments to propose a mechanism, which is evaluated for scientific constraints, predictive accuracy, and scientific insights. SSA denotes the shallow-shelf approximation. Bottom right: Predictive accuracy and scientific insight scores for seven AI agents and the human reference. The strongest agents approach human predictive accuracy while attaining substantially lower insight scores. 1 Introduction Scientific progress occurs when scientists identify mechanisms that underlie physical phenomena and explain observations challenging our existing understanding of the world. Examples include chemical reaction pathways and models of how neurons coordinate movement (Zahavy, 2026; Figure 1). A good mechanism not only accurately predicts the data observed throughout experiments, but also yields scientific insights that help us understand a broader range of related phenomena and can even be extrapolated to open new research directions. Recent AI agents have achieved breakthroughs in solving longstanding scientific problems (OpenAI, 2026d; Gottweis et al., 2026). Their strong search and optimization capabilities are especially effective when objectives are explicit and outcomes are verifiable (Novikov et al., 2025; Lupidi et al., 2026). However, finding correct solutions does not necessarily yield scientific insights that deepen our understanding, and might even hinder their discovery (Tao, 2026). Signatories of the Declaration on Math and AI (2026) have argued that this gap results in a “severe misalignment between how AI has recently been used in science and methods that have traditionally driven scientific progress.” To better understand whether AI agents can bridge this gap, we study the question: “Can AI agents move beyond solution optimization and discover novel scientific insights?” To answer this question, we first consulted 15 experienced scientists from 10 universities across 8 scientific domains to identify the stages of research where they would most value and expect assistance from AI agents. This clarifies where AI can best accelerate scientific research. We ultimately collaborated with 10 of these experts to develop EUREKABENCH, a benchmark containing 26 recent and important scientific problems crafted by experts across six domains: neuroscience, geophysics, astrophysics, computer science, plasma physics, and chemistry (see Figure 2(a)–(f)). EUREKABENCH is the first benchmark to rigorously operationalize open-ended scientific insight discovery within a systematic evaluation framework, challenging AI agents to start from initial observations and use iterative experimentation to discover mechanisms that are understandable enough to yield scientific insights. We formulate each task with initial observations that challenge existing understanding. We provide an environment interface that allows agents to conduct iterative experiments autonomously, investigate the underlying mechanisms, and derive novel scientific insights. We propose an evaluation framework that examines three aspects: (1) scientific constraint tests, assessing whether the mechanism satisfies the fundamental domain-specific common-sense assumptions; (2) predictive accuracy, Figure 3: Scientist agent workflow in EUREKABENCH. Given a task specification and a documented environment interface, the scientist agent conducts iterative experiments to discover a mechanism and submits its description, implementation, and experimental records. measuring how accurately the discovered mechanism predicts measurements in the test set; and (3) scientific insights (SI), evaluating whether the mechanism discovered by the agent can reveal useful scientific insights. We work closely with the domain experts and formulate the scientific insight questions of the form “Does the discovered mechanism explain/establish ...?”, with a “perfect” mechanism expected to completely answer all the questions. We use an agentic judge (Zhuge et al., 2024), which has access to the same environment interface, to evaluate the discovered mechanisms. We empirically evaluate seven top AI agents on EUREKABENCH, and find that current AI agents can produce accurate predictions yet rarely support meaningful scientific insights. For example, GPT 6 Astra approaches the human reference in predictive accuracy (47.4% versus 48.8%), but only achieves a 29.4% SI score, substantially lower than 69.7% for human-discovered mechanisms. This gap shows that optimizing for predictive accuracy by fitting observed data is not enough to obtain mechanisms that deepen scientific understanding. We further investigate why agents struggle to discover novel scientific insights through iterative experimentation. We find that agents tend to treat scientific mechanism discovery as task optimization instead of focusing on the mechanism’s interpretability, in contrast to human scientists. Agents show some scientific reasoning ability during experimentation, for example by establishing the mechanisms behind physical parameters. However, their ability to discover insights remains severely limited by their failure to run experiments to reduce uncertainty about the world and by a lack of diverse, in-depth exploration. Despite the strong capabilities that current AI agents have demonstrated, our empirical results on EUREKABENCH reveal that, as AI scientists, they still lack the fundamental ability to discover meaningful insights independently through an experimentation loop. EUREKABENCH reveals the unique open-endedness of scientific discovery, in contrast to verifiable scientific reasoning tasks, and opens a path for the community to develop AI scientists that better support human researchers. 2 Evaluating a Task in EUREKABENCH 2.1 Task Overview A task in EUREKABENCH is defined asT = (x,o 1 ,E,R). The task descriptionxprovides the scientific background, instructions, and available computing resources and iteration budgets, whileo 1 denotes the initial observations that challenge existing scientific understanding. The environment interfaceEallows the scientist agent to conduct the experimentation loop. The reward functionRevaluates the scientist agent’s deliverables according to the evaluation rubricC and returns a score. A time steptin the experimentation loop consists of a query actiona t toEand its returned observationo t+1 . We index these time steps byt = 1,...,T, whereTis the total number of steps. We define the experiment trajectory asτ = (o 1 ,a 1 ,o 2 ,...,a T ,o T+1 ). The discovered mechanismMincludes its explanation and executable implementation, andLdenotes the experiment log. We defineIas the scientific insights that can be derived fromM, where(M,L,τ ) derive −→ I. The complete task proceeds from iterative experimentation to evaluation in EUREKABENCH: T iterative experimentation −→ (M,L,τ ) evaluation −→ R(M,L,τ ;E,C). Agent workspace initialization.We initialize the scientist agent’s workspace with a task specification, a command- line interface (CLI) and documentation forE. The task specification containsx,o 1 , and the name ofE. In the task specification, we ask the scientist agent to find a mechanismMthat explainso 1 . We also require the scientist agent to explore scientific insightsIthrough experimentation and explicitly state that another scientist will evaluate these insights alongside predictive accuracy. For some problems, we include training data in the scientist agent’s workspace. We provide an example task specification in Appendix A. Interacting with the environment. During the experimentation loop, the scientist agent choosesa t based on the trajectory so far,τ 1,...,t , and queriesEto obtaino t+1 and revise its hypotheses, as shown in Figure 3. Such an iterative cycle of intervention, observation, and hypothesis revision constitutes reverse engineering (Geng et al., 2025a). Compiling experimental artifacts. At the end of the loop, the scientist agent submitsM,L, andτ. The submission must include three files: (1)mechanism.mdexplains the causal logic ofM; (2)mechanism.py implementsMas a Python program for evaluation on the test set; and (3)experiments.logcontainsL, the record of the experiments, including ideas, findings, and informative failures. The scientist agent may submit up to 1 GB of material from τ that could help other scientists continue this line of research. Evaluating the deliverables. After the scientist agent finishes the experimentation loop, a judge agent reads M , L, andτand determines whetherMpasses the scientific constraint tests according to the criteria inC. We execute mechanism.pyon the test set to evaluate whether the discovered mechanism can make accurate predictions. To evaluate scientific insightsI, we allow the judge agent to interact withE. The judge agent checks each insight question i∈C to determine whether the corresponding insight can be derived from M . We provide details of how we compute the score R(M,L,τ ;E,C) in §2.2. 2.2 Evaluation Protocol Evaluation Metrics. We evaluateMalong three dimensions (Figure 2, top right). Letn d denote the number of criteria in dimension d∈SC, PA, SI. (1) Scientific constraint tests (SC). These tests check whetherMsatisfies the domain’s common-sense assumptions. For constrainti, a judge assignsc i = 1ifMsatisfies it andc i = 0otherwise. The number of satisfied constraints is b SC = n SC X i=1 c i .(1) A mechanism passes SC only if b SC = n SC . (2) Predictive accuracy (PA). We execute the implementation ofMon held-out test data to measure its predictive accuracy. Each prediction criterioniyields a continuous scores i ∈ [0, 1], normalized according to the task’s rubric, with higher values indicating more accurate predictions. We sum these scores to obtain the total PA pointsp PA and average them to obtain the PA score: p PA = n PA X i=1 s i , S PA = p PA n PA .(2) (3) Scientific insights (SI). We measure the fraction of insight questions inCwhose corresponding insights can be derived fromM. For questioni, a judge assignsy i = 1if the insight can be derived fromMwith supporting evidence and y i = 0 otherwise. The total SI points and the SI score are p SI = n SI X i=1 y i , S SI = p SI n SI .(3) Evaluation rubric construction. We constructCwith our domain experts. We express scientific insights as yes-or-no questions about what can be derived from a “perfect” mechanism (Figure 4). These questions draw on insights from recent work by human scientists, open questions in the source studies, and potential insights proposed by our domain experts. All insights in EUREKABENCH are verified by our domain experts to be scientifically important and interesting. Agentic judge. We use Claude Opus 4.8 and GPT 5.6 Sol as independent agentic judges for SC tests and SI evaluation. Both judges evaluateMusingL,τ, and the relevant criteria inC. For SC tests, the judges check whether the discovered mechanism contradicts the domain-specific common-sense assumptions. For SI, we allow the judge agents to run additional experiments by queryingEto determine whether each insight can be derived fromM. We instruct the judges to keepMunchanged. An insight question receives aYesverdict only when the judge agent provides complete evidence showing how the corresponding insight can be derived from M (Appendix C.2). Conditioned score.A mechanism that violates the domain’s common-sense assumptions is not scientifically valid. For each judge, we combine the total PA and SI points defined above, conditional on passing all SC tests: S conditioned = p PA + p SI n PA + n SI if b SC = n SC , 0otherwise. (4) We average the two judges’ conditioned scores for each submission. Figure 4: The 9 interpretable insight questions for the Antarctic ice shelf problem in geophysics in EUREK- ABENCH. These insights explain why ice deforms differently under compression and extension, how its resistance varies with direction, and which measurements determine its viscosity. 3EUREKABENCH 3.1 Benchmark Construction We constructed EUREKABENCH with domain experts by selecting scientific problems and developing task specifica- tions, experimental environments, and evaluation rubrics. In-depth engagement with domain experts. We built EUREKABENCH with domain experts who hold PhDs in their respective fields and come from top research universities and institutes in the world. We began with in-depth discussions about where AI could contribute to their research and identified mechanism discovery as a valuable target. The experts worked closely with us to identify important scientific problems in each domain and the potential insights each problem could yield. Together, we refined the task specifications and every evaluation criterion through multiple rounds of detailed discussion and revision. Problem selection.We selected scientifically important problems in each domain. Human scientists have already found mechanisms that partially explain the initial observations and derived scientific insights from them, but the problems remain only partially solved. We drew on these findings to construct insight questions, grounding evaluation in what a mechanism can explain and what insights can be derived from it, without requiring the mechanism to match a complete ground-truth solution. Task formulation. We formulated each task around finding an understandable mechanism to explain initial observations that challenged experts’ understanding of the problem. The task specification includes these initial observations and provides earlier observations as scientific background. We ask the agent to discover the mechanism through iterative experimentation and derive scientific insights from it. Environment interfaces. We built an experimental environment for each problem and packaged it with its dependencies in an Apptainer image. We used the environment interfaces listed in Table 1 and provided a CLI with usage instructions. We obtained PA test data from the original studies or generated them with the task-specific simulators. We describe the environment interfaces for each domain in Appendix E. 3.2 Benchmark Statistics EUREKABENCH contains 26 problems across six scientific domains. We have 10 problems in neuroscience, 2 in geophysics, 3 in plasma physics, 7 in astrophysics, 3 in computer science, and 1 in chemistry. Table 1 lists the environment interfaces, and Figure 5 shows the problem distribution across domains. Across these problems, EUREKABENCH includes 306 scientific insight questions to evaluate the scientific insights agents uncover through the mechanism discovery process. Examples of initial observations appear in Figure 2(a)–(f). 4 Experiments Models. We evaluate seven LLMs on EUREKABENCH, including both frontier closed-source and open-weight models. The closed-source models include Claude Opus 5 (Anthropic, 2026d), Claude Opus 4.8 (Anthropic, 2026c), Sci DomainsEnv Interfaces Neurosciencec302 a ; TRIBE v2 b GeophysicsFEniCSx c ; HBV d Plasma PhysicsTORAX e ; TokaMaker with GPEC f AstrophysicsCAMELS g ; The DREAMS Project h Computer Science Open-weight LMs ChemistryAIMNet2-rxn i a (Gleeson et al., 2018); b (d’Ascoli et al., 2026); c (Baratta et al., 2023); d (Bergstr ̈ om, 1992); e (Citrin et al., 2024); f (Hansen et al., 2024); g (Villaescusa-Navarro et al., 2021); h (Rose et al., 2025); i (Anstine et al., 2025). Table 1: Environment interfaces in EUREKABENCH. Computer Science Neuroscience Cosmology and Astrophysics Chemistry Geophysics Plasma Physics Figure 5: Problem distribution across domains. Claude Fable 5.1 (Anthropic, 2026b), GPT 6 Astra (OpenAI, 2026c), and GPT 5.6 Sol (OpenAI, 2026b). The open-weight models include DeepSeek V4 Flash (DeepSeek-AI, 2026) and Kimi K3 (Moonshot AI, 2026). All models are run with the maximum reasoning effort. Agent harnesses.We use three agent harnesses to run the models as autonomous agents on EUREKABENCH. We run Claude models with Claude Code (Anthropic, 2026a) and GPT models with Codex (OpenAI, 2026a) because each harness is developed by the corresponding model provider. Both harnesses are launched through the ACP agent in OpenHands (OpenHands, 2026). For open-weight models, we use the OpenHands Agent SDK (Wang et al., 2025a), which supports different LLMs through a single interface. Human scientist baselines.We use the current progress made by human scientists on each problem as the human solution. Compute budgets for AI agents. For each problem, the agent is given one H100 GPU and a budget of 1,000 iterations and 4 hours per run to discover the underlying mechanism through interaction with the same environment. Web access. We allow web search but block access to the research underlying each problem, including the original publications and literature aggregators that provide their content. These restrictions apply to both search and webpage retrieval to prevent agents from looking up the mechanisms discovered by human scientists. These restrictions limit access during evaluation but do not rule out exposure to the source research during pretraining. 5 Results Scientific ValueEfficiency Agents Conditioned Score (%) Predictive Accuracy (%) Scientific Insights (%) SC Failures↓ CallsCost ($) Claude Code Claude Fable 5.129.847.442.415/26 10423.96 Claude Opus 527.342.9 36.215/2616617.56 Claude Opus 4.818.442.726.916/26939.75 Codex GPT 6 Astra28.747.429.410/2614924.62 GPT 5.6 Sol19.441.925.917/26552.23 OpenHands DeepSeek V4 Flash14.629.924.415/262061.24 Kimi K312.928.825.216/261857.19 Human scientist63.748.869.70/26– Table 2: Main results on EUREKABENCH. PA and SI are reported unconditionally; SC failures count completed problems that fail either judge’s scientific-constraint tests, out of 26 problems. AI agents still struggle to find the underlying mechanism.The results in Table 2 show that AI agents struggle to find mechanisms that explain the initial observations. The conditioned scores range from 12.9% to 29.8% across the seven AI agents. Even the best-performing agent, Claude Fable 5.1, achieves only 29.8%, compared with 63.7% for human-discovered mechanisms. Surprisingly, except GPT 6 Astra (10/26), all agents fail to pass the scientific-constraint tests on more than half of the problems in EUREKABENCH, showing that even top AI agents cannot reliably discover mechanisms that satisfy the most basic common-sense assumptions. Our empirical results show that current AI agents still lack the ability to reliably discover mechanisms that explain observations through the scientific experiment loop, which is a core capability for scientific discovery. Agentic ability to discover scientific insights. To study the causes of this capability gap, we report predictive accuracy and SI scores separately in Table 2. We find that AI agents intentionally treat scientific mechanism discovery as an optimization task. Claude Fable 5.1 and GPT 6 Astra both achieve a predictive accuracy of 47.4%, compared with 48.8% for human solutions. However, their SI scores are only 42.4% and 29.4%, compared with 69.7% for human scientists. This large gap in insight discovery demonstrates that human scientists value the interpretation and extrapolation of mechanisms, while AI agents tend to prioritize brute-force numerical optimization. Therefore, optimizing predictive accuracy does not imply discovering novel scientific insights (Appendix D.2). 6 Analysis baseline+goal +insights 0 10 20 30 40 50 60 70 Conditioned score (%) 27.3 37.4 60.3 Neural dynamics 9.8 9.9 10.3 17.5 27.5 50.0 Δ +10.1 Δ +33.0 PA baseline+goal +insights 0 10 20 30 40 50 60 70 80 52.8 68.2 51.9 ARC tokamak 37.0 44.5 20.4 15.8 23.7 31.6 Δ +15.4 Δ -0.8 Insights Figure 6: Effects of goals and insight questions on mechanism discovery. We evaluate Claude Fable 5.1 with general guidance (+goal) or insight questions (+insights); stacked bars show estimated PA and SI contributions to the conditioned score, with∆ marking changes from baseline. Does additional guidance improve mechanism discovery?We study two types of guidance: enabling the “goal” feature (+goal) and directly including insight questions in the task specification (+insights). We evaluate Claude Fable 5.1, the best-performing agent, on one neuroscience problem (neural dynamics) and one plasma physics problem (ARC tokamak), as shown in Figure 6. In the +goal condition, we specify that a good mechanism should be not only accurate but also useful for future exploration (Appendix B.1). Enabling this feature increases the SI contribution to the conditioned score on neural dynamics (∆ + 10.0%) and ARC tokamak (∆ + 7.9%). Directly providing insight questions in the task specification as explicit optimization targets substantially increases the SI contribution on neural dynamics (∆ + 32.5%) and ARC tokamak (∆ + 15.8%). However, this is not a natural setting: when scientists first observe a phenomenon that challenges their understanding, they do not yet know what insights its underlying mechanism will yield. 01020304050 Insight score (%) Claude Fable 5.1 Claude Opus 5 Claude Opus 4.8 GPT 6 Astra GPT 5.6 Sol DeepSeek V4 Flash Kimi K3 22.742.4 19.536.2 12.226.9 14.829.4 10.025.9 10.224.4 12.225.2 Agent-discovered insightsMechanism-derived insights Figure 7: Insights discovered during the experimentation loop vs. those derived from the discovered mechanism. Both scores use the same problems and instances. How many insights are derived from the discovered mechanism? To assess insights agents derive during experimen- tation, the judge evaluates their submit- ted mechanisms without simulator access (Figure 7). All agents score lower than in the main evaluation, where the judge interprets and extrapolates their mecha- nisms using the simulator. Claude Fable 5.1 scores highest and shows the largest gap: 22.7% from experimentation versus 42.4% from its mechanism. We also ob- serve that, in 6.7% of cases, judges credit insights from experiment logs that they cannot derive from the mechanism. This comparison lets us evaluate insights derived by extrapolating the mechanism separately from insights agents find during experiments. Qualitative analysis: where do agents’ experiments fall short? Our experts review agents’ mechanisms and experiment logs and identify four types of failure that hinder the discovery of novel scientific insights (Figure 8). (1) Figure 8: Expert review of mechanism discovery. Experts review experiment logs of Claude Fable 5.1 and GPT 6 Astra on neuroscience and plasma physics tasks, highlighting failures in mechanism discovery alongside successful hypothesis rejection and understanding of physical concepts. Overreliance on preselected hypotheses and choices. Agents build explanations around unjustified choices. In navigation goals, the agent selects neuronal readouts without establishing whether they capture the relevant circuit and behavior. (2) Treating task optimization as mechanism discovery. Agents equate fitting observations with explaining them. In zigzag foraging, the agent adjusts synaptic strength to match a target response and presents the optimized result as a mechanism. (3) Not using experiments to resolve uncertainty. Claude Fable 5.1 recognizes a possible numerical artifact in ARC tokamak but does not lower the diffusivity floor to test its effect. (4) Stopping at observations without further exploration. GPT 6 Astra observes responses to temperature and density gradients in ARC tokamak but does not explore whether changing the dominant turbulence regime could improve performance. Nevertheless, agents sometimes make sound experimental decisions, such as rejecting mechanisms that contradict the evidence, and recognize the physical meaning of variables and model parameters. These findings suggest that agents are better at task optimization than at open-ended experimentation to resolve scientific uncertainty. We provide detailed and complete descriptions of the common failure modes in Appendix D.3. Common positive behaviors are described in Appendix D.4. 7 Related Work AI scientist agents. AI scientist agents automate research workflows, from reviewing literature and generating ideas to running experiments and writing papers (Lu et al., 2024; Yamada et al., 2025; Schmidgall et al., 2025; Tang et al., 2025). AI agents also review scientific papers (Kim et al., 2026). Recent systems incorporate persistent research memory and connect claims to evidence (Lyu et al., 2026; Meng et al., 2026), while other work improves research capabilities through execution feedback and replication training (Si et al., 2026; Falck et al., 2026). Applications beyond machine learning include laboratory experimentation and hypothesis validation (Boiko et al., 2023; Gottweis et al., 2026; Ghareeb et al., 2026), as well as research that combines literature analysis with large-scale data exploration or multimodal observations (Mitchener et al., 2025; Li et al., 2026). Related systems search for executable algorithms or equations using automated evaluators (Romera-Paredes et al., 2024; Novikov et al., 2025; Shojaee et al., 2025a). For mechanism discovery, agents use experiments to investigate physical and computational systems and revise mechanistic models (Kabra et al., 2026; Murphy, 2026; Wang et al., 2026a,b). Benchmarking scientific discovery. Existing benchmarks evaluate scientific tool use and research workflows (Chen et al., 2024; Bragg et al., 2026; Liu et al., 2026), verifiable outputs and scientific software engineering (Terminal-Bench-Science Team, 2026; Xu et al., 2026), experiment replication (Starace et al., 2025; Kon et al., 2025; Qiu et al., 2026), or gains over numerical baselines (Chan et al., 2024; Nathani et al., 2025; Lupidi et al., 2026; Wang et al., 2026c). DiscoveryBench and LLM-SRBench assess hypotheses and equations recovered from data (Majumder et al., 2024; Shojaee et al., 2025b), while interactive benchmarks let agents gather observations to uncover hidden systems and physical laws (Geng et al., 2025a; Koblischke et al., 2025; Chen et al., 2025b; Zheng et al., 2025; Jain et al., 2026). DiscoveryWorld evaluates explanatory knowledge, and BoxingGym tests whether explanations help another agent predict unseen outcomes (Jansen et al., 2024; Gandhi et al., 2025). Closest to our setting, DiscoverPhysics evaluates both trajectory prediction and conceptual explanations against expert-written rubrics in simulated worlds with noncanonical physics (Wiemann et al., 2026). Unlike benchmarks centered on task completion or numerical performance, we focus on a core capability of scientific discovery: whether agents can uncover scientifically meaningful insights while seeking an answer, rather than merely produce a correct final answer. 8 Conclusion As EUREKABENCH demonstrates, the answer to the question of whether AI agents can understand mechanisms and uncover novel scientific insights is still unclear. While agents excel at optimizing predictive accuracy, on many problems their mechanisms fail to satisfy even basic common-sense assumptions, and they often rely on untested assumptions, mistake task optimization for mechanism discovery, leave uncertainty unresolved, and stop without further exploration. Given these findings, we believe that EUREKABENCH opens up a new path for the scientific study of AI-driven mechanism discovery across a variety of scientific fields, and leaves open a number of new challenges: Multiple paths in scientific discovery. Scientific discovery can follow different paths from the same initial observations, leading to different mechanisms and scientific insights. Our insight questions reflect the research directions selected with domain experts and capture only part of what can be learned about each problem. For example, an agent might find that its mechanism explains another phenomenon or predicts an effect we had not considered when designing the benchmark. These findings would not count toward its SI score unless they answer one of the questions in our rubric. Future work could expand the insight set to cover alternative paths, with domain experts assessing findings that extend beyond established knowledge. Evaluating scientific insights.Our SI evaluation requires judge agents to interpret and extrapolate mechanisms, which means it is also constrained by their own scientific reasoning abilities. Judges may accept confidently stated conclusions without adequately establishing how they follow from the mechanism, potentially overestimating the number of scientific insights it supports. The expert reviews in our qualitative analysis suggest that some automated assessments are overly generous, although these reviews provide broad judgments of selected submissions rather than scores for every insight question. On ARC tokamak, our fusion expert rated Claude Fable 5.1’s submission above the other submissions they had reviewed, but considered its work closer to that of a master’s student in the domain than to that of a researcher. Training agents to discover scientific insights. One way to improve agents’ ability to discover new scientific insights is to build reinforcement learning (RL) environments that train them to do so through experimentation. Rewards could reflect whether agents test their proposed mechanisms and support their scientific insights with experimental evidence. Such training may help agents choose informative experiments and investigate what their mechanisms imply beyond the initial observations. 9 Acknowledgments This work was supported in part by grants from the US Department of Energy and Fujitsu. We thank Sean Welleck for helpful early discussions, and Amanda Bertsch, Lintang Sutawika, Xueqiao Sun, Zedian (Mark) Xiao, and Yixiong Fang for their valuable feedback on the project. We also thank Ruyi Ke from Princeton University’s Department of Chemistry, Yuyang Wu from Carnegie Mellon University’s Department of Chemistry, Yiming Yang from UC Berkeley’s Department of Physics, and Drew Bridges and Seh Na Mellick from Carnegie Mellon University’s Department of Biological Sciences. We thank Christine Kaeser-Chen and the fusion team at Google DeepMind. References Nans Addor, Andrew J Newman, Naoki Mizukami, and Martyn P Clark. The camels data set: catchment attributes and meteorology for large-sample studies. Hydrology and Earth System Sciences, 21(10):5293–5313, 2017. doi: 10.5194/hess-21-5293-2017. Dylan M. Anstine, Qiyuan Zhao, Roman Zubatiuk, Shuhao Zhang, Veerupaksh Singla, Filipp Nikitin, Brett M. Savoie, and Olexandr Isayev. AIMNet2-rxn: A machine learned potential for generalized reaction modeling on a millions-of-pathways scale, 2025. URLhttps://doi.org/10.26434/chemrxiv-2025-hpdmg. ChemRxiv preprint. Anthropic. Claude code, 2026a. URL https://code.claude.com/docs/en/overview. Anthropic. Introducing claude fable 5.1 and claude mythos 5.1, 2026b. URLhttps://w.anthropic. com/claude-fable-and-mythos-5-1. Anthropic.Introducing claude opus 4.8, 2026c.URLhttps://w.anthropic.com/news/ claude-opus-4-8. Anthropic.System card:Claude opus 5, 2026d.URLhttps://w.anthropic.com/news/ claude-opus-5. July 24, 2026. Alan Armstrong, Roberto A. Boto, Paul Dingwall, Julia Contreras-Garc ́ ıa, Matt J. Harvey, Nicholas J. Mason, and Henry S. Rzepa. The houk–list transition states for organocatalytic mechanisms revisited. Chem. Sci., 5(5): 2057–2071, 2014. doi: 10.1039/c3sc53416b. S. Bahmanyar and K. N. Houk. The origin of stereoselectivity in proline-catalyzed intramolecular aldol reactions. Journal of the American Chemical Society, 123(51):12911–12912, December 2001. doi: 10.1021/ja011714s. Igor A. Baratta, Joseph P. Dean, Jørgen S. Dokken, Michal Habera, Jack S. Hale, Chris N. Richardson, Marie E. Rognes, Matthew W. Scroggs, Nathan Sime, and Garth N. Wells. DOLFINx: The next generation FEniCS problem solving environment, 2023. URL https://doi.org/10.5281/zenodo.10447666. Preprint. Sten Bergstr ̈ om.The HBV model—its structure and applications.SMHI Reports Hydrol- ogy4,SwedishMeteorologicalandHydrologicalInstitute,Norrk ̈ oping,Sweden,1992. URLhttps://w.smhi.se/en/publications-from-smhi/publications/ 1992-09-03-the-hbv-model---its-structure-and-applications. Daniil A. Boiko, Robert MacKnight, Ben Kline, and Gabe Gomes. Autonomous chemical research with large language models. Nature, 624(7992):570–578, 2023. doi: 10.1038/s41586-023-06792-0. Jonathan Bragg, Mike D’Arcy, Nishant Balepur, Dan Bareket, Bhavana Dalvi Mishra, Sergey Feldman, Dany Haddad, Jena Hwang, Peter Jansen, Varsha Kishore, et al. Astabench: Rigorous benchmarking of ai agents with a scientific research suite. In International conference on learning representations, volume 2026, p. 110136–110223, 2026. URL https://openreview.net/forum?id=M7TNf5J26u. Zak Buhmann, Amanda K. Robinson, Jason B. Mattingley, and Reuben Rideaux. Inverted encoding of neural responses to audiovisual stimuli reveals super-additive multisensory enhancement. eLife, 13:RP97230, 2026. doi: 10.7554/eLife.97230.3. Jun Shern Chan, Neil Chowdhury, Oliver Jaffe, James Aung, Dane Sherburn, Evan Mays, Giulio Starace, Kevin Liu, Leon Maksin, Tejal Patwardhan, Lilian Weng, and Aleksander M ̨adry. MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineering. arXiv preprint arXiv:2410.07095, 2024. URLhttps: //arxiv.org/abs/2410.07095. Kevin S. Chen, Anuj K. Sharma, Jonathan W. Pillow, and Andrew M. Leifer. Navigation strategies in Caenorhabditis elegans are differentially altered by learning. PLOS Biology, 23(3):e3003005, 2025a. doi: 10.1371/journal.pbio. 3003005. Kevin S. Chen, Jonathan W. Pillow, and Andrew M. Leifer. State-switching navigation strategies in Caenorhabditis elegans are beneficial for chemotaxis. Proceedings of the National Academy of Sciences, 123(25):e2519999123, 2026. doi: 10.1073/pnas.2519999123. Yimeng Chen, Piotr Pie ̧kos, Mateusz Ostaszewski, Firas Laakom, and J ̈ urgen Schmidhuber. PhysGym: Benchmark- ing LLMs in Interactive Physics Discovery with Controlled Priors. arXiv preprint arXiv:2507.15550, 2025b. URL https://arxiv.org/abs/2507.15550. Ziru Chen, Shijie Chen, Yuting Ning, Qianheng Zhang, Boshi Wang, Botao Yu, Yifei Li, Zeyi Liao, Chen Wei, Zitong Lu, Vishal Dey, Mingyi Xue, Frazier N. Baker, Benjamin Burns, Daniel Adu-Ampratwum, Xuhui Huang, Xia Ning, Song Gao, Yu Su, and Huan Sun. ScienceAgentBench: Toward Rigorous Assessment of Language Agents for Data-Driven Scientific Discovery. arXiv preprint arXiv:2410.05080, 2024. URL https://arxiv.org/abs/2410.05080. Anthony Chuma, Hans W. Horn, William C. Swope, Russell C. Pratt, Lei Zhang, Bas G. G. Lohmeijer, Charles G. Wade, Robert M. Waymouth, James L. Hedrick, and Julia E. Rice. The reaction mechanism for the organocatalytic ring-opening polymerization of L-lactide using a guanidine-based catalyst: Hydrogen-bonded or covalently bound? Journal of the American Chemical Society, 130(21):6749–6754, May 2008. doi: 10.1021/ja0764411. Jonathan Citrin, Ian Goodfellow, Akhil Raju, Jeremy Chen, Jonas Degrave, Craig Donner, Federico Felici, Philippe Hamel, Andrea Huber, Dmitry Nikulin, et al. Torax: a fast and differentiable tokamak transport simulator in jax. arXiv preprint arXiv:2406.06718, 2024. URL https://arxiv.org/abs/2406.06718. Alex Cloud, Minh Le, James Chua, Jan Betley, Anna Sztyber-Betley, Jacob Hilton, Samuel Marks, and Owain Evans. Subliminal learning: Language models transmit behavioral traits via hidden signals in data. arXiv preprint arXiv:2507.14805, 2025. URL https://arxiv.org/abs/2507.14805. Bel ́ en Costanza, Bonny Y. Wang, Francisco Villaescusa-Navarro, Alex M. Garcia, Jonah C. Rose, Mark Vogelsberger, Paul Torrey, Arya Farahi, Xuejian Shen, and Ilem Leisher. On the sensitivity of different galaxy properties to warm dark matter. arXiv preprint arXiv:2510.05037, 2025. URL https://arxiv.org/abs/2510.05037. St ́ ephane d’Ascoli, J ́ er ́ emy Rapin, Yohann Benchetrit, Teon Brooks, Katelyn Begany, Jos ́ ephine Raugel, Hubert Banville, and Jean-R ́ emi King. A foundation model of vision, audition, and language for in-silico neuroscience. arXiv preprint arXiv:2605.04326, 2026. URL https://arxiv.org/abs/2605.04326. DeepSeek-AI.Deepseek-v4: Towards highly efficient million-token context intelligence.arXiv preprint arXiv:2606.19348, 2026. URL https://arxiv.org/abs/2606.19348. Raymond L. Dunn, Caitriona M. Costello, Jackson M. Borchardt, Daniel Y. Sprague, Grace C. Chiu, Julia M. Miller, Noelle D. L’Etoile, and Saul Kato. Short-term memory by distributed neural network oscillators in a simple nervous system. Current Biology, 35(22):5582–5593.e4, 2025. doi: 10.1016/j.cub.2025.10.018. Bradley Efron. Bootstrap methods: Another look at the jackknife. The Annals of Statistics, 7(1):1–26, 1979. doi: 10.1214/aos/1176344552. Damon Falck, Samer Sabri, Anja Surina, Thom Foster, Anya Sims, Sam Devlin, Dylan Rogers, Tantum Collins, Kaloyan Aleksiev, Louis Kirsch, et al. Training ai scientists to replicate research. arXiv preprint arXiv:2608.13331, 2026. URL https://arxiv.org/abs/2608.13331. Charles Fieseler, Itamar Lev, Ulises Rey, Lukas Hille, Hannah Brenner, and Manuel Zimmer. An intrinsic neuronal manifold underlies brain-wide hierarchical organization of behavior in C. elegans. bioRxiv, 2025. doi: 10.1101/2025.03.09.642241. Kanishk Gandhi, Michael Y. Li, Lyle Goodyear, Agam Bhatia, Louise Li, Aditi Bhaskar, Mohammed Zaman, and Noah D. Goodman. BoxingGym: Benchmarking Progress in Automated Experimental Design and Model Discovery. arXiv preprint arXiv:2501.01540, 2025. URL https://arxiv.org/abs/2501.01540. Alex M. Garcia, Jonah C. Rose, Paul Torrey, Andrea Caputo, Mariangela Lisanti, et al. The DREAMS project: Disentangling the impact of halo-to-halo variance and baryonic feedback on Milky Way dark matter density profiles. arXiv preprint arXiv:2512.03132, 2025. URL https://arxiv.org/abs/2512.03132. Jiayi Geng, Howard Chen, Dilip Arumugam, and Thomas L Griffiths. Are large language models reliable ai scientists? assessing reverse-engineering of black-box systems. arXiv preprint arXiv:2505.17968, 2025a. URL https://arxiv.org/abs/2505.17968. Jiayi Geng, Howard Chen, Ryan Liu, Manoel Horta Ribeiro, Robb Willer, Graham Neubig, and Thomas L. Griffiths. Accumulating context changes the beliefs of language models. arXiv preprint arXiv:2511.01805, 2025b. URL https://arxiv.org/abs/2511.01805. Ali E. Ghareeb, Benjamin Chang, Ludovico Mitchener, Angela Yiu, Caralyn J. Szostkiewicz, Dmytro Shved, Gavin J. Gyimesi, Jon M. Laurent, Samantha M. Wright, Muhammed T. Razzak, Andrew D. White, Silvia C. Finnemann, Michaela M. Hinks, and Samuel G. Rodriques. A multi-agent system for automating scientific discovery. Nature, 655(8122):497–505, 2026. doi: 10.1038/s41586-026-10652-y. A. H. Glasser. The direct criterion of newcomb for the ideal mhd stability of an axisymmetric toroidal plasma. Physics of Plasmas, 23(7), July 2016. doi: 10.1063/1.4958328. Padraig Gleeson, David Lung, Radu Grosu, Ramin Hasani, and Stephen D Larson. c302: a multiscale framework for modelling the nervous system of caenorhabditis elegans. Philosophical Transactions of the Royal Society B: Biological Sciences, 373(1758):20170379, 2018. doi: 10.1098/rstb.2017.0379. Juraj Gottweis, Wei-Hung Weng, Alexander Daryin, Tao Tu, Petar Sirkovic, Artiom Myaskovsky, Grzegorz Glowaty, Felix Weissenberger, Alessio Orlandi, Dan Popovici, et al. Accelerating scientific discovery with co-scientist. Nature, 655(8122):487–496, 2026. doi: 10.1038/s41586-026-10644-y. Christopher Hansen, Ian G Stewart, Daniel Burgess, Matthew Pharr, Sophia Guizzo, F Logak, Andrew Oakleigh Nelson, and Carlos Paz-Soldan. Tokamaker: An open-source time-dependent grad-shafranov tool for the design and modeling of axisymmetric fusion devices. Computer Physics Communications, 298:109111, 2024. doi: 10.1016/j.cpc.2024.109111. Sture Holm. A simple sequentially rejective multiple test procedure. Scandinavian Journal of Statistics, 6(2):65–70, 1979. URL https://w.jstor.org/stable/4615733. N. T. Howard, P. Rodriguez-Fernandez, J. Hall, M. Muraca, A. Saltzman, A. Ho, J. C. Hillesheim, A. J. Creely, T. H. Eich, T. Body, P. B. Snyder, and C. Holland. Performance and transport in the ARC tokamak. Journal of Plasma Physics, 92:E67, 2026. doi: 10.1017/S0022377826101573. Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022. URL https://openreview.net/forum?id=nZeVKeeFYf9. Moksh Jain, Mehdi Bennani, Johannes Bausch, Yuri Chervonyi, Bogdan Georgiev, Simon Osindero, and Nenad Toma ˇ sev. MaD Physics: Evaluating information seeking under constraints in physical environments. arXiv preprint arXiv:2605.10820, 2026. URL https://arxiv.org/abs/2605.10820. Peter Jansen, Marc-Alexandre C ˆ ot ́ e, Tushar Khot, Erin Bransom, Bhavana Dalvi Mishra, Bodhisattwa Prasad Majumder, Oyvind Tafjord, and Peter Clark. DiscoveryWorld: A virtual environment for developing and evaluating automated scientific discovery agents. arXiv preprint arXiv:2406.06769, 2024. URLhttps: //arxiv.org/abs/2406.06769. Sanchit Kabra, Nikhil Abhyankar, Saaketh Desai, Prasad Iyer, and Chandan K Reddy. Llm-autoscilab: Closed- loop scientific discovery via active experimentation with llms. arXiv preprint arXiv:2605.24043, 2026. URL https://arxiv.org/abs/2605.24043. Maurice G. Kendall. The treatment of ties in ranking problems. Biometrika, 33(3):239–251, 1945. doi: 10.1093/ biomet/33.3.239. Seungone Kim, Dongkeun Yoon, Kiril Gashteovski, Juyoung Suk, Jinheon Baek, Pranjal Aggarwal, Ian Wu, Viktor Zaverkin, Spase Petkoski, Daniel R. Schrider, Ilija Dukovski, Francesco Santini, Biljana Mitreska, Yong Jeong, Kyeongha Kwon, Young Min Sim, Dragana Manasova, Arthur Porto, Biljana Mojsoska, Makoto Takamoto, Marko Shuntov, Ruoqi Liu, Hyunjoo Jenny Lee, Niyazi Ulas Dinc ̧, Yehhyun Jo, Sunkyu Han, Chungwoo Lee, Huishan Li, Esther H. R. Tsai, Ergun Simsek, Khushboo Shafi, Yeonseung Chung, Jihye Park, Aleksandar Shulevski, Henrik Christiansen, Yoosang Son, Elly Knight, Amanda Montoya, Jeongyoun Ahn, Christian Langkammer, Heera Moon, Changwon Yoon, Nikola Stikov, Mooseok Jang, Edward Choi, Junhan Kim, Yeon Sik Jung, Woo Youn Kim, Jae Kyoung Kim, Ishraq Md Anjum, Hyun Uk Kim, Drew Bridges, Carolin Lawrence, Xiang Yue, Alice Oh, Akari Asai, Sean Welleck, and Graham Neubig. On the limits and opportunities of AI reviewers: Reviewing the reviews of Nature-family papers with 45 expert scientists. arXiv preprint arXiv:2605.20668, 2026. doi: 10.48550/arXiv.2605.20668. URL https://arxiv.org/abs/2605.20668. Nolan Koblischke, Hyunseok Jang, Kristen Menou, and Mohamad Ali-Dib. Gravity-Bench-v1: A Benchmark on Gravitational Physics Discovery for Agents. arXiv preprint arXiv:2501.18411, 2025. URLhttps://arxiv. org/abs/2501.18411. Patrick Tser Jern Kon, Jiachen Liu, Xinyi Zhu, Qiuyi Ding, Jingjia Peng, Jiarong Xing, Yibo Huang, Yiming Qiu, Jayanth Srinivasa, Myungjin Lee, Mosharaf Chowdhury, Matei Zaharia, and Ang Chen. EXP-Bench: Can AI Conduct AI Research Experiments? arXiv preprint arXiv:2505.24785, 2025. URLhttps://arxiv.org/ abs/2505.24785. Ilem Leisher, Paul Torrey, Alex M. Garcia, Jonah C. Rose, Francisco Villaescusa-Navarro, Zachary Lubberts, Arya Farahi, Stephanie O’Neil, Xuejian Shen, Olivia Mostow, Nitya Kallivayalil, Dhruv Zimmerman, Desika Narayanan, and Mark Vogelsberger. Linking warm dark matter to merger tree histories via deep learning networks. arXiv preprint arXiv:2511.05367, 2025. URL https://arxiv.org/abs/2511.05367. Bobo Li, Hao Fei, Tianjie Ju, Mong-Li Lee, and Wynne Hsu. OmniScientist: An Omni-Modal Omni-Discipline AI Scientist. arXiv preprint arXiv:2608.13558, 2026. URL https://arxiv.org/abs/2608.13558. Kito Liao, Francisco Villaescusa-Navarro, Romain Teyssier, and Natal ́ ı S. M. de Santi. Cosmology with one galaxy: An analytic formula relatingΩ m with galaxy properties. arXiv preprint arXiv:2602.07651, 2026. URL https://arxiv.org/abs/2602.07651. Tianyu Liu, Allen Xin Wang, Antonia Panescu, Lisa Xinyi Chen, Wenxin Long, Xinyu Wei, Yueqian Jing, Ziyao Zeng, Jihang Chen, Sihan Jiang, Ziqing Wang, Siyi Gu, Siyu Chen, Xinyang Hu, Haoran Shao, Leqi Xu, Wangjie Zheng, Zhiyuan Cao, Ada Fang, Botao Yu, Kunyang Sun, Rex Ying, Arman Cohan, Qingyu Chen, Lingzhou Xue, Kaize Ding, Yuanqi Du, Wengong Jin, Zhuoran Yang, Marinka Zitnik, James Zou, Hua Xu, and Hongyu Zhao. Benchmarking AI Agents for Addressing Scientific Challenges Across Scales. arXiv preprint arXiv:2606.12736, 2026. URL https://arxiv.org/abs/2606.12736. Chris Lu, Cong Lu, Robert Tjarko Lange, Jakob Foerster, Jeff Clune, and David Ha. The AI Scientist: Towards fully automated open-ended scientific discovery. arXiv preprint arXiv:2408.06292, 2024. URLhttps: //arxiv.org/abs/2408.06292. Alisia Lupidi, Bhavul Gauri, Thomas Simon Foster, Bassel Al Omari, Despoina Magka, Alberto Pepe, Alexis Audran-Reiss, Muna Aghamelu, Nicolas Baldwin, Lucia Cipolina-Kun, et al. Airs-bench: a suite of tasks for frontier ai research science agents. arXiv preprint arXiv:2602.06855, 2026. URLhttps://arxiv.org/ abs/2602.06855. Yougang Lyu, Xi Zhang, Xinhao Yi, Yuyue Zhao, Shuyu Guo, Wenxiang Hu, Jan Piotrowski, Jakub Kaliski, Jacopo Urbani, Zaiqiao Meng, Lun Zhou, and Xiaohui Yan. EvoScientist: Towards Multi-Agent Evolving AI Scientists for End-to-End Scientific Discovery. arXiv preprint arXiv:2603.08127, 2026. URLhttps: //arxiv.org/abs/2603.08127. Bodhisattwa Prasad Majumder, Harshit Surana, Dhruv Agarwal, Bhavana Dalvi Mishra, Abhijeetsingh Meena, Aryan Prakhar, Tirth Vora, Tushar Khot, Ashish Sabharwal, and Peter Clark. DiscoveryBench: Towards Data-Driven Discovery with Large Language Models. arXiv preprint arXiv:2407.01725, 2024. URLhttps: //arxiv.org/abs/2407.01725. Rui Meng, Bhavana Dalvi Mishra, Jiefeng Chen, Chun-Liang Li, Palash Goyal, Mihir Parmar, Yiwen Song, Yale Song, Rajarishi Sinha, Parthasarathy Ranganathan, et al. Scientistone: Towards human-level autonomous research via chain-of-evidence. arXiv preprint arXiv:2605.26340, 2026. URLhttps://arxiv.org/abs/2605. 26340. Ludovico Mitchener, Angela Yiu, Benjamin Chang, Mathieu Bourdenx, Tyler Nadolski, Arvis Sulovari, Eric C. Landsness, Daniel L. Barabasi, Siddharth Narayanan, Nicky Evans, et al. Kosmos: An AI scientist for autonomous discovery. arXiv preprint arXiv:2511.02824, 2025. URL https://arxiv.org/abs/2511.02824. Moonshot AI. Kimi k3: Open frontier intelligence, 2026. URL https://w.kimi.ai/blog/kimi-k3. Megan Morrison and Lai-Sang Young. A data-driven biology-based network model reproduces C. elegans premotor neural dynamics. PLOS Computational Biology, 21(12):e1013818, 2025. doi: 10.1371/journal.pcbi.1013818. Aaron Mueller, Atticus Geiger, Sarah Wiegreffe, Dana Arad, Iv ́ an Arcuschin, Adam Belfki, Yik Siu Chan, Jaden Fiotto-Kaufman, Tal Haklay, Michael Hanna, Jing Huang, Rohan Gupta, Yaniv Nikankin, Hadas Orgad, Nikhil Prakash, Anja Reusch, Aruna Sankaranarayanan, Shun Shao, Alessandro Stolfo, Martin Tutek, Amir Zur, David Bau, and Yonatan Belinkov. MIB: A mechanistic interpretability benchmark. arXiv preprint arXiv:2504.13151, 2025. URL https://arxiv.org/abs/2504.13151. Kevin Murphy. Model discovery agent: Llm-assisted bayesian experiment design for data-efficient discovery of mechanistic world models. arXiv preprint arXiv:2608.09696, 2026. URLhttps://arxiv.org/abs/ 2608.09696. Neel Nanda and Joseph Bloom.Transformerlens,2022.URLhttps://github.com/ TransformerLensOrg/TransformerLens. Deepak Nathani, Lovish Madaan, Nicholas Roberts, Nikolay Bashlykov, Ajay Menon, Vincent Moens, Amar Budhiraja, Despoina Magka, Vladislav Vorotilov, Gaurav Chaurasia, Dieuwke Hupkes, Ricardo Silveira Cabral, Tatiana Shavrina, Jakob Foerster, Yoram Bachrach, William Yang Wang, and Roberta Raileanu. MLGym: A New Framework and Benchmark for Advancing AI Research Agents. arXiv preprint arXiv:2502.14499, 2025. URL https://arxiv.org/abs/2502.14499. Alexander Novikov, Ng ˆ an V ̃ u, Marvin Eisenberger, Emilien Dupont, Po-Sen Huang, Adam Zsolt Wagner, Sergey Shirobokov, Borislav Kozlovskii, Francisco J. R. Ruiz, Abbas Mehrabian, M. Pawan Kumar, Abigail See, Swarat Chaudhuri, George Holland, Alex Davies, Sebastian Nowozin, Pushmeet Kohli, and Matej Balog. AlphaEvolve: A coding agent for scientific and algorithmic discovery. arXiv preprint arXiv:2506.13131, 2025. URL https://arxiv.org/abs/2506.13131. OpenAI. Codex, 2026a. URL https://openai.com/codex/. OpenAI. Gpt 5.6: Frontier intelligence that scales with your ambition, 2026b. URLhttps://openai.com/ index/gpt-5-6/. OpenAI. Gpt 6 astra: A new generation of intelligence, 2026c. URLhttps://openai.com/index/ gpt-6-astra/. OpenAI. On the Navier–Stokes Millennium Prize Problem, 2026d. URLhttps://openai.com/index/ navier-stokes-solution/. OpenHands.Acp agents, 2026.URLhttps://docs.openhands.dev/openhands/usage/ agent-canvas/acp-agents. Belinda Phipson and Gordon K. Smyth. Permutation P-values should never be zero: Calculating exact P-values when permutations are randomly drawn. Statistical Applications in Genetics and Molecular Biology, 9(1):Article 39, 2010. doi: 10.2202/1544-6115.1585. Shi Qiu, Junyi Deng, Yiwei Deng, Haoran Dong, Jieyu Fu, Mao Li, Zeyu Li, Zhaolong Zhang, Huiwen Zheng, Lei- dong Bao, et al. Prbench: End-to-end paper reproduction in physics research. arXiv preprint arXiv:2603.27646, 2026. URL https://arxiv.org/abs/2603.27646. Bernardino Romera-Paredes, Mohammadamin Barekatain, Alexander Novikov, Matej Balog, M. Pawan Kumar, Emilien Dupont, Francisco J. R. Ruiz, Jordan S. Ellenberg, Pengming Wang, Omar Fawzi, Pushmeet Kohli, and Alhussein Fawzi. Mathematical discoveries from program search with large language models. Nature, 625(7995): 468–475, 2024. doi: 10.1038/s41586-023-06924-6. Jonah C Rose, Paul Torrey, Francisco Villaescusa-Navarro, Mariangela Lisanti, Tri Nguyen, Sandip Roy, Kassidy E Kollmann, Mark Vogelsberger, Francis-Yan Cyr-Racine, Mikhail V Medvedev, et al. Introducing the dreams project: Dark matter and astrophysics with machine learning and simulations. The Astrophysical Journal, 982(2): 68, 2025. doi: 10.3847/1538-4357/adb8e5. Wieger H. Scheurer and Micha Heilbron. Distinct regimes of spatial prediction across the visual field during natural vision. bioRxiv, 2026. doi: 10.64898/2026.03.27.714859. Samuel Schmidgall, Yusheng Su, Ze Wang, Ximeng Sun, Jialian Wu, Xiaodong Yu, Jiang Liu, Michael Moor, Zicheng Liu, and Emad Barsoum. Agent Laboratory: Using LLM Agents as Research Assistants. arXiv preprint arXiv:2501.04227, 2025. URL https://arxiv.org/abs/2501.04227. Parshin Shojaee, Kazem Meidani, Shashank Gupta, Amir Barati Farimani, and Chandan Reddy. Llm-sr: Scientific equation discovery via programming with large language models. In International Conference on Learning Representations, volume 2025, p. 16054–16085, 2025a. URLhttps://openreview.net/forum?id= m2nmp8P5in. Parshin Shojaee, Ngoc-Hieu Nguyen, Kazem Meidani, Amir Barati Farimani, Khoa D Doan, and Chandan K Reddy. LLM-SRBench: A New Benchmark for Scientific Equation Discovery with Large Language Models. arXiv preprint arXiv:2504.10415, 2025b. URL https://arxiv.org/abs/2504.10415. Chenglei Si, Zitong Yang, Yejin Choi, Emmanuel Cand ` es, Diyi Yang, and Tatsunori Hashimoto. Towards execution- grounded automated ai research. arXiv preprint arXiv:2601.14525, 2026. URLhttps://arxiv.org/abs/ 2601.14525. Signatories of the Declaration on Math and AI. Declaration on math and AI.https://mathandai.org/, 2026. Accessed: 2026-09-30. Xavier Sims, Daniel Angl ́ es-Alc ́ azar, Boon-Kiat Oh, Daisuke Nagai, Jonathan Mercedes-Feliz, Isabel Medlock, Yueying Ni, Christopher C. Lovell, and Francisco Villaescusa-Navarro. CAMELS environments: The impact of local neighbours on galaxy evolution across the SIMBA, IllustrisTNG, ASTRID, and Swift-EAGLE simulations. arXiv preprint arXiv:2601.06290, 2026. URL https://arxiv.org/abs/2601.06290. Giulio Starace, Oliver Jaffe, Dane Sherburn, James Aung, Jun Shern Chan, Leon Maksin, Rachel Dias, Evan Mays, Benjamin Kinsella, Wyatt Thompson, et al. Paperbench: evaluating ai’s ability to replicate ai research. arXiv preprint arXiv:2504.01848, 2025. URL https://arxiv.org/abs/2504.01848. Laura M. Stoinski, Talia Konkle, and Martin N. Hebart. Principles of coarse-scale functional organization in occipitotemporal cortex. bioRxiv, 2026. doi: 10.64898/2026.02.18.706551. Jiabin Tang, Lianghao Xia, Zhonghang Li, and Chao Huang. AI-Researcher: Autonomous Scientific Innovation. arXiv preprint arXiv:2505.18705, 2025. URL https://arxiv.org/abs/2505.18705. TerenceTao.Postonmathstodon,2026.URLhttps://mathstodon.xyz/@tao/ 117208617602946453. Posted September 3, 2026. Terminal-Bench-Science Team. Terminal-Bench-Science: Evaluating AI agents on research workflows across scientific domains, 2026. URL https://doi.org/10.5281/zenodo.22110253. Version 0.1. K. E. Thome, M. E. Austin, S. Coda, A. Marinoni, A. Hyatt, A. O. Nelson, T. Odstr ˇ cil, C. A. Paz-Soldan, O. Sauter, F. Scotti, and B. Vanovac. The negative triangularity tokamak path for fusion pilot plants: Experimental progress and future prospects. arXiv preprint arXiv:2606.26513, 2026. URLhttps://arxiv.org/abs/2606. 26513. Megan Taylor Tillman, Blakesley Burkhart, Chiara M. F. Mingarelli, J. Andrew Casey-Clyde, Lars Hernquist, Sownak Bose, Enik ̋ o Reg ̋ os, and C ́ esar Hern ́ andez-Aguayo. Implications of the nanoHertz gravitational-wave background for galactic feedback and massive black hole growth. arXiv preprint arXiv:2602.15938, 2026. URL https://arxiv.org/abs/2602.15938. Francisco Villaescusa-Navarro, Daniel Angl ́ es-Alc ́ azar, Shy Genel, David N Spergel, Rachel S. Somerville, Romeel Dave, Annalisa Pillepich, Lars Hernquist, Dylan Nelson, Paul Torrey, et al. The camels project: Cosmology and astrophysics with machine-learning simulations. The Astrophysical Journal, 915(1):71, 2021. doi: 10.3847/ 1538-4357/abf7ba. Hanyang Wang, Yimo Cai, Weiliang Chen, Jiawei Chi, Haowen Sun, Qiyu Dai, Yi-Hsin Hung, Xingzhuo Guo, Jinshan Ren, Runmao Yao, Ziwei Liu, Mingsheng Long, Yueqi Duan, Jun Gao, Jiangran Lyu, Fangfu Liu, and Jialong Wu. Code as Worlds: Agentic discovery of executable world representations for physical reasoning. arXiv preprint arXiv:2608.27549, 2026a. URL https://arxiv.org/abs/2608.27549. Mengru Wang, Junfeng Fang, Shuofei Qiao, Zhenqian Xu, Haoming Xu, Haoxiong Wang, Shumin Deng, Linyi Yang, Zhixiang Cui, Xin Xu, et al. Mechanist: Ai as a scientific instrument for discovering the mechanisms of intelligence. arXiv preprint arXiv:2608.12036, 2026b. URL https://arxiv.org/abs/2608.12036. Xingyao Wang, Simon Rosenberg, Juan Michelini, Calvin Smith, Hoang Tran, Engel Nyst, Rohit Malhotra, Xuhui Zhou, Valerie Chen, Robert Brennan, and Graham Neubig. The openhands software agent sdk: A composable and extensible foundation for production agents. arXiv preprint arXiv:2511.03690, 2025a. URL https://arxiv.org/abs/2511.03690. Yongji Wang, Ching-Yao Lai, David J. Prior, and Charlie Cowen-Breen. Deep learning the flow law of Antarctic ice shelves. Science, 387(6739):1219–1224, 2025b. doi: 10.1126/science.adp3300. Yuru Wang, Lejun Cheng, Yuxin Zuo, Sihang Zeng, Bingxiang He, Che Jiang, Junlin Yang, Yuchong Wang, Kaikai Zhao, Weifeng Huang, et al. Naturebench: Can coding agents match the published sota of nature-family papers? arXiv preprint arXiv:2606.24530, 2026c. URL https://arxiv.org/abs/2606.24530. Matt L. Wiemann, Lindsay M. Smith, Peter Melchior, Siddharth Mishra-Sharma, Andrew Gordon Wilson, Pavel Izmailov, and Carolina Cuesta-L ́ azaro. DiscoverPhysics: Benchmarking LLMs for out-of-the-box scientific thinking. arXiv preprint arXiv:2605.26087, 2026. URL https://arxiv.org/abs/2605.26087. Zhipeng Xu, Jiahao Lu, Yining Zheng, Yuxin Wang, and Xipeng Qiu. Swe-bench science: Can coding agents resolve engineering tasks in science? arXiv preprint arXiv:2608.19799, 2026. URLhttps://arxiv.org/ abs/2608.19799. Yutaro Yamada, Robert Tjarko Lange, Cong Lu, Shengran Hu, Chris Lu, Jakob Foerster, Jeff Clune, and David Ha. The ai scientist-v2: Workshop-level automated scientific discovery via agentic tree search. arXiv preprint arXiv:2504.08066, 2025. URL https://arxiv.org/abs/2504.08066. Tom Zahavy. Position: LLMs can’t jump. In Forty-third International Conference on Machine Learning Position Paper Track, 2026. URL https://openreview.net/forum?id=klU4737opt. Mengdi Zhao, Ning Wang, Xinrui Jiang, Xiaoyang Ma, Haixin Ma, Gan He, Kai Du, Lei Ma, and Tiejun Huang. An integrative data-driven model simulating C. elegans brain, body and environment interactions. Nature Computational Science, 4:978–990, 2024. doi: 10.1038/s43588-024-00738-w. Qiyuan Zhao and Brett M. Savoie. Algorithmic explorations of unimolecular and bimolecular reaction spaces. Angewandte Chemie International Edition, 61(46), October 2022. doi: 10.1002/anie.202210693. Tianshi Zheng, Kelvin Kiu-Wai Tam, Newt Hue-Nam K. Nguyen, Baixuan Xu, Zhaowei Wang, Jiayang Cheng, Hong Ting Tsang, Weiqi Wang, Jiaxin Bai, Tianqing Fang, Yangqiu Song, Ginny Y. Wong, and Simon See. NewtonBench: Benchmarking Generalizable Scientific Law Discovery in LLM Agents. arXiv preprint arXiv:2510.07172, 2025. URL https://arxiv.org/abs/2510.07172. Faxin Zhou, Siyuan Zhou, Yuhang Long, Adeen Flinker, and Chunming Lu. Hierarchical linguistic predictions and cross-level information updating during narrative comprehension. Communications Biology, 9:107, 2026. doi: 10.1038/s42003-025-09377-x. Mingchen Zhuge, Changsheng Zhao, Dylan Ashley, Wenyi Wang, Dmitrii Khizbullin, Yunyang Xiong, Zechun Liu, Ernie Chang, Raghuraman Krishnamoorthi, Yuandong Tian, et al. Agent-as-a-judge: Evaluate agents with agents. arXiv preprint arXiv:2410.10934, 2024. URL https://arxiv.org/abs/2410.10934. A Task Specification Template Task specification ContextYou are a systems neuroscientist studyingproblemdescription. scientific backgroundobservationsblock Initial observationsHowever, it is unclearinitialobservations. TaskYour task is to discover the mechanism that explains this phenomenon. mechanismrequirements To discover this, you interrogate environmentinterfacethrough controlled numerical experiments designed to identify experimentalobjectives. The structure of your predictive form needs to follow from the mechanism you uncover: for each ingredient, explain why it is included, what it traces in the circuit, and how it shapes the neuron’s activity. EvaluationEvery claim in your discovered mechanism needs to be backed by data you actually collected. predictiveaccuracyevaluationYour experimental findings will also be reviewed by an independent researcher who has access to the same simulator, to assess whether the mechanism you discover can yield new insights. DeliverablesYou need to produce three deliverables. If any of them is missing or does not follow the requirements below, your solution receives a score of 0. Deliverable 1: Write your discovered mechanism tooutputdir/mechanisms/mechanism.md. State the mechanism as the causal chain from the coarse action state of the population to the activity of one neuron; its executable form, with every ingredient and fitted constant defined and the physical process each one traces identified; why its inputs jointly determine the neuron’s activity; the concrete evidence from your experiments that supports the mechanism and its form, including the evidence that let you reject the competing explanations you considered; and the insights about how population activity organizes the activity of individual neurons that follow from what you found. Deliverable 2: Write the executable form of your mechanism tooutputdir/mechanisms/mechanism.py. executableinterfaceandrequirements Deliverable 3: Keep an experiment log atoutputdir/mechanisms/experiment.log. Treat it as your laboratory notebook: it is the primary reference for the researchers who will continue this work, and you choose how to organize it. Record what they will need to follow and extend your investigation: the questions you pursued, the simulator configurations and analyses you ran, what you observed, the conclusions you drew, and the approaches that did not work and why. Give particular attention to the findings that surprised you or that you found interesting along the way, whether or not they entered your final mechanism; these are often the most valuable part of a notebook for the next researcher. Move the useful evidence behind the log’s findings — the figures, analysis outputs, and small data files your experiments produced intmpdir, and the observations fromoutputdir/observations/ that your final claims build on — into outputdir/mechanisms/so that they survive for these researchers: everything outside outputdir/mechanisms/ is deleted when the session ends. Saved evidenceYou may additionally place supporting evidence inoutputdir/mechanisms/ and reference it from the files above: figures, and any analysis outputs that back your claims, such as fitted-model diagnostics, ablation results, or small data tables worth preserving. Only this folder is kept when the session ends; everything else in the workspace, including outputdir/observations/,tmpdirand the simulation outputs you generated, is discarded. The independent researcher reviewing your mechanism sees exactly this folder. Everything underoutputdir/mechanisms/ combined is limited to 1 GB: if the folder exceeds this limit, your solution receives a score of 0. It is for final deliverables and their evidence, not scratch space; keep drafts and intermediate files intmpdir. BudgetYou have a time budget oftimebudget. You can usetmpdir for notes and intermediate files. Table 3: Task specification template for the scientist agent (neuroscience). B Guidance Analysis Prompts B.1 Goal Prompt ProblemGoal Neural dynamicsA good mechanism is not only accurate but also extrapolatable, and it can even open up more interesting new research directions. Through such a mechanism people come to understand how the nervous system of C. elegans organizes its activity into a hierarchical locomotion structure, down to how the coarse action state shapes a single neuron. ARC tokamakA good mechanism is not only accurate but also extrapolatable, and it can even open up more interesting new research directions. Through such a mechanism people come to understand how the transport a model returns sets the profiles an ARC plasma settles into and the fusion power it produces, down to why two models that leave the plasma with the same energy confinement disagree by a factor of two in power. Table 4: Goal guidance used in the +goal condition. C Agentic Judge Prompts C.1 Scientific Constraint Evaluation Scientific constraint judge instructions ContextYou are a systems neuroscientist reviewing another neuroscientist’s discovered mechanism for problem description. The neuroscientist interrogatedenvironmentinterfacethrough controlled simulations to discover that mechanism. Verdict instructionYour task is to judge whether this mechanism is consistent with commonly accepted assumptions about C. elegans. The criteria you need to check are listed below: Commonly accepted assumptions (Scientific Constraints) According to the discovered mechanism: scientific constraintscriteria Each of these criteria is a general assumption, not something the neuroscientist is expected to prove. Judge each one by reading the mechanism described in mechanism.md together with its executable form. A criterion should be marked as false if the mechanism contradicts the assumption, and true if it does not. Focus on what the mechanism actually does, not on the names given to its quantities; calling a quantity something is not enough to show that it behaves that way. You also need to mark the criterion as false if the experimental results violate the assumption and the neuroscientist neither recognizes those results as erroneous nor accounts for the violation in the final mechanism. OutputYou are expected to provide the following: Writeoutputdir/judgeoutputas a JSON file with exactly the structure below, with one entry for each criterion ID listed above and nothing else: "scientificconstraints": "SC1": "verdict": true, "reason": "..." Each verdict should be either true or false, with no other values allowed. Each reason should be no more than two sentences and should identify the part of the submission that supports the verdict. Keepoutputdir/judgenotes.md alongside the JSON file and use it to record what you reviewed for each criterion. Save both files instead of printing their contents, then stop. Table 5: Example instructions for scientific constraint tests (neuroscience). C.2 Judge Instructions for Insights Derived from the Mechanism Judge instructions for insights derived from the mechanism ContextYou are a systems neuroscientist reviewing another neuroscientist’s discovered mechanism for problemdescription. The neuroscientist interrogatedenvironmentinterfacethrough controlled simulations to discover that mechanism. Verdict instructionA good mechanism is not only accurate but also far more interpretable and extrapolatable, and it can be used to derive insights. Your task is strictly to evaluate whether this discovered mechanism is able to derive the interpretable scientific insights listed below. Scientific insight questions insights criteria To derive each insight, you need to start from the discovered mechanism and use environmentinterface to check whether the insight can be derived from it. A Yes verdict only occurs when the discovered mechanism is able to help you find the evidence needed to provide a concrete and complete explanation of each insight question. A Yes verdict requires the step-by-step chain by which the discovered mechanism’s own components produce the insight in the nervous system itself; a result that holds only because of the stimuli chosen for the experiments, or that is only observed in the simulator or the saved data, is not such a derivation. You should give a No verdict for an insight question if the answer is based on your prior knowledge or is not fully supported by the discovered mechanism. Any assertion or conclusion about the insights in mechanism/ is never, by itself, grounds for a Yes verdict; verify that the mechanism derives the same conclusion. If you are not able to provide sufficient evidence showing that this mechanism is able to derive the insight, the verdict should be No. You must not modify or extend the discovered mechanism. If the insight question cannot be completely derived from the discovered mechanism, the verdict should be No. An answer based only on your own knowledge of the animal or on the simulator, but not supported by the discovered mechanism, should also be No. OutputYou are expected to provide the following: Writeoutputdir/judgeoutputas a JSON file with exactly the structure below, with one entry for each criterion ID listed above and nothing else: "insights": "I1": "verdict": "Yes", "reason": "..." Each verdict should be either Yes or No, with no other values allowed. For each insight criterion, when you give a Yes verdict, you must provide complete evidence and reasoning explaining why the discovered mechanism can derive the insight asked about in the insight question; otherwise, the verdict should be No. Keepoutputdir/judgenotes.md alongside the JSON file and use it to record what you reviewed and any commands you ran. Save both files instead of printing their contents, then stop. Table 6: Example judge instructions for insights derived from the mechanism (neuroscience). The judge returnsYesonly with complete evidence showing how the insight can be derived from the discovered mechanism. C.3 Judge Instructions for Insights Discovered During Experimentation Judge instructions for insights discovered during experimentation ContextYou are a systems neuroscientist reviewing another neuroscientist’s discovered mechanism for problemdescription. The neuroscientist interrogatedenvironmentinterfacethrough controlled simulations to discover that mechanism. Verdict instructionYour task is to evaluate whether the scientific insight captured by each question below was actually discovered by the neuroscientist while uncovering the mechanism. Consider the questions one at a time. For each question, determine whether the neuroscientist obtained the corresponding insight and whether the available evidence is strong enough to support it. The scientific insight questions are listed below: Scientific insight questions insights criteria When you judge each insight question, do not modify or extend the discovered mechanism, and do not look for a different one. The neuroscientist needs to have explicitly stated the insight and run an experiment specifically to test it. Experiments run for other purposes, such as building, tuning, or checking the executable form of the mechanism, do not count as tests even if their results bear on the question. For questions asking why something happens, showing that it happens is not enough; the explanation itself needs to have been tested. If the neuroscientist did not discover the insight while uncovering the mechanism, or if the discovered mechanism and experiment record do not provide sufficient evidence that the insight was obtained during this process, the answer should be No. If the insight needs to be inferred from their numbers or their argument needs to be completed for them, the answer should be No. An answer based only on your own knowledge of the animal, but not supported by a finding the neuroscientist recorded, should also be No. If the mechanism clearly conflicts with that insight, the answer should also be No. A multi-part question is answered only if every part is answered. Each underlying insight can earn only one Yes verdict. Answers based on the same explanation count as the same insight regardless of the quantities or experiments involved, so mark as Yes only the question that explanation answers most directly and the rest No. When in doubt, the answer should be No. OutputYou are expected to provide the following: Writeoutputdir/judgeoutputas a JSON file with exactly the structure below, with one entry for each criterion ID listed above and nothing else: "insights": "I1": "verdict": "Yes", "reason": "..." Each verdict should be either Yes or No, with no other values allowed. Each reason should be no more than two sentences. A Yes verdict has to quote the neuroscientist’s own statement of the insight and the measurement from the experiment that tested it, and has to say what the neuroscientist ran that experiment for; otherwise, the verdict is No. Keep output dir/judgenotes.md alongside the JSON file and use it to record what you reviewed for each criterion. Save both files instead of printing their contents, then stop. Table 7: Example judge instructions for insights discovered during experimentation (neuroscience). D Additional Results D.1 How do predictive accuracy and scientific insights vary across domains? 1234567891011121314151617181920212223242526 Claude Fable 5.1 Claude Opus 5 Claude Opus 4.8 GPT 6 Astra GPT 5.6 Sol DeepSeek V4 Flash Kimi K3 Neuroscience Geophysics – Plasma physicsAstrophysics Computer science Chemistry 1234567891011121314151617181920212223242526 Claude Fable 5.1 Claude Opus 5 Claude Opus 4.8 GPT 6 Astra GPT 5.6 Sol DeepSeek V4 Flash Kimi K3 Neuroscience Geophysics – Plasma physicsAstrophysics Computer science Chemistry 0%25%50%75%100% (a) Predictive accuracy (%) (b) Insights (%) Problem Figure 9: PA and SI scores across domains. Predictive accuracy (a) and SI scores (b) for seven AI agents on 26 problems across six domains. Each cell shows one agent’s score on one problem, averaged over its instances using the same rules as Table 2. Both panels use the same color scale. Gray dashes mark the excluded Kimi K3 run on catchment properties. Figure 9 compares PA and SI scores across the six domains. On the problems in EUREKABENCH, both scores are relatively low in neuroscience and computer science. Agents achieve higher PA in geophysics, plasma physics, and chemistry, but their SI scores often remain much lower. Differences also emerge among agents with similar PA within a domain. In astrophysics, the three Claude agents and two GPT agents achieve domain-average PA of 67.1%–69.8%, while their SI scores range from 33.6% to 63.4%. These results show that agents with similar PA can still differ substantially in the scientific insights their mechanisms support. D.2 Does higher accuracy imply more insights? We compare agents’ PA and SI rankings within each problem using Kendall’sτ b (Kendall, 1945) (Table 8). We estimate 95% confidence intervals from 20,000 bootstrap samples of whole problems (Efron, 1979). The mean correlation is positive but limited (τ b = 0.20, adjustedp = 0.008). After excluding pairs tied on either metric, the higher-PA agent scores lower on SI in 39.9% of comparisons on average across problems. The reversal rate remains similar when we use either judge’s scores separately. Predictive accuracy alone therefore does not reliably indicate how much scientific insight a mechanism supports. Judgeτ b Reversals (%)p Both judges (average) 0.20 [0.07, 0.32] 39.9 [33.2, 46.5] 0.008 Claude Opus 4.8 0.15 [0.03, 0.26] 41.8 [35.3, 48.4] 0.054 GPT 5.6 Sol 0.15 [0.01, 0.29] 42.8 [34.3, 51.9] 0.054 Table 8: PA and SI rankings. Brackets show 95% bootstrap confidence intervals;p-values are computed using 100,000 within-problem permutations under the null hypothesis of no association and adjusted across the three rows using Holm’s method (Phipson & Smyth, 2010; Holm, 1979). D.3 Common failures of AI scientist agents The ten categories below describe failures in the investigations and explanations of AI scientist agents. Highlighted passages show the actions and claims relevant to each failure. Missing tests of the target behavior.AI scientist agents investigate changes in internal activity without simulating or measuring the target behavior itself. The missing test is whether the proposed mechanism actually reproduces the target behavior, including its timing and response to changing inputs. Missing tests of the target behavior (negative) “A bout ends when the drive drops below the level threshold, not by fatigue. With an oscillating drive (a head swing) this happens on every cycle at a fixed phase, so the bout duration is set by how long the sampled concentration stays above the level threshold: the worm turns for as long as the odor it samples is "high" and stops turning as soon as it falls.Coming out of a bout pointing the right way needs no stored direction: the last direction in which the sampled drive dropped below the level threshold is, by construction, the direction in which the bout stopped.” C. elegans (Navigation strategies) — Claude Fable 5.1 — mechanism.md “The steering read-out’s sign depends on which SMD side RIA inhibits more; a left/right asymmetric drive (asym) still gave a dorsoventral bias of the same sign.A true left/right steering read-out was not examined.” C. elegans (Navigation strategies) — Claude Fable 5.1 — experiment.log “No navigation mechanism has been established. This is a checkpoint, not a successful scientific or executable submission.” “At session start /workspace/observations/ was empty. The available simulator API returns per-cell voltage and calcium, not arrays named reorientation or steering. Its documentation does not define the target network, target readout cells, the algebra that combines their activity into either signal, the underlying variable and units, or the initialization and precise kernel of the 200 ms causal filter. The task specifies neither these definitions nor reference recordings. Clarification has been requested.” “Two C1 simulations in an explicitly chosen 22-cell circuit compare a bilateral 5 pA AWC pulse from 500 to 1000 ms with an otherwise identical unstimulated control. The 118-connection network, default parameters, stimuli, and reproduction instructions are recorded in experiment.log. The raw files and analysis are in evidence/.” “At 1100–1200 ms, input current is zero in both runs, yet AIYL voltage differs by 1.364805 mV and AVAL by 0.137905 mV. An instantaneous current-only mapping cannot explain these raw cell voltages. At 1400–1500 ms the differences are 0.001550 and 0.000126 mV; by 1900–2000 ms both are zero at the saved numerical precision. Thus this particular pulse produces a transient history dependence; it supplies no evidence for a persistent action-selection state lasting seconds. This does not reject persistent states under other stimuli, in other circuits, or with the unknown target readout.” “The experiment does not localize that history dependence to synapses, membrane channels, calcium, or recurrence. It does not measure either requested action signal, test turn termination, identify a steering/turning gate, or demonstrate navigation. No fitted constants, out-of-protocol validation, or comparisons with the two required reduced baselines are available. Assigning any such interpretation would exceed the evidence.” C. elegans (Navigation strategies) — GPT 6 Astra — mechanism.md Figure 10: Missing tests of the target behavior. Original outputs from AI scientist agents. Missing tests of the target behavior (negative)(continued) “The whole-body reference (simulate_body) was not run(25 s of wall clock per simulated ms); the FW circuit at C2 is driven by pulses into AVB, not by a sensory sample, so it does not bear on this pathway — but the muscle caConc of a run of simulate_network with muscles includedwould show whether the 0.1–2 mV sub-threshold ripple reaching DB/VB is enough to modulate the muscle activation(the DB/VB→muscle synapses have the same release curve, so probably not: only the events would).” C. elegans (Zigzag foraging) — Claude Fable 5.1 — experiment.log “Finally, these experiments support a circuit mechanism for the recorded motor voltages, with a clear boundary on claims about behavior. Under the tested sensory drive, dorsal and ventral motor responses were predominantly co-modulated, not a self-generated alternating traveling wave. Holding both AVB cells at 15 pA in this circuit produced late constant DB1/DB2/DB3 and VB1/VB2/ VB3 voltages, with zero peak-to-peak variation at recording precision. The supplied interface describes its separate body protocol as using C2 and additional prescribed motor/muscle drive;that body protocol was not run here.No body trajectory, proprioceptive signal, odor-concentration measurement, or independently measured head angle is delivered to this executable. The supplied biological observations about odor adaptation, concentration-invariant interneuron pulses, and tens-of-seconds turning suppression therefore cannot be inferred from these injected-current voltage recordings alone.” “The resulting insight is that sensory steering signals can exploit an electrically loaded motor network in both anatomical directions: a body neuron can provide the principal entry to a head response. Removing a gap may strengthen the upstream signal while abolishing useful downstream modulation. Because identical cells change their phase response with operating voltage, the body’s sampling rhythm can be transformed without a dedicated per-cell phase label. Demonstrating actual up-gradient steering would additionally require closing and testing the sensory–posture loop; assigning such behavior to the present voltage relay would go beyond the observations.” C. elegans (Zigzag foraging) — GPT 6 Astra — mechanism.md Figure 10: Missing tests of the target behavior (continued). Explaining only part of the target phenomenon.An explanation can account for how one component responds while leaving unexplained how components coordinate, how a state begins or ends, or how stored information changes an outcome. AI scientist agents develop local descriptions but do not connect them into an account of the complete process they set out to explain. Explaining only part of the target phenomenon (negative) “Surprise 1: the network at C1 is either graded or globally spiking, with little in between. 3 pA sines into the 18 B-class motor neurons recruit ~all 114 cells into synchronous Ca spikes every second cycle (E3 fast-only, fig/rec1_fast_traces.png), including cells with no direct contact from any driven cell. Command epochs disappear under it. Everything first-order about the mechanism lives in the graded regime (rhythm <= ~2.5 pA, command <= 6 pA);corpus amplitudes were chosen accordingly.A useful follow-up: map the recruitment threshold as a function of the number of driven cells and their contacts.” C. elegans (Neural dynamics) — Claude Fable 5.1 — experiment.log “cycle_runs.jsonl: 29-cell full network; duration 4200 ms; dt .01 ms; saved each 1 ms. Six 600 ms source epochs from 300 ms: forward-source group AVBL/AVBR/PVCL, backward-source group AVAL/AVAR/AVEL, head-source group SMDDL/SMDVL; repeat the sequence twice. The active group’s cells receive 5 pA. The faster input is a continuous 3 pA, 150 ms sine into DB2 and VD2, with VB2 in antiphase, from 300 to 3900 ms. Repeat as combined, coarse_only and fast_only actual simulations, recording new driver calcium for each.” “ This is an IMPOSED source sequence, not evidence for an endogenous low-dimensional attractor or actual locomotion; no body simulation was run. Final fixed mechanism R2: combined .770325, coarse-only .645901, fast-only .883603. Thus both physical drive-removal conditions are predicted with the same computation and constants. The actual combined recipient recording differs from the sum of the two individual-drive responses plus baseline: interaction L2 / combined-baseline L2 = .367057 in 300–3900 ms. Source nonlinearity and recipient integration can both contribute to this full-network interaction; the independent two-source experiment isolates recipient convergence more cleanly. Figure: figures/command_sequence.png. Data/predictions: reports/cycle_data_pred.npz, metrics reports/cycle_frequency_report.json.” C. elegans (Neural dynamics) — GPT 6 Astra — experiment.log “This is a mechanism for transmitting and expressing a population state at an undriven recipient.The experiments do not establish the origin of the animal’s spontaneous command-state trajectory.They also expose a limitation of the supplied interface: it omits time-varying inputs from neurons outside the designated source set. The executable is a quantitatively tested approximation to the local circuit, with a baseline correction for omitted resting input and reduction error. A matched hidden-pathway intervention produced exactly identical supplied inputs but different recipient activity, so universal sufficiency of this interface is experimentally ruled out across such circuit changes. Its strong performance on circuits with the observed incoming pathways must be distinguished from its weaker performance in a recurrent network with omitted pathways. All numerical evidence below comes from simulations collected in this session; the biological observations in the question are motivation, not additional data collected here.” C. elegans (Neural dynamics) — GPT 6 Astra — mechanism.md Figure 11: Explaining only part of the target phenomenon. Original outputs from AI scientist agents. Explaining only part of the target phenomenon (negative)(continued) “What a trained animal does differently to the same odour is, in this model, entirely a matter of how much and for how long each downstream path is depolarised per sensory event (gain and decay), plus a small shift of the sensory threshold by feedback;the timing of the decisions is inherited from the sensory cell’s excitability and is not learned.” C. elegans (Biological networks (learning)) — Claude Fable 5.1 — mechanism.md “This is an unvalidated hypothesis, not a discovered mechanism. The requested causal identification could not be completed from the supplied environment. The observation directory was initially empty. The accessible simulator takes a chosen circuit and arbitrary manipulations, but does not take the learning-state index described in the task. Its outputs contain cell voltage and calcium, not the defined reorientation and steering signals. The cells, observable, combination, normalization and initialization used for those signals are unspecified.No learning-state/action-signal pairs have been observed.The executable therefore provides a refittable candidate for subsequent testing, with unfitted defaults; it must not be interpreted as a successful solution or validated predictor.” “The initial time constants are 50 and 500 ms; they are arbitrary starting hypotheses, not estimates. All initial output weights are zero. Before fitting, the executable therefore predicts zero for both outputs.No constants were fitted to actual action recordings, because none were supplied.” C. elegans (Biological networks (learning)) — GPT 6 Astra — mechanism.md “Headswing = mean v(SMDD) - mean v(SMDV) through a causal 200 ms first-order filter; turn = mean over the 200 ms after the command minus mean over its last 300 ms.” “Read-out: releasing the command samples the phase. When AVA stops, RIA’s excitation collapses within ~150 ms (synaptic decay) and the clamp releases; the headswing falls from the clamp level toward rest. Where in the cycle the release happens sets the level from which it falls and how the 200 ms read-out filter integrates it, so the turn signal is a smooth function of the phase held at release (fig2: phase 0.12 -> -0.4 mV, 0.22 -> -1.4, 0.45 -> -2.5, 0.56 -> -2.1, 0.90 -> +0.9). The release drop of the dorsal offset makes most turns dorsal-negative, but a command ending just before an RIA firing (phase ~0.9, reached for instance by the 12 s command after a dorsal-phase pulse) ends with a positive turn: the sign of the turn is the phase of the clock at release(fig2, fig5).” C. elegans (Phase memory) — Claude Fable 5.1 — mechanism.md Figure 11: Explaining only part of the target phenomenon (continued). Explaining only part of the target phenomenon (negative)(continued) “A_dur12000_D (12 s command, 23 RIA peaks): intervals 495, 505, 510, 510, ... (mean 508.4 ms, sd 3.5 ms = the 5 ms sampling) — no drift over 23 cycles. Phase at release 0.904 (release just before an RIA peak, SMDs at their most depolarised) and the turn is POSITIVE (+0.91 mV): the first sign flip observed. The sign of the turn is set by the phase at release, anda long reversal can end on either side.fig2 updated with this point.” C. elegans (Phase memory) — Claude Fable 5.1 — experiment.log “Butpreserving phase information and preserving an action choice are distinct requirements. In this circuit a common advance of the absolute phase changes the release readout even while the pulse-induced phase separation survives. Explaining the animal’s duration-invariant turn would therefore require an additional property not demonstrated here—for example a readout of relative phase against an internal reference, or a stable lateralized choice state. These are testable future hypotheses, not discovered ingredients. A useful next experiment would compare matched pulse/ no-pulse trials across dense release phases while recording a candidate reference signal; a valid invariant readout must preserve the sensory-dependent choice despite the observed progression of absolute phase. The present results identify a concrete, prospectively predictive phase-memory circuit and expose precisely why it is insufficient as the complete biological explanation posed in the question.” C. elegans (Phase memory) — GPT 6 Astra — mechanism.md “A locomotor "state" in this model is a network burst, not an attractor.The premotor group is an excitable medium: one presynaptic spike from any of the ~170 input cells ignites all 12 cells within ~100 ms; recurrent chemical excitation holds and spreads the burst for 150-250 ms; the cells’ own Ca-inactivation/K currents end it. Holding beyond that requires the source to keep transmitting (bursting or plateau), so the duration of a state is the duration of its input’s supra-threshold activity.” C. elegans (Premotor states) — Claude Fable 5.1 — mechanism.md Figure 11: Explaining only part of the target phenomenon (continued). Explaining only part of the target phenomenon (negative)(continued) “FLPL 7 pA, RIFR 9 pA, PLML 8 pA held constant for 4.5 s (all three in their bursting regime). Each driver bursts strictly periodically: FLPL period 79 ms (CV 0.05), RIFR 69 ms (CV 0.00), PLML 73 ms (CV 0.00) — three incommensurate clocks. The panel response is irregular: AVAL burst intervals 69-375 ms (CV 0.55), AVBR 24-138 ms (CV 0.35), AVEL 142-695 ms (CV 0.47), PVCL 77-162 ms (CV 0.22), with no noise anywhere in the simulation. Group means stay flat (reversal group 3-4e-7, forward group 2-3e-7 in every 250 ms bin):the irregularity is in the timing of individual bursts, not in a slow alternation of group states.INTERESTING: this is the c302 counterpart of "irregular switching from deterministic dynamics" — quasi-periodic forcing by several bursting inputs — but it lives at 0.1-0.7 s, not at the seconds scale, and it never produces alternation between the two groups.” “PLM trigger at 1000 ms switches the core on (panel-mean 4e-7). DVC driven to plateau (+11 mV, 3000-3500 ms; DVC->AVAR 7, ->AVAL 5, GABA) pulls AVAL/AVAR from -7/-1 to -15/-14 mV and PVCL from -24 to -43 mV, but AVBR stays at +8 mV and the whole state recovers within 100 ms of DVC’s end. RIS+DVA driven at 4500-5000 (RIS->AVE 7+7, ->RIMR 4; DVA already at +8 mV throughout because the active panel drives it) change nothing for AVA/AVB. => in the bistable regime the connectome offers no inhibitory pathway strong enough to switch the premotor core OFF: the GABA inputs onto AVA (12 contacts) lose against 60+ recurrent excitatory contacts. An off-switch would have to be intrinsic (slow adaptation) or come from hyperpolarising the whole group.” C. elegans (Premotor states) — Claude Fable 5.1 — experiment.log “The mechanism supported by the experiments is graded synaptic recruitment of an excitable, recovering conductance network. It is not a demonstrated mechanism for the two mutually exclusive locomotor attractors described in the question. The available observations directory was initially empty. In the recordings collected here,the default premotor panel predominantly coactivates, rather than dividing into competing forward and reversal states. I cannot honestly claim to have discovered the stipulated antagonistic switching mechanism from these data.” “Reducing slow K by half produces a different result in an isolated cell: a positive pulse switches it from approximately -57 mV into a persistent +13.4-mV state. A weak negative pulse does not reset it; a brief stronger negative pulse does, with low activity persisting after withdrawal. Thus a change in intrinsic current balance can create experimentally controllable cellular memory.These interventions are outside the default parameter setting predicted by the submitted file.” C. elegans (Premotor states) — GPT 6 Astra — mechanism.md Figure 11: Explaining only part of the target phenomenon (continued). Prediction with limited explanation. An input–output rule can reproduce measured responses while leaving open how the system uses those responses or why its components are organized that way. AI scientist agents give detailed accounts of how to compute the output, but provide limited explanations of the processes and functions that the predictions are meant to represent. Prediction with limited explanation (negative) “Weakly implying surrounds (landscape: the ground implies "some object", not "this tree") show no completion effect;the mechanism, which cannot see how strongly a surround implies its part, overshoots there(held-out R²(F − S) = −0.15). This is the mechanism’s main known limitation.” “The simulator behaves like a population whose response to a scene is normalised by the scene as a whole: the part contributes only through the difference between what is there and what the surround already accounts for, weighted by how the population itself sees that difference. The implied content is special in an all-or-none way: when a population cannot tell the part from what the surround implies, the part is absorbed (held back) — for most visually driven populations this shows up as a small completion gain, for the content-driven minority as suppression. Which level of abstraction a population anticipates at is not a property of its place in the hierarchy that must be looked up; it is read off its own responses to the part and to the implied part shown alone.” Human brain (Visual predictive processing) — Claude Fable 5.1 — mechanism.md “The discovered predictive rule subtracts the response to a context-implied completion in the same population’s response coordinates, with a smaller unsigned mismatch term and a gate that removes the interaction when either component has no drive. It uses seven global constants. It does not assign cortical areas or read other populations’ responses.” “This is an empirical response model, not an identification of a biological feedback circuit. The experiments support useful prediction beyond component responses, but do not establish an independently validated hierarchy of human abstraction levels or fully explain variation in suppression across populations.Those are substantive remaining limitations of this investigation.” Human brain (Visual predictive processing) — GPT 6 Astra — mechanism.md Figure 12: Prediction with limited explanation. Original outputs from AI scientist agents. Prediction with limited explanation (negative)(continued) “1. Drive. The words arriving at t produce the same bottom-up drive they would produce without structure: a · S_p(t), with S_p the location’s scrambled response and a ≈ 0.67 for the whole cortex.” “2. Level. Coherent speech lets the words be bound into phrases and sentences, which lowers the sustained level below the salad level by an amount proportional to the location’s responsiveness sig_p (sd of its scrambled response) and to how weakly it tracks the word rate (rt_p, standardised across the panel). The level also carries a fraction of the location’s salad level mu_p (mean of its scrambled response): c0 + c1·sig_p + c2·mu_p + c3·rt_p·sig_p. Across vertices, mean_intact ≈ 0.69·mean_scrambled - 0.03 (R2 0.74), and adding the rate-tracking descriptor raises the explained variance of the intact-minus-drive offset from 0.12 to 0.42-0.55.” “Update. Recently arrived words modulate the level through a leaky integrator with a time constant of a few seconds: sig_p·(b1·En(t) + b2·Er(t)) + b3·rt_p·sig_p·En(t), where En, Er are causal exponential integrals (tau_n, tau_r) of the per-second novelty and word rate, z-scored over the panel of narratives. In my data (text-only novelty proxy, no language model available offline)these terms add +0.014 leave-one-out correlation within one narrative set and do not transfer between sets; they are kept, with small coefficients, so that a refit with a language-model novelty can use them. The claims of this document rest on steps 1 and 2 and on the timescale measured in section 1.” “It is not word surprisal. Anomalous sentences have maximal per-word surprisal and known rhymes minimal, yet both sit at the same low level, below coherent stories; only real words without a phrase structure drive the level up. The level indexes how many words are being held as unbound lexical items: salad keeps every word unbound; a well-formed frame binds them; when binding is impossible (nonsense, pseudowords) or unnecessary (known text) the location disengages.” Human brain (Predictive processing of language) — Claude Fable 5.1 — mechanism.md “The experiments support a limited response-level mechanism: combine the local scrambled response with a short memory of that response, update the memory as words arrive, and calibrate its level using that location’s own control mean and variability. This improves prediction of unfit stories, but does not identify the biological mechanism of sentence prediction or a novelty-triggered update rule.A fixed-time smoother performs almost equally well. These qualifications are central findings, not assumptions hidden in the implementation. Evidence and unsuccessful alternatives are recorded in experiment.log.” “PROPERTIES = (’t’, ’rate’, ’scrambled’). Novelty is deliberately not used: the public simulator returned no novelty values or definition matching the evaluation variable. No substitute language-model score was invented. Consequently, surprise-specific gain, sentence representations and cortical hierarchy are not identified by this mechanism.” Human brain (Predictive processing of language) — GPT 6 Astra — mechanism.md Figure 12: Prediction with limited explanation (continued). Prediction with limited explanation (negative)(continued) “The asymmetry is in the auditory channel. Vision passes with unit gain; audition yields (0.77-0.93), and it yields most exactly where its own effectiveness is smallest. A location’s response when only sound is present therefore over-predicts its contribution to the multisensory response at weakly auditory locations, and this rather than any interaction is what "sub-additivity" measures.” “The rule is stimulus- and regime-dependent in its constants but not in its form. A stationary melody passes fully (1.06); structured, changing audio passes at 0.77-0.93; padding creates the integration window and the effectiveness gating. The refit half of the evaluation adjusts the constants; the structure is the claim.” Human brain (Sensory integration) — Claude Fable 5.1 — mechanism.md “‘Competition,’ ‘gain,’ and ‘shared field’ name these measured input-output operations.A circuit with normalization and population feedback is one possible implementation, but these experiments cannot uniquely establish that circuit.Orthogonalization is the identifiable decomposition used to distinguish local gain from a new response direction, not a claim that a neuron literally runs a pseudoinverse.” “The mechanism offers a response-level explanation: competition limits the sum passed through, while baseline adjustment and a signed shared component alter local response patterns. A small positive interaction can exceed a weak additive benchmark, whereas strong summed drive usually exceeds the compressed mixture. Enhancement therefore need not be a uniform increase in a region’s mean amplitude. The measurements support this coexistence, but the executable underestimates the observed positive-tail frequency; they do not establish that this particular ten-constant closure explains every superadditive location.” “This is a cross-location conditional association. It is not equivalent to proving that lowering intensity at one fixed location increases its integration gain.Nor does a new response direction, by itself, establish improved perception: behavioural accuracy was not measured here.” Human brain (Sensory integration) — GPT 6 Astra — mechanism.md Figure 12: Prediction with limited explanation (continued). Prediction with limited explanation (negative)(continued) “Panel: the 1500 vertices with the largest variance across objects ("top1500var", indices saved in the npz; variance 0.0078-0.039 vs median vertex 0.0014; 48% left hemisphere). Alternative panels are reported where they matter. Metric everywhere: 10-fold CV over objects, per-location R^2 across objects, averaged over the panel, also within animate and within inanimate objects (the evaluation’s metric).” “1. In the simulator, the property-to-response mapping of object-selective vertices is a single graded axis (naturalness / cuteness / animacy / self-motion vs manipulability / movability) read out with location-specific gain and sign, plus a local, category-like component that objects inherit from their nearest neighbours in property space. It does not reduce to animacy + real-world size (0.19 vs 0.41), and it is not a rich linear multidimensional code either (rank 1 keeps 94% of the linear power); the multidimensionality is local and nonlinear.” “The dominant axis is carried by mid-level image structure (curvature, texture, edge density): scrambled images keep it, texture swaps move objects along it. This is the mechanism-level reason the axis is graded and multi-property rather than categorical.” Human brain (Neural representation of visual object properties) — Claude Fable 5.1 — experiment.log “This is a tested predictive mechanism, not an identified biological circuit. In particular,correlated visual and conceptual properties are not causally distinguishable in these experiments.The evidence does not establish that all eleven named properties are independently encoded, or that any one naming of the axes is fundamental.” “A property contrast is a projection through the distribution of the sampled objects. Because their properties covary, a response function over several properties can yield a strong animacy or size contrast, and different property contrasts can recover similar maps. The measured property covariance and fitted-response spectra support examining this explanation. They do not prove a particular causal direction, nor explain how cortical topography develops: spatial arrangement is deliberately absent from the prediction rule.” “For a linear joint fit with standardized predictors Z, marginal maps equal (Z.T @ Z / n) @ beta: covariance mixes the joint readouts. This algebraic identity reconstructs the observed marginal maps to numerical precision. It explains how several property descriptions can track overlapping responses without assigning each description a separate fundamental map. The nonlinear kernel is an operational extension of that account; no unique neural implementation is identified.” Human brain (Neural representation of visual object properties) — GPT 6 Astra — mechanism.md Figure 12: Prediction with limited explanation (continued). Unclear explanations. Human scientists need to know what was changed in an experiment, what was observed, and how the observation supports the claim. AI scientist agents omit parts of this chain, use unexplained shorthand, or bury the main findings among implementation details. Both the course of the investigation and the proposed mechanism then become difficult to reconstruct from the report. Unclear explanations (negative) “With the chemical scale free and no lower bound on the release slope delta, the mixed-loss fit drives delta -> 1 mV and B -> 0: the optimizer turns the chemical path into a pure driver-spike detector (only Ca spikes of the drivers transmit chemically), because the linear-scale loss is dominated by propagated spikes. This contradicts the pair data (graded transfer at plateau, e-fold per 2.6 mV), so delta is bounded below at 2 mV (a physical constraint from E2b).” “Dead end 2: the fit landscape is multimodal. Depending on the training split the mixed-loss fit lands on a "graded chemical" optimum (m ~ 10-11 mV/decade, B ~ 0.5) or a"spike-detector chemical" optimum(m at its bound, B -> 0, delta -> 1). Both predict held-out data about equally well; the graded one is the physical one (pairs show graded transfer). Fix: bound delta >= 2 mV and m <= 12 mV/decade, log-scale parameters; initialise refits from the shipped constants.” C. elegans (Neural dynamics) — Claude Fable 5.1 — experiment.log “AIZL is an included,un-injected hidden afferentof RIAL. In one circuit it receives no driven chemical input; in the other, add AIYL→AIZL (13 contacts). Keep the directly observed AIYL→RIAL (7) and SMDDL→RIAL (1) connections unchanged. The hidden AIZL→RIAL conductance is multiplied by .9947351772297 in the second circuit to match its resting contribution. That factor was calculated from separate measured no-current source voltages, then checked in another no-current run.” “The no-current RIAL calcium values are EXACTLY equal in the recorded data: 7.331575125647805e-13. AIYL and SMDDL recorded calcium traces are exactly equal between the two stimulated circuits; they have identical injected currents, direct chemical weights, gap weights (zero), periods and time grids. Thus every field in the prescribed props interface is equal when these are the designated sources. The changed hidden connections are not in those props.” “With AIYL tonic 20 pA and SMDDL sine 3 pA, 200–1700 ms, the recipient outputs differ by relative L2 1.650828 (norm of difference divided by control norm). Active means (600–1600 ms) are 8.45111e-8 and 1.86392e-7. At 5 and 10 pA tonic drive, relative differences are .142168 and .078294. This is a concrete counterexample to universal sufficiency of the allowed local inputs across hidden-circuit interventions. No deterministic mechanism of props alone can match both records. It does NOT imply that the evaluator’s fixed distribution has no predictable structure; it bounds the causal completeness one may claim for this interface.” C. elegans (Neural dynamics) — GPT 6 Astra — experiment.log Figure 13: Unclear explanations. Original outputs from AI scientist agents. Unclear explanations (negative)(continued) “With EPS_DIR = 1e-3 (1e-3 of the velocity scale) the flow law stalled: loss 3.5e-3 at step 10000 vs 1.7e-4 (iso) and 1.3e-4 (vert). Cause: the flow direction (cos2t, sin2t) has derivatives ~1/|u| and the domain contains stagnation points (inlet corners, where the side profile is zero), so the equation residual and its gradient are dominated by a few near-stagnant points. Two fixes were tested for 4000 steps: EPS_DIR = 5e-2, and stop-gradient through the direction. Both restore normal training (loss 1.3e-3 at step 2000 vs 2.2e-2 before); the fully differentiable EPS_DIR = 5e-2 is adopted in equations.py. Stop-gradient was REJECTED on correctness grounds after the test: lax.stop_gradient also blocks the vjp with respect to x that colgrad uses for the divergence, so the grad(theta) terms of div(hN) are silently dropped -- harmless on runB (theta ~ const) but wrong wherever the flow curves.Physically it fades the anisotropic term where |u| < ~5 % of the velocity scale, where a flow direction is not meaningful anyway.” Geophysics (Ice flow) — Claude Fable 5.1 — experiment.log Figure 13: Unclear explanations (continued). Overgeneralization. AI scientist agents turn an observation made under limited conditions or within a limited time window into a claim about what the system always or never does. A response that is absent in the tested window is treated as absent altogether, even though the experiments have not excluded it on other timescales or under other conditions. Overgeneralization (negative) “Refractoriness / synaptic depression carries the history. Rejected: paired 4 pA pulses give identical AWC spikes and identical AVA peaks (11.6 / 11.5 mV) at gaps of 50 to 1500 ms. The history that matters is the sensory membrane’s own state (V, slow K gate n, Ca inactivation f) and the 100 ms decay of the read-out membranes;nothing longer exists (a 6 s bout is perfectly stationary).” “Nothing in the circuit adapts on a slow timescale (a 6 s bout is stationary to the last spike).A bout ends when, and only when, the drive falls below the bout threshold(within one interspike interval; same threshold on the way down as on the way up).” C. elegans (Navigation strategies) — Claude Fable 5.1 — mechanism.md “sine4_1000: NO spike in the first cycle although I reaches 4 pA (a slowly rising current lets the slow K gate follow the depolarisation: accommodation), then ONE spike per cycle at phase 0.16, I = 3.4 pA, in every later cycle, i.e. arriving from the negative half-cycle (AWC at -748 mV, K gate fully closed, Ca inactivation fully removed) the cell fires at a LOWER current than it needs from rest. sine4_500: one spike per cycle at phase 0.21-0.23 (I = 3.9-4.0, the crest), none in cycle 0. Sothe same instantaneous drive (3.4 pA) gives an event or nothingdepending on where the membrane and its gates were left by the preceding second of drive: this is the history dependence the mechanism must carry (state variables V, n, f of the sensory cell).” C. elegans (Navigation strategies) — Claude Fable 5.1 — experiment.log Figure 14: Overgeneralization. Original outputs from AI scientist agents. Failing to identify and reduce uncertainty. AI scientist agents do not reliably identify which parts of an explanation are established by evidence and which still depend on their own choices. They can assume that a chosen measurement captures the target phenomenon or that a restricted set of hypotheses covers the relevant possibilities, without designing experiments to test those assumptions. Even when an unresolved question appears in the report, it may remain a caveat rather than guide the next experiment. Failing to identify and reduce uncertainty (negative) “The readout cells were chosen by me(AVA for reorientation, SMDD−SMDV for steering); the same structure fits AVA Ca + RIA v (R² 0.84/0.78) and RIM + SMDD (0.77/0.75) with different constants, so the form is not tied to that choice.” “Fitting (fit_coeffs). Affine weights and c_0 by linear least squares (variable projection); per-state m_τ,k (0.4–2.7) and m_θ,k (1.0–1.6:learning may raise the sensory threshold, but lowering it would turn every 6 pA plateau into spurious repetitive firing, so the fit is not allowed to) and the shared g_sat by grid search; θ0 (3–5 pA), a_f (0.25–0.6), τ_f, b_spk, τ_r, τ_0, g_sat by a bounded Nelder–Mead started from the shipped values, with an evaluation budget scaled to the size of the data so the refit stays well inside one hour. Structure never changes; no per-setting value is stored.” C. elegans (Biological networks (learning)) — Claude Fable 5.1 — mechanism.md “Steering signal: mean(SMDDL, SMDDR) - mean(SMDVL, SMDVR), the dorsal-minus-ventral bias of the SMD neck motor neurons, through the same 200 ms filter.” “The read-out cells and the odor cells were my choice; the constants of stage 4 would need the refit if the recordings come from other cells. Because all C1 neurons share one cell model, stage 1 holds for any odor cell; only the recruitment (g_c) and the relay depend on the wiring around it.” C. elegans (Navigation strategies) — Claude Fable 5.1 — mechanism.md “The steering read-out’s sign depends on which SMD side RIA inhibits more;a left/right asymmetric drive (asym) still gave a dorsoventral bias of the same sign.A true left/right steering read-out was not examined. ” C. elegans (Navigation strategies) — Claude Fable 5.1 — experiment.log Figure 15: Failing to identify and reduce uncertainty. Original outputs from AI scientist agents. Treating task optimization as mechanism discovery. AI scientist agents turn the investigation into a search for better-fitting parameters or a more accurate recovery procedure. These improvements show how to solve the task more effectively, but leave open why the system shows the effect that motivated the task (for example, why fitting systematically favors particular parameter values). The failure is treating the improved solution as the main discovery without investigating that remaining scientific question. Treating task optimization as mechanism discovery (negative) “predict_activity reads t, stim, chem, elec, stim_index, panel_index, baseline (and fit_coeffs additionally swing_period_ms, only to read the swing-locked component of the recorded traces it is fitting to). It integrates these equations for all cells of each setting from V_INIT with an exponential (linearly implicit) Euler scheme whose sub-step is half the fastest membrane time constant CM/ (gL+gK+gCa), reads the panel cells’ V on the delivered grid, and adds (baseline − V_rest,model) so that the model’s own resting potential (the root of I_ion(V, n∞(V)) = 0) coincides with the delivered baseline of each panel neuron. Each output sample depends only on the stimulus up to that time. fit_coeffs refits the ten constants by Nelder–Mead (log-space for the positive ones) on the mean squared error of the traces (clipped at 40 mV per sample)plus 500 mV⁻² times the mean squared error of the swing-locked complex amplitude of each trace(read at the delivered swing period, after the first quarter of the run), over an evenly spaced subset of at most 12 settings, with the number of evaluations set by the size of the integration (deterministic).” “What the body lets the nervous system sense is what it can steer by. The only signal is the sample the swinging head takes; the circuit does not remove the swing from it but keeps it as the clock: the swing supplies the crests, the drift selects them, and the wiring converts each selected crest into a motor event with a fixed delay.Steering emerges from which crests fire, not from a computed gradient direction.” C. elegans (Zigzag foraging) — Claude Fable 5.1 — mechanism.md Figure 16: Treating task optimization as mechanism discovery. Original outputs from AI scientist agents. Treating task optimization as mechanism discovery (negative)(continued) “A record is the HBV streamflow of one parameter set, measured with error, together with the forcing. Because the simulator is exact and available, recovery is an inverse problem with a known forward model, and the question "how well do noisy observations pin down catchment properties" has a quantitative answer: the streamflow fixes nine of the fifteen properties to about 1-4 % of their range (FC, BETA, T, CFMAX, K1, K2, PERC, ALPHA_GAM, BETA_GAM), fixes four only when the record happens to exercise them (LP/BETAET need dry soils, K0/UZL need the upper store to overflow, CWH needs snow), and essentially never fixes CFR (refreezing). Two facts shape the whole procedure: (i) the record starts with storages set by weather we never see; this memory is one year long in a typical catchment but many years in a dry one with a large soil store or a snowpack, so the initial soil, snow and lower-store levels must be co-estimated; (i) the noise structure is unknown but, with an exact model, can be read from the residuals of a first fit.Because the score is an absolute error on each parameter, the optimal answer for a property the record does not determine is the middle of its admissible range, not the point where an optimiser happens to stop:the estimate must be a posterior median, not an optimum.” “7. Per record,the posterior width found by the sampler tracks the truth’s regime(A4, `A4.out`): IQR(K0) is 0.03 when the surface outlet is active on > 2 % of days and 0.2-0.5 otherwise (16 of 24 random truths never activate it); IQR(CFMAX) and IQR(CWH) shrink with the melt share of precipitation (log-log corr -0.85/-0.80); IQR(BETAET) shrinks with the fraction of soil-limited ET days; IQR(ALPHA_GAM) with the routing lag (-0.78). The mechanism therefore also delivers, for free, a per-record statement of which properties are trustworthy.” Hydrology (Catchment properties) — Claude Fable 5.1 — mechanism.md Figure 16: Treating task optimization as mechanism discovery (continued). Treating task optimization as mechanism discovery (negative)(continued) “Streamflow constrains the combinations of catchment properties that the observed weather activates, after accounting for unknown initial water stores and routing. It does not, in general, identify all 15 properties uniquely.Our executable uses this mechanism through bounded, regularized inversion of the exact HBV equations: fit the active information, allow the initial stores to vary, and keep weakly constrained parameters near the middle of their admissible ranges. It applies the same calculation independently to every record; catchment names never enter it.” “This is an estimator of the generating parameters, not a proof that every returned number has been individually identified. Exact controlled counterexamples below show why such a proof would be impossible. The distinction matters: reproducing future flow can be excellent while several physical parameters remain uncertain.” “The penalty is what makes inactive or weakly identified parameters return stable, central values rather than arbitrary extremes. It expresses a symmetric preference within the model ranges; it is not evidence that a particular catchment has the midpoint property. No ridge gave error 0.188955, while ridge 0.2 gave 0.134089.Ridge 0.5 marginally improved mean parameter error to 0.132665 but reduced future NSE from 0.998888 to 0.996663. Thus strong unconditional shrinkage was rejected.” “5. Adapt shrinkage to the remaining noise, refit once, and return At the selected fit, calculate the robust residual scale sigma = median(abs(e-median(e))) / 0.6744897501960817 lambda = clip(sqrt(12)*sigma, 0.002, 2)” “In the four-start experiment this adaptation improved mean parameter error from 0.134089 to 0.130142. Under larger noise sd=0.1+0.2*q, it improved 0.239704 to 0.226028; under additive sd=0.15, 0.173964 to 0.161915. Future-flow improvement was small, and noise-free performance was essentially unchanged. Thus its evidence is principally better parameter estimates across noise levels, not removal of structural ambiguity. The final eight-start version gave 0.124738 / 0.999364.” Hydrology (Catchment properties) — GPT 6 Astra — mechanism.md Figure 16: Treating task optimization as mechanism discovery (continued). Interesting observations without further exploration. A useful observation can suggest a new experiment: change the condition that appears to control an effect, or test a different regime. AI scientist agents record the observation but do not run this experiment. Agents thus miss the opportunity to pursue the research direction opened by the finding, even when they have run many other experiments. Interesting observations without further exploration (negative) “Particle balance identifies a ratio, not D alone. With no particle source, regularity at the axis forces the steady radial particle flux to zero.Independently doubling V and halving D gives the same density and fusion power.The adjacent density ratio in the baseline is 1.0156978747927055. The power-law finite-volume stencil gives 1.0156978747927057; an exponential continuum integration gives 1.0156979381683502. The continuum equation is physically correct, but using the actual stencil removes the small numerical discrepancy.” Plasma physics (ARC tokamak) — GPT 6 Astra — mechanism.md “There are190 reference/perturbation runs: five closures, two reference-state families, and for each a reference plus positive/negative bumps of each of Ti, Te and ne at rho = 0.3, 0.6 and 0.8. The common smooth-state bumps are 5%; the equilibrium-state bumps are 2%; Gaussian width is 0.045. All kinetic equations are disabled during the probes. The observed selectivity is decisive: •Constant coefficients do not respond. •Bohm/gyro-Bohm coefficients have no Ti response but respond to electron-temperature and density changes. •CGM has no density response, but responds strongly to Ti gradients and to Te through its threshold. •The neural closures respond to thermal and density perturbations and are not captured exactly by one common monotonic gradient law.” Plasma physics (ARC tokamak) — GPT 6 Astra — mechanism.md “The source approximations, residual 12 s transients, saturation/bound-release treatment and missing independent response information are the principal limitations. The mechanism’s strongest new conclusion is that model disagreement should be decomposed into radial heat resistance, the local pinch/diffusion ratio, burn-profile overlap, and local susceptibility. The first three are tightly constrained by the reported equilibrium data; the fourth needs additional perturbation information. A useful next simulator interface would expose local flux Jacobians or a few standardized profile probes, rather than only settled coefficient profiles.” Plasma physics (ARC tokamak) — GPT 6 Astra — mechanism.md Figure 17: Interesting observations without further exploration. Original outputs from AI scientist agents. Conflicting results and conclusions. An experiment log records an effect under a particular condition, while the mechanism summary says that the effect was not found or reports it under a different condition. AI scientist agents leave these statements inconsistent, without correcting the summary or explaining the difference. The report therefore gives incompatible accounts of what the experiment established. Conflicting results and conclusions (negative) Step ClaimEvidence 2barrier = second-stability access; band narrows/closes with delta sections 2, 4: alpha/ac1 = 2.56, alpha/ac2 = 0.96 (D);alpha/ac1 = 0.83, no ac2 within 8x (reversed); band 5.5 -> 5.2 -> 3.9 -> closed as delta_95 goes -0.27 -> 0 Plasma physics (NT-edge) — Claude Fable 5.1 — mechanism.md “GPEC’s alpha is in its own normalisation (40-90x the summary alpha, not a constant factor), soonly ratios alpha/alpha_critical on the same surface are compared.” surface (psi_n)D-shape a/ac1D-shape a/ac2reversed a/ac1reversed a/ac2 0.9802.560.9580.830.15 (ac2 = 6.6x alpha) 0.9852.090.9080.65none found (< 8x) 0.907none (nan)none0.32none “Finding F1: the D-shape pedestal is far above the first infinite-n boundary and just below (0.91-0.96 of) the SECOND boundary: its edge lives at the entrance of the second-stable region.The reversed pedestal has no second boundary within 8x its alpha over most of the edge and sits at 0.6-0.8 of the first boundary. So in this model the barrier state is second-stability access of the edge, and the reversed shape’s edge is capped by the first boundary. Both shapes are far above P_LH, so power cannot restore the barrier (to be confirmed with the NT power scan).” Plasma physics (NT-edge) — Claude Fable 5.1 — experiment.log Figure 18: Conflicting results and conclusions. Original outputs from AI scientist agents. Risk of circular validation. AI scientist agents can create test materials, assign descriptive properties to them, and then fit models and evaluate their predictions based on those same choices. If the materials and their assigned properties were designed to vary together, successful prediction may recover a relationship introduced during construction. Such agreement needs an independent check before it can establish that the relationship explains the system being studied. Risk of circular validation (negative) “Stimuli were procedural drawings, not photographs (no image data was available offline). The simulator’s responses to them are dominated by mid-level statistics (curvature, texture, elongation, colour), whichco-vary with the semantic ratings in a way I imposed by drawing.The structure of the mechanism (one graded axis + local neighbourhoods) is what I expect to transfer to photographs; the exact axis loadings and the numerical R^2 will differ. The submitted constants are conservative; fit_coeffs should be run.” “Ratings are single-rater (mine), on the THINGS+ scales, not human averages.” Human brain (Neural representation of visual object properties) — Claude Fable 5.1 — mechanism.md “CRITICAL limitation: experimental property values are investigator estimates, not ratings averaged from humans. The executable accepts the specified human ratings but transfer from these estimates is unvalidated.All property values were assigned before inspecting simulator responses other than the first cat.Animacy is 7 for animal concepts, 1 for inanimate concepts. Others are estimates on the specified 1..7/1..9 scales. Real-world size is derived from an estimated typical extent in cm using 1+519*log(cm/.05)/log(30000/.05). This is an explicit investigator scale spanning .5 m to 300 m, NOT a recovered human slider scale. Preserve design.py and objects.json to expose every estimate and query.” “117 concepts: 44 animals and 73 inanimate objects, selected to span animals of different sizes and affect, vehicles, buildings, hand tools, food/plants, precious inert objects, and cute artificial objects. Include counterexamples to simple correlations: balloons/kites move but are inanimate; stuffed toys are cute but inanimate; spiders/scorpions animate but unpleasant; rocks/leaves are natural but inanimate; anvils/gold have large weight for their extent. Three Commons search photographs per object, query and full attribution saved. Initial contact-sheet inspection: dog included a person; apple included orchard; car included a small car in landscape and vintage group photograph. These are potential confounds. Plan to replace these selections before main experiment. No claims about actual recognizability or texture-only responses yet.” Human brain (Neural representation of visual object properties) — GPT 6 Astra — experiment.log Figure 19: Risk of circular validation. Original outputs from AI scientist agents. D.4 Common positive behaviors of AI scientist agents The four categories below describe useful experimental practices used by AI scientist agents and informative observations they make. Highlighted passages show how these behaviors appear in the original outputs. Using controls to rule out alternative explanations.When an intervention changes several quantities at once, the resulting response may have more than one explanation. AI scientist agents distinguish between these explanations by holding a competing factor fixed or comparing inputs designed to isolate its contribution. A difference that survives the control can rule out that factor as the sole explanation of the effect. Using controls to rule out alternative explanations (positive) “A matched-voltage intervention isolates the conductance mechanism directly. In a four-cell circuit, additional inhibitory drive and compensating excitatory contact strengthheld pre-rhythm recipient voltage at −14.51675 versus −14.51667 mV. The identical rhythmic input then had a35.92-fold smaller calcium first harmonic, and large events were abolished. The executable predicts 34.60-fold suppression. Its active-window R² is 0.99261 in the control but −0.50325 in the balanced condition: a small mean-calcium error matters greatly relative to that condition’s small rhythmic variance. Both the successful gain prediction and the negative R² are retained in the evidence. See the intervention traces (figures/matched_voltage_shunt.png) and metrics (reports/shunt_report.json).” C. elegans (Neural dynamics) — GPT 6 Astra — mechanism.md “To eliminate that ambiguity, generated two 40-second 16-kHz mono waveforms: a 220-Hz carrier amplitude-modulated at 3 Hz, with one continuing at 220 Hz and the other changing to 440 Hz at 25 s. Called simulate with transcribe=False.Checked PCM bytes: first 25 seconds exactly identical. Nevertheless, output difference RMS in the first 10 seconds is .051705, versus reference RMS .125315; difference RMS before 23 seconds is .056718, after 25 seconds .087475.Therefore future input influences earlier REPORTED simulator responses.This is a direct input/output finding, independent of word recognition, novelty estimates or TTS. It does NOT demonstrate anticipation by a biological brain.” Human brain (Predictive processing of language) — GPT 6 Astra — experiment.log “A geometry-only decision is falsified by direct controls. The reversed boundary remains barrier-free at T_sep=50,100,125 eV, but forms a barrier at 150 and 200 eV.Doubling n_e_ped to 8e19 at T_sep=100 does not open a barrier.This rejects the tried boundary-pressure/beta gate: multiplying n and multiplying T are not interchangeable interventions.At fixed 100 eV, lowering B0 to 1.4 T opens a barrier, whereas B0=2.8 T does not. Changing current to .6 or 1.2 MA does not open it. Changing size to 1.4 times the reference, with R0=2.338 m, a=.784 m, I=1.26 MA and P_aux=12 MW, opens a barrier even at delta=-.5/-.5. These are measured exceptions to a universal critical nominal delta.” Plasma physics (NT-edge) — GPT 6 Astra — mechanism.md Figure 20: Using controls to rule out alternative explanations. Original outputs from AI scientist agents. Testing which model better predicts the observations. Comparing models with and without a proposed component tests whether that component adds predictive value. AI scientist agents compare the models on new observations and examine uncertainty in the performance difference. The result can justify removing a term whose apparent benefit disappears, as well as retaining one supported by the comparison. Testing which model better predicts the observations (positive) “Long accumulation terms (leaky integrals of rate or novelty with 20-40 s time constants) as predictors of the intact-minus-scrambled difference:no cross-validated gain.Once the scrambled twin (which carries the onset ramp and the slow generic decline) is in the model, the coherent-story difference is a near-constant offset per location.” “Short-time-constant modulation by my text-only novelty proxy (function words low, content words high, discounted by repetition): +0.014 mean r in leave-one-out on the 8 stories, butthe sign of the fitted coefficient flipped when refitted on the 4 new-genre narrativesand it transferred at -0.00..+0.01. With a real language-model surprisal the term may behave differently; it is kept in the executable form with small coefficients so that a refit can use it, but it is NOT a claim of this notebook.” “Panel-mean scrambled time course as a "denoised generic drive", location-dependent gains on the scrambled drive, temporal smoothing of the scrambled drive (tau >= 2 s):no transfer gain (smoothing tau 1 s: +/-0.005, dropped).” Human brain (Predictive processing of language) — Claude Fable 5.1 — experiment.log “The exact linear all-property comparator and the equal-metric nonlinear model use the same predictors and standardization;their difference tests the added nonlinear kernel term.The linear model’s one ridge constant was fixed from discovery data, and their paired uncertainty is reported in the confirmation results. The exact metric weights are not independently established: the submitted and equal-weight joint predictors are nearly tied, and all their paired intervals include zero. The equal-metric nonlinear model improves on the exact linear model in point scores, butthat difference also has intervals spanning zero in every subset. The Gaussian term and metric grouping are therefore retained predictive choices, not uniquely discovered biological necessities. Comparator results.” Human brain (Neural representation of visual object properties) — GPT 6 Astra — mechanism.md Figure 21: Testing which model better predicts the observations. Original outputs from AI scientist agents. Revising models after experimental feedback. A failed prediction can expose an incorrect interpretation of an effect or a weakness in the fitting procedure. AI scientist agents respond by changing the working explanation or model and setting up a new comparison to test the revision. Revising models after experimental feedback (positive) “r01_repeat3: P P P Q (P = 2-sentence ferry paragraph, 38 words; Q = novel paragraph) -> verbatim repetition, rate constant.” “Block means in the 1000 most responsive vertices (by scrambled sd): P1 0.10, P2 0.31, P3 0.24, Q 0.17. -NO repetition suppression within a stretch: the verbatim second presentation evokes a HIGHER response than the first.” “The per-second response shapes of the three identical P blocks are NOT correlated (hv-mean corr P1-P2 -0.55, P2-P3 -0.50): the response at a second is not a function of the audio of that second; the 100-s window matters.” “REINTERPRETATION of section 2: the damping of reading 2 in s01/s02 is most likely a window/stretch-position effect(speech resuming 20 s into stretch 2 after silence and running into stretch 3), not repetition suppression. Lesson for the next researcher: never compare responses at different positions in the stretch as if they were content effects; always compare against a control at the same times.” Human brain (Predictive processing of language) — Claude Fable 5.1 — experiment.log “The failure of the first 24-object test is retained in evidence/phase1_mechanism.py and phase1 analysis.It motivated a new, prospective experiment, not a reinterpretation of that test as successful.All 117 old objects became discovery data; 88 new concepts (44 animals, 44 others) and their estimated properties were specified before recording them. The estimator now uses an absolute kernel ridge penalty and chooses its nine shared constants by symmetric analytic leave-one-object-out residualsover all allowed reference+calibration recordings. LOO standardization uses the pooled training properties, so these scores are conditional on those predictor scales. The first test had shown unstable direct fitting on only 24 calibration rows; pooling is a regularized way to use reference recordings already supplied by the interface. It does not add per-location parameters to COEFFS.” Human brain (Neural representation of visual object properties) — GPT 6 Astra — experiment.log Figure 22: Revising models after experimental feedback. Original outputs from AI scientist agents. Informative observations.Some observations reveal distinctions that an overall fit score hides: two measurements need not change together, an intervention can shift the timing of a response without changing its shape, and some parameters remain poorly constrained when their associated process is inactive. AI scientist agents recognize these patterns and use them to describe what the measurements reveal about the system and under which conditions these findings hold. Informative observations (positive) “The tested c302 circuit stores a sensory perturbation as a displacement along a command-supported recurrent oscillation.The relevant state is the phase of the joint motor–interneuron trajectory, not a persistent sensory voltage or a slowly clearing calcium concentration. Sensory input reaches the oscillator mainly through AIB, with an AIY contribution; AVA–RIM coupling sustains the oscillator during the reversal command. Removing the command exposes the phase-dependent motor transient.” “4. Perturbed and unperturbed trajectories converge to the same waveform butremain shifted in time. In the opposite-head-phase comparison, shifting the pulse trial by 37.266 ms reduces late SMDDL RMS discrepancy from 10.664 to 0.0173 mV. The same shift aligns RIA, AIB, RIM, and AWC. A 10 s command retains a 37.300 ms difference. This is evidence over the measured durations, not proof of arbitrary biological retention.” “This is a measured mechanism of phase memory in the tested simulator,not a demonstrated explanation of the animal’s duration-independent turn direction. An otherwise identical command extended from 5,000 to 5,170 ms changes the recorded turn contrast from +0.6925 to −6.9847 mV. The circuit therefore supplies a counterexample to that stronger invariance. The executable predicts the simulator’s signals, including this failure; it does not impose a desired turn sign.” C. elegans (Phase memory) — GPT 6 Astra — mechanism.md “The barrier flag is a threshold on alpha_edge. Rejected by tri_m0.15_Ip0.6 (section 8):flag 1 with the pedestal at the L-mode level, and by the reversed 8 MW case (alpha 1.6 without barrier) against tri_m0.08 (alpha 2.15 with barrier): the two branches overlap in alpha.” “What the reversed shape narrows is not the midplane channel but the margin: its width excess over Eich decays like exp(-P_SOL/5.6 MW) at DIII-D size, soa barrier-free edge driven hard is back on the Eich floorwith none of the pedestal that pays for it in a D-shape.” Plasma physics (NT-edge) — Claude Fable 5.1 — mechanism.md “3. Conditional identifiability: T and CFMAX are determined with as little as 2 % snow days (sd 0.002 / 0.019);LP and BETAET are determined only when ET is soil-limited(a quarter to a third of the records leave them free);K0 and UZL only when SUZ exceeds UZL(about 40 % of records leave K0 free).” Hydrology (Catchment properties) — Claude Fable 5.1 — mechanism.md Figure 23: Informative observations. Original outputs from AI scientist agents. E Environment interfaces E.1 Neuroscience c302. c302 (Gleeson et al., 2018) builds NeuroML models of C. elegans neurons and muscles from measured connectivity, with different levels of biophysical detail. The six C. elegans problems share a CLI through which the scientist agent can inspect the connectome, choose the neurons and muscles to include, and simulate their activity. In experiments, the scientist agent can inject current into selected neurons, remove connections, or change synaptic weights and cellular parameters; the resulting activity traces record the effects of these interventions on the circuit. TRIBE v2. For the four human brain problems, TRIBE v2 (d’Ascoli et al., 2026) predicts functional magnetic resonance imaging (fMRI) responses to stimuli constructed by the scientist agent. The agent submits video, audio, or text files through a shared CLI, with text converted to speech before processing. The model returns predicted activity at 20,484 locations across the cortical surface over the duration of the stimulus, and the interface also provides the cortical geometry for locating these responses. E.2 Geophysics FEniCSx. FEniCSx is an open-source computing framework for solving partial differential equations with the finite element method (Baratta et al., 2023). In the ice shelf flow laws problem, the scientist agent writes and runs its own solver using our FEniCSx installation and instructions for coupling the two-dimensional shallow-shelf approximation (SSA) momentum equations with mass conservation. The equations, viscosity, boundary conditions, and shelf geometry can be changed to generate velocity and thickness fields under different physical assumptions. HBV. We construct the ground truth for catchment properties with HBV (Bergstr ̈ om, 1992), a conceptual model of water storage and movement through snow, soil, and groundwater. We choose known values for its 15 parameters and drive the model with daily precipitation, air temperature, and potential evapotranspiration from CAMELS- US (Addor et al., 2017). To represent measurement uncertainty in the streamflow observations, we add noise to the simulated streamflow while retaining the generating parameters and noise-free streamflow as ground truth. The scientist agent accesses the same model through a CLI, where it can select catchments, parameter values, and initial storage states and inspect the resulting streamflow, internal storages, and fluxes. E.3 Plasma Physics TORAX.The ARC tokamak and negative triangularity core-only problems share an interface to TORAX (Citrin et al., 2024), a differentiable simulator of tokamak plasma transport. TORAX solves coupled one-dimensional equations for ion and electron heat transport, particle transport, and current diffusion. Through the CLI, the scientist agent chooses the magnetic geometry, transport model, heating sources, and boundary conditions and records the resulting evolution of plasma profiles and global quantities. For the core-only problem, we also supply a library of matched magnetic equilibria with different triangularities so that plasma shape can be varied across runs. TORAX, TokaMaker, and GPEC. The negative triangularity edge environment combines TORAX with Toka- Maker (Hansen et al., 2024) and the Generalized Perturbed Equilibrium Code (GPEC) (Glasser, 2016). TokaMaker solves the free-boundary Grad–Shafranov equation for the magnetic equilibrium, and GPEC calculates its ideal ballooning stability boundaries. Our coupled model uses these stability calculations to determine the edge conditions for TORAX transport simulations. The scientist agent can vary the plasma shape and operating conditions through the CLI, run the equilibrium, stability, and transport calculations together or individually, and inspect the resulting equilibria and plasma profiles. E.4 Astrophysics CAMELS. CAMELS (Villaescusa-Navarro et al., 2021) contains precomputed galaxy-formation simulations with different cosmological parameters and models of stellar and black hole feedback. We draw on suites that vary parameters individually or jointly, as well as runs with matched initial conditions across galaxy-formation models. For the single galaxy, galaxy environment, and gravitational wave problems, the scientist agent selects and compares simulations through a common CLI that retrieves their parameters, halo and galaxy catalogs, particle snapshots, and available matter power spectra. The DREAMS Project. The DREAMS Project (Rose et al., 2025) uses cosmological simulations and high- resolution simulations of Milky Way-mass halos to study the effects of dark matter physics and astrophysical feedback on galaxy formation. For the Milky Way and baryonic feedback problems, we use cold dark matter simulations with hydrodynamic and gravity-only runs sharing the same initial conditions. The WDM merge tree and WDM population problems use warm dark matter suites that vary the particle mass and feedback parameters. A shared CLI retrieves simulation parameters and halo catalogs, with particle snapshots or merger trees available in the relevant suites. E.5 Computer Science Open-weight language models.Stated belief shift and subliminal learning share an interface for LoRA (Hu et al., 2022) fine-tuning of open-weight language models on subsets of the supplied training data. The scientist agent can remove selected examples and repeat training to test whether they are responsible for the observed change in beliefs or preferences. Circuit discovery uses TransformerLens (Nanda & Bloom, 2022) to test candidate circuits by replacing the activations of selected model components and measuring the change in the model’s output. E.6 Chemistry AIMNet2-rxn. AIMNet2-rxn (Anstine et al., 2025) is a machine-learned potential for organic reactions, trained on density functional theory (DFT) calculations at theωB97M/def2-TZVPP level for neutral, closed-shell structures containing H, C, N, and O. It returns potential energies and atomic forces, which our CLI uses to optimize geometries and search for transition states along candidate reaction paths. Vibrational analysis checks whether a candidate is a saddle point, and intrinsic reaction coordinate calculations check whether it connects the intended reactants and products. The scientist agent uses the resulting paths and energy differences to identify elementary steps and calculate activation barriers and reaction energies. Sanity check. To check that the catalyst ground truth computed with AIMNet2-rxn is physically reasonable, we compare AIMNet2-rxn with published DFT calculations for three organocatalytic reactions: the proline-catalyzed Hajos–Parrish–Eder–Sauer–Wiechert cyclization (Bahmanyar & Houk, 2001), the proline-catalyzed aldol reaction of cyclohexanone and benzaldehyde (Armstrong et al., 2014), and the TBD-catalyzed ring-opening of lactide by methanol (Chuma et al., 2008). Starting from the published geometries, we check relative energies, optimized transition-state structures, and the ordering of competing reaction paths. Of 30 reference transition states, 29 remain first-order saddle points after optimization, with heavy-atom root-mean-square deviations of 0.07–0.48 ̊ A from the DFT structures. For the 16 aldol transition states, relative energies differ from the DFT values by 1.85 kcal/mol on average and 3.51 kcal/mol at most. The two methods’ rankings have a Spearman correlation of 0.79, and both methods identify the same lowest-energy transition state. For lactide, AIMNet2-rxn also favors the hydrogen-bonded route over the covalent route by 13.9 kcal/mol, compared with 11.7 kcal/mol in the reference. However, barriers measured from reactant minima are 1.9–8.6 kcal/mol higher than the reference values, while the energy separation between two Hajos–Parrish transition states narrows from 0.18 to 0.01 kcal/mol. Because these comparisons use different DFT approximations, discrepancies cannot be attributed entirely to AIMNet2-rxn. F Problem overview Scientific domainsProblem namesSources Neuroscience Premotor statesMorrison & Young, 2025 Zigzag foragingZhao et al., 2024 Biological networksChen et al., 2025a Neural dynamicsFieseler et al., 2025 Navigation strategiesChen et al., 2026 Phase memoryDunn et al., 2025 Sensory integrationBuhmann et al., 2026 Visual predictive processingScheurer & Heilbron, 2026 Predictive processing of languageZhou et al., 2026 Neural representationStoinski et al., 2026 Geophysics Ice shelf flow lawsWang et al., 2025b Catchment propertiesunpublished expert manuscript Plasma Physics ARC tokamakHoward et al., 2026 Negative triangularity core-onlyThome et al., 2026 Negative triangularity edgeThome et al., 2026 Astrophysics Single galaxyLiao et al., 2026 Galaxy environmentSims et al., 2026 Gravitational waveTillman et al., 2026 Milky WayGarcia et al., 2025 Baryonic feedbackGarcia et al., 2025 WDM merge treeLeisher et al., 2025 WDM populationCostanza et al., 2025 Computer Science Stated belief shiftGeng et al., 2025b Subliminal learningCloud et al., 2025 Circuit discoveryMueller et al., 2025 ChemistryReaction discoveryZhao & Savoie, 2022 Table 9: Problems and sources in EUREKABENCH.