Paper deep dive
EarthVerse: Benchmarking Scientific Agents Across Dynamic Earth Systems and Natural Hazards
Zhiqing Cui, Xinxiang Yin, Yihong Tang, Xinglang Zhang, Yuanzhe Hu, Siru Zhong, Weidong Tang, Yuxuan Liang, Weijia Li, Ming Jin, Shirui Pan, Yuhao Kang, Dingyi Zhuang, Jinhua Zhao
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Earth-system analysis reconstructs changing physical processes from observations that differ in source, scale, timing, and modality. Natural hazards make this work consequential because incomplete evidence can change estimates of severity, exposure, and mechanism. We introduce EarthVerse, a benchmark that evaluates scientific agents through package-scoped investigations. Its 405 reproducible tasks are grounded in 199 documented events and 19 hazard families. Agents inspect heterogeneous event packages, choose compatible evidence, execute transparent calculations, reconcile source differences, and preserve provenance in the final answer. We provide executable ground truth that decomposes each task into fine-grained answer units, together with task-specific rubrics that assess the supporting research process while allowing multiple valid paths. We evaluate 25 model and agent systems under a controlled tool-using protocol, then use controlled studies to locate failures in evidence access, tool selection, memory, reasoning, interaction, and scientific execution. Across systems, the best mean answer-unit accuracy is 84.65%, while the highest Strict@95 is only 34.81%. The gap shows that current agents often complete individual steps without maintaining a consistent chain across evidence, scales, units, calculations, and physical interpretation. EarthVerse provides a reproducible basis for measuring end-to-end scientific reliability in dynamic Earth systems.
Tags
Links
- Source: https://arxiv.org/abs/2608.23525v1
- Canonical: https://arxiv.org/abs/2608.23525v1
Trouble viewing inline? Open PDF directly →
Full Text
155,957 characters extracted from source content.
Expand or collapse full text
EARTHVERSE: BENCHMARKING SCIENTIFIC AGENTS ACROSS DYNAMIC EARTH SYSTEMS AND NATURAL HAZARDS Zhiqing Cui 1 Xinxiang Yin 2 Yihong Tang 3 Xinglang Zhang 4 Yuanzhe Hu 5 Siru Zhong 4 Weidong Tang 6 Yuxuan Liang 4 Weijia Li 7 Ming Jin 8 Shirui Pan 8 Yuhao Kang 9 Dingyi Zhuang 10,† Jinhua Zhao 10 1 NUIST 2 HKU 3 McGill 4 HKUST(GZ) 5 Georgia Tech 6 NUS 7 Tsinghua 8 Griffith 9 UT Austin 10 MIT zhiqing@nuist.edu.cn dingyi@mit.edu † Corresponding author Hugging FaceGitHubWebsite ABSTRACT Earth-system analysis reconstructs changing physical processes from observations that differ in source, scale, timing, and modality. Natural hazards make this work consequential because incomplete evidence can change estimates of severity, exposure, and mechanism. We introduce EarthVerse, a benchmark that evaluates scientific agents through package-scoped investigations. Its 405 reproducible tasks are grounded in 199 documented events and 19 hazard families. Agents inspect heterogeneous event packages, choose compatible evidence, execute transparent calcu- lations, reconcile source differences, and preserve provenance in the final answer. We provide executable ground truth that decomposes each task into fine-grained answer units, together with task-specific rubrics that assess the supporting research process while allowing multiple valid paths. We evaluate 25 model and agent systems under a controlled tool-using protocol, then use controlled studies to locate failures in evidence access, tool selection, memory, reasoning, interaction, and scientific execution. Across systems, the best mean answer-unit accuracy is 84.65%, while the highest Strict@95 is only 34.81%. The gap shows that current agents often complete individual steps without maintaining a consistent chain across evidence, scales, units, calculations, and physical interpretation. EarthVerse provides a reproducible basis for measuring end-to-end scientific reliability in dynamic Earth systems. (a) Scientific coverage.(b) Leaderboard landscape. Figure 1: Coverage and model landscape. Task and answer-unit coverage across hazards and capabilities (left); mean answer-unit accuracy and token use across 25 systems (right). arXiv:2608.23525v1 [cs.AI] 24 Aug 2026 EarthVerse: Benchmarking Scientific Agents Across Dynamic Earth Systems and Natural Hazards 1 Introduction Earth-system science asks how the atmosphere, land, ocean, ecosystems, and human activity shape one another. These processes govern water, food, health, infrastructure, and climate risk. Natural hazards make the question immediate: weather, climate, and water extremes have caused extensive mortality and economic loss, while compound events can propagate across sectors and regions [1–4]. Timely analysis matters, but so does scientific traceability. A plausible explanation is not enough when it may inform disaster assessment or response. Natural-hazard analysis applies Earth-system reasoning to decisions about event severity, exposure, physical mechanism, and response. It requires researchers to reconstruct an evolving process from observations collected for different purposes and at different scales. The difficulty begins with observation. No single instrument records an evolving Earth system in full. Agency reports, stations, satellites, reanalyses, maps, and impact databases measure different variables over different footprints and time windows. Combining them requires scientific judgment before calculation: the analyst must determine which records describe the same process, which scale is appropriate, and what uncertainty remains. Work on machine learning for Earth systems and model–data integration has made these records easier to use, but has not removed the need to reconcile them [5–8]. Foundation models and tool-using agents now make a broader form of analysis possible. Systems such as Google Earth AI combine imagery, population, and environmental models through a reasoning agent, while scientific agents can inspect files, execute code, and revise a result after new observations arrive [9–13]. This changes the evaluation target. A system has to decide what evidence to seek, bind each value to the correct source and scale, and let tool feedback alter its account when the evidence disagrees. Most Earth-science benchmarks begin after that decision has been made. They test knowledge, figure interpretation, remote sensing, or geospatial reasoning over a supplied passage, image, or named asset [14–16]. Newer multimodal and agent benchmarks broaden the interface, but generally continue to declare the observation or data layer in advance [17,18]. Frontier systems already approach or exceed published expert references on some of these formats. Such results do not show whether the same system can construct a compatible evidence base for an open question, carry calculations across several sources, and preserve support for every part of its conclusion. Position. Earth-science analysis requires systems to maintain the correspondence between heterogeneous observations and scientific claims. Satellite products, station records, reanalyses, and event reports can support a shared physical explanation only when their scales, variable meanings, time windows, units, and transformations remain aligned. EarthVerse evaluates this evolving claim–evidence state: a single broken binding can make a locally correct calculation describe the wrong Earth process [19, 20]. EarthVerse contains 405 reproducible investigations grounded in 199 real disasters and extreme events. Each task opens an event collection assembled from 80 external source and access families. A typical event contains about 34 files, and solving a task requires evidence from more than seven distinct sources on average. Every file is connected to the event, so source choice remains a scientific judgment. The difficulty lies in deciding which records answer the question, whether their time windows and spatial support are compatible, and how much weight each source should carry. Working through these materials takes sustained interaction. Across the leaderboard, each task takes about 17 model rounds on average. A full evaluation of a frontier model costs more than $2,500. Figure 2 samples this breadth, pairing the global event collection with representative investigations that move from heterogeneous observations to an auditable conclusion. EarthVerse uses transparent mathematics: aggregation, ratios, thresholds, spatial overlap, weighted indices, and bounded counterfactuals. The difficulty lies in combining these operations with compatible evidence. A wrong time window changes the total, which can alter both the comparison and the physical explanation. Tasks therefore combine process reconstruction, quantitative analysis, competing hypotheses, rankings, and open synthesis. Fine-grained answer units expose each obligation, while Strict@95 measures whether the resulting claim–evidence state is nearly complete. The results show how demanding this composition is. Across 25 systems, the best mean answer-unit accuracy is 84.65%, whereas the highest Strict@95 is only 34.81%. Even the strongest system leaves roughly two thirds of investigations with at least one consequential error or omission. This is the benchmark’s central reliability gap: an agent may recover most local facts and calculations while one unsupported source, mismatched scale, or missing mechanism breaks the account as a whole. Figure 1 places the benchmark’s scientific coverage beside the resulting model landscape. 2 EarthVerse: Benchmarking Scientific Agents Across Dynamic Earth Systems and Natural Hazards Figure 2: Benchmark overview. Global event coverage and four representative investigations. Each case follows a different model trajectory, and performance is scored against the answer units defined for that task. In Case 2, the linked-event score is a deterministic composite derived from package evidence. Contributions. We contribute (i) EarthVerse, a benchmark of reproducible multi-source investigations with exe- cutable answer units and task-specific process rubrics; (i) a controlled evaluation of 25 model and agent systems that separates average scientific competence from reliable completion; and (i) controlled experiments across evidence localization, tool selection, memory, reasoning, interaction modes, and professional scientific environments that identify the main failure points along the research chain. Research questions. We organize the study around three questions. RQ1: Can current systems complete an auditable multi-source hazard investigation, and which missing links separate high average accuracy from reliable completion? RQ2: When does reasoning improve a scientific investigation, and which controls over evidence access, tools, memory, stopping, and execution keep the claim–evidence state open to correction? RQ3: How much of scientific-agent performance comes from interpreting supplied observations, and how much depends on constructing and maintaining the evidence base? 2 Related Work 3 EarthVerse: Benchmarking Scientific Agents Across Dynamic Earth Systems and Natural Hazards Figure 3: Expert roles and scientific tools for investi- gation authoring. Earth observation and scientific multimodality. Most Earth-science benchmarks isolate a particular form of interpre- tation: scientific figures, atmospheric processes, geospatial rela- tions, or remote-sensing imagery [15–18,21,22]. Complemen- tary work studies how representations transfer across sensors, regions, and downstream tasks [23–26]. These studies establish strong perception and forecasting settings, but the observation set is generally prepared before the model begins its analysis. Disaster intelligence. Disaster benchmarks have followed a similar progression, from classification and damage mapping over supplied media to tool-supported emergency analysis [27– 29]. DORA moves closer to an operational workflow through expert-authored tasks, typed tools, and replayable reference calls, while retaining a declared set of relevant observations and lay- ers [30]. Tool-using and scientific agents. Agent research sup- plies the missing interaction machinery. Early work established reasoning–action loops and learned API use [10,31,32]; later benchmarks placed agents in executable environments and scientific workflows involving code, data, and long-horizon orchestration [11,33–37]. BLADE and DiscoveryBench are especially relevant because they allow multiple valid analysis paths while keeping the resulting work evaluable [38, 39]. EarthVerse couples expert authoring with a scientific tool suite (Figure 3). Experts frame and verify each investigation; numerical, atmospheric, and geospatial tools support executable analysis. Evaluation then scores source choice, intermediate work, and physical interpretation separately from the final answer. Geospatial and deep-research systems. Recent systems bring these strands together through remote-sensing tools, coordinated geospatial operations, iterative retrieval, or domain-specific pretraining [17,40–50]. EarthVerse evaluates the combined research problem: whether an agent can choose compatible evidence, use scientific tools appropriately, and maintain a coherent account as an event analysis develops. 3 The EarthVerse Benchmark EarthVerse contains 405 investigations grounded in 199 documented disasters and extreme events, covering 19 hazard families and 6,709 local files. Each task follows the rhythm of real post-event inquiry: establish what happened, identify the observations that can resolve the question, test the relevant quantities and mechanisms, reconcile disagreement, and state what the evidence supports. Packages preserve the complexity of the event without adding large collections of unrelated files merely to obstruct retrieval. Multi-source misalignment is part of the scientific problem. Reports, stations, satellite products, reanalyses, exposure layers, and impact records may use different windows, footprints, variables, and units. A solver must inspect these differences before choosing evidence, then manage the dependency between tools, sources, calculations, and physical reasoning. Computation is frequent but deliberately transparent: aggregation, normalization, ratios, thresholds, lags, spatial overlap, weighted indices, and bounded counterfactuals. EarthVerse does not test recall of undocumented formulas or background-heavy differential equations. It tests whether simple, checkable operations can support a complex investigation. 3.1 Design principles Four rules govern task construction. Event-scale scientific iteration. The solver first identifies the event window, geographic extent, hazard evolution, evidence families, and major inconsistencies. It then alternates between targeted evidence access, calculation, and physical interpretation, revising the event picture when an observation conflicts with the current account. Evidence discovery. The prompt names the event and scientific decision but not the relevant files. A released task requires 7.26 distinct sources on average, selected from a package with a median of 34 candidates. Across the release, evidence comes from dozens of distinct product and API families rather than one fixed archive. Tasks that reveal a path, filename, or answer-identifying quantity are rejected. 4 EarthVerse: Benchmarking Scientific Agents Across Dynamic Earth Systems and Natural Hazards Scientific checkability. Conclusions are scored with their supporting values, units, source roles, and mechanism tests. The benchmark tests whether an operation is attached to the right evidence, not whether specialized mathematics can be reproduced from memory. Path flexibility under package grounding. Report-first and time-series-first investigations can both receive full credit if they establish the required quantities and mechanism. Versioned packages keep the evidence and computation reproducible. 3.2 Fidelity to investigative practice Post-event studies usually begin with an archive rather than a curated figure. Analysts compare agency bulletins, station records at different aggregations, satellite and reanalysis products, impact databases, and incomplete metadata. One source may refine the time window; another may challenge the assumed mechanism; a third may reveal a spatial mismatch. EarthVerse reproduces this investigative sequence in a fixed, replayable setting. The scientific question defines the decision, answer units specify what must be established, and the rubric scores source choice, calculation, revision, and causal argument. The benchmark stays close to disaster practice without relying on live feeds or unstated specialist knowledge. 3.3 Task construction and answer contracts Each event contributes two tasks, with a third for seven flagship events. The public prompt states a scientific decision but does not identify the relevant files. Every task includes an expert solution, structured ground truth, a deterministic compute_gt.py program, and a task-specific process rubric. The task and evaluation flow is (q i ,P i ) π −→ τ i = (a 1 ,o 1 ,...,a T ,o T )−→ ˆy i , E i = [f (ˆy i ,y ⋆ i ),g(τ i ;s i ,B i )]. (1) Figure 4: Task interface. Metadata, observations, and an auditable research trajectory. For taski, the solver sees questionq i and evidence col- lectionP i . Policyπproduces trajectoryτ i and answerˆy i . Functionfcompares the answer with structured ground truthy ⋆ i ;gevaluates the trajectory against expert anal- ysiss i and rubricB i without prescribing a tool order. The release contains 10,879 answer units. We use units because a single choice cannot show whether a system recovered the correct window, ranking, quantity, mecha- nism, or qualification. Each unit is a nontrivial scientific obligation, and tasks can combine numerical results with orderings and open-ended conclusions. This remains discriminative when frontier models already perform ex- tremely well on simpler multiple-choice and short-answer formats. A worked example. One task examines the 2021 Pa- cific Northwest heat wave. The solver must distinguish a direct hourly record from a lower-resolution aggregate, identify the matching heat window, and compute heat and humidity statistics on the same temporal support. The arithmetic is elementary; the scientific work lies in choosing compatible observations and carrying that choice into the conclusion. 3.4 Capabilities and research environment Capability coverage. Each task receives one to three manually reviewed capability labels: physical-mechanism reasoning, spatiotemporal reconstruction, quantitative calculation, multi-source synthesis, causal-chain reasoning, ranking or decision, and remote-sensing or geospatial interpretation. Labels may overlap. Conditioned scores are diagnostic and do not change the overall metric. Figure 1(a) shows that these capabilities recur across hazard families rather than forming separate task silos. Figure 4 illustrates how a solver links task metadata, observations, and its research trajectory. 5 EarthVerse: Benchmarking Scientific Agents Across Dynamic Earth Systems and Natural Hazards Research tools. We selected six general operations that are sufficient to express every released workflow: discover files, read a source, search local text, inspect structured data, execute scoped Python, and finalize the answer. This common interface makes system comparisons consistent while leaving the scientific strategy open. The registry also supports custom tool registration and retains 170 reusable operations for reports, time series, vector and exposure analysis, raster and Earth observation, meteorology and climate, fire and geophysics, infrastructure, and output assembly. Structured returns separate execution failure from scientific disagreement. 3.5 Quality assurance and benchmark comparison Construction and review. Construction proceeds through event verification, package assembly, task authoring, executable validation, and release review (Figure 5). Automated tools support the process, but every released task is checked and corrected by people. Review covers event identity, source reliability, temporal and spatial fit, the public question, expert solution, ground truth, executable program, capability labels, and rubric. Package-local data must regenerate every deterministic value, and the visible assignment must match the hidden scoring target. Appendix C.6 describes the human and agent review process; Appendix C.7 records the seven release gates. A separate LLM pairwise comparison offers a second check on score validity. Its win–tie–loss pattern broadly follows the Core leaderboard across system families (Figure 7). Systems with higher Core are also preferred when complete answers and traces are compared directly, so the ranking is not driven solely by the number of satisfied scoring fields. Position relative to prior benchmarks. Table 1 compares what the model is asked to do, what evidence it receives, how experts participate, and what is scored. EarthSE, MSEarth, and GeoMMBench usually provide one selected passage, figure, or image [15–17]. ThinkGeo, Earth-Agent, OpenEarthAgent, and DORA add tools around declared imagery, assets, or layers [30,40–42]. EarthVerse instead asks the system to construct and document a multi-source evidence base whose contents depend on the disaster and the scientific question. Table 1: Where the scientific work begins. The two text columns identify the starting evidence and analytical work; marks show full, partial, or absent/unreported coverage. BenchmarkStart evidenceScientific workTools Find evidence Multi- source Compute Unit score Trace score Human review EarthSEPaperSci. QA× MSEarthFigureFigure QA× GeoMMBenchImageGeo reasoning× ThinkGeoRS imageTool use✓×✓ Earth-AgentEO assetsEO planning✓×✓ OpenEarthAgentImageryMultimodal✓×✓× DORALayersCrisis analysis✓×✓ EarthVersePackageEvent synthesis✓ ✓fullpartial×absent/unreportedEarthVerse scale: 34 files/package; 7.26 sources and 26.86 units/task. EarthVerse places the difficulty in scientific assembly rather than specialized mathematics. Fine-grained answer units state what must be established; the process rubric tests whether the chosen path supports those claims. 4 Evaluation Protocol EarthVerse fixes the scientific question, visible evidence boundary, execution budget, and answer contract across systems. During a run, the system selects evidence, performs calculations, records provenance, and submits a structured scientific answer through the same interface. The full inference configuration, model modalities, and controlled-study manifests are reported in Appendix C. 4.1 Outcome, process, and reliability metrics For taski, letU i be the percentage of required answer units satisfied andH i the holistic answer judgment, both on a 0–100 scale. Answer correctness isA i = min(H i ,U i ): a fluent answer cannot outrank the scientific obligations it actually completes. Let P i be the task-specific process-rubric score on the same scale. The primary score is Combined Core = 1 2N N X i=1 (A i + P i ).(2) 6 EarthVerse: Benchmarking Scientific Agents Across Dynamic Earth Systems and Natural Hazards Figure 5: Benchmark construction. Event grounding, package assembly, investigation authoring, and expert audit. It therefore gives equal weight to endpoint correctness and the documented research process. Answer units are necessary because one investigation may require a time window, several quantities, a source comparison, and a physical mechanism. A single holistic grade would hide which obligation failed and could let a fluent conclusion mask missing scientific work. Mean unit accuracy averagesU i across tasks and captures partial completion of required scientific claims and output fields; numerical units use task-specific tolerances. Strict@95 is the percentage of tasks withU i ≥ 95and measures near- complete task reliability rather than average progress. Capability-conditioned scores apply the same Core calculation to expert-tagged task slices. Trajectory statistics report interaction, evidence access, computation, latency, and token demand, but never enter the score. Solver failures receive zero; judge-service failures are rerun until a valid judgment is obtained. Path pluralism and diagnostic validity. EarthVerse does not prescribe a tool order. Earth-Agent and DORA instead evaluate whether an agent follows a predefined tool or operational sequence. This is useful when a workflow has a canonical procedure, but complex multi-source investigations rarely admit a single defensible trajectory ground truth: report-first and time-series-first analyses may reach the same well-supported conclusion. EarthVerse therefore scores what the system establishes and whether the observed evidence and calculations support it. Tools and traces serve reasoning, review, and diagnosis; reproducing a prescribed path earns no credit by itself. 4.2 Evaluated systems Table 2 compares hosted systems, open-weight models, and research frameworks through one package interface. Some backbones are text-only, while others support native multimodal input; the shared leaderboard does not grant extra evidence to either group. All systems receive the same question, package boundary, answer contract, and execution budget. Appendix C.1 gives the complete roster, modality treatment, controller exceptions, and judge configuration. 5 Main Evaluation The results follow the three questions. The leaderboard and reliability diagnostics in Section 5.1 answer RQ1. Reasoning, interaction, and evidence-path interventions in Section 5.2 answer RQ2. Section 5.3 addresses RQ3 by comparing supplied-observation performance with evidence construction. Appendix B.1 develops the resulting position. 7 EarthVerse: Benchmarking Scientific Agents Across Dynamic Earth Systems and Natural Hazards 5.1 Overall system performance RQ1 concerns completion at the investigation level. We begin by asking whether current systems can keep the whole account valid when most individual parts are correct. Setting. Table 2 reports all 25 systems under the package-scoped interactive protocol. The Harness column names the controller used to run each model. Core averages Answer correctness and Process-rubric score; Strict@95 counts tasks satisfying at least 95% of answer units. Trajectory columns are per-task means and do not affect quality scores. Systems are grouped by family and ordered from lower to higher Core. Table 2: EarthVerse leaderboard. Harness names the controller. Scores are percentages; failed, timed-out, or malformed runs remain zero, and batch reads may cover multiple files. Within each family, Core increases downward; bold blue and underlined tealmark the top two outcomes. System configurationOutcomeTrajectory mean per task SystemHarnessAnswer Process Core Strict@95 (%) Unit acc. Rounds Tool calls Reads Files Python Latency (s) Tokens (k) Agent frameworks (own controllers over the same package interface) OpenResearcherOpenResearcher30.5131.06 30.794.94 32.7439.73 33.62 15.00 17.035.38521538.7 GPT-5.5GeoMMAgent65.8670.84 68.3512.35 67.185.008.28 11.938.590.9922137.9 Open-weight and Earth-specialized models Intern-S1-miniEarthVerse10.7812.67 11.720.25 11.604.993.431.802.390.075739.4 Nemotron NanoEarthVerse12.3014.01 13.150.25 12.802.592.071.261.770.1527953.2 MiroThinker 8BEarthVerse12.7913.62 13.212.96 13.4227.30 25.986.07 11.422.00367318.0 GeoGPTEarthVerse22.1527.32 24.740.25 23.6130.41 29.37 19.33 11.862.65565502.1 Qwen3-4BEarthVerse25.1124.94 25.020.00 26.135.063.792.553.660.0713237.7 GeohazardGPTEarthVerse28.8833.33 31.100.49 30.2730.33 29.34 13.38 15.708.44456333.7 Qwen3-235BEarthVerse31.9834.64 33.310.99 32.6613.47 12.805.337.413.50195274.3 Hosted general-purpose systems GPT-5.6 LunaCodex35.6835.63 35.6611.11 36.286.585.392.374.291.05189156.9 GPT-4oEarthVerse37.3541.11 39.230.49 38.6918.818.954.046.840.6955127.4 MiniMax M2.7EarthVerse43.6645.77 44.717.41 44.3517.27 16.087.118.942.22205285.8 Gemini 3.1 Flash Lite EarthVerse57.6761.66 59.662.96 58.677.376.103.595.691.082470.1 Doubao Seed 2.0 Mini EarthVerse61.1464.84 62.995.93 62.2213.71 12.707.218.861.08113175.1 Qwen 3.7 PlusEarthVerse69.3371.37 70.3518.27 70.1617.15 15.726.51 11.253.31313204.3 Claude Haiku 4.5EarthVerse70.7673.70 72.2314.07 71.7021.31 19.86 10.08 11.723.96324477.0 Tencent Hy3EarthVerse71.2774.26 72.7716.30 72.2219.47 17.778.32 12.704.41311208.7 DeepSeek V4 FlashEarthVerse72.3274.77 73.5420.25 73.2527.92 26.81 11.00 16.384.07333389.7 Kimi K2.5EarthVerse73.8976.76 75.3214.57 74.7024.25 22.67 10.01 16.004.79439302.9 Claude Sonnet 4.6EarthVerse74.8877.44 76.1620.00 75.7617.54 16.547.72 11.824.49343331.0 GPT-5.5EarthVerse76.8579.28 78.0617.04 77.4815.17 14.206.83 10.923.74210224.9 GLM-5.2EarthVerse78.1480.19 79.1722.96 78.6218.32 17.277.52 13.364.31259196.1 GPT-5.6 TerraCodex80.1280.00 80.0627.16 80.8112.18 11.134.429.462.71173182.6 GPT-5.6 SolCodex81.43 81.9481.6834.81 82.1213.88 12.794.889.652.87205205.9 Claude Fable 5Claude Code83.9985.9584.9725.9384.6514.4513.455.9210.022.61247161.6 The landscape in Figure 1(b) shows that similar token budgets can produce sharply different answer-unit accuracy. Mean competence versus conjunctive reliability. Claude Fable 5 leads on Core (84.97), while GPT-5.6 Sol leads on Strict@95 (34.81%). Fable satisfies 84.65% of answer units on average but reaches 95% completion on only 25.93% of tasks; Sol leaves fewer investigations with a consequential omission. Missing windows, provenance, alternative mechanisms, and unit conversions are coupled errors: a wrong window changes the calculation and its physical interpretation. Low Strict@95 rates show that current systems often know most of the analysis without keeping the whole analysis valid at once. Finding. Reliability requires the whole evidence chain. High average accuracy can hide one missing source, window, unit, calculation, or mechanism. In disaster analysis, that single break can invalidate an otherwise plausible account. Coupling reasoning with interaction. Most leaderboard systems use the EarthVerse harness, a common multi- round controller configured for evidence discovery, calculation, provenance, recovery, and structured finalization. 8 EarthVerse: Benchmarking Scientific Agents Across Dynamic Earth Systems and Natural Hazards Holding this scaffold fixed makes backbone comparisons fair, while framework rows retain their native controllers. Interaction volume still does not predict quality. OpenResearcher uses 39.73 rounds and 538.7k tokens per task, yet trails Claude Fable 5 by 54.18 Core points because weak sources and stale values survive into late revisions. GeoMMAgent is too brief in the opposite direction: five rounds leave evidence undiscovered and Strict@95 reaches only 12.35%. Useful interaction updates the claim–evidence state and carries the revision into the answer. Finding. Useful interaction changes the scientific state. A tool call matters when its observation changes the chosen source, calculation, or physical account. Longer traces that preserve the same mistake add cost, not quality. Domain knowledge and frontier reasoning. Earth-specialized pretraining improves terminology and process recognition [48–50], but the evaluated open models remain 40–70 Core points behind frontier general systems. GeohazardGPT averages 30.33 rounds and 8.44 Python calls per task yet reaches 31.10 Core. The gap appears after recognition, in source selection, provenance, revision, and stopping. Domain training remains useful, but in these systems it has not yet produced a reliable research loop. Capability slices in Appendix D.1 sharpen this result. Several systems rise above their own mean on calculation, spatiotemporal reconstruction, or remote sensing, yet fall on causal reasoning and ranking. Overall strength does not produce a uniform scientific profile. Computational cost. Evaluating a frontier model costs more than $2,500, while flash models remain below $45. GPT-5.6 Terra retains 98% of GPT-5.6 Sol’s Core at less than half the cost. Cost depends on both model pricing and controller behavior, so neither trace length nor token count alone is an efficiency measure. Terra offers the strongest quality–cost balance among the top systems, whereas low-cost Flash models are useful for broad screening but do not match frontier reliability. Appendix D.3.3 reports the full comparison. 5.2 Reasoning, interaction, and evidence control Table 3: Reasoning and evidence access. GPT-5.5 across four access modes and five effort settings; diagnostics are per-task means. OutcomeInteractionEvidenceResources EffortAnswer Process Core Unit acc. Rounds Calls Files Refs Tokens (k) Latency (s) Compressed direct none41.3745.1043.2442.21–13.955.6 low24.9929.32 27.16 25.43–14.9120.5 medium22.8825.69 24.29 23.26–15.8152.4 high24.1728.17 26.17 24.28–16.671.5 xhigh16.6716.67 16.67 16.67–17.688.1 All-text direct none27.6631.2229.4428.27–81.632.1 low13.2616.20 14.73 13.36–82.744.1 medium9.7912.18 10.99 10.05–83.561.0 high8.809.00 8.90 8.82–85.478.5 xhigh12.0015.00 13.50 12.00–87.794.4 EarthVerse interactive none48.8953.70 51.30 49.469.68 8.68 7.00 9.9496.9140 low56.9760.05 58.51 57.5314.70 13.70 9.44 14.70162.4259 medium61.4564.80 63.13 62.3517.20 16.20 10.71 18.12206.4424 high72.4175.1973.8072.5914.9613.969.9217.19174.3571 xhigh69.6667.50 68.58 69.8311.25 10.25 10.25 16.50100.0674 Interactive with final review none53.7056.48 55.09 54.0611.62 9.62 7.42 10.46130.7108 low61.9365.50 63.72 62.7818.18 16.18 10.60 17.02227.0253 medium67.1669.22 68.19 67.9120.81 18.81 11.88 21.02266.3552 high69.8274.71 72.27 70.7320.63 18.63 12.13 21.10255.11,105 xhigh84.3688.2186.2984.4017.5715.5710.1418.29217.71,493 For RQ2, we separate additional deliberation from interventions that let the system revise its claim–evidence state. Tables 3 and 4 ex- amine complementary parts of that loop: how evidence can be used during reasoning, and what the system can locate or act on. Table 3 shows a sharp split between fixed and editable evidence. With compressed or all-text direct input, the best result comes from no addi- tional reasoning; high effort lowers Core from 43.24 to 26.17 and from 29.44 to 8.90, respec- tively. Once the system can retrieve evidence and act on what it finds, Core rises from 51.30 with no extra effort to 73.80 at high effort. Final review changes the best operating point again: xhigh reaches 86.29, compared with 68.58 with- out review. Reasoning helps when it can trigger retrieval, recomputation, or correction. Delib- erating longer over the same snapshot often deepens the initial mistake. Resource use reinforces this distinction. High effort in the interactive setting reaches 73.80 Core with 14.96 rounds and 174.3k tokens; medium effort uses more rounds and tokens but reaches only 63.13. Additional steps help when they change the evidence set, repair a calculation, or resolve a conflict that reaches the final answer. Trace length alone says little about the quality of the investigation. Final review also has a conditional effect. At xhigh, it raises Core from 68.58 to 86.29, but at high effort it changes 73.80 to 72.27. Review is most useful when the investigation still contains evidence dependencies or calculations that 9 EarthVerse: Benchmarking Scientific Agents Across Dynamic Earth Systems and Natural Hazards can be checked. Once the claim–evidence state is coherent, another pass may add cost without improving the answer. This favors targeted verification over a fixed rule that every task needs more reasoning. To locate the remaining bottlenecks, Table 4 uses answer-free oracles that remove one obstacle at a time: they reveal the relevant files, evidence roles, useful tool classes, or a high-level plan, but never the target answer. The same paired tasks vary tool presentation, package evidence, and access to meteorological operations. The meteorological conditions compare a fixed compact suite with a cue-routed subset; tool use remains optional, so this probe asks whether specialist access alone improves the investigation. Table 4: Evidence-path interventions. Complete paired outcome, effect, and capability grid relative to the Top-20 router. Intervals are paired 95% bootstrap CIs. OutcomePaired effectCore by capability InterventionAnswer Process Core ∆ Core 95% CI Physical mech. Spatio- temporal Quant. calc. Multi- source Causal chain Rank sort RS / geo. Top-20 semantic-router baseline66.9065.8866.390.00–65.1564.4073.4390.0065.8950.8453.94 Answer-free information oracles Relevant-file oracle79.4682.75 81.11 +14.72 [+5.48, +24.69]80.6191.47 87.57 90.45 76.53 60.70 72.09 Evidence-map oracle76.9678.50 77.73 +11.34 [+2.34, +21.25]76.6990.50 83.83 97.50 73.35 58.16 61.25 Tool-subset oracle68.5368.62 68.58 +2.19 [−3.55, +7.28]67.3265.67 72.60 92.50 69.54 59.14 59.75 Planning oracle65.1966.50 65.84−0.54 [−7.07, +4.35]64.3157.10 71.24 95.00 67.93 54.76 60.25 Tool-interface presentation Hierarchical catalog66.7667.62 67.19 +0.80 [−9.18, +11.86]65.8166.72 67.83 93.50 70.78 56.60 61.67 Descriptive tool names66.8866.75 66.82 +0.43 [−3.55, +4.05]65.0761.21 72.92 100.00 66.98 57.59 55.23 Dynamic descriptions61.3961.75 61.57−4.82 [−15.02, +4.33]59.8161.56 63.81 95.00 61.27 54.58 60.75 Opaque tool names56.7259.50 58.11−8.28 [−23.57, +6.33]56.1648.88 58.54 95.22 59.76 54.01 80.00 Evidence perturbations Equivalent unit change67.6171.12 69.37 +2.98 [−9.75, +15.53]68.0266.93 71.44 95.00 71.29 55.47 76.38 Added distractor files65.9167.75 66.83 +0.44 [−6.80, +7.27]65.4660.47 72.15 92.95 65.16 59.98 70.62 Missing evidence63.9764.75 64.36−2.03 [−12.05, +7.08]62.4960.31 67.82 100.00 66.63 54.74 55.12 Truncated evidence63.1763.38 63.27−3.12 [−14.13, +7.47]61.7855.67 70.53 91.70 64.24 41.52 68.92 Conflicting evidence63.0363.12 63.08−3.31 [−12.05, +2.97]61.5067.97 66.63 92.95 60.46 55.19 54.50 Optional meteorological access Compact meteorology suite62.6965.38 64.03−2.36 [−11.92, +5.89]62.1461.97 66.67 100.00 65.22 56.15 58.47 Cue-routed meteorology57.2259.88 58.55−7.84 [−19.40, +3.93]56.7458.87 59.25 92.95 60.28 50.45 57.25 Finding. Reasoning needs an editable claim–evidence state. Additional reasoning helps when it can change the investigation. Interactive retrieval and final review let the system replace a weak source, recompute a quantity, and carry the correction into the final answer. With a fixed evidence snapshot, the same effort can reinforce an early error. Table 4 identifies what changes that investigation. Relevant-file localization and the evidence map raise Core by 14.72 and 11.34 points, and they are the only interventions whose confidence intervals exclude zero. A curated tool subset adds 2.19 points, while the planning oracle slightly lowers performance. Renaming tools, normalizing units, and exposing optional meteorological tools also fail to reproduce the localization gain. More tools or more procedural guidance are therefore not enough. The agent must find the observations that determine the scientific object before it can reason reliably about that object. The capability columns make the mechanism more precise. Relevant-file localization lifts spatiotemporal Core from 64.40 to 91.47, physical-mechanism Core from 65.15 to 80.61, and quantitative-calculation Core from 73.43 to 87.57. Multi-source performance changes little because the baseline already covers many sources; the gain comes from selecting and aligning the right ones. Missing, truncated, or conflicting evidence lowers overall quality, although the paired intervals remain wide. In this study, evidence localization is the clearest reproducible bottleneck, and downstream reasoning quality depends on whether that step succeeds. 5.3 Cross-benchmark transfer and benchmark adaptation RQ3 separates the ability to interpret a supplied observation from the longer task of constructing and maintaining the evidence base. We evaluate GPT-5.5 under the public protocols of Earth-Bench, EarthSE, MSEarth, and GeoMMBench. Each comparison begins with a direct response, then repairs clearly erroneous benchmark items or judgments, and finally applies the structured EarthVerse procedure. This procedure is answer-free. We summarize recurring question types across the benchmark and provide a general guide for identifying the relevant variables, choosing the necessary 10 EarthVerse: Benchmarking Scientific Agents Across Dynamic Earth Systems and Natural Hazards calculation, and checking whether the conclusion follows from the supplied observation. The guide contains no item answer, target label, or item-specific evidence. Figure 6 summarizes a benchmark-native score for each stage and includes the relevant published reference [16,17]. Because the benchmarks define different metrics, the figure supports comparisons within each benchmark, not rankings across the four groups. Complete native metrics and source-paper rows accompany the release. Figure 6: Cross-benchmark protocol sensitivity. Benchmark-native scores across three GPT-5.5 stages and published references. The broad pattern is consistent, although the source of the gain differs by benchmark. On Earth-Bench, direct GPT-5.5 already clears the published line and improves further when the answer contract is made explicit. EarthSE is less uniform: multiple-choice and free-response results are already high, while fill-in accuracy rises from 20.29 to 61.90 after schema and answer-granularity corrections. MSEarth and GeoMMBench show the clearest recovery from interface mismatch. Direct GPT-5.5 trails the displayed reference, whereas the corrected and EarthVerse stages close or reverse the gap. The improvement comes from making the problem legible to the model, not from revealing a solution or teaching new Earth-science knowledge during evaluation. Two changes are especially clear: EarthSE fill-in accuracy rises from 20.29 to 61.90, while GeoMMBench accuracy rises from 84.52 to 98.01, above both the published GeoMMAgent result of 88.40 and the reported human reference of 86.50. Under their native metrics, the tested frontier model handles these supplied-observation tasks well once the question structure is explicit. These results concern tasks in which a passage, figure, or named observation has already been selected and the required output is clear. Current LLMs and agents are less reliable when they must decide which sources matter, align records with different support, verify calculations, preserve provenance, and revise downstream conclusions. This difference explains why adapted performance can be high while EarthVerse Strict@95 remains below 35%. Finding. Long-horizon evidence construction remains unresolved. Answer-free structural guidance brings a frontier model to or above published reference levels on supplied-observation benchmarks, including the displayed human lines. EarthVerse Strict@95 remains below 35% because the agent must construct, check, and revise the evidence base itself. 6 Conclusion RQ1: scientific reliability depends on the complete claim–evidence state. Current systems often recover much of an analysis, yet one unsupported source, scale mismatch, or missing mechanism can change the result. Progress depends on preserving the links among claims, sources, scales, and calculations throughout the investigation. 11 EarthVerse: Benchmarking Scientific Agents Across Dynamic Earth Systems and Natural Hazards RQ2: reasoning helps when control keeps the claim–evidence state editable. More effort over a fixed serialization does not recover interactive performance. Evidence localization and targeted review help when they let the system replace a weak source, recompute a quantity, and carry the correction into its final account. The controller must preserve claim–source bindings, check method applicability, and stop once the required evidence is complete. RQ3: interpreting evidence is easier than constructing it. Within benchmarks that supply the observation, structured adaptation reaches or exceeds published reference lines. EarthVerse remains substantially harder because the system must decide what belongs in the analysis and keep the selected evidence compatible through calculation and synthesis. EarthVerse contributes an executable benchmark and controlled diagnostics for multi-source hazard research. The leaderboard and interventions point to the same limitation: agents often recover valid local facts but fail to carry source changes and corrected quantities into the final account. By tracing that failure across evidence, computation, and reporting, EarthVerse connects model performance to the reliability of the resulting scientific analysis. Author Contributions Zhiqing Cui led the project and was responsible for benchmark conceptualization, methodology, data curation, ex- perimentation, and writing the original manuscript. Xinxiang Yin and Xinglang Zhang developed the figures and visual presentation. Yuanzhe Hu and Weidong Tang coordinated review and quality control. Siru Zhong contributed to conceptual development and scientific discussion. Yihong Tang contributed methodological guidance, benchmark conceptualization, and scientific discussion. Dingyi Zhuang provided senior supervision, methodological guidance, and funding acquisition. Jinhua Zhao provided overall supervision, and project leadership. Yuxuan Liang, Weijia Li, Ming Jin, Shirui Pan, and Yuhao Kang contributed scientific discussion and critical review. All authors reviewed and approved the manuscript. References [1]World Meteorological Organization. Atlas of mortality and economic losses from weather, climate and water- related hazards, 1970–2021. Technical report, World Meteorological Organization, 2023. URLhttps://public .wmo.int/publication-series/atlas-of-mortality-and-economic-losses-from-weather-cli mate-and-water-related-hazards-1970-2021. [2]World Meteorological Organization. State of the global climate 2024. Technical Report WMO-No. 1368, World Meteorological Organization, 2025. URLhttps://wmo.int/publication-series/state-of-global-c limate/state-of-global-climate-2024. [3] Intergovernmental Panel on Climate Change (IPCC). Climate Change 2022: Impacts, Adaptation and Vulnerability. Contribution of Working Group I to the Sixth Assessment Report of the Intergovernmental Panel on Climate Change. Cambridge University Press, 2022. doi: 10.1017/9781009325844. URLhttps://w.ipcc.ch/repo rt/ar6/wg2/. [4]Jakob Zscheischler, Seth Westra, Bart J. J. M. van den Hurk, Sonia I. Seneviratne, Philip J. Ward, Andy Pitman, Amir AghaKouchak, David N. Bresch, Michael Leonard, Thomas Wahl, and Xuebin Zhang. Future climate risk from compound events. Nature Climate Change, 8(6):469–477, 2018. doi: 10.1038/s41558-018-0156-3. [5]Markus Reichstein, Gustau Camps-Valls, Bjorn Stevens, Martin Jung, Joachim Denzler, Nuno Carvalhais, and Prabhat. Deep learning and process understanding for data-driven Earth System Science. Nature, 566:195–204, 2019. doi: 10.1038/s41586-019-0912-1. [6]Yannis Markonis, Christoforos Pappas, Martin Hanel, and Simon Michael Papalexiou. A cross-scale framework for integrating multi-source data in Earth System Sciences. Environmental Modelling & Software, 139:104997, 2021. doi: 10.1016/j.envsoft.2021.104997. [7] Andrew Gettelman, Alan J. Geer, Richard M. Forbes, Greg R. Carmichael, Graham Feingold, Derek J. Posselt, Graeme L. Stephens, Susan C. van den Heever, Adam C. Varble, and Paquita Zuidema. The future of Earth System Prediction: Advances in model–data fusion. Science Advances, 8(14):eabn3488, 2022. doi: 10.1126/sciadv.abn34 88. [8]Zhiqing Cui, Binwu Wang, Qingxiang Liu, Yeqiang Wang, Zhengyang Zhou, Yuxuan Liang, and Yang Wang. Augur: Modeling covariate causal associations in time series via large language models. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 764–787. Association for Computational Linguistics, 2026. doi: 10.18653/v1/2026.acl- long.32. URL https://aclanthology.org/2026.acl-long.32/. 12 EarthVerse: Benchmarking Scientific Agents Across Dynamic Earth Systems and Natural Hazards [9]Aaron Bell, Amit Aides, Amr Helmy, Arbaaz Muslim, Aviad Barzilai, Aviv Slobodkin, Bolous Jaber, et al. Earth AI: Unlocking geospatial insights with foundation models and cross-modal reasoning. arXiv preprint arXiv:2510.18318, 2025. [10]Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. ReAct: Synergizing reasoning and acting in language models. In International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=WE_vluYUL-X. [11] Ziru Chen, Shijie Chen, Yuting Ning, Qianheng Zhang, Boshi Wang, Botao Yu, Yifei Li, Zeyi Liao, Chen Wei, Zitong Lu, et al. ScienceAgentBench: Toward rigorous assessment of language agents for data-driven scientific discovery. In International Conference on Learning Representations, 2025. URLhttps://openreview.net/f orum?id=6z4YKr0GK6. [12]Ming Hu, Chenglong Ma, Wei Li, Wanghan Xu, Jiamin Wu, Jucheng Hu, Tianbin Li, Guohang Zhuang, Jiaqi Liu, Yingzhou Lu, et al. A survey of scientific large language models: From data foundations to agent frontiers. arXiv preprint arXiv:2508.21148, 2025. [13]Niloufar Alipour Talemi, Julia Boone, and Fatemeh Afghah. Agentic AI in remote sensing: Foundations, taxonomy, and emerging systems. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision Workshops, pages 786–799, 2026. doi: 10.1109/WACVW68408.2026.00088. URLhttps: //openaccess.thecvf.com/content/WACV2026W/GeoCV/html/Talemi_Agentic_AI_in_Remote_Sen sing_Foundations_Taxonomy_and_Emerging_Systems_WACVW_2026_paper.html. [14]Zicheng Zhang, Junying Wang, Farong Wen, Yijin Guo, Xiangyu Zhao, Xinyu Fang, Shengyuan Ding, Ziheng Jia, Jiahao Xiao, Ye Shen, et al. Large multimodal models evaluation: a survey. Science China Information Sciences, 68(12):221301, 2025. doi: 10.1007/s11432-025-4676-4. [15]Wanghan Xu, Xiangyu Zhao, Yuhao Zhou, Xiaoyu Yue, Ben Fei, Fenghua Ling, Wenlong Zhang, and Lei Bai. EarthSE: A benchmark evaluating earth scientific exploration capability for large language models. In International Conference on Learning Representations, 2026. URL https://openreview.net/forum?id=jyYE06FL8G. [16]Xiangyu Zhao, Wanghan Xu, Bo Liu, Yuhao Zhou, Fenghua Ling, Ben Fei, Xiaoyu Yue, Lei Bai, Wenlong Zhang, and Xiao-Ming Wu. MSEarth: A multimodal benchmark for earth science phenomenon discovery with MLLMs. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 5270–5301, 2026. doi: 10.18653/v1/2026.acl- long.239. URLhttps: //aclanthology.org/2026.acl-long.239/. [17]Aoran Xiao, Shihao Cheng, Yonghao Xu, Yexian Ren, Hongruixuan Chen, and Naoto Yokoya. GeoMMBench and GeoMMAgent: Toward expert-level multimodal intelligence in geoscience and remote sensing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 34843–34853, 2026. [18]Chenyue Li, Wen Deng, Mengqian Lu, and Binhang Yuan. AtmosSci-Bench: evaluating the recent advance of large language model for atmospheric science. In Advances in Neural Information Processing Systems, volume 38, 2025. doi: 10.52202/085713-4850. [19]Mingxuan Du, Benfeng Xu, Chiwei Zhu, Licheng Zhang, Xiaorui Wang, and Zhendong Mao. DeepResearch Bench: A comprehensive benchmark for deep research agents. In International Conference on Learning Repre- sentations, 2026. URL https://openreview.net/forum?id=hQ0K2Hhq7H. [20]Yigeng Jiang, Tengchao Yang, Taoyong Cui, Jiaxing Wan, Yuan Wang, Weida Wang, Zhiyu Liu, Chuyi Peng, Binzhao Luo, Maoli Gao, et al. Deep research in physical sciences: A multi-agent framework and comprehensive benchmark. arXiv preprint arXiv:2606.18648, 2026. [21]Muhammad Sohail Danish, Muhammad Akhtar Munir, Syed Roshaan Ali Shah, Kartik Kuckreja, Fahad Shahbaz Khan, Paolo Fraccaro, Alexandre Lacoste, and Salman Khan. GEOBench-VLM: Benchmarking vision-language models for geospatial tasks. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7132–7142, 2025. doi: 10.1109/ICCV51701.2025.00670. [22] Fengxiang Wang, Hongzhen Wang, Zonghao Guo, Di Wang, Yulin Wang, Mingshuo Chen, Qiang Ma, Long Lan, Wenjing Yang, Jing Zhang, et al. XLRS-Bench: Could your multimodal LLMs understand extremely large ultra-high-resolution remote sensing imagery? In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 14325–14336, 2025. doi: 10.1109/CVPR52734.2025.01336. [23]Daniela Szwarcman, Sujit Roy, Paolo Fraccaro, Þorsteinn Elí Gíslason, Benedikt Blumenstiel, Rinki Ghosal, Pedro Henrique de Oliveira, João Lucas de Sousa Almeida, Rocco Sedona, Yanghui Kang, et al. Prithvi- EO-2.0: A versatile multitemporal foundation model for Earth Observation applications. IEEE Transactions on Geoscience and Remote Sensing, 64:1–20, 2026. doi: 10.1109/TGRS.2025.3642610. URLhttps: //ieeexplore.ieee.org/document/11296896/. 13 EarthVerse: Benchmarking Scientific Agents Across Dynamic Earth Systems and Natural Hazards [24]Johannes Jakubik, Felix Yang, Benedikt Blumenstiel, Erik Scheurer, Rocco Sedona, Stefano Maurogiovanni, Jente Bosmans, Nikolaos Dionelis, Valerio Marsocci, Niklas Kopp, et al. TerraMind: Large-scale generative multimodality for earth observation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7383–7394, 2025. doi: 10.1109/ICCV51701.2025.00693. [25]Naomi Simumba, Nils Lehmann, Paolo Fraccaro, Hamed Alemohammad, Geeth De Mel, Salman Khan, Manil Maskey, Nicolas Longepe, Xiao Xiang Zhu, Hannah Kerner, et al. GEO-Bench-2: From performance to capability, rethinking evaluation in geospatial AI. arXiv preprint arXiv:2511.15658, 2025. [26] Zhiqing Cui, Siru Zhong, Ming Jin, Shirui Pan, Qingsong Wen, and Yuxuan Liang. Breaking the regional barrier: Inductive semantic topology learning for worldwide air quality forecasting. arXiv preprint arXiv:2601.21899, 2026. doi: 10.48550/arXiv.2601.21899. URL https://arxiv.org/abs/2601.21899. [27]Firoj Alam, Ferda Ofli, and Muhammad Imran. CrisisMMD: Multimodal Twitter datasets from natural disasters. In Proceedings of the International AAAI Conference on Web and Social Media, volume 12, pages 465–473. AAAI Press, 2018. doi: 10.1609/icwsm.v12i1.14983. [28]Ritwik Gupta, Bryce Goodman, Nirav Patel, Ricky Hosfelt, Sandra Sajeev, Eric T. Heim, Jigar Doshi, Keane Lucas, Howie Choset, and Matthew E. Gaston. Creating xBD: A dataset for assessing building damage from satellite imagery. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pages 10–17, 2019. URLhttps://openaccess.thecvf.com/content_CVPRW_2019/html/cv 4gc/Gupta_Creating_xBD_A_Dataset_for_Assessing_Building_Damage_from_Satellite_CVPRW_ 2019_paper.html. [29] Maryam Rahnemoonfar, Tashnim Chowdhury, Argho Sarkar, Debvrat Varshney, Masoud Yari, and Robin Roberson Murphy. FloodNet: A high resolution aerial imagery dataset for post flood scene understanding. IEEE Access, 9: 89644–89654, 2021. doi: 10.1109/ACCESS.2021.3090981. [30]Junjue Wang, Weihao Xuan, Heli Qi, Pengyu Dai, Kunyi Liu, Hongruixuan Chen, Zhuo Zheng, Junshi Xia, Stefano Ermon, and Naoto Yokoya. Can LLM agents respond to disasters? benchmarking heterogeneous geospatial reasoning in emergency operations. arXiv preprint arXiv:2605.11633, 2026. [31]Timo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves to use tools. In Advances in Neural Information Processing Systems, volume 36, pages 68539–68551, 2023. doi: 10.52202/07528 0-2997. [32]Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, et al. ToolLLM: Facilitating large language models to master 16000+ real-world APIs. In International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=dHng2O0Jjr. [33]Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, et al. AgentBench: Evaluating LLMs as agents. In International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=zAdUB0aCTQ. [34]Grégoire Mialon, Clémentine Fourrier, Craig Swift, Thomas Wolf, Yann LeCun, and Thomas Scialom. GAIA: A benchmark for general AI assistants. In International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=fibxvahvs3. [35]Tianbao Xie, Danyang Zhang, Jixuan Chen, Xiaochuan Li, Siheng Zhao, Ruisheng Cao, Toh Jing Hua, Zhoujun Cheng, Dongchan Shin, Fangyu Lei, et al. OSWorld: Benchmarking multimodal agents for open-ended tasks in real computer environments. In Advances in Neural Information Processing Systems, volume 37, pages 52040–52094, 2024. doi: 10.52202/079017-1650. [36]Yubo Ma, Zhibin Gou, Junheng Hao, Ruochen Xu, Shuohang Wang, Liangming Pan, Yujiu Yang, Yixin Cao, and Aixin Sun. SciAgent: Tool-augmented language models for scientific reasoning. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 15701–15736, 2024. doi: 10.18653/v1/2024.emnlp-main.880. [37]Yujiong Shen, Yajie Yang, Zhiheng Xi, Binze Hu, Huayu Sha, Qiyuan Peng, Jiazheng Zhang, Junlin Shang, Jixuan Huang, Yutao Fan, Jingqi Tong, Ming Zhang, Shihan Dou, Zhenfei Yin, Xingjun Ma, Lei Bai, Tao Gui, Qi Zhang, Xuanjing Huang, and Yu-Gang Jiang. SciAgentGym: Benchmarking multi-step scientific tool-use in LLM agents. In International Conference on Machine Learning, 2026. URLhttps://icml.c/virtual/2026/poster/ 66785. [38]Ken Gu, Ruoxi Shang, Ruien Jiang, Keying Kuang, Richard-John Lin, Donghe Lyu, Yue Mao, Youran Pan, Teng Wu, Jiaqian Yu, Yikun Zhang, Tianmai M. Zhang, Lanyi Zhu, Mike A. Merrill, Jeffrey Heer, and Tim Althoff. BLADE: Benchmarking language model agents for data-driven science. In Findings of the Association for 14 EarthVerse: Benchmarking Scientific Agents Across Dynamic Earth Systems and Natural Hazards Computational Linguistics: EMNLP 2024, pages 13936–13971, 2024. doi: 10.18653/v1/2024.findings-emnlp.815. URL https://aclanthology.org/2024.findings-emnlp.815/. [39]Bodhisattwa Prasad Majumder, Harshit Surana, Dhruv Agarwal, Bhavana Dalvi Mishra, Abhijeetsingh Meena, Aryan Prakhar, Tirth Vora, Tushar Khot, Ashish Sabharwal, and Peter Clark. DiscoveryBench: Towards data- driven discovery with large language models. In International Conference on Learning Representations, 2025. URLhttps://proceedings.iclr.c/paper_files/paper/2025/hash/0d70af566e69f1dfb687791e cf955e28-Abstract-Conference.html. [40] Akashah Shabbir, Muhammad Akhtar Munir, Akshay Dudhane, Muhammad Umer Sheikh, Muhammad Haris Khan, Paolo Fraccaro, Juan Bernabe Moreno, Fahad Shahbaz Khan, and Salman Khan. ThinkGeo: Evaluating tool-augmented agents for remote sensing tasks. arXiv preprint arXiv:2505.23752, 2025. [41]Peilin Feng, Zhutao Lv, Junyan Ye, Xiaolei Wang, Xinjie Huo, Jinhua Yu, Wanghan Xu, Wenlong Zhang, Lei Bai, Conghui He, and Weijia Li. Earth-Agent: Unlocking the full landscape of Earth Observation with agents. In International Conference on Learning Representations, 2026. URLhttps://proceedings.iclr.c/paper_ files/paper/2026/hash/5b4a459db23e6db9be2a128380953d96-Abstract-Conference.html. [42]Akashah Shabbir, Muhammad Umer Sheikh, Muhammad Akhtar Munir, Hiyam Debary, Mustansar Fiaz, Muham- mad Zaigham Zaheer, Paolo Fraccaro, Fahad Shahbaz Khan, Muhammad Haris Khan, Xiao Xiang Zhu, et al. OpenEarthAgent: A unified framework for tool-augmented geospatial agents. arXiv preprint arXiv:2602.17665, 2026. [43] Riyang Bao, Cheng Yang, Dazhou Yu, Zhexiang Tang, Gengchen Mai, and Liang Zhao. Spatial-Agent: Agentic geo-spatial reasoning with scientific core concepts. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 14896–14911, 2026. doi: 10.18653/v1/2026.acl-l ong.679. URL https://aclanthology.org/2026.acl-long.679/. [44]Yuxiang Zheng, Dayuan Fu, Xiangkun Hu, Xiaojie Cai, Lyumanshan Ye, Pengrui Lu, and Pengfei Liu. Deep- Researcher: Scaling deep research via reinforcement learning in real-world environments. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 414–431, 2025. doi: 10.18653/v1/2025.emnlp-main.22. URL https://aclanthology.org/2025.emnlp-main.22/. [45] Zhuofeng Li, Dongfu Jiang, Xueguang Ma, Haoxiang Zhang, Ping Nie, Yuyu Zhang, Kai Zou, Jianwen Xie, Yu Zhang, and Wenhu Chen. OpenResearcher: A fully open pipeline for long-horizon deep research trajectory synthesis. arXiv preprint arXiv:2603.20278, 2026. [46]MiroMind Team, Song Bai, Lidong Bing, Carson Chen, Guanzheng Chen, Yuntao Chen, Zhe Chen, Ziyi Chen, Jifeng Dai, Xuan Dong, Wenhan Dou, et al. MiroThinker: Pushing the performance boundaries of open-source research agents via model, context, and interactive scaling. arXiv preprint arXiv:2511.11793, 2025. [47]Tao Yu, Yiming Ding, Shenghua Chai, Minghui Zhang, Zhongtian Luo, Xinming Wang, Xinlong Chen, Zhaolu Kang, Junhao Gong, Yuxuan Zhou, Haopeng Jin, Zhiqing Cui, Jiabing Yang, YiFan Zhang, Hongzhu Yi, Zheqi He, Xi Yang, Yan Huang, and Liang Wang. Omni-DeepSearch: A benchmark for audio-driven omni-modal deep search. arXiv preprint arXiv:2605.08762, 2026. URL https://arxiv.org/abs/2605.08762. [48] Lei Bai, Zhongrui Cai, Yuhang Cao, Maosong Cao, Weihan Cao, Chiyu Chen, Haojiong Chen, Kai Chen, Pengcheng Chen, Ying Chen, et al. Intern-S1: A scientific multimodal foundation model. arXiv preprint arXiv:2508.15763, 2025. [49]Yifan Zhang, Cheng Wei, Zhengting He, and Wenhao Yu. GeoGPT: An assistant for understanding and processing geospatial tasks. International Journal of Applied Earth Observation and Geoinformation, 131:103976, 2024. doi: 10.1016/j.jag.2024.103976. [50]Qi Ge, Pengfa Li, Yinhao Dai, Jin Li, Ni An, Yang Yu, Qing Lv, and Hongyue Sun. GeohazardGPT: Towards large language models for geohazards. Under review, 2025. [51] OpenAI. GPT-4o system card. arXiv preprint arXiv:2410.21276, 2024. [52] OpenAI. GPT-5.5 system card. https://openai.com/index/gpt-5-5-system-card/, 2026. [53] OpenAI. GPT-5.6 system card. https://deploymentsafety.openai.com/gpt-5-6, 2026. [54] Anthropic. Model system cards. https://w.anthropic.com/system-cards, 2026. [55] Google DeepMind. Gemini 3.1 Flash-Lite. Google DeepMind model card,https://deepmind.google/mode ls/model-cards/gemini-3-1-flash-lite/, 2026. [56]MiniMax. MiniMax M2.7: Early echoes of self-evolution.https://w.minimax.io/news/minimax-m27 -en, 2026. 15 EarthVerse: Benchmarking Scientific Agents Across Dynamic Earth Systems and Natural Hazards [57]ByteDance Seed Team. Seed 2.0 official launch.https://seed.bytedance.com/en/blog/seed2-0-%25E6 %25AD%25A3%25E5%25BC%258F%25E5%258F%2591%25E5%25B8%2583, 2026. [58] Qwen. Qwen3.7-Plus. Official model directory, https://chat.qwen.ai/legal-agreement/models, 2026. [59]Tencent. Tencent unveils Hy3 preview; model enhances agent capabilities and real-world usability.https: //w.tencent.com/en-us/articles/2202320.html, 2026. [60]DeepSeek-AI. DeepSeek-V4: Towards highly efficient million-token context intelligence. arXiv preprint arXiv:2606.19348, 2026. [61] Kimi Team. Kimi K2.5: Visual agentic intelligence. arXiv preprint arXiv:2602.02276, 2026. [62] Z.ai. GLM-5.2: Built for long-horizon tasks. https://z.ai/blog/glm-5.2, 2026. [63]An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 2025. [64]NVIDIA. NVIDIA Nemotron Nano 2: An accurate and efficient hybrid mamba-transformer reasoning model. arXiv preprint arXiv:2508.14444, 2025. 16 EarthVerse: Benchmarking Scientific Agents Across Dynamic Earth Systems and Natural Hazards Appendix Guide A Appendix A Controlled Studies18 How reasoning, evidence access, memory, tools, and execution conditions change research quality. Answer mode • reasoning effort • routing • context • robustness • scientific environments B Appendix B Discussion, Broader Impact, and Limitations21 What the results imply for scientific agents, disaster analysis, oversight, and responsible use. Research position • broader impact • system design • limitations C Appendix C Experimental Setup and Reproducibility22 How models, modalities, inference, scoring, study manifests, and release decisions are controlled. System roster • modality interface • main configuration • sampling • audit D Appendix D Extended Diagnostics27 Where capability, trajectory, error-path, resource, intervention, and tool-compliance analyses are reported. Capability scores • error paths • costs • ablations • uncertainty E Appendix E Prompts and Execution Contracts36 What the solver, verifier, judge, and benchmark-construction reviewers are instructed to do. Runtime visibility • solver contract • final review • judging • construction review F Appendix F Representative Case Set40 Which systems and tasks were selected to represent the benchmark’s main scientific workflows and failure modes. Five successful cases • one partial failure • system family • task scope • score G Appendix G Detailed Research Trajectories40 How selected systems discover evidence, calculate results, recover from errors, and finalize an answer. Full task • ordered trace • computed ledger • evidence reconciliation • failure analysis Reading routes Experiments A + DReproduction C + EInterpretation B + F + G 17 EarthVerse: Benchmarking Scientific Agents Across Dynamic Earth Systems and Natural Hazards A Controlled Studies RQ2 asks when reasoning improves an investigation and which controls preserve or break scientific continuity. We therefore vary answer mode, reasoning effort, evidence localization, tool routing, context, stopping, and professional scientific execution. Unless noted, every condition uses GPT-5.5 under the protocol in Section 4.2, with identical task manifests within each experiment. A.1 Reasoning effort and answer mode We cross four answer modes with five reasoning settings on a fixed, difficulty-stratified manifest. Compressed direct uses a bounded evidence summary, while all-text direct serializes every readable file. Both interactive conditions use the same EarthVerse harness; the review variant adds one evidence-and-calculation audit before finalization. The compact view below retains the outcome columns so that each reasoning curve can be compared without the denser interaction and resource diagnostics in Table 3. Table 5: Outcome view of the reasoning sweep. GPT-5.5; bold blue and shading mark the best setting in each access mode. Compressed direct (selected evidence) EffortAnswerProcessCore Unit acc. none41.3745.1043.2442.21 low24.9929.32 27.16 25.43 medium22.8825.69 24.29 23.26 high24.1728.17 26.17 24.28 xhigh16.6716.67 16.67 16.67 All-text direct (budgeted evidence) EffortAnswerProcessCore Unit acc. none27.6631.2229.4428.27 low13.2616.20 14.73 13.36 medium9.7912.18 10.99 10.05 high8.809.008.908.82 xhigh12.0015.00 13.50 12.00 EarthVerse interactive EffortAnswerProcessCore Unit acc. none48.8953.70 51.30 49.46 low56.9760.05 58.51 57.53 medium61.4564.80 63.13 62.35 high72.4175.1973.8072.59 xhigh69.6667.50 68.58 69.83 EarthVerse interactive + review EffortAnswerProcessCore Unit acc. none53.7056.48 55.09 54.06 low61.9365.50 63.72 62.78 medium67.1669.22 68.19 67.91 high69.8274.71 72.27 70.73 xhigh84.3688.2186.2984.40 A.2 Tools, context, and interaction budget On one fixed paired manifest, we vary catalog size, semantic routing, tool composition, context retention, and interaction budget. Every condition uses the default medium reasoning setting and the same answer and process scores. Appendix D.4 reports trajectory diagnostics. A.2.1 Tool-catalog scale Nested fixed catalogs expose increasingly broad subsets of the same 170-tool registry (panel (a) of Table 6). The six-tool condition contains package discovery, reading, search, tabular inspection, Python execution, and finalization; each larger catalog adds domain tools without changing the task prompt. The full catalog loses 9.80 Core points relative to the six-tool foundation while using 19% more tokens. The decline is not monotonic, which rules out a simple capacity penalty. Extra tools add nearby analytical paths with different assumptions about variables, units, aggregation, and applicability. When the controller cannot retain why it chose one path, later outputs from another tool can enter the same ledger as if they were comparable. Catalog breadth then creates scientific ambiguity. Reporting a tool count without the navigation policy misses this failure mode. A.2.2 Semantic routing and scientific composition The routing study selects tools from task text and package metadata before interaction (panel (b) of Table 6). Top-k routing exposes the highest-scoring individual tools, whereas workflow routing first selects a domain workflow. A separate composition study compares an expert-only suite, a mixed general/expert suite, and atmospheric Python and Fortran suites. Top-20 routing recovers 11.21 Core points over the full catalog with 34% fewer tokens. Its advantage comes from narrowing the next choice without deciding the investigation in advance. Workflow routing is more brittle because it 18 EarthVerse: Benchmarking Scientific Agents Across Dynamic Earth Systems and Natural Hazards can commit to a domain before the evidence warrants that commitment; the expert-only suite has the opposite problem, lacking ordinary file and calculation operations that connect specialist output to a scored claim. Routing should act as a revisable prior: preserve a small general core, introduce specialist methods when their required variables appear, and allow the evidence to overturn the initial route. A.2.3 Context retention We vary the amount of tool history preserved verbatim before older content is summarized (panel (c) of Table 6). Aggressive compression keeps a narrow recent window; short and long policies progressively retain more observations; full history disables summarization within the tested budget. Long bounded retention gains 13.66 Core points over aggressive compression and 5.27 over full replay. Too little history forces the agent to rediscover files and often severs intermediate values from their provenance. Full replay retains rejected hypotheses and superseded numbers alongside active values, so discarded information can reappear in later calculations. Scientific memory therefore needs a compact claim–evidence state that binds claims to sources, units, temporal and spatial support, and unresolved conflicts while archiving replaced values. Finding. Scientific memory is selective. Long bounded retention beats both aggressive compression and full replay. The useful state is a compact ledger of active claims, sources, units, and unresolved conflicts, not a verbatim transcript. A.2.4 Interaction budget The final study changes only the EarthVerse-harness turn limit. The unbounded condition still uses the common wall-clock budget, so it tests productive stopping rather than unlimited execution. Twelve turns usually end before discovery, calculation, and verification can all occur. Additional turns improve mean quality, but the gain depends on what those turns accomplish. Extra interaction helps when it closes a named evidence gap and hurts when the agent reopens a settled calculation or follows an unproductive branch. Stopping is therefore a coverage decision: the system should stop when required claims are supported, units and provenance are checked, and material contradictions have been resolved or disclosed. Table 6: System ablations. GPT-5.5 at medium reasoning; bold blue marks panel bests. OutcomeCoverageCore by capabilityResource ConditionAnswer Process Core Unit acc. Physical mech. Spatio- temporal Quant. calc. Multi- source Causal chain Ranking RS / geo. Tokens (a) Tool catalog: nested subsets of 170 tools Fixed tools (6)64.3465.62 64.9864.6963.2766.2270.34 97.5063.3551.78 58.12229,557 Fixed tools (10)60.2257.10 58.6660.3556.7458.3661.04 95.2255.7758.32 61.25194,047 Fixed tools (20)60.5859.38 59.9860.9358.1370.5061.67 95.0054.9056.50 55.62232,912 Fixed tools (40)58.5561.00 59.7759.4358.1647.8162.03 90.4563.6359.57 54.12232,993 Fixed tools (70)58.7661.25 60.0159.1958.5455.6863.89 87.9560.1454.21 54.62233,759 Fixed tools (100)63.9663.12 63.5464.1161.8158.9365.84 96.3765.7960.82 51.00278,936 Full catalog (170)54.4855.88 55.1855.4655.5855.7254.96 47.5052.7563.15 55.00273,216 (b) Routing and tool composition Top-10 semantic router64.1266.25 65.1964.5564.3760.6467.93 80.7467.5057.20 59.17216,227 Top-20 semantic router66.9065.88 66.3967.1465.1564.4073.43 90.0065.8950.84 53.94180,010 Workflow-profile router56.0556.75 56.4056.4754.7452.6358.60 87.9555.3955.00 60.62220,232 Expert-only suite59.2860.88 60.0859.5058.4861.0861.71 90.4557.7961.08 57.62295,490 General + expert suite59.0763.00 61.0459.9559.2566.0163.21 95.0061.3449.27 51.38195,092 Atmospheric Python suite59.4860.12 59.8059.6758.2659.6460.69 89.2060.9458.31 49.25198,802 Atmospheric Fortran suite57.4657.88 57.6757.5155.8151.0561.19 92.9557.3553.88 60.75205,262 (c) Context retained before summarization Aggressive compression48.7551.62 50.1949.0048.0757.5553.73 90.4544.6545.44 47.00197,038 Short retained context56.3156.38 56.3456.5254.1657.3055.54 97.7257.8560.08 42.25209,827 Long retained context64.3263.38 63.8564.7562.2159.4368.29 95.0064.7258.79 48.25227,536 Full retained history58.9058.25 58.5859.4256.9855.3962.15 89.0057.2552.59 61.97187,700 (d) EarthVerse-harness turn budget under one wall clock 12-turn budget44.8449.50 47.1744.9544.6545.3646.27 95.0047.7357.50 35.13106,263 24-turn budget54.1052.88 53.4954.3251.3157.8855.46 95.0050.9349.13 50.92161,404 96-turn budget59.7260.12 59.9260.1657.9462.7660.53 97.5062.1054.08 44.38182,798 Wall-clock limited62.6564.50 63.5762.9162.1662.7266.66 90.4567.1156.91 33.38221,180 19 EarthVerse: Benchmarking Scientific Agents Across Dynamic Earth Systems and Natural Hazards A.3 Oracle access, discoverability, and evidence robustness These conditions reuse one paired task set and the Top-20 router baseline. Table 4 reports the outcomes; Appendix D.5 retains trajectory and resource diagnostics. A.3.1 Oracle bottleneck localization Each oracle removes one information bottleneck while leaving the model, budget, tools, and scoring unchanged. It is a diagnostic upper bound rather than a proposed deployment setting, and none of the conditions reveals the target answer. The relevant-file oracle identifies only the package-relative files used by the reference analysis. The tool-subset oracle names useful tool classes; the planning oracle supplies a short answer-free plan; and the evidence-map oracle adds source roles and intermediate quantities without stating the conclusion. A.3.2 Tool naming and discoverability We hold tool implementations fixed and alter only how the catalog is presented (panel (b) of Table 4). Descriptive names add intended-use and exclusion cues; hierarchical discovery chooses a domain before exposing tools; dynamic descriptions retrieve candidate tools at each step; opaque names remove semantic hints. A.3.3 Evidence perturbation and robustness Perturbations preserve the question and answer schema while modifying the package evidence (panel (c) of Table 4): irrelevant files are added, a secondary source is made inconsistent, one supporting file is removed, long text is truncated, or units are changed while preserving physical equivalence. A.3.4 Compact meteorological access This probe tests access rather than compulsory execution. One condition exposes the same compact shortlist of meteorological operations to every task; the other uses task and package cues to expose at most three. The agent may ignore them in either condition. This separates tool availability from the professional-environment study below, where repeated specialist use is required. A.4 Professional meteorological environments This study asks a stricter question: what happens when a task must be completed through a named scientific environment? Each condition requires repeated use of one of 13 libraries or runtimes under the same protocol. Adapters expose only implemented operations and record whether calls succeed, allowing the results to separate useful scientific fit from technically successful but unsuitable method use. Table 7: Required scientific environments. GPT-5.5; ∆Core is relative to the Top-20 router. OutcomeChangeCoverage EnvironmentAnswerProcessCore ∆ Core Unit acc. Top-20 router (ref.)66.9065.8866.390.0067.14 (a) Python scientific libraries MetPy68.6970.3869.53+3.1468.80 xclim67.7769.6268.70+2.3167.95 thermofeel66.7868.7567.76+1.3767.47 PyET65.3467.3866.36−0.0365.89 climate-indices64.5066.7565.62−0.7765.58 xskillscore63.3764.1263.75−2.6463.90 pyextremes59.0260.0059.51−6.8859.27 (b) Compiled and domain-language environments Julia71.7770.2571.01+4.6271.99 R climate68.5571.3869.96+3.5769.44 CDO66.9169.5068.20+1.8167.35 NCO67.3965.0066.19−0.2067.49 NCL65.0365.0065.01−1.3865.27 Fortran64.4664.2564.35−2.0464.52 20 EarthVerse: Benchmarking Scientific Agents Across Dynamic Earth Systems and Natural Hazards Julia, R climate, MetPy, xclim, and CDO improve Core because their implemented operations match common package structures such as time series, gridded fields, and atmospheric profiles. Julia gives the largest gain (+4.62), but the ranking is less informative than the conditions under which the tools help. They compress many low-level numerical steps into an inspectable transformation without hiding the variables or aggregation. That advantage disappears when the package lacks the temporal support or paired variables assumed by the method. Required use exposes a failure that optional-tool studies can miss. Pyextremes loses 6.88 Core points because many event packages do not contain a long, homogeneous series suitable for return-level estimation; forecast-verification methods face the same mismatch when paired forecasts and observations are absent. Under a hard requirement, the agent sometimes bends the available data toward the method instead of rejecting the method. A scientific interface needs an explicit refusal path, backed by minimum record length, sampling assumptions, missing-data rules, and a statement of what the output can support. Finding. Execution success is not scientific validity. A specialist method helps only when its required variables and sampling support exist. Otherwise, a successful call can produce a precise result that the package cannot justify. B Discussion, Broader Impact, and Limitations B.1 Research position EarthVerse evaluates a complete research loop over a bounded event archive. The agent must decide what counts as evidence, transform it, and revise an event-scale explanation while retaining enough provenance for another analyst to replay the work. This is more constrained than open-ended research, but it begins earlier than supplied-observation QA and asks more of the controller than a tool-execution benchmark. Strong systems often interpret a selected observation correctly. Reliability falls when they must choose the record, its scale, the compatible variables, and the point at which the analysis is complete. Many failures originate in evidential binding: a plausible value is attached to the wrong aggregation, time window, unit, or source role, and later reasoning treats the mismatch as established fact. This explains why stronger reasoning, more tools, and longer traces can all fail to help. They operate on the state the controller has preserved. If that state no longer records what a value means, additional reasoning only makes the wrong account more elaborate. Complex Earth-system investigations rarely have a unique valid research path. EarthVerse therefore prescribes neither a tool order nor a reference trajectory. Different paths are valid when they recover the required windows, quantities, source roles, and mechanisms, and preserve support for the resulting claims [38, 39]. B.2 Broader impact Scientific practice. Assistants can already reduce the mechanical burden of first-pass hazard analysis: inventorying an archive, locating candidate records, harmonizing units, repeating calculations, and surfacing source disagreement. The experiments also show where this assistance becomes risky. A fluent system may complete each local operation while carrying a scale mismatch through the full analysis. Experts still need to judge comparability, uncertainty, and consequences; package-local provenance gives them something concrete to inspect. Reliability in high-stakes settings. High average unit accuracy can hide one missing source, time window, calculation, or mechanism. In disaster analysis, that single omission may change estimated severity, exposed population, or the causal account used to justify action. The benchmark does not certify operational readiness. It makes these omissions visible before a result is treated as operational evidence. Access and evaluation practice. A local package reduces dependence on changing APIs and lets researchers inspect the same evidence. Evaluating frontier systems across hundreds of long trajectories is still expensive and may exclude smaller groups. Cost should be reported alongside quality, and releases should include compact diagnostic subsets, task-level scores, and traces. Subset results should not be presented as full-benchmark estimates. B.3 System design implications For Earth-system analysis, a trajectory is useful only when it preserves the support for each claim. A record of calls without source bindings cannot show whether the final explanation follows from the evidence. 21 EarthVerse: Benchmarking Scientific Agents Across Dynamic Earth Systems and Natural Hazards Claim–evidence state. A transcript records chronology but does not identify which claims and values remain active. The controller needs a claim–source graph in which every derived value retains its source, unit, time window, transformation, and relation to competing explanations. Corrections should replace active values without erasing the history that explains the change. Scientific operations. Scientific tools need to expose applicability as part of their output. Record length, sampling support, missing-data rules, and estimator assumptions determine whether a computed value can support the claim. Training that rewards successful calls alone encourages the model to force data into an available method; justified refusal should receive credit when the inputs are inadequate. Verification and oversight. Final review should operate on claims rather than prose. For each required conclusion, the system should expose coverage, dimensional checks, source conflicts, and the mechanism tests it rejected. Experts can then inspect the points where judgment entered the analysis and decide whether the result is fit for use. A polished report without this record is not auditable. B.4 Limitations Coverage and ecological validity. EarthVerse is a controlled approximation of research. Its 199 packages cover many natural hazards, but not the full range of Earth-system science, long-term attribution, or live forecasting. Package isolation improves reproducibility and blocks web leakage, at the cost of open-ended discovery and institutional data-quality judgments. Masked products also prevent tests of absolute geolocation. Ground truth and judge validity. Structured ground truth necessarily chooses one decomposition of a scientific answer. Process scores also rely on an LLM judge that compares each trace with a reference solution. Separate answer and process blocks reduce score leakage but cannot remove judge error. Future releases should measure agreement with domain experts on a stratified subset and retain criterion-level decisions. Data provenance and maintenance. The assembled event packages, task formulations, answer units, and derived targets were created for EarthVerse and were not available online before release, which reduces the chance of exact benchmark memorization. Some underlying historical events and source documents may still have appeared in pretraining. Package isolation, hidden scoring artifacts, and package-specific calculations make shortcut answering harder. Statistical scope of controlled studies. The controlled studies use paired subsets, and several intervals in Appendix D.5 include zero. Large contrasts, such as the relevant-file oracle, support architectural hypotheses. Smaller changes need replication across more packages and backbones. Cross-benchmark revision measures recoverability under feedback, not zero-shot superiority. C Experimental Setup and Reproducibility This section collects implementation details that are necessary for reproduction but not for interpreting the main findings. It records the evaluated systems, the common evidence interface, the leaderboard configuration, controlled-study manifests, scoring isolation, and retained audit artifacts. C.1 Systems, controllers, and model modality The leaderboard contains 25 complete systems. Hosted endpoints include the OpenAI GPT-4o, GPT-5.5, and GPT-5.6 tiers [51–53]; Anthropic Claude tiers [54]; Gemini [55]; MiniMax M2.7 [56]; Seed 2.0 [57]; Qwen3.7 [58]; Tencent Hy3 [59]; DeepSeek-V4 [60]; Kimi K2.5 [61]; and GLM-5.2 [62]. The open-weight group includes Qwen3 at 4B and 235B scales [63], Nemotron Nano [64], MiroThinker 8B [46], GeohazardGPT, GeoGPT, and Intern-S1-mini [48–50]. Most systems run in the EarthVerse harness, which manages package discovery, tool use, context, recovery, and finalization. GeoMMAgent uses its adapted five-stage coordinator with GPT-5.5 as the worker model in every stage; OpenResearcher retains its research-loop controller [17,45]. These rows therefore describe a model–controller system, not a bare backbone. The roster mixes text-only models with natively multimodal endpoints. Direct image interpretation is required by only a small fraction of EarthVerse tasks; most remote-sensing evidence is represented by product metadata, raster statistics, change indices, or other auditable tool outputs. Raw pixels are not selectively sent to models with vision support. This 22 EarthVerse: Benchmarking Scientific Agents Across Dynamic Earth Systems and Natural Hazards keeps the evidence interface comparable, although the resulting scores should not be read as a standalone test of visual perception. C.2 Shared experimental configuration Every run exposes a scientific question and one evidence root while keeping the solution, rubric, ground truth, scoring code, prior outputs, and web evidence hidden. Reads, writes, and Python execution stay inside the active evidence collection. Reports and tables are returned as local text or structured summaries; image and raster operations return recorded metadata and measurements. The EarthVerse harness is the common controller for the standard model rows. To study the attainable system-level ceiling, we also evaluate the GPT-5.6 family in Codex and Claude Fable 5 in Claude Code, the strongest model- native harnesses available for those systems. GeoMMAgent pairs GPT-5.5 with its five-stage coordinator, while OpenResearcher keeps its own controller. The Harness column in Table 2 makes these differences explicit. The common execution limit is 48 turns, 1,680 seconds for research, and 1,800 seconds overall. The controller retains 16 recent messages and a 24,000-character research state; individual tool observations are capped at 12,000 characters. Package-scoped Python has a 60-second call limit. Up to two schema repairs are allowed, with 120 seconds reserved for finalization and a separate 120-second final-call timeout. We retain provider-native reasoning and generation controls when available, use endpoint defaults otherwise, and do not impose one temperature across providers. The judge is a fixedgpt-4.1endpoint, and randomized procedures use seed 2026. Endpoint access dates and local model revisions are retained in the release registry. C.3 Package evidence sources and access interfaces The 199 package manifests contain 6,397 provenance records, 835 rawsource_namelabels, and 6,709 local files from 107 external host domains. Many raw labels are event-specific filenames, response formats, endpoint variants, or repeated queries to the same service rather than distinct sources. We therefore group aliases at the product/API-family level while keeping distinct providers, sensors, product versions, and access semantics separate. This yields 80 external source and access families. We also list nine package-local derivative families used directly in computation; they do not count as additional upstream sources. PACKAGE SOURCES Distinct source and access families across 199 event packages Event discovery, reports, and authoritative anchors29 families Provider or productAccess formRole in a package GDACSevent-list REST/GeoJSON API; RSS archive Cross-hazard event discovery, dates, alert levels, and identifiers. NASA EONET v3REST search by date, category, or keywordWildfire, flood, storm, and general natural-event catalog records. ReliefWebpublic update and report searchHumanitarian situation reports, response updates, and impact narratives. HDXCKAN package_search API Candidate humanitarian datasets and country- or event-specific impact layers. GDELTevent-window article searchSecondary news discovery used only when the package retains the result. WikipediaREST page-summary and event-search interfaces Event-name cross-checks and compact encyclopedic context. Wikidataentity-search APIStructured event and place identifiers used for identity checks. NASA Earth Observatoryevent articles and image pagesLocked event anchors and satellite-based physical narratives. NASA Scientific Visualization Studiovisualization pages and media APIOfficial imagery, animations, and explanatory visual products. NASA Disasters and Applied Sciencesactivation and event pagesSatellite-based disaster-response context and mapped observations. World Meteorological Organizationreports, topic pages, and bulletinsAuthoritative global weather, climate, and extreme-event summaries. World Meteorological Centre Beijingevent reports and operational bulletinsRegional meteorological summaries and event chronology. NOAA Climate.govevent trackers and climate reportsReviewed US climate-event narratives and physical context. US National Weather Service and WPCforecast discussions, storm summaries, and precipitation pages Operational weather chronology, rainfall totals, and warning context. 23 EarthVerse: Benchmarking Scientific Agents Across Dynamic Earth Systems and Natural Hazards PACKAGE SOURCES Distinct source and access families across 199 event packagescontinued NOAA NESDIS and SSDsatellite product and event pagesOperational satellite imagery and environmental monitoring context. Copernicus EMSactivation and rapid-mapping archivesEvent confirmation and links to emergency-mapping products. UN OCHAsituation reports and humanitarian updatesReported impacts, needs, access constraints, and response activity. UNOSATsatellite-assessment reports and mapsMapped damage, flood, and exposure evidence. UN country teams and UN Genevacountry and event updatesLocal consequence and response narratives retained as event evidence. IFRCemergency appeals and operational updatesHumanitarian consequences, needs, and Red Cross response records. AHA Centreregional disaster updates and reportsASEAN event impacts and coordinated response context. World Health Organizationhealth-emergency reportsMortality, morbidity, service disruption, and public-health consequences. UNICEFhumanitarian situation reportsImpacts on children, water, health, education, and relief delivery. FAOfood, agriculture, and livelihood reportsCrop, livestock, food-security, and rural-impact evidence. World Bankassessment and recovery documentsInfrastructure, economic loss, and reconstruction context. World Weather Attributionevent-study pages and reports Attribution context and documented physical or societal drivers. South Asian operational agenciesIMD/RSMC New Delhi, DHM Nepal, and DMC Sri Lanka Official cyclone, rainfall, flood, and warning records. National meteorological servicesBoM, HKO, UK Met Office, MetService, and SAWS National weather reports, warnings, and climate summaries. Scientific institutesESSL, ICIMOD, NGI/NGU, ICL, OSTIHail, glacier, landslide, heat, and other specialist event accounts. Meteorology, climate, hydrology, ocean, and air quality13 families Provider or productAccess formRole in a package ERA5-Landhourly reanalysis; package-local aggregates/stacks Temperature, precipitation, wind, and land-surface background. CHIRPS v2daily precipitation archiveEvent accumulation, wet spells, climatology, and rainfall anomalies. NASA GPM IMERG V07Earthdata CMR discovery and precipitation granules Satellite precipitation totals, timing, and spatial structure. Open-Meteo Archivehistorical hourly/daily REST APIPoint temperature, humidity, apparent temperature, rain, snow, and wind. Open-Meteo Air Qualityhistorical air-quality REST APIPM 2.5 , PM 10 , aerosol, and related atmospheric fields. NASA POWERdaily point REST APITemperature, corrected precipitation, and 10-m wind cross-checks. NOAA PSL climate indicesdownload tables for ONI, SOI, and related modes Large-scale climate-state and teleconnection context. NOAA CPC ONIASCII and tabular index archiveOperational Oceanic Niño Index values and episode timing. IRI and Australian BoMENSO monitoring archivesIndependent ENSO phase and coupled-ocean background. GPCCprecipitation-anomaly productsGauge-based anomaly context alongside CHIRPS. NOAA OISST v2 high resolutionPSL THREDDS/ERDDAP catalogSea-surface temperature and marine-heatwave context. Indian Ocean Dipole indexDMI archiveIndian Ocean climate-mode state for regional events. NOAA ocean servicestide gauges and Ocean Prediction Center analyses Coastal water levels, surge context, and marine storm structure. Remote sensing and geospatial context12 families Provider or productAccess formRole in a package NASA Worldview/GIBSsnapshot API and MODIS/VIIRS browse imagery Pre-event and in-event true-color or thematic visual context. MODIS/VIIRS vegetation productsWorldview or archive previewsVegetation condition and drought-stress context. MODIS/VIIRS MAIAC AODaerosol and smoke previewsSmoke transport and aerosol loading context. 24 EarthVerse: Benchmarking Scientific Agents Across Dynamic Earth Systems and Natural Hazards PACKAGE SOURCES Distinct source and access families across 199 event packagescontinued Sentinel-1 GRDVV pre/post imagery and change productsFlood, deformation, and surface-change comparison. Sentinel-2 SR Harmonizedsurface reflectance and dNBR derivativesBurn severity and optical before/after consistency. Google Satellite Embedding / AlphaEarthannual 64-band embeddingsAnnual state vectors, cosine-change statistics, and change rasters. Copernicus EMS Rapid MappingEMSR/GFM maps and archive productsMapped flood, fire, earthquake, and exposure footprints. Microsoft Planetary ComputerSTAC metadata searchAOI/date discovery for registered Earth-observation assets. Natural EarthAdmin-0 and physical basemap filesLow-resolution geopolitical and physical reference layers. geoBoundariesadministrative-boundary APIOpen administrative geometries for package AOIs. GHSLbuilt-up and settlement layersUrban extent and settlement structure. Copernicus DEM and SRTMelevation and derived slopeTerrain constraints for flood, landslide, and glacier analyses. Exposure, infrastructure, and human impact9 families Provider or productAccess formRole in a package WorldPop GP 100 mpopulation grids and AOI summariesPopulation exposure, density, and resampled 1-km context. OpenStreetMap Nominatimgeocoding REST APIEvent location, fallback coordinates, and AOI checks. OpenStreetMap Overpassbounded feature-query APIRoads, buildings, services, shelters, and critical amenities. OpenStreetMap Geofabrikregional extracts Broader road and infrastructure context where bounded queries are insufficient. FEWS NET and IPCfood-security products and reportsDrought exposure, food-security phases, and response pressure. NYC Open DataSocrata 311 service-request APIFlood complaints and city-service stress during urban rainfall. Harris County and Houston open dataflood-warning and 311 interfacesGauge conditions, complaints, and local flood impacts. Hong Kong public feedswarnings, incident, and transport recordsLocal disruption and response evidence for typhoon events. Brazil CEMADEN and civil defencealerts and municipal reportsRainfall-triggered hazard warnings and local consequences. Hazard-specific scientific and operational sources17 families Provider or productAccess formRole in a package USGS ComCatearthquake-event REST APIOrigin time, magnitude, depth, location, and event products. USGS ShakeMap and PAGERevent-product feedsInstrumental intensity, shaking footprint, and loss context. USGS Did You Feel It?community-intensity productReported shaking distribution and populated-area impacts. NOAA NCEI/WDSGlobal Historical Tsunami DatabaseTsunami source, run-up, and historical event records. NOAA NHCTropical Cyclone Reports archiveTrack, intensity, rainfall, surge, and impact summaries. NOAA HRD H*Windsurface-wind analysis archiveStorm wind structure and peak-wind calibration. NOAA Coral Reef Watch5-km thermal-stress productsHotSpot, Degree Heating Week, and bleaching-risk context. Copernicus GloFAS/GFM/EFASflood-monitoring and event productsRiver-flood extent, timing, and European flood context. Dartmouth Flood ObservatoryGlobal Flood Monitor/archiveIndependent flood occurrence and extent context. Smithsonian GVPvolcano reports and event pagesEruption chronology, volcano status, and physical interpretation. Volcanic Ash Advisory Centresoperational ash advisoriesAsh extent, altitude, movement, and aviation context. GLIMS and Randolph Glacier Inventoryglacier outlines and catalog interfacesGlacier geometry and inventory context. NSIDC and ICIMODglacier and cryosphere reportsRegional ice change and glacier-hazard evidence. NASA Global Landslide Catalog / COOLRlandslide catalog searchOccurrence records and landslide-event cross-checks. USGS landslide releasesevent pages, publications, and data releasesSite evidence for landslides and landslide-generated tsunamis. ESSL and IEM/NWS archives severe-weather summaries and Local Storm Reports Hail, wind, tornado, and convective-event observations. Australian fire and smoke productsofficial extent and smoke recordsFire perimeter, smoke, and air-quality context. Package-local derivatives retained for computation9 derived families Provider or productAccess formRole in a package Compact event AOIevent-bounding-box derivativeCommon spatial support for package layers. 25 EarthVerse: Benchmarking Scientific Agents Across Dynamic Earth Systems and Natural Hazards PACKAGE SOURCES Distinct source and access families across 199 event packagescontinued ERA5-Land aggregate and stackhourly statistics and raster bundleCompact numerical access without replacing the upstream provenance. GPM and CHIRPS event accumulationsderived totals and AOI rastersComparable precipitation ledgers across satellite and gauge products. WorldPop summariesAOI sum and 1-km resampleDeterministic population exposure inputs. Sentinel-1 changepre/post statistics and rasterAuditable radar change metrics. Sentinel-2 dNBRsummary and rasterAuditable burn-severity calculation. Satellite-embedding changeannual statistics and rasterNormalized annual surface-change evidence. Bounded OSM slicepackage-local JSONStable infrastructure and service counts. Locked event anchorpackage-local report copyFixed event identity, time window, and provenance for replay. The inventory is exhaustive at this level of grouping, although each event uses only a hazard- and availability-specific subset. The 80 external rows cover distinguishable upstream sources and access methods; the nine derived rows record transformations reused across packages. Per-package manifests retain the raw label, exact URL, acquisition status, package-relative output, byte count, and checksum for record-level provenance. C.4 Scoring, isolation, and audit protocol The judge receives the task, structured ground truth, final answer, reference process, rubric, and trajectory. It scores required answer units separately from evidence selection, calculation, source reconciliation, and mechanism analysis. Combined Core is the mean of these blocks, and Strict@95 records near-complete tasks. Capability columns use overlapping expert tags as diagnostic slices, not additive categories. Controlled comparisons use paired manifests and task-level bootstrap resampling. Solvers may list, read, search, and compute only within the active event package. The reference solution, rubric, structured ground truth, and ground-truth program remain hidden. Tools record each path and operation before shortening observations, which preserves provenance through context compression. Finalization receives the accumulated research state and required schema but no answer-bearing hint. Released traces cannot reconstruct provider-hidden reasoning, proprietary routing, or undocumented cache accounting. C.5 Controlled-study sampling design The reasoning-effort study uses a fixed manifest containing historically difficult tasks and a fixed-seed random draw from the rest of the release, preferring distinct events. Tool, routing, context, and interaction studies use a smaller paired manifest drawn from it. Oracle and robustness conditions reuse the same task IDs and Top-20 router outputs. Paired 95% intervals use 5,000 bootstrap resamples of task-level score differences with seed 2026. Professional-environment conditions use GPT-5.5 with the default medium reasoning setting and 48 turns. Core libraries retain the paired manifest; input-dependent environments draw a difficulty-balanced compatible manifest with seed 2026. Their compliance contract requires at least two successful, distinct specialist calls, targets three, and counts at most four. These paired ablations are diagnostic and should not be read as exact full-leaderboard effects. Artifacts retained for audit. • Task identity. Version, manifest, package checksum, and question reproduce the scientific input. • System identity. Endpoint revision, controller, tool profile, and context policy reproduce the executable system. • Research trace. Actions, arguments, observations, errors, and timing reconstruct the investigation, including failed calls. • Submission. Raw text, parsed object, repair attempts, and final status keep malformed or missing answers in the denominator. • Reference and judgment. Ground truth, solution, computation, rubric, judge prompt, and criterion decisions support answer-unit and aggregate recomputation. • Paired comparison. Task IDs, intervention, seed, and bootstrap resamples identify the controlled change. C.6 Human evaluation and agent-assisted review Human review begins when a task is conceived and continues through release. The team included six doctoral-level experts in Earth science and twenty undergraduates in related fields. Doctoral reviewers set the scientific direction, 26 EarthVerse: Benchmarking Scientific Agents Across Dynamic Earth Systems and Natural Hazards checked physical validity, and approved the final scoring specification. Undergraduate reviewers followed a common protocol for source and metadata checks, package replay, calculation verification, and secondary review of answer units. Reviewers contributed more than 50 hours per person on average. Task direction and scoring checkpoints. For every task, a doctoral-level reviewer first analyzed the event and specified the scientific question, hazard process, spatial and temporal support, admissible evidence, required calculations, and answer fields. LLM agents then explored the package for plausible source combinations, alternative calculations, ambiguities, and likely failure modes. Reviewers used these passes to refine the reference analysis and define the final answer units and process checks. Agents proposed candidates; human experts decided which checks were scientifically necessary, supported by the package, and suitable for scoring. Joint verification and correction. Every candidate task underwent agent replay followed by human inspection. Reviewers confirmed that package evidence could answer the public question, thatcompute_gt.pyregenerated the structured targets, and that the solution, answer units, and rubric referred to the same quantities and physical claims. Any mismatch returned the task to construction. A small number of sources had minor spatial offsets between the data footprint and the documented disaster location. Human experts removed the affected answer units rather than score a quantity the package could not support. Iteration scale. For difficult cases, constructing and reviewing a single question consumed more than 20 million tokens across repeated agent passes. These passes covered evidence exploration, draft construction, calculation checks, replay, discrepancy analysis, and revision. Human reviewers used them to align the scientific question, evidence, expert solution, deterministic program, answer units, and process rubric before release. C.7 Benchmark construction and release gates Each construction stage produces an auditable artifact and an explicit release decision. A failed gate returns the task to the responsible stage. Seven release gates. 1. Event anchoring: verify identity, hazard type, time window, spatial scale, and event–source agreement. 2. Package assembly: build a typed relative-path manifest and confirm that the required evidence roles are readable. 3. Task authoring: require package-local resolution without revealing a file path or answer-bearing value. 4. Reference analysis: align claims, units, mechanisms, and defensible alternatives; trace every load-bearing claim. 5. Executable grounding: regenerate structured ground truth deterministically and pass schema and tolerance checks. 6. Rubric design: map the task obligations to five to eight criteria totaling exactly 20 points. 7. Final review: align the question, solution, ground truth, program, and rubric before release approval. D Extended Diagnostics Table 8 breaks Combined Core down by seven overlapping capability tags. Because a task may carry several tags, the columns are diagnostic views rather than additive partitions. The strongest systems lead across nearly every tag, but their residual errors concentrate in the connections between results. Among the top five, causal-chain scores trail quantitative calculation by 4.9–7.9 points. A single defensible data path may be enough to compute a total or ratio. An evolving hazard account is harder to stabilize: rainfall, exposure, damage, and response evidence must refer to compatible windows and scales, and the proposed mechanism must survive their disagreement. GeoMMAgent breaks the otherwise stable ordering. Its short, structured search works well on tabular evidence but often bypasses raster intermediates, leaving remote sensing at 58.82. Capability scores thus capture the organization of an investigation as much as the knowledge of its backbone. D.1 Capability-conditioned performance 27 EarthVerse: Benchmarking Scientific Agents Across Dynamic Earth Systems and Natural Hazards Table 8: Core by capability. Tags overlap; system groups follow Table 2. Bold blue marks column bests. Earth-process analysisEvidence synthesis System Physical mech. Spatio- temporal Quant. calc. Multi- source Causal chain Ranking RS / geo. Agent frameworks GeoMMAgent67.7272.9468.8570.5666.1567.00 58.82 OpenResearcher30.4635.2431.3534.6325.3530.41 33.18 Open-weight and Earth-specialized models Qwen3-235B31.9040.3534.0335.5129.0029.29 32.32 GeohazardGPT31.7731.3128.0430.4232.1239.04 29.16 Qwen3-4B24.6322.7124.7024.3225.8924.06 32.61 GeoGPT25.3425.7024.6623.1423.6326.28 22.77 MiroThinker 8B13.5114.3914.6814.528.616.76 21.96 Nemotron Nano11.7914.4112.3017.8912.4213.26 15.48 Intern-S1-mini12.0410.7011.9910.4510.5713.21 16.50 Hosted general-purpose systems Claude Fable 585.1585.5187.1483.4582.2080.98 89.99 GPT-5.6 Sol81.8183.3185.4282.1377.5471.63 88.50 GPT-5.6 Terra80.8379.8382.9679.0676.2676.12 86.27 GLM-5.280.2577.3080.2476.8580.0878.34 79.27 GPT-5.577.2178.3881.9176.7974.3375.76 83.77 Claude Sonnet 4.675.4973.6078.3677.9273.4379.73 75.93 Kimi K2.575.0779.1077.8175.3470.4870.82 77.94 DeepSeek V4 Flash73.6376.3375.1672.5968.9172.33 77.75 Tencent Hy372.6775.9273.7170.3269.7270.23 79.41 Claude Haiku 4.569.6072.4476.0973.5067.7366.76 79.66 Qwen 3.7 Plus70.4172.4071.1369.2767.1968.08 78.26 Doubao Seed 2.0 Mini63.2466.4362.6759.3060.4263.51 70.67 Gemini 3.1 Flash Lite60.1861.4060.3856.9956.3758.35 68.49 MiniMax M2.742.7652.9148.8049.5339.9924.21 42.28 GPT-4o39.9738.8638.7734.9337.5643.70 48.30 GPT-5.6 Luna34.7442.9138.5836.4831.7721.17 38.21 D.2 Preference structure and diagnostic error paths Pairwise judgments broadly recover the leaderboard order while exposing uncertainty hidden by Core alone. Figure 7 separates wins, losses, and ties in opponent-balanced comparisons. Leaders win consistently, adjacent frontier systems remain close, and ties are uncommon. The drop beyond the leading group agrees with low Strict@95: partial success is common, while consistently complete research remains rare. Position. Reliability in dynamic Earth systems depends on revising the claim–evidence state as observations change. A system must detect conflicts, isolate local errors, and carry corrections into derived quantities, causal interpretations, and the final answer. Systems differ not only in how often they err, but also in whether those errors are found, contained, and repaired. Longer traces, more tool calls, and higher execution success do not measure that capacity [11, 19]. 28 EarthVerse: Benchmarking Scientific Agents Across Dynamic Earth Systems and Natural Hazards 1007550250255075100 Share of pairwise judgments (%) GPT-5.6 Sol Fable 5 GPT-5.6 Terra GLM-5.2 GPT-5.5 Sonnet 4.6 DeepSeek V4 Haiku 4.5 GeoMMAgent Qwen 3.7 Tencent Hy3 Kimi K2.5 GPT-5.6 Luna MiniMax M2.7 Qwen3-235B Doubao 2.0 Gemini 3.1 GPT-4o OpenResearcher Qwen3-4B Nemotron GeohazardGPT GeoGPT Intern-S1 MiroThinker +67.4 +45.1 +53.5 +53.5 +56.9 +57.7 +38.9 +30.3 +28.1 +26.4 +25.7 +20.8 +14.6 +12.1 -7.1 -10.0 -27.1 -40.3 -42.9 -44.3 -47.1 -61.4 -75.0 -86.4 -90.0 NetLossWin LossTieWin Figure 7: Pairwise win, tie, and loss shares. Net is win share minus loss share. 020406080100 Task Core Frontier model Agent framework Open weight Earth specialized 024 Errors per task Figure 8: Task Core and diagnosed errors by system family. Figure 8 shows the same separation at task level. Frontier models have a dense high-score mode but retain a long tail toward zero. Agent frameworks cluster at moderately high scores, while open-weight and Earth-specialized systems shift downward and rarely approach complete solutions. Error counts overlap far more than Core scores. Strong systems still make mistakes; their advantage lies in catching or containing them before the final answer. Error frequency therefore describes the research process, while Core records the damage that remains. The family-level flow in Figure 9 sharpens this result. Computation and answer errors occur in every family, yet their consequences differ. Frontier systems more often keep affected tasks in the 40–70 or above-70 bands; open-weight and Earth-specialized systems more often fall below 40. Recovery, cross-source checking, and final synthesis determine whether a local mistake remains local or spreads through the answer. 29 EarthVerse: Benchmarking Scientific Agents Across Dynamic Earth Systems and Natural Hazards Frontier model Agent framework Open weight Earth specialized Core < 40 40-70 > 70 System familyPrimary error Task Core Figure 9: System family, primary error family, and task-level Core band. Figure 10 examines the same pattern within systems. The largest Core losses accompany premature stopping, failed recovery, tool failure, and missing required outputs—failures that interrupt the research loop itself. Numerical and schema errors are common, but their average effect is smaller because some runs catch or absorb them before finalization. The inset connects missing evidence, output omissions, schema mistakes, and calculation errors into one cluster. These are often successive stages of a failed investigation, not independent defects. 30 EarthVerse: Benchmarking Scientific Agents Across Dynamic Earth Systems and Natural Hazards −60−40−20020 Within-system Core difference Premature stopping Overthinking/recovery Tool selection/execution Output missing Mechanism misidentified Alternative untested Evidence not found Formula/metric substitution Resolution mismatch Unsupported claim Threshold/baseline error Event/entity mismatch Wrong source Causal direction error Cross-source synthesis Insufficient coverage Decision/ranking error Evidence conflict ignored Unit/dimension error Final-answer inconsistency Scale mismatch Schema misread Calculation error Wrong time window Spatial mismatch Scenario/counterfactual Citation mismatch Unverified computation Output missing Calculation error Decision error Evidence not found Schema misread Evidence Scale Computation Reasoning Answer Process Figure 10: Within-system Core differences and the strongest error co-occurrences. D.3 Reasoning modes and resource accounting Tables 9–11 report trajectory, latency, token use, and evaluation cost. They separate interactive retrieval from direct serialization and model activity from successful evidence acquisition. D.3.1 Interactive trajectory diagnostics Higher reasoning effort changes the treatment of evidence more than the amount collected. At xhigh, the EarthVerse harness opens about as many files as at medium effort and uses fewer rounds, yet takes almost twice as long. The additional work occurs between observations: the model compares interpretations, revisits calculations, and decides whether its account is complete. Explicit review intensifies this pattern. From harness-high to review-xhigh, output nearly doubles while file reads barely move. Some of that deliberation repairs omissions; some only restates a settled path. Rounds, tokens, and latency therefore measure different kinds of work, and none is a reliable stand-in for scientific progress. 31 EarthVerse: Benchmarking Scientific Agents Across Dynamic Earth Systems and Natural Hazards Table 9: Interactive trajectory diagnostics. Per-task means; call success is the share of tool calls that return normally. ExecutionEvidenceResources ModeEffortRounds Tool calls Success (%) Reads Files Evidence refs. Input (k) Output (k) Total (k) Cached (k) Latency (s) EarthVerse none9.68 8.6892.083.84 7.009.94 94.32.6 96.910.6140 low14.70 13.7093.675.64 9.4414.70 157.45.1 162.415.4259 medium17.20 16.2095.396.84 10.7118.12 198.67.9 206.417.9424 high14.96 13.9697.195.15 9.9217.19 164.89.5 174.313.2571 xhigh11.25 10.25 100.004.75 10.2516.50 90.69.4 100.015.9674 Reviewnone11.62 9.6294.264.96 7.4210.46 126.93.8 130.718.8108 low18.18 16.1894.946.84 10.6017.02 219.08.0 227.025.1253 medium20.81 18.8194.786.56 11.8821.02 253.812.5 266.325.2552 high20.63 18.6395.376.50 12.1321.10 239.215.9 255.122.21,105 xhigh17.57 15.5799.295.86 10.1418.29 198.918.8 217.725.61,493 D.3.2 Direct package-serialization diagnostics Direct conditions receive a fixed package serialization before generation. Compressed direct includes more files in far fewer tokens than all-text, yet neither mode lets the model return to a source after discovering that a window, unit, or variable was wrong. That restriction changes the task. The model can reason about the supplied snapshot, but it cannot conduct the next measurement implied by its own reasoning. Larger inputs and longer outputs fail to rescue these baselines because the evidence schedule is fixed before the analysis begins. The missing operation is revision of the observation set, not another pass over the same text. Table 10: Direct-serialization resources. Direct modes inject a fixed file set before generation; values are per-task means. ModeEffort Injected files Evidence refs. InputOutputTotal Latency (s) Compressed none17.9224.56 13,016878 13,89455.61 low17.9225.31 13,0331,857 14,890120.45 medium17.9225.72 12,8383,004 15,842152.36 high17.9326.73 12,9583,630 16,58871.47 xhigh17.6725.00 12,8784,726 17,60488.14 All textnone7.5713.89 80,842751 81,59332.06 low7.6814.00 80,8861,854 82,73944.10 medium7.4613.90 80,9412,557 83,49861.02 high8.7315.13 81,6233,735 85,35978.49 xhigh8.0010.00 83,6554,013 87,66894.38 D.3.3 Evaluation cost Table 11 reports the cost of the evaluated runs for models with a comparable USD list price. Table 11: Evaluation cost. Rows descend by total cost. SystemCore Tokens/task (k) Price (USD/M) Cost (USD) Claude Fable 584.97161.650.003,271.57 GPT-5.578.06224.930.002,732.09 GPT-5.6 Sol81.68205.930.002,501.58 Claude Sonnet 4.676.16331.015.002,011.07 GPT-5.6 Terra80.06182.615.001,109.03 Claude Haiku 4.572.23477.05.00966.02 GPT-4o39.23127.410.00515.92 GPT-5.6 Luna35.66156.96.00381.31 DeepSeek V4 Flash73.54389.70.2844.19 Gemini 3.1 Flash Lite59.6670.11.5042.61 32 EarthVerse: Benchmarking Scientific Agents Across Dynamic Earth Systems and Natural Hazards Quality and cost do not form a simple ladder. The strongest frontier evaluations exceed $2,500, while GPT-5.6 Terra retains about 98% of GPT-5.6 Sol’s Core at 44% of its cost. The two Flash systems remain below $45 but differ by almost 14 Core points. A long trace from a low-price model may still cost less than a short frontier run, so quality and cost should be reported together when planning replication. D.4 Tool, memory, and interaction ablations Table 6 reports outcome and capability scores for each ablation; Table 12 shows how the same runs searched, computed, and consumed context. Token use is descriptive and does not affect scoring. D.4.1 Research behavior and context consumption The trajectory diagnostics expose two different kinds of false productivity. Under aggressive compression, the agent lists files more than four times as often as with long retention and records more evidence references, yet receives the lowest process score. It is rebuilding context that the controller discarded, not broadening the analysis. The expert-only suite looks efficient for the opposite reason: it has the highest call success while inspecting the fewest distinct files. Its tools run cleanly, but the investigation is narrow. Activity counts become meaningful only when tied to state changes: a new source should resolve a claim, a computation should test a mechanism, and a repeated read should have a reason. D.4.2 Capability-conditioned effects The largest capability losses occur when evidence must remain connected across several steps. With 170 visible tools, multi-source performance falls to 47.50 even though most other catalog configurations stay near or above 88. Aggressive compression loses 20.07 points in causal-chain reasoning relative to long retention. The two interventions damage continuity in different places. A large catalog makes the next action unstable; compression makes the meaning of earlier actions unstable. A single calculation can survive either disturbance because its inputs are local. A physical account cannot: its validity depends on remembering why several observations belong together. 33 EarthVerse: Benchmarking Scientific Agents Across Dynamic Earth Systems and Natural Hazards Table 12: System-ablation trajectories. Per-task means for the panels in Table 6. ToolsEvidenceActionsProcessTokens Condition Calls OK Calls failed Success (%) Files Evidence refs. Lists Searches Python Process (%) Input OutputTotal Cached (a) Tool-catalog scale Fixed tools (6)15.80 2.1592.52 12.7519.80 1.954.455.4565.62 219,639 9,918 229,557 25,882 Fixed tools (10)15.25 1.7592.22 11.0517.40 1.303.303.3557.10 185,888 8,159 194,047 25,408 Fixed tools (20)15.55 2.0591.74 12.0517.15 1.453.104.6059.38 223,666 9,246 232,912 25,267 Fixed tools (40)15.75 1.9592.81 11.9519.70 1.554.154.1061.00 224,175 8,818 232,993 24,346 Fixed tools (70)15.00 1.9092.39 11.4018.35 1.253.352.8561.25 224,853 8,906 233,759 26,445 Fixed tools (100)16.35 2.0092.54 12.0020.25 1.103.903.6063.12 268,527 10,409 278,936 32,294 Full tool catalog (170)13.90 1.5095.04 9.5513.15 1.051.852.5555.88 264,983 8,233 273,216 33,331 (b) Semantic routing and tool composition Top-10 dynamic router14.65 2.1593.16 12.4018.65 1.403.504.7066.25 205,908 10,319 216,227 29,286 Top-20 dynamic router12.75 1.6592.98 11.3016.15 1.352.604.0565.88 170,972 9,038 180,010 21,696 Workflow-aware router15.20 1.9593.38 12.2521.65 1.404.653.4556.75 210,928 9,304 220,232 26,381 Expert-only suite17.70 0.6097.59 7.3512.60 0.000.000.0060.88 283,502 11,988 295,490 24,192 General + expert suite14.05 1.5594.43 10.4015.40 1.403.053.4563.00 186,216 8,875 195,092 31,437 Atmospheric Python suite 14.60 1.3594.78 10.3518.40 1.454.102.8560.12 191,208 7,593 198,802 25,344 Atmospheric Fortran suite 14.55 1.8593.68 11.2517.10 1.553.353.3057.88 195,439 9,823 205,262 31,078 (c) Context retention Aggressive compression24.60 1.9593.88 12.5034.90 5.406.254.0051.62 187,492 9,545 197,038 10,765 Short retained context20.20 2.2593.64 13.7526.80 2.706.405.1056.38 198,062 11,765 209,827 14,682 Long retained context13.35 1.8592.77 11.8016.95 1.203.704.4563.38 218,347 9,189 227,536 48,998 Full retained context12.15 1.3093.84 11.5018.00 1.053.903.4058.25 180,517 7,182 187,700 80,819 (d) Interaction budget 12-turn budget9.95 0.3097.36 8.5013.05 1.152.701.3049.50 101,826 4,437 106,263 23,002 24-turn budget12.90 1.6593.11 10.8016.90 1.354.203.3052.88 153,261 8,143 161,404 32,346 96-turn budget13.60 1.3094.12 11.9018.45 1.603.854.5560.12 173,925 8,873 182,798 27,763 Unbounded turn budget15.20 1.8094.22 12.3021.40 1.605.154.3564.50 211,751 9,429 221,180 30,131 D.5 Oracle, interface, and evidence robustness Table 4 consolidates outcome scores, paired uncertainty, and capability slices for each intervention. Table 13 separately reports trajectory and resource effects. Confidence intervals resample paired task-level Core differences. Oracles provide guidance without answers, while robustness conditions alter only package evidence. D.5.1 Paired intervention uncertainty Only the two evidence-localization oracles have intervals that exclude zero: relevant files [+5.48, +24.69] and the evidence map [+2.34, +21.25]. The stable gain comes from knowing where evidence is, not from receiving a generic plan. A plan names familiar analytical moves; localization resolves an event-specific uncertainty that scientific knowledge alone cannot settle. The other interventions remain suggestive because their intervals cross zero. On a diagnostic manifest, a few packages whose structure favors one interface can still move the mean substantially. D.5.2 Trajectory and resource effects Oracle guidance shortens search without necessarily reducing cost. The evidence map lowers rounds and file reads, but the injected structure itself consumes context and takes time to interpret. Distractors produce the reverse pattern: the agent performs more actions and records more evidence references with almost no quality loss. Obvious clutter is therefore mostly an efficiency tax. A compact evidence description poses a different problem. The agent must decide whether to trust the summary or reopen the underlying source, and the latter is often necessary to recover metadata, aggregation choices, or conflicts. Search length and evidential control are related, but they are not interchangeable. 34 EarthVerse: Benchmarking Scientific Agents Across Dynamic Earth Systems and Natural Hazards Table 13: Intervention trajectories. Panels match Table 4; values are per-task means. ExecutionEvidence and computeLatencyTokens InterventionRounds Tool calls Success (%) Reads Files Evidence refs. Python SecondsInput OutputTotal Cached (a) Answer-free information oracles Relevant-file oracle15.30 14.3092.17 5.15 11.3016.104.15 217.26 181,939 5,572 187,512 19,507 Evidence-map oracle11.10 10.1093.28 3.10 8.5511.102.75 436.56 290,368 5,050 295,418 62,950 Tool-subset oracle17.80 16.8093.18 6.60 13.3521.603.90 212.74 205,252 5,628 210,880 27,738 Planning oracle16.95 15.9591.49 5.45 12.6518.304.05 185.18 196,489 5,817 202,306 25,190 (b) Tool-interface presentation Hierarchical tool catalog16.60 15.6091.97 5.75 12.4517.753.85 197.77 182,350 6,000 188,351 20,416 Descriptive tool names18.25 17.2593.42 5.40 12.2519.653.65 150.14 202,390 5,462 207,853 28,749 Dynamic tool descriptions17.55 16.5592.12 4.00 12.8519.953.65 182.42 199,673 5,036 204,709 23,053 Opaque tool names17.90 16.9092.44 7.00 10.8516.504.30 234.10 215,650 6,294 221,943 23,027 (c) Evidence perturbations Equivalent unit change16.45 15.4590.70 5.20 13.2519.253.90 176.16 184,679 5,153 189,832 19,904 Added distractor files19.60 18.6089.97 6.20 13.1022.504.15 162.54 220,098 5,658 225,756 25,792 Missing evidence17.40 16.4089.12 5.65 12.0521.203.55 181.47 198,173 5,219 203,392 19,546 Truncated evidence16.10 15.1091.65 4.25 12.7522.103.55 265.79 184,244 5,827 190,071 16,781 Conflicting evidence16.50 15.5092.04 5.65 12.5519.703.25 155.59 182,815 5,875 188,689 18,624 (d) Optional meteorological access Compact meteorology suite17.95 16.9594.90 5.85 12.7520.604.05 223.20 201,057 5,822 206,879 27,571 Cue-routed meteorology suite15.65 14.6592.50 5.65 11.5017.853.15 238.22 170,431 5,875 176,306 25,882 D.5.3 Capability-conditioned intervention effects The effective oracles help most with spatiotemporal reconstruction: both exceed 90, compared with 64.40 at baseline. File localization directly removes uncertainty about which records contain the event window and region. Ranking and decision improve less because the oracle does not say how competing quantities should be weighted. Opaque tool names damage the same capability most strongly. Reconstructing a window rarely depends on one call; it requires a sequence of filtering, reading, aggregation, and comparison. Losing the semantic role of any operation makes the resulting timeline harder to recover and harder to audit. D.6 Professional meteorological environments The professional-environment study separates method availability from execution. Compliance records whether the required interactions occurred; call success records whether they completed without adapter, argument, or runtime failure. D.6.1 Compliance, execution reliability, and implemented scope Nearly every run attempts the required specialist interaction, so willingness to call the tool is not the bottleneck. Execution is. Specialist-call success ranges from 20.29% for PyET to 97.56% for R climate, and high-success environments usually need fewer repair rounds and smaller contexts. The ranking should not be read as a comparison of languages. It reflects the overlap between the audited operations and the data actually present in the packages. R climate, Julia, and CDO often accept the available time series or grids directly; other adapters require variables, record lengths, or sampling structures that many event packages do not contain. 35 EarthVerse: Benchmarking Scientific Agents Across Dynamic Earth Systems and Natural Hazards Table 14: Scientific-environment outcomes and execution. Final Answer, Process, and Core scores are reported with per-task compliance, successful specialist calls, resource use, and implemented operations. OutcomeExecutionResources EnvironmentOperationsAnswer Process Core Comply (%) Success (%) Rounds Tokens Latency (s) (a) Python scientific libraries MetPyrain summary; profiles; kinematics68.6970.38 69.5310038.8325.00 311,043 255.60 xclimheat/rain extremes; wet/dry spells67.7769.62 68.709538.7824.50 299,074 268.44 climate-indices SPI; SPEI; PET64.5066.75 65.629522.6030.55 374,199 306.69 PyETPenman–Monteith; Hargreaves65.3467.38 66.369520.2932.30 387,917 315.96 thermofeelUTCI; heat index; wind chill66.7868.75 67.7610026.8528.55 346,924 357.88 pyextremesblock maxima; POT; return levels59.0260.00 59.5110034.1928.05 363,839 289.16 xskillscoreRMSE; correlation; Brier; CRPS63.3764.12 63.759557.1423.70 297,724 231.47 (b) Compiled and domain-language environments Juliasummary; integration; sensitivity71.7770.25 71.0110091.1121.50 256,852 204.47 R climatesummary; SPEI; GEV; trend68.5571.38 69.9610097.5621.70 259,984 190.20 CDOtemporal and field aggregates66.9169.50 68.2010081.6322.30 268,906 197.90 NCOinspect; weighted mean; derive field67.3965.00 66.1910035.5926.65 321,773 230.95 NCLsummary only65.0365.00 65.019540.6225.70 308,559 270.94 Fortranrain; kinematics; moisture transport64.4664.25 64.359543.4825.70 301,236 222.60 Applicability and reproducibility. Environment names refer to the audited adapters in Table 14, not to every feature of the upstream packages. The outcome columns make the distinction explicit: R climate has the highest call success, while Julia has the highest Core. A successful call proves only that code ran. It does not establish that the estimator fits the record, that the sampling geometry supports the comparison, or that the returned value answers the question. Those checks sit between execution and scientific use, and the gap between execution reliability and final quality shows why they cannot be collapsed into one metric. Each adapter therefore records normalized arguments, status, output, and errors. It also declares the operations it implements and validates package-scoped inputs before execution. This makes a justified failure reproducible: another reviewer can see that a method was rejected because the record was too short or a required variable was absent. Merely rerunning a command is weaker. Scientific reproducibility requires reconstructing why the operation was admissible and how its output entered the claim. E Prompts and execution contracts The prompts are grouped by role: package-scoped solving and review, answer–process evaluation, and benchmark construction. Placeholders mark values supplied by the harness. The solver sees no demonstration, hidden reference, or recommended file. Appendix G presents representative runs. Runtime visibility. The solver sees the public question, event-package root, exposed tool schemas, and package- relative observations. Ground truth, reference solutions, rubrics, computation programs, judge records, and other model answers remain hidden. The trace retains both successful and failed tool calls. Depending on the condition, the controller may request two format repairs, run a bounded evidence review, require a specialist operation, or reserve the final call for a schema-valid submission. None of these transitions adds scientific evidence. Appendix C.2 records their limits. PROMPTRUNTIMEPackage-scoped research suite SOLVER EarthVerse interactive solver SYSTEM You are an autonomous extreme-event benchmark solver. You must solve the task by planning, selecting local files, executing tools, reading observations, and then producing a structured final answer. Use only local event-package evidence returned by tools. Do not use web knowledge. Never read computed_gt.json, solution_en.md, review.json, or compute_gt.py. Prefer deterministic Python calculations when the task asks for metrics. You must make real tool calls before finalizing. Every assistant message must follow one of the two response contracts below. TASK MESSAGE 36 EarthVerse: Benchmarking Scientific Agents Across Dynamic Earth Systems and Natural Hazards PROMPTRUNTIMEPackage-scoped research suitecontinued Task id: <<TASK_ID>> Event id: <<EVENT_ID>> Question path: <<QUESTION_PATH>> Event package path: <<PACKAGE_ROOT>> Question: <<PUBLIC_QUESTION>> Evidence boundary: Strict package-only mode: only the current question and event package evidence are visible. Tools: <<REGISTERED_TOOL_CATALOGUE>> TOOL ACTION CONTRACT Action tool_call Tool one registered tool name Arguments only the arguments required by that tool Reason why this call is needed for the current analysis FINAL ANSWER CONTRACT Action final_answer Answer the exact structure requested by the public question Confidence one numeric confidence value Evidence for each claim: package-relative source file and support type (direct metric, report claim, metadata, image context, table value, derived metric, or unsupported boundary) Research a brief ordered summary of the reasoning Limitations important unsupported or unresolved elements Start by inspecting the package inventory, then choose only the files needed for this event-specific disaster analysis. REVIEW Final evidence review Begin a final evidence-review phase before submitting the answer. Treat the prior answer as a draft: re-check its evidence provenance, formulas, units, intermediate values, competing physical explanations, and every required output field against the package. Use additional tools when verification is useful. Return either one tool action or a revised final answer; do not merely endorse the draft. You have at most <<MAX_REVIEW_TURNS>> review turns. PROMPTEVALUATIONJoint answer and process judge JUDGE Answer and research-process evaluation SYSTEM You are a strict but fair evaluator for a local extreme-event benchmark. Your job is to judge answer correctness and task-rubric quality. Use partial 0-100 scores and avoid old deterministic exact matching. CALIBRATION CORE Evaluate this model response for an extreme-event evidence-and-reasoning benchmark. Make one combined evaluation and produce two core 0-100 scores. - answer_correctness_score is the direct partial-credit comparison between the model's final answer and the ground truth. Treat the answer as a set of units: required fields, labels, decisions, values, units, orderings, alternatives, stage boundaries, and required conclusions. Missing required fields are incorrect; correct prose cannot replace a missing structured field. - llm_rubric_score must come from the task's own 20-point rubric. Compare the answer and visible trace against every criterion, assign earned points out of 20, and convert the result to percent. - Do not invent extra credit. A plausible narrative, many tool calls, or reading the right files is not by itself a correct solution process. Evidence, 37 EarthVerse: Benchmarking Scientific Agents Across Dynamic Earth Systems and Natural Hazards PROMPTEVALUATIONJoint answer and process judgecontinued intermediate values, transformations, and conclusions must agree. - Keep the two scores distinct. A correct-looking final answer cannot substitute for missing process evidence, and an attempted method cannot receive full rubric credit when its result is wrong. - Use only the supplied package evidence, ground truth, task solution, rubric, final answer, and model trace. Do not rely on web knowledge. Runtime payload: <<TASK_METADATA>> <<QUESTION>> <<GROUND_TRUTH>> <<SOLUTION_AND_20_POINT_RUBRIC>> <<MODEL_FINAL_ANSWER>> <<MODEL_TOOL_TRACE>> <<MODEL_RUN_NOTES>> EVALUATION RECORD Scores answer correctness and rubric-based process quality Answer units each required unit and whether it was satisfied Rubric ledger criterion-level points earned out of 20 Diagnosis verdict, critical errors, missing elements, unsupported claims Rationale one sentence explaining the score pair PROMPTCONSTRUCTIONBenchmark construction suite QUALITY Task construction acceptance review ROLE You are the final quality reviewer for one extreme-event benchmark task. INPUTS - The public question (question_en.md) - The expert solution (solution_en.md) - The executable ground-truth program (compute_gt.py) - The generated scoring contract (computed_gt.json) - The optional review record (review.json) TASK Verify that the five artifacts describe one event-specific scientific target. Do not approve a rewritten public question when the solution or GT still scores an older package-inspection objective. ACCEPTANCE CHECKS 1. The public question reads like a real disaster-analysis assignment. 2. It contains no local path, visible CSX identifier, package inventory, available-material wording, AOI, quality-gate, evidence-boundary, or benchmark phrasing, and it does not reveal the useful files. 3. The solution contains Final Answer, Key Computations, Reasoning Path, Computed Interpretation, and Scoring Rubric. 4. computed_gt.json declares schema_version 1.0, task and event identifiers, task type, evaluation mode, primary GT, key values, and a top-level rubric. 5. Deterministic tasks contain compact machine-checkable targets. Open expert tasks contain concrete judge anchors rather than a fabricated exact essay. 6. The rubric has 5-8 criteria whose points sum exactly to 20. 7. Former distractors are rejected through numeric tests, inequalities, process timing, or competing-mechanism checks rather than option prose. 8. compute_gt.py runs successfully and regenerates the scoring contract. OUTPUT Return changed or reviewed file paths, event and hazard type, task type, evaluation mode, acceptance status for every check, and any remaining ambiguity, missing-data risk, temporal/spatial mismatch, or unsupported overclaim. RUBRIC Event-specific rubric construction ROLE 38 EarthVerse: Benchmarking Scientific Agents Across Dynamic Earth Systems and Natural Hazards PROMPTCONSTRUCTIONBenchmark construction suitecontinued You are constructing the hidden scoring contract for one benchmark task. TASK Create a 20-point rubric that can be applied without guessing the author's intent. Use 5-8 small, independently judgeable criteria. Every criterion must contain name, points, description, and partial_credit. The point values must sum exactly to 20. REQUIRED COVERAGE - 3-4 points: final conclusion, numeric target, threshold state, proof result, ranking, or concise canonical stance. - 3-5 points: required quantitative extraction or calculation, including units, tolerances, and windows where relevant. - 2-4 points: formula, index, process-window, causal-chain, or score-ledger logic. - 2-4 points: fusion of reports, time series, rasters, imagery, exposure, infrastructure, or trajectory evidence into checkable anchors. - 1-3 points: a compact event-specific consequence that follows from the computed evidence. - 1-3 points: rejection of plausible competing mechanisms or overclaims. - 1-2 points: complete, concise output in the requested structure. HARD RULES Do not use vague criteria such as "good reasoning" or "uses evidence well." Do not award points for choosing an option after a converted task has become a numeric or structured analysis. A criterion that depends on an incorrect scientific result cannot receive full process credit. For llm_judge tasks, include canonical_stance, required_claims, required_metrics, forbidden_claims, and acceptable_variants in primary_gt. For hybrid tasks, preserve both the deterministic core and the reasoning anchors. RUBRIC RECORD Total points exactly 20 Criteria 5-8 independently judgeable items Each item event-specific name; points; full-credit requirement; partial-credit and error rule SOURCE Event authenticity and source audit ROLE You are auditing the real-event anchor of one event package. QUESTIONS 1. Did the event occur? Require at least one traceable source confirming the event name, time window, or principal affected region. 2. Is the source reliable? Prefer government agencies, international organizations, peer-reviewed work, and authoritative scientific institutions; then disaster catalogues and major news. Encyclopedias and generic search pages are supporting evidence only. 3. Does the source actually correspond to this event? Check event name, time, place, and hazard type. A source about the same hazard family but a different time or place is not a strong match. OUTPUT FIELDS - existence_verdict: confirmed | likely | weak | not_confirmed - temporal_match: exact | near | broad | mismatch | unknown - spatial_match: exact | regional | broad | mismatch | unknown - hazard_match: exact | compound_related | broad_family | mismatch | unknown - source_reliability: high | medium | low | mixed - release_decision: keep | keep_with_minor_fix | needs_stronger_source | replace_event - concise evidence and repair notes DECISION RULES Use keep when the event is confirmed, at least two of temporal/spatial/hazard matching are exact or regional, and the preferred evidence is authoritative. Use keep_with_minor_fix when the event is real but URL, naming, or scale needs a small correction. Use needs_stronger_source when current support is generic, secondary, or overly broad. Use replace_event when the event cannot be confirmed or the cited evidence clearly belongs to another event. 39 EarthVerse: Benchmarking Scientific Agents Across Dynamic Earth Systems and Natural Hazards F Representative case set The six cards below index remote-sensing reconciliation, urban pluvial causality, evidence substitution, multi-agent quantitative decomposition, integrated heat research, and a partial scientific failure. Together they cover five system families without repeating the full records that follow. REPRESENTATIVE CASE SET FRONTIER HOSTEDGPT-5.6 SolCSX-189_Q1Core 99.50 Uljin wildfire consistency. Report claims, wind and precipitation, Sentinel-2 dNBR, annual change, smoke transport, and bounded impact claims. TRACE Sixteen successful calls, including a failed semantic search recovered by a filename query and a final provenance check. GENERAL HOSTEDTencent Hy3CSX-059_Q2Core 100.00 NYC Ida pluvial overload. An hourly rainfall record inside a two-day rain shield, population load, surface change, and a controlled future perturbation. TRACE Ten evidence calls followed by one deterministic calculation of every ratio, normalization term, and scenario score. OPEN-WEIGHT LARGEQwen3-235BCSX-006_Q2Core 93.93 Heat-source substitution audit. Variable-level admissibility of an ERA5-Land aggregate substituted for local hourly apparent-heat and warm-night data. TRACE Eight calls covering inventory, source comparison, schema checks, corrected heat metrics, and numerical source impact. MULTI-AGENT FRAMEWORKGeoMMAgentCSX-063_Q1Core 83.50 Zhengzhou threshold ledger. Reported burst and multiday rainfall, product ordering, exposure thresholds, and a conjunctive flood diagnosis. TRACE Four coordinator roles and nine package-tool calls for retrieval, calculation, scientific synthesis, and schema assembly. RESEARCH FRAMEWORKOpenResearcherCSX-018_Q2Core 86.92 Sahel heat-response model. Hourly heat, wet bulb, warm nights, exposure, services, roads, surface stability, two indices, and a +1 C counterfactual. TRACE Twenty-one calls, including one failed heat script, an explicit repair, and independent checks of peaks, duration, density, scores, and deltas. GENERAL HOSTEDGPT-4oCSX-312_Q2Core 41.25 Partial scientific failure. The model identifies the coupled Kilauea mechanism but reports lava effusion rate instead of the required summit-collapse-to-flow volume ratio. TRACE Nineteen model rounds and nine tool calls: seven succeed, two guessed paths fail, and the final answer earns Answer 40.00 and Rubric 42.50. G Detailed research trajectories Appendix F identifies the six selected systems and tasks. This section expands each card into its scientific target, evidence sequence, calculations, and decision. The Uljin case retains the complete task and action order; the remaining records focus on the steps that determine the result. G.1 Full case: Uljin report-to-burn consistency QUESTION CSX-189_Q1 SCIENTIFIC TARGET Uljin wildfire report-to-burn consistency ledger A technical review team is checking whether the March 4−13, 2022 Uljin wildfire is numerically consistent with a dry, wind−assisted burn−scar diagnosis rather than a late−rain−control or image−baseline alternative. EVIDENCE SCOPE Use only package−local evidence. Do not use point−weather products for this task; build the ledger from the event report, package gridded precipitation and wind summaries, Sentinel−2 dNBR, and annual embedding−change context. 40 EarthVerse: Benchmarking Scientific Agents Across Dynamic Earth Systems and Natural Hazards QUESTION CSX-189_Q1continued REQUIRED COMPUTATION Compute these ledger values: − report_dry_wind_flag: 1 if the report links the event to strong winds and dry weather, otherwise 0. − smoke_transport_flag: 1 if the report states that smoke moved toward southern Japan, otherwise 0. − era5_mean_wind_speed_mps: vector speed from mean 10 m u and v components. − era5_wind_bearing_to_deg: direction toward which the mean wind vector points, degrees clockwise from north. − precip_mean_spread_m: spread between ERA5−Land, GPM IMERG, and CHIRPS event precipitation means. − dnbr_mean: mean dNBR over the event burn−change window. − dnbr_max_to_mean_ratio: dNBR maximum divided by dNBR mean. − dnbr_to_annual_change_mean_ratio: dNBR mean divided by annual 1−minus−cosine embedding−change mean. − reported_charred_area_km2: reported hectares converted to square kilometers. MECHANISM CHECKSApply these tests: − report_dry_wind_mechanism_test: pass if report_dry_wind_flag = 1. − smoke_transport_context_test: pass if smoke_transport_flag = 1. − positive_heterogeneous_burn_test: pass if dnbr_mean > 0 and dnbr_max_to_mean_ratio >= 2.0. − late_rain_control_test: pass only if rainfall is the dominant control; otherwise fail. − nuclear_damage_overclaim_test: pass if the answer avoids inferring nuclear− plant damage or radiation release. OUTPUT Evidence ledger − Report/dry−wind flag and smoke−transport flag − Mean ERA5 wind speed and bearing − Precipitation−mean spread across the three gridded products − Mean dNBR, dNBR maximum−to−mean ratio, and dNBR/annual−change ratio − Reported charred area in square kilometres Decision checks − Report mechanism; smoke context; positive heterogeneous burn − Late−rain control; bounded nuclear−impact claim INTERPRETATION − One or two sentences tied directly to the computed ledger CASE Research record: CSX-189_Q1 COMPLETE ACTION ORDER; OBSERVATION BODIES ABRIDGED 1list_package_filesDiscover Enumerated all 39 package files. 2 read_text_fileAnchor Confirmed Uljin, South Korea, 4–13 March 2022. 3search_package_filesMechanism Located report language linking strong winds and dry weather. 4 search_package_filesTransport Located the statement that smoke moved toward southern Japan. 5search_package_filesImpact quantity Recovered nearly 17,000 charred hectares. 6read_text_fileERA5-Land Read mean precipitation and mean 10 m wind components. 7read_text_fileGPM IMERG Read event precipitation mean of 2.884 m. 8 read_text_fileCHIRPS Read event precipitation mean of 4.456 m. 9read_text_fileSentinel-2 dNBR Read mean 0.303255 and maximum 0.708527. 10search_package_filesEmbedding search Initial semantic query returned no match. 41 EarthVerse: Benchmarking Scientific Agents Across Dynamic Earth Systems and Natural Hazards CASE Research record: CSX-189_Q1continued 11search_package_filesRecovery A filename-oriented query located annual cosine-change statistics. 12read_text_fileAnnual context Read mean 1-minus-cosine change of 0.013761. 13search_package_filesSmoke check Confirmed the report-side smoke passage. 14 read_text_fileFull report Read the authoritative HTML surrounding all extracted claims. 15 python_execDeterministic ledger Computed wind speed 1.392 m/s, bearing 331.253 degrees, precipitation spread 6.389 m, dNBR heterogeneity 2.336, dNBR/annual-change ratio 22.037, and 170 km2 charred area. 16 search_package_filesFinal provenance check Re-queried the most easily misquoted smoke-transport claim before finalization. COMPUTED LEDGER DRY-WIND FLAG 1SMOKE FLAG 1MEAN WIND 1.392043 m/sBEARING 331.252581 deg PRECIPITATION SPREAD 6.389107 mmMEAN dNBR 0.303255dNBR MAX/MEAN 2.336407 dNBR/ANNUAL CHANGE 22.037202CHARRED AREA 170.0 km 2 PASSReport mechanismPASSSmoke contextPASSHeterogeneous burn FAILLate-rain controlPASSBounded nuclear-impact claim EVIDENCE RECONCILIATION The burn scar is not accepted on appearance alone. The trace anchors two report claims, reconciles three precipitation products, and keeps event-window dNBR separate from annual embedding change. Step 10 also preserves the failed semantic search and the filename-based recovery that follows. CASE NYC Ida: urban pluvial overload TASK SCIENTIFIC TARGETReconstruct the New York City phase of post-tropical Cyclone Ida as a record hourly burst embedded in a broader two-day rain shield. EVIDENCE SCOPEKeep station-scale hourly rainfall distinct from the wider gridded rainfall field, then add population exposure and surface-change context. REQUIRED COMPUTATIONCompute the baseline response stress and repeat it under the prescribed future perturbation. RESEARCH TRAJECTORY: TEN EVIDENCE CALLS AND ONE CALCULATION 1list + anchorWindow Enumerated 39 files and fixed the inclusive 1–2 September 2021 event window. 2 read reportLocal forcing Recovered the Central Park hourly record and the conservative 10-inch regional reference. 3read 3 productsRain shield Read GPM IMERG, CHIRPS, and ERA5-Land event means and maxima. 4 read exposureContext Read 5.013 million exposed people, Sentinel-1 mean change of 0.234 dB, and annual embedding change of 0.022. 5python_execLedger + scenario Converted units, reconciled products, computed ratios and normalized terms, and reran the weighted index under the prescribed perturbation. EVIDENCE, METRICS, AND DECISION 42 EarthVerse: Benchmarking Scientific Agents Across Dynamic Earth Systems and Natural Hazards CASE NYC Ida: urban pluvial overloadcontinued WINDOW 2 daysRECORD HOUR 88.138 mmREGIONAL REFERENCE 254.000 mmGRID PEAK 136.888 m GRID MEAN 53.427 mmHOUR/PEAK 0.644REGIONAL/PEAK 1.856BURST/AREAL DAY 3.299 POPULATION 5.013 millionRAIN LOAD 267.812 million-person-mmSURFACE CHANGE 0.337 DECISION The baseline response-stress index is 72.6; it rises to 80.1 under the future perturbation (+7.5). The local hourly record is not treated as a substitute for the wider gridded rain shield. CASE Heat-source substitution audit TASK SCIENTIFIC TARGETDetermine whether a same-window ERA5-Land aggregate can replace the local hourly heat source. EVIDENCE SCOPE Audit variable availability for apparent-temperature duration and warm-night exposure rather than judging a source by filename. DECISION RULEAccept the substitution only when the replacement preserves every variable needed for the requested metrics. RESEARCH TRAJECTORY: EIGHT SUCCESSFUL CALLS 1list_package_filesDiscover Enumerated 37 files and identified the two competing heat products. 2read anchorWindow Fixed 31 March–4 April 2024. 3read sourcesCompare The local source contains hourly temperature, humidity, apparent temperature, and daily minima; the aggregate lacks the last two required roles. 4python_execSchema audit Confirmed that apparent temperature is the first missing field and daily minimum temperature is also absent. 5python_execCorrect + compare Recomputed the target heat metrics and measured the peak-temperature error introduced by the substitute. EVIDENCE, METRICS, AND DECISION MAX AIR 44.6 CMAX APPARENT 43.2 CHOURS APPARENT≥ 40 C 16WARM NIGHTS≥ 27 C 5 AGGREGATE MAX 35.3 CPEAK DIFFERENCE 9.3 C MISSING FIELDS apparent temperature; daily minimum SOURCE SWAP FAILSThe aggregate cannot compute apparent-heat duration or warm nights.The corrected label is sustained_apparent_heat_with_warm_night_exposure. CASE Zhengzhou: coordinated threshold ledger TASK SCIENTIFIC TARGETTest whether the 17–23 July 2021 Zhengzhou flood combined an exceptional hourly burst, multiday extreme rainfall, and dense urban exposure. REQUIRED COMPUTATIONBuild the rainfall-ratio, product-order, population, and infrastructure threshold ledger. DECISION RULEAssign the final label only when all four numerical thresholds pass. RESEARCH TRAJECTORY: FOUR ROLES AND NINE PACKAGE-TOOL CALLS 43 EarthVerse: Benchmarking Scientific Agents Across Dynamic Earth Systems and Natural Hazards CASE Zhengzhou: coordinated threshold ledgercontinued 1Agent 0: retrieveEvidence Read the event report, three precipitation products, WorldPop, and OSM with strict provenance. 2Agent 1: computeLedger Calculated rainfall ratios, product order, and the four threshold flags. 3Agent 2: synthesizeMechanism Joined physical rainfall evidence with population and road exposure. 4Agent 3: assembleOutput Built the requested schema without changing the shared evidence boundary. 5python_execVerify Recovered hour share 0.3272, annual ratio 0.963, and four true threshold flags. EVIDENCE, METRICS, AND DECISION THREE-DAY RAIN 617.1 mmONE-HOUR RAIN 201.9 mmHOUR SHARE 0.3272ANNUAL RATIO 0.963 MEAN ORDER GPM > CHIRPS > ERA5-LandMAX GRID MEAN 460.5514 mmMAX GRID 484.255 m POPULATION 5.2625 millionHIGHWAYS 618AMENITIES 50TUNNELS 2 ALL FOUR THRESHOLDS PASS Hour share≥ 0.30, annual ratio≥ 0.90, population≥ 5million, and highway features≥ 200. Final label: short_burst_multiday_urban_exposure. CASE Sahel heat: response load and warming scenario TASK SCIENTIFIC TARGET Model a five-day Sahel heat episode and its response burden. EVIDENCE SCOPECombine hourly heat and Stull wet-bulb temperature, warm nights, population, services, roads, gridded context, and surface stability. REQUIRED COMPUTATIONCalculate both response indices at baseline and under a +1 C counterfactual. RESEARCH TRAJECTORY: TWENTY-ONE CALLS 1list + openCore inputs Read hourly heat, WorldPop, AOI geometry, and OSM infrastructure. 2 python: exposureCounts Computed schools, health facilities, shelters, major roads, AOI area, and population density. 3 python: heatFailed attempt The first heat script failed because it assumed the wrong hourly field names. 4 python: repairSchema recovery Re-read the actual fields and computed Stull wet-bulb temperature and warm nights. 5open productsContext Read ERA5-Land temperature, GPM and CHIRPS precipitation, and Sentinel-2 dNBR. 6python checksHeat metrics Independently verified air and apparent peaks, duration, degree-hours, and density. 7 python scenarioIndices Rechecked baseline and +1 C scores, threshold counts, and deltas. HEAT, EXPOSURE, AND SURFACE CONTEXT MAX AIR 44.6 CMAX APPARENT 43.2 CMAX WET BULB 16.3605 CHOURS≥ 40 C 16DEGREE-HOURS 24.4 WARM NIGHTS 5 POPULATION 170,311.46DENSITY 3.46 km −2 SCHOOLS 6HEALTH FACILITIES 2SHELTERS 1MAJOR ROADS 57 SURFACE CHANGE 0.024472SURFACE STABILITY 0.97553 44 EarthVerse: Benchmarking Scientific Agents Across Dynamic Earth Systems and Natural Hazards CASE Sahel heat: response load and warming scenariocontinued INDICES AND +1 C SCENARIO BASE HEAT RESPONSE 77.344912BASE PRIORITY 86.374648 FUTURE MAX APPARENT 44.2 CFUTURE HOURS≥ 40 C 25FUTURE DEGREE-HOURS 40.4 FUTURE HEAT RESPONSE 84.444912FUTURE PRIORITY 93.094648DELTAS +7.1; +6.72 INTERPRETATION The explicit failed computation and repair remain in the record; the final scenario is supported by independently rechecked heat, exposure, and response terms. CASE Partial failure: correct mechanism, wrong physical index TASK SCIENTIFIC TARGETDiagnose the 2018 Kilauea eruption as isolated lower-rift, isolated summit, rainfall-triggered, or coupled summit-drainage/lower-rift activity. MECHANISM CHECKUse the evidence to distinguish the coupled explanation from all three alternatives. OUTPUTReturn one physical index and value, three evidence clues, and an operational implication. MODEL ROUNDS 19TOOL CALLS 9SUCCESS 7ERRORS 2UNIQUE EVIDENCE FILES 4 OBSERVED RESEARCH SEQUENCE 1list_package_filesDiscover Enumerated all 34 package files. 2 read USGS reportPrimary evidence Opened the authoritative summit-collapse and Lower East Rift Zone report. 3search packageWeak recovery Two semantic searches returned no direct match for the requested coupling language. 4read anchorEvent bounds Confirmed the 3 May–4 August 2018 event window and Kilauea location. 5read guessed pathsTool errors Two nonexistent guessed files produced the run’s only tool errors. 6 search + rereadFinal extraction Recovered downrift propagation and effusion above 100 m 3 /s, then finalized without computing the 0.8/0.8 volume ratio. RETURNED ANSWER DIAGNOSIS coupled summit drainage and lower-rift effusion PHYSICAL INDEX Lava Effusion RateVALUE 100 m 3 /s CLUES Downrift magma propagation after summit collapse; lava discharge above 100 m 3 /s; coordinated summit and rift monitoring. REQUIRED ANSWER DIAGNOSIS coupled summit drainage and LERZ effusion PHYSICAL INDEX summit-collapse-to-lava-flow volume ratioVALUE 1.0 The index follows from approximately 0.8 km 3 of summit collapse and 0.8 km 3 of lava flow. A complete answer also uses partial summit drainage, near-daily collapses, andM w 4.7–5.4 energy release, then rejects isolated-rift, isolated-summit, and rainfall-triggered alternatives. ASSESSMENT 45 EarthVerse: Benchmarking Scientific Agents Across Dynamic Earth Systems and Natural Hazards CASE Partial failure: correct mechanism, wrong physical indexcontinued ANSWER 40.00RUBRIC 42.50CORE 41.25 PARTIAL FAILURE The model identifies the correct coupled mechanism and gives a useful operational implication, but substitutes a real yet irrelevant rate for the required volume ratio. It also omits the summit evidence and explicit alternative-mechanism rejection. The failure is scientific and evidential, not merely a formatting breakdown. 46