Paper deep dive
MDArena: Evaluating Coding Agents on Realistic Molecular Dynamics Workflows
Nithishwer Mouroug Anand, Wei-Tse Hsu, Kyle Vaccaro, Eden James Gage, Jonathan David Colburn, Linda Xi Phan, Minjoon Seo, Kevin Guan, Philip C. Biggin
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 92%
Last extracted: 8/5/2026, 3:42:21 AM
Summary
The paper introduces MDArena, a benchmark for evaluating coding agents on realistic molecular dynamics (MD) workflows. It comprises 50 containerized tasks across 29 molecular systems and 14 research protocols. The study evaluates six model/harness configurations, finding that Codex GPT-5.5 (XHigh) performs best with 48% strict success, while hard tasks like membrane-protein preparation remain largely unsolved, highlighting a gap between supervised assistance and autonomous reliability.
Entities (10)
Relation Signals (6)
MDArena → contains → 50 containerized tasks
confidence 95% · MDArena, a benchmark of 50 containerized tasks drawn from active biomolecular simulation projects
MDArena → evaluates → Coding Agents
confidence 95% · MDArena: Evaluating Coding Agents on Realistic Molecular Dynamics Workflows
Codex GPT-5.5 XHigh → achievesbestperformanceon → MDArena
confidence 90% · Codex GPT-5.5 at extra-high reasoning effort performs best, reaching 24/50 Strict-Pass@1 successes
Harbor → implements → MDArena
confidence 90% · MDArena is implemented in the Harbor framework
Membrane-protein system preparation → ishardfor → Coding Agents
confidence 90% · Hard tasks remain largely unsolved, particularly membrane-protein system preparation
GROMACS → isusedin → MDArena
confidence 90% · The benchmark set comprises... simulation and trajectory analysis (GROMACS
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Accelerating scientific discovery is among the most consequential applications of AI, and computational biomolecular simulation stands out as a particularly promising target within this broader effort. Coding agents promise to automate significant portions of this workflow, yet their reliability on realistic molecular dynamics (MD) tasks remains poorly characterized. To address this issue, we introduce MDArena, a benchmark of 50 containerized tasks drawn from active biomolecular simulation projects, spanning 29 molecular systems and 14 broad research protocols, including trajectory analysis, complex system preparation, free-energy protocols, and enhanced sampling. We evaluate six model/harness configurations spanning Codex and OpenCode. Among the evaluated configurations, Codex GPT-5.5 at extra-high reasoning effort performs best, reaching 24/50 Strict-Pass@1 successes (48%), followed by Codex GPT-5.5 Medium with 21/50, and OpenCode Gemini Flash 3.5 with 20/50. Average correctness and process rewards are substantially higher than strict success rates across all configurations, indicating that agents frequently make meaningful partial progress but fail on the fine-grained details required for reproducible scientific workflows. Hard tasks remain largely unsolved, particularly membrane-protein system preparation and alchemical free-energy setup, both unsolved or near-unsolved by every evaluated configuration. MDArena thus exposes a substantial gap between the usefulness of coding agents as supervised assistants and their reliability as autonomous MD researchers, while providing a reproducible and extensible platform for tracking progress toward closing it.
Tags
Links
- Source: https://arxiv.org/abs/2608.02642v1
- Canonical: https://arxiv.org/abs/2608.02642v1
Trouble viewing inline? Open PDF directly →
Full Text
53,369 characters extracted from source content.
Expand or collapse full text
MDARENA: EVALUATING CODING AGENTS ON REAL- ISTIC MOLECULAR DYNAMICS WORKFLOWS Nithishwer Mouroug Anand 1∗† , Wei-Tse Hsu 1∗ , Kyle Vaccaro 1,2 , Eden James Gage 1 , Jonathan David Colburn 1 , Linda Xi Phan 1 , Minjoon Seo 1 , Kevin Guan 1 & Philip C. Biggin 1† 1 Department of Biochemistry, University of Oxford 2 Scripps Research Institute nithishwer.mourouganand@reuben.ox.ac.uk philip.biggin@bioch.ox.ac.uk ABSTRACT Accelerating scientific discovery is among the most consequential applications of AI, and computational biomolecular simulation stands out as a particularly promis- ing target within this broader effort. Coding agents promise to automate significant portions of this workflow, yet their reliability on realistic molecular dynamics (MD) tasks remains poorly characterized. To address this issue, we introduce MDArena, a benchmark of 50 containerized tasks drawn from active biomolecular simulation projects, spanning 29 molecular systems and 14 broad research protocols, includ- ing trajectory analysis, complex system preparation, free-energy protocols, and enhanced sampling. We evaluate six model/harness configurations spanning Codex and OpenCode. Among the evaluated configurations, Codex GPT-5.5 at extra-high reasoning effort performs best, reaching 24/50 Strict-Pass@1 successes (48%), followed by Codex GPT-5.5 Medium with 21/50, and OpenCode Gemini Flash 3.5 with 20/50. Average correctness and process rewards are substantially higher than strict success rates across all configurations, indicating that agents frequently make meaningful partial progress but fail on the fine-grained details required for reproducible scientific workflows. Hard tasks remain largely unsolved, particularly membrane-protein system preparation and alchemical free-energy setup, both un- solved or near-unsolved by every evaluated configuration. MDArena thus exposes a substantial gap between the usefulness of coding agents as supervised assistants and their reliability as autonomous MD researchers, while providing a reproducible and extensible platform for tracking progress toward closing it. 1INTRODUCTION Molecular dynamics (MD) provides a computational microscope for studying the motion of molecular systems. Over the past decades, it has been widely used across scientific disciplines and delivered insights into molecular mechanisms, dynamics, and functions in numerous studies (Hollingsworth & Dror, 2018; Huggins et al., 2019). While its underlying theory is well-established, carrying out a scientifically defensible MD workflow remains technically demanding, as it often requires meticulous system preparation, parameterization, selection of appropriate sampling protocols, and analysis tailored to the scientific question. These activities require substantial expertise and are often repeated across systems and projects, creating a strong incentive to automate their execution. Recent advances in large language models (LLMs) and coding agents offer a promising route toward such automation (Zhao et al., 2023; Liu et al., 2024b). In particular, MD is well-suited to agentic assistance because its workflows are mature, modular, and well-supported by a variety of simulation and analysis software (Abraham et al., 2015; Eastman et al., 2023; Michaud-Agrawal et al., 2011). Equipped with language-based reasoning and the capability to directly interact with computational environments, an agent could in principle translate a scientific objective into an actionable workflow, ∗ These authors contributed equally. † Correspondence: N.M.A. and P.C.B. 1 arXiv:2608.02642v1 [physics.chem-ph] 31 Jul 2026 orchestrate available tools, and execute iteratively adapting protocols given intermediate results or failures. Still, reliable automation of MD workflows and its own evaluation are non-trivial. The challenge lies in the sequential coupling between workflow stages: an error introduced at a certain step, such as an incorrect protonation state or ligand topology, can propagate and invalidate subsequent calculations. Compounding this problem, many mistakes are silent, as a successfully executed simulation may still produce plausible results even when the underlying system or protocol is flawed. A benchmark must therefore assess not only whether an agent can execute MD software successfully, but also whether its workflow and outputs are scientifically valid, especially in realistic scenarios that mirror actual research projects or drug-discovery campaigns. Existing benchmarks and evaluations often fall short on this front (Kumar et al., 2026; Campbell et al., 2026), relying on isolated toy problems and narrow research contexts while providing limited support to evaluate the validity of an agent’s workflow. To address this need, we introduce MDArena, an executable benchmark for evaluating coding agents on realistic molecular-simulation workflows. The initial benchmark contains 50 expert-curated tasks derived from real research projects, each accompanied by a containerized environment and formulated as a bounded scientific problem with a clearly defined objective and verifiable outcome. Characterized by three difficulty levels, the benchmark tasks span diverse biomolecular systems and simulation software, covering 14 method classes that include membrane-protein preparation, parameterization of non-standard residues/molecules, enhanced-sampling protocols, quantum-chemistry workflows, and advanced data analysis. For evaluation, MDArena reports two scores using a two-part verifier: a correctness score, reported by deterministic checks that examine the properties of resulting artifacts, and a process score, assigned by an LLM-as-judge that scrutinizes the agent’s execution trajectory against predefined criteria covering scientific decision-making and result interpretation. Reported separately, these two complementary scores distinguish valid solutions from superficially convincing workflows and reveal cases in which a sound process nevertheless produces an incorrect result. Using MDArena, we evaluate six contemporary model–harness configurations through Harbor (Harbor Framework Team, 2026) under a common execution protocol and two-part verification scheme. We find that the evaluated agents can solve many well-specified tasks, but remain brittle on long-horizon workflows. Even the strongest configuration solves fewer than half of the tasks, with membrane-protein preparation and other complex workflows remaining largely unsolved. Across models, agents frequently produce plausible intermediate artifacts and make substantial progress, yet overlook details that prevent a fully correct solution. These results characterize the frontier of agentic assistance in MD at the time of evaluation: the strongest evaluated coding agents are already useful for bounded tasks under supervision, but cannot yet be relied upon to conduct complete research workflows autonomously. 2RELATED WORKS The growing ubiquity of AI agents in scientific computing has driven a parallel effort to benchmark them on realistic scientific workflows, and molecular dynamics is no exception. Yet most existing benchmarks remain narrow in scope and fail to capture the breadth of high-level tasks relevant to MD workflows. MDCrow (Campbell et al., 2026) was the first to evaluate LLM agents with 40+ tools on 25 MD tasks. However, this benchmark is limited in scope, and modern coding agents now nearly saturate this suite. This was followed by NAMD-Agent (Chandrasekhar & Barati Farimani, 2025) which automates the CHARMM-GUI interface (Jo et al., 2008) via Gemini (Team et al., 2023) and Selenium. NAMD-Agent primarily serves as a system demonstration and does not introduce an accompanying benchmark. ChemGraph (Pham et al., 2026) applies multi-agent orchestration to computational chemistry workflows, including geometry optimization, reaction enthalpy, and Gibbs free energy. It reports 100% accuracy on all tasks tested, although the task suite is relatively small and focuses on textbook-scale problems. DynaMate (Guilbert et al., 2025) was the first to target biomolecular MD with realistic tasks, evaluating a 12-system benchmark and automating the industry-relevant M/GBSA binding-affinity estimation, though it remains a system demonstration rather than a held-out benchmark. MDAgent2 (Shi et al., 2026) contributes MD-EvalBench, the first dedicated 2 MD benchmark suite, alongside a domain-adapted Qwen3-8B model (Yang et al., 2025) trained via continued pretraining, supervised fine-tuning, and reinforcement learning with execution as the reward signal. Its scope, however, is restricted to LAMMPS (Thompson et al., 2022) code generation, with no biomolecular tasks. PolyJarvis (Zhao et al., 2026) extends this direction to polymer MD, orchestrating polymer MD toolkits via Model Context Protocol (MCP) servers, but remains restricted to polymer MD. MDGym (Kumar et al., 2026) is a dedicated benchmark in this line, evaluating agents on GROMACS (Abraham et al., 2015) and LAMMPS (Thompson et al., 2022) workflows rather than merely demonstrating agent capability. MDGym provides an important foundation, while also leaving several opportunities for broader evaluation. MDGym’s current biomolecular tasks score agents against simulation output directly, which can conflate protocol correctness with sampling noise. The task suite currently focuses on simulation execution given pre-built structures and topologies, leaving system preparation and parameterization as a promising direction for future extension. It also concentrates on well-known soluble protein systems, which leaves room to broaden coverage in follow-up work. Additionally, tasks are framed as property prediction problems with fixed accuracy thresholds (e.g., 5% deviation), an approach that could be complemented by grading strategies better suited to settings without a well-defined ground truth. The reported failure-mode taxonomy provide a useful starting point that could be complemented by more fine-grained, case-level analysis in future iterations. MDArena departs from these works along three axes. First, provenance: tasks are drawn from active research projects in a working MD group rather than authored as benchmark artifacts, allowing continual addition of novel research workflows as they arise. Second, scope: coverage extends beyond prior work into alchemical free-energy perturbation (FEP) (Mey et al., 2020) workflows, membrane- protein systems, protein-ligand complexes, and enhanced-sampling methods such as replica exchange of expanded ensembles (REXEE) (Hsu & Shirts, 2024) and alchemical metadynamics (Hsu et al., 2023). Third, evaluation: agent outputs and execution trajectories are graded against a dual-pronged scheme combining human-authored, criterion-anchored rubrics with deterministic correctness checks designed to catch subtle but common agent errors that rubric grading alone can miss. Together, these yield a more robust and diagnostically informative evaluation than execution- or output-matching alone. 3BENCHMARK CONSTRUCTION MDArena was designed around three goals. Realism: tasks are drawn from authentic MD workflows used in active research projects rather than toy examples. Diversity: tasks span a broad range of molecular systems, methods, analysis routines, and simulation packages (see Figure 1). Robustness: verifiers are built to capture a broad range of agent failure modes, with tasks and rubrics specifically designed to catch subtle errors. 3.1DATASET CONSTRUCTION AND CURATION An initial collection round yielded more than 100 candidate MD tasks from group members. The associated systems, trajectories, and intermediate files were drawn from completed or ongoing research projects (Hsu et al., 2022; Joshi et al., 2026), grounding the benchmark in realistic scientific workflows. Candidate problems were then filtered against two criteria: alignment with the dataset’s design goals, and difficulty sufficient to remain challenging for modern coding agents. This narrowed the pool to 50 high-quality tasks. The filtering criteria were chosen to probe the specific factors limiting AI agents in autonomously conducting MD-based biomolecular research. These include: negative examples, where the requested action is impossible or scientifically inappropriate, to probe judgement rather than execution alone; long-horizon tasks, to stress-test context-length limitations; and underspecified tasks, which lack full problem detail, to evaluate an agent’s ability to infer research intent from realistic but incompletely specified descriptions. The final dataset spans 29 systems, 22 software packages and toolkits, and 14 broad method classes (see Table A1 for per-task tags). The tasks exercise a broad range of established scientific software, spanning simulation and trajectory analysis (GROMACS (Abraham et al., 2015), OpenMM (Eastman et al., 2023), MDAnalysis (Michaud-Agrawal et al., 2011)), system preparation and small-molecule 3 Figure 1: Composition of the MDArena dataset. Pie charts showing the distribution of tasks across three axes: method class, system type, and software package. parameterization (AmberTools (Case et al., 2023), ACPYPE (Sousa da Silva & Vranken, 2012), the OpenFF Toolkit and Sage small-molecule force field (Boothroyd et al., 2023), RDKit (Landrum et al., 2013), Open Babel (O’Boyle et al., 2011)), free-energy and enhanced sampling workflows (OpenFE (Zhang et al., 2023), alchemlyb (Wu et al., 2024), PLUMED (Tribello et al., 2014)), quantum chemistry (ORCA (Neese et al., 2020)), and validation (physicalvalidation (Merz & Shirts, 2018; Merz et al., 2022)). Each task was assigned an easy, medium, or hard difficulty label by the task author and subsequently reviewed, based on factors including workflow length, required domain knowledge, and technical complexity for modern coding agents. The benchmark set comprises 11 easy, 23 medium, and 16 hard tasks. 3.2VERIFIER DESIGN MD workflows, like many computational science workflows, are difficult to verify reliably, largely due to the absence of a ground truth and the inherently stochastic nature of simulation (Wan et al., 2021; Communications Biology, 2023). MDArena therefore focuses on the stages surrounding simulation execution: preprocessing and system setup on one end, and analysis and post-processing on the other. Importantly, these are in fact the stages where much of the technical difficulty lies and where outputs can often be evaluated deterministically. Once a valid simulation input, such as a GROMACS TPR file, has been generated, launching the calculation is comparatively straightforward. To maximize verifier coverage without imposing overly rigid constraints on agents, we adopted a two-pronged verification scheme combining deterministic correctness checks with LLM-as-judge process checks. The deterministic correctness checks confirm that expected outputs are present and valid, and satisfy numerical, structural, or software-level requirements. The associated correctness reward is defined as the fraction of deterministic criteria satisfied. The process checks instead assess whether the broader, goal-oriented steps taken by the agent are reasonable, without penalizing agents that reach the correct answer through an unorthodox route. These checks are kept minimal, covering only the steps strictly necessary to reach a correct answer given the task instructions. Across the benchmark, the process verifier contains 243 author-defined criteria, with a median of four per task and a range of one to sixteen. Most are graded as binary pass/fail criteria, while a few use a Likert scale. 3.3IMPLEMENTATION FRAMEWORK To maximize adoption, MDArena is implemented in the Harbor framework (Harbor Framework Team, 2026), an agent-evaluation framework from the makers of Terminal-Bench (Terminal-Bench Team, 2025), making it straightforward for researchers both to evaluate their own agents against the 4 benchmark and to extend MDArena by adding new self-contained tasks. Each Harbor task is paired with a dedicated Docker image, giving every task an isolated, reproducible environment (Moreau et al., 2023). For every hard task and most medium tasks (31 of 50 in total), we additionally provide a reference solution: an executable script that Harbor’s built-in Oracle agent runs in place of an agent (Harbor Framework Team, 2026). Each reference solution is required to pass all correctness and process checks (reward 1), which serves two purposes — it confirms that the task is well-posed and solvable, and it validates the verifier itself, ensuring that a correct workflow is scored as a full success rather than penalized by an overly strict check. LLM-as-judge evaluation uses Claude Opus 4.8 as the judge across all tasks, chosen to ensure consistent grading. For each task, the judge is given the agent’s full execution trajectory (trajectory.json) together with a task-specific set of output files, and scores the task’s process criteria under a 300s to 600s judge timeout. 4EXPERIMENTAL SETUP 4.1HARNESS AND MODEL SELECTION A coding agent consists of a base language model paired with a harness, the software layer that provides tools, controls execution, and mediates access to the task environment. Recent research suggests that harness design can substantially influence coding-agent performance alongside the capabilities of the underlying model (Yang et al., 2024; Wang et al., 2025; Lee et al., 2026; Lin et al., 2026b;a). To examine the contribution of both model and harness choice, we evaluated agents using two distinct harnesses: OpenCode, an open-source harness, and Codex, OpenAI’s proprietary closed-source harness. Within OpenCode, we tested four models spanning proprietary and open- weight families and a range of inference costs: Gemini-3.5-Flash and Gemini-3.1-Pro-Preview (both proprietary), and DeepSeek-V3.2-MAAS (Liu et al., 2024a) and Qwen3-235B-A22B-Instruct-2507- MAAS (Yang et al., 2025) (both open-weight). Within Codex, we evaluated GPT-5.5 at medium and extra-high (XHigh) reasoning effort to probe frontier-model performance. Each model–harness pair was evaluated once per task, yielding a single-run Pass@1 estimate. 4.2EXECUTION ENVIRONMENTS AND RUNTIME LIMITS Each task in MDArena was implemented in a Docker container with allocated 2–4 CPUs, 8 GB RAM, 10 GB storage, no GPUs, and internet access enabled. By default, tasks use a 900-second agent wall-clock timeout and a 900-second verifier timeout. We treat the agent timeout as part of the benchmark protocol rather than as an infrastructure specification, as it directly affects reported performance. A uniform 900-second budget disadvantaged larger, reasoning-heavy “thinking” models at hard tasks, which often required more time to plan and execute before producing an answer. To prevent runtime budget from confounding task performance, we extended the agent timeout for select computationally intensive tasks. Verifier timeouts were similarly extended for specific tasks. Timeout and other runtime failures were counted as unsuccessful tasks for Strict-Pass@1, but are reported separately from verifier-scored failures to distinguish operational errors from incorrect completed workflows. 5RESULTS 5.1OVERALL ACCURACY The overall performance on MDArena is summarized in Figure 2 and Table 1. Full success is measured by Strict-Pass@1, a binary metric requiring all process and correctness checks to pass, while partial credit is captured separately via average correctness and process rewards. Strict-Pass@1 rates are modest across the board: the best-performing configuration, Codex GPT-5.5 XHigh, solves 24/50 tasks (48%), followed by Codex GPT-5.5 Medium at 21/50 (42%), OpenCode with Gemini Flash 3.5 at 20/50 (40%), and Gemini Pro 3.1 at 18/50 (36%). Open-weight models trail substantially, with DeepSeek-V3.2-MAAS solving 6/50 tasks (12%) and Qwen3-235B-A22B-Instruct-2507-MAAS solving just 1/50 (2%). The gap between Strict-Pass@1 and the average reward metrics indicates that agents often possess strong underlying capability but fall just short of a fully correct solution in these multi-step tasks. GPT- 5 Table 1: Overall baseline performance on the 50-task MDArena benchmark. Cost is the mean recorded API cost per task. AgentStrict-Pass@1CorrectnessProcessCost GPT-5.5 XHigh24/50 (48%)0.8380.856USD 2.10 GPT-5.5 Medium21/50 (42%)0.8360.778USD 0.96 Gemini Flash 3.520/50 (40%)0.6840.713USD 1.55 Gemini Pro 3.118/50 (36%)0.6480.604USD 1.25 DeepSeek-V3.26/50 (12%)0.5990.323USD 0.28 Qwen3-235B1/50 (2%)0.2140.129USD 0.13 5.5 Medium and XHigh achieve similar average correctness rewards (0.836 and 0.838, respectively), but XHigh attains a notably higher average process reward (0.856 versus 0.778), consistent with its three additional strict successes. This suggests the higher-effort model is more deliberate in its execution, accounting for edge cases that are predominantly captured by the process checks. Gemini- 3.5-Flash and Gemini-3.1-Pro performed similarly in terms of the Strict-Pass@1 rate and average correctness reward, though Gemini-3.5-Flash slightly outperformed Gemini-3.1-Pro in the average process reward. DeepSeek shows a similar dissociation at lower absolute performance: a moderate average correctness reward (0.599) alongside a much lower process reward (0.323), suggesting it often produces locally plausible artifacts that fail to satisfy the broader scientific workflow. Qwen, by contrast, frequently fails before even reaching the substantive stages of the workflow, leaving little or no scientifically usable output. Figure 2: Overall agent performance across all MDArena tasks. (a) Strict-Pass@1: a binary metric scored as success only if all process and correctness checks for a task pass. (b) Average correctness reward and (c) average process reward: each task defines multiple correctness and process checks, and we report the mean pass rate across these checks, averaged over all MDArena tasks. Error bars show standard errors across tasks, using binomial standard errors for pass/fail success rates 5.2COSTS AND EFFICIENCY Performance and cost breakdown by difficulty is summarized in Figure 3. The benchmark contains 11 easy, 23 medium, and 16 hard tasks. All agents perform reasonably on easy tasks relative to their overall scores, but hard tasks remain largely unsolved. Both GPT-5.5 configurations solve 9/11 easy tasks and just 1/16 hard tasks; XHigh’s overall advantage comes mainly from medium tasks, where it solves 14/23 compared with Medium’s 11/23. A similar pattern holds for Gemini: both configurations solve the same 7 easy tasks, but Gemini Flash 3.5 solves more medium tasks than Gemini Pro 3.1 (12/23 versus Pro’s 9/23), while Pro reaches slightly more hard-task successes (2/16 versus Flash’s 1/16). Gemini 3.5 Flash’s stronger overall performance than Gemini 3.1 Pro is not unexpected, as Flash is positioned specifically for coding, tool use, and long-horizon agentic workflows, whereas Pro is positioned more strongly around advanced reasoning and complex problem solving. What is notable is that Pro’s reasoning-oriented profile does not confer an advantage on MDArena. Its higher 6 action count and more iterative behavior suggest that sustained tool use and execution may matter more than reasoning strength alone for these workflows. Cost rises with difficulty for every configuration, tracking increases in both time on task and token usage. GPT-5.5 XHigh costs roughly USD 2.10 per task on average, more than twice GPT-5.5 Medium’s USD 0.96, for a six-percentage-point gain in Strict-Pass@1. Gemini Flash 3.5 is likewise more expensive than Gemini Pro 3.1 (USD 1.55 versus USD 1.25 per task) but achieves higher success, correctness, and process rewards; this follows from Flash taking more actions per task and behaving more agentically overall, which may contribute to both its higher cost and stronger performance. This cost-performance reversal is a reminder that cheaper or nominally stronger base models do not necessarily yield better agentic performance once harness behavior, tool use, and failure recovery are taken into account. Success generally improves with cost (Figure 3), as expected, but GPT-5.5 Medium occupies a favorable cost-performance position among the evaluated configurations: within the Codex harness, it costs roughly 23% less than Gemini Pro 3.1 and 38% less than Gemini Flash 3.5 under OpenCode, while still achieving higher success on both counts, suggesting that the complete model–harness configuration, not just the base model alone, shapes where a configuration lands on this frontier. Figure 3: Model performance and cost across task difficulty. (a) Strict-Pass@1 success rate by difficulty. (b) Average cost per task by difficulty: mean API cost per task for each model, split across easy, medium, and hard MDArena tasks. (c) Strict-Pass@1 versus average recorded API cost across model–harness configurations. 5.3PERFORMANCE ACROSS TASK CATEGORIES Performance varies sharply by task category (Figure 4). Agents are strongest on compact validation and troubleshooting-style tasks, though several of these categories have small sample sizes. System preparation, parameterization, and trajectory analysis are partly solved by the strongest agents, but are not fully solved by any evaluated configuration: the best-performing configurations solve 5/7 system preparation tasks (GPT-5.5 Medium and XHigh), 3/5 parameterization tasks (GPT-5.5 Medium, GPT-5.5 XHigh, Gemini Pro), and 6/13 trajectory analysis tasks (GPT-5.5 Medium, GPT-5.5 XHigh, Gemini Flash). Membrane-protein system preparation remains unsolved by every evaluated configuration. Higher-level tasks like free-energy planning further separate agents: GPT-5.5 XHigh solves 4/6 tasks in this category, while GPT-5.5 Medium, Gemini Flash 3.5, and Gemini Pro 3.1 each solve 2/6. Correctness and process rewards, however, paint a somewhat different picture, with stronger models achieving noticeably higher scores even on tasks they ultimately fail to solve (see Figure A2). These 7 category-level results illustrate a broader pattern: as tasks grow more complex and multi-step, the likelihood that an agent overlooks a key step rises accordingly, and errors compound across steps, increasing the chance of eventual task failure even when individual steps are handled competently. Figure 4: Agent success rate by task category. Strict-Pass@1 success rate for each agent, broken down across MDArena task categories. Error bars show standard errors across tasks, using binomial standard errors for pass/fail success rates 5.4OBSERVATIONS ON AGENT PERFORMANCE Because MDArena tasks vary across systems, methods, packages, and instruction styles, it is impracti- cal to isolate each dimension systematically. Instead, this section presents several representative case studies that capture broad, recurring trends that we consider most indicative of agent performance, focusing on dimensions along which success and failure patterns are particularly clear. 5.4.1WELL-SPECIFIED TASKS ARE COMPARATIVELY TRACTABLE Well-specified tasks are comparatively tractable for AI agents. Even multi-step tasks are handled well when each step is defined in fine detail. While we deliberately left some hard tasks underspec- ified to probe agent judgment, we observed that certain easy and medium tasks, despite involving what would be a tedious search through a large parameter space for a human, were comparatively easy for agents. Several such tasks are solved by most or all evaluated configurations, including 8 concentration/trajectory compatibility checks, alternate-location cleanup, clash checking, simple GROMACS parameter edits, and some ligand parameterization tasks. These tasks are compact, have clearly defined outputs, and typically admit direct validation with standard tools. They represent settings in which agents can already be useful under supervision. 5.4.2LONG-HORIZON, HIGH-LEVEL WORKFLOWS REMAIN BRITTLE Long-horizon, high-level workflows, especially those with multiple valid solutions, remain brittle. The hardest tasks combine multiple packages and long dependency chains, such as membrane-protein system preparation, alchemical free-energy setup, and enhanced-sampling workflows. Failures typically emerge in the middle to late stages of an agent’s trajectory but compound as they propagate through the remaining steps. A recurring driver of this failure mode is that agents tend to write bespoke scripts rather than rely on established packages such as MDAnalysis (Michaud-Agrawal et al., 2011), even when those packages are already available in the containerized environment. Although quick to write, ad hoc agent-written scripts often overlook file-format conventions and edge cases handled by mature software. One representative example: an agent adding residues from a PDB file into a GRO file with its own script failed to account for differing residue-numbering conventions between the two formats. In isolation, this would be a minor and easily fixed error. Left unaddressed, however, it compounded into topology-coordinate mismatches later in the pipeline that agents consistently struggle to resolve. This illustrates how a reflexive preference for writing new code, rather than reusing robust, purpose-built tools, can backfire over the course of a long workflow. 5.4.3OVERLOOKING FINE DETAILS UNRELATED TO THE PRIMARY TASK A distinct failure mode, most pronounced in weaker models, involves overlooking task-relevant details that fall outside the immediate focus of the instruction. Higher-reasoning, stronger models tend to avoid this pitfall by checking their outputs extensively before concluding a task; weaker models instead declare success without robust verification. This can also be read as a manifestation of agents having a tendency to act without fully reasoning through why each step is necessary, itself a symptom of the underspecification issue discussed above. For example, Task 12 asks the agent to transfer crystal waters from a PDB file into a GRO file. Crystal waters in PDB files are conventionally represented by the oxygen atom alone; correctly completing the task requires isolating these waters, parameterizing them with a compatible force field, and transferring the resulting waters back with hydrogens included. Instead, agents frequently transfer only the oxygen atoms and report the task complete. Qwen exhibits this failure mode heavily on easy tasks, more so than on medium or harder tasks, its failures are more fundamental, typically reflecting misinterpretation of the task or breakdowns in planning rather than oversight. On easy tasks, it typically completes the core objective but overlooks a peripheral detail: in a topology preparation task, for instance, Qwen correctly stages the resulting .top and .gro files into the output directory but omits the associated .itp files referenced by the topology, causing the correctness check to fail. This illustrates that even nominally successful task completion can fail on details incidental to, but required by, the stated objective, underscoring the gap between an agent appearing to finish a task and actually satisfying its full specification. 5.4.4OPERATIONAL STABILITY VARIES ACROSS AGENTS AND HARNESSES Operational failures were also strongly agent and harness dependent. Across 300 runs, 67 ended with an exception rather than a normal, verifier-scored outcome. Timeouts were the dominant failure mode, accounting for 36/67 errors overall, and were especially concentrated in Gemini Pro 3.1, where all 9 errored runs were timeouts. Qwen3-235B was the least operationally stable configuration, with 32/50 runs ending in error (Figure A1); its dominant failure mode was a nonzero agent exit, typically caused by the provider’s context-limit failures rather than a genuine verifier failure. By contrast, the Codex configurations were operationally stable throughout: GPT-5.5 XHigh produced only 1 errored run, and GPT-5.5 Medium just 4. Taken together, these patterns suggest operational instability is often a harness-model coupling issue rather than a purely intrinsic model weakness: models that consume more context, through verbose tool calls or extended reasoning, are more likely to exceed time or context budgets regardless of underlying reasoning quality. 9 6DISCUSSION Performance on MDArena shows that while coding agents can already be useful within clearly bounded task chunks, they remain far from autonomous computational researchers. Even the strongest evaluated configuration, GPT-5.5 XHigh, solves only 24/50 tasks (48%) at Strict-Pass@1, despite comparatively strong average correctness (0.838) and process (0.856) scores, and near-perfect operational stability (1/50 error runs). This gap between high partial credit and modest full success is the frontier for the next generation of coding agents: while some of it may be closed with better tooling and domain-specific skills, we believe true generalizable improvement will require gains in raw model intelligence. The evaluated open-weight models, meanwhile, remain far from competitive on this benchmark, and closing that gap will likely require substantial further progress before they are viable for MD research use. MDArena has several limitations. Each model-harness pair was evaluated only once per task, so our results reflect a practical baseline rather than repeated-run reliability, and stochastic variation across runs remains uncharacterized. The process verifier relies on an LLM judge, which, despite criterion-anchored rubrics, introduces its own model dependence. The task distribution is drawn from a single research group and may reflect that group’s particular expertise and conventions rather than the full diversity of MD practice. Because task selection deliberately emphasized workflows expected to expose known agent failure modes, the resulting success rates should not be interpreted as estimates of performance over all routine MD tasks. Finally, the benchmark intentionally emphasizes system setup and analysis over long-running, stochastic production simulation; extending MDArena to evaluate end-to-end simulation execution is an important direction for future work, provided this is done without collapsing into the trap of grading stochastic simulation output directly. Importantly, MDArena is intended as an extensible, longitudinal evaluation framework rather than a fixed snapshot. The initial task suite provides a versioned reference point, while future releases can incorporate additional systems, methods, and community-contributed workflows. Evaluations of recently released frontier models (e.g., GPT 5.6-Sol, Claude Opus-5, and Gemini-3.6-Flash) are already underway and will be soon incorporated into the next versioned release. More broadly, applying a consistent evaluation protocol across successive model–harness generations will allow MDArena to track both overall capability gains and persistent category-specific weaknesses, thereby meaningfully guiding the development of future AI agents for biomolecular simulations. DATA AND CODE AVAILABILITY The MDArena benchmark, including task definitions and inputs, Docker specifications, veri- fier implementations, reference solutions, and evaluation configurations, is publicly available at https://github.com/weitse-hsu/MDArena. The version corresponding to the results reported in this manuscript is archived as v0.1. ACKNOWLEDGEMENTS NMA is funded by the MRC via an iCASE studentship with Vertex Pharmaceuticals Europe Ltd with contributions from Reuben College, Oxford. We thank Fabio Zuccotto, Ewa Chudyk and Ron Knegtel for useful discussions. We also acknowledge the BBSRC (JDC, PCB) and the Wellcome Trust (JDC, PCB). EJG is supported by a Watson Scholarship and Corpus Christi College. MS is in receipt of a Department of Biochemistry scholarship. KV acknowledges the Scripps/Oxford programme. W-TH and LXP thank Ineos for support. REFERENCES Mark James Abraham, Teemu Murtola, Roland Schulz, Szil ́ ard P ́ all, Jeremy C Smith, Berk Hess, and Erik Lindahl. Gromacs: High performance molecular simulations through multi-level parallelism from laptops to supercomputers. SoftwareX, 1:19–25, 2015. Simon Boothroyd, Pavan Kumar Behara, Owen C Madin, David F Hahn, Hyesu Jang, Vytautas Gapsys, Jeffrey R Wagner, Joshua T Horton, David L Dotson, Matthew W Thompson, et al. 10 Development and benchmarking of open force field 2.0. 0: the sage small molecule force field. Journal of Chemical Theory and Computation, 19(11):3251–3275, 2023. Quintina Campbell, Sam Cox, Jorge Medina, Brittany Watterson, and Andrew D. White. MDCrow: Automating molecular dynamics workflows with large language models. Machine Learning: Science and Technology, 7(2):025037, 2026. doi: 10.1088/2632-2153/ae4b07. URLhttps: //doi.org/10.1088/2632-2153/ae4b07. David A Case, Hasan Metin Aktulga, Kellon Belfon, David S Cerutti, G Andr ́ es Cisneros, Vin ́ ıcius Wilian D Cruzeiro, Negin Forouzesh, Timothy J Giese, Andreas W Gotz, Holger Gohlke, et al. Ambertools. Journal of Chemical Information and Modeling, 63(20):6183–6191, 2023. Achuth Chandrasekhar and Amir Barati Farimani. Automating MD simulations for proteins us- ing large language models: NAMD-Agent, 2025. URLhttps://arxiv.org/abs/2507. 07887. Communications Biology. Reliability and reproducibility checklist for molecular dynamics sim- ulations. Communications Biology, 6:268, 2023. doi: 10.1038/s42003-023-04653-0. URL https://doi.org/10.1038/s42003-023-04653-0. Peter Eastman, Raimondas Galvelis, Ra ́ ul P Pel ́ aez, Charlles RA Abreu, Stephen E Farr, Emilio Gallicchio, Anton Gorenko, Michael M Henry, Frank Hu, Jing Huang, et al. Openmm 8: molecular dynamics simulation with machine learning potentials. The Journal of Physical Chemistry B, 128 (1):109–116, 2023. Salom ́ e Guilbert, Cassandra Masschelein, Jeremy Goumaz, Bohdan Naida, and Philippe Schwaller. DynaMate: An autonomous agent for protein-ligand molecular dynamics simulations, 2025. URL https://arxiv.org/abs/2512.10034. Harbor Framework Team. Harbor: A framework for evaluating and optimizing agents and models in container environments, 2026. URL https://doi.org/10.5281/zenodo.20953922. Scott A Hollingsworth and Ron O Dror. Molecular dynamics simulation for all. Neuron, 99(6): 1129–1143, 2018. Wei-Tse Hsu and Michael R Shirts. Replica exchange of expanded ensembles: A generalized ensemble approach with enhanced flexibility and parallelizability. Journal of Chemical Theory and Computation, 20(14):6062–6081, 2024. Wei-Tse Hsu, Dominique A Ramirez, Tarek Sammakia, Zhongping Tan, and Michael R Shirts. Identifying signatures of proteolytic stability and monomeric propensity in o-glycosylated insulin using molecular simulation. Journal of Computer-Aided Molecular Design, 36(4):313–328, 2022. Wei-Tse Hsu, Valerio Piomponi, Pascal T Merz, Giovanni Bussi, and Michael R Shirts. Alchemical metadynamics: Adding alchemical variables to metadynamics to enhance sampling in free energy calculations. Journal of Chemical Theory and Computation, 19(6):1805–1817, 2023. David J Huggins, Philip C Biggin, Marc A D ̈ amgen, Jonathan W Essex, Sarah A Harris, Richard H Henchman, Syma Khalid, Antonija Kuzmanic, Charles A Laughton, Julien Michel, et al. Biomolec- ular simulations: From dynamics and mechanisms to computational assays of biological activity. Wiley Interdisciplinary Reviews: Computational Molecular Science, 9(3):e1393, 2019. Sunhwan Jo, Taehoon Kim, Vidyashankara G Iyer, and Wonpil Im. Charmm-gui: a web-based graphical user interface for charmm. Journal of Computational Chemistry, 29(11):1859–1865, 2008. Chitra Joshi, Justin C Deme, Yoshinobu Nakamura, Wei-Tse Hsu, Jonathan D Goult, Takafumi Kato, Joanne L Parker, Philip C Biggin, Susan M Lea, Takeo Nakanishi, et al. Structural basis for prostaglandin and drug transport via slco2a1. Nature Communications, 17(1):2285, 2026. Vinay Kumar, Satyendra Rajput, Mausam, and N. M. Anoop Krishnan. MDGYM: Benchmarking AI agents on molecular simulations, 2026. URL https://arxiv.org/abs/2605.08941. Greg Landrum et al. Rdkit documentation. Release, 1(1-79):4, 2013. 11 Yoonho Lee, Roshen Nair, Qizheng Zhang, Kangwook Lee, Omar Khattab, and Chelsea Finn. Meta- Harness: End-to-end optimization of model harnesses, 2026. URLhttps://arxiv.org/ abs/2603.28052. Jiahang Lin, Shichun Liu, Chengjun Pan, Lizhi Lin, Shihan Dou, Zhiheng Xi, Xuanjing Huang, Hang Yan, Zhenhua Han, Tao Gui, and Yu-Gang Jiang. Agentic harness engineering: Observability- driven automatic evolution of coding-agent harnesses, 2026a. URLhttps://arxiv.org/ abs/2604.25850. Minhua Lin, Juncheng Wu, Zijun Wang, Zhan Shi, Yisi Sang, Bing He, Zewen Liu, Tianxin Wei, Zongyu Wu, Zhiwei Zhang, Dakuo Wang, Xiang Zhang, Benoit Dumoulin, Cihang Xie, Yuyin Zhou, Suhang Wang, and Hanqing Lu. Harness updating is not harness benefit: Disentangling evolution capabilities in self-evolving LLM agents, 2026b. URLhttps://arxiv.org/abs/ 2605.30621. Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. DeepSeek-V3 technical report, 2024a. URL https://arxiv.org/abs/2412.19437. Junwei Liu, Kaixin Wang, Yixuan Chen, Xin Peng, Zhenpeng Chen, Lingming Zhang, and Yiling Lou. Large language model-based agents for software engineering: A survey. ACM Transactions on Software Engineering and Methodology, 2024b. Pascal T Merz and Michael R Shirts. Testing for physical validity in molecular simulations. PLOS ONE, 13(9):e0202764, 2018. Pascal T Merz, Wei-Tse Hsu, Matt W Thompson, Simon Boothroyd, Chris C Walker, and Michael R Shirts. physicalvalidation: A python package to assess the physical validity of molecular simula- tion results. Journal of Open Source Software, 7(69):3981, 2022. Antonia SJS Mey, Bryce K Allen, Hannah E Bruce Macdonald, John D Chodera, David F Hahn, Maximilian Kuhn, Julien Michel, David L Mobley, Levi N Naden, Samarjeet Prasad, et al. Best practices for alchemical free energy calculations [article v1. 0]. Living Journal of Computational Molecular Science, 2(1):18378, 2020. Naveen Michaud-Agrawal, Elizabeth J Denning, Thomas B Woolf, and Oliver Beckstein. Mdanalysis: a toolkit for the analysis of molecular dynamics simulations. Journal of Computational Chemistry, 32(10):2319–2327, 2011. David Moreau, Kristina Wiebels, and Carl Boettiger. Containers for computational reproducibility. Nature Reviews Methods Primers, 3:50, 2023. doi: 10.1038/s43586-023-00236-9. URLhttps: //doi.org/10.1038/s43586-023-00236-9. Frank Neese, Frank Wennmohs, Ute Becker, and Christoph Riplinger. The orca quantum chemistry program package. The Journal of Chemical Physics, 152(22), 2020. Noel M O’Boyle, Michael Banck, Craig A James, Chris Morley, Tim Vandermeersch, and Geoffrey R Hutchison. Open babel: An open chemical toolbox. Journal of Cheminformatics, 3(1):33, 2011. Thang D. Pham, Aditya Tanikanti, and Murat Kec ̧eli. ChemGraph as an agentic framework for computational chemistry workflows. Communications Chemistry, 9:33, 2026. doi: 10.1038/ s42004-025-01776-9. URL https://doi.org/10.1038/s42004-025-01776-9. Zhuofan Shi, Hubao A, Yufei Shao, Dongliang Huang, Hongxu An, Chunxiao Xin, Haiyang Shen, Zhenyu Wang, Yunshan Na, Gang Huang, and Xiang Jing. MDAgent2: Large language model for code generation and knowledge Q&A in molecular dynamics, 2026. URLhttps://arxiv. org/abs/2601.02075. Alan W Sousa da Silva and Wim F Vranken. Acpype-antechamber python parser interface. BMC Research Notes, 5(1):367, 2012. Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: A family of highly capable multimodal models, 2023. URL https://arxiv.org/abs/2312.11805. 12 Terminal-Bench Team. Terminal-Bench: A benchmark for AI agents in terminal environments, April 2025. URL https://github.com/harbor-framework/terminal-bench. Aidan P Thompson, H Metin Aktulga, Richard Berger, Dan S Bolintineanu, W Michael Brown, Paul S Crozier, Pieter J In’t Veld, Axel Kohlmeyer, Stan G Moore, Trung Dac Nguyen, et al. Lammps-a flexible simulation tool for particle-based materials modeling at the atomic, meso, and continuum scales. Computer Physics Communications, 271:108171, 2022. Gareth A Tribello, Massimiliano Bonomi, Davide Branduardi, Carlo Camilloni, and Giovanni Bussi. Plumed 2: New feathers for an old bird. Computer Physics Communications, 185(2):604–613, 2014. Shunzhou Wan, Robert C. Sinclair, and Peter V. Coveney. Uncertainty quantification in classical molecular dynamics. Philosophical Transactions of the Royal Society A, 379(2197):20200082, 2021. doi: 10.1098/rsta.2020.0082. URLhttps://doi.org/10.1098/rsta.2020. 0082. Xingyao Wang, Boxuan Li, Yufan Song, Frank F. Xu, Xiangru Tang, Mingchen Zhuge, Jiayi Pan, Yueqi Song, Bowen Li, Jaskirat Singh, Hoang Tran, Fuqiang Li, Ren Ma, Mingzhang Zheng, Bill Qian, Daniel Shao, Niklas Muennighoff, Yizhe Zhang, Binyuan Hui, Junyang Lin, Robert Brennan, Hao Peng, Heng Ji, and Graham Neubig. OpenHands: An open platform for AI software developers as generalist agents. In International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id=OJd3ayDDoF. Zhiyi Wu, David L Dotson, Irfan Alibay, Bryce K Allen, Mohammad Soroush Barhaghi, J ́ er ˆ ome H ́ enin, Thomas T Joseph, Ian M Kenney, Hyungro Lee, Haoxi Li, et al. alchemlyb: The simple alchemistry library. Journal of Open Source Software, 9(101):6934, 2024. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report, 2025. URLhttps://arxiv. org/abs/2505.09388. John Yang, Carlos E. Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik R. Narasimhan, and Ofir Press. SWE-agent: Agent-computer interfaces enable automated software engineering. In Advances in Neural Information Processing Systems, 2024. URLhttps://arxiv.org/ abs/2405.15793. Tianping Zhang, Zheyu Aqa Zhang, Zhiyuan Fan, Haoyan Luo, Fengyuan Liu, Qian Liu, Wei Cao, and Li Jian. Openfe: Automated feature generation with expert-level performance. In International Conference on Machine Learning, p. 41880–41901. PMLR, 2023. Alexander Zhao, Achuth Chandrasekhar, and Amir Barati Farimani. PolyJarvis: An LLM-orchestrated agent for automated all-atom molecular dynamics of amorphous homopolymers, 2026. URL https://arxiv.org/abs/2604.02537. Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, et al. A survey of large language models, 2023. URL https://arxiv.org/abs/2303.18223. AAPPENDIX A.1TASK METADATA TAGS Table A1 lists the task-level metadata used to summarize MDArena’s benchmark coverage. Package, system, and method cells contain semicolon-separated tags when a task belongs to multiple categories; blank package entries indicate that no explicit package/tool tag was assigned in the source package table. 13 Table A1: Task metadata tags for MDArena. TaskPackage tagsSystem tagsMethod tags 01similes2simGROMACS / gmxGeneric solvated ligand from SMILES System preparation / model building 02concdiffsys traj GROMACS / gmxChignolin; T4 lysozymeSimulation compatibility / sanity checks 03idbpresid–Adenosine A2A receptor / A2AR Structure analysis / validation 04trajanadist–Unspecified residue-distance trajectory system Trajectory analysis 05trajanahbonds MDAnalysisUnspecified protein-ligand H-bond trajectory Trajectory analysis 06trajana rotamers –p38 alphaTrajectory analysis 07 tyk2equilGROMACS / gmxTYK2System preparation / model building 08clashcheck–TYK2Structure analysis / validation; Simulation compatibility / sanity checks 09holo2apoGROMACS / gmxmGlu5 receptorSystem preparation / model building 10 trajanamg coord –Isocitrate lyase / ICL familyTrajectory analysis 11amb2gmxemGROMACS / gmxGlycosylated insulin / glycoinsulin System preparation / model building 12 xtalwatersMDAnalysisIsocitrate lyase / ICL familyStructure analysis / validation 13altlocclean–OxymyoglobinSystem preparation / model building 14trajanalig rmsd MDAnalysisSLCO2A1Trajectory analysis 15grompptcoupGROMACS / gmxGeneric GROMACS protein system Topology / force-field editing; Simulation compatibility / sanity checks 16gaffparamGROMACS / gmx; ACPYPEAla-Phe ligandLigand / residue parameterization 17prolifint–Isocitrate lyase / ICL familyTrajectory analysis 18scalechargeGROMACS / gmxWater-decane ion topologyTopology / force-field editing 19trajanaopen–TPC2 membrane proteinTrajectory analysis 20fepabfedhdl analysis alchemlybAdenosine A2A receptor / A2AR FEP / alchemical free-energy setup 21watertransfer ox2r MDAnalysisOX2R receptorStructure analysis / validation 22trajanapept2 pcacv suitability PLUMEDPepT2 / beta-lactam systemsCollective-variable analysis / PCA 23a2armembrane equilibration GROMACS / gmx; OpenMM; ACPYPE; Open Babel; PDBFixer Adenosine A2A receptor / A2AR System preparation / model building 24 ligand parameterization smirnoffopenfe OpenFF Toolkit / SMIRNOFF workflow; OpenMM; openmmforcefields; AmberTools; ParmEd Generic SMIRNOFF/OpenFF ligand parameterization Ligand / residue parameterization 25 mglu5membrane setup GROMACS / gmx; OpenFF Toolkit / SMIRNOFF workflow; OpenMM; openmmforcefields; AmberTools; ParmEd; PDBFixer mGlu5 receptorSystem preparation / model building; Ligand / residue parameterization 26d3receptor membranesetup GROMACS / gmx; MDAnalysis; OpenMM; ACPYPE; Open Babel; PDBFixer Dopamine D3 receptorSystem preparation / model building; Ligand / residue parameterization 27 a2ar6gt3md setup GROMACS / gmx; OpenMM; ACPYPE; Open Babel; PDBFixer Adenosine A2A receptor / A2AR System preparation / model building; Ligand / residue parameterization 28trajanapept2 pcaplumedsetup PLUMEDPepT2 / beta-lactam systemsCollective-variable analysis / PCA; Umbrella sampling setup 29pept2beta lactammembrane setup GROMACS / gmx; ACPYPE; Open Babel PepT2 / beta-lactam systemsSystem preparation / model building; Ligand / residue parameterization 30 camumbrella pore6nq0 GROMACS / gmx; MDAnalysis6NQ0 pore / ion-conductance system Umbrella sampling setup 31trajanana conductance6nq0 GROMACS / gmx6NQ0 pore / ion-conductance system Trajectory analysis 32 fepopenfe rbfedhodh OpenFE; OpenFF Toolkit / SMIRNOFF workflow; OpenMM; openmmtools; AmberTools; ParmEd; RDKit DHODHFEP / alchemical free-energy setup 33fepopenfe rbfed3 OpenFE; OpenFF Toolkit / SMIRNOFF workflow; OpenMM; openmmtools; AmberTools; ParmEd; RDKit Dopamine D3 receptorFEP / alchemical free-energy setup Continued on next page 14 Table A1: Task metadata tags for MDArena (continued). TaskPackage tagsSystem tagsMethod tags 34fepopenfe rbfedefox2s1 OpenFE; OpenFF Toolkit / SMIRNOFF workflow; OpenMM; openmmtools; AmberTools; ParmEd; RDKit OX2R receptorFEP / alchemical free-energy setup 35 fepopenfe abfed3 OpenFE; OpenFF Toolkit / SMIRNOFF workflow; OpenMM; openmmtools; AmberTools; ParmEd; RDKit Dopamine D3 receptorFEP / alchemical free-energy setup 36 fepopenfe abfemglu5r OpenFE; OpenFF Toolkit / SMIRNOFF workflow; OpenMM; openmmtools; AmberTools; ParmEd; RDKit mGlu5 receptorFEP / alchemical free-energy setup 37fepopenfe abfea2ar MDAnalysis; OpenFE; OpenFF Toolkit / SMIRNOFF workflow; OpenMM; openmmtools; AmberTools; ParmEd; RDKit Adenosine A2A receptor / A2AR FEP / alchemical free-energy setup 38pept2qm clusterorca MDAnalysis; ORCA; MPI / OpenMPI / mpi4py PepT2 / beta-lactam systemsQM / quantum chemistry setup 39 fepgmxd3abfe GROMACS / gmx; MDAnalysis; MDRestraintsGenerator Dopamine D3 receptorFEP / alchemical free-energy setup; Boresch restraints 40 trajanameng samrmsd –MenG membrane/SAM systemTrajectory analysis 41rexeeargon hydrationsetup GROMACS / gmx; ensemble md; MPI / OpenMPI / mpi4py Argon hydrationREXEE 42expanded ensemblemdp GROMACS / gmx Generic expanded-ensemble MDP system Topology / force-field editing; Expanded ensemble / Wang-Landau 43 cb710weight updatingee GROMACS / gmxCB7-10 host-guest complexExpanded ensemble / Wang-Landau 44trajanaatp8b rmsd MDAnalysisATP8B1-CDC50A membrane complex Trajectory analysis 45 atp8b insertiondepth –ATP8B1-CDC50A membrane complex Trajectory analysis 46mengdmkff patch GROMACS / gmxMenG membrane/SAM systemTopology / force-field editing 47 d2presidue parameterization GROMACS / gmx Dianionic phosphoaspartate / D2P pentapeptide Ligand / residue parameterization 48 cb710 alchemical metadynamics GROMACS / gmx; PLUMEDCB7-10 host-guest complexAlchemical metadynamics 49 slco2a1 physical validation GROMACS / gmx; physicalvalidation SLCO2A1Structure analysis / validation; Simulation compatibility / sanity checks 50glycoinsulin betasheet propensity MDAnalysis; MatplotlibGlycosylated insulin / glycoinsulin Trajectory analysis A.2ADDITIONAL OUTCOME BREAKDOWN A.3CATEGORY-WISE CHECK SCORES 15 Figure A1: Task outcomes by model. Stacked bars show the percentage of MDArena tasks ending in a full pass, failed evaluation, agent timeout, or other runtime error. 16 Figure A2: Category-wise correctness and process rewards. Average check-level reward by task category for each model/harness configuration, separated into (a) correctness checks and (b) process checks. 17