Paper deep dive

Evaluating Verified Autonomy in Quantum Engineering

Naixu Guo, Changhao Li, Siyu Cheng, Qicheng Tang, Binzhao Luo, Bikun Li, Yuxuan Du, Shihao Ru, Jiaqi Cai

Year: 2026Venue: arXiv preprintArea: quant-phType: PreprintEmbeddings: 51

Intelligence

Status: not_run | Model: - | Prompt: - | Confidence: 0%

Entities (0)

No extracted entities yet.

Relation Signals (0)

No relation signals yet.

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Reliable quantum engineering is essential for turning quantum phenomena into practical technologies. As quantum platforms grow in scale and complexity, their characterization and operation require increasing human effort and coordination. Scientific artificial intelligence agents, which can plan experiments, operate instruments, and analyze observations, offer a promising route towards autonomous quantum engineering. Yet whether current agents can perform reliably in this setting has not been systematically established. To fill this gap, we developed Quantum-Harbor, a virtual laboratory that provides a controlled execution environment for agents to interact with quantum systems. This design enables direct verification of both the actions taken and the conclusions drawn. Building on this framework, we introduce QIQCBench, a benchmark of $49$ expert-authored tasks spanning multiple layers including calibration and control, error correction and compilation, sensing and networking. Across $17$ frontier agentic systems, QIQCBench reveals wide variation in verified performance. These results expose a substantial gap between demonstrating capability and achieving reliable operation, and establish Quantum-Harbor as a foundation for measuring progress towards verified autonomy in quantum engineering.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

47,748 characters extracted from source content.

Expand or collapse full text

Evaluating Verified Autonomy in Quantum Engineering Naixu Guo ∗,† , 1 Changhao Li ∗,† , 2 Siyu Cheng ∗ , 3 Qicheng Tang ∗ , 4 Binzhao Luo, 5 Bikun Li, 6 Yuxuan Du, 7 Shihao Ru † , 8 and Jiaqi Cai †5 1 Centre for Quantum Technologies, National University of Singapore, 117543, Singapore 2 Unitary Foundation, San Francisco, CA, USA 3 Department of Physics, Boston College, Chestnut Hill, Massachusetts 02467, USA 4 School of Physics, Georgia Institute of Technology, Atlanta, GA 30332, USA 5 GaugeForge PTE. LTD., 049422, Singapore 6 Chicago Quantum Institute and Pritzker School of Molecular Engineering, University of Chicago, Chicago, Illinois 60637, USA 7 College of Computing and Data Science, Nanyang Technological University, Singapore 639798, Singapore 8 School of Electrical and Electronic Engineering, Nanyang Technological University, Singapore 639798, Singapore (Dated: September 16, 2026) Reliable quantum engineering is essential for turning quantum phenomena into practical tech- nologies. As quantum platforms grow in scale and complexity, their characterization and operation require increasing human effort and coordination. Scientific artificial intelligence agents, which can plan experiments, operate instruments, and analyze observations, offer a promising route towards autonomous quantum engineering. Yet whether current agents can perform reliably in this setting has not been systematically established. To fill this gap, we developed Quantum-Harbor, a virtual laboratory that provides a controlled execution environment for agents to interact with quantum systems. This design enables direct verification of both the actions taken and the conclusions drawn. Building on this framework, we introduce QIQCBench, a benchmark of 49 expert-authored tasks spanning multiple layers including calibration and control, error correction and compilation, sensing and networking. Across 17 frontier agentic systems, QIQCBench reveals wide variation in veri- fied performance. These results expose a substantial gap between demonstrating capability and achieving reliable operation, and establish Quantum-Harbor as a foundation for measuring progress towards verified autonomy in quantum engineering. INTRODUCTION Quantum technologies, spanning computing, simula- tion, communication, and sensing [1–4], are moving from proof-of-principle demonstrations toward practical appli- cations [5–16]. Quantum engineering enables this transi- tion by integrating quantum platforms and control meth- ods as well as software into reliable systems. Building and operating quantum platforms has traditionally relied heavily on human experts, who assess device information, interpret experimental results, and make the necessary adjustments [17]. As these platforms scale, the number of components, parameters, and workflows that must be coordinated and optimized grows, making manual diag- nosis increasingly impractical [18, 19]. Several individual engineering tasks have therefore been automated [20–22]. Scientific artificial intelligence (AI) agents offer a way to extend this automation to longer workflows [17, 23]. These agents can plan actions, use experimental and computational tools, and adapt their strategies in re- sponse to feedback [24]. Recent efforts have begun to ∗ These authors contributed equally to this work. † Corresponding authors: naixug@u.nus.edu (N.G.), chang- haoli96@gmail.com (C.L.), shihao.ru.2018@gmail.com (S.R.), jiaqi.cai@gauge-forge.com (J.C.) apply such agents to quantum experiments and software workflows [25–33]. Yet whether current agents can reli- ably carry out complete quantum engineering workflows remains unclear. Evaluating this capability requires more than check- ing whether an agent produces the correct answer or completes a predefined task. In practical quantum engi- neering, critical information often becomes available only during operation [34, 35]. Because device information may be only partially known, an agent must choose and execute experiments or computations, interpret the re- sulting observations, and decide how to proceed [36, 37]. Even with the relevant scientific knowledge, an agent may rely on information it did not measure, misinterpret ex- perimental data, or produce a result that fails on fresh data [38]. We therefore focus on verified autonomy, the ability of an agent to complete an engineering task and support its conclusions with evidence generated during execution. Outputs that cannot be verified directly from recorded data, such as calibrations or designs, must in- stead be tested under fresh hidden conditions. Existing benchmarks typically assess final outputs without sys- tematically checking whether those results are supported by the agent’s execution record [39, 40]. To make verified autonomy measurable, we de- velop Quantum-Harbor, a virtual quantum laboratory that provides experimental and computational interfaces arXiv:2609.17439v1 [quant-ph] 15 Sep 2026 2 a quantum platforms quantum simulators superconducting circuits trapped ions neutral atoms photonic systems, solid-state spins, quantum networks, quantum sensors, ... submit Independent verification quantum engineering applications & algorithms logical control & compilation error correction gates & calibration device characterization b 114-qubit chipseam correlationsretuned decoder verified on fresh syndromes logical memorylogical cnot error < 10 -3 error < 0.175 c 60-qubit circuitdata collectionsurrogate prediction evaluation on 40 held-out inputs predictionsreference (hidden) compare 40 inputs × 1,770 pairs | RMSE ≤ 0.0025 q 0 q 1 q 2 p 0 p 1 p 2 HHR z (θ) R x (φ) R z (θ) R x (φ) 퐹 휃 휃 S f t |ψ C ⟩ |+⟩ |ψ T ⟩ M Z M X Z X = 푅 푥 (휃 1 ) 푅 푦 (휃 2 ) 푅 푥 (휃 3 ) 푅 푦 (휃 4 ) 푅 푥 (휃 5 ) 푅 푦 (휃 6 ) basesbitstrings XZY... 0101100101... YXX... 1100101010... ZYZ... 0010011101... ⋮ inputθ reference (hidden) surrogate prediction θ 2 θ 1 predictionsreference (hidden) plan run analyze adapt AI agent FIG. 1. AI agents for quantum engineering and representative workflows. a. Agent-operated quantum engineering. The agent iterates planning, execution, analysis, and adaptation while interacting with quantum laboratories. The workflow operates across the layers shown on the right, with individual tasks typically engaging multiple layers, and its outputs are submitted for independent verification. b. Decoder calibration for a lattice-surgery logical CNOT gate. The agent measures error correlations across the merge seam between surface-code patches and submits a retuned decoder, verified on fresh syndrome records. The public acceptance criteria require a distance-5 logical-memory error below 10 −3 per cycle and, for eight logical-CNOT test circuits spanning both measurement bases, a mean error below 0.175 with each circuit-specific error below 0.185. c. Surrogate modeling of a hidden 60-qubit parameterized quantum circuit. The agent collects measurement data and constructs a surrogate model. Once the measurement record is fixed, 40 held-out inputs are revealed, for which the agent predicts 1,770 two-qubit correlations for each input. The predictions are compared with hidden reference correlations, with accuracy evaluated by the root mean square error. while keeping device properties and grading ground truth hidden. For each task, the agent receives only a high-level objective and a finite action budget. The agent must decide how to interact with the environment, conduct experiments or computations, and submit a final result. Quantum-Harbor records the agent’s actions and the re- sulting data and uses this record to verify the submitted result. The grader verifies numerical results by recom- puting them from raw experimental data, while for cal- ibration and design outputs, it retests them under fresh hidden conditions. Building on this environment, we con- struct a benchmark comprising 49 expert-designed quan- tum engineering tasks covering hardware and software workflows at multiple architectural layers. We use this benchmark to systematically evaluate verified autonomy in frontier agentic AI systems. RESULTS Representative quantum research tasks Agent-operated quantum engineering places the AI agent in direct interaction with quantum laborato- ries, closing the loop between experiments and analy- sis (Fig. 1a). The laboratory may host quantum sim- ulators or quantum platforms such as superconducting circuits, trapped ions, neutral atoms and photonic sys- tems. During execution, the agent chooses experimental or computational actions, interprets the resulting data, and revises its plan as new information is acquired. The workflow therefore evolves adaptively as execution pro- gresses. A single workflow may span multiple layers of quantum engineering, from device characterization and 3 gate calibration to error correction, compilation, and ap- plications. The resulting execution record provides the evidence needed to assess and verify the agent’s outputs. We now examine two representative quantum research tasks that require distinct agent capabilities (Fig. 1b,c). The first task focuses on the logical CNOT gate which is a basic building block of fault-tolerant quantum com- putation. Lattice surgery realizes this gate by merging and splitting error-correcting code patches [41, 42]. Re- cent experiments have demonstrated this approach in trapped-ion [43] and superconducting processors [44, 45]. However, tuning the decoder to account for correlated noise not captured by hardware datasheets remains an expert-driven process [5, 46]. In this task, the agent op- erates a simulated 114-qubit transmon grid containing three distance-3 surface-code patches, a routing region used to fuse them during surgery, and a distance-5 mem- ory patch. It submits decoder configurations that the hidden grader replays on fresh syndrome data. A success- ful configuration must clear two criteria that bound the per-cycle error of the distance-5 memory and the logical- CNOT error across test circuits in both measurement bases, as shown in Fig. 1b. Both criteria follow from a hidden noise model that is set at the physical error rates of current below-threshold processors [5]. The task also embeds a failure mode taken from real laboratories. The lab notebook records earlier memory measurements on separate patches and recommends reusing the resulting decoder calibration for the logical operation. Merging the patches introduces correlated errors along their shared boundary, which are not visible in those earlier measure- ments, so the calibrated decoder can fail during surgery. The strongest agents instead measure the merged con- figuration directly and clear both criteria, with the best run reaching a memory error of 6.65× 10 −4 per cycle and a mean logical-CNOT error of 0.131 on fresh syndrome records. The surrogate model task [47–49] requires the agent to design informative measurement protocols and con- struct a predictive model of an unknown quantum sys- tem (Fig. 1c). The agent interacts with a 60-qubit circuit U (θ) containing Clifford and non-Clifford gates, with 18 non-Clifford rotation gates controlled by six independent parameters. Within a fixed experimental budget, it de- termines how to sample the parameter space and allocate shots across measurement bases. After the measurement phase ends, 40 previously hidden inputs are revealed, for which the agent must predict 1,770 pairwise correlations without further measurements. The predictions are com- pared with reference values generated from the same cir- cuit and readout noise model. Success therefore requires learning from limited measurements and generalizing to unseen parameter settings. Failed runs often expend sub- stantial experimental resources but fall short because of uninformative sampling, unreliable validation or incom- plete execution. Many of these runs progress through much of the workflow, while consistent end-to-end com- pletion remains challenging. These examples illustrate both the emerging capabili- ties and current limitations of agentic quantum engineer- ing. The lattice surgery task requires identifying noise that emerges only in the merged configuration, whereas the surrogate model task requires learning from limited measurements to predict unseen inputs. In both cases, success depends on information acquired during execu- tion. Evaluating such workflows therefore requires pre- serving the interaction record and, where conclusions cannot be checked directly, testing them on fresh data or unseen inputs. These requirements motivate a con- trolled environment for systematic evaluation. The Quantum-Harbor framework Quantum-Harbor provides the large language model (LLM) agent with a sandbox environment to interact with quantum hardware and complete quantum research tasks (Fig. 2).It builds on Harbor [50], a general framework for sandboxed agent tasks such as terminal shell use and software development [51, 52]. Quantum- Harbor adds three features to Harbor: a laboratory in- terface for quantum experiments, interchangeable quan- tum hardware backends, and evidence-bound grading. Each Quantum-Harbor run launches two isolated envi- ronments, namely the agent environment and the sidecar. The agent environment (Fig. 2a) runs the language model harness such as Codex and Claude Code, and stores the workfiles used by the agent. The agent reads the working files and writes code to interact with the public surface exposed in the agent environment. The public surface is a set of tools exposed via the Model Context Protocol (MCP) [53], a published standard that lets a language model call an external service the way a program calls a function. Through them the agent can read the hard- ware spec, write the lab notebook whose calibration en- tries may be outdated, and submit experiment jobs, and its final answer. The agent can also carry out the exper- iment through the Qiskit [54] or QCoDeS [55] libraries that a human experimentalist would use. Every experi- ment submission returns a job handle to the agent and completes asynchronously, and the job results are raw records, such as readout points or per-shot bitstrings. Behind the public surface is the sidecar (Fig. 2b,c). The agent can only access and interact with the public surface, and the sidecar is hidden and isolated from the agent. The sidecar hosts the detailed implementation of the quantum hardware or simulated hardware. To sup- port a variety of quantum platforms, the sidecar contains different quantum hardware types called qtypes (Fig. 2b) to accommodate the variety among different quantum hardware. A qtype abstracts the physics of one class of experimental platform, such as a multilevel transmon un- 4 abc agent environment public surfaces codes LLM work files MCP tools hardware spec lab notebook submit answer jobs 01001101 bitstrings hardware / simulated hardware qtypes transmontrapped ion neutral atom... 33 more qtypes simulator hidden parameters noise seeds replay / live device evidence log grader 1 recompute 2 bind 3 score pass / fail FIG. 2. Architecture and evidence-bound grading in Quantum-Harbor. a. The agent environment contains the language model, work files, and agent-written code. Through the public interfaces, the agent consults hardware specifications and lab notebooks, submits experiment jobs, retrieves raw measurement records (bitstrings shown), and submits its final answer. Experiment jobs execute asynchronously. b. The hardware layer provides quantum hardware abstractions (qtypes) for transmons, neutral atoms, trapped ions, and other platforms. Experiments run on simulators governed by hidden device and noise parameters, or on replay and live-device backends where supported. An evidence log records experimental requests and outcomes for subsequent verification. c. The grader recomputes relevant quantities, binds submitted claims to the recorded experiments, and applies task-specific scoring criteria (pass/fail shown). For calibration tasks, submitted configurations can be evaluated on fresh simulated data. The shaded region and dashed boundary indicate components inaccessible to the agent, including hidden parameters, the private evidence log, and grading logic. der pulse control, a trapped-ion chain, or a neutral-atom array. Qtypes can simulate quantum hardware, with pre- defined hidden parameters such as the coherence times, anharmonicities, readout confusion matrices, noise corre- lations, and random seeds (see Supplementary Material for details about the implementation of simulation). It also allows live connection to real quantum hardware, such as superconducting systems from IBM, Rigetti, and ion trap systems from IonQ and Quantinuum. The re- play backend is the dry-run mode for the live quantum hardware and runs on recorded data. The simulator, real quantum hardware, and the replay backend are inter- changeable. Switching among them within one qtype does not require any change to task definitions or public surfaces. The evidence log inside the sidecar records every exper- iment job executed on either simulator or real quantum hardware. An LLM might report a number it never mea- sured [56, 57]. Therefore, the grader (Fig. 2c) is used to evaluate LLMs’ performance on quantum research tasks and is bound to the evidence log. The grader verifies the task result by recomputing it from the raw data in the evidence log. When the submission is a calibration and not a number, the grader replays it against fresh draws from the hidden parameters. This prevents the agent from being rewarded for submitting false results. The QIQCBench benchmark Building on the Quantum-Harbor framework, we next introduce QIQCBench, which comprises 49 tasks built on 36 qtypes, all contributed and reviewed by domain experts.Their expertise spans the working breadth of quantum engineering. Fig. 3a shows the composi- tion of the suite, with tasks organized into six cate- gories. Representative measurement and characteriza- tion tasks include estimating the relaxation time of a drifted transmon and reconstructing correlated gate er- rors [34, 59], while quantum control tasks range from pulse design to Hamiltonian engineering [37, 60]. Er- ror correction and mitigation tasks include problems such as decoder calibration and logical-memory protec- tion [5, 46], and compilation and resource-optimization tasks map target computations to trapped-ion, neutral- atom, and superconducting architectures [18, 21]. Tasks in simulation and many-body physics include inferring hidden Hamiltonians and open-system dynamics [22, 61], while communication and sensing covers, for example, entanglement purification across repeater arrays [62] and GHZ-assisted frequency estimation under non-Markovian noise [63]. The tasks also cover the full quantum comput- ing stack [58], as shown in the five layers in Fig. 3b. In our benchmark, the agent needs to reason across different layers. As shown in Fig. 3, the tasks run on simulated hard- ware. Several tasks require no specific device and are 5 a measurement & characterization quantum control error correction & mitigation compilation & resource optimization simulation & many-body physics communication & sensing task index platform evaluation type 12345678910111213141516171819202122232425262728293031323334353637383940414243444546474849 btask-layer mapping mapped layernot mapped application & algorithms logical & compilation error correction gates & calibration physical device platformevaluation type superconducting neutral atom trapped ion spin/NV, semiconductor & bosonic platform-agnostic device-less binary verified-science gate racing (Elo-rated) test case, excluded from science scores FIG. 3. Task composition and quantum computing stack coverage of QIQCBench. a. The 49 tasks grouped into six scientific categories. Each column represents one task. Colors identify the hardware platform, with separate labels for platform-agnostic and device-less tasks. Evaluation types are indicated by filled squares for the 39 tasks assessed by a binary pass–fail evaluation, orange diamonds for the 7 racing tasks ranked by Elo rating, and open squares for the 3 infrastructure test cases excluded from science scores. b. Mapping of each task to five layers of the quantum computing stack, adapted from Ref. [58]: physical device, gates and calibration, error correction, logical and compilation, and application and algorithms. Dark blue cells mark the layers engaged by each task. Pale cells indicate unmapped layers. Columns share the task indices and category boundaries of panel a, showing how individual tasks connect multiple layers of quantum engineering. marked device-less. Orthogonally, each task is rated in one of two modes: i) pass–fail, where a hidden verifier returns a Boolean pass or fail, or i) racing, where sub- missions are ranked by a Bradley–Terry Elo fit (see Meth- ods). Finally, three deliberately simple test cases (e.g., measuring the T 1 of a transmon qubit) stress-test the benchmark infrastructure and are excluded from all sci- ence scores. We further note that per-task hidden param- eters and pass thresholds stay out of all published ma- terial, so future systems cannot learn the answers from this paper. We established three principles during the develop- ment of this benchmark. First, every task is designed as an end-to-end laboratory mini-project whose deliverable is a significant final result, such as a physical quantity, a calibrated artifact, or an optimized design. Second, we require that a task’s difficulty come from the physics and not from obscurity. Each task has an intended solution path that a competent experimentalist can follow, and a task is hard only when that path turns on a nonobvious inference. Third, to increase the difficulty and mimic the laboratory setting, contributors tend to design traps (or tricks) that can mislead the model. However, we require that every trap be honest, of a kind that arises in realis- tic experimental environments. For example, stale note- books are labeled as notebooks and budgets are stated explicitly. The agent is not punished for missing infor- mation due to our infrastructure. Benchmark results We evaluated seventeen agentic systems from eight vendors, three from OpenAI (GPT-5.6-sol, GPT- 5.6-terra, and GPT-5.6-luna), three from DeepSeek (DeepSeek-V4.1-Flash, V4-Pro, and V4-Flash, hereafter DS-V4.1-Flash and so on), three from Z.ai (GLM-5.3, 5.3- Flash, and 5.2), two from Anthropic (Claude Opus 4.8 and Haiku 4.5), two from Alibaba (Qwen3.8-Max, 3.8- Flash), two from Tencent (Hunyuan T3 and 4 preview, denoted by Hunyuan T4*), one from Moonshot (Kimi K3), and one from ByteDance (Seed 2.1 Pro). The two Anthropic systems ran under the Claude Code agent har- ness and the three OpenAI systems under the Codex CLI harness. The remaining twelve systems ran under the OpenCode harness. Every system ran at its extra high (xhigh) or higher reasoning-effort setting and attempted each task three times independently. Fig. 4a ranks the seventeen systems by pass rate on the pass–fail tasks. Pass rates run from 78% down to 3% with performance decreasing steadily across the ranked systems and no clear separation into groups. We note that systems scoring within a narrow band on a frontier agentic benchmark spread across almost the whole range of QIQCBench. For example, on AutomationBench-A, the independent run of Zapier’s AutomationBench [64] carried out by Artificial Analysis [65], eight of our sev- enteen systems score between 54.0% and 62.2%. For 6 abc end-to-end tasksall 7 racesper task 0255075100 GPT-5.6-sol GPT-5.6-terra Opus 4.8 GPT-5.6-luna Hunyuan T4* DS-V4.1-Flash Hunyuan T3 DS-V4-Flash DS-V4-Pro Kimi K3 Qwen3.8-Flash GLM-5.3 GLM-5.3-Flash Haiku 4.5 Qwen3.8-Max Seed 2.1 Pro GLM-5.2 pass rate (%) 0255075100 GPT-5.6-sol GPT-5.6-terra Opus 4.8 GPT-5.6-luna Hunyuan T4* DS-V4.1-Flash Hunyuan T3 DS-V4-Flash DS-V4-Pro Kimi K3 Qwen3.8-Flash GLM-5.3 GLM-5.3-Flash Haiku 4.5 Qwen3.8-Max Seed 2.1 Pro GLM-5.2 trials (%) 500100015002000 GPT-5.6-sol GPT-5.6-terra Opus 4.8 GPT-5.6-luna Hunyuan T4* DS-V4.1-Flash Hunyuan T3 DS-V4-Flash DS-V4-Pro Kimi K3 Qwen3.8-Flash GLM-5.3 GLM-5.3-Flash Haiku 4.5 Qwen3.8-Max Seed 2.1 Pro GLM-5.2 Elo rating 78 57 51 46 36 21 21 15 15 14 12 9 8 7 7 7 3 d measurement & characterization 45678 quantum control 9101112131415161718 error correction & mitigation 192021222324272829 compilation & resource optimization 33363738 simulation & many-body physics 39404142434445 communication & sensing 46474849 GPT-5.6-sol GPT-5.6-terra Opus 4.8 GPT-5.6-luna Hunyuan T4* DS-V4.1-Flash Hunyuan T3 DS-V4-Flash DS-V4-Pro Kimi K3 Qwen3.8-Flash GLM-5.3 GLM-5.3-Flash Haiku 4.5 Qwen3.8-Max Seed 2.1 Pro GLM-5.2 3 2 1 0 passscienceno submission contractunclassified passes (of 3) FIG. 4. Benchmark results and analysis of failure and pass patterns. a. Pass rate on the pass–fail tasks. Error bars: Wilson 68% intervals. b. Outcome composition of the trials, including passes, science failures, contract failures, and no-submission trials. c. Bradley–Terry Elo ratings over the 7 racing tasks (filled circles: games-weighted aggregate; open circles: per-task ratings). d. Per-task passes for every system, grouped by the scientific categories of Fig. 3, easiest to hardest within each group. Only pass-fail tasks are shown here. GDPval-A v2 [66], a widely adopted evaluation, six of them sit within 31 Elo points of one another. We conclude that our benchmark can separate systems that perform alike on other frontier agentic benchmarks. We classify failures into three categories (Fig. 4b), in- cluding no submission when no answer arrives, science failure when the hidden grader rejects the physics, and contract failure when the answer breaks the submission protocol. These categories correspond to three require- ments of autonomous experimentation: completing the task, reaching a scientifically valid conclusion, and re- turning that conclusion in a form that can be verified. No submission is the largest failure category, accounting for 937 of 1,510 failures. Seven systems exhaust their budget before submitting an answer in more than 60% of trials. These trials are, for the most part, neither crashes nor idle hangs. When timed out, the agents are often still running experiments. Among the remaining failures, 54% are science failures and the rest are contract failures. Across the pool, many failures arise from how agents conduct experiments, including relying on stale priors, mismanaging budgets, and discarding in-hand re- 7 sults at submission time. The racing tasks score performance on a continuous metric, such as required resources or run time (Fig. 4c). Every trial races pairwise against the trials of every other system. A trial that clears the task-specific feasibility gate always ranks above one that does not. Among gate- passing trials, the ranking is determined by the recom- puted metric (see Methods). Of the 357 race trials, 125 cleared their gates. A Bradley–Terry fit spreads the field from 1694 down to 582, with the anchor fixed at 1000. The ordering on the racing tasks corroborates that on the pass–fail tasks, with the same top three systems, but produces wider margins by incorporating design quality alongside reliability. We examine the pass pattern in Fig. 4d. Thirty-seven of the 39 tasks were solved by at least one system. For the two exceptions, a surrogate modeling on 60 qubits and a net-zero tunable-coupler CZ gate design, expert re- viewers confirmed the intended solution paths. Six tasks were solved only by the three highest-ranked systems. Per-task ratings, gate outcomes, the failure-category def- initions, and the audit protocols are tabulated in the Sup- plementary Material. We observe several interesting patterns in the bench- mark result. We can observe from Fig. 4d that simu- lation and many-body physics is the hardest category for the pool, with a passing rate of only 15%. The top three systems pass 64% of its trials and the other fourteen pass only 4%, also the largest such gap of any category. We found no significant correlation between difficulty and stack depth, as the pass rate is uncorre- lated with the number of layers a task spans (Spearman ρ = 0.14, p = 0.40, see Supplementary Material) and both unsolved tasks sit in the bottom two layers, gates & calibration and physical device. Success is intermit- tent, with only 34% of the 249 system and task pairs that pass at least once passing all three attempts. DISCUSSION The present benchmark results show that current agents exhibit meaningful but uneven capability on de- manding quantum engineering tasks. Large performance gaps between systems and substantial variability across repeated attempts indicate that reliable completion re- mains a central challenge for autonomous quantum en- gineering. The failure patterns point to experimental judgment as an important direction for improvement. Choosing informative measurements, revising stale as- sumptions, and deciding when to stop are recurring de- mands when agents must reach supported conclusions un- der finite resources. These failures are not explained by limitations in scientific knowledge or laboratory execu- tion alone, suggesting that improvements to agent har- nesses could substantially increase reliability. Quantum-Harbor makes the verified autonomy of AI agents measurable by evaluating not only their outputs but also whether those outputs are supported by infor- mation acquired during execution. By placing agentic systems in controlled quantum environments, it provides a framework for examining how agents gather evidence, revise hypotheses, allocate experimental resources and arrive at verifiable conclusions. QIQCBench realizes this framework across a broad range of cross-layer quantum engineering tasks, revealing differences between agentic systems that are not apparent from domain-knowledge and coding benchmarks alone. Beyond the present benchmark, Quantum-Harbor could support controlled studies of why agents succeed or fail. Task conditions, operating instructions and resource constraints could be varied systematically to disentangle the effects of model capability, harness design and scien- tific decision-making. Such studies could track progress beyond aggregate pass rates and guide improvements in the reliability and efficiency of autonomous quantum en- gineering. We note that the present results are limited to sim- ulated environments and characterize complete agentic systems operating through different harnesses under fixed resource budgets. Future work should determine how these findings transfer across experimental conditions and physical devices, and whether increasingly capable agents can reduce the measurement cost and human ef- fort required for quantum engineering. METHODS Task construction and review. Each task follows a com- mon construction and review pipeline. A domain expert prepares a task-design brief specifying the scientific ob- jective, the intended solution approach, and the task- specific pitfalls. A maintainer then builds the task on the Quantum-Harbor runtime, implementing the public materials, the hidden configuration, and the grader as separate artifacts, with the runtime enforcing the public– hidden boundary (Supplemental Material). Before the task is included in the suite, a second expert reviews the package. We verify the task end-to-end on Quantum- Harbor and use coding agents (Claude Code or Open- Code with Fable and Opus 5) to audit selected agents’ outcomes as a sanity check. Evaluation protocol. Each trial runs the agent con- tainer and the hardware-sidecar container. After the agent’s session ends, the sidecar container grades the run. The per-task time budget is 3 hours for all tasks. The agent container has internet access, allowing the system to consult public sources during the run. All tasks ran against the simulation backend. We used coding agents to audit every trial’s complete action record. Trials af- fected by confirmed infrastructure faults were quaran- 8 tined and rerun afterward. Failure categories. We distinguish three failure cat- egories for pass-fail tasks, as stated in the main text. The science failure occurs when an admissible submis- sion fails the task’s scientific checks. Depending on the grading contract, the hidden grader checks the submis- sion against hidden ground truth, recomputes reported results from logged evidence, or evaluates the submission in a replay on fresh data. The contract failure occurs when a submission is rejected at an admissibility gate be- fore its scientific content is scored, owing to a schema vi- olation, missing mandatory evidence, or delivery outside the published submission channel. The no-submission trial ends without a final answer being delivered. Cat- egories were assigned from each failing trial’s complete action record, by expert transcript audit or directly from the trial’s scorer and termination records, with the source recorded per trial. Rating of racing tasks.Each racing task runs on a fixed instance, so the grader-recomputed race met- rics of different systems’ submissions are directly com- parable. Every trial is compared pairwise against ev- ery trial of every other system on the task. A trial that passes the task’s feasibility gate beats a gate-failing trial. Two gate-passing trials are ordered by the recom- puted race metric. Exact ties and pairs of gate fail- ures are excluded from the fit. Thus, a gate-failing trial contributes a loss against each gate-passing trial of an- other system, but gate failures are not ranked among themselves. For the retained comparisons, the Bradley– Terry model assigns each system i a latent strength θ i and models the probability that system i beats system j as σ(θ i − θ j ), where σ(x) = 1/(1 + e −x ) is the lo- gistic function. Let w ij denote the number of retained comparisons won by system i against system j. For each task, we fit a Bradley–Terry model by maximizing P i,j w ij lnσ(θ i − θ j )− λ 2 ∥θ∥ 2 . The ridge penalty, with λ = 0.5, keeps the fitted strengths finite even for systems with only wins or only losses. We report strengths on the Elo scale, Elo i = 1000 + (400/ ln 10)(θ i − ̄ θ), where ̄ θ is the mean fitted strength across systems rated on that task. This scaling gives each task a mean rating of 1000. A 400-point rating advantage corresponds to model-implied win odds of 10 : 1. A system’s aggregate rating is the weighted mean of its per-task ratings, with each task weighted by that system’s number of retained comparisons (wins plus losses). Tasks on which the sys- tem has no retained comparisons are excluded from its aggregate. The rating combines how often a system pro- duces a feasible design with the quality of its feasible designs. Statistics. A system’s pass rate in Fig. 4a is k/n, the number k of passing trials over its n = 117 pass–fail trials (39 tasks, three repetitions each). The error bars are Wilson score intervals, k + z 2 /2 n + z 2 ± z n + z 2 r k(n− k) n + z 2 4 , with z = 1, corresponding to a nominal confidence level of approximately 68%. These intervals use a pooled bi- nomial approximation and do not quantify uncertainty due to the choice of tasks. Acknowledgement The authors thank Feng Pan, Zhan Yu, Yunlong Xiao and Ikko Hamamura for helpful discussions. [1] F. Arute et al., Quantum supremacy using a pro- grammable superconducting processor, Nature 574, 505 (2019). [2] A. J. Daley, I. Bloch, C. Kokail, S. Flannigan, N. Pearson, M. Troyer, and P. Zoller, Practical quantum advantage in quantum simulation, Nature 607, 667 (2022). [3] K. Azuma, S. E. Economou, D. Elkouss, P. Hilaire, L. Jiang, H.-K. Lo, and I. Tzitrin, Quantum repeaters: From quantum networks to the quantum internet, Rev. Mod. Phys. 95, 045006 (2023). [4] C. L. Degen, F. Reinhard, and P. Cappellaro, Quantum sensing, Rev. Mod. Phys. 89, 035002 (2017). [5] Google Quantum AI and Collaborators, Quantum error correction below the surface code threshold, Nature 638, 920 (2025). [6] D. Bluvstein et al., Logical quantum processor based on reconfigurable atom arrays, Nature 626, 58 (2024). [7] A. Paetznick et al., Improved quantum processor logical error rates via correction and detection, Nature 654, 349 (2026). [8] Y. Kim et al., Evidence for the utility of quantum com- puting before fault tolerance, Nature 618, 500 (2023). [9] Google Quantum AI and Collaborators, Observation of constructive interference at the edge of quantum ergod- icity, Nature 646, 825 (2025). [10] A. L. Shaw, Z. Chen, J. Choi, D. K. Mark, P. Scholl, R. Finkelstein, A. Elben, S. Choi, and M. Endres, Bench- marking highly entangled states on a 60-atom analogue quantum simulator, Nature 628, 71 (2024). [11] W.-Z. Liu, Y.-B. Zhou, J.-P. Chen, B. Wang, A. Teng, X.-W. Han, G.-C. Liu, Z.-J. Zhang, Y. Yang, F.-G. Liu, C.-H. Xue, B.-W. Yang, J. Yang, C. Zeng, D.-R. Pan, M.-Y. Zheng, X. Zhang, S. Cao, Y.-Z. Zhen, Y. Xiao, H. Li, L. You, X. Ma, Q. Zhao, F. Xu, Y. Wang, Y. Wan, Q. Zhang, and J.-W. Pan, Long-lived remote ion–ion en- tanglement for scalable quantum repeaters, Nature 652, 51 (2026). [12] Y. Zheng, H. Wang, X. Jia, J. Huang, H. Yuan, C. Zhai, J. Dai, J. Shi, L. Zhang, X. Zhang, M. Zhuang, J. Liu, J. Mao, T. Dai, Z. Fu, Y. Jiao, Y. Shi, D. Dai, X. Wang, Y. Li, Q. Gong, Z. Yuan, L. Chang, and J. Wang, Large- scale quantum communication networks with integrated photonics, Nature 651, 68 (2026). 9 [13] P.-J. Stas, Y.-C. Wei, M. Sirotin, Y. Q. Huan, U. Yazlar, F. Abdo Arias, E. Knyazev, G. Baranes, B. Machielse, S. Grandi, D. Riedel, J. Borregaard, H. Park, M. Lonˇcar, A. Suleymanzade, and M. D. Lukin, Entanglement- assisted non-local optical interferometry in a quantum network, Nature 651, 326 (2026). [14] X. Zhou, M. Wang, X. Ye, H. Sun, Y. Guo, S. Han, Z. Chai, W. Ji, K. Xia, F. Shi, Y. Wang, and J. Du, Entanglement-enhanced nanoscale single-spin sensing, Nature 647, 883 (2025). [15] J. Rovny, S. Kolkowitz, and N. P. de Leon, Multi-qubit nanoscale sensing with entanglement as a resource, Na- ture 647, 876 (2025). [16] W. Ji, Z. Liu, Y. Guo, Z. Hu, J. Zhou, S. Dai, Y. Chen, P. Yu, M. Wang, K. Xia, F. Shi, Y. Wang, and J. Du, Correlated sensing with a solid-state quantum multisen- sor system for atomic-scale structural analysis, Nat. Pho- tonics 18, 230 (2024). [17] Y. Alexeev, M. H. Farag, T. L. Patti, M. E. Wolf, N. Ares, A. Aspuru-Guzik, S. C. Benjamin, Z. Cai, S. Cao, C. Chamberland, et al., Artificial intelligence for quantum computing, Nat. Commun. 16, 10829 (2025). [18] M. E. Beverland et al., Assessing requirements to scale to practical quantum advantage (2022), arXiv:2211.07629. [19] M. Mohseni et al., How to build a quantum supercom- puter: Scaling from hundreds to millions of qubits (2024), version 3, revised March 2026, arXiv:2411.10406. [20] H. Moon et al., Machine learning enables completely au- tomatic tuning of a quantum device faster than human experts, Nat. Commun. 11, 4161 (2020). [21] F. T. Chong, D. Franklin, and M. Martonosi, Program- ming languages and compiler design for realistic quantum hardware, Nature 549, 180 (2017). [22] J. Wang, S. Paesani, R. Santagati, S. Knauer, A. A. Gen- tile, N. Wiebe, M. Petruzzella, J. L. O’Brien, J. G. Rarity, A. Laing, and M. G. Thompson, Experimental quantum Hamiltonian learning, Nat. Phys. 13, 551 (2017). [23] Y. Du, Y. Zhu, Y.-H. Zhang, M.-H. Hsieh, P. Rebentrost, W. Gao, Y.-Z. You, J. Eisert, G. Chiribella, D. Tao, B. C. Sanders, and Y.-D. Wu, Artificial intelligence for repre- senting and characterizing quantum systems, Nat. Rev. Phys. 8, 579 (2026). [24] S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, K. R. Narasimhan, and Y. Cao, ReAct: Synergizing reasoning and acting in language models, in The Eleventh Interna- tional Conference on Learning Representations (2023). [25] S. Cao, Z. Zhang, M. Alghadeer, S. D. Fasciati, M. Piscitelli, M. Bakr, P. Leek, and A. Aspuru-Guzik, Automating quantum computing laboratory experiments with an agent-based AI framework, Patterns 6, 101372 (2025). [26] H. Xu, J. Han, S. Ou, et al., Vibe calibration: Au- tonomous bring-up of a 112-qubit superconducting quan- tum processor by a skill-orchestrating language agent (2026), arXiv:2606.22376. [27] S. Arlt, X. Gu, and M. Krenn, Towards autonomous quantum physics research using LLM agents with access to intelligent tools (2025), arXiv:2511.11752. [28] T. Isogawa, R. Okabe, N. Phadetsuwannukun, M. Li, and P. Cappellaro, Agentic AI for scientific reasoning in autonomous quantum sensing experiments (2026), arXiv:2607.25145 [quant-ph]. [29] C. Dalyac, A. Dauphin, L. Henriet, and C. Jurczak, Lowering the implementation barrier of neutral-atom quantum computing with agentic workflows (2026), arXiv:2607.25834. [30] Z. Fu, L. Jiang, Y. Xu, G. Huang, and F. Chen, QA- gent: An LLM-based multi-agent system for autonomous OpenQASM programming (2025), arXiv:2508.20134 [cs.AI]. [31] W. Li, J. Ren, L. Cheng, and C. Gong, Autonomous quantum simulation through large language model agents (2026), arXiv:2601.10194 [quant-ph]. [32] I. Gustin, L. Mantilla Calder ́on, J. B. P ́erez-S ́anchez, C. Crebolder, J. F. Gonthier, M. Ghazi Vakili, Y. Naka- mura, K. Panicker, M. Ramprasad, A. Yin, et al., El agente cu ́antico: automating quantum simulations, Rep. Prog. Phys. 89, 077602 (2026). [33] M. Shiraishi, I. Hamamura, T. Ishigaki, and T. Kad- owaki, A Model Context Protocol server for quantum ex- ecution in hybrid quantum-HPC environments, in 2026 International Conference on Quantum Communications, Networking, and Computing (QCNC) (IEEE, 2026) p. 1–6. [34] T. Proctor, M. Revelle, E. Nielsen, K. Rudinger, D. Lob- ser, P. Maunz, R. Blume-Kohout, and K. Young, Detect- ing and tracking drift in quantum information processors, Nat. Commun. 11, 5396 (2020). [35] V. Sivak, A. Morvan, M. Broughton, R. G. Corti ̃nas, J. Bausch, A. W. Senior, M. Neeley, A. Eickbusch, N. Shutty, L. A. Beni, et al., Reinforcement learning con- trol of quantum error correction, Nature 655, 879–884 (2026). [36] D. T. Lennon, H. Moon, L. C. Camenzind, L. Yu, D. M. Zumb ̈uhl, G. A. . D. Briggs, M. A. Osborne, E. A. Laird, and N. Ares, Efficiently measuring a quantum device us- ing machine learning, npj Quantum Information 5, 79 (2019). [37] Y. Baum et al., Experimental deep reinforcement learn- ing for error-robust gate-set design on a superconducting quantum computer, PRX Quantum 2, 040324 (2021). [38] S. Cao et al., QCalEval: Benchmarking vision-language models for quantum calibration plot understanding (2026), arXiv:2604.25884. [39] S. Minami, T. Ishigaki, I. Hamamura, et al., Quan- tumBench: A benchmark for quantum problem solving (2025), arXiv:2511.00092. [40] R. Yang, Z. Wang, Y. Gu, Y. Liang, and T. Li, QCir- cuitBench: A large-scale dataset for benchmarking quan- tum algorithm design, in Advances in Neural Information Processing Systems, Vol. 38 (2025) p. 48750–48801. [41] D. Horsman, A. G. Fowler, S. Devitt, and R. Van Meter, Surface code quantum computing by lattice surgery, New J. Phys. 14, 123011 (2012). [42] D. Litinski, A game of surface codes: Large-scale quan- tum computing with lattice surgery, Quantum 3, 128 (2019). [43] A. Erhard et al., Entangling logical qubits with lattice surgery, Nature 589, 220 (2021). [44] I. Besedin et al., Lattice surgery realized on two distance- three repetition codes with superconducting qubits, Nat. Phys. 22, 189 (2026). [45] W. Lin et al., Surface code logical operations on a super- conducting quantum processor (2026), arXiv:2607.01473 [quant-ph]. [46] J. Bausch et al., Learning high-accuracy error decoding for quantum processors, Nature 635, 834 (2024). 10 [47] F. J. Schreiber, J. Eisert, and J. J. Meyer, Classical sur- rogates for quantum learning models, Phys. Rev. Lett. 131, 100803 (2023). [48] Y. Du, M.-H. Hsieh, and D. Tao, Efficient learning for linear properties of bounded-gate quantum circuits, Nat. Commun. 16, 3790 (2025). [49] W.-Y. Liao, Y. Du, X. Wang, T.-C. Tian, Y. Luo, B. Du, D. Tao, and H.-L. Huang, Demonstration of efficient predictive surrogates for large-scale quantum processors, Nat. Commun. 17, 4731 (2026). [50] Harbor Framework Team, Harbor: A framework for eval- uating and optimizing agents and models in container en- vironments, https://doi.org/10.5281/zenodo.21878 893 (2026), version 0.21.0, https://github.com/harbo r-framework/harbor. [51] M. A. Merrill, A. G. Shaw, N. Carlini, et al., Terminal- Bench: Benchmarking agents on hard, realistic tasks in command line interfaces (2026), arXiv:2601.11868 [cs.SE]. [52] Terminal-Bench-Science Team, Terminal-Bench-Science: Evaluating AI agents on research workflows across scien- tific domains, Zenodo, https://doi.org/10.5281/zeno do.22110253 (2026). [53] Anthropic, Introducing the Model Context Protocol, ht tps://w.anthropic.com/news/model-context-pro tocol (2024). [54] A. Javadi-Abhari, M. Treinish, K. Krsulich, C. J. Wood, J. Lishman, J. Gacon, S. Martiel, P. D. Nation, L. S. Bishop, A. W. Cross, B. R. Johnson, and J. M. Gambetta, Quantum computing with Qiskit (2024), arXiv:2405.08810 [quant-ph]. [55] QCoDeS Contributors, QCoDeS: Python-based data ac- quisition framework, Zenodo, https://doi.org/10.528 1/zenodo.596989 (2026). [56] T. Islam, D. Wadekar, and Z. Zhou, gwBenchmarks: Stress-testing LLM agents on high-precision gravitational wave astronomy (2026), arXiv:2605.11269 [gr-qc]. [57] N. Chowdhury, D. Johnson, V. Huang, J. Steinhardt, and S. Schwettmann, Investigating truthfulness in a pre- release o3 model, Transluce technical report, https://tr ansluce.org/investigating-o3-truthfulness (2025), accessed 8 August 2026. [58] N. C. Jones, R. Van Meter, A. G. Fowler, P. L. McMahon, J. Kim, T. D. Ladd, and Y. Yamamoto, Layered archi- tecture for quantum computing, Phys. Rev. X 2, 031007 (2012). [59] M. Sarovar, T. Proctor, K. Rudinger, K. Young, E. Nielsen, and R. Blume-Kohout, Detecting crosstalk errors in quantum information processors, Quantum 4, 321 (2020). [60] C. P. Koch, U. Boscain, T. Calarco, G. Dirr, S. Fil- ipp, S. J. Glaser, R. Kosloff, S. Montangero, T. Schulte- Herbr ̈uggen, D. Sugny, and F. K. Wilhelm, Quantum op- timal control in quantum technologies. strategic report on current status, visions and goals for research in eu- rope, EPJ Quantum Technol. 9, 19 (2022). [61] I. de Vega and D. Alonso, Dynamics of non-markovian open quantum systems, Rev. Mod. Phys. 89, 015001 (2017). [62] H.-J. Briegel, W. D ̈ur, J. I. Cirac, and P. Zoller, Quan- tum repeaters: The role of imperfect local operations in quantum communication, Phys. Rev. Lett. 81, 5932 (1998). [63] A. W. Chin, S. F. Huelga, and M. B. Plenio, Quantum metrology in non-markovian environments, Phys. Rev. Lett. 109, 233601 (2012). [64] D. Shepard and R. Salimans, AutomationBench (2026), benchmark by Zapier; scores quoted in this work are from the Artificial Analysis independent run, not from this paper, whose strict whole-task completion metric reports different values, arXiv:2604.18934. [65] Artificial Analysis, AutomationBench-A: Independent evaluation of AutomationBench, https://artificialan alysis.ai/evaluations/automationbench-a (2026), independent run by Artificial Analysis on a private held- out set of 657 tasks across six business domains; the head- line metric is the average share of each task’s objectives completed without guardrail violations. Accessed 2026- 09-15. [66] Artificial Analysis, GDPval-A v2: Independent evalua- tion of OpenAI’s GDPval gold set, https://artificial analysis.ai/evaluations/gdpval-a (2026), Elo rat- ings from blind pairwise comparisons; agents run with shell and browsing access via Stirrup. Accessed 2026-09- 15.

Instant research discovery