Paper deep dive
Are LLMs Good For Quantum Software, Architecture, and System Design?
Sourish Wawdhane, Poulami Das
Intelligence
Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 96%
Last extracted: 3/31/2026, 1:58:22 AM
Summary
This paper evaluates the performance of nine frontier Large Language Models (LLMs) from OpenAI, Google, and Anthropic on quantum computing system design tasks, comparing them against graduate students. The study finds that while reasoning-capable LLMs perform well on tasks like error decoder design, they struggle with complex architectural challenges such as flag-proxy network optimization. The authors conclude that LLMs show promise for quantum software and architecture development but require domain-specific training, curated datasets, and human expert oversight.
Entities (6)
Relation Signals (3)
LLMs → strugglewith → Flag-proxy networks
confidence 100% · All the LLMs performed poorly on a quantum reasoning task designed around optimal design of flag-proxy networks.
GPT-5.4 → outperforms → Human Students
confidence 95% · GPT 5.4 Thinking... attains the overall highest score of 83, outperforming the students by a significant margin.
Human Experts → improvesperformanceof → LLMs
confidence 90% · The performance of all three model families used in our case study improves when the reasoning models are assisted by the instructor
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Quantum computers promise massive computational speedup for problems in many critical domains, such as physics, chemistry, cryptanalysis, healthcare, etc. However, despite decades of research, they remain far from entering an era of utility. The lack of mature software, architecture, and systems solutions capable of translating quantum-mechanical properties of algorithms into physical state transformations on qubit devices remains a key factor underlying the slow pace of technological progress. The problem worsens due to significant reliance on domain-specific expertise, especially for software developers, computer architects, and systems engineers. To address these limitations and accelerate large-scale high-performance quantum system design, we ask: Can large language models (LLMs) help with solving quantum software, architecture, and systems problems? In this work, we present a case study assessing the performance of LLMs on quantum system reasoning tasks. We evaluate nine frontier LLMs and compare their performance to graduate UT Austin students on a set of quantum computing problems. Finally, we recommend several directions along which research and engineering development efforts must be pursued.
Tags
Links
- Source: https://arxiv.org/abs/2603.26904v1
- Canonical: https://arxiv.org/abs/2603.26904v1
Trouble viewing inline? Open PDF directly →
Full Text
12,621 characters extracted from source content.
Expand or collapse full text
NSF Workshop on Systems Research at the Quantum-AI Frontier, 2026 Are LLMs Good For Quantum Software, Architecture, and System Design? Sourish Wawdhane, Poulami Das The University of Texas at Austin sourishw, poulami.das@utexas.edu 1. Introduction Quantum computers promise massive computational speedup for problems in many critical domains, such as physics, chem- istry, cryptanalysis, healthcare, etc [1–3]. However, despite decades of research, they remain far from entering an era of utility. The lack of mature software, architecture, and systems solutions capable of translating quantum-mechanical proper- ties of algorithms into physical state transformations on qubit devices remains a key factor underlying the slow pace of tech- nological progress. Algorithmic research at the top, such as designing efficient algorithms to map applications or building quantum error correction (QEC) protocols to tolerate device errors, are largely limited to theoretical studies. In contrast, device level research at the bottom is either driven by indus- try that focus on a single qubit modality or academic labs that over-optimize qubit devices at a small scale; however, these optimizations are often fragile and tend to break down as system sizes increase. The problem worsens due to sig- nificant reliance on domain-specific expertise, especially for software developers, computer architects, and systems engi- neers. To address these limitations and accelerate large-scale high-performance quantum system design, we ask: Can large language models (LLMs) help with solving quan- tum software, architecture, and systems problems? 2. Background: A Case Study To answer this question, we perform a preliminary case study. We are the instructors (primary instructor and teaching assis- tant) for the "Introduction to Quantum Computing Systems" course at UT Austin [4]. This course: (1) introduces quantum computing basics to ECE/CS students, (2) familiarizes them with on-going research in the field (with particular emphasis on programming, compiler optimizations, architectural and systems models, based on recently published research papers at premier IEEE and ACM conferences, such as ISCA, MI- CRO, ASPLOS, HPCA, QCE), and (3) trains them to optimize quantum software and architecture. A critical component of the course focuses on in-class exams in which the students use their existing knowledge to solve quantum system design problems. The exam duration is one hour and thirty minutes and students are allowed to bring in calculators and a cheat- sheet written by themselves (although the students generally claim that cheat sheets are not so helpful for them to attempt the questions). For our case study, we take one of these exams and study the performance of different LLMs from OpenAI, Google, and Anthropic. Table 1 offers a detailed summary of these model families and versions. Table 1: Summary of LLMs used. For each provider and model family, we use two variants- a lightweight version for producing fast outputs and an advanced version with reasoning abilities. ProviderModel FamilyLightweightReasoning OpenAIGPT [5]5.35.4-Thinking GoogleGemini [6]3 Fast3.1-Pro AnthropicClaude [7]Sonnet 4.6Opus 4.6 The exam covered topics related fault-tolerant quantum systems, including trade-offs in QEC codes, decoder de- signs [8, 9], synchronizing QEC cycles [10], and design- ing flag-proxy networks [11]. To compare the performance against human experts, we show the test scores of four stu- dent participants. Note that these students willingly partic- ipated in the study and FERPA laws are not violated. The performance of the models and students were both manu- ally graded by the instructors. To measure performance, we use the exam score out of a total of 100 points. The exam, rubric, solutions, and exam scores are available at https://tinyurl.com/arellmsgoodatquantum. 3. Preliminary Results Figure 1 shows the performance (exam score) for different model configurations and compares against the performance of the students. We make the following observations. 1.All six LLMs used performed well with an average score of57.33. The minimum is28using the Gemini-3 Fast model while the maximum is 71 using the GPT-5.4 Thinking model. 2.Reasoning models outperform lightweight variants with an average score of66compared to48.67. Reasoning models increase scores by1.3×and2.1×respectively for the GPT and Gemini models, compared to only1.1×for the Claude models. GPT 5.4 Thinking used the largest reasoning traces (1m 57s without papers and 4m 18s with papers). Gemini Pro used 56s without papers and 1m 23 seconds with papers. In contrast, Claude reasoning models chose not to reason. 3.Assisting the reasoning LLMs to use relevant research papers [8, 10, 11] increases the average score further to72.33. The improvements are much more pronounced for the GPT and Claude models than the Gemini model. 4.The GPT models outperform the other two LLM families for each configuration and attains the overall highest score of 83, outperforming the students by a significant margin. 5. LLMs performed poorly on a question requiring test-takers to map QEC codes to hardware. LLMs struggled to discover optimal mappings and reason about error masking. arXiv:2603.26904v1 [quant-ph] 27 Mar 2026 5.35.4 Thinking 5.4 Thinking 3 Fast 3.1 Pro 3.1 Pro Sonnet 4.6 Opus 4.6 Opus 4.6 Student 1 Student 2 Student 3 Student 4 0 25 50 75 100 Exam Score 55 71 83 28 58 59 63 69 75 39 40 47 57 GPTGeminiClaudeHuman Model provided with relevant research papers during evaluation Figure 1: Performance of different large language model families for quantum reasoning tasks, especially related to software, architecture, and system design. Overall, all the model families perform well on the exam, especially the reasoning variants of each model. The performance also increases when assisted by a human expert. 4. Future: Quantum Systems Aided by AI Based on this preliminary case-study, we think research at the quantum and AI frontier is a promising and critical direction. A Promising Outlook: The overall performance of LLMs on the exam (which the instructors as well as the students con- sider to be significantly difficult) is promising and highlights the possibility of using LLMs to accelerate quantum software, architecture, and systems development. A fundamental re- search question on this front remains: how do we build agentic workflows specifically for quantum system design? Creating Quantum Specific LLMs: Modern day LLMs are primarily designed for coding tasks, content creation, summa- rization, Q&A settings. However, we lack LLMs specifically trained for quantum reasoning tasks. Creating dedicated mod- els or fine-tuning existing LLMs have the potential to improve performance even further. We should spend research and en- gineering efforts into this stream to effectively use LLMs for quantum reasoning tasks. In particular, all the models used in our study show excellent performance in error decoder design questions in the context of QEC and surface codes. Accurate, fast, and scalable decoders are critical to enable QEC at scale. Using quantum-specific LLMs have the potential to further assist in their development. Need for Training Datasets: We understand that the current case-study is done in a very limited setting and requires more involved expert participation to refine the scope and assess the headroom for improvement. It also requires advanced training recipes to steer the model to generate more accurate and nuanced outputs. For example, a similar study designed and led by researchers from Harvard University along with participants from various top-ranked computer engineering programs [12] show that using curated datasets for computer architecture problems improve the reasoning capability of tra- ditional LLMs. We anticipate a similar methodology to remain effective even for quantum reasoning tasks. We recommend concerted efforts in this space to expand the capabilities of LLMs for quantum tasks. Role of Human Experts Remain Critical: The performance of all three model families used in our case study improves when the reasoning models are assisted by the instructor to use the right research paper. Especially in a field like quantum computing and topics involving software, architecture, and systems design, the human expert still plays a critical role in the adoption of LLMs. It also remains unclear if moving to larger models with more parameters and advanced reasoning capabilities can close the gap by reducing the reliance on the expert or we would hit a parameter wall. This also poses a more crucial question- how do we maximize the utility of the human experts given the limited workforce in this domain and the criticality of their inputs. LLMs Struggle On Certain Advanced Topics: All the LLMs performed poorly on a quantum reasoning task designed around optimal design of flag-proxy networks. Flag proxy net- works optimally use flag and proxy qubits to reduce error propagation and overcome the limited connectivity of super- conducting quantum systems. For higher resource efficiency and error performance, designing these networks with minimal qubits is critical. Six models failed to generate valid flag con- nectivity configurations that minimized flag usage or required swaps. Moreover, all the LLMs under study were unable to reason about multi-error scenarios, where error propagation can mask syndromes via interference. These findings suggest that current LLMs struggle with certain complex reasoning tasks, motivating the use and improvement of multi-modal LLMs to solve these problems with higher accuracy. 5. Conclusion LLM-assisted workflows and AI-based frameworks promise revolutionary impact in traditional software ecosystems. In this paper, we show how LLMs can also be used to power quantum software, architecture, and systems development by evaluating the performance of traditional LLMs on quantum reasoning tasks. Based on our study, we have recommended several directions along which research and engineering devel- opment efforts must be pursued. 2 Acknowledgments We thank Shashank Nag, Allison Seigler, Dongwhee Kim, and Pawan Kashyap, for their participation in this study. We thank Avinash Kumar for generating Claude responses. We have open sourced the exam as well as the rubric and we encourage others to contribute to the dataset as well. References [1]IBM. Quantum protein folding algorithms.https://protein- folding-demo.mybluemix.net/, year=2016. [2] Seth Lloyd. Universal quantum simulators. Science, 1996. [3]Peter W Shor. Algorithms for quantum computation: discrete log- arithms and factoring. In Proceedings 35th annual symposium on foundations of computer science. IEEE, 1994. [4]Poulami Das. Ece-382v: Introduction to quantum computing systems: From a software and architecture perspective. [5]OpenAI.Openai api and model documentation.https:// platform.openai.com/docs/models, 2024. [6] Gemini Team and Google DeepMind. Gemini: A family of highly capable multimodal models. arXiv preprint arXiv:2312.11805, 2024. [7] Anthropic.Claudemodeldocumentation.https: //docs.anthropic.com, 2024. [8]Narges Alavisamani, Suhas Vittal, Ramin Ayanzadeh, Poulami Das, and Moinuddin Qureshi. Promatch: Extending the reach of real-time quantum error correction with adaptive predecoding. In Proceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 3, pages 818–833, 2024. [9]Poulami Das et al. Afs: Accurate, fast, and scalable error-decoding for fault-tolerant quantum computers. In 2022 IEEE International Symposium on High-Performance Computer Architecture (HPCA), pages 259–273. IEEE, 2022. [10] Satvik Maurya and Swamit Tannu. Synchronization for fault-tolerant quantum computers. In Proceedings of the 52nd Annual International Symposium on Computer Architecture, pages 1370–1385, 2025. [11] Suhas Vittal, Ali Javadi-Abhari, Andrew W Cross, Lev S Bishop, and Moinuddin Qureshi. Flag-proxy networks: Overcoming the architec- tural, scheduling and decoding obstacles of quantum ldpc codes. In 2024 57th IEEE/ACM International Symposium on Microarchitecture (MICRO), pages 718–734. IEEE, 2024. [12]Shvetank Prakash, Andrew Cheng, Jason Yik, Arya Tschand, Radhika Ghosal, Ikechukwu Uchendu, Jessica Quaye, Jeffrey Ma, Shreyas Grampurohit, Sofia Giannuzzi, Arnav Balyan, Fin Amin, Aadya Pipersenia, Yash Choudhary, Ankita Nayak, Amir Yazdanbakhsh, and Vijay Janapa Reddi. Quarch: A question-answering dataset for ai agents in computer architecture, 2025. 3