Paper deep dive
Qlippy: A Retrieval-Augmented GenAI Assistant for Reproducible Quantum Workflows and Experiment Tracking
Mahee Gamage, Vlad Stirbu
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 91%
Last extracted: 9/8/2026, 1:35:42 AM
Summary
The paper introduces Qlippy, a retrieval-augmented generation (RAG) assistant integrated into Visual Studio Code to support reproducible quantum software development. Qlippy grounds its responses in a curated corpus of quantum software engineering knowledge, explaining reproducibility and provenance concepts while automatically augmenting Qiskit programs with MLflow-based experiment tracking aligned to the QProv schema. The system aims to reduce the cognitive load and expertise barrier for developers, particularly novices, by providing context-aware, in-editor assistance that separates knowledge from model parameters, enabling low-cost, privacy-preserving local deployment.
Entities (12)
Relation Signals (10)
Qlippy â implementstrackingwith â MLflow
confidence 95% · augments existing Qiskit programs with MLflow-based experiment tracking aligned to the QProv schema.
Qlippy â integrateswith â Visual Studio Code
confidence 95% · integrated into Visual Studio Code, designed to support the adoption of reproducible quantum software practices.
Qlippy â supports â reproducible quantum software development
confidence 95% · Qlippy: A Retrieval-Augmented GenAI Assistant for Reproducible Quantum Workflows and Experiment Tracking
Qlippy â uses â RAG
confidence 95% · Qlippy is a retrieval-augmented GenAI assistant... RAG mitigates these limitations by integrating external, curated knowledge sources
Qlippy â alignswith â QProv
confidence 90% · augments existing Qiskit programs with MLflow-based experiment tracking aligned to the QProv schema.
Qlippy â augments â Qiskit
confidence 90% · augments existing Qiskit programs with MLflow-based experiment tracking
Qlippy â evaluatedusing â RAGAS
confidence 90% · We conducted a preliminary evaluation of the advisory scenario using the RAGAS framework
Qlippy â implementedwith â FastAPI
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Quantum software development is iterative and error-prone. Noisy hardware and repeated re-execution make experiment tracking, provenance, and reproducibility essential, yet these practices are hard to adopt because of tooling complexity and the specialized knowledge they demand. General-purpose language models can help but tend to hallucinate and lack grounding in domain-specific tooling. We present Qlippy, a retrieval-augmented GenAI assistant embedded in the development environment that grounds its responses in a curated corpus of quantum-software-engineering knowledge. Qlippy explains reproducibility and provenance concepts in context and augments existing Qiskit programs with MLflow-based experiment tracking aligned to the QProv schema. By separating knowledge from model parameters, grounding gives explicit control over the scope and provenance of the assistant's responses and reduces reliance on model scale, which points toward low-cost, privacy-preserving local deployment.
Tags
Links
- Source: https://arxiv.org/abs/2609.05039v1
- Canonical: https://arxiv.org/abs/2609.05039v1
Trouble viewing inline? Open PDF directly â
Full Text
21,360 characters extracted from source content.
Expand or collapse full text
Qlippy: A Retrieval-Augmented GenAI Assistant for Reproducible Quantum Workflows and Experiment Tracking Mahee Gamage University of Jyv Ì askyl Ì a Jyv Ì askyl Ì a, Finland mahee.s.hewagamage@jyu.fi Vlad Stirbu University of Jyv Ì askyl Ì a Jyv Ì askyl Ì a, Finland vlad.a.stirbu@jyu.fi AbstractâQuantum software development is iterative and error-prone. Noisy hardware and repeated re-execution make experiment tracking, provenance, and reproducibility essential, yet these practices are hard to adopt because of tooling com- plexity and the specialized knowledge they demand. General- purpose language models can help but tend to hallucinate and lack grounding in domain-specific tooling. We present Qlippy, a retrieval-augmented GenAI assistant embedded in the develop- ment environment that grounds its responses in a curated corpus of quantum-software-engineering knowledge. Qlippy explains reproducibility and provenance concepts in context and augments existing Qiskit programs with MLflow-based experiment tracking aligned to the QProv schema. By separating knowledge from model parameters, grounding gives explicit control over the scope and provenance of the assistantâs responses and reduces reliance on model scale, which points toward low-cost, privacy-preserving local deployment. Index Termsâquantumsoftwareengineering,retrieval- augmented generation, experiment tracking, provenance, repro- ducibility, LLM-assisted development I. INTRODUCTION Quantum computing is emerging as an interdisciplinary field that attracts practitioners from physics, mathematics, and computer science across research, industry, and education. As access to quantum programming frameworks and cloud-based quantum hardware expands, more developers are building and running quantum algorithms in practice. However, the devel- opment of quantum software is not only a matter of algorithm design, but also involves practices rooted in modern software engineering, such as experiment tracking, reproducibility, and structured data management. These practices are essential for ensuring that experimental results can be validated, compared, and iteratively improved. Despite the wide adoption of these practices in classical computing, adopting them in quantum software development remains challenging. Practitioners entering the field often pos- sess strong theoretical backgrounds but have limited exposure to software engineering concepts, particularly those related to reproducibility and lifecycle management, an issue that is This work has been supported by Business Finland projects EM4QS (155/31/2024) and the Research Council of Finland project DEQSE (349945). especially pronounced for novice developers. As a result, ex- periment tracking and provenance collection are either applied superficially or omitted entirely, leading to ad hoc development practices and reduced transparency in experimental results. Furthermore, the complexity of integrating external tools into development workflows increases cognitive load and creates friction for developers. While recent advances in generative AI offer opportunities to support this work, generic language models lack grounding in domain-specific materials and may produce explanations or code that are misaligned with domain practices, limiting their effectiveness in specialized settings. In this paper, we present a retrieval-augmented GenAI assistant integrated into Visual Studio Code, designed to sup- port the adoption of reproducible quantum software practices. The assistant leverages curated knowledge sources, includ- ing tools documentation and relevant research literature, to provide context-aware explanations and code augmentation that help developers apply experiment tracking and collect structured provenance information. Through a guided sce- nario, we demonstrate how retrieval-augmented generation can bridge the gap between quantum algorithm development and modern software engineering practices, reducing cognitive overhead while reinforcing understanding through immediate application. I. BACKGROUND Experiment tracking is particularly important in quantum software development due to the intrinsic instability of current hardware, where noise, calibration drift, and backend variabil- ity can significantly affect results across executions. These conditions make quantum development inherently iterative, requiring practitioners to repeatedly refine circuits, parameters, and execution strategies while carefully observing marginal improvements [1]. In this context, classical machine learning tools such as MLflow [2] provide a structured way to log parameters, results, and artifacts, supporting reproducibility and comparative analysis. However, despite its importance, ex- periment tracking is rarely adopted systematically in quantum software development, leaving practitioners to rely on ad hoc practices. arXiv:2609.05039v1 [quant-ph] 4 Sep 2026 Prior work on laboratory practices and reporting related to quantum software experiments [3], highlights comprehensive approaches to structuring experiments and reporting results, yet also illustrates that the current bar for systematic track- ing and documentation remains high for novice developers. This gap reinforces the need for more accessible, integrated approaches that embed experiment tracking into everyday de- velopment workflows rather than treating it as an advanced or optional practice. Beyond basic logging, effective experiment tracking in quantum software requires structured metadata and detailed execution context, including circuits, compilation steps, backend settings, and noise conditions. Without this, results are difficult to reproduce or compare across runs. The QProv schema addresses this by proposing a comprehensive attribute schema on how provenance is captured, enabling more reliable and interpretable experimentation [4]. Adopting these practices is challenging in its own right: developers must reason simultaneously about quantum algo- rithms, hybrid execution environments, and supporting in- frastructure, while tools such as MLflow and provenance approaches like QProv add further layers of abstraction [5]. These are experiential skills that are difficult to acquire from documentation alone, which motivates tool-supported approaches embedded directly in the development workflow. Generative AI, particularly Large Language Models (LLMs), is increasingly used as an assistant in software engineering tasks due to its ability to interpret natural language and generate contextually relevant responses [6]. However, LLMs rely on parametric memory encoded during pre-training, which leads to well-documented limitations such as hallucina- tions, outdated knowledge, and lack of grounding in domain- specific facts [7]. These issues are particularly problematic in specialized domains like quantum software development, where correctness and traceability are essential. Retrieval- Augmented Generation (RAG) mitigates these limitations by integrating external, curated knowledge sources into the generation process, allowing the model to retrieve relevant information and ground its responses in verifiable content [8]. By separating knowledge from model parameters and enabling controlled, domain-specific knowledge bases, RAG supports more reliable, transparent, and domain-appropriate interactions. I. DESIGN A. System Architecture The system architecture comprises two distinct environ- ments interconnected via a REST API, as illustrated in Fig. 1. On the developerâs local workstation, a Visual Studio Code extension provides a chat-based interface, enabling natural language interaction with the assistant. The research environment hosts two primary services: an MLflow tracking server for managing experimental metadata and a RAG (Retrieval-Augmented Generation) service. Im- plemented in Python using FastAPI 1 and orchestrated via 1 https://fastapi.tiangolo.com/ Fig. 1. The system architecture (newly developed components in gray) LangChain 2 , the RAG service operates through three core components. First, a ChromaDB 3 vector database stores cu- rated documentation on quantum software experiment track- ing, serving as the systemâs non-parametric memory. The cor- pus comprises five research papers, two synthetically authored reference documents, and a curated subset of the MLflow doc- umentation, ingested via a pipeline consisting of file parsing, chunking, and embedding stages. Second, a retriever identifies and extracts relevant context from this database based on user input. Finally, a generator synthesizes the retrieved context and the user query to produce natural-language explanations or modified code segments. The service is designed to be LLM- agnostic, ensuring seamless interoperability with both com- mercial APIs and local open-weight models without requiring modifications to the core pipeline. B. Developer interaction model The system follows an interaction loop tightly integrated into the developer workflow. A developer begins by writing or exploring quantum code within the development environment and then poses a question to the assistant in natural language. In response, the assistant performs three complementary ac- tions: it explains the underlying concept, augments or modifies the code where appropriate, and references relevant sources to ground the explanation. Because each explanation is coupled with an immediate code-level change delivered in context, conceptual understanding and practical implementation re- inforce each other rather than being deferred to external documentation or tutorials. C. Integration with Tracking The assistant integrates directly with MLflow as a real- world experiment tracking system, ensuring that the generated outputs align with tools and practices expected in professional and research settings. Instead of introducing a simplified or simulated mechanism, the system augments user code by automatically inserting MLflow-based tracking calls, enabling the recording of userâs experiment runs, parameters, metrics, and artifacts in the same manner as used in research and industry environments. This makes the resulting workflow directly transferable to real-world practice. 2 https://w.langchain.com/ 3 https://w.trychroma.com/ Fig. 2. Qlippy in the advisory scenario for experiment tracking. IV. DEMONSTRATION A. Advisory Scenario: In-Context Explanation of Experiment Tracking The first demonstration scenario focuses on advisory sup- port, where the artifact helps quantum software practitioners, including novices, acquire knowledge about experiment track- ing concepts without requiring prior exposure to tools such as MLflow or provenance models like QProv. In this scenario, the user interacts with the system through natural language queries embedded within the development environment (e.g., VSCode), depicted in Fig. 2. The system retrieves relevant information from curated sources, including MLflow docu- mentation and QProv specifications, and generates context- aware responses that explain both conceptual and practical aspects of experiment tracking. This interaction is iterative and exploratory: users can refine questions, request examples, and connect abstract provenance concepts with concrete implementation practices. The system effectively acts as a domain-specific advisor, bridging the gap between quantum software development expertise and experiment tracking practices. The success of this scenario is demonstrated when the system provides coherent, grounded explanations that enable users to understand what to track (e.g., circuit properties, execution parameters) and why these are important in the NISQ context. This directly addresses the identified knowledge barrier and supports understanding through interaction rather than static documentation. B. Augmentation Scenario: AI-Assisted Injection of Experi- ment Tracking Code The second demonstration scenario focuses on augmenta- tion, where the artifact actively modifies existing quantum programs to include experiment tracking capabilities. In this case, the user provides a functional Qiskit-based quantum pro- gram along with a natural language instruction describing the tracking requirements (e.g., aligning with QProv attributes). The system retrieves the necessary knowledge and generates a modified version of the program with integrated MLflow logging. Fig. 3. Using the Qlippy in the injection of experiment tracking code Unlike the advisory scenario, which supports understanding, this scenario reduces the need for manual implementation by automating the translation of conceptual knowledge into executable code. The generated output preserves the original program logic while augmenting it with tracking functionality, enabling the recording of parameters, metrics, and provenance data during execution. The demonstration is considered suc- cessful when the modified program executes correctly and logs experiment data consistent with QProv guidelines into the MLflow tracking system. This scenario illustrates how the artifact shifts from a passive advisory role to an active development assistant, low- ering the operational barrier to adopting structured experiment tracking in quantum workflows. V. DISCUSSION A. Preliminary evaluation We conducted a preliminary evaluation of the advisory scenario using the RAGAS framework [9] (an LLM-as-a- Judgeâbased evaluation approach), over 20 questionâanswer pairs spanning five subdomains of experiment tracking and four question types (factual, summary, reasoning, and unan- swerable), with GPT-5.4-mini as judge 4 . Responses were scored on faithfulness, context recall, answer relevancy, and factual correctness (recall value based), excluding unanswer- able questions from the means since evaluation metrics are not designed to meaningfully evaluate that question type. Across four generator models (GPT-4o-mini and GPT-5.4 via the OpenAI API; Gemma4:e4b and Qwen3.5:9b, quan- tized and served locally via Ollama), the locally deployable Qwen3.5:9b achieved the highest answer relevancy (0.917), while GPT-5.4 achieved the highest factual correctness (0.908), only 12% higher than Qwen3.5:9bâs (0.814). Given the scale of the question set and the use of an LLM as judge, these results are best read as relative comparisons between configurations. A more rigorous evaluation is needed before drawing firm conclusions about system quality. 4 OpenAI API, accessed April 2026. B. Benefits for quantum software development A retrieval-augmented assistant embedded in the develop- ment environment offers several practical benefits for adopt- ing reproducible quantum software practices. First, it low- ers the expertise barrier: developers can apply experiment tracking and provenance collection without deep familiarity with MLflow, the QProv schema, or the surrounding DevOps tooling. Second, support is delivered in context and just in time, at the point in the workflow where a practice becomes relevant, which reduces the friction of adopting unfamiliar tools. Third, and in contrast to general-purpose coding assis- tants, grounding the model in a curated corpus gives explicit control over the scope and provenance of its knowledge, keeping responses aligned with a target practice rather than the open-ended behavior of an ungrounded model. Education is one setting where these benefits apply directly, helping instruc- tors introduce advanced practices without mastering the full tooling ecosystem, but the same properties support research and industry teams onboarding reproducible workflows. C. Operational concerns A central consideration in deploying such an assistant is the tradeoff between cost, performance, and control over the ex- ecution environment. As the preliminary evaluation indicates, locally hosted models such as qwen3.5:9b can approach commercial offerings like GPT-5.4 on answer relevancy and factual coverage when used within a Retrieval-Augmented Generation (RAG) pipeline. This is largely because RAG shifts the burden of knowledge from the modelâs parametric memory to curated external sources, effectively reducing the role of the generator to synthesis and contextualisation. As a result, the advantage of large parametric models is less pronounced in such constrained, domain-specific scenarios. From a cost perspective, local deployment eliminates re- curring API usage fees, which can become significant under continuous or large-scale usage. It also provides greater control over data flows, addressing privacy concerns that are especially salient when proprietary code or unpublished research artifacts are involved. These properties make locally hosted solutions attractive across research, industry, and teaching settings alike, wherever cost, confidentiality, or control over the execution environment are priorities. However, these benefits come with their own tradeoffs, including the need to provision and maintain adequate com- putational infrastructure, manage model updates, and ensure system reliability. The decision between commercial and local models is therefore not only a performance comparison but a broader socio-technical choice involving cost structures, governance requirements, and operational capabilities. D. Threats to validity This study has several limitations, identified according to Wohlin et al [10] as follows. First, the evaluation is based on a prototype implementation, which demonstrates feasibility but may not reflect performance and usability in real-world deployments. Second, the system depends on curated knowl- edge sources; any gaps or biases in these sources directly affect retrieval quality and response correctness. Third, the artifact has not been extensively validated with real users and development teams, limiting conclusions about its adoption in practice. Finally, despite grounding through RAG, residual LLM limitations remain, including occasional hallucinations and variability in responses, particularly for complex queries. VI. CONCLUSION This paper presented Qlippy, a retrieval-augmented GenAI assistant that embeds experiment tracking and provenance practices directly into the quantum software development workflow. By grounding its responses in curated sources and coupling explanations with automated MLflow instrumentation aligned to the QProv schema, the assistant helps developers apply reproducible practices as they write code, one use case being novice practitioners learning these concepts for the first time. Our preliminary evaluation indicates that, within a RAG pipeline, a locally hosted model can approach a commercial one on answer relevancy and factual coverage, making low- cost, privacy-preserving deployment realistic across research, industry, and teaching settings alike. Since the evaluation remains preliminary, future work will extend the question set with multi-expert validation, add a formal assessment of provenance completeness for the augmentation scenario, and compare the grounded assistant against an ungrounded model of equal size and against general-purpose coding assistants. REFERENCES [1] O. Kinanen, A. D. Mu Ì noz-Moller, V. Stirbu, J. M. Murillo, and T. Mikkonen, âToolchain for faster iterations in quantum software development,â Computing, vol. 107, no. 4, p. 99, 2025. [2] M. Zaharia, A. Chen, A. Davidson, A. Ghodsi, S. A. Hong, A. Konwin- ski, S. Murching, T. Nykodym, P. Ogilvie, M. Parkhe et al., âAccelerat- ing the machine learning lifecycle with mlflow,â IEEE Data Eng. Bull., vol. 41, no. 4, p. 39â45, 2018. [3] E. Moguel, J. A. Parejo, A. Ruiz-Cort Ì es, J. Garcia-Alonso, and J. M. Murillo, âQuantum software experiments: A reporting and laboratory package structure guidelines proposal,â in 2025 IEEE International Conference on Quantum Software (QSW). IEEE, 2025, p. 185â194. [4] B. Weder, J. Barzen, F. Leymann, M. Salm, and K. Wild, âQprov: A provenance system for quantum computing,â IET Quantum Communi- cation, vol. 2, no. 4, p. 171â181, 2021. [5] J. Reinikainen, V. Stirbu, T. Heinosaari, V. Lappalainen, and T. Mikko- nen, âQuantum computing for all: Online courses built around an interactive visual quantum circuit simulator,â IEEE Computer Graphics and Applications, vol. 44, no. 5, p. 67â75, 2024. [6] T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell et al., âLanguage models are few-shot learners,â in Advances in Neural Information Processing Systems, vol. 33, 2020, p. 1877â1901. [7] P. Lewis, E. Perez, A. Piktus, F. Petroni, V. Karpukhin, N. Goyal, H. K Ì uttler, M. Lewis, W.-t. Yih, T. Rockt Ì aschel, S. Riedel, and D. Kiela, âRetrieval-augmented generation for knowledge-intensive NLP tasks,â in Advances in Neural Information Processing Systems, vol. 33, 2020, p. 9459â9474. [8] Y. Gao, Y. Xiong, X. Gao, K. Jia, J. Pan, Y. Bi, Y. Dai, J. Sun, M. Wang, and H. Wang, âRetrieval-augmented generation for large language models: A survey,â arXiv:2312.10997 [cs], Mar. 2024. [9] S. Es, J. James, L. Espinosa-Anke, and S. Schockaert, âRAGAs: Au- tomated evaluation of retrieval augmented generation,â in Proc. EACL: Syst. Demonstrations, 2024, p. 150â158. [10] C. Wohlin, P. Runeson, M. H Ì ost, M. C. Ohlsson, B. Regnell, and A. Wessl Ì en, Experimentation in Software Engineering. Springer, 2012.