Paper deep dive
Generative Replay Mitigates Sample Starvation in Quantum Architecture Search
Akash Kundu, Amit Kumar Jaiswal, Sebastian Feld, Prayag Tiwari
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 93%
Last extracted: 9/11/2026, 5:30:41 AM
Summary
The paper introduces GenQAS, a reinforcement learning framework for quantum architecture search (QAS) that mitigates sample starvation by combining tensor network-guided initialization (Matrix Product State warm-start) with prioritized generative replay. A learned local transition model generates synthetic circuit transitions from real state-action seeds, mixing them with real experience during Double Deep Q-Network updates. GenQAS demonstrates significant improvements in success probability and sample efficiency across chemical Hamiltonian benchmarks (BeH2, H2O) and a transverse field Ising model, outperforming passive replay baselines, especially at larger qubit counts (up to 15 qubits).
Entities (10)
Relation Signals (8)
GenQAS → evaluatedon → H2O
confidence 95% · Across chemical Hamiltonian benchmarks from 6 to 12 qubits, GenQAS improves fixed-budget success probability...
GenQAS → evaluatedon → BeH2
confidence 95% · In a noisy 6-qubit BeH2 transfer experiment, generative replay reduces the steps to chemical accuracy by 92.7%.
GenQAS → evaluatedon → Transverse Field Ising Model
confidence 95% · On a 15-qubit transverse field Ising model, GenQAS increases success probability from 12% to 21%.
GenQAS → uses → Generative Replay
confidence 95% · Here we introduce GenQAS, a tensor network-guided RL framework that combines a fixed matrix product state warm-start with prioritized generative replay.
GenQAS → uses → Double Deep Q-Network
confidence 95% · A learned local transition model generates synthetic circuit transitions on demand and mixes them with real experience during Double Deep Q-Network updates.
GenQAS → outperforms → TensorRL-QAS
confidence 92% · At 12 qubits, it improves final success probability by up to 7.0× over passive replay.
Generative Replay → mitigates → Sample Starvation
confidence 90% · These results show that generative replay can mitigate sample starvation in quantum architecture search...
GenQAS → →
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Reinforcement learning (RL) can automate quantum architecture search, but its scalability is limited when useful circuit trajectories become rare in the rapidly expanding search space. Existing replay mechanisms reuse observed transitions; the proposed learned model produces additional predicted one step transitions from real state-action seeds. Here we introduce GenQAS, a tensor network-guided RL framework that combines a fixed matrix product state warm-start with prioritized generative replay. A learned local transition model generates synthetic circuit transitions on demand and mixes them with real experience during Double Deep Q-Network updates. Under a random exploration analysis, near ground state circuits occupy a rapidly shrinking region of the accessible state space. We investigate whether real data anchored synthetic replay can improve the effective training signal in this regime. Across chemical Hamiltonian benchmarks from 6 to 12 qubits, GenQAS improves fixed-budget success probability and identifies compact circuits at competitive energy error. At 12 qubits, it improves final success probability by up to $7.0\times$ over passive replay. On a 15-qubit transverse field Ising model, GenQAS increases success probability from $12\%$ to $21\%$. In a noisy 6-qubit BeH$_2$ transfer experiment, generative replay reduces the steps to chemical accuracy by $92.7\%$. These results show that generative replay can mitigate sample starvation in quantum architecture search and support more resource efficient circuit discovery.
Tags
Links
- Source: https://arxiv.org/abs/2609.11248v1
- Canonical: https://arxiv.org/abs/2609.11248v1
Trouble viewing inline? Open PDF directly →
Full Text
149,392 characters extracted from source content.
Expand or collapse full text
Generative Replay Mitigates Sample Starvation in Quantum Architecture Search Akash Kundu Email: a.kundu@tudelft.nl Affiliation: Delft University of Technology, Delft, The Netherlands Affiliation: Quantum Computing Division, QuTech, Delft, The Netherlands Amit Kumar Jaiswal Email: amit.chr@iitbhu.ac.in Affiliation: Indian Institute of Technology (BHU), Varanasi, India Sebastian Feld Email: s.feld@tidelft.nl Affiliation: Delft University of Technology, Delft, The Netherlands Affiliation: Quantum Computing Division, QuTech, Delft, The Netherlands Prayag Tiwari Email: prayag.tiwari@h.se Affiliation: Halmstad University, Halmstad, Sweden Abstract Reinforcement learning (RL) can automate quantum architecture search, but its scalability is limited when useful circuit trajectories become rare in the rapidly expanding search space. Existing replay mechanisms reuse observed transitions; the proposed learned model produces additional predicted one step transitions from real state–action seeds. Here we introduce GenQAS, a tensor network-guided RL framework that combines a fixed matrix product state warm-start with prioritized generative replay. A learned local transition model generates synthetic circuit transitions on demand and mixes them with real experience during Double Deep Q-Network updates. Under a random exploration analysis, near ground state circuits occupy a rapidly shrinking region of the accessible state space. We investigate whether real data anchored synthetic replay can improve the effective training signal in this regime. Across chemical Hamiltonian benchmarks from 6 to 12 qubits, GenQAS improves fixed-budget success probability and identifies compact circuits at competitive energy error. At 12 qubits, it improves final success probability by up to 7.0×7.0× over passive replay. On a 15-qubit transverse field Ising model, GenQAS increases success probability from 12%12\% to 21%21\%. In a noisy 6-qubit BeH2 transfer experiment, generative replay reduces the steps to chemical accuracy by 92.7%92.7\%. These results show that generative replay can mitigate sample starvation in quantum architecture search and support more resource efficient circuit discovery. keywordsConditional generative model, Quantum architecture search, Replay buffer, Reinforcement learning, Tensor network †equal-contributors: These authors contributed equally to this work.†equal-contributors: These authors contributed equally to this work. 1 Introduction The performance of near-term and early fault tolerant quantum algorithms are limited by qubit count, imperfect gate execution and hardware noise Preskill (2018); Eisert and Preskill (2025); Katabarwa et al. (2024). Gate-based quantum processors provide a leading platform for their implementation Tacchino et al. (2020); Kjaergaard et al. (2020); Wright et al. (2019); Fernández de Fuentes et al. (2026); Wang et al. (2024); Petit et al. (2022); Tosato et al. (2026). However, increasing the number of physical qubits alone does not guarantee practical utility, as achievable performance also depends on error rates, connectivity, control quality, and the reliably executable circuit depth Blume-Kohout and Young (2020); Cross et al. (2019). This issue is particularly important for variational quantum algorithms Cerezo et al. (2021), in which a parameterized quantum circuit (PQC) U(,)U( θ,C), defined by a discrete architecture C and continuous parameters θ, prepares the state |ψ(,)⟩=U(,)|0⟩⊗N ψ( θ,C)=U( θ,C) 0 N. The parameters are optimized by minimizing an objective such as C(,)=⟨ψ(,)|H|ψ(,)⟩C( θ,C)= ψ( θ,C)H ψ( θ,C) utilizing powerful classical optimizers accessed through SciPy Virtanen et al. (2020). The structure of a PQC strongly influences the accuracy, trainability, and hardware cost of a VQAs. Two broad PQC design strategies are commonly used in literature. Problem-inspired ansätze encode prior knowledge of the target problem, for example by constructing parameterized circuits from Trotterized evolutions under terms of the problem Hamiltonian, as in the Hamiltonian variational ansatz Wecker et al. (2015), or from physically motivated excitation operators, as in unitary coupled-cluster Anand et al. (2022) and adaptive VQE methods Grimsley et al. (2019). In contrast, hardware-efficient architectures consist of layers of native single- and two-qubit operations arranged to match device connectivity and minimize compilation overhead. The architectures are guided either by problem structure Peruzzo et al. (2014) or hardware constraints Kandala et al. (2017). Figure 1: GenQAS converts passive replay into active experience generation. a, Under weakly informed exploration, high-quality circuit trajectories become increasingly sparse as the system size grows. b, Uniform and prioritized replay can only resample previously observed transitions, so the density of useful experience declines as the search space expands. c, GenQAS starts from an MPS-derived circuit, retains real transitions, learns a local transition model and generates synthetic transitions on the fly. Real and generated samples are mixed during Double-DQN updates, supporting the search for low-error circuits with fewer entangling gates. Although such fixed PQC architecutre can be effective, they may be either insufficiently expressive to access the relevant region of Hilbert space and often stuck in the local minima. To address these limitations a third way was introduced through quantum architecture search (QAS) Zhang et al. (2022); Du et al. (2022); Kuo et al. (2021). The core mechanism of QAS is to automatically constructing PQC architectures from a prescribed gate set, enabling the circuit structure to adapt to the target task and hardware constraints. Reinforcement learning (RL) has emerged as a versatile framework for quantum technologies, including the adaptive control of quantum error correction, where an agent uses error-detection events to continuously adjust hardware control parameters during computation Sivak et al. (2026). RL also provides a natural formulation for automated quantum circuit design and optimization. In the early fault tolerant setting, AlphaTensor-Quantum Ruiz et al. (2025) formulates the optimization of Clifford+T circuits as a symmetric tensor-decomposition problem and uses RL to search for low-rank decompositions corresponding to circuits with reduced T-count. In VQAs, RL can instead be used for QAS, in which an agent sequentially constructs a parameterized quantum circuit by selecting gate types and their locations Ostaszewski et al. (2021); Kuo et al. (2021); Fösel et al. (2021); Moro et al. (2021); Zhang et al. (2020). This flexibility creates a rapidly expanding decision space. Let 1G_1 and 2G_2 denote the available single- and two-qubit gate families, respectively, and let ℰNE_N denote the set of hardware-allowed two-qubit couplings on an N-qubit device. The number of available actions at each circuit-construction step is |N|=|1|N⏟1-qubit+|2||ℰN|⏟2-qubit+1, _N = _1 N_1-qubit+ _2 _N _2-qubit+1, (1) where the final term denotes termination. For a fully connected device with undirected couplings, |ℰN|=N(N−1)/2 _N =N(N-1)/2. Consequently, a construction process of maximum depth T can encounter up to |N|T _N ^T gate sequences, while the dimension of the underlying Hilbert space grows as 2N2^N. Evaluating each candidate architecture generally further requires continuous parameter optimization and repeated objective estimation using a quantum simulator or processor. RL-QAS must therefore learn from a limited number of costly interactions within a search space that grows combinatorially with circuit depth and rapidly with system size. For overviews of RL-based quantum circuit optimization and design, we refer the reader to Refs. Kundu et al. (2026); Bukov and Marquardt (2026). Recent methods mitigate different aspects of this scaling challenge. Curriculum-based RL-QAS adapts the difficulty of the search task during training Patel et al. (2024), while training-free approaches use proxy metrics to prioritize candidate architectures before costly variational evaluation He et al. (2024). One-shot methods, such as QuantumNAS Wang et al. (2022), reuse parameters across candidate architectures and incorporate hardware-aware objectives. These approaches reduce the cost of evaluating or prioritizing candidate circuits, but do not directly augment the transition data available to an RL agent during sequential architecture construction. TensorRL-QAS Kundu and Mangini (2026) addresses a complementary aspect of scalability through physically informed initialization. It uses a matrix product state (MPS), obtained with density matrix renormalization group calculations, to construct a problem-informed initial circuit that is subsequently refined by a double deep Q-network agent. The MPS warm start places the agent in a more relevant region of circuit space and reduces exploration of clearly unpromising architectures. However, after initialization, the agent still learns only from transitions explicitly collected through interactions with the environment. GenQAS addresses this remaining limitation by learning a generative transition model from real experience and using it to produce on-demand synthetic transitions for DDQN updates, thereby increasing the effective training signal without additional quantum-classical circuit evaluations. Experience replay is commonly used to improve sample efficiency by reusing these interactions. Uniform replay samples stored transitions without regard to their learning value, whereas prioritized experience replay increases the sampling frequency of transitions associated with a larger temporal-difference error or another relevance signal Mnih et al. (2013); Schaul et al. (2015). Prioritization can therefore make better use of a useful transition after it has been observed. Nevertheless, both approaches remain passive with respect to the set of experiences available for learning: they can change how frequently an observed transition is reused, but they cannot create informative experience that was never collected. As the circuit search space expands, useful trajectories constitute an increasingly small fraction of the agent’s interaction history, and the learning signal becomes diluted by a much larger number of low-value alternatives. We refer to this decline in the density of informative experience as sample starvation. Our theoretical analysis formalizes this limitation under the assumptions stated in the Results. For weakly informed exploration over sufficiently expressive circuit ensembles, we show that the measure of near ground state experience decreases exponentially with the dimension of the accessible Hilbert space. This statement characterizes the geometry of random exploration rather than assuming that every learned policy follows the Haar distribution. It nevertheless exposes an important limitation of passive replay during the early stages of search: reweighting previously collected transitions cannot alter the probability that informative transitions were encountered in the first place. One approach to mitigating this limitation is to augment, rather than only reorder, the transitions used in value function updates. Generative replay provides such a mechanism by learning a model of the local transition dynamics from real interactions Sutton (1991); Ha and Schmidhuber (2018); Janner et al. (2019); Feinberg et al. (2018); Wang et al. (2025). Quantum circuit construction is particularly suitable for this approach because the current circuit and the selected gate constrain the subsequent circuit state, the associated reward and whether the construction process terminates. A learned transition model can therefore generate additional training samples around observed state-action pairs without requiring a new quantum-classical evaluation for every generated transition. The main challenge is model bias: synthetic transitions are beneficial only when they remain sufficiently consistent with the real environment to support stable off-policy learning. Generative replay must consequently remain anchored in real experience and control how strongly generated samples influence policy optimization. Here we introduce GenQAS, a tensor network-guided reinforcement learning framework that augments passive experience replay with on-demand synthetic transitions for QAS (Fig. 1). GenQAS combines the fixed matrix product state (MPS) initialization and Double-DQN search backbone of TensorRL-QAS with a learned local transition model. Given a circuit state and gate action, the model predicts the resulting circuit state, reward and termination probability. These predictions define synthetic one-step transitions that are generated from real state-action seeds and mixed with real transitions during value-function updates. The MPS initialization and generative replay provide complementary inductive biases. The MPS-derived circuit places the agent in a physically meaningful low-energy region of circuit space, whereas generative replay increases the amount of training signal extracted from subsequent interactions. Crucially, synthetic transitions are generated on demand and are not stored as replacements for real experience. The replay buffer therefore remains grounded in environment interactions, while the generation ratio GrG_r controls the contribution of synthetic experience to each update. This design seeks to improve sample efficiency without requiring additional quantum-classical parameter optimization calls. Under the random exploration and Hamiltonian assumptions considered here, we show that near ground state circuit trajectories become rapidly sparse with increasing system size This creates a regime in which resampling observed transitions alone does not increase the support of the observed transition distribution. Within this analytical setting, MPS-seeded generative replay improves the effective density of informative training samples relative to uniform and prioritized replay. The benefit depends on maintaining sufficient agreement between the learned transition model and the underlying QAS environment, thereby motivating the use of real transitions as the persistent replay distribution. We evaluate GenQAS under matched training budgets on 6-qubit BeH2BeH_2, 8-, 10- and 12-qubit H2OH_2O, and a 15-qubit transverse-field Ising model. At 12 qubits, GenQAS increases final success probability from 12%12\% for the strongest passive baseline to 77%77\%, corresponding to a 7.0×7.0× improvement. At 15 qubits, it improves the best success probability from 15%15\% with prioritized replay to 21%21\%, a 40% relative increase. Across the tested systems, GenQAS also identifies circuits with competitive energy errors, reduced CNOT counts and circuit depth. In a noisy 6-qubit BeH2BeH_2 experiment, transfer of the generative replay mechanism reduces the number of search steps required to obtain the first chemical-accuracy solution from 90539053 to 660660. We further evaluate GenQAS on a 4-qubit Clifford-synthesis task, in which the agent must construct a circuit matching a target Clifford operation without variational parameter optimization. In this fully discrete setting, GenQAS exceeds the performance of the passive RL baseline, and the generation frequency provides an additional control over the trade-off between synthetic experience and stable off-policy learning. Together, these results show that model-augmented replay can mitigate sample starvation in RL-QAS and can improve circuit discovery when informative environment interactions are sparse and expensive. 2 Results Quantum architecture search becomes increasingly difficult as the number of qubits grows because both the circuit construction space and the Hilbert space expand rapidly. Our theoretical analysis formalizes one aspect of this difficulty. Under the weakly informed exploration and Hamiltonian assumptions specified in Section 4.3, Theorems 4.1 and 4.2 show that the probability of encountering an ϵε-accurate low energy circuit through passive exploration decreases rapidly with system size. Consequently, uniform replay and prioritized experience replay can only resample a progressively smaller set of useful transitions. Lemmas 4.4 and 4.5 establish that the MPS derived warm start provides a controlled initial circuit approximation under the stated assumptions, while Theorem 4.6 characterizes the potential reduction in episode complexity when model generated transitions remain concentrated in a high fidelity region of the search space. Detailed proofs, assumptions, and derivations are provided in Supplementary Information Section 5. The experiments below test whether the proposed generative replay mechanism produces the corresponding practical benefits in learning speed, circuit quality, robustness to noise, and generalization beyond molecular ground state preparation. We compare GenQAS with TensorRL-QAS using uniform and prioritized replay, as well as with hardware efficient ansatz and UCCSD baselines where applicable. Molecular benchmarks include 6 qubit BEH2 BEH_2 and 8, 10, and 12 qubit H2O H_2 O. We additionally consider a 15 qubit transverse field Ising model and a 4 qubit Clifford synthesis task. Within each benchmark, all RL methods use the same environment, tensor network warm start, action space, and benchmark specific training budget. They differ only in the replay mechanism. Full hyperparameters, molecular Hamiltonian specifications, and simulation details are given in Supplementary Information Sections 8, 8.3, and 8.4. (a) Scaling performance over 5 random initializations of neural network. (b) Training dynamics. Figure 2: Fixed-budget performance and training dynamics of GenQAS. (a) At a fixed budget of 60006000 training episodes, GenQAS consistently outperforms TensorRL-QAS with uniform and PER replay across N∈6,8,10,12N∈\6,8,10,12\, while passive baselines degrade sharply with system size. The observed gains increase with qubit number, reaching the strongest advantage at 12 qubits and showing an increasing relative advantage over the evaluated 6 to 12 qubit range, consistent with the proposed signal-density mechanism. (b) Training trajectories for 6, 8, 10 and 12 qubits show that GenQAS converges faster and attains higher success probability than TensorRL-QAS baselines. The Gr=0.8G_r=0.8 setting is the most robust at larger system sizes, maintaining high success even at 12 qubits. 2.1 Generative replay improves fixed budget scaling Theorem 4.6 predicts that, under the assumptions of the theoretical analysis, generative replay can reduce the number of environment interactions required to reach a target accuracy relative to passive replay. Here, we test the empirical consequence of this prediction under matched training budgets at each system size: whether the advantage of GenQAS over passive replay becomes more pronounced as the QAS problem increases from 6 to 12 qubits. At each DDQN update, GenQAS mixes real transitions from replay memory with synthetic transitions generated on demand by the learned dynamics model. The generation ratio Gr∈[0,1)G_r∈[0,1) specifies the synthetic fraction of a total update batch of size B: Breal=⌊(1−Gr)B⌋,Bsyn=B−Breal.B_real= (1-G_r)B , B_syn=B-B_real. (2) Thus, Gr=0.4G_r=0.4 produces an update batch with approximately 60%60\% real and 40%40\% synthetic transitions, whereas Gr=0.8G_r=0.8 produces approximately 20%20\% real and 80%80\% synthetic transitions. Synthetic samples are generated from state action seeds drawn from real replay memory and are not stored persistently. The theoretical mechanism can be expressed through the useful-transition density in an update batch. If ρreal(N) _real(N) and ρsyn(N) _syn(N) denote the densities of useful real and synthetic transitions, respectively, then the mixed update distribution has density ρmix(N)=(1−Gr)ρreal(N)+Grρsyn(N). _mix(N)=(1-G_r) _real(N)+G_r _syn(N). (3) Under the assumptions of the theoretical analysis, Theorems 4.1 and 4.2 show that ρreal(N) _real(N) decreases rapidly under passive exploration. Lemma 4.5 establishes a fidelity bound for the MPS derived warm start, while Theorem 4.6 characterizes the conditions under which generative replay can maintain a nonvanishing useful-transition contribution to the mixed update distribution. The full signal-density derivation and its assumptions are provided in Supplementary Information Section 8.5. Figure 2(a) reports the final success probability across 6, 8, 10, and 12 qubits, averaged over five random neural-network initializations. At 6 qubits, TensorRL-QAS with uniform and prioritized replay reaches success probabilities of approximately 0.570.57 and 0.600.60, respectively. GenQAS with Gr=0.4G_r=0.4 reaches 0.880.88, whereas the more synthetic configuration, Gr=0.8G_r=0.8, reaches 0.280.28. At 8 qubits, the passive baselines reach approximately 0.540.54 and 0.550.55, while GenQAS reaches 0.840.84 for Gr=0.4G_r=0.4 and 0.340.34 for Gr=0.8G_r=0.8. The same qualitative pattern persists at 10 qubits. The uniform and prioritized TensorRL-QAS baselines attain final success probabilities of approximately 0.510.51 and 0.550.55, respectively. GenQAS with Gr=0.4G_r=0.4 reaches 0.840.84, corresponding to an approximately 1.5×1.5× improvement over the strongest passive baseline, whereas GenQAS with Gr=0.8G_r=0.8 reaches 0.370.37. These 6 to 10 qubit results show that an intermediate synthetic fraction provides the strongest performance when real transitions remain sufficiently available to anchor value learning. At 12 qubits, the passive replay baselines deteriorate sharply, reaching success probabilities of approximately 0.100.10 and 0.110.11. In contrast, GenQAS with Gr=0.4G_r=0.4 attains 0.380.38, corresponding to an approximately 3.4×3.4× improvement over the strongest passive baseline. The Gr=0.8G_r=0.8 configuration reaches 0.770.77, corresponding to an approximately 7.0×7.0× improvement. Thus, the generation ratio that is suboptimal at smaller system sizes becomes the strongest configuration when the useful real-transition density is most severely depleted. The fixed budget advantage over the evaluated instances shown in Figure 2(a) is therefore not strictly monotonic in GrG_r. At 6, 8, and 10 qubits, Gr=0.4G_r=0.4 provides the highest final success probability, indicating that a substantial real-data component remains beneficial. At 12 qubits, however, Gr=0.8G_r=0.8 becomes superior, consistent with the hypothesis that a larger synthetic contribution is increasingly valuable once passive replay enters a stronger sample-starvation regime. Although the four investigated system sizes are insufficient to establish an asymptotic scaling law empirically, the increasing relative advantage of GenQAS is consistent with the signal-density mechanism in Eq. (3). Figure 2(b) shows the corresponding learning trajectories. At 6 qubits, GenQAS with Gr=0.4G_r=0.4 rapidly improves after approximately 3000 episodes and reaches the highest final success probability. At 8 and 10 qubits, the same configuration again converges faster and reaches a higher final plateau than uniform replay, prioritized replay, and the Gr=0.8G_r=0.8 configuration. At 12 qubits, the learning dynamics change qualitatively: the passive baselines remain near zero success probability throughout most of training, while GenQAS with Gr=0.8G_r=0.8 steadily improves and reaches the highest final success probability. These trajectories show that the preferred real to synthetic replay mixture depends on problem size, with higher generation ratios becoming advantageous in the largest QAS instance examined. 2.2 Generative replay improves noisy replay buffer transfer We next evaluate whether replay information collected in a noiseless setting remains useful when the same QAS task is evaluated under gate noise. We consider the 6-qubit BeH2 benchmark and follow the replay buffer transfer setting Kundu and Feld (2026). TensorRL-QAS and GenQAS are first trained in the noiseless environment using the same reward construction and training protocol as in the molecular experiments. The resulting replay memories are subsequently used to initialize training in a noisy environment. Only the replay-buffer information is transferred, network weights and an additional ϵε-greedy pretraining phase are not transferred. The noisy environment applies a depolarizing channel after every gate. Each single-qubit gate is followed by depolarizing noise of strength p1=0.01p_1=0.01, and each two-qubit gate is followed by depolarizing noise of strength p2=0.05p_2=0.05. TensorRL-QAS uses a transferred uniform replay buffer, whereas GenQAS uses the transferred prioritized generative replay mechanism trained with Gr=0.4G_r=0.4. (a) Relative improvement after noiseless replay-buffer transfer. Run Steps to chemical accuracy CX ROT Depth TensorRL-QAS Kundu and Mangini (2026) 13 623 13 4 10 TensorRL-QAS with transfer 9 520 9 4 10 GenQAS 9 053 9 9 8 GenQAS with transfer 660 6 3 6 (b) Raw metrics for the noisy 6-qubit BeH2 task. Figure 3: Replay buffer transfer in the noisy 6-qubit BEH2 BEH_2 experiment. A depolarizing channel with strength p1=0.01p_1=0.01 is applied after each single-qubit gate and a channel with strength p2=0.05p_2=0.05 is applied after each two-qubit gate. (a) Relative improvement in steps to chemical accuracy, CNOT count, rotation count, and circuit depth after transfer of replay information collected in the noiseles environment. (b) Corresponding raw metrics for TensorRL-QAS and GenQAS with and without replay-buffer transfer. Figure 3 shows that transferred replay information improves both methods, but the improvement is substantially larger for GenQAS. For TensorRL-QAS, transfer of the uniform replay buffer reduces the steps to the first chemical-accuracy solution from 13623 to 9520, a 30.1%30.1\% reduction. The transferred buffer also reduces the CNOT count from 13 to 9, a 30.8%30.8\% reduction, but leaves the rotation count and circuit depth unchanged at 4 and 10, respectively. For GenQAS, transfer of the generative replay mechanism reduces the steps to chemical accuracy from 9053 to 660, corresponding to a 92.7%92.7\% reduction. The transferred GenQAS configuration also decreases the CNOT count from 9 to 6, the rotation count from 9 to 3, and the circuit depth from 8 to 6. These correspond to relative reductions of 33.3%33.3\%, 66.7%66.7\%, and 25.0%25.0\%, respectively. The result shows that, in this noiseless to noisy transfer experiment, the GenQAS replay representation retains more useful information for subsequent noisy architecture search than a passive uniform replay buffer. In particular, the transferred GenQAS configuration reaches chemical accuracy with fewer search steps while also identifying a more compact circuit. This finding is consistent with the hypothesis that model-generated replay can retain locally useful circuit-transition structure across a moderate change in the evaluation environment. It does not establish noise-model independence, but it demonstrates a substantial transfer benefit under the depolarizing noise setting considered here. 2.3 Generative replay diagnostics reveal a generation-ratio trade-off The fixed-budget results show that generative replay can improve policy performance, particularly as useful real transitions become sparse. However, synthetic replay is beneficial only when the learned transition model remains sufficiently accurate for stable value learning. We therefore examine the generator losses and weighted DDQN Huber losses at 6 and 10 qubits for generation ratios Gr∈0.4,0.6,0.8G_r∈\0.4,0.6,0.8\. These diagnostics do not directly measure the density of low-energy synthetic transitions. Instead, they assess whether the learned dynamics model and value function remain numerically well behaved under different fractions of generated replay. Figure 4: Generator loss decomposition at 6 and 10 qubits. Total, transition, reward, and termination losses of the learned dynamics model for generation ratios Gr∈0.4,0.6,0.8G_r∈\0.4,0.6,0.8\. The transition loss decreases or remains low across the investigated settings. At 10 qubits, the Gr=0.4G_r=0.4 configuration exhibits a late increase in total loss, whereas the Gr=0.6G_r=0.6 and Gr=0.8G_r=0.8 configurations remain more bounded over their respective training intervals. Figure 5: Weighted DDQN Huber loss under different generation ratios. Replay Q-learning loss over training steps at 6 and 10 qubits for Gr∈0.4,0.6,0.8G_r∈\0.4,0.6,0.8\. At 6 qubits, all configurations remain in a low bounded range after the initial training stage. At 10 qubits, Gr=0.4G_r=0.4 exhibits late high-variance loss spikes, whereas the Gr=0.6G_r=0.6 and Gr=0.8G_r=0.8 configurations remain comparatively more stable over the displayed training intervals. Figure 4 decomposes the dynamics-model objective into transition, reward, and termination components. At 6 qubits, the transition loss decreases substantially for all generation ratios, reaching values near 10−310^-3, while the total loss remains bounded. The reward and termination components exhibit stochastic fluctuations, but no sustained divergence is observed. These results show that the local transition model can be trained stably while generated replay constitutes between 40%40\% and 80%80\% of each DDQN update batch. At 10 qubits, the dependence on GrG_r is more pronounced. For Gr=0.6G_r=0.6 and Gr=0.8G_r=0.8, the transition loss remains low and the total generator loss stays within a relatively bounded range. In contrast, the Gr=0.4G_r=0.4 configuration shows a late increase in the total loss, driven primarily by the reward and termination components. This behaviour indicates that generator stability is not determined by the generation ratio alone; it also depends on the evolving state distribution, reward landscape, and training horizon of the underlying QAS task. Figure 5 reports the weighted DDQN Huber loss computed from mixed batches of real and synthetic transitions. At 6 qubits, the losses for all three generation ratios decrease after the initial learning regime and remain within a low range. The Gr=0.8G_r=0.8 setting displays moderately larger early fluctuations, but its loss remains bounded over the reported training interval. At 10 qubits, the Gr=0.4G_r=0.4 configuration develops large late-stage Q-loss spikes. By comparison, the Gr=0.6G_r=0.6 and Gr=0.8G_r=0.8 configurations retain a lower and more bounded loss profile, apart from isolated fluctuations. The loss diagnostics therefore identify a generation-ratio trade-off: a moderate synthetic fraction can yield strong final success probability, as observed in the 10-qubit scaling experiment, while a larger synthetic fraction can provide a more stable critic-loss trajectory in the later stages of training. Together, Figures 4 and 5 show that on-demand synthetic replay can be incorporated into DDQN training without systematic divergence across the investigated settings. They also show that the relative stability of different generation ratios changes with system size. This motivates treating GrG_r as a task-dependent control parameter, rather than assuming that a fixed real-to-synthetic replay mixture is optimal across all QAS instances. Table 1: Circuit statistics for BeH2 (6 qubits) and H2O (8, 10, and 12 qubits). For each configuration we report two-qubit gate count (CNOT), single-qubit rotation count (ROT), circuit depth, and error for the best seed. GenQAS consistently attains comparable or lower error than TensorRL-QAS, HEA, and UCCSD while using substantially fewer two-qubit gates, fewer rotations, or reduced depth. HEA errors are taken from the best of 10 random seeds per ansatz depth (1, 2, 3 layers); UCCSD statistics are from a single deterministic run. System Method CNOT ROT Depth Error 6-qubit BeH2 TensorRL-QAS (Uniform) 9 9 9 7.22×10−57.22× 10^-5 TensorRL-QAS (PER) 16 3 14 7.89×10−57.89× 10^-5 GenQAS (Gr=0.2G_r=0.2) 14 5 14 7.62×10−57.62× 10^-5 GenQAS (Gr=0.4G_r=0.4) 5 2 6 7.18×10−57.18× 10^-5 GenQAS (Gr=0.6G_r=0.6) 5 10 5 2.37×10−52.37× 10^-5 GenQAS (Gr=0.8G_r=0.8) 4 12 6 7.59×10−57.59× 10^-5 HEA (1-layer) 6 24 10 5.92×10−35.92× 10^-3 HEA (2-layer) 12 36 18 1.98×10−31.98× 10^-3 HEA (3-layer) 18 48 26 3.37×10−43.37× 10^-4 UCCSD 272 620 475 5.92×10−35.92× 10^-3 8-qubit H2O TensorRL-QAS (Uniform) 14 6 8 1.24×10−31.24× 10^-3 TensorRL-QAS (PER) 14 2 11 1.26×10−31.26× 10^-3 GenQAS (Gr=0.2G_r=0.2) 12 2 10 2.22×10−32.22× 10^-3 GenQAS (Gr=0.4G_r=0.4) 3 2 4 1.19×10−31.19× 10^-3 GenQAS (Gr=0.6G_r=0.6) 7 10 10 1.22×10−31.22× 10^-3 GenQAS (Gr=0.8G_r=0.8) 2 13 8 1.20×10−31.20× 10^-3 HEA (1-layer) 8 32 12 2.63×10−32.63× 10^-3 HEA (2-layer) 16 48 22 2.63×10−32.63× 10^-3 HEA (3-layer) 24 64 32 2.96×10−32.96× 10^-3 UCCSD 1312 2596 2148 2.63×10−32.63× 10^-3 10-qubit H2O TensorRL-QAS (Uniform) 15 17 17 4.15×10−44.15× 10^-4 TensorRL-QAS (PER) 23 14 19 4.17×10−44.17× 10^-4 GenQAS (Gr=0.2G_r=0.2) 4 5 7 4.17×10−44.17× 10^-4 GenQAS (Gr=0.4G_r=0.4) 14 15 20 3.92×10−43.92× 10^-4 GenQAS (Gr=0.6G_r=0.6) 19 19 20 3.93×10−43.93× 10^-4 GenQAS (Gr=0.8G_r=0.8) 26 36 24 4.32×10−44.32× 10^-4 HEA (1-layer) 10 40 14 4.69×10−34.69× 10^-3 HEA (2-layer) 20 60 26 4.84×10−34.84× 10^-3 HEA (3-layer) 30 80 38 1.14×10−21.14× 10^-2 UCCSD 3440 5932 5336 4.69×10−34.69× 10^-3 12-qubit H2O TensorRL-QAS (Uniform) 18 2 6 2.22×10−22.22× 10^-2 TensorRL-QAS (PER) 17 2 11 2.48×10−22.48× 10^-2 GenQAS (Gr=0.2G_r=0.2) 14 2 10 2.33×10−22.33× 10^-2 GenQAS (Gr=0.4G_r=0.4) 18 2 4 2.28×10−22.28× 10^-2 GenQAS (Gr=0.6G_r=0.6) 3 10 10 2.45×10−22.45× 10^-2 GenQAS (Gr=0.8G_r=0.8) 2 4 2 2.44×10−22.44× 10^-2 HEA (1-layer) 12 48 16 2.52×10−22.52× 10^-2 HEA (2-layer) 24 72 30 3.29×10−23.29× 10^-2 HEA (3-layer) 36 96 44 6.31×10−26.31× 10^-2 UCCSD 6976 10628 10356 2.51×10−22.51× 10^-2 2.4 GenQAS identifies compact circuits at competitive error Table 1 compares GenQAS with TensorRL-QAS using uniform and prioritized experience replay, as well as with fixed hardware-efficient ansatz and UCCSD reference circuits. The RL based methods share the same MPS derived warm start, DDQN backbone, environment, and action space. Thus, the comparison between GenQAS and TensorRL-QAS isolates the effect of replacing passive replay with prioritized generative replay. For each RL configuration, the table reports the most accurate circuit among the evaluated random seeds Across the molecular benchmarks, GenQAS identifies circuits on a competitive accuracy-resource frontier. At 6 qubits, GenQAS with Gr=0.6G_r=0.6 achieves the lowest reported error, 2.37×10−52.37× 10^-5, at depth 5 using 5 CNOTs. At 8 qubits, GenQAS with Gr=0.4G_r=0.4 reaches an error of 1.19×10−31.19× 10^-3 using 3 CNOTs, 2 rotations, and depth 4, compared with 14 CNOTs, 6 rotations, and depth 8 for TensorRL-QAS with uniform replay. At 10 qubits, GenQAS with Gr=0.2G_r=0.2 matches the error of prioritized replay while using 4 CNOTs, 5 rotations, and depth 7. The 12 qubit H2O task illustrates the accuracy-resource trade-off. Uniform TensorRL-QAS reaches the lowest reported error, 2.22×10−22.22× 10^-2, whereas GenQAS with Gr=0.8G_r=0.8 identifies the most compact circuit, using 2 CNOTs, 4 rotations, and depth 2 with error 2.44×10−22.44× 10^-2. GenQAS with Gr=0.4G_r=0.4 achieves an intermediate solution with error 2.28×10−22.28× 10^-2 and depth 4. Thus, generative replay does not uniformly dominate passive replay for every individual metric; instead, it provides a practical means of navigating the trade-off between accuracy and circuit resources. The fixed HEA and UCCSD circuits provide reference points for the size of the resource savings. At 12 qubits, UCCSD requires 6976 CNOTs, 10628 single-qubit rotations, and depth 10356, whereas the GenQAS circuits use at most tens of gates and depth at most 10. HEA circuits are substantially more compact than UCCSD but remain less gate efficient than the compact GenQAS solutions at comparable error. These comparisons indicate that generative replay can support the discovery of low resource circuit architectures under the same RL interaction budget. 2.5 Generative replay generalizes across quantum optimization tasks The molecular benchmarks establish that generative replay can improve fixed-budget learning and identify compact circuits for electronic structure Hamiltonians. We next test whether this behaviour extends beyond molecular ground state preparation. We consider a 15-qubit transverse field Ising model (TFIM), which preserves the variational ground state objective but changes the physical problem class, and a 4-qubit Clifford synthesis task, which replaces variational energy minimization with exact discrete target matching. Spin-model ground state preparation For the 15 qubit TFIM, we evaluate GenQAS using the TensorRL-QAS backbone under a matched training budget. Figure 6(a) shows that GenQAS attains equal or higher final success probability than TensorRL with uniform or prioritized replay for the tested generation ratios. The final energy errors remain in the narrow range from 1.53×10−21.53× 10^-2 to 1.54×10−21.54× 10^-2 across all methods, indicating that the replay modification does not degrade final solution quality in this benchmark. The generation ratio determines the resource profile of the resulting circuits. GenQAS with Gr=0.4G_r=0.4 matches the error and CNOT count of the uniform TensorRL baseline while reducing the rotation count from 8 to 6. GenQAS with Gr=0.8G_r=0.8 identifies a circuit with only 3 CNOTs at comparable error, although it uses 13 single-qubit rotations. Thus, GrG_r provides a (a) Fixed-budget performance. Method Error CNOT ROT TensorRL (uniform) 1.53×10−21.53× 10^-2 10 8 TensorRL (PER) 1.54×10−21.54× 10^-2 10 8 GenQAS (Gr=0.2G_r=0.2) 1.53×10−21.53× 10^-2 7 10 GenQAS (Gr=0.4G_r=0.4) 1.53×10−21.53× 10^-2 10 6 GenQAS (Gr=0.6G_r=0.6) 1.54×10−21.54× 10^-2 14 6 GenQAS (Gr=0.8G_r=0.8) 1.53×10−21.53× 10^-2 3 13 (b) Circuit statistics. Figure 6: 15-qubit TFIM performance: fixed-budget success (a) and best circuit statistics (b). practical control parameter for navigating the trade-off between entangling gates, single-qubit rotations, and learning behaviour in a non-molecular ground-state problem. Discrete clifford synthesis We further evaluate GenQAS on a 4-qubit Clifford synthesis task, in which the agent sequentially constructs a circuit that exactly matches a target Clifford operation using a discrete gate library Cliff=Hq,Sq,Sq†,CNOTc,tA_Cliff= \ H_q,\, S_q,\, S_q ,\, CNOT_c,t \, where q∈0,…,N−1q∈\0,…,N-1\, and c≠tc≠ t. Unlike the molecular and TFIM benchmarks, this task requires neither variational parameter optimization nor energy expectation-value estimation. Rewards are determined by exact target matching, allowing us to test generative replay in a fully discrete circuit construction setting. The target generation procedure, action space construction, reward design, and evaluation protocol are provided in Supplementary Information Section 7. Figure 7: 4-qubit clifford synthesis. Having 4-qubit clifford synthesis a relatively easier problem to solve here we wanted to show that a small generation ratio (GrG_r) makes a big difference in performance. As we do not have the variational quantum-classical framework in this experiment, this is the exact place where we can analyse the generation frequency by the conditional generative model implemented inside the genQAS. Figure 7(a) shows that GenQAS with Gr=0.2G_r=0.2 matches or modestly exceeds the success probability of the passive RL-QAS baseline. Figure 7(b) further shows that the generator update frequency affects learning performance. The intermediate configuration gen_freq=5 gen\_freq=5 achieves the highest average cumulative reward under the shared training budget, whereas very infrequent and very frequent generation provide smaller gains. Together, the TFIM and Clifford results show that the benefit of generative replay is not restricted to molecular Hamiltonians. It can also improve sequential circuit construction in a spin-model ground-state task and in a fully discrete exact-synthesis task. At the same time, both experiments show that the generation ratio and update frequency are task-dependent hyperparameters rather than universally optimal constants. 3 Discussion This work addresses sample starvation in reinforcement learning for quantum architecture search. Under the random-exploration and Hamiltonian assumptions of our theoretical analysis, Theorems 4.1 and 4.2 show that the probability of encountering an ϵε-accurate low-energy circuit decreases rapidly as the number of qubits increases. In this regime, uniform replay and prioritized experience replay can change how often observed transitions are reused, but cannot alter the probability that informative transitions were collected in the first place. GenQAS addresses this limitation by combining a fixed MPS derived warm start with a learned local transition model that generates additional on-demand replay transitions from real state-action seeds. Under the conditions stated in Lemma 4.5 and Theorem 4.6, this mechanism can maintain a nonvanishing useful transition contribution to the mixed replay distribution. The complete signal-density derivation and its assumptions are provided in Supplementary Information Section 8.5. The empirical results in Section 2 are consistent with this mechanism. At 12 qubits, GenQAS increases the final success probability from 12%12\% for the strongest passive replay baseline to 77%77\%, corresponding to a 7.0×7.0× improvement. At 15 qubits, GenQAS reaches a success probability of 21%21\%, compared with 12%12\% for the baseline. The observed fixed-budget advantage is consistent with the proposed mechanism; the finite set of problem sizes does not establish an asymptotic scaling law. The improved success probability is accompanied by competitive accuracy resource trade-offs. Across the molecular benchmarks and the transverse field Ising model, GenQAS identifies circuits that often match passive replay energy errors while using fewer CNOT gates, rotations, or circuit layers. This should be interpreted as a Pareto trade-off rather than universal dominance on every metric. For example, at 12 qubits, uniform TensorRL-QAS attains the lowest reported energy error, whereas GenQAS identifies substantially more compact circuits at comparable error. Relative to fixed reference ansatzes, the reduction in circuit resources is substantial: for 12-qubit H2O H_2 O, UCCSD requires nearly 7000 CNOT gates, whereas the GenQAS circuits use only a small number of entangling gates. These results suggest that generative replay can guide RL based search towards lower resource regions of the circuit-design space. The noisy 6-qubit BEH2 BEH_2 transfer experiment further shows that the GenQAS replay mechanism remains useful after transfer from noiseless to depolarizing circuit evaluation. Transfer of the generative replay information reduces the steps to the first chemical accuracy solution from 9053 to 660, a 92.7%92.7\% reduction, while also reducing the CNOT count, rotation count, and circuit depth. In comparison, transfer of the uniform TensorRL-QAS replay buffer yields a smaller reduction in search steps and no reduction in rotation count or circuit depth. This result demonstrates a substantial benefit in the specific depolarizing noise setting considered here. It does not, however, establish robustness to arbitrary noise models, hardware drift, or finite-shot measurement noise. Finally, the 15-qubit TFIM and 4-qubit Clifford synthesis experiments indicate that the replay mechanism is not restricted to molecular electronic structure tasks. The TFIM benchmark retains the variational ground state objective but changes the physical problem class, while the Clifford task replaces energy minimization with exact discrete target matching. In the Clifford benchmark, GenQAS matches or modestly exceeds the passive RL baseline, and an intermediate generator update frequency yields the highest cumulative reward. These results suggest that generative replay can be useful for both variational ground state preparation and fully discrete circuit-construction tasks. Several limitations remain. The present study uses state-vector simulation, a simple depolarizing-noise model, and a limited range of system sizes. The MPS warm-start analysis also relies on locality, spectral, and entanglement assumptions that need not apply to arbitrary Hamiltonians. In addition, the learned transition model is evaluated through local replay losses and downstream policy performance rather than through a direct long-horizon model accuracy analysis. Future work should test finite-shot and hardware evaluations, hardware-specific connectivity constraints, larger system sizes, cross-task transfer, and uncertainty-aware generative models for controlling model bias. A further limitation is that GenQAS has not yet been evaluated for quantum error correction or fault-tolerant circuit discovery. Such tasks require an environment that represents syndrome information, physical error propagation, ancilla resources, connectivity constraints, and explicit fault-tolerance criteria, rather than a variational energy objective alone. Recent work has shown that reinforcement learning can discover compact fault-tolerant logical state preparation circuits under hardware constraints Zen et al. (2025). Extending GenQAS to this setting would test whether generative replay can improve the sample efficiency of searching over sparse, constraint heavy fault-tolerant circuit spaces. In addition, the present GenQAS implementation operates with a fixed elementary action space. This restricts the agent to discovering useful higher-level circuit motifs through repeated low-level gate insertions. An important direction is to augment the action space with learned composite operations, or gadgets, extracted from high-performing circuits on simpler instances. Gadget-based reinforcement learning has shown that reusable circuit fragments can be transferred to more difficult optimization problems by dynamically expanding the agent’s action space Kundu et al. (2026). Combining GenQAS with learned gadgets could reduce the effective search horizon and provide a complementary route to scalability. However, this extension would require principled gadget selection, hardware-aware decomposition, and controls against an excessively large or poorly structured action space. 4 Methods GenQAS combines the fixed tensor network initialization of TensorRL-QAS Kundu and Mangini (2026) with prioritized generative replay Wang et al. (2025). The tensor network module provides a problem informed fixed circuit prefix, whereas the model based replay mechanism generates synthetic transitions to improve the sample efficiency of reinforcement learning based quantum architecture search (QAS). The MPS derived warm start circuit is compiled into the native gate set and retained as a fixed prefix throughout training. GenQAS subsequently refines this initialization across all evaluated systems. For example, for the 15 qubit transverse field Ising model, GenQAS reduces the warm start energy error from 1.0251.025 to 1.5×10−21.5× 10^-2, corresponding to a 98.5%98.5\% relative reduction in energy error. The construction, resource cost, and refinement of the fixed warm start are detailed in Supplementary Information Sections 6 and 6.6. 4.1 GenQAS We formulate QAS as a Markov decision process in which the state sts_t encodes the circuit constructed by the agent at step t, and an action ata_t specifies the insertion of a gate at a valid circuit location. States use the discrete circuit representation introduced in TensorRL-QAS Kundu and Mangini (2026), defined over an N qubit, D layer brickwork layout Leone et al. (2024); Kandala et al. (2017). Optional continuous gate angles and auxiliary descriptors, such as the current energy estimate, can be appended to this representation. GenQAS adopts the fixed TensorRL-QAS initialization. Specifically, a tensor network derived circuit UTNU_TN prepares a fixed warm start state, while the agent constructs a learnable circuit suffix VtV_t. The circuit evaluated by the environment is Ut=VtUTN.U_t=V_tU_TN. (4) The warm start circuit is not encoded in the RL state and its parameters remain fixed throughout training. Consequently, the agent observes and searches only over the appended architecture VtV_t. This design reduces the observation dimension and separates the tensor network state preparation from the RL based circuit refinement. A complete description of the DMRG calculation, MPS representation, variational MPS to circuit mapping, and Riemannian optimization procedure is provided in Supplementary Information Sections 6.1 to 6.4. The discrete action space A comprises gate placement operations from the fixed gate set RX,RY,RZ,CNOT\ RX, RY, RZ, CNOT\ and their valid qubit locations. Invalid actions are masked during action selection. In addition to operations that violate the maximum circuit depth or hardware connectivity constraints, we prohibit redundant single qubit gate insertions and repeated entangling gates. A single qubit action on qubit q at layer m is illegal when the corresponding location is already occupied, namely, Gm,q=Gm−1,q.G_m,q=G_m-1,q. (5) Likewise, a CNOT acting on qubit pair (q1,q2)(q_1,q_2) is illegal when the same CNOT was applied in the preceding layer, Gm,(q1,q2)=CNOT∧Gm−1,(q1,q2)=CNOT.G_m,(q_1,q_2)= CNOT\; \;G_m-1,(q_1,q_2)= CNOT. (6) We employ a Double Deep Q Network (DDQN) Mnih et al. (2013), parameterized by Qϕ(s,a)Q_φ(s,a), for problems up to 8 qubits. For larger problems, we use a DDQN with variable step sizes in an n step trajectory rollout update Sutton et al. (1998), with n=5n=5. The Q network is implemented as a multilayer perceptron. The agent follows an ϵε greedy policy, selecting a random valid action with probability ϵε and otherwise selecting the valid action with the largest predicted Q value. The exploration rate decays from ϵ=1.0ε=1.0 to a prescribed minimum of ϵmin=0.05 _ =0.05. For a transition (st,at,rt,st+1,dt)(s_t,a_t,r_t,s_t+1,d_t), DDQN uses the online network to select the next action and a target network Qϕ−Q_φ^- to evaluate it: yt=rt+γ(1−dt)Qϕ−(st+1,at+1⋆),at+1⋆=argmaxa∈validQϕ(st+1,a).y_t=r_t+γ(1-d_t)Q_φ^-(s_t+1,a _t+1), a _t+1= _a _validQ_φ(s_t+1,a). (7) The online network is optimized using the Huber loss between Qϕ(st,at)Q_φ(s_t,a_t) and yty_t with the Adam optimizer Kingma and Ba (2015). For variable depth search spaces, we define the per step discount factor as γ=(γfinal)1/D,γ= ( _final )^1/D, (8) such that the effective discount over the full circuit depth remains fixed. Full agent and environment hyperparameters are reported in Supplementary Information Section 8. 4.2 Generative replay In addition to real transitions collected from the QAS environment, GenQAS learns a neural dynamics model from replay memory. Given a state action pair (st,at)(s_t,a_t), the model predicts the next state, reward, and termination probability, (s^t+1,r^t,d^t)=fψ(st,at).( s_t+1, r_t, d_t)=f_ψ(s_t,a_t). (9) The model consists of separate multilayer perceptron heads for transition, reward, and termination prediction. The transition head contains two hidden layers of dimension 256256 with LeakyReLU activations and dropout probability 0.10.1. The reward and termination heads use hidden dimensions 256256 and 128128, with the termination head producing a probability through a sigmoid output. The discrete action index is concatenated with the state representation before being provided as input to each prediction head. The dynamics model is trained only on real replay transitions. In the n step setting, these transitions contain the aggregated discounted reward, terminal state, and effective discount factor associated with the observed trajectory segment. The model is optimized using Adam with learning rate 3×10−43× 10^-4 and gradient norm clipping at 1.01.0. Its composite training objective is ℒgen=2ℒtrans+ℒrew+0.5ℒdone,L_gen=2L_trans+L_rew+0.5L_done, (10) where ℒtransL_trans and ℒrewL_rew are mean squared errors for next state and reward prediction, respectively, and ℒdoneL_done is the binary cross entropy of the predicted termination probability. During DDQN updates, GenQAS draws real transitions from the prioritized replay memory. Synthetic samples are generated on demand by uniformly sampling seed state action pairs from the real replay memory and applying the learned dynamics model, τ~t=(st,at,r^t,s^t+1,d^t). τ_t=(s_t,a_t, r_t, s_t+1, d_t). (11) The predicted termination probability is thresholded at 0.50.5 to obtain the binary termination indicator d^t d_t. Each synthetic transition inherits the n step horizon and corresponding discount factor of its real seed transition. Synthetic transitions are not stored persistently in replay memory; the replay buffer therefore remains a record of real environment interactions only. For a DDQN update batch of total size B, the generation ratio GrG_r controls the fraction of synthetic transitions: Breal=⌊(1−Gr)B⌋,Bsyn=B−Breal.B_real= (1-G_r)B , B_syn=B-B_real. (12) We evaluate Gr∈0.2,0.4,0.6,0.8G_r∈\0.2,0.4,0.6,0.8\. Thus, for example, Gr=0.4G_r=0.4 produces an update batch containing approximately 60%60\% real transitions and 40%40\% generated transitions. Real transitions are sampled according to prioritized replay, whereas generated transitions are assigned the mean importance sampling weight of the real samples in the same update batch. Only real transitions have their priorities updated using the DDQN temporal difference error. The generative model is updated every Tgen=10T_gen=10 DDQN updates (if not stated otherwise) using the real transitions sampled in that update. This restriction prevents the model from training on its own outputs and reduces the risk of compounding model error. Further implementation details, including the replay capacity, batch size, generation ratios, and logging configuration, are reported in Supplementary Information Section 8. 4.3 Theoretical guarantees We establish the theoretical guarantees that together motivate and bound the performance of GenQAS. First, we prove that sample starvation is statistically inevitable for any passive replay strategy, regardless of how cleverly transitions are prioritised. Second, we show that MPS initialisation is not merely a warm-start heuristic but a provable conditional generation process with quantifiable fidelity bounds. Third, we derive an exponential speedup in episode complexity under generative replay. These results collectively answer the question: why is a generative buffer the right tool for QAS? 4.3.1 The manifold of quantum circuits Quantum circuits constructed gate-by-gate are discrete combinatorial objects, yet their effect, a unitary transformation is a continuous point in the unitary group (2N)U(2^N). This duality is central to genQAS, a generative model can learn the continuous manifold of reachable unitaries from discrete transition data, and then synthesize new transitions that lie geometrically between known high-quality circuits, even when those circuits differ structurally. Consider the space of all N-qubit quantum circuits of depth D, denoted N,DC_N,D. The RL agent’s policy π(a|s)π(a|s) operates on a state space S encoding the current circuit, with action space size ||=O(N2)|A|=O(N^2) due to two-qubit gate placements, and state space size ||∝||D|S| |A|^D growing factorially with depth. The distribution of useful circuits, those approximating the ground state is far from uniform on (2N)U(2^N): it concentrates around low-dimensional ravines aligned with the target Hamiltonian H Cerezo et al. (2021). The set of unitaries U satisfying |⟨ψ0|U†HU|ψ0⟩−E0|<ϵ| _0|U HU| _0 -E_0|<ε is exponentially sparse in the full unitary group. 4.3.2 Sample starvation in training Theorem 4.1 (Exponential sparsity of high-relevance quantum transitions). Consider the space of N-qubit parameterized quantum circuits of depth D=poly(N)D=poly(N) acting on the initial state |0⟩⊗N|0 N. Let H be a local Hamiltonian with ground state energy E0E_0 and ground state |ψ0⟩| _0 . For a given precision ϵ>0ε>0, define the set of good unitaries ϵ=U∈(2N):⟨0|U†HU|0⟩−E0<ϵ.G_ε=\U (2^N): 0|U HU|0 -E_0<ε\. The probability of uniformly sampling a circuit from the architecture space (or equivalently, sampling a state uniformly under the Haar measure) that belongs to ϵG_ε is bounded by, P(ϵ)≤exp(−c⋅2N⋅log(1ϵ))P(G_ε)≤ (-c· 2^N· ( 1ε ) ) for some constant c>0c>0, whenever ϵ<ϵ0ε< _0 for some threshold ϵ0 _0 depending on the Hamiltonian gap. We establish that without generative densification, an RL agent working with a standard replay buffer faces a fundamental sample complexity barrier that scales as exp(Ω(2N)) ( (2^N)). The prioritized generative replay framework overcomes this by generating synthetic transitions that target the high-fidelity cap ϵG_ε circumventing the volume bottleneck. A detailed proof is deferred in appendix 5.1. The following caters to non-local Hamiltonians with k-body interactions of strength J across M terms. Theorem 4.2 (Sparsity of low energy manifold). Let H=∑j=1MhjH= _j=1^Mh_j be an N-qubit Hamiltonian composed of M local terms, each hjh_j acting nontrivially on at most k qubits and satisfying ‖hj‖≤J\|h_j\|≤ J. Let E0=⟨ψ0|H|ψ0⟩E_0= _0|H| _0 be the ground state energy. For any ϵ<12min‖hj‖:hj≠0ε< 12 \\|h_j\|:h_j≠ 0\, define the set of ϵε-approximate ground states, ϵ=|ψ⟩∈ℂℙ2N−1:⟨ψ|H|ψ⟩−E0<ϵ. _ε=\|ψ ^2^N-1: ψ|H|ψ -E_0<ε\. Then the probability of sampling a state in ϵG_ε under the Haar (Fubini-Study) measure is bounded by P(ϵ)≤exp(−2NM(Nk)⋅ϵ28J2+O(N)). P(G_ε)≤ \! (- 2^NM Nk· ε^28J^2+O(N) ). In the typical regime where k=O(1)k=O(1) and M=poly(N)M= (N), P(ϵ)≤exp(−c⋅2NNk⋅ϵ2)=exp(−Ω(2Npoly(N))) P(G_ε)≤ \! (-c· 2^NN^k·ε^2 )= \! (- \! ( 2^N (N) ) ) for some absolute constant c>0c>0. We establish that low-energy states form an exponentially small subset of the Hilbert space because satisfying the energy condition requires simultaneous near-optimality on a large number of approximately independent local constraints. The Lévy concentration bound provides the sharp Gaussian tail for each local term, and the union over an independent set of terms yields the overall probability. The double-exponential form P≤exp(−eΩ(2N/poly(N)))P≤ (-e (2^N/ (N))) reflects the extreme sparsity, the good region is super-exponentially small in the dimension. This theorem provides the theoretical foundation for why generative replay is essential in QAS without a mechanism to focus the experience of an agent on the minuscule relevant subset of the state space, learning is information-theoretically impossible for large N. A detailed proof is deferred in appendix 5.2. Theorems 4.1-4.2 establish that no passive buffer strategy; uniform, PER, or otherwise; can avoid exponential sample starvation as N grows. The only remedy is a buffer that generates high-quality experience rather than merely storing it. 4.3.3 MPS initialization as a generative prior PGR learns a conditional generative model G(τ|c)G(τ|c) over transitions τ=(s,a,r,s′)τ=(s,a,r,s ), guided by a relevance signal. In QAS, the energy reduction ΔE=E(s′)−E(s) E=E(s )-E(s) is a natural relevance function, it is a strict Lyapunov function for ground state search, providing a principled signal unavailable in generic RL domains. The buffer contains states s representing circuits, the generative model can synthesize circuits s′s that interpolate geometrically between known high-quality circuits on the Stiefel manifold Tagare (2011); Boumal (2020) of parameterised gates, even when those circuits differ structurally. Definition 4.3 (Transition operator). Let sts_t be a quantum circuit. An action ata_t via gate UgateU_gate yields st+1=Ugate⋅sts_t+1=U_gate· s_t, defining the transition τ=(st,at,st+1,ΔE)τ=(s_t,a_t,s_t+1, E). Lemma 4.4 (MPS initialization as guided generation). Let H be an N-qubit Hamiltonian whose ground state obeys the area law of entanglement entropy, and let |ΨMPS⟩| _MPS be the DMRG approximation with bond dimension χ. The Riemannian optimisation mapping |ΨMPS⟩| _MPS to a brickwork unitary UTNU_TN satisfying UTN|0⟩⊗N≈|ΨMPS⟩U_TN|0 N≈| _MPS constitutes a single-step conditional generation process GMPS(τ|c=H)G_MPS(τ|c\!=\!H), where the generated transition τ=(∅,∅,UTN,ΔEMPS)τ=( , ,U_TN, E_MPS) initialises the circuit. The MPS to circuit mapping via Riemannian optimization is a conditional generative model. The condition c is the Hamiltonian H, encoded through the DMRG solution. The generated output is a high-quality initialization transition τinit _init that seeds the RL replay buffer. This provides a basis for why TensorRL-QAS can be viewed as an instantiation of the PGR framework, where the priority is determined by the energy expectation value. We provide the proof detailed in appendix 5.3. Lemma 4.5 (Synthetic circuit bound). Let |Ψ0⟩| _0 be the true ground state of an N-qubit gapped local Hamiltonian H, |ΨMPS⟩| _MPS be its matrix product state approximation with bond dimension χ. and UTNU_TN be the unitary obtained by mapping |ΨMPS⟩| _MPS to a brickwork quantum circuit via Riemannian optimization over the Stiefel manifold (4)×mU(4)^× m. For any target trace-distance threshold δ>0δ>0, there exists a bond dimension χ≤poly(N,1/δ)χ (N,1/δ) and a number of optimization steps TRiemT_Riem polynomial in N, such that the generated state UTN|0⟩U_TN|0 satisfies 12‖UTN|0⟩⟨0|UTN†−|Ψ0⟩⟨Ψ0|‖1≤δ. 12 \|U_TN|0 0|U_TN -| _0 _0| \|_1≤δ. (13) The theoretical basis of Tensor network initialization is equivalent to a one-step conditional generative model with a provable error bound. We provided a detailed proof reported in appendix 5.4. In this, the total pre-training cost, classical DMRG and Riemannian optimization is polynomial in N and 1/δ1/δ. Therefore, the circuit UTNU_TN constitutes a high accuracy, synthetically generated quantum state that resides within the δ-neighborhood of the optimal manifold, and it is obtained before the RL agent executes its first quantum circuit in the environment. 4.3.4 Generative replay provides exponential speedup Theorem 4.6 (Convergence Acceleration via Synthetic Replay). Let πbase _base be a policy learning purely from online interaction (Vanilla RL-QAS) using a replay buffer realD_real, and πPGR _PGR be a policy trained with prioritized generative replay. The generative model GψG_ψ is conditioned on the energy reduction −ΔE- E and is pre-trained to densify the replay buffer around a high accuracy state sTNs_TN. There exists a constant κ>1κ>1 dependent on the Hamiltonian gap and noise level such that the sample complexity S to reach chemical accuracy satisfies πPGR(ϵc)≤1κNπbase(ϵc).S_ _PGR( _c)≤ 1κ^NS_ _base( _c). (14) Theorem 4.6 shows that the episode-complexity gap between passive and generative replay grows exponentially in the qubit count, the same rate at which Theorems 4.1 and 4.2 predict passive buffers to fail. The three results are therefore mutually consistent, the generative buffer is the minimal corrective for the geometric barrier identified above. A detailed proof of Theorem 4.6 is provided in appendix 5.5. 4.4 Training protocol Algorithm 1 summarizes the GenQAS training procedure. Before RL training, DMRG produces an MPS approximation of the target state, which is compiled into the fixed warm start circuit UTNU_TN. At the beginning of each episode, the agent initializes an empty learnable suffix V0V_0 and sequentially selects valid gate insertion actions. Each action extends the suffix to form Vt+1V_t+1, while the environment evaluates the complete circuit Ut+1=Vt+1UTNU_t+1=V_t+1U_TN. The resulting real transition is stored in replay memory. Algorithm 1 GenQAS: Training with fixed MPS warm start and generative replay 1: Hamiltonian H, MPS bond dimension χ, maximum RL depth D, replay memory ℳM, Q network QϕQ_φ, target network Qϕ−Q_φ^-, generative model fψf_ψ, generation ratio GrG_r, synthetic replay frequency TsynT_syn 2: Obtain |ψMPS⟩χ _MPS_χ using DMRG for H 3: Map |ψMPS⟩χ _MPS_χ to a brickwork circuit UTNU_TN using Riemannian optimization 4: Initialize ℳ←∅M← and Qϕ−←QϕQ_φ^-← Q_φ 5: for each training episode do 6: Initialize the learnable circuit suffix V0←IV_0← I and the RL state s0s_0 7: for t=0,…,D−1t=0,…,D-1 do 8: Select a valid action ata_t using ϵε greedy exploration 9: Append the selected gate to the suffix: Vt+1←atVtV_t+1← a_tV_t 10: Evaluate Ut+1=Vt+1UTNU_t+1=V_t+1U_TN and observe (rt,st+1,dt)(r_t,s_t+1,d_t) 11: Store the real transition (st,at,rt,st+1,dt)(s_t,a_t,r_t,s_t+1,d_t) in ℳM 12: Sample a real batch ℬrealB_real from ℳM 13: ℬ←ℬrealB _real 14: if tmodTsyn=0t T_syn=0 then 15: Sample Gr|ℬreal|G_r|B_real| state action pairs from ℬrealB_real 16: Generate ℬsyn=(s,a,r^,s^′,d^)B_syn=\(s,a, r, s , d)\ using fψ(s,a)f_ψ(s,a) 17: ℬ←ℬreal∪ℬsynB _real _syn 18: end if 19: Update QϕQ_φ using ℬB 20: if generative model update step then 21: Update fψf_ψ using only real transitions from ℳM 22: end if 23: if target network update step then 24: Set Qϕ−←QϕQ_φ^-← Q_φ 25: end if 26: if dt=1d_t=1 then 27: break 28: end if 29: end for 30: end for 31: return best circuit U⋆=V⋆UTNU =V U_TN At each DDQN update, a batch of real transitions is sampled from replay memory. The generative model is periodically updated from real transitions, and synthetic transitions are generated from sampled real state action pairs. The Q network is then optimized using the combined real and synthetic batch. The target Q network is updated at a separate fixed interval. An episode terminates when the target criterion is reached or when the maximum allowed circuit depth is attained. The complete training procedure is given in Algorithm 1, and benchmark specific hyperparameters are reported in Supplementary Information Section 8. To benchmark GenQAS we consider the problem of preparing the ground state of chemical Hamiltonian of varying size generate using the quantum chemistry module of PennyLane Bergholm et al. (2018); Arrazola et al. (2021). The GenQAS tensor network warm start was implemented using Quimb and Qiskit Gray (2018); Javadi-Abhari et al. (2024), while quantum circuit simulation and energy expectation value evaluation were performed using the Qulacs simulator Suzuki et al. (2021). Detailed molecular Hamiltonian construction, active-space definitions, tensor network implementation, Hamiltonian validation, and simulation settings are provided in Supplementary Information Sections 8.3 and 8.4. Acknowledgements A.K. and S.F. acknowledge the use of computational resources of the DelftBlue supercomputer, provided by Delft High Performance Computing Centre (https://w.tudelft.nl/dhpc). Declarations 4.5 Funding S.F. and A.K. gratefully acknowledge financial support by QDNL via the National Growth Fund KAT-1 program. 4.6 Conflict of interest/Competing interests The authors have no conflict of interest. 4.7 Ethics approval and consent to participate Not applicable. 4.8 Consent for publication Not appicable. 4.9 Data availability The data can be generated from the code itself but if reuqired the corresponding author can share the data upon reasonable request. 4.10 Code availability The code will be available with the final version upon publication 4.11 Author contribution A.K. conceptualize this idea. A.K. and A.K.J together prepare the code and theory for the paper. S.F. and P.T. supervised the project and together with A.K. and A.K.J investigated and wrote the manuscript. References (DHPC) (2024) D. H. P. C. C. (DHPC) DelftBlue Supercomputer (Phase 2). Note: https://w.tudelft.nl/dhpc/ark:/44463/DelftBluePhase2 Cited by: §8.2. Anand et al. (2022) A. Anand, P. Schleich, S. Alperin-Lea, P. W. Jensen, S. Sim, M. Díaz-Tinoco, J. S. Kottmann, M. Degroote, A. F. Izmaylov, and A. Aspuru-Guzik A quantum computing view on unitary coupled cluster theory. Chemical Society Reviews 51 (5), p. 1659–1684. Cited by: §1. Arrazola et al. (2021) J. M. Arrazola, S. Jahangiri, A. Delgado, J. Ceroni, J. Izaac, A. Száva, U. Azad, R. A. Lang, Z. Niu, O. Di Matteo, et al. Differentiable quantum computational chemistry with pennylane. arXiv preprint arXiv:2111.09967. Cited by: §4.4, §8.3. Baydin et al. (2018) A. G. Baydin, B. A. Pearlmutter, A. A. Radul, and J. M. Siskind Automatic differentiation in machine learning: a survey. Journal of machine learning research 18 (153), p. 1–43. Cited by: §6.3. Berezutskii et al. (2025) A. Berezutskii, M. Liu, A. Acharya, R. Ellerbrock, J. Gray, R. Haghshenas, Z. He, A. Khan, V. Kuzmin, D. Lyakh, et al. Tensor networks for quantum computing. Nature Reviews Physics 7 (10), p. 581–593. Cited by: §6.2. Bergholm et al. (2018) V. Bergholm, J. Izaac, M. Schuld, C. Gogolin, S. Ahmed, V. Ajith, M. S. Alam, G. Alonso-Linaje, B. AkashNarayanan, A. Asadi, et al. Pennylane: automatic differentiation of hybrid quantum-classical computations. arXiv preprint arXiv:1811.04968. Cited by: §4.4, §8.3. Blume-Kohout and Young (2020) R. Blume-Kohout and K. C. Young A volumetric framework for quantum computer benchmarks. Quantum 4, p. 362. Cited by: §1. Boumal (2020) N. Boumal An introduction to optimization on smooth manifolds. To appear with Cambridge University Press,(Apr 2022). Cited by: §4.3.3, §5.4. Bukov and Marquardt (2026) M. Bukov and F. Marquardt Reinforcement learning for quantum technology. arXiv preprint arXiv:2601.18953. Cited by: §1. Cerezo et al. (2021) M. Cerezo, A. Arrasmith, R. Babbush, S. C. Benjamin, S. Endo, K. Fujii, J. R. McClean, K. Mitarai, X. Yuan, L. Cincio, et al. Variational quantum algorithms. Nature Reviews Physics 3 (9), p. 625–644. Cited by: §1, §4.3.1. Cross et al. (2019) A. W. Cross, L. S. Bishop, S. Sheldon, P. D. Nation, and J. M. Gambetta Validating quantum computers using randomized model circuits. Physical Review A 100 (3), p. 032328. Cited by: §1. Du et al. (2022) Y. Du, T. Huang, S. You, M. Hsieh, and D. Tao Quantum circuit architecture search for variational quantum algorithms. npj Quantum Information 8 (1), p. 62. Cited by: §1. Eisert and Preskill (2025) J. Eisert and J. Preskill Mind the gaps: the fraught road to quantum advantage. arXiv preprint arXiv:2510.19928. Cited by: §1. Feinberg et al. (2018) V. Feinberg, A. Wan, I. Stoica, M. I. Jordan, J. E. Gonzalez, and S. Levine Model-based value estimation for efficient model-free reinforcement learning. arXiv preprint arXiv:1803.00101. External Links: 1803.00101, Document, Link Cited by: §1. Fernández de Fuentes et al. (2026) I. Fernández de Fuentes, E. Raymenants, B. Undseth, O. Pietx-Casas, S. Philips, M. Mądzik, S. De Snoo, S. Amitonov, L. Tryputen, A. Schmitz, et al. Running a six-qubit quantum circuit on a silicon spin-qubit array. PRX Quantum 7 (1), p. 010308. Cited by: §1. Fösel et al. (2021) T. Fösel, M. Y. Niu, F. Marquardt, and L. Li Quantum circuit optimization with deep reinforcement learning. arXiv preprint arXiv:2103.07585. Cited by: §1. Gray (2018) J. Gray Quimb: a python package for quantum information and many-body calculations. Journal of Open Source Software 3 (29), p. 819. Cited by: §4.4, §8.4. Grimsley et al. (2019) H. R. Grimsley, S. E. Economou, E. Barnes, and N. J. Mayhall An adaptive variational algorithm for exact molecular simulations on a quantum computer. Nature communications 10 (1), p. 3007. Cited by: §1. Ha and Schmidhuber (2018) D. Ha and J. Schmidhuber World models. arXiv preprint arXiv:1803.10122. External Links: 1803.10122, Link Cited by: §1. Hastings (2007) M. B. Hastings An area law for one-dimensional quantum systems. Journal of statistical mechanics: theory and experiment 2007 (08), p. P08024–P08024. Cited by: §5.4. He et al. (2024) Z. He, M. Deng, S. Zheng, L. Li, and H. Situ Training-free quantum architecture search. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 38, p. 12430–12438. External Links: Document Cited by: §1. Janner et al. (2019) M. Janner, J. Fu, M. Zhang, and S. Levine When to trust your model: model-based policy optimization. In Advances in Neural Information Processing Systems, Vol. 32. Cited by: §1. Javadi-Abhari et al. (2024) A. Javadi-Abhari, M. Treinish, K. Krsulich, C. J. Wood, J. Lishman, J. Gacon, S. Martiel, P. D. Nation, L. S. Bishop, A. W. Cross, et al. Quantum computing with qiskit. arXiv preprint arXiv:2405.08810. Cited by: §4.4, §8.3, §8.4. Kandala et al. (2017) A. Kandala, A. Mezzacapo, K. Temme, M. Takita, M. Brink, J. M. Chow, and J. M. Gambetta Hardware-efficient variational quantum eigensolver for small molecules and quantum magnets. nature 549 (7671), p. 242–246. Cited by: §1, §4.1, §6.2. Katabarwa et al. (2024) A. Katabarwa, K. Gratsea, A. Caesura, and P. D. Johnson Early fault-tolerant quantum computing. PRX quantum 5 (2), p. 020101. Cited by: §1. Kingma and Ba (2015) D. P. Kingma and J. Ba Adam: a method for stochastic optimization. International Conference on Learning Representations. Cited by: §4.1. Kjaergaard et al. (2020) M. Kjaergaard, M. E. Schwartz, J. Braumüller, P. Krantz, J. I. Wang, S. Gustavsson, and W. D. Oliver Superconducting qubits: current state of play. Annual Review of Condensed Matter Physics 11 (1), p. 369–395. Cited by: §1. Kremer et al. (2024) D. Kremer, V. Villar, H. Paik, I. Duran, I. Faro, and J. Cruz-Benito Practical and efficient quantum circuit synthesis and transpiling with reinforcement learning. arXiv preprint arXiv:2405.13196. Cited by: §7. Kundu and Feld (2026) A. Kundu and S. Feld Replay-buffer engineering for noise-robust quantum circuit optimization. arXiv preprint arXiv:2604.21863. Cited by: §2.2. Kundu and Mangini (2026) A. Kundu and S. Mangini TensorRL-qas: reinforcement learning with tensor networks for improved quantum architecture search. Advances in Neural Information Processing Systems 38, p. 120896–120930. Cited by: §1, 3(b), §4.1, §4, §5.3, Figure 8, Figure 8, §6. Kundu et al. (2026) A. Kundu, A. Sarkar, P. Tiwari, and S. Feld Reinforcement learning for quantum circuit optimization: a review. Openreview. Cited by: §1, §3. Kuo et al. (2021) E. Kuo, Y. L. Fang, and S. Y. Chen Quantum architecture search via deep reinforcement learning. arXiv preprint arXiv:2104.07715. Cited by: §1, §1. Leone et al. (2024) L. Leone, S. F. Oliviero, L. Cincio, and M. Cerezo On the practical usefulness of the hardware efficient ansatz. Quantum 8, p. 1395. Cited by: §4.1, §6.2. Luchnikov et al. (2021) I. A. Luchnikov, M. E. Krechetov, and S. N. Filippov Riemannian geometry and automatic differentiation for optimization problems of quantum physics and quantum technologies. New Journal of Physics 23 (7), p. 073006. Cited by: §5.4, §6.3, §6.3. Mnih et al. (2013) V. Mnih, K. Kavukcuoglu, D. Silver, A. Graves, I. Antonoglou, D. Wierstra, and M. Riedmiller Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602. Cited by: §1, §4.1. Moro et al. (2021) L. Moro, M. G. Paris, M. Restelli, and E. Prati Quantum compiling by deep reinforcement learning. Communications Physics 4 (1), p. 178. Cited by: §1. Orús (2014) R. Orús A practical introduction to tensor networks: matrix product states and projected entangled pair states. Annals of physics 349, p. 117–158. Cited by: §6.1. Ostaszewski et al. (2021) M. Ostaszewski, L. M. Trenkwalder, W. Masarczyk, E. Scerri, and V. Dunjko Reinforcement learning for optimization of variational quantum circuit architectures. Advances in neural information processing systems 34, p. 18182–18194. Cited by: §1. Patel et al. (2024) Y. J. Patel, A. Kundu, M. Ostaszewski, X. Bonet-Monroig, V. Dunjko, and O. Danaci Curriculum reinforcement learning for quantum architecture search under hardware errors. In International Conference on Learning Representations, p. 32298–32329. Cited by: §1. Peruzzo et al. (2014) A. Peruzzo, J. McClean, P. Shadbolt, M. Yung, X. Zhou, P. J. Love, A. Aspuru-Guzik, and J. L. O’brien A variational eigenvalue solver on a photonic quantum processor. Nature communications 5 (1), p. 4213. Cited by: §1. Petit et al. (2022) L. Petit, M. Russ, G. H. Eenink, W. I. Lawrie, J. S. Clarke, L. M. Vandersypen, and M. Veldhorst Design and integration of single-qubit rotations and two-qubit gates in silicon above one kelvin. Communications Materials 3 (1), p. 82. Cited by: §1. Preskill (2018) J. Preskill Quantum computing in the nisq era and beyond. Quantum 2, p. 79. Cited by: §1. Ran (2020) S. Ran Encoding of matrix product states into quantum circuits of one-and two-qubit gates. Physical Review A 101 (3), p. 032310. Cited by: §6.2. Ruiz et al. (2025) F. J. R. Ruiz, T. Laakkonen, J. Bausch, M. Balog, M. Barekatain, F. J. H. Heras, A. Novikov, N. Fitzpatrick, B. Romera-Paredes, J. van de Wetering, A. Fawzi, K. Meichanetzidis, and P. Kohli Quantum circuit optimization with alphatensor. Nature Machine Intelligence 7, p. 374–385. External Links: Document Cited by: §1. Schaul et al. (2015) T. Schaul, J. Quan, I. Antonoglou, and D. Silver Prioritized experience replay. arXiv preprint arXiv:1511.05952. Cited by: §1. Schollwöck (2011) U. Schollwöck The density-matrix renormalization group in the age of matrix product states. Annals of physics 326 (1), p. 96–192. Cited by: §6.1. Schön et al. (2005) C. Schön, E. Solano, F. Verstraete, J. I. Cirac, and M. M. Wolf Sequential generation of entangled multiqubit states. Physical review letters 95 (11), p. 110503. Cited by: §6.2. Sivak et al. (2026) V. Sivak, A. Morvan, M. Broughton, R. G. Cortiñas, J. Bausch, A. W. Senior, M. Neeley, A. Eickbusch, N. Shutty, L. A. Beni, et al. Reinforcement learning control of quantum error correction. Nature, p. 1–6. Cited by: §1. Sutton et al. (1998) R. S. Sutton, A. G. Barto, and A. Barto Reinforcement learning: an introduction. Vol. 1, MIT press Cambridge. Cited by: §4.1. Sutton (1991) R. S. Sutton Dyna, an integrated architecture for learning, planning, and reacting. ACM SIGART Bulletin 2 (4), p. 160–163. External Links: Document Cited by: §1. Suzuki et al. (2021) Y. Suzuki, Y. Kawase, Y. Masumura, Y. Hiraga, M. Nakadai, J. Chen, K. M. Nakanishi, K. Mitarai, R. Imai, S. Tamiya, et al. Qulacs: a fast and versatile quantum circuit simulator for research purpose. Quantum 5, p. 559. Cited by: §4.4, §8.4. Tacchino et al. (2020) F. Tacchino, A. Chiesa, S. Carretta, and D. Gerace Quantum computers as universal quantum simulators: state-of-the-art and perspectives. Advanced Quantum Technologies 3 (3), p. 1900052. Cited by: §1. Tagare (2011) H. D. Tagare Notes on optimization on stiefel manifolds. Yale University, New Haven. Cited by: §4.3.3. Tosato et al. (2026) A. Tosato, A. Elsayed, F. Poggiali, L. E. A. Stehouwer, D. Costa, K. L. Hudson, D. Degli Esposti, and G. Scappucci A crossbar chip for benchmarking semiconductor spin qubits. Nature Electronics 9 (3), p. 324–333. Cited by: §1. Virtanen et al. (2020) P. Virtanen, R. Gommers, T. E. Oliphant, M. Haberland, T. Reddy, D. Cournapeau, E. Burovski, P. Peterson, W. Weckesser, J. Bright, et al. SciPy 1.0: fundamental algorithms for scientific computing in python. Nature methods 17 (3), p. 261–272. Cited by: §1. Wang et al. (2024) C. Wang, V. John, H. Tidjani, C. X. Yu, A. S. Ivlev, C. Déprez, F. van Riggelen-Doelman, B. D. Woods, N. W. Hendrickx, W. I. Lawrie, et al. Operating semiconductor quantum processors with hopping spins. Science 385 (6707), p. 447–452. Cited by: §1. Wang et al. (2022) H. Wang, Y. Ding, J. Gu, Y. Lin, D. Z. Pan, F. T. Chong, and S. Han Quantumnas: noise-adaptive search for robust quantum circuits. In 2022 IEEE International Symposium on High-Performance Computer Architecture (HPCA), p. 692–708. Cited by: §1. Wang et al. (2025) R. Wang, K. Frans, P. Abbeel, S. Levine, and A. Efros Prioritized generative replay. In International Conference on Learning Representations, Vol. 2025, p. 47151–47169. Cited by: §1, §4. Wecker et al. (2015) D. Wecker, M. B. Hastings, and M. Troyer Progress towards practical quantum variational algorithms. Physical Review A 92 (4), p. 042303. Cited by: §1. White (1992) S. R. White Density matrix formulation for quantum renormalization groups. Physical review letters 69 (19), p. 2863. Cited by: §6.1. Wright et al. (2019) K. Wright, K. M. Beck, S. Debnath, J. Amini, Y. Nam, N. Grzesiak, J. Chen, N. Pisenti, M. Chmielewski, C. Collins, et al. Benchmarking an 11-qubit quantum computer. Nature communications 10 (1), p. 5464. Cited by: §1. Yang et al. (2026) R. Yang, Z. Wang, Y. Gu, Y. Liang, and T. Li QCircuitBench: a large-scale dataset for benchmarking quantum algorithm design. Advances in Neural Information Processing Systems 38. Cited by: §7.1. Zen et al. (2025) R. Zen, J. Olle, L. Colmenarez, M. Puviani, M. Müller, and F. Marquardt Quantum circuit discovery for fault-tolerant logical state preparation with reinforcement learning. Physical Review X 15 (4), p. 041012. Cited by: §3. Zhang et al. (2022) S. Zhang, C. Hsieh, S. Zhang, and H. Yao Differentiable quantum architecture search. Quantum Science & Technology 7 (4), p. 045023. Cited by: §1. Zhang et al. (2020) Y. Zhang, P. Zheng, Y. Zhang, and D. Deng Topological quantum compiling with reinforcement learning. Physical Review Letters 125 (17), p. 170501. Cited by: §1. 5 Theory 5.1 Proof of Theorem 4.1 The unitary group (2N)U(2^N) acts transitively on the space of pure quantum states, which is isomorphic to the complex projective space =(2N)(1)×(2N−1)≅ℂℙ2N−1, = U(2^N)U(1)×U(2^N-1) ^2^N-1, with real dimension dimℝ()=2N+1−2 _R(S)=2^N+1-2. The natural probability measure is the Fubini-Study volume dμFSd _FS, with total volume VolFS()=π2N−1(2N−1)!.Vol_FS(S)= π^2^N-1(2^N-1)!. Let |ψ0⟩| _0 be the true ground state of H and define the good circuit set ϵ=|ψ⟩:⟨ψ|H|ψ⟩<E0+ϵG_ε=\|ψ : ψ|H|ψ <E_0+ε\. Since H is a local Hamiltonian with spectral gap Δ=E1−E0>0 =E_1-E_0>0, decomposing |ψ⟩|ψ in the eigenbasis |ϕk⟩\| _k \ gives ⟨ψ|H|ψ⟩ ψ|H|ψ =|c0|2E0+∑k≥1|ck|2Ek =|c_0|^2E_0+ _k≥ 1|c_k|^2E_k ≥|c0|2E0+(1−|c0|2)E1. ≥|c_0|^2E_0+(1-|c_0|^2)E_1. Requiring this to be less than E0+ϵE_0+ε results |c0|2>1−ϵ/Δ|c_0|^2>1-ε/ , i.e., the fidelity F=|⟨ψ0|ψ⟩|2F=| _0|ψ |^2 must satisfy F>1−δF>1-δ where δ=ϵ/Δδ=ε/ . In projective space, ϵG_ε is a geodesic ball of angular radius θϵ≲2δ _ε 2 δ, with volume VolFS(Bθϵ)∼Cd⋅θϵ2(2N−1).Vol_FS(B_ _ε) C_d· _ε^2(2^N-1). Hence, P(ϵ) P(G_ε) =VolFS(Bθϵ)VolFS(ℂℙ2N−1)≤C⋅(θϵπ)2(2N−1) = Vol_FS(B_ _ε)Vol_FS(CP^2^N-1)≤ C· ( _επ )^2(2^N-1) ≤C⋅(2πϵΔ)2(2N−1)=C⋅exp(2(2N−1)⋅log(2πϵΔ)). ≤ C· ( 2π ε )^2(2^N-1)=C· \! (2(2^N-1)· \! ( 2π ε ) ). Since ϵε is small the logarithm is negative, and for large N the factor 2(2N−1)2(2^N-1) dominates, giving P(ϵ)≤exp(−c⋅2N⋅log1ϵ),c>0. P(G_ε)≤ \! (-c· 2^N· 1ε ), c>0. By standard unitary t-design theory, a brickwork circuit of depth D=poly(N)D=poly(N) induces a measure exponentially close (in total variation) to the Haar measure on (2N)U(2^N), the probability of finding a good circuit via random action selection is therefore proportional to the Haar probability above. It follows that [Trials]≥1P(ϵ)≥exp(c⋅2N⋅log1ϵ)E[Trials]≥ 1P(G_ε)≥ \! (c· 2^N· 1ε ), and the signal-to-noise ratio in the replay buffer decays super-exponentially as exp(−c⋅2N). (-c· 2^N). 5.2 Proof of Theorem 4.2 Initially, we map the energy condition to a condition on the real and imaginary parts of the state vector. The space of pure quantum states of N qubits is the complex projective space ℂℙd−1CP^d-1 with d=2Nd=2^N. Equivalently, a pure state can be represented by a unit vector |ψ⟩∈ℂd|ψ ^d with ‖|ψ⟩‖=1\||ψ \|=1, modulo an overall phase. The real dimension of this manifold is 2d−2=2N+1−22d-2=2^N+1-2. The Haar measure on (d)U(d) induces the Fubini-Study measure μFS _FS on ℂℙd−1CP^d-1. Under this measure, a random state |ψ⟩|ψ can be generated as |ψ⟩=|ϕ⟩‖|ϕ⟩‖,|ψ = |φ \||φ \|, where |ϕ⟩=(x1+iy1,…,xd+iyd)T|φ =(x_1+iy_1,…,x_d+iy_d)^T is a complex Gaussian vector with each xℓ,yℓ∼(0,1/2)x_ ,y_ (0,1/2) independently. Also, |ψ⟩|ψ is distributed uniformly over the real unit sphere S2d−1⊂ℝ2dS^2d-1 ^2d. Now, we apply Lévy’s lemma to control concentration of the local terms. We fix a single term hjh_j of the Hamiltonian, without loss of generality, and assuming hjh_j is traceless (any trace part shifts the energy by a constant and does not affect the gap structure). Since ‖hj‖≤J\|h_j\|≤ J, the function fj(|ψ⟩)=⟨ψ|hj|ψ⟩f_j(|ψ )= ψ|h_j|ψ is a real-valued Lipschitz function on the sphere. For unit vectors |u⟩,|v⟩|u ,|v , |f(|u⟩)−f(|v⟩)| |f(|u )-f(|v )| =|⟨u|A|u⟩−⟨v|A|v⟩| =| u|A|u - v|A|v | =|Tr(A(|u⟩⟨u|−|v⟩⟨v|))| =|Tr(A(|u u|-|v v|))| ≤‖A‖∞⋅‖|u⟩⟨u|−|v⟩⟨v|‖1 ≤\|A\|_∞·\||u u|-|v v|\|_1 ≤2‖A‖∞⋅‖|u⟩−|v⟩‖2, ≤ 2\|A\|_∞·\||u -|v \|_2, where the last inequality follows from the trace-norm bound for rank-1 projectors. Thus Lip(fj)≤2JLip(f_j)≤ 2J. Since |ψ0⟩| _0 is the ground state of H, it is also an eigenstate of each hjh_j in some local decomposition. We do not assume this, instead, we use the fact that E0=⟨ψ0|H|ψ0⟩E_0= _0|H| _0 is the global minimum. The deviation of fj(|ψ⟩)f_j(|ψ ) from its ground state expectation is what matters. We define the centered function for each term gj(|ψ⟩)=⟨ψ|hj|ψ⟩−⟨ψ0|hj|ψ0⟩. g_j(|ψ )= ψ|h_j|ψ - _0|h_j| _0 . Then Lip(gj)≤2JLip(g_j)≤ 2J as well. Note that ∑jgj(|ψ⟩)=⟨ψ|H|ψ⟩−E0 _jg_j(|ψ )= ψ|H|ψ -E_0. The median of gjg_j under the Haar measure depicted as MjM_j satisfies a property, for any traceless local observable of small support, the Haar expectation is zero and the distribution is strongly concentrated around zero. More precisely, we need the following concentration inequality. Lemma 5.1. Let f:Sn−1→ℝf:S^n-1 be L-Lipschitz. Then for any t>0t>0, P(|f(x)−Mf|≥t)≤2exp(−nt24L2), P (|f(x)-M_f|≥ t )≤ 2 \! (- nt^24L^2 ), where MfM_f is the median of f. Proof. Applying Lemma 5.1 to gjg_j on S2d−1S^2d-1 (so n=2d=2N+1n=2d=2^N+1), P(|gj(|ψ⟩)−Mj|≥t)≤2exp(−2Nt28J2).P (|g_j(|ψ )-M_j|≥ t )≤ 2 \! (- 2^Nt^28J^2 ). (15) ∎ For each local term hjh_j, we can bound its median MjM_j. Since hjh_j is a sum of Pauli strings of weight at most k, and Haar[⟨ψ|hj|ψ⟩]=Tr(hj)dE_ [ ψ|h_j|ψ ]= Tr(h_j)d. For a traceless hjh_j, the expectation is zero. The median and expectation of Lipschitz functions on the sphere are close, |Mj−[gj]|≤4Ln=8J2(N+1)/2≤8J⋅2−N/2. |M_j-E[g_j]|≤ 4L n= 8J2^(N+1)/2≤ 8J· 2^-N/2. Thus, for N≥2N≥ 2, |Mj|≤8J⋅2−N/2≤ϵ/2|M_j|≤ 8J· 2^-N/2≤ε/2 for ϵε larger than an exponentially small threshold. For our choice of ϵε, we can safely bound |Mj|≤ϵ/2|M_j|≤ε/2 for sufficiently large N. For the state to be in ϵG_ε, we need ∑j=1Mgj(|ψ⟩)<ϵ _j=1^Mg_j(|ψ )<ε. A sufficient condition is that each individual term is bounded, if gj(|ψ⟩)<ϵ/Mg_j(|ψ )<ε/M for all j, then the sum is <ϵ<ε. The converse is not true, but this gives a valid upper bound on the probability. Thus, P(ϵ) P(G_ε) ≤P(⋂j=1Mgj(|ψ⟩)<ϵ/M) ≤ P ( _j=1^M\g_j(|ψ )<ε/M\ ) ≤P(⋂j=1M|gj(|ψ⟩)−Mj|<ϵ/M−|Mj|) ≤ P ( _j=1^M\|g_j(|ψ )-M_j|<ε/M-|M_j|\ ) ≤P(⋂j=1M|gj(|ψ⟩)−Mj|<ϵ/(2M)), ≤ P ( _j=1^M\|g_j(|ψ )-M_j|<ε/(2M)\ ), (16) where the last step assumes |Mj|≤ϵ/(2M)|M_j|≤ε/(2M), which holds for sufficiently large N as argued above. The complement event is P(ϵc)≥1−P(ϵ)P(G_ε^c)≥ 1-P(G_ε). However, using the union bound P(ϵ) P(G_ε) =1−P(∃j:|gj(|ψ⟩)−Mj|≥ϵ/(2M)) =1-P(∃ j:|g_j(|ψ )-M_j|≥ε/(2M)) ≥1−∑j=1MP(|gj(|ψ⟩)−Mj|≥ϵ/(2M)). ≥ 1- _j=1^MP(|g_j(|ψ )-M_j|≥ε/(2M)). (17) gives a lower bound on P(ϵ)P(G_ε). We need an upper bound to show the sparsity. Now, let’s P(ϵ)P(G_ε) is the probability that the sum is small. Since the sum involves M nonnegative (after shifting) terms, a necessary condition for the sum to be <ϵ<ε is that no individual term exceeds ϵε. That is ϵ⊆⋂j=1M|gj(|ψ⟩)−Mj|<ϵ+|Mj|. _ε _j=1^M\|g_j(|ψ )-M_j|<ε+|M_j|\. But this trivial inclusion leads to a upper bound (P≤1P≤ 1). Instead, we use the fact that for the state to have low energy, a significant fraction of the local terms must be individually small. Quantitatively, if ∑gj<ϵΣ g_j<ε, then at most M/2M/2 terms can exceed 2ϵ/M2ε/M. This indicates that the Hamiltonian terms are not independent, they share overlapping supports. However, we can exploit the locality to get a stronger bound. Since each term hjh_j acts on at most k qubits, a single qubit can be involved in at most (N−1k−1) N-1k-1 terms. The total number of terms is at most (Nk) Nk. We can select a subset ⊆1,…,MJ \1,…,M\ of terms acting on disjoint sets of qubits. The maximum size of such an independent set is ||≥M(Nk)⋅1k|J|≥ M Nk· 1k. For simplicity, we use the factor M(Nk) M Nk (ignoring the constant 1/k1/k, which only affects constants). These |||J| terms are statistically independent under the Haar measure because they act on disjoint subsystems. For each j∈j , P(gj(|ψ⟩)≥ϵ||)≥1−exp(−2N+1(ϵ/||)24(2J)2)=1−exp(−2Nϵ28J2||2). P (g_j(|ψ )≥ ε|J| )≥ 1- \! (- 2^N+1(ε/|J|)^24(2J)^2 )=1- \! (- 2^Nε^28J^2|J|^2 ). We need the probability that all terms in J are small simultaneously. Since they are independent: P(∀j∈:gj(|ψ⟩)≤δ)=∏j∈P(gj(|ψ⟩)≤δ). P (∀ j :g_j(|ψ )≤δ )= _j P(g_j(|ψ )≤δ). For each term based on Lemma 5.1, P(gj(|ψ⟩)≤δ)≤P(|gj−Mj|≤δ+|Mj|)≤1−exp(−2Nδ28J2)(for δ≫2−N/2). P(g_j(|ψ )≤δ)≤ P(|g_j-M_j|≤δ+|M_j|)≤ 1- \! (- 2^Nδ^28J^2 ) (for δ 2^-N/2). Thus, P(ϵ)≤∏j∈P(gj≤ϵ/||)≤(1−exp(−2Nϵ28J2||2))||.P(G_ε)≤ _j P(g_j≤ε/|J|)≤ (1- \! (- 2^Nε^28J^2|J|^2 ) )^|J|. Using 1−x≤e−x1-x≤ e^-x for x∈[0,1]x∈[0,1], P(ϵ) P(G_ε) ≤exp(−||⋅exp(−2Nϵ28J2||2)) ≤ \! (-|J|· \! (- 2^Nε^28J^2|J|^2 ) ) =exp(−exp(ln||−2Nϵ28J2||2)). = \! (- \! ( |J|- 2^Nε^28J^2|J|^2 ) ). (18) This double-exponential form is standard for sparsity bounds. To maximize the inner argument and thus for the tightest bound, we choose |||J| to balance ln|| |J| against 2Nϵ2/||22^Nε^2/|J|^2. However, |||J| is constrained by the geometry ||≈M/(Nk)|J|≈ M/ Nk. For M=poly(N)M= (N) and k=O(1)k=O(1), we have ||=Θ(N−k⋅poly(N))=poly(N)|J|= (N^-k· (N))= (N). Then 2Nϵ28J2||2≫ln||, 2^Nε^28J^2|J|^2 |J|, since 2N2^N grows exponentially while |||J| grows only polynomially. Hence the inner exponential dominates, ln||−2Nϵ28J2||2≈−2Nϵ28J2||2. |J|- 2^Nε^28J^2|J|^2≈- 2^Nε^28J^2|J|^2. Substituting ||=M(Nk)|J|= M Nk, P(ϵ) P(G_ε) ≤exp(−exp(−2Nϵ28J2⋅(Nk)2M2)) ≤ \! (- \! (- 2^Nε^28J^2· Nk^2M^2 ) ) ≈exp(−2Nϵ28J2⋅(Nk)2M2)(since e−x≈x for small x). ≈ \! (- 2^Nε^28J^2· Nk^2M^2 ) (since e^-x≈ x for small x). (19) Actually, the correct leading-order behavior from the binomial expansion (1−x)n≈e−nx(1-x)^n≈ e^-nx when x≪1x 1 is P(ϵ)≤exp(−||⋅e−2Nϵ28J2||2). P(G_ε)≤ \! (-|J|· e^- 2^Nε^28J^2|J|^2 ). For N large, the exponent 2Nϵ28J2||2→∞ 2^Nε^28J^2|J|^2→∞ provided ϵ≫2−N/2ε 2^-N/2. Thus, the inner exponential is exponentially small in 2N2^N, resulting a super-exponentially small outer probability. Precisely, P(ϵ) P(G_ε) ≤exp(−M(Nk)⋅exp(−2Nϵ2(Nk)28J2M2)) ≤ \! (- M Nk· \! (- 2^Nε^2 Nk^28J^2M^2 ) ) ≤exp(−exp(lnM(Nk)−2Nϵ2(Nk)28J2M2)). ≤ \! (- \! ( M Nk- 2^Nε^2 Nk^28J^2M^2 ) ). (20) For ϵε fixed and N large, the term 2Nϵ2(Nk)28J2M2 2^Nε^2 Nk^28J^2M^2 grows as Ω(2N/poly(N)) (2^N/ (N)) and dominates ln(M/(Nk))=O(logN)M/ (M/ Nk)=O( N). Hence, P(ϵ)≤exp(−exp(Ω(2Npoly(N)))). P(G_ε)≤ \! (- \! ( \! ( 2^N (N) ) ) ). This is the strongest possible form. Our Theorem states a slightly looser but more practical single-exponential bound, which follows from the same analysis by considering that the probability is certainly bounded by P(ϵ)≤exp(−2NM(Nk)⋅ϵ28J2+O(N)), P(G_ε)≤ \! (- 2^NM Nk· ε^28J^2+O(N) ), which is equivalent to the stated bound after adjusting constants. When k=O(1)k=O(1), we have (Nk)=Θ(Nk) Nk= (N^k). With M=poly(N)M= (N) 2NM(Nk)=2Npoly(N)⋅Θ(Nk)=Θ(2NNk+deg(M))=Ω(2Npoly(N)). 2^NM Nk= 2^N (N)· (N^k)= \! ( 2^NN^k+ (M) )= \! ( 2^N (N) ). Thus, P(ϵ)≤exp(−c⋅2NNk+deg(M)⋅ϵ2)=exp(−Ω(2Npoly(N))), P(G_ε)≤ \! (-c· 2^NN^k+ (M)·ε^2 )= \! (- \! ( 2^N (N) ) ), where c=1/(8J2⋅constants)c=1/(8J^2·constants). This completes the proof. 5.3 Proof of Lemma 4.4 In PGR, a conditional generator Gθ(z|c)G_θ(z|c) maps latent variable z and condition c to a sample x∼pdata(⋅|c)x p_data(·|c). Here x is a quantum transition τ=(s,a,s′,r)τ=(s,a,s ,r), c=Hc=H, and z encodes the entanglement structure of the state. We aim to show that the MPS to circuit mapping is exactly such a generator. An MPS with bond dimension χ has the form |ΨMPS⟩=∑i1,…,iN=01Ai1[1]Ai2[2]⋯AiN[N]|i1i2⋯iN⟩, | _MPS = _i_1,…,i_N=0^1A^[1]_i_1A^[2]_i_2·s A^[N]_i_N|i_1i_2·s i_N , with Aik[k]∈ℂχ×χA^[k]_i_k ^χ×χ and total parameter count O(Nχ2)O(Nχ^2) exponentially smaller than a generic state.DMRG solves |ΨMPS⟩=argmin|ψ⟩∈MPSχ⟨ψ|H|ψ⟩| _MPS = _|ψ _χ ψ|H|ψ , implicitly encoding c=Hc=H, we may view it as DMRG:(H,χ)↦|ΨMPS⟩DMRG:(H,χ) | _MPS . The Riemannian step then finds UTN=argmaxU∈brick|⟨ΨMPS|U|0⟩⊗N|,U_TN= U _brick | _MPS|U|0 N |, maximising overlap exactly as a generative model minimises its reconstruction loss, with ℒ(U)=|Tr(U†ρMPS)|L(U)=|Tr(U _MPS)| and ρMPS=|ΨMPS⟩⟨0|⊗N _MPS=| _MPS 0| N.Here, brickU_brick is the set of unitaries that can be expressed as a product of 2-qubit gates in a brickwork pattern, brick=∏ℓ=1L∏(i,j)∈EℓUij(ℓ):Uij(ℓ)∈(4), _brick= \ _ =1^L _(i,j)∈ E_ U_ij^( ):U_ij^( ) (4) \, where L is the number of layers and EℓE_ defines the qubit connectivity at layer ℓ . The objective function is ℒ(U)=|⟨ΨMPS|U|0⟩⊗N|=|Tr(U†ρMPS)| (U)=| _MPS|U|0 N|=|Tr(U _MPS)| In a standard conditional generative model (for instance, a conditional VAE or diffusion model), the generator is trained to maximize the likelihood of the data given the condition, θ∗=argmaxx∼pdata(⋅|c)[logpθ(x|c)].θ^*= θ \ E_x p_data(·|c)[ p_θ(x|c)]. For quantum state preparation, our data is the target state |ΨMPS⟩| _MPS , and the model distribution is the pushforward of the initial state |0⟩⊗N|0 N under the circuit U, pU(|ψ⟩)=|⟨ψ|U|0⟩⊗N|2.p_U(|ψ )=| ψ|U|0 N|^2. The log-likelihood of generating the target state |ΨMPS⟩| _MPS under this model is logpU(|ΨMPS⟩)=log|⟨ΨMPS|U|0⟩⊗N|2=2log|⟨ΨMPS|U|0⟩⊗N|. p_U(| _MPS )= | _MPS|U|0 N|^2=2 | _MPS|U|0 N|. The maximization of this log-likelihood is equivalent to maximizing the overlap |⟨ΨMPS|U|0⟩⊗N|| _MPS|U|0 N|, which is precisely the objective of the Riemannian optimization Kundu and Mangini (2026). Therefore, argmaxU∈brickℒ(U)=argmaxU∈brick|ΨMPS⟩[logpU(|ψ⟩|c=H)]. U _brick L(U)= U _brick \ E_| _MPS [ p_U(|ψ |c=H)]. The fidelity of this generation is Fgen=|⟨ΨMPS|UTN|0⟩⊗N|2.F_gen=| _MPS|U_TN|0 N|^2. By the properties of Riemannian optimization on the Stiefel manifold, the Cayley retraction method guarantees monotonic convergence of the objective. Also, the algorithm produces a sequence U(k)\U^(k)\ such that ℒ(U(k+1))≥ℒ(U(k))∀k,L(U^(k+1)) (U^(k)) ∀ k, and converges to a local maximum ℒ∗L^* of the overlap function on brickU_brick. The output of this process is a transition τinit _init: τinit=(∅,∅,UTN,rinit), _init=( , ,U_TN,r_init), where rinit=−⟨0|UTN†HUTN|0⟩r_init=- 0|U_TN HU_TN|0 is the negative energy expectation. This transition is injected into the replay buffer synD_syn before the RL agent begins training. It is a synthetic transition generated conditionally on H, exactly analogous to the synthetic transitions generated by the diffusion model in PGR when conditioned on a relevance function ℱ(τ)F(τ). The DMRG procedure can be run with different initializations or slightly different hyperparameters to produce an ensemble of MPS states |ΨMPS(j)⟩j=1K\| _MPS^(j) \_j=1^K, all approximating the ground state. Each can be mapped to a circuit UTN(j)U_TN^(j), populating the synthetic buffer with diverse yet high-quality starting points. This is analogous to the diversity induced by classifier-free guidance in PGR, where the guidance scale ω trades off between fidelity and diversity. 5.4 Proof of Lemma 4.5 Since H is a gapped local Hamiltonian in one spatial dimension, it satisfies the area law for entanglement entropy Hastings (2007). Also, there exists an MPS |ΨMPS⟩| _MPS of bond dimension χ such that ‖|ΨMPS⟩−|Ψ0⟩‖2≤ϵMPS(χ),\|| _MPS -| _0 \|_2≤ _MPS(χ), (21) where ϵMPS(χ)=CNexp(−αχ1/κ) _MPS(χ)=CN (-αχ^1/κ) for some constants C,α,κ>0C,α,κ>0 independent of N. By inverting this relation, choosing χ=O([log(CNδ/2)]κ)χ=O ( [ ( CNδ/2 ) ]^κ ) (22) assures ϵMPS≤δ/2 _MPS≤δ/2. Moreover, this χ scales only polylogarithmically with N and 1/δ1/δ, hence χ≤poly(N,1/δ)χ (N,1/δ). The mapping from the MPS to a brickwork circuit UTN=∏k=1mUkU_TN= _k=1^mU_k (with Uk∈(4)U_k (4)) is performed by maximizing the loss function ℒ(Uk)=|⟨ΨMPS|UTN|0⟩|,L(\U_k\)= | _MPS|U_TN|0 |, (23) over the Stiefel manifold (4)×mU(4)^× m. Using the Riemannian Adam optimizer with Cayley retraction Luchnikov et al. (2021), the updates maintain strict unitarity. Since the loss function is smooth and the manifold is compact, Riemannian gradient descent converges to a stationary point with polynomial iteration complexity TRiem=O(poly(m,1/δ))T_Riem=O(poly(m,1/δ)) Boumal (2020). The number of 2-qubit gates m required to represent an MPS with bond dimension χ in a brickwork structure is m=O(Nχ2)m=O(Nχ^2). Combined with the aforementioned result of MPS approximation, m=O(N⋅polylog(N/δ))m=O(N·poly (N/δ)), which is polynomial in N. This optimization yields a unitary UTNU_TN such that the overlap satisfies |⟨ΨMPS|UTN|0⟩|≥1−ϵopt, | _MPS|U_TN|0 |≥ 1- _opt, (24) which implies a 22-norm bound ‖|ΨMPS⟩−UTN|0⟩‖2≤2ϵopt\|| _MPS -U_TN|0 \|_2≤ 2 _opt. By tuning the optimization hyperparameters, we can ensure ϵopt≤δ2/8 _opt≤δ^2/8, and thus the 22-norm error is bounded by δ/2δ/2. Using the triangle inequality for the 22-norm and the relation between the 22-norm and the trace distance (12∥⋅∥1≤∥⋅∥2 12\|·\|_1≤\|·\|_2), we combine the errors from MPS approximation and Riemannian optimization. 12‖UTN|0⟩⟨0|UTN†−|Ψ0⟩⟨Ψ0|‖1 12\|U_TN|0 0|U_TN -| _0 _0|\|_1 ≤‖UTN|0⟩−|Ψ0⟩‖2 ≤\|U_TN|0 -| _0 \|_2 ≤‖UTN|0⟩−|ΨMPS⟩‖2+‖|ΨMPS⟩−|Ψ0⟩‖2 ≤\|U_TN|0 -| _MPS \|_2+\|| _MPS -| _0 \|_2 ≤δ2+δ2=δ. ≤ δ2+ δ2=δ. (25) This establishes the required bound. 5.5 Proof of Theorem 4.6 We define the true state-value function for a policy π starting from state s as Vπ(s)=π[∑t=0∞γtRt|s0=s]V^π(s)=E_π[ _t=0^∞γ^tR_t|s_0=s]. For the base algorithm, the initial value of the empty circuit is Vπbase(s0)≈E¯/(1−γ)V _base(s_0)≈ E/(1-γ). For PGR, the effective initial state distribution of the agent is shifted. The experience replay buffer D is augmented with synthetic data centered on sTNs_TN. The initial value estimate is thus biased towards VπPGR(sTN)≈ETN/(1−γ)V _PGR(s_TN)≈ E_TN/(1-γ). By Lemma 4.5, the initial suboptimality gap for PGR is Δ0PGR=ETN−E0=ΔEχ _0^PGR=E_TN-E_0= E_χ. For the baseline, it is Δ0base=E¯−E0∝N _0^base= E-E_0 N. The ratio of initial gaps is R0=Δ0PGRΔ0base=ΔEχE¯−E0.R_0= _0^PGR _0^base= E_χ E-E_0. (26) Since ΔEχ E_χ is system size independent for fixed bond dimension χ and gapped systems, while E¯−E0 E-E_0 scales as O(N)O(N), the ratio R0R_0 decays as O(1/N)O(1/N). The standard Q-learning update minimizes the squared Bellman residual ℒ(θ)=(s,a,r,s′)∼[(Qθ(s,a)−r−γmaxa′Qθ¯(s′,a′))2].L(θ)=E_(s,a,r,s ) [(Q_θ(s,a)-r-γ _a Q_ θ(s ,a ))^2 ]. (27) For the base algorithm, the buffer realD_real is dominated by trajectories starting from s0s_0. As per Lemma 4.2, the variance of the target value r+γmaxa′Q(s′,a′)r+γ _a Q(s ,a ) is exponentially small due to the barren plateau, which implies a vanishingly small gradient ∇θℒ _θL. For the PGR algorithm, the generative buffer synD_syn consists of transitions (s,a,s′,r)(s,a,s ,r) where s is sampled from a distribution concentrated around the high energy gradient region near sTNs_TN. The reward signal r=ℰ(s)−ℰ(s′)r=E(s)-E(s ) is deterministic and large (proportional to the energy gradient at sTNs_TN), bypassing the barren plateau. Let σgrad2(s)=Var[∇θQ(s,a)] _grad^2(s)=Var[ _θQ(s,a)]. We have πbase _base: s∼real[σgrad2(s)]∝exp(−N)E_s _real[ _grad^2(s)] (-N). And, πPGR _PGR: s∼syn[σgrad2(s)]∝‖∇ℰ(sTN)‖2E_s _syn[ _grad^2(s)] \| (s_TN)\|^2, which is O(1)O(1). This prevents the learning dynamics from vanishing. Consider the optimal Bellman operator T (Q)(s,a)=R(s,a)+γs′∼P(⋅|s,a)[maxa′Q(s′,a′)].(TQ)(s,a)=R(s,a)+ _s P(·|s,a)[ _a Q(s ,a )]. (28) It is well-known that T is a γ-contraction in the supremum norm, ‖Q1−Q2‖∞≤γ‖Q1−Q2‖∞\|TQ_1-TQ_2\|_∞≤γ\|Q_1-Q_2\|_∞. After k iterations, the error is bounded by ‖Qk−Q∗‖∞≤γk‖Q0−Q∗‖∞\|Q_k-Q^*\|_∞≤γ^k\|Q_0-Q^*\|_∞. The number of iterations (and thus samples) required to reach an error ϵε is K=logγ(ϵ/‖Q0−Q∗‖∞)K= _γ(ε/\|Q_0-Q^*\|_∞). For the base algorithm, ‖Q0−Q∗‖∞base≈|E¯−E0|/(1−γ)∈O(N)\|Q_0-Q^*\|_∞^base≈| E-E_0|/(1-γ)∈ O(N). For the PGR algorithm, the generative seeding provides an initial Q-function Q0PGRQ_0^PGR whose error is bounded by the energy suboptimality of the seed state ‖Q0−Q∗‖∞PGR≈ΔEχ/(1−γ)∈O(1)\|Q_0-Q^*\|_∞^PGR≈ E_χ/(1-γ)∈ O(1). The ratio of required iterations is, KPGRKbase=log(ϵ/ΔEχ)log(ϵ/N)≈log(1/ΔEχ)logN. K_PGRK_base= (ε/ E_χ) (ε/ N)≈ (1/ E_χ) N. (29) Each iteration requires interaction with the environment (or the generative model) to collect a batch of samples. The main bottleneck in QAS is the episode length T. For πbase _base, to discover a path from s0s_0 to s∗∈ℳϵcs^* _ _c, the agent must perform a random walk in the discrete circuit space of depth D. The expected number of actions to reach the target is Tbase∝||D=(O(N2))poly(N)T_base |A|^D=(O(N^2)) (N). For πPGR _PGR, the generative model GψG_ψ directly outputs transitions (s,a)(s,a) that are high relevance. If we consider the synthetic replay as performing counterfactual reasoning generating the next gate action conditioned on the target energy, it effectively reduces the depth of the required search. The agent only needs to refine the pre-existing circuit sTNs_TN, requiring a constant depth ΔD D independent of N. Thus TPGR∝||ΔD∈poly(N)T_PGR |A| D∈ (N). Combining the iteration count K and the episode length T, the total sample complexity is =K×TS=K× T. Taking the ratio, PGRbase=(log(1/ΔEχ)logN)×(||ΔD||poly(N))=O(1||poly(N))=O(1N2poly(N)). S_PGRS_base= ( (1/ E_χ) N )× ( |A| D|A| (N) )=O ( 1|A| (N) )=O ( 1N^2 (N) ). (30) This is a super-exponentially decaying ratio. There exists a constant κ>1κ>1 such that this ratio is strictly bounded above by κ−Nκ^-N for sufficiently large N, which completes the proof. 6 GenQAS warm-start from TensorRL-QAS GenQAS builds on the TensorRL (fixed) configuration of TensorRL-QAS Kundu and Mangini (2026). This component provides a problem-informed quantum state initialization before reinforcement learning (RL) architecture search begins. In contrast to TensorRL (trainable) the tensor network-derived warm start circuit is not included in the RL observation and its parameters are not subsequently optimized by the agent. Instead, it is retained as a fixed state-preparation prefix, while GenQAS searches for an additional circuit suffix. Thus, the complete circuit at RL step t is Ut=VtUTN,U_t=V_tU_TN, (31) where UTNU_TN is the fixed tensor network initialization and VtV_t is the circuit constructed by the GenQAS agent up to time t. This design separates the physically motivated state preparation from the learned architecture refinement. In particular, the RL state encodes only VtV_t, whereas the environment evaluates the full circuit in Eq. (31). Consequently, GenQAS retains the reduced RL-state dimensionality and reduced trainable parameter set of TensorRL (fixed), while augmenting its real experience with synthetic transitions generated by the learned dynamics model. 6.1 MPS approximation of the target state For a target Hamiltonian H, we first obtain an approximate ground state using DMRG White (1992); Schollwöck (2011); Orús (2014): |ψMPS⟩≈argmin|ψ⟩⟨ψ|H|ψ⟩⟨ψ|ψ⟩. _MPS≈ _ ψ ψH ψ ψ|ψ . (32) The resulting state is represented as an N-site matrix product state, |ψMPS⟩=∑i1,…,iNA1i1A2i2⋯ANiN|i1i2⋯iN⟩, _MPS= _i_1,…,i_NA_1^i_1A_2^i_2·s A_N^i_N i_1 i_2 ·s i_N, (33) where ik∈0,1i_k∈\0,1\ and the tensors Akik∈ℂdk−1×dkA_k^i_k ^d_k-1× d_k have maximum bond dimension χ. The boundary dimensions are d0=dN=1d_0=d_N=1, and dk≤χd_k≤χ for interior bonds. The bond dimension controls the entanglement that can be represented by the MPS and therefore determines the trade-off between classical preprocessing cost and warm-start accuracy. DMRG optimizes the local MPS tensors through iterative sweeps until the energy in Eq. (32) converges. We use the resulting MPS only to construct the initial circuit; the MPS itself is not passed to the GenQAS agent. TensorRL-QAS employed this procedure to generate problem-aware, compact starting circuits before RL-QAS refinement. Figure 8 contrasts the trainable and fixed TensorRL-QAS initializations; GenQAS adopts the fixed configuration shown in Fig. 8(b). 6.2 Variational MPS-to-circuit mapping We compile |ψMPS⟩ _MPS into a shallow parameterized circuit UTNU_TN acting on |0⟩⊗N 0 N Berezutskii et al. (2025); Schön et al. (2005); Ran (2020). The circuit follows a nearest neighbour brickwork layout compatible with linearly connected quantum devices Kandala et al. (2017). It comprises M two-qubit unitary blocks, UTN=UMUM−1⋯U1,Uk∈U(4),U_TN=U_MU_M-1·s U_1, U_k (4), (34) where each UkU_k acts on an adjacent pair of qubits. We determine these blocks by maximizing the overlap between the target MPS and the prepared state, equivalently minimizing ℒ(Ukk=1M)=1−|⟨ψMPS|UM⋯U1|0⟩⊗N|2.L(\U_k\_k=1^M)=1- | _MPSU_M·s U_1 0 N |^2. (35) The brickwork layout is a one-dimensional hardware-efficient ansatz (HEA) consisting of alternating layers of nearest-neighbour two-qubit operations. Its local connectivity makes it compatible with devices arranged in a linear or effectively one-dimensional topology and reduces the need for additional routing operations. However, the practical suitability of an HEA depends on both its depth and the entanglement properties of the target-state family: deep HEAs can exhibit barren plateaus, whereas shallow HEAs can remain trainable in favourable regimes, particularly for states obeying an area-law entanglement structure Leone et al. (2024). In GenQAS, the shallow brickwork circuit is not used as a generic variational ansatz initialized at random; rather, its two-qubit blocks are optimized to reproduce the DMRG-derived MPS. This tensor network-informed initialization provides a physically meaningful starting point for the subsequent RL-based circuit refinement. 6.3 Riemannian optimization on the unitary manifold The optimization variables in Eq. (35) belong to the unitary manifold U(4)U(4): Uk†Uk=I4,k=1,…,M.U_k U_k=I_4, k=1,…,M. (36) We therefore use a Riemannian version of Adam with Cayley retraction, as in Appendix B of TensorRL-QAS. Let Gk=∇UkℒG_k= _U_kL denote the Euclidean gradient obtained by automatic differentiation Baydin et al. (2018). For unitary matrices, the Riemannian gradient can be written as gradℒ(Uk)=Gk−UkGk†Uk.gradL(U_k)=G_k-U_kG_k U_k. (37) This expression lies in the tangent space of U(4)U(4) at UkU_k. For each block, Riemannian Adam maintains first- and second-moment estimates Luchnikov et al. (2021), mk(t+1) m_k^(t+1) =β1mk(t)+(1−β1)gradℒ(Uk(t)), = _1m_k^(t)+(1- _1)gradL(U_k^(t)), (38) vk(t+1) v_k^(t+1) =β2vk(t)+(1−β2)‖gradℒ(Uk(t))‖2, = _2v_k^(t)+(1- _2) \|gradL(U_k^(t)) \|^2, (39) followed by the bias-corrected tangent direction Vk(t)=−ηtm^k(t+1)v^k(t+1)+ε.V_k^(t)=- _t m_k^(t+1) v_k^(t+1)+ . (40) Here, ηt _t is the learning rate, β1 _1 and β2 _2 are the Adam decay parameters, and division by the scalar denominator is applied elementwise to the matrix-valued first moment. To update the unitary while preserving Eq. (36), we apply Cayley retraction. Defining the skew-Hermitian matrix Wk(t)=Vk(t)Uk(t)†−Uk(t)Vk(t)†,W_k^(t)=V_k^(t)U_k^(t) -U_k^(t)V_k^(t) , (41) the updated block is Uk(t+1)=(I4−12Wk(t))−1(I4+12Wk(t))Uk(t).U_k^(t+1)= (I_4- 12W_k^(t) )^-1 (I_4+ 12W_k^(t) )U_k^(t). (42) Because Wk(t)W_k^(t) is skew-Hermitian, the Cayley map in Eq. (42) preserves unitarity up to numerical precision Luchnikov et al. (2021). The momentum variables are subsequently transported to the tangent space at the updated point before the next iteration. All two-qubit blocks are updated simultaneously until the overlap loss converges. The resulting MPS-derived brickwork circuit provides the warm-start circuit used by both TensorRL-QAS configurations in Fig. 8; GenQAS retains only the fixed variant in Fig. 8(b). Figure 8: Tensor network initializations in TensorRL-QAS. (a) TensorRL (trainable): the DMRG-derived MPS is mapped to a brickwork circuit and encoded in the RL state. The RL agent can subsequently modify the warm start circuit and optimize its parameters jointly with those of the newly appended gates. (b) TensorRL-fixed TN-init: the MPS-derived circuit prepares a fixed warm-start state |ψMPS⟩χ _MPS_χ, while the RL state encodes only the circuit appended by the agent. The fixed initialization is excluded from the agent observation and remains non-trainable throughout RL search. GenQAS uses the TensorRL (fixed) configuration in (b), upon which the agent constructs the learnable circuit suffix VtV_t. Adapted from TensorRL-QAS Kundu and Mangini (2026). 6.4 Fixed initialization within GenQAS After convergence of the mapping objective, the two-qubit unitaries are decomposed into the native search and execution gate set. As illustrated in Fig. 8(b), UTNU_TN prepares the warm-start state but is excluded from the RL-state and is not modified by the GenQAS agent. In the TensorRL-QAS configuration adopted here, the compiled circuit is expressed in terms of single-qubit rotations and entangling gates, namely RX,RY,RZ,CNOT\ RX, RY, RZ, CNOT\. The resulting circuit is denoted UTNU_TN and remains unchanged throughout a GenQAS training run. Specifically, the initial quantum state used by the environment is |ψ0⟩=UTN|0⟩⊗N. _0=U_TN 0 N. (43) At time t, an action selected by the agent appends a gate to the learnable suffix VtV_t. The energy, reward, and termination condition are evaluated using Et=⟨0|UTN†Vt†HVtUTN|0⟩.E_t= 0U_TN V_t HV_tU_TN 0. (44) Crucially, the circuit representation presented to the agent contains only the appended architecture VtV_t. If the RL suffix has maximum depth D, the state dimension is therefore independent of the warm start depth DTND_TN. In contrast, a trainable TN initialization would encode both the TN prefix and the subsequently generated gates, increasing the representation from the suffix-only form to one that includes DTN+D_TN+D circuit layers. Retaining UTNU_TN as a fixed prefix reduces the observation size, avoids optimizing its parameters during each RL episode, and maintains a lower-cost environment for the generative replay agent. 6.5 Relation to the GenQAS contribution The tensor network warm start is inherited from TensorRL-QAS (fixed) and serves only as a fixed, problem informed initialization. GenQAS differs from TensorRL-QAS in the experience-learning mechanism after initialization: instead of training exclusively on transitions sampled from the real replay memory, GenQAS learns a transition-reward-termination model from real experience and uses it to generate synthetic transitions on demand. Accordingly, the tensor network module affects the initial state distribution and the circuit evaluation prefix, whereas the generative replay module affects the distribution of transitions used for DDQN updates. The two components are complementary: UTNU_TN reduces the burden of discovering a useful initial circuit from scratch, while model-based generative replay increases the effective density of training data in the remaining architecture search problem. 6.6 Warm-start resource cost and GenQAS refinement Table 2 reports the resource requirements of the fixed MPS-derived warm-start circuits after decomposition into the native gate set RX,RY,RZ,CNOT\ RX, RY, RZ, CNOT\. Here, ROT=NRX+NRY+NRZROT=N_ RX+N_ RY+N_ RZ denotes the total number of single-qubit rotations. The compiled warm-start circuits have a fixed depth of 27 across the considered systems, while the number of two-qubit gates increases from 15 CNOTs for 6-qubit BeH2BeH_2 to 42 CNOTs for the 15-qubit transverse field Ising model (TFIM). Table 2: Resource cost and energy error of the fixed MPS-derived warm-start circuits. The MPS obtained by DMRG is mapped to a brickwork circuit and decomposed into the native gate set RX,RY,RZ,CNOT\ RX, RY, RZ, CNOT\. ROT denotes the total number of single-qubit rotations. The initialization error is ϵTN=|ETN−E0| _TN=|E_TN-E_0|, and ϵGenQAS _GenQAS is the lowest error obtained by GenQAS. System ROT CNOT Depth ϵTN _TN ϵGenQAS _GenQAS RϵR_ε 6-BeH2BeH_2 93 15 27 5.96×10−35.96× 10^-3 2.3×10−52.3× 10^-5 99.6%99.6\% 8-H2OH_2O 129 21 27 2.70×10−32.70× 10^-3 1.2×10−31.2× 10^-3 55.6%55.6\% 10-H2OH_2O 165 27 27 4.77×10−34.77× 10^-3 4.1×10−44.1× 10^-4 91.4%91.4\% 12-H2OH_2O 196 32 27 5.07×10−15.07× 10^-1 2.2×10−22.2× 10^-2 95.7%95.7\% 15-TFIM 255 42 27 1.0251.025 1.5×10−21.5× 10^-2 98.5%98.5\% To quantify refinement beyond the fixed initialization, we define the warm-start and final GenQAS errors relative to the reference ground-state energy E0E_0 as ϵTN=|ETN−E0|,ϵGenQAS=|EGenQAS−E0|. _TN= |E_TN-E_0 |, _GenQAS= |E_GenQAS-E_0 |. (45) The absolute error reduction achieved by GenQAS is Δϵ=ϵTN−ϵGenQAS, ε= _TN- _GenQAS, (46) and the relative error reduction is Rϵ=100×ϵTN−ϵGenQASϵTN.R_ε=100× _TN- _GenQAS _TN. (47) GenQAS improves upon the fixed tensor network initialization across all evaluated systems. For example, for 12-qubit H2OH_2O, GenQAS reduces the MPS-to-circuit mapping error from 5.07×10−15.07× 10^-1 to 2.2×10−22.2× 10^-2. For the 15-qubit TFIM, GenQAS reduces the initialization error from 1.0251.025 to 1.5×10−21.5× 10^-2, corresponding to an absolute reduction of 1.0101.010 and a relative reduction of 98.5%98.5\%. These results demonstrate that the MPS-derived circuit provides a useful but not necessarily sufficient warm start, which is subsequently refined through RL-based architecture search. 7 Clifford synthesis We evaluate GenQAS on a discrete 4-qubit Clifford synthesis task in which the agent constructs a circuit that exactly implements a target Clifford operator. This experiment complements the molecular and TFIM benchmarks by removing variational parameter optimization and energy-expectation evaluation. The target is instead specified by an exact Clifford transformation, and success is determined through exact equality of the synthesized and target Clifford operators. Our formulation is conceptually related to prior reinforcement learning approaches to Clifford circuit synthesis, which also frame synthesis as a sequential decision problem over a Clifford gate set and use stabilizer tableau representations of the target operator Kremer et al. (2024). However, our environment is independently implemented and differs in several important respects. In particular, we construct circuits forward from the identity, represent both the current and target Clifford operators in the observation, use QASM-defined target circuits, include the tableau phase information, and train the GenQAS DDQN agent with prioritized generative replay. 7.1 Target circuits and action space Target Clifford circuits are loaded from QASM files organized by qubit number and circuit depth category and generated using the QCircuitBench dataset Yang et al. (2026). At the beginning of each episode, one target circuit is sampled uniformly from the dataset entries. The target circuit is converted to a Qiskit Clifford object, denoted CtarC_tar, while the agent starts from the identity circuit with Clifford representation C0=IC_0=I. The agent sequentially appends gates to the current circuit until the synthesized Clifford operator exactly equals CtarC_tar or the maximum synthesis horizon TmaxT_ is reached. We use Tmax=20T_ =20 in the reported Clifford synthesis experiments. The action space contains Cliff=Hq,Sq,Sq†,CNOTc,t,q∈0,…,N−1,c≠t.A_Cliff= \ H_q,\, S_q,\, S_q ,\, CNOT_c,t \, q∈\0,…,N-1\, c≠ t. (48) The resulting action-space size is |Cliff|=3N+N(N−1).|A_Cliff|=3N+N(N-1). (49) For the 4-qubit benchmark, this gives 2424 possible actions: 1212 single-qubit Clifford actions and 1212 directed CNOT actions. We apply local action masking to suppress immediately repeated or cancelling gate sequences on the same qubit. In particular, the environment masks consecutive identical H, S, or S†S actions on a qubit, masks consecutive S and S†S actions on the same qubit, and masks a repeated CNOT with the same ordered control-target pair. All actions are masked once the maximum episode length is reached. These constraints reduce locally redundant trajectories while retaining the full Clifford synthesis objective. 7.2 State representation A Clifford operator on N qubits is represented using the stabilizer tableau and a phase vector obtained by the Qiskit Clifford class. Let (C) T(C) denote the binary tableau representation of a Clifford operator C, and let (C)p(C) denote the associated phase vector. We define the vectorized Clifford representation as (C)=[vec((C)),vec((C))].v(C)= [vec( T(C)),vec(p(C)) ]. (50) Unlike representations that omit phase information, the present environment retains both tableau and phase entries because exact operator equality, including phase, defines successful synthesis. At time step t, the observation supplied to the agent consists of the target Clifford representation, the current Clifford representation, and a normalized progress variable: st=[(Ctar),(Ct),tTmax].s_t= [v(C_tar),v(C_t), tT_ ]. (51) This construction makes the target transformation, the operator synthesized so far, and the remaining effective episode horizon available to the DDQN agent. 7.3 Reward and termination To provide dense feedback before exact synthesis is achieved, we compare the current and target Clifford tableaux entrywise. Let t T_t, tp_t and tar T_tar, tarp_tar denote the tableau and phase vector of the current and target Clifford operators, respectively. The tableau-match fraction is defined as mt=∑i[t,i=tar,i]+∑j[pt,j=ptar,j]|t|+|t|.m_t= _iI [ T_t,i= T_tar,i ]+ _jI [p_t,j=p_tar,j ]| T_t|+|p_t|. (52) The associated tableau-distance proxy is dt=1−mt.d_t=1-m_t. (53) The reward after applying action ata_t is rt=−λstep−λCX[at∈CNOT]+λpartial(mt+1−mt)+Rsolve[Ct+1=Ctar],r_t=- _step- _CXI[a_t ]+ _partial (m_t+1-m_t )+R_solveI[C_t+1=C_tar], (54) where λstep=0.01,λCX=0.02,λpartial=0.5,Rsolve=10. _step=0.01, _CX=0.02, _partial=0.5, R_solve=10. (55) The first term penalizes long circuits, the second applies an additional cost to entangling operations, the third rewards local improvement in tableau agreement, and the final term provides a sparse terminal reward for exact Clifford synthesis. An episode terminates when Ct=CtarC_t=C_tar (56) or when t=Tmaxt=T_ . Exact equality is evaluated using the Qiskit Clifford representation, rather than through an approximate state or unitary-distance threshold. Therefore, a successful episode corresponds to exact synthesis of the target Clifford operation within the allowed number of construction steps. 8 GenQAS implementation details 8.1 Hyperparameters GenQAS was implemented as a prioritized generative replay extension of a Double Deep Q-Network (DDQN) agent as discussed in the main text Sec 4. The agent used a five-layer multilayer perceptron with 10001000 neurons per hidden layer, LeakyReLU activations, no dropout, and an ADAM learning rate of 3×10−43× 10^-4. All experiments employed a replay memory of 2000020000 transitions and mini-batches of 10001000 transitions. The target network was updated every 100100 optimization steps. The replay buffer was prioritized, and the conditional generative transition model was trained every 1010 steps using real replay transitions. Synthetic experience was generated on-the-fly and mixed with real transitions at generation ratios Gr∈0.2,0.4,0.6,0.8G_r∈\0.2,0.4,0.6,0.8\; all other agent and environment settings were kept fixed within each system size. For the 10- and 12-qubit H2O tasks, we used an n-step DDQN update with n=5n=5; the same n=5n=5 roll-out setting was used throughout the reported GenQAS experiments for consistency. The exploration policy was initialized with ϵ=1.0ε=1.0 and decayed multiplicatively with factor 0.999950.99995 until reaching ϵmin=0.05 _ =0.05. We used a per-layer discount factor derived from a final discount value of γfinal=0.005 _final=0.005. Circuit construction was initialized from a tensor-network prior obtained using bond dimension χ=2χ=2, with no zero-parameter initialization. All noiseless results were obtained using exact expectation-value evaluations (nshots=0n_shots=0), without error mitigation or stochastic early termination. Molecular Hamiltonians were constructed using the Jordan-Wigner mapping with tapering enabled, and continuous circuit parameters were optimized after each architectural modification using COBYLA. 8.2 Computational resources All experiments were executed on the DelftBlue high-performance computing system (DHPC) (2024). The GenQAS training and simulation workloads used the gpu-a100 partition, comprising nodes with two Intel Xeon Gold 6448Y processors (3232 cores per processor), four NVIDIA Tesla A100 GPUs with 8080 GB memory per GPU, 500500 GB system memory, and 1.51.5 TB local storage. Unless otherwise stated, each training run used 22 NVIDIA A100 GPU(s) and 44 CPU core(s). For some of the comparative analysis for GenQAS, we utilise a single node system equipped with NVIDIA RTX A6000. The molecular Hamiltonian construction, tensor network initialization, RL training, and Qulacs statevector simulations were executed using the computational resources allocated to each run. The source code, complete hyperparameter configurations, random seeds, and instructions for reproducing the experiments will be made available in the associated code repository upon publication. Table 3: Main GenQAS hyperparameters for the noiseless molecular benchmarks. The generation ratio GrG_r was swept over 0.2,0.4,0.6,0.8\0.2,0.4,0.6,0.8\ while all remaining settings were held fixed for a given qubit number. Hyperparameter 6-qubit 8-qubit 10-qubit 12-qubit System BeH2 H2O H2O H2O Training episodes 40004000 70007000 50005000 50005000 Maximum circuit layers 4747 4747 107107 7171 Acceptance error 1.6×10−31.6× 10^-3 1.6×10−31.6× 10^-3 1.6×10−31.6× 10^-3 2.54×10−22.54× 10^-2 Generation ratio, GrG_r 0.2,0.4,0.6,0.8\0.2,0.4,0.6,0.8\ 0.2,0.4,0.6,0.8\0.2,0.4,0.6,0.8\ 0.2,0.4,0.6,0.8\0.2,0.4,0.6,0.8\ 0.2,0.4,0.6,0.8\0.2,0.4,0.6,0.8\ Replay memory size 2000020000 2000020000 2000020000 2000020000 Batch size 10001000 10001000 10001000 10001000 Hidden layers [1000]5[1000]^5 [1000]5[1000]^5 [1000]5[1000]^5 [1000]5[1000]^5 Learning rate 3×10−43× 10^-4 3×10−43× 10^-4 3×10−43× 10^-4 3×10−43× 10^-4 Generator training frequency 1010 1010 1010 1010 Prioritized replay Yes Yes Yes Yes n-step return 55 55 55 55 Target-network update 100100 100100 100100 100100 ϵε decay / minimum 0.99995/0.050.99995/0.05 0.99995/0.050.99995/0.05 0.99995/0.050.99995/0.05 0.99995/0.050.99995/0.05 Final discount, γfinal _final 0.0050.005 0.0050.005 0.0050.005 0.0050.005 TN initialization / bond dimension Yes / 22 Yes / 22 Yes / 22 Yes / 22 Optimizer COBYLA COBYLA COBYLA COBYLA 8.3 Molecular Hamiltonian construction We generated the molecular electronic-structure Hamiltonians using the quantum chemistry module of PennyLane (qml.qchem) Bergholm et al. (2018); Arrazola et al. (2021). For each molecule, we specify the atomic symbols, Cartesian nuclear coordinates, total charge Q=0Q=0, spin multiplicity mult=1mult=1, and an active-space definition characterized by the number of active electrons and active orbitals. The resulting second-quantized electronic Hamiltonian was mapped to a qubit Hamiltonian using the Jordan-Wigner transformation, H=∑jcjPj,H= _jc_jP_j, (57) where cj∈ℝc_j are Pauli coefficients and Pj∈I,X,Y,Z⊗NP_j∈\I,X,Y,Z\ N are N-qubit Pauli words. The Pauli decomposition (cj,Pj)j\(c_j,P_j)\_j was stored and used as the target Hamiltonian in the variational energy objective, E(,)=⟨0|U†(,)HU(,)|0⟩,E( θ,A)= 0|U ( θ,A)HU( θ,A)|0 , (58) where A denotes the circuit architecture generated by the RL agent. We considered BEH2 BEH_2, and H2O H_2 O molecules. For BEH2 BEH_2, the nuclear geometry was specified as H(0,0,−1.330)(0,0,-1.330)-Be(0,0,0)(0,0,0)-H(0,0,1.330)(0,0,1.330), with four active electrons in three active orbitals. For H2O H_2 O, we used the geometry H(−0.021,−0.002,0.000)(-0.021,-0.002,0.000), O(0.835,0.452,0.000)(0.835,0.452,0.000), and H(1.477,−0.273,0.000)(1.477,-0.273,0.000), with four active electrons. The standard H2O instances used four active orbitals; larger active-space instances used the 6-31g basis to support the corresponding higher 10-qubit Hamiltonians. Unless a larger H2O active space was required, molecular integrals were calculated in the STO-3G basis. Our implementation used PennyLane for molecular Hamiltonian generation, PennyLane NumPy for differentiable array handling, and SciPy linear algebra routines for exact diagonalization of the Hamiltonian matrix. We additionally constructed an equivalent SparsePauliOp representation using Qiskit Quantum Information Javadi-Abhari et al. (2024). This enabled an independent consistency check between the PennyLane and Qiskit representations: we verified the agreement of the dense Hamiltonian matrices and their ground-state eigenvalues before storing each Hamiltonian. The final data file contained the dense Hamiltonian matrix, its eigenvalue spectrum, the Pauli strings, the associated coefficients, and a zero energy shift. This validation ensures that the Pauli Hamiltonian used by the RL-QAS environment is consistent with the molecular Hamiltonian generated by the quantum chemistry workflow. 8.4 Quantum simulators Although the molecular Hamiltonians were generated using PennyLane, the tensor-network warm-start component inherited from TensorRL-QAS was implemented using Qiskit Javadi-Abhari et al. (2024) and Quimb Gray (2018). In particular, Quimb was used to construct and optimize the matrix product state (MPS) approximation of the target ground state with bond dimension χ=2χ=2. The resulting MPS was subsequently converted into a quantum circuit representation using the TensorRL-QAS warm-start procedure, implemented through Qiskit. This circuit provides the physics-informed initial architecture for the RL agent before sequential gate level architecture search begins. All quantum state simulations and energ expectation value evaluations reported in this work were performed using the Qulacs Suzuki et al. (2021). For a circuit architecture A and variational parameters θ, Qulacs prepares the state |ψ(,)⟩=U(,)|0⟩⊗N, ψ( θ,A)=U( θ,A) 0 N, (59) and evaluates the molecular energy as E(,)=⟨ψ(,)|H|ψ(,)⟩=∑jcj⟨ψ(,)|Pj|ψ(,)⟩.E( θ,A)= ψ( θ,A)H ψ( θ,A)= _jc_j ψ( θ,A)P_j ψ( θ,A). (60) Here, H=∑jcjPjH= _jc_jP_j is the Jordan-Wigner-mapped Pauli Hamiltonian. Qulacs was used consistently throughout the RL-training loop. 8.5 Mechanistic signal-density analysis This section provides a mechanistic interpretation of how the generation ratio controls the effective density of useful transitions during GenQAS training. The analysis is conditional on the assumptions of Theorems 4.1 and 4.2, Lemma 4.5, and Theorem 4.6. It does not assert that every generated transition is useful; rather, it identifies the conditions under which synthetic replay can compensate for the decreasing density of useful real transitions in large QAS problems. Let ϵG_ε denote the set of circuit states with energy within ϵε of the ground state energy E0E_0. For a replay distribution D, we define the useful-transition density as ρ()=Prτ∼[s(τ)∈ϵ],ρ(D)= _τ [s(τ) _ε ], (61) where s(τ)s(τ) denotes the circuit state associated with transition τ. This quantity measures the probability that a transition sampled for a DDQN update originates from the target low-energy region. For passive replay, the replay buffer contains only transitions collected through interaction with the QAS environment. Let ρreal(N) _real(N) denote the useful-transition density in this distribution for an N qubit problem. Under the weakly informed exploration and Hamiltonian assumptions used in Theorems 4.1 and 4.2, this density satisfies ρreal(N)≤exp[−c 2Nlog(1ϵ)],c>0. _real(N)≤ [-c\,2^N ( 1ε ) ], c>0. (62) Thus, as the number of qubits increases, useful transitions constitute a rapidly decreasing fraction of the real interaction history. Uniform replay and prioritized experience replay can alter the sampling frequency of previously observed transitions, but neither can change the probability that a useful transition was collected initially. At each DDQN update, GenQAS constructs a batch of total size B by mixing real transitions sampled from the prioritized replay memory with synthetic transitions generated on demand. The generation ratio Gr∈[0,1)G_r∈[0,1) denotes the synthetic fraction of the total update batch: Breal=⌊(1−Gr)B⌋,Bsyn=B−Breal.B_real= (1-G_r)B , B_syn=B-B_real. (63) Consequently, Gr=0.4G_r=0.4 produces an update batch containing approximately 60%60\% real and 40%40\% synthetic transitions, while Gr=0.8G_r=0.8 produces approximately 20%20\% real and 80%80\% synthetic transitions. The synthetic samples are generated from state action seeds drawn from real replay memory and are not stored persistently. The replay buffer therefore remains a record of real environment interactions. Let ρsyn(N) _syn(N) denote the useful transition density among synthetic samples. Lemma 4.5 shows that the MPS derived warm start can be prepared within trace distance δ of the relevant low energy manifold under the stated MPS approximation and circuit mapping conditions. Let q∈[0,1]q∈[0,1] denote the probability that the learned local transition model preserves this useful neighborhood when generating a synthetic transition. Under this model alignment condition, ρsyn(N)≥q(1−δ). _syn(N)≥ q(1-δ). (64) The factor q separates the fidelity of the MPS derived initialization from the local predictive accuracy of the learned transition model. Accordingly, Eq. (64) is a theoretical condition, not a guarantee that follows automatically from neural network training. The generator and critic diagnostics in Figures 4 and 5 assess whether learning remains stable for the investigated generation ratios. The useful-transition density of the mixed GenQAS update distribution is ρmix(N)=(1−Gr)ρreal(N)+Grρsyn(N). _mix(N)=(1-G_r) _real(N)+G_r _syn(N). (65) Combining Eqs. (62) and (64) gives ρmix(N)≥(1−Gr)ρreal(N)+Grq(1−δ). _mix(N)≥(1-G_r) _real(N)+G_rq(1-δ). (66) The corresponding gain relative to passive replay is ρmix(N)ρreal(N) _mix(N) _real(N) ≥(1−Gr)+Grq(1−δ)1ρreal(N) ≥(1-G_r)+G_rq(1-δ) 1 _real(N) ≥(1−Gr)+Grq(1−δ)exp[c 2Nlog(1ϵ)]. ≥(1-G_r)+G_rq(1-δ) [c\,2^N ( 1ε ) ]. (67) Equation (67) describes the proposed mechanism of generative replay. If the MPS warm start remains sufficiently accurate, such that δ is small, and the learned dynamics model preserves local useful-transition structure with nonzero probability q, the synthetic contribution does not vanish with system size. This contrasts with the useful density of passive replay, which decreases rapidly under the assumptions of Eq. (62). Finally, consider a regime in which a policy requires K useful replay transitions to achieve a fixed reduction in value-estimation error. The expected number of sampled replay transitions required to obtain these transitions scales approximately as [M]≈Kρ.E[M]≈ Kρ. (68) Therefore, [Mreal][Mmix]≈ρmix(N)ρreal(N). E[M_real]E[M_mix]≈ _mix(N) _real(N). (69) Together, Eqs. (67) and (69) provide an intuitive connection between useful-transition densification and the episode complexity result in Theorem 4.6. The finite-size results in Figure 2(a) do not establish an asymptotic scaling law, but they are consistent with the predicted increase in the relative value of generative replay as the passive replay signal becomes sparse.