Paper deep dive
Replay-buffer engineering for noise-robust quantum circuit optimization
Akash Kundu, Sebastian Feld
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 96%
Last extracted: 4/26/2026, 9:07:19 PM
Summary
The paper introduces 'Replay-buffer engineering' as a method to optimize quantum circuit design using Deep Reinforcement Learning (RL). It addresses three bottlenecks: unreliable TD targets, high evaluation costs in curriculum learning, and the loss of noiseless training data when moving to noisy hardware. The authors propose three key innovations: ReaPER++, an annealed replay rule that transitions from TD-error prioritization to reliability-aware sampling; OptCRLQAS, which amortizes expensive quantum-classical evaluations over multiple architectural edits; and a lightweight buffer-only transfer scheme that reuses noiseless trajectories to warm-start learning in noisy environments. Experimental results show significant gains in sample efficiency, wall-clock time reduction, and noise robustness across quantum compilation and molecular task benchmarks.
Entities (8)
Relation Signals (4)
Akash Kundu → affiliatedwith → Delft University of Technology
confidence 100% · Akash Kundu1,2 ... 1Delft University of Technology
ReaPER++ → improves → Sample Efficiency
confidence 95% · achieving 4-32x gains in sample efficiency over fixed PER, ReaPER, and uniform replay
OptCRLQAS → reduces → Wall-clock Time
confidence 95% · cutting wall-clock time per episode by up to 67.5% on a 12-qubit optimization problem
Replay-buffer transfer → enables → Noise-robustness
confidence 90% · reduces steps to chemical accuracy by up to 85-90% and final energy error by up to 90% over from-scratch baselines
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Deep reinforcement learning (RL) for quantum circuit optimization faces three fundamental bottlenecks: replay buffers that ignore the reliability of temporal-difference (TD) targets, curriculum-based architecture search that triggers a full quantum-classical evaluation at every environment step, and the routine discard of noiseless trajectories when retraining under hardware noise. We address all three by treating the replay buffer as a primary algorithmic lever for quantum optimization. We introduce ReaPER$+$, an annealed replay rule that transitions from TD error-driven prioritization early in training to reliability-aware sampling as value estimates mature, achieving $4-32\times$ gains in sample efficiency over fixed PER, ReaPER, and uniform replay while consistently discovering more compact circuits across quantum compilation and QAS benchmarks; validation on LunarLander-v3 confirms the principle is domain-agnostic. Furthermore we eliminate the quantum-classical evaluation bottleneck in curriculum RL by introducing OptCRLQAS which amortizes expensive evaluations over multiple architectural edits, cutting wall-clock time per episode by up to $67.5\%$ on a 12-qubit optimization problem without degrading solution quality. Finally we introduce a lightweight replay-buffer transfer scheme that warm-starts noisy-setting learning by reusing noiseless trajectories, without network-weight transfer or $\epsilon$-greedy pretraining. This reduces steps to chemical accuracy by up to $85-90\%$ and final energy error by up to $90\%$ over from-scratch baselines on 6-, 8-, and 12-qubit molecular tasks. Together, these results establish that experience storage, sampling, and transfer are decisive levers for scalable, noise-robust quantum circuit optimization.
Tags
Links
- Source: https://arxiv.org/abs/2604.21863v1
- Canonical: https://arxiv.org/abs/2604.21863v1
Trouble viewing inline? Open PDF directly →
Full Text
96,551 characters extracted from source content.
Expand or collapse full text
Replay-buffer engineering for noise-robust quantum circuit optimization Akash Kundu1,2 Sebastian Feld1,2 1Delft University of Technology, Delft, The Netherlands 2Quantum Computing Division, QuTech, The Netherlands A.kundu, S.Feld@tudelft.nl Corresponding author. Abstract Deep reinforcement learning (RL) for quantum circuit optimization faces three fundamental bottlenecks: replay buffers that ignore the reliability of temporal-difference (TD) targets, curriculum-based architecture search that triggers a full quantum-classical evaluation at every environment step, and the routine discard of noiseless trajectories when retraining under hardware noise. We address all three by treating the replay buffer as a primary algorithmic lever for quantum optimization. We introduce ReaPER++, an annealed replay rule that transitions from TD error-driven prioritization early in training to reliability-aware sampling as value estimates mature, achieving 4-32× gains in sample efficiency over fixed PER, ReaPER, and uniform replay while consistently discovering more compact circuits across quantum compilation and QAS benchmarks; validation on LunarLander-v3 confirms the principle is domain-agnostic. Furthermore we eliminate the quantum-classical evaluation bottleneck in curriculum RL by introducing OptCRLQAS which amortizes expensive evaluations over multiple architectural edits, cutting wall-clock time per episode by up to 67.5%67.5\% on a 12-qubit optimization problem without degrading solution quality. Finally we introduce a lightweight replay-buffer transfer scheme that warm-starts noisy-setting learning by reusing noiseless trajectories, without network-weight transfer or ϵε-greedy pretraining. This reduces steps to chemical accuracy by up to 85-90% and final energy error by up to 90%90\% over from-scratch baselines on 6-, 8-, and 12-qubit molecular tasks. Together, these results establish that experience storage, sampling, and transfer are decisive levers for scalable, noise-robust quantum circuit optimization. 1 Introduction Optimization is one of the most important application areas of quantum computing Abbas et al. (2024); Farhi et al. (2014), both as a target problem class and as an internal algorithmic task in the design and execution of quantum algorithms. On current quantum hardware, quantum computation is realized through sequences of elementary logic gates arranged into quantum circuits Devoret and Schoelkopf (2013); Croot et al. (2025); Kjaergaard et al. (2020), and their depth, gate count, and hardware compatibility of these circuits directly determines what can be achieved in practice Kandala et al. (2017); Kim et al. (2023). This makes circuit optimization a central challenge across both near-term and fault-tolerant regimes. In the near term Preskill (2018); Bharti et al. (2022), limited coherence, connectivity, and high gate noise place strong pressure on circuit depth, two-qubit gate count, and hardware compatibility. In the longer term Fowler et al. (2012); Forster et al. (2025); Zimborás et al. (2025), fault-tolerant quantum computing will relax some hardware constraints but will not remove the need for efficient circuit synthesis, compilation, and task-adapted circuit design. Across both settings, the ability to construct compact and effective quantum circuits remains a key determinant of performance, resource cost, and practical utility. Figure 1: Overview of replay-buffer engineering for quantum optimization. (Left) Buffer engineering improves experience reuse through replay design and sampling. (Middle) Amortized learning reduces the cost of curriculum RL-based quantum architecture search by performing expensive quantum-classical updates only every m steps. (Right) Noise-aware transfer warm-starts the RL-training in noisy environment by reusing trajectories collected in noiseless training. Reinforcement learning (RL) Sutton et al. (1998) has emerged as a promising framework for this problem because it treats circuit design as a sequential decision process, enabling agents to construct gate sequences or parameterized circuit architectures step by step Ostaszewski et al. (2021); Bukov and Marquardt (2026). RL has been successfully applied to quantum compilation Moro et al. (2021); Wang et al. (2024) and variational quantum circuit design Ostaszewski et al. (2021); Kundu (2024). However, its practical use remains limited by a basic inefficiency: past experience is often poorly reused. In most existing workflows Ostaszewski et al. (2021); Patel et al. (2024); Sweke et al. (2021); Reuer et al. (2023), each new setting is treated as a fresh training problem, so trajectories collected in a noiseless simulator are discarded when the task is revisited under hardware noise which is an especially costly choice in quantum optimization LeBlond et al. (2024), where the noiseless-to-noisy gap represents a shift between fundamentally distinct computational substrates that is physically motivated, practically unavoidable, and grows in severity with system size, making experience reuse both more challenging and more valuable than in classical transfer settings. In the standard variational framework Cerezo et al. (2021), a parameterized quantum circuit U(,)U( θ,A) is defined by a discrete architecture A and continuous parameters θ, preparing a state |ψ(,)⟩=U(,)|0⟩⊗n ψ( θ,A)=U( θ,A) 0 n, with cost C(,)=⟨ψ(,)|H|ψ(,)⟩C( θ,A)= ψ( θ,A)H ψ( θ,A) estimated on the QPU and a classical optimizer updating θ toward targets d(U(,),Utar)≤εcompd(U( θ,A),U_tar)≤ _comp or C(,)−E0≤εchemC( θ,A)-E_0≤ _chem. Each RL step may trigger a full quantum-classical evaluation, so wall-clock time grows rapidly with system size. While TensorRL-QAS Kundu and Mangini (2025) narrows the search space via tensor-network warm-starts, it still incurs substantial per-step cost and relies on standard replay buffers without explicit experience transfer; similarly, transfer in RL-optimized error-correcting codes Zen et al. (2025) does not treat replay-buffer design as the primary transfer vehicle. As a result, the practicality of RL-based quantum optimization depends not only on the agent architecture, but critically on how experience is stored, prioritized, and transferred across the noiseless-to-noisy scenario which is a direction that is not explored yet in the scope of quantum optimization. We address these bottlenecks by treating the replay buffer as a primary algorithmic lever for making RL practical in quantum optimization. This is motivated by evidence from classical RL that buffer composition and sampling strongly affect stability and sample efficiency Hester et al. (2018); Zhou et al. (2025); Lee et al. (2021); Guo et al. (2024); Song et al. (2025). Figure 1 summarizes our replay-buffer engineering framework, which combines annealed replay, amortized curriculum learning, and lightweight noiseless-to-noisy transfer. Concretely, we make three contributions: 1. Annealed replay for quantum optimization. We systematically study replay design for quantum compiling and architecture search under a fixed DQN Mnih et al. (2013)/DDQN Van Hasselt et al. (2016) setup, isolating the effect of the buffer from the agent or environment. Based on this analysis, we introduce ReaPER++, an annealed replay rule that transitions from TD-error prioritization Schaul et al. (2015) early in training to reliability-aware replay Pleiss et al. (2026) later on, improving sample efficiency by 4×4×-32×32× over fixed PER, fixed ReaPER, and uniform replay while finding more compact circuits. A classical RL validation on LunarLander-v3 (Appendix E) confirms that ReaPER++ is not domain-specific. 2. Amortized curriculum learning. We introduce OptCRLQAS, an amortized variant of CRLQAS Patel et al. (2024) that reuses expensive quantum-classical evaluations across multiple architectural edits. On 12-qubit H2O H_2 O ground-state preparation, OptCRLQAS reduces average wall-clock time per episode by 67.5%67.5\% (3×3× faster) without degrading final energy error or gate count. 3. Replay-based transfer to noise. We propose a lightweight transfer scheme that reuses noiseless trajectories to warm-start learning in noisy settings, without network-weight transfer or long ϵε-greedy pretraining. On 6-qubit BEH2 BEH_2, and 8- and 12-qubit H2O H_2 O molecular tasks, this buffer-only transfer cuts the steps needed to reach chemical accuracy by up to 8585-90%90\% and improves final energy error by up to 90%90\% over from-scratch noisy baselines, yielding a transfer advantage that grows with system size. 2 Related work Quantum circuit optimization Abbas et al. (2024) has been approached via adaptive ansatz construction Grimsley et al. (2019), differentiable architecture search Zhang et al. (2022); Wu et al. (2023), evolutionary heuristics Rasconi and Oddi (2019); Sünkel et al. (2023), Bayesian optimization Nicoli et al. (2023), Monte Carlo tree search Wang et al. (2023), neural predictors Zhang et al. (2021), and sampling-based strategies Du et al. (2022), demonstrating automatic circuit structure discovery but revealing persistent challenges in scaling and noise robustness. Within this landscape, reinforcement learning has emerged as a flexible framework for both quantum compiling Moro et al. (2021) and quantum architecture search Ostaszewski et al. (2021); Fösel et al. (2021); Kuo et al. (2021), with value-based and policy-gradient methods applied to variational ground-state preparation Ostaszewski et al. (2021); Patel et al. (2024); Kundu and Mangini (2025), entangled-state generation Kuo et al. (2021); Kundu et al. (2024), and hardware-aware circuit design Kremer et al. (2024); Kundu and Sarra (2026). We refer the reader the curated list Kundu (2025) for a complete list of automated quantum circuit optimization approaches. However, existing RL-based frameworks treat each new setting as a fresh training instance, when a task moves from a noiseless simulator to noisy hardware, the agent is retrained from scratch and accumulated experience is discarded, incurring substantial GPU and CPU cost per retraining cycle. Experience replay is a central component of off-policy deep RL Mnih et al. (2013), with uniform replay, hindsight experience replay Andrychowicz et al. (2017), prioritized experience replay Schaul et al. (2015), and reliability-aware variants Pleiss et al. (2026) providing increasingly structured trade-offs between coverage, informativeness, and target reliability. In classical RL, recent offline-to-online methods Zhou et al. (2025); Lee et al. (2021); Guo et al. (2024); Song et al. (2025) demonstrate that buffer composition and sampling rules are critical for stable, sample-efficient fine-tuning. In quantum optimization Patel et al. (2024); Zen et al. (2025), however, replay buffers have largely been treated as fixed implementation choices rather than as a primary algorithmic design lever. This is the gap our work directly addresses by treating the replay buffer as the central object of design for quantum circuit optimization. 3 Methods We introduce a replay-buffer engineering algorithm for quantum circuit optimization as depicted in Figure 1 with three components: (1) annealed replay (ReaPER+, described in the next paragraph), (2) amortized curriculum learning (OptCRLQAS), and (3) lightweight noiseless-to-noisy buffer transfer. To isolate the effect of replay design from other algorithmic choices, all experiments use a common off-policy deep Q-learning (for quantum compiling) and double deep Q-learning (for quantum architecture search) agent with fixed state representation, action space, reward, and training protocol; only the replay mechanism is varied. ReaPER++. Among existing strategies, PER and ReaPER emerge as complementary baselines. PER (see Equation 11) prioritizes transitions with large TD errors, providing aggressive early exploration but potentially amplifying noisy targets. ReaPER (see Equation 13) discounts transitions whose downstream TD errors indicate unreliable targets, better matching the long-horizon structure of episodic quantum optimization but converging more slowly in the early phase. This complementary behavior motivates a hybrid that transitions from PER-like exploration to ReaPER-like refinement over the course of training. To exploit the best of both regimes, we introduce ReaPER+, a replay strategy that transitions smoothly from PER-like prioritization to ReaPER-like prioritization over the course of training. Specifically, at training step τ we define Ψt(+,τ)=Rtωτ(δt+)α,μt(+,τ)=Ψt(+,τ)∑iΨi(+,τ), _t^(+,τ)=R_t _τ( _t^+)^α, _t^(+,τ)= _t^(+,τ) _i _i^(+,τ), (1) where the annealing exponent ωτ∈[0,1] _τ∈[0,1] is non-decreasing in τ; ωτ=0 _τ=0 recovers PER and ωτ=1 _τ=1 recovers ReaPER exactly. In practice, we use the linear schedule ωτ=ωmin+(ωmax−ωmin)min(τTann, 1), _τ= _ +( _ - _ ) \! ( τT_ann,\,1 ), (2) which is monotone and controlled by a single interpretable timescale TannT_ann, avoiding the rapid early transitions of cosine or exponential schedules when reliability estimates are still poorly calibrated. We set ωmin=0.1>0 _ =0.1>0 so that a residual TD-error signal is retained at initialization, where ℛtR_t carries little meaning under a random Q-function, and ωmax=0.7<1 _ =0.7<1 to prevent premature over-commitment to reliability scores before Q-function convergence (values ωmax≥0.9 _ ≥ 0.9 produced slower convergence in both settings; see Appendix I). TannT_ann is set to half the total training budget (5×1055× 10^5 for compilation, 5×1045× 10^4 for LunarLander-v3; Table 9), so the transition completes by mid-training when value estimates are sufficiently stable. Early in training sampling is thus driven by TD error; later, the influence of RtR_t grows, biasing replay toward transitions that are both informative and reliable. ReaPER+ therefore preserves the sample-efficiency advantages of PER at the beginning of learning while inheriting the stability of ReaPER once value estimates mature, a design choice directly motivated by the empirical finding that PER excels early while ReaPER is preferable once estimates stabilize this is confirmed by our systematic benchmark of uniform replay, HER Andrychowicz et al. (2017), PER Schaul et al. (2015), and ReaPER Pleiss et al. (2026) under a fixed agent across all tasks. A theoretical justification is given in Appendix C. Lightweight buffer transfer. In addition to replay-buffer engineering, we study a replay buffer transfer scheme for noise-robust quantum optimization. The idea is to first collect trajectories in a source environment, typically the noiseless version of a given compiling or QAS task, and then use the resulting replay memory to initialize training in the corresponding noisy target environment. Let ℬsrcB_src denote the replay buffer obtained after source training and ℬtgt(0)B_tgt^(0) the initial replay buffer of the target task. We initialize the target buffer directly as ℬtgt(0)←ℬsrc,B^(0)_tgt _src, (3) that is, all stored transitions (St,At,Rt,St+1,dt)(S_t,A_t,R_t,S_t+1,d_t) are copied from the source buffer to the target buffer without modification, relabeling, or filtering. This is valid because the noiseless and noisy environments share identical state and action spaces: the RL-state and the gate action set A are unchanged by the introduction of noise, which only affects the transition dynamics and reward statistics. Buffer-only transfer of this kind, where source experience warm-starts a target replay buffer without sharing network weights, has recently proved effective in classical deep RL. Zhou et al. Zhou et al. (2025) show that seeding the online replay buffer with offline transitions suffices for sample-efficient fine-tuning without retaining the offline dataset or transferring network parameters. Lee et al. Lee et al. (2021) study balanced mixing of offline and online replay in the offline-to-online setting, finding that the buffer composition critically determines early learning speed. Our scheme follows the same instance-transfer paradigm You et al. (2022): experience is transferred at the trajectory level rather than the parameter level, deliberately decoupling the two mechanisms so that their individual contributions can be isolated through controlled ablations that separately compare buffer transfer, network initialization transfer, and their combination. Consequently, trajectories that are informative in the noiseless setting can still provide a strong initialization for learning under noise by improving early buffer coverage and accelerating the discovery of high-quality circuits. In Appendix D we provide a formal argument for why replay-buffer transfer is natural in quantum optimization settings under the assumption that source and target tasks share approximately similar state and action spaces. OptCRLQAS. Finally, to scale while operating with CRLQAS we realized the time per episode scales rapidly with qubit count, and training the RL-agent with CRLQAS beyond 10-qubit can take a substantial amount of GPU compute. To address this issue we introduce OptCRLQAS, an efficiency-oriented variant of curriculum reinforcement learning-based quantum architecture search. In standard CRLQAS, each environment step triggers a full quantum-classical evaluation of the current circuit, including variational parameter optimization and cost-function estimation, so for an episode of length T, the total evaluation cost scales as T⋅CevalT· C_eval, where CevalC_eval denotes the cost of a single quantum-classical call. OptCRLQAS reduces this by amortizing each evaluation over m consecutive architectural edits: rather than invoking a new variational optimization at every step, the agent accumulates m local gate modifications before triggering a single evaluation, spreading the cost CevalC_eval across m steps. Formally, let uτ∈0,1u_τ∈\0,1\ denote the update indicator, uτ=1,if τmodm=0 or the episode terminates,0,otherwise,u_τ= cases1,&if τ m=0 or the episode terminates,\\ 0,&otherwise, cases (4) so that a full architecture evaluation is performed only when uτ=1u_τ=1. Hence, for an episode of length T, the number of expensive quantum-classical evaluations is reduced from T in CRLQAS to approximately ⌈T/m⌉ T/m in OptCRLQAS, yielding an expected reduction by a factor of about m when episode lengths are sufficiently large. Beyond the computational saving, accumulating m edits before evaluating also improves the learning signal: single-gate modifications are often too small to produce distinguishable rewards when variational parameters θ can compensate local changes, leading to weakly separated TD targets and slow value propagation. Judging a block of m edits jointly increases reward contrast and allows useful gate combinations to be credited as a unit. A more detailed discussion of both effects is provided in Section 4.2. Throughout the paper, OptCRLQAS refers to m=10m=10 unless stated otherwise. Figure 2: 1-qubit compiling of Haar-random target unitaries with RX,RY,RZ(±π/128) RX, RY, RZ(±π/128) gates. (Left) Success probability and mean fidelity at tolerances 0.990.99, 0.9990.999, and 0.99990.9999, where ReaPER++ performs best overall. (Right) mean circuit length with std. dev. error bars versus tolerance; although all methods require deeper circuits at higher accuracy and exhibit a similar growth rate with tightening tolerance, ReaPER++ maintains a consistently lower circuit-length offset, giving the best accuracy-length tradeoff across all tolerance levels. Tol Method Success (%) ↑ Avg. fidelity ↑ 0.99 ReaPER+ (Ours) 89.30 0.94 PER Schaul et al. (2015) 85.81 0.94 ReaPER Pleiss et al. (2026) 85.15 0.93 HER Andrychowicz et al. (2017) 76.49 0.89 PPO Schulman et al. (2017) 75.40 0.88 0.999 ReaPER+ (Ours) 85.30 0.95 PER 82.35 0.94 ReaPER 80.97 0.93 HER 74.44 0.90 PPO 71.60 0.87 0.9999 ReaPER+ (Ours) 81.40 0.95 PER 72.66 0.94 ReaPER 64.35 0.93 HER 63.30 0.90 PPO 69.90 0.87 4 Results We evaluate replay-buffer design in quantum compilation, where the agent must synthesize a target unitary with encoded in an RL-state of size 2×2N2× 2^N by sequentially appending gates from a fixed action space defined in Section B.2.1. In every training episode the environment resets to the |0⟩ 0 and a fresh Haar-random 1-qubit target unitary is sampled; the agent acts until the target fidelity tolerance is met or the maximum episode length L is reached. All agents are trained for 5×1045× 10^4 episodes and evaluated on 10510^5 independently sampled Haar-random target unitaries. Full hyperparameter details are given in Appendix M. 4.1 Quantum compiling We evaluate replay-buffer design in quantum compiling, where the agent synthesizes a target unitary by sequentially appending gates from the action spaces defined in Section B.2.1. All agents are trained for 5×1045× 10^4 episodes and evaluated on 10510^5 Haar-random target unitaries (hyperparameters in Appendix M). 1-qubit compiling. We evaluate two settings: (i) the small-rotation basis RX,RY,RZ(±π/128) RX, RY, RZ(±π/128) and (i) the discrete HRC gate set Harrow et al. (2002), following Ref. Moro et al. (2021). Figure 2 reports success probability and fidelity over tolerances 0.990.99-0.99990.9999 (40 seeds). ReaPER++ achieves the highest success at every tolerance (89.30%89.30\%, 85.30%85.30\%, 81.40%81.40\%), outperforming PER (85.81%85.81\%, 82.35%82.35\%, 72.66%72.66\%) and fixed ReaPER (85.15%85.15\%, 80.97%80.97\%, 64.35%64.35\%), while showing the slowest circuit-length growth, yielding the best success-fidelity-length tradeoff overall. For the HRC basis, Table 4 shows ReaPER++ reaches 100%100\% success with mean fidelity 0.9950.995 and shortest circuits (14.30±7.8914.30± 7.89 gates) at 1.56×1061.56× 10^6 steps, ∼24% \!24\% faster than fixed ReaPER, ∼26% \!26\% faster than PER, and ∼72% \!72\% faster than our tuned HER baseline (k=5k=5 relabelings, 100-episode warm-up), which despite being stronger than the original Moro et al. (2021) still plateaus at 95%95\% success. All subsequent “HER” references denote this tuned implementation. 2-qubit compiling. Using target unitaries sampled via Algorithm 1, we test whether agents can approximate a Z(π) Z(π) gate at fidelity threshold 0.99140.9914, matching the benchmark of Ref. Moro et al. (2021). Table 1 shows ReaPER++ reaches fidelity 0.99200.9920 in only 2.5×1042.5× 10^4 episodes, a 4×4× reduction over fixed ReaPER, PER, and HER (all at 10510^5 episodes), and a ×32× reduction over PPO Moro et al. (2021) (8×1058× 10^5 episodes). Although fixed ReaPER attains a marginally higher best fidelity (0.99310.9931) given its longer budget, ReaPER++ reaches a comparable fidelity regime far more efficiently, the operationally relevant metric when quantum-classical simulation time is limited. Table 1: ReaPER+ outperforms all baselines on ZZ(π)Z(π) approximation using 2-qubit gates. ReaPER++ matches or exceeds fidelity at 32×32× fewer episodes than PPO Moro et al. (2021). Method Episodes ↓ Min. gates ↓ Best fidelity ↑ ReaPER+ (Ours) 2.5×2.5× 10^4 123 0.99200.9920 ReaPER Pleiss et al. (2026) 10510^5 126126 0.99310.9931 PPO (Moro et al. Moro et al. (2021)) 8×1058× 10^5 122122 0.99140.9914 PER Schaul et al. (2015) 10510^5 127127 0.99180.9918 HER Andrychowicz et al. (2017) 10510^5 NA <0.9914<0.9914 4.2 Quantum architecture search We evaluate replay buffers in QAS, where the agent seeks a parameterized circuit that prepares the ground state of a target molecular Hamiltonian. Following CRLQAS Patel et al. (2024); Ostaszewski et al. (2021), we use the gate set in Eq. 19 and reward in Eq. 22, benchmarking molecular ground-state preparation tasks of increasing scale (molecule geometries and hyperparameters in Appendices J and K respectively). Table 2: OptCRLQAS + ReaPER++ vs. non-RL baselines. Our method achieves the lowest energy error across all systems while using competitive or fewer gates. The 5-qubit Heisenberg model is described in Appendix J.2. For 5-Heisenberg we utilize an action space consists of X(θxx),Y(θyy),Z(θzz),RX(θx),RY(θy),RZ(θz)\ X( _x), Y( _y), Z( _z), RX( _x), RY( _y), RZ( _z)\ gates. Problem Method Min error (Ha) Total gates CNOT 5-Heisenberg OptCRLQAS + ReaPER++ (ours) 5.9×−5.9× 10^-4 41 NA DQAS Zhang et al. (2022) 1.1×10−11.1× 10^-1 35 NA GQAS He et al. (2024b) 7.1×10−47.1× 10^-4 35 NA TF-QAS He et al. (2024a) 1.2×10−31.2× 10^-3 35 NA 6-BEH2 BEH_2 OptCRLQAS + ReaPER++ (ours) 5.8×−5.8× 10^-5 54 12 TF-QAS He et al. (2024a) 1.8×10−31.8× 10^-3 57 NA SA-QAS Lu et al. (2023) 5.6×10−35.6× 10^-3 73 45 8-H2O H_2 O OptCRLQAS + ReaPER++ (ours) 1.2×−1.2× 10^-4 134 52 quantumDARTS Wu et al. (2023) 1.7×10−41.7× 10^-4 219 68 SA-QAS Lu et al. (2023) 2.6×10−32.6× 10^-3 95 69 Figure 3 compares replay strategies on the smaller-scale systems. ReaPER++ achieves the lowest energy error among prioritized methods with competitive circuit compactness; fixed ReaPER (ω=0.4ω=0.4 for BEH2 BEH_2, ω=0.6ω=0.6 for H2O H_2 O) produces the most compact circuits, reflecting the longer-horizon credit assignment at larger scale (full ω sensitivity in Appendix I). Uniform replay yields superficially shorter circuits but at substantially higher energy error, indicating early trapping in local minima. As shown in Table 2, OptCRLQAS with ReaPER++ (denoted as “OptCRLQAS + ReaPER++”) achieves the lowest energy error across 5-, 6-, and 8-qubit QAS problems, outperforming non-RL baselines such as DQAS Zhang et al. (2022), GQAS He et al. (2024b), TF-QAS He et al. (2024a), SA-QAS Lu et al. (2023), and quantumDARTS Wu et al. (2023) while using competitive or fewer gates. For the 5-qubit problem, we target the ground-state energy of the Heisenberg model (Hamiltonian given in Appendix J.2). Since this problem is well-suited to the gateset RXX(θxx),RYY(θyy),RZZ(θzz),RX(θx),RY(θy),RZ(θz)\ RXX( _x),\, RYY( _y),\, RZZ( _z),\, RX( _x),\, RY( _y),\, RZ( _z)\, we employ an alternative RL state encoding described in Appendix L. Figure 3: Replay-buffer design controls circuit compactness in QAS. For 6-BEH2 BEH_2 and 8-H2O H_2 O, ReaPER++ variants yield the lowest total, CNOT, and rotation gate counts compared to PER and uniform replay (mean ± std over seeds). ω=0ω=0 recovers PER; ω=1ω=1 gives fully reliability-adjusted replay. At 12-qubit scale, standard CRLQAS is prohibitively expensive since every step triggers a full variational optimization. OptCRLQAS amortizes this cost over m steps, reducing evaluations per episode from T to ⌈T/m⌉ T/m and cutting average wall-clock time by 67.5%67.5\% (∼3× 3×) without degrading solution quality (Fig. 4). Batching m edits also yields a more separable learning signal by crediting meaningful architectural blocks rather than nearly indistinguishable single-step edits. Figure 4: Efficiency and performance on 12-qubit H2O. (Left) OptCRLQAS reduces wall-clock time per episode by 67.5%67.5\% over CRLQAS Patel et al. (2024). (Right) ReaPER achieves the lowest minimum energy error and fastest convergence across all replay baselines. Method Min err. ↓ Gates ↓ CNOT ↓ Steps ↓ ReaPER 1.7×−1.7× 10^-2 196 109 1.6×1.6× 10^4 ReaPER++ 2.3×10−22.3× 10^-2 241 94 3.7×1043.7× 10^4 PER 2.5×10−22.5× 10^-2 121 66 9.2×1049.2× 10^4 Vanilla 2.5×10−22.5× 10^-2 151 91 1.7×1041.7× 10^4 With OptCRLQAS enabling 12-qubit training, ReaPER attains the lowest minimum energy error (1.7×10−21.7× 10^-2 Ha) in the fewest steps (1.6×1041.6× 10^4), while PER reaches a similar floor at 5.7×5.7× the cost and vanilla replay converges rapidly to a worse solution. Appendix H further shows OptCRLQAS can reduce quantum simulation time by up to 89%89\% and classical optimization time by up to 85%85\% in matched comparisons. 4.3 Noise-robust learning through buffer transfer We evaluate the transfer scheme of Section 3, in which a buffer ℬsrcB_src collected in a noiseless source environment initializes training in a related noisy target, to our knowledge, the first demonstration that a lightweight, weight-free replay buffer alone suffices to transfer noiseless experience to realistic depolarizing-noise settings, with advantages that grow with system size up to 12-qubit. All experiments use OptCRLQAS with uniform replay to isolate the effect of transfer. We study (i) noiseless-to-noiseless and (i) noiseless-to-noisy transfer (depolarizing noise with single-qubit strength p1p_1, two-qubit strength p2p_2) on molecular ground-state preparation benchmarks of increasing scale. The source buffer is obtained by training a vanilla agent for a fixed 12 GPU-hour budget; to exploit the warm start, we reduce initial exploration from ϵ=1.0ε=1.0 to 0.550.55 and tighten the curriculum (Appendix K). We quantify transfer via a multi-objective score S=w1Δsteps+w2ΔROT+w3ΔCNOT+w4ΔerrS=w_1\, +w_2\, +w_3\, CNOT+w_4\, (weights 0.4,0.1,0.2,0.30.4,0.1,0.2,0.3), measuring relative improvement over the no-transfer baseline in steps to chemical accuracy, rotation and CNOT gate counts, and best energy error Ikhtiarudin et al. (2025). Figure 5: Weighted transfer matrix for BEH2 BEH_2 under noiseless and noisy transfer. Buffer transfer reduces steps to chemical accuracy by 4747-58%58\% and improves final energy by up to 90.2%90.2\% across all noise settings, yielding composite scores of 19.219.2-35.8%35.8\%. The strongest score (35.8%35.8\%) is driven by the largest energy improvement at p2=0.001p_2=0.001. Figure 6: Weighted transfer matrix for H2O H_2 O under noiseless and noisy transfer. Step reductions range from 49.8%49.8\% to 84.8%84.8\%, and energy improvements reach 46.7%46.7\% under combined noise (p1=0.001p_1=0.001, p2=0.005p_2=0.005), yielding the highest score of 28.7%28.7\%. Figure 7: Weighted transfer matrix for 12-qubit H2O H_2 O under noiseless and noisy transfer. Transfer reduces steps to achieve a similar accuracy as non-transfer by 88.2%88.2\% and CNOT count by 57.6%57.6\%, under combined depolarizing noise (p1=0.001p_1=0.001, p2=0.005p_2=0.005). Across all systems and noise settings, buffer transfer consistently reduces steps to chemical accuracy and improves final energy error. At the smallest scale (Figure 5), step reductions of 4747-58%58\% and energy improvements of up to 90.2%90.2\% yield composite scores of 19.219.2-35.8%35.8\%. At 8-qubit scale (Figure 6), step reductions reach 84.8%84.8\% under combined noise. Crucially, as shown in Figure 7, the transfer advantage strengthens with system size: at 12-qubit under combined depolarizing noise, buffer transfer reduces steps by 88.2%88.2\% and achieves the highest composite score of 51.0%51.0\% across all benchmarks, demonstrating that lightweight replay-buffer transfer is not only effective but becomes more beneficial as the quantum system grows. All gains require no architectural changes, no reward relabeling, and no network weight sharing. The noiseless and noisy tasks share identical state and action spaces, so their optimal value functions differ only by a bounded perturbation. Classical transfer analyses Taylor and Stone (2009) show that high-value source trajectories remain informative under modest dynamics shifts; recent Bellman-alignment theory Chai et al. (2026) formalizes this, proving that small one-step Bellman mismatch suffices for provable sample-complexity reduction via source replay, corrected online. 4.4 ReaPER++ generalizes beyond quantum domain To confirm that the annealing mechanism of ReaPER++ is not specific to quantum reward structure, we benchmark all three replay strategies on LunarLander-v3 Towers et al. (2024) using identical DQN agents and the same ω schedule (ωmin=0.1→ωmax=0.7 _ =0.1\!→\! _ =0.7, Tann=5×104T_ann=5×10^4 steps; full details in Appendix E). ReaPER++ achieves a 9%9\% gain in normalized cumulative-return AUC over both baselines and reaches first solve in 9.3%9.3\% fewer environment steps than fixed ReaPER (3.8×1053.8× 10^5 vs. 42×10542× 10^5), while sustaining a higher asymptotic success rate (60%60\% vs. 5050-55%55\% at episode 45004500), confirming that the PER → ReaPER annealing principle generalizes across reward regimes. 5 Conclusion We introduced a replay-buffer engineering framework for quantum circuit optimization built on three complementary components: ReaPER++, which improves sample efficiency and circuit compactness through annealed PER → ReaPER prioritization, OptCRLQAS, which cuts wall-clock time per episode by up to 67.5%67.5\% by amortizing expensive quantum-classical evaluations and lightweight buffer transfer, which reduces steps to chemical accuracy by up to 88%88\% and improves final energy error by up to 90%90\% over from-scratch noisy baselines, without network-weight sharing or reward relabeling. Across quantum compiling and QAS benchmarks, these results show that how experience is stored, sampled, and transferred is a primary algorithmic lever for scalable, noise-robust quantum circuit optimization. A classical validation on LunarLander-v3 further confirms that ReaPER++’s annealing principle is domain-agnostic, yielding a +9%+9\% AUC improvement over PER and fixed ReaPER with identical agents and schedules as shown in Appendix E. References [1] A. Abbas, A. Ambainis, B. Augustino, A. Bärtschi, H. Buhrman, C. Coffrin, G. Cortiana, V. Dunjko, D. J. Egger, B. G. Elmegreen, et al. (2024) Challenges and opportunities in quantum optimization. Nature Reviews Physics 6 (12), p. 718–735. Cited by: §1, §2. [2] M. Andrychowicz, F. Wolski, A. Ray, J. Schneider, R. Fong, P. Welinder, B. McGrew, J. Tobin, O. Pieter Abbeel, and W. Zaremba (2017) Hindsight experience replay. Advances in neural information processing systems 30. Cited by: §B.1, §B.1, §2, Figure 2, §3, Table 1. [3] K. Bharti, A. Cervera-Lierta, T. H. Kyaw, T. Haug, S. Alperin-Lea, A. Anand, M. Degroote, H. Heimonen, J. S. Kottmann, T. Menke, et al. (2022) Noisy intermediate-scale quantum algorithms. Reviews of Modern Physics 94 (1), p. 015004. Cited by: §1. [4] M. Bukov and F. Marquardt (2026) Reinforcement learning for quantum technology. arXiv preprint arXiv:2601.18953. Cited by: §1. [5] M. Cerezo, A. Arrasmith, R. Babbush, S. C. Benjamin, S. Endo, K. Fujii, J. R. McClean, K. Mitarai, X. Yuan, L. Cincio, et al. (2021) Variational quantum algorithms. Nature Reviews Physics 3 (9), p. 625–644. Cited by: §B.2.2, §1. [6] J. Chai, E. Zhang, E. Chen, and Y. Yan (2026) Optimistic transfer under task shift via bellman alignment. arXiv preprint arXiv:2601.21924. Cited by: Appendix D, §4.3. [7] X. Croot, K. Nowrouzi, C. Spitzer, C. G. Almudever, A. Blais, M. Carroll, J. Chow, D. Friedman, M. Tokunari, E. Charbon, et al. (2025) Enabling technologies for scalable superconducting quantum computing. arXiv preprint arXiv:2512.15001. Cited by: §1. [8] M. H. Devoret and R. J. Schoelkopf (2013) Superconducting circuits for quantum information: an outlook. Science 339 (6124), p. 1169–1174. Cited by: §1. [9] Y. Du, T. Huang, S. You, M. Hsieh, and D. Tao (2022) Quantum circuit architecture search for variational quantum algorithms. npj Quantum Information 8 (1), p. 62. Cited by: §2. [10] E. Farhi, J. Goldstone, and S. Gutmann (2014) A quantum approximate optimization algorithm. arXiv preprint arXiv:1411.4028. Cited by: §1. [11] T. V. Forster, N. Quetschlich, and R. Wille (2025) Quantum circuit optimization for the fault-tolerance era: do we have to start from scratch?. In 2025 IEEE International Conference on Quantum Computing and Engineering (QCE), Vol. 1, p. 584–590. Cited by: §1. [12] T. Fösel, M. Y. Niu, F. Marquardt, and L. Li (2021) Quantum circuit optimization with deep reinforcement learning. arXiv preprint arXiv:2103.07585. Cited by: §B.2, §2. [13] A. G. Fowler, M. Mariantoni, J. M. Martinis, and A. N. Cleland (2012) Surface codes: towards practical large-scale quantum computation. Physical Review A—Atomic, Molecular, and Optical Physics 86 (3), p. 032324. Cited by: §1. [14] H. R. Grimsley, S. E. Economou, E. Barnes, and N. J. Mayhall (2019) An adaptive variational algorithm for exact molecular simulations on a quantum computer. Nature communications 10 (1), p. 3007. Cited by: §2. [15] S. Guo, L. Zou, H. Chen, B. Qu, H. Chi, P. S. Yu, and Y. Chang (2024) Sample efficient offline-to-online reinforcement learning. IEEE Transactions on Knowledge and Data Engineering 36 (3), p. 1299–1310. External Links: Document Cited by: §1, §2. [16] A. W. Harrow, B. Recht, and I. L. Chuang (2002) Efficient discrete approximations of quantum gates. Journal of Mathematical Physics 43 (9), p. 4445–4451. Cited by: §B.2.1, Appendix F, §4.1. [17] Z. He, M. Deng, S. Zheng, L. Li, and H. Situ (2024) Training-free quantum architecture search. In Proceedings of the AAAI conference on artificial intelligence, Vol. 38, p. 12430–12438. Cited by: §4.2, Table 2, Table 2. [18] Z. He, J. Wei, C. Chen, Z. Huang, H. Situ, and L. Li (2024) Gradient-based optimization for quantum architecture search. Neural Networks 179, p. 106508. Cited by: §4.2, Table 2. [19] T. Hester, M. Vecerik, O. Pietquin, M. Lanctot, T. Schaul, B. Piot, D. Horgan, J. Quan, A. Sendonaris, I. Osband, et al. (2018) Deep q-learning from demonstrations. In Proceedings of the AAAI conference on artificial intelligence, Vol. 32. Cited by: §1. [20] A. Ikhtiarudin, A. Das, P. Thakkar, and A. Kundu (2025) BenchRL-qas: benchmarking reinforcement learning algorithms for quantum architecture search. In Proceedings of the AAAI Symposium Series, Vol. 7, p. 358–367. Cited by: §4.3. [21] A. Kandala, A. Mezzacapo, K. Temme, M. Takita, M. Brink, J. M. Chow, and J. M. Gambetta (2017) Hardware-efficient variational quantum eigensolver for small molecules and quantum magnets. nature 549 (7671), p. 242–246. Cited by: §1. [22] Y. Kim, A. Eddins, S. Anand, K. X. Wei, E. Van Den Berg, S. Rosenblatt, H. Nayfeh, Y. Wu, M. Zaletel, K. Temme, et al. (2023) Evidence for the utility of quantum computing before fault tolerance. Nature 618 (7965), p. 500–505. Cited by: §1. [23] M. Kjaergaard, M. E. Schwartz, J. Braumüller, P. Krantz, J. I. Wang, S. Gustavsson, and W. D. Oliver (2020) Superconducting qubits: current state of play. Annual Review of Condensed Matter Physics 11 (1), p. 369–395. Cited by: §1. [24] D. Kremer, V. Villar, H. Paik, I. Duran, I. Faro, and J. Cruz-Benito (2024) Practical and efficient quantum circuit synthesis and transpiling with reinforcement learning. arXiv preprint arXiv:2405.13196. Cited by: §B.2, §2. [25] A. Kundu and S. Mangini (2025) TensorRL-qas: reinforcement learning with tensor networks for improved quantum architecture search. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, Cited by: Appendix A, §1, §2. [26] A. Kundu, A. Sarkar, and A. Sadhu (2024) Kanqas: kolmogorov-arnold network for quantum architecture search. EPJ Quantum Technology 11 (1), p. 76. Cited by: §2. [27] A. Kundu and L. Sarra (2026) Reinforcement learning with learned gadgets to tackle hard quantum problems on real hardware. Communications Physics. Cited by: §2. [28] A. Kundu (2024) Reinforcement learning-assisted quantum architecture search for variational quantum algorithms. arXiv preprint arXiv:2402.13754. Cited by: §1. [29] awesome-QAS: A curated list of resources for quantum architecture search External Links: Link Cited by: §2. [30] E. Kuo, Y. L. Fang, and S. Y. Chen (2021) Quantum architecture search via deep reinforcement learning. arXiv preprint arXiv:2104.07715. Cited by: §2. [31] T. LeBlond, C. Dean, G. Watkins, and R. Bennink (2024) Realistic cost to execute practical quantum circuits using direct clifford+ t lattice surgery compilation. ACM Transactions on Quantum Computing 5 (4), p. 1–28. Cited by: §1. [32] S. Lee, Y. Seo, K. Lee, P. Abbeel, and J. Shin (2021) Offline-to-online reinforcement learning via balanced replay and pessimistic q-ensemble. In 5th Annual Conference on Robot Learning, External Links: Link Cited by: §1, §2, §3. [33] X. Lu, K. Pan, G. Yan, J. Shan, W. Wu, and J. Yan (2023) Qas-bench: rethinking quantum architecture search and a benchmark. In International conference on machine learning, p. 22880–22898. Cited by: §4.2, Table 2, Table 2. [34] V. Mnih, K. Kavukcuoglu, D. Silver, A. Graves, I. Antonoglou, D. Wierstra, and M. Riedmiller (2013) Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602. Cited by: §B.1, item 1, §2. [35] L. Moro, M. G. Paris, M. Restelli, and E. Prati (2021) Quantum compiling by deep reinforcement learning. Communications Physics 4 (1), p. 178. Cited by: §B.1, §B.2.1, §B.2.3, Table 4, Table 4, Table 4, Appendix F, Appendix F, §1, §2, §4.1, §4.1, Table 1, Table 1. [36] K. Nicoli, C. J. Anders, L. Funcke, T. Hartung, K. Jansen, S. Kühn, K. Müller, P. Stornati, P. Kessel, and S. Nakajima (2023) Physics-informed bayesian optimization of variational quantum circuits. Advances in Neural Information Processing Systems 36, p. 18341–18376. Cited by: §2. [37] I. Osband, C. Blundell, A. Pritzel, and B. Van Roy (2016) Deep exploration via bootstrapped dqn. Advances in neural information processing systems 29. Cited by: Table 8. [38] M. Ostaszewski, L. M. Trenkwalder, W. Masarczyk, E. Scerri, and V. Dunjko (2021) Reinforcement learning for optimization of variational quantum circuit architectures. Advances in neural information processing systems 34, p. 18182–18194. Cited by: §B.2.3, §B.2, §1, §2, §4.2. [39] Y. J. Patel, A. Kundu, M. Ostaszewski, X. Bonet-Monroig, V. Dunjko, and O. Danaci (2024) Curriculum reinforcement learning for quantum architecture search under hardware errors. In The Twelfth International Conference on Learning Representations, External Links: Link Cited by: Appendix K, §B.2.2, §B.2.3, §B.2, Appendix H, Table 5, item 2, §1, §2, §2, Figure 4, Figure 4, §4.2. [40] L. S. Pleiss, T. Sutter, and M. Schiffer (2026) Reliability-adjusted prioritized experience replay. In The Fourteenth International Conference on Learning Representations, External Links: Link Cited by: §B.1, §B.1, 2nd item, Table 4, item 1, §2, Figure 2, §3, Table 1, Proposition 1. [41] M. J. Powell (1994) A direct search optimization method that models the objective and constraint functions by linear interpolation. In Advances in optimization and numerical analysis, p. 51–67. Cited by: Table 8, Appendix K. [42] J. Preskill (2018) Quantum computing in the nisq era and beyond. Quantum 2, p. 79. Cited by: §1. [43] R. Rasconi and A. Oddi (2019) An innovative genetic algorithm for the quantum circuit compilation problem. In Proceedings of the AAAI conference on artificial intelligence, Vol. 33, p. 7707–7714. Cited by: §2. [44] K. Reuer, J. Landgraf, T. Fösel, J. O’Sullivan, L. Beltrán, A. Akin, G. J. Norris, A. Remm, M. Kerschbaum, J. Besse, et al. (2023) Realizing a deep reinforcement learning agent for real-time quantum feedback. Nature Communications 14 (1), p. 7138. Cited by: §1. [45] F. J. Ruiz, T. Laakkonen, J. Bausch, M. Balog, M. Barekatain, F. J. Heras, A. Novikov, N. Fitzpatrick, B. Romera-Paredes, J. Van De Wetering, et al. (2025) Quantum circuit optimization with alphatensor. Nature Machine Intelligence 7 (3), p. 374–385. Cited by: §B.2. [46] T. Schaul, J. Quan, I. Antonoglou, and D. Silver (2015) Prioritized experience replay. arXiv preprint arXiv:1511.05952. Cited by: §B.1, §B.1, Appendix C, 1st item, Table 4, item 1, §2, Figure 2, §3, Table 1, Proposition 1. [47] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov (2017) Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347. Cited by: Figure 2. [48] C. Song, J. Lee, and J. Park (2025) Adaptive replay buffer for offline-to-online reinforcement learning. External Links: 2512.10510, Link Cited by: §1, §2. [49] L. Sünkel, D. Martyniuk, D. Mattern, J. Jung, and A. Paschke (2023) Ga4qco: genetic algorithm for quantum circuit optimization. arXiv preprint arXiv:2302.01303. Cited by: §2. [50] R. S. Sutton, A. G. Barto, et al. (1998) Reinforcement learning: an introduction. Vol. 1, MIT press Cambridge. Cited by: §1. [51] R. Sweke, M. S. Kesselring, E. P. van Nieuwenburg, and J. Eisert (2021) Reinforcement learning decoders for fault-tolerant quantum computation. Machine Learning: Science and Technology 2 (2), p. 025005. Cited by: §1. [52] M. E. Taylor and P. Stone (2009) Transfer learning for reinforcement learning domains: a survey.. Journal of Machine Learning Research 10 (7). Cited by: §4.3. [53] M. Towers, A. Kwiatkowski, J. Terry, J. U. Balis, G. De Cola, T. Deleu, M. Goulão, A. Kallinteris, M. Krimmel, A. KG, et al. (2024) Gymnasium: a standard interface for reinforcement learning environments. arXiv preprint arXiv:2407.17032. Cited by: Appendix E, §4.4. [54] H. Van Hasselt, A. Guez, and D. Silver (2016) Deep reinforcement learning with double q-learning. In Proceedings of the AAAI conference on artificial intelligence, Vol. 30. Cited by: item 1. [55] P. Wang, M. Usman, U. Parampalli, L. C. Hollenberg, and C. R. Myers (2023) Automated quantum circuit design with nested monte carlo tree search. IEEE Transactions on Quantum Engineering 4, p. 1–20. Cited by: §2. [56] R. Wang, K. Frans, P. Abbeel, S. Levine, and A. A. Efros (2025) Prioritized generative replay. In The Thirteenth International Conference on Learning Representations, External Links: Link Cited by: Appendix A. [57] Z. Wang, Q. Chen, Y. Du, Z. Yang, X. Cai, K. Huang, J. Zhang, K. Xu, J. Du, Y. Li, et al. (2024) Quantum compiling with reinforcement learning on a superconducting processor. arXiv preprint arXiv:2406.12195. Cited by: §1. [58] W. Wu, G. Yan, X. Lu, K. Pan, and J. Yan (2023) Quantumdarts: differentiable quantum architecture search for variational quantum algorithms. In International conference on machine learning, p. 37745–37764. Cited by: §2, §4.2, Table 2. [59] H. You, T. Yang, Y. Zheng, J. Hao, E. Taylor, et al. (2022) Cross-domain adaptive transfer reinforcement learning based on state-action correspondence. In Uncertainty in Artificial Intelligence, p. 2299–2309. Cited by: §3. [60] R. Zen, J. Olle, L. Colmenarez, M. Puviani, M. Müller, and F. Marquardt (2025) Quantum circuit discovery for fault-tolerant logical state preparation with reinforcement learning. Physical Review X 15 (4), p. 041012. Cited by: §1, §2. [61] S. Zhang, C. Hsieh, S. Zhang, and H. Yao (2021) Neural predictor based quantum architecture search. Machine Learning: Science and Technology 2 (4), p. 045027. Cited by: §2. [62] S. Zhang, C. Hsieh, S. Zhang, and H. Yao (2022) Differentiable quantum architecture search. Quantum Science & Technology 7 (4), p. 045023. Cited by: §2, §4.2, Table 2. [63] Z. Zhou, A. Peng, Q. Li, S. Levine, and A. Kumar (2025) Efficient online reinforcement learning fine-tuning need not retain offline data. In The Thirteenth International Conference on Learning Representations, External Links: Link Cited by: §1, §2, §3. [64] Z. Zimborás, B. Koczor, Z. Holmes, E. Borrelli, A. Gilyén, H. Huang, Z. Cai, A. Acín, L. Aolita, L. Banchi, et al. (2025) Myths around quantum computation before full fault tolerance: what no-go theorems rule out and what they don’t. arXiv preprint arXiv:2501.05694. Cited by: §1. Appendix A Limitations and future work Limitations. All quantum experiments use a fixed DQN/DDQN backbone; whether ReaPER++’s annealing advantage persists under policy-gradient or actor-critic agents remains an open question. The replay-buffer transfer scheme assumes shared state and action spaces between source and target tasks, which may not hold when porting across substantially different hardware topologies or gate sets. Future work. Immediate extensions include: (i) combining OptCRLQAS with tensor-network warm-starts [25] to further reduce search cost at 2020-qubit scale; (i) hardware-aware replay, where buffered circuits are ranked using limited QPU evaluations before transfer to realistic noise models; and (i) prioritized generative replay [56] to synthesize high-value transitions for richer buffer initialization under depolarizing noise. Appendix B Preliminaries Here we discuss the preliminaries based on which the methods (in Section 3) are based on. This discussion includes a through description of the state-of-the-art replay buffer. Also t includes the encoding methods through which we represent the action space, reward functions and the RL-state. B.1 Replay buffers in deep reinforcement learning Experience replay is a central component of off-policy deep RL because it determines which past transitions are revisited during optimization. Uniform replay [34], hindsight experience replay [2], prioritized experience replay [46], and more recent reliability-aware variants [40] provide increasingly structured ways to trade off coverage, informativeness, and target reliability. In quantum optimization, however, replay buffers have largely been treated as fixed implementation choices rather than as a primary object of algorithmic design. To formalize the replay mechanisms studied in this work, we consider an episodic reinforcement-learning setting. An episode is a sequence D=Ctt=1nD=\C_t\_t=1^n of transitions, where each transition Ct=(St,At,Rt,St+1,dt)C_t=(S_t,A_t,R_t,S_t+1,d_t) records the state St∈S_t , action At∈A_t , scalar reward RtR_t, next state St+1S_t+1, and terminal flag dt∈0,1d_t∈\0,1\, which equals 11 if the episode ends at step t and 0 otherwise. The agent maintains an online Q-network QθQ_θ with parameters θ and a periodically updated target network Qθ¯Q_ θ with parameters θ¯ θ, and γ∈(0,1]γ∈(0,1] denotes the discount factor. For each stored transition, we define four quantities that replay strategies use to assign sampling priorities. The TD target YtY_t is the bootstrapped return estimate used to update QθQ_θ: Yt=Rt+γ(1−dt)maxa′Qθ¯(St+1,a′).Y_t=R_t+γ(1-d_t) _a Q_ θ(S_t+1,a ). (5) The TD error δt _t measures how far the current value estimate deviates from this target: δt=Yt−Qθ(St,At), _t=Y_t-Q_θ(S_t,A_t), (6) and its magnitude δt+=|δt| _t^+=| _t| serves as a proxy for how much a transition still has to teach the agent. The true value error ete_t measures the deviation of the current estimate from the optimal Q-function Q⋆Q : et=Qθ(St,At)−Q⋆(St,At).e_t=Q_θ(S_t,A_t)-Q (S_t,A_t). (7) Finally, the target bias εt _t captures the error introduced by the target network itself: εt=Qθ¯(St,At)−Q⋆(St,At). _t=Q_ θ(S_t,A_t)-Q (S_t,A_t). (8) Replay strategies differ in how they assign sampling preference to stored transitions. For each transition CiC_i in the replay buffer, let Ψi _i denote its unnormalized replay priority and μi=Ψi/∑j=1NΨj _i= _i/ _j=1^N _j the corresponding sampling probability. Different replay rules are then specified by different choices of Ψi _i. Uniform replay (Vanilla). A uniform replay buffer samples all stored transitions with equal probability. If the current buffer contains Cii=1N\C_i\_i=1^N, then Ψiuni=1,μiuni=1N. _i^uni=1, _i^uni= 1N. (9) Hindsight experience replay (HER). HER augments the replay buffer by relabeling goals from achieved future states within the same trajectory [2]. The key idea is that even a failed trajectory, one that did not reach the intended goal g, can be treated as a successful one with respect to a different goal g′g , typically the state actually reached. Concretely, given a goal-conditioned transition (St,At,Rt,St+1,dt,g)(S_t,A_t,R_t,S_t+1,d_t,g), HER adds an additional relabeled copy to the buffer, (St,At,Rt′(g′),St+1,dt′(g′),g′), (S_t,A_t,R_t (g ),S_t+1,d_t (g ),g ), (10) where g′g is the relabeled goal and Rt′(g′)R_t (g ), dt′(g′)d_t (g ) are the reward and terminal flag recomputed with respect to g′g . Unless stated otherwise, HER samples uniformly from the augmented buffer. HER is utilized in the quantum compilation task in Ref. [35]. Prioritized experience replay (PER). PER samples transitions according to their absolute TD error [46]: ΨiPER=(δi+)α,μiPER=ΨiPER∑j=1NΨjPER, _i^PER=( _i^+)^α, _i^PER= _i^PER _j=1^N _j^PER, (11) where α>0α>0 controls the strength of prioritization. Reliability-adjusted prioritized experience replay (ReaPER). ReaPER was recently introduced in ref. [40]. It discounts transitions whose targets are unreliable due to large downstream TD errors. For an episode D=Ctt=1nD=\C_t\_t=1^n, the reliability score is Rt=1−∑i=t+1nδi+∑i=1nδi+,R_t=1- _i=t+1^n _i^+ _i=1^n _i^+, (12) and the corresponding priorities are ΨtReaPER=Rtω(δt+)α,μtReaPER=ΨtReaPER∑i=1nΨiReaPER, _t^ReaPER=R_t^ω( _t^+)^α, _t^ReaPER= _t^ReaPER _i=1^n _i^ReaPER, (13) where ω∈[0,1]ω∈[0,1] controls the strength of reliability weighting (ω=0ω=0 recovers PER and ω=1ω=1 corresponds to fully reliability-adjusted replay). Thus, relative to PER, high-error and low-reliability transitions are down-weighted. B.2 Reinforcement learning for quantum optimization Reinforcement learning has been applied to both quantum compilation and parameterized quantum circuit design. Existing work includes RL-based gate-sequence synthesis [45, 12, 24], architecture search for molecular ground-state preparation [38], and hardware-aware variational circuit construction [39], demonstrating that RL can discover nontrivial circuit structures across a range of quantum tasks. At the same time, prior methods typically operate under substantial sample and compute demands, especially when each environment step requires an expensive variational optimization or when the target setting includes hardware noise. To tackle this, we formulate both quantum compilation and quantum architecture search as Markov decision processes in which an agent sequentially constructs a quantum circuit. At time step t, the agent observes St∈S_t , selects At∈A_t , receives reward RtR_t, and transitions to St+1S_t+1. Episodes terminate when a target threshold is reached or when a maximum circuit length is exceeded. B.2.1 RL-state and action for quantum compilation In quantum compilation, the agent aims to approximate a target unitary UtarU_tar by appending gates from a finite base G. Following Ref. [35], the system is initialized in the all-zero state |0⟩⊗n 0 n, and the circuit unitary after t steps is Ut=∏j=1tAjU_t= _j=1^tA_j, so that the prepared state at step t is Ut|0⟩⊗nU_t 0 n. The observation is based on the residual unitary Ot=Ut†Utar,O_t=U_t U_tar, (14) whose real and imaginary entries are provided to the agent. 1-qubit compiling For 1-qubit compiling, the chosen gateset base is rot(1)=Ri(θj)|i∈x,y,z,θj∈+π128,−π128,G^(1)_rot= \R_i\! ( _j )\; |\;i∈\x,y,z\,\; _j∈ \+ π128,- π128 \ \, (15) where Ri(θj)R_i( _j) denotes a single-qubit rotation by angle θj _j about axis i. The rotation angle is thus restricted to a binary set of small fixed increments ±π/128±π/128, making each action a minimal discrete rotation rather than a continuous parameter choice. In an alternate setting, we also consider the Harrow-Recht-Chuang (HRC) [16] efficient universal basis, V1=15(12i2i1)V_1= 1 5 pmatrix1&2i\\ 2i&1 pmatrix, V2=15(12−21)V_2= 1 5 pmatrix1&2\\ -2&1 pmatrix, V3=15(1+2i001−2i)V_3= 1 5 pmatrix1+2i&0\\ 0&1-2i pmatrix, with action space HRC(1)=V1,V2,V3.A^(1)_HRC=\V_1,V_2,V_3\. (16) 2-qubit compiling. For the 2-qubit setting, we use the action space: (2)=Rz(θj)⊗,⊗Rz(θj),X=X(±π128),Y=Y(±π128),A^(2)=\R_z( _j) ,I R_z( _j),X=X\! (± π128 ),Y=Y\! (± π128 )\, (17) B.2.2 RL-state and action for quantum architecture search For quantum architecture search (QAS), the agent searches over parameterized quantum circuits to minimize a task-dependent cost, typically the expectation value [5]. Following CRLQAS [39], the state is represented by a tensor-based binary encoding of the circuit: QAS=(t,ct)|t∈0,1Ndepth×Nwires×Ngate_type,ct∈ℝ,S_QAS= \(E_t,c_t)\; |\;E_t∈\0,1\^N_depth× N_wires× N_gate\_type,\;c_t \, (18) where tE_t encodes the circuit structure and ctc_t summarizes its current performance by calculating the current cost function i.e. the expectation value of Hamiltonian. Using the gate set QAS=RX(θx),RY(θy),RZ(θz),CNOTG_QAS=\ RX( _x), RY( _y), RZ( _z), CNOT\, the action space consists of placing a gate at a particular circuit location. A compact description is QAS=a=(ℓ,i,g)|ℓ∈1,…,Nlayers,i∈1,…,Nwireskg,g∈QAS,A_QAS= \a=( ,i,g)\; |\; ∈\1,…,N_layers\,\;i∈\1,…,N_wires\^k_g,\;g _QAS \, (19) where ℓ is the layer index, i specifies the qubit(s) the gate acts on, and kgk_g is the number of qubits required by gate g which isequal to 11 for single-qubit gates RX,RY,RZ\ RX, RY, RZ\ and 22 for the two-qubit gate CNOT. Illegal actions are masked out during training. B.2.3 Reward structure In both tasks, an episode ends either when a target threshold is met or when the maximum number of steps is reached. quantum compilation. Following Ref. [35], we use two reward designs depending on the gate base. For bases small-rotation (i.e ±π/128±π/128), we use the dense reward rtcomp=(L−t)+1,if d(Ut,Utar)<ε,−d(Ut,Utar)/L,otherwise,r_t^comp= cases(L-t)+1,&if d(U_t,U_tar)< ,\\[4.0pt] -d(U_t,U_tar)/L,&otherwise, cases (20) where L is the maximum episode length and ε is the target accuracy threshold. For discrete bases such as HRC, we use the sparse reward rtcomp=0,if d(Ut,Utar)<ε,−1/L,otherwise.r_t^comp= cases0,&if d(U_t,U_tar)< ,\\[4.0pt] -1/L,&otherwise. cases (21) Quantum architecture search. Here we use the reward used as in [38, 39]. Let CtC_t be the optimized variational cost at step t, ξ the target threshold, CminC_ the desired minimum cost, and TseT_s^e the maximum number of steps in episode e. Then RtQAS=5,if Ct<ξ,−5,if t≥Tse and Ct≥ξ,max(Ct−1−CtCt−1−Cmin,−1),otherwise.R_t^QAS= cases5,&if C_t<ξ,\\[4.0pt] -5,&if t≥ T_s^e and C_t≥ξ,\\[6.0pt] \! ( C_t-1-C_tC_t-1-C_ ,\,-1 ),&otherwise. cases (22) This reward provides a positive terminal signal for success, a negative terminal signal for failure, and otherwise a normalized improvement-based intermediate reward. Appendix C Justification for ReaPER+ Here we provide an argument on why the proposed ReaPER++ replay rule can be expected to outperform a fixed replay strategy over the full course of training. Recall that ReaPER++ defines the replay priority of transition t at training step τ as Ψt(+,τ)=Rtωτ(δt+)α,μt(+,τ)=Ψt(+,τ)∑iΨi(+,τ), _t^(+,τ)=R_t _τ( _t^+)^α, _t^(+,τ)= _t^(+,τ) _i _i^(+,τ), (23) where δt+=|δt| _t^+=| _t| is the absolute temporal-difference error, Rt∈[0,1]R_t∈[0,1] is the reliability score, α>0α>0 is the prioritization exponent, and ωτ∈[0,1] _τ∈[0,1] is a non-decreasing annealing parameter. When ωτ=0 _τ=0, the scheme reduces to PER, whereas ωτ=1 _τ=1 recovers ReaPER. Proposition 1. Assume that: (i) larger reliability RtR_t corresponds to smaller bias in the TD target of transition t; (i) early in training, reliability estimates are noisy and only weakly correlated with true target quality; and (i) later in training, reliability estimates become more informative. Then a schedule ωτ _τ that starts near 0 and increases toward 11 induces a replay distribution that is better aligned with the needs of the learning process than either fixed PER [46] or ReaPER [40] used throughout training. Justification. Consider two transitions i and j. Their relative sampling probability under ReaPER++ is μi(ω)μj(ω)=Riω(δi+)αRjω(δj+)α=(RiRj)ω(δi+δj+)α. _i^(ω) _j^(ω)= R_i^ω( _i^+)^αR_j^ω( _j^+)^α= ( R_iR_j )^ω ( _i^+ _j^+ )^α. (24) If Ri>RjR_i>R_j, then for any ω2>ω1 _2> _1 we have μi(ω2)μj(ω2)>μi(ω1)μj(ω1). _i^( _2) _j^( _2)> _i^( _1) _j^( _1). (25) Hence increasing ω monotonically shifts replay mass toward transitions with higher reliability while preserving the PER-style dependence on TD error. Figure 8: ReaPER+ progressively concentrates buffer mass toward higher-fidelity transitions (fidelity ≥0.95≥ 0.95) while retaining broader early-training coverage, consistent with its annealed transition from PER-like exploration to ReaPER-like reliability-aware sampling. PER maintains broader low-fidelity coverage throughout training, while ReaPER shows intermediate concentration behavior. This monotonicity explains the role of the annealing schedule. At the beginning of training, the value function is inaccurate and the resulting reliability estimates can be unstable. In this regime, setting ωτ≈0 _τ≈ 0 prevents the learner from overcommitting to a noisy reliability signal and recovers the PER [46] behavior that emphasizes large TD-error transitions. Later in training, once the critic becomes more accurate, the reliability score carries more information about whether a large TD error corresponds to a genuinely useful correction or to a misleading target. Increasing ωτ _τ then down-weights high-error but low-reliability transitions and shifts the replay distribution toward more trustworthy updates. Equivalently, one may examine the log-priority logΨt(+,τ)=ωτlogRt+αlogδt+. _t^(+,τ)= _τ R_t+α _t^+. (26) The annealing coefficient ωτ _τ controls the contribution of logRt R_t relative to logδt+ _t^+. Early in training, the replay rule is dominated by the TD-error term; later, the reliability term gradually regularizes prioritization by suppressing transitions whose downstream targets are less trustworthy. Therefore, ReaPER+ inherits the strong early-learning behavior of PER and the late-stage stability of ReaPER. The argument above does not claim universal dominance for every environment or every annealing schedule. Rather, it shows that under mild assumptions on the quality of reliability estimates across training, ReaPER+ provides a natural continuation method between two useful replay regimes. As shown in Figure 8, during the 1-qubit compiling task, ReaPER++ progressively shifts the fidelity distribution toward higher-fidelity, more reliable replay samples significantly faster than both PER and ReaPER. Appendix D Replay buffer transfer under source-target task similarity We next justify why replay-buffer transfer from a noiseless source environment to a corresponding noisy target environment can accelerate learning in quantum optimization. Let ℳsrcM_src and ℳtgtM_tgt denote two Markov decision processes with the same state and action spaces, where the target task differs from the source task by noise in the transition dynamics, reward evaluations, or both. Let ℬsrcB_src be a replay buffer collected in the source task, and let the target replay buffer be initialized as ℬtgt(0)←(ℬsrc),B_tgt^(0) (B_src), (27) where T is a transfer operator that copies compatible transitions from source to target. Proposition 2. Assume that: (i) the source and target tasks share the same state and action spaces; (i) the target task is a bounded perturbation of the source task in the sense that rewards and transition kernels do not differ arbitrarily; and (i) high-value trajectories in the source task remain informative, though not necessarily optimal, in the target task. Then initializing the target replay buffer with transferred source transitions can improve early target learning relative to training from an empty buffer. Justification. Let the source and target Bellman targets for a transition (s,a)(s,a) be Ysrc(s,a)=rsrc(s,a)+γs′∼Psrc(⋅|s,a)[maxa′Q(s′,a′)],Y^src(s,a)=r^src(s,a)+ _s P^src(·|s,a) [ _a Q(s ,a ) ], (28) and Ytgt(s,a)=rtgt(s,a)+γs′∼Ptgt(⋅|s,a)[maxa′Q(s′,a′)].Y^tgt(s,a)=r^tgt(s,a)+ _s P^tgt(·|s,a) [ _a Q(s ,a ) ]. (29) If the source and target tasks are sufficiently similar in rewards and transition dynamics, then the induced Bellman targets differ by a bounded amount, as suggested by classical simulation-lemma arguments and recent Bellman-alignment analyses for transfer reinforcement learning [6]. For example, if |rtgt(s,a)−rsrc(s,a)|≤εr|r^tgt(s,a)-r^src(s,a)|≤ _r (30) and the transition mismatch is bounded so that |Ptgt[maxa′Q(s′,a′)]−Psrc[maxa′Q(s′,a′)]|≤εp, |E_P^tgt\! [ _a Q(s ,a ) ]-E_P^src\! [ _a Q(s ,a ) ] |≤ _p, (31) then |Ytgt(s,a)−Ysrc(s,a)|≤εr+γεp.|Y^tgt(s,a)-Y^src(s,a)|≤ _r+γ _p. (32) Thus, a transition that is informative for the source task remains approximately informative for the target task whenever the source-target shift is moderate. The second in the proposition 2 is improved coverage. At the start of training in the noisy environment, an empty replay buffer contains little information about which gate sequences, partial circuits, or architectural motifs are promising. In contrast, a transferred buffer already contains trajectories concentrated in regions of state-action space that were useful in the source task. Since the noisy and noiseless tasks share the same circuit-building structure, these transitions provide a more informative initial replay distribution than random target experience alone. Consequently, the learner can begin updating from semantically meaningful trajectories before it has independently rediscovered them under noise. Whereas the third mechanism in proposition 2 is self-correction during continued training. As the agent interacts with the target environment, new target-domain transitions are added to the replay buffer and progressively adjust the replay distribution toward the true noisy task. Hence transfer does not lock the agent into the source domain; rather, it supplies a warm start that is gradually refined by on-target experience. In prioritized replay schemes, this correction can be even more effective because target transitions that carry larger TD errors or higher reliability automatically gain greater replay probability. The above argument explains why replay-buffer transfer is especially natural in quantum optimization settings where the noisy and noiseless problems share the same circuit representation, action space, and objective structure, but differ in the stochastic distortion introduced by hardware noise. Under such source-target similarity, transferred source trajectories provide a useful initial memory for the target learner, improving early sample efficiency while still allowing subsequent adaptation to the noisy environment. Appendix E Classical RL validation: LunarLander-v3 The three contributions presented in the main text ReaPER++, OptCRLQAS, and replay-buffer transfer, are motivated by and evaluated on quantum circuit optimization tasks. To verify that the annealing schedule underlying Reaper++ is not specific to the sparse, long-horizon reward structure of quantum environments, we provide a supplementary validation on LunarLander-v3 [53], a well-established continuous-control benchmark from classical deep RL with dense rewards and a standard solved threshold of +200+200 cumulative return. We stress that this experiment is not intended as a primary result; its sole purpose is to confirm that the PER→ annealing idea generalizes beyond the quantum domain. Figure 9: LunarLander-v3 validation of ReaPER++. (Left) rolling success rate (300-episode window). (Middle) ReaPER++ (blue) reaches a higher success rate faster and maintains a higher asymptotic level than fixed ReaPER (red) and PER (green). (Right) normalized cumulative-return AUC. ReaPER++ accumulates +9%+9\% more return over the full training run, confirming improved sample efficiency on a dense-reward classical benchmark. All methods use identical DQN agents, only the replay mechanism differs. We benchmark three replay strategies using identical DQN agents (two hidden layers of 128 units, SELU activations, Table 3): • PER [46]: prioritization by absolute TD error, α=0.6α=0.6, β0=0.4 _0=0.4 annealed to 1.01.0 over 5×1045×10^4 frames. • ReaPER [40]: reliability-adjusted prioritization with fixed ω=0.4ω=0.4, α=0.4α=0.4. • ReaPER++ (ours): annealed ω schedule ωmin=0.1→ωmax=0.7 _ =0.1\!→\! _ =0.7 over Tann=5×104T_ann=5×10^4 steps (identical schedule to the quantum compiling experiments), α=0.4α=0.4. All agents are trained for 5,0005,000 episodes on LunarLander-v3 with γ=0.99γ=0.99, ε -greedy exploration (ε0=1.0→εmin=0.05 _0=1.0\!→\! _ =0.05, decay 0.99950.9995 per episode), learning rate 10−310^-3, batch size 6464, replay capacity 10510^5, and target-network synchronization every 1010 episodes. Four gradient updates are performed per episode after a 2020-episode warm-up. The same agent architecture and training code used for the quantum experiments are employed without modification; only the environment is swapped. Results are reported for 3 different initializations of neural network. Figure 9 summarizes two complementary metrics. Left: the 300300-episode rolling success rate (fraction of episodes exceeding the +200+200 solved threshold). ReaPER++ reaches a higher success rate earlier in training and sustains a higher asymptotic level (≈60%≈60\% at episode 45004500) compared with fixed ReaPER (≈50%≈50\%) and PER (≈55%≈55\%). Right: the normalized area under the reward curve (AUC), computed as the per-episode running mean of the cumulative return, shifted and normalized to [0,1][0,1]. ReaPER++ achieves a +9%+9\% AUC advantage over both baselines by episode 50005000, indicating better sample efficiency throughout training. These results are consistent with the mechanism described in Section 3 and Appendix C: early in training, ω≈0ω≈ 0 recovers PER-style exploration; as value estimates stabilize, rising ω down-weights unreliable high-error transitions and concentrates replay on trustworthy updates. This two-phase behavior is beneficial regardless of whether rewards are sparse and episodic (quantum compilation) or dense and continuous (LunarLander), supporting the view that ReaPER++ is a general-purpose annealing strategy rather than a domain-specific heuristic. Table 3: Hyperparameters for the LunarLander-v3 validation experiment. Shared parameters apply to all three replay strategies; method-specific parameters are listed in the lower block. Parameter Value Shared (all methods) Network architecture 2 × 128 hidden units, SELU Discount γ 0.990.99 Learning rate 10−310^-3 Batch size 6464 Replay capacity 10510^5 ε schedule 1.0→0.051.0→ 0.05, decay 0.99950.9995/episode Gradient updates/ep. 44 (after 2020-episode warm-up) Training episodes 50005000 Solved threshold +200+200 cumulative return PER α 0.60.6 β0 _0 0.40.4, annealed to 1.01.0 over 5×1045×10^4 frames ReaPER α 0.40.4 ω 0.40.4 (fixed) β0 _0 0.40.4, annealed to 1.01.0 over 5×1045×10^4 frames ReaPER++ α 0.40.4 ωmin _ 0.10.1 ωmax _ 0.70.7 TannT_ann 5×1045×10^4 steps β0 _0 0.40.4, annealed to 1.01.0 over 5×1045×10^4 frames Appendix F 1-qubit compiling with HRC gateset For a direct comparison with Ref. [35], we evaluate the discrete HRC basis [16] using the action space and sparse reward defined in Eq. 16 and Eq. 21. Table 4 shows that ReaPER++ reaches 100%100\% success with a mean fidelity of 0.9950.995 and the shortest average circuit length (14.30±7.8914.30± 7.89 gates), achieving this at 1.56×1061.56× 10^6 steps. This is slightly earlier than fixed ReaPER (1.60×1061.60× 10^6 steps), clearly earlier than PER (2.10×1062.10× 10^6 steps), and substantially earlier than our tuned HER baseline (5.50×1065.50× 10^6 steps), while also producing shorter circuits than all three. Table 4: 1-qubit compiling of Haar-random targets using the HRC basis. Success rate, mean fidelity, and circuit length for different replay strategies. All methods except the original HER baseline [35] reach 100% success. ReaPER++ yields the shortest circuits (14.30±7.8914.30± 7.89 gates) and reaches peak performance ∼24% 24\%, ∼26% 26\%, and ∼72% 72\% faster than fixed ReaPER, PER, and tuned HER, respectively. Method Success (%) ↑ Fidelity ↑ Mean length ↓ At step ↓ ReaPER++ (Ours) 100.0100.0 0.9950.995 14.30±7.8914.30± 7.89 1.56×1.56× 10^6 ReaPER [40] 100.0100.0 0.9950.995 20.34±16.1620.34± 16.16 1.60×1061.60× 10^6 PER [46] 100.0100.0 0.9950.995 19.54±14.6119.54± 14.61 2.10×1062.10× 10^6 HER (tuned) 100.0100.0 0.9920.992 28.60±21.5828.60± 21.58 5.50×1065.50× 10^6 HER (Moro et al. [35]) 95.095.0 0.9900.990 3535 – In contrast, the original unoptimized HER implementation from Moro et al. [35] plateaus at 95%95\% accuracy. Note that throughout the remainder of this paper, any reference to the HER baseline indicates our tuned implementation rather than the original formulation of Ref. [35], ensuring all comparisons are made against the strongest possible version of the baseline. Appendix G Target dataset for 2-qubit compiling task For the compiling experiments with the 1-qubit small-rotation basis and the 2-qubit gate set, we generate target unitaries synthetically by sampling random circuits from the corresponding elementary gate library. Concretely, a target unitary is constructed by first sampling a circuit length uniformly at random and then composing gates drawn uniformly from the same basis used by the agent during training. This yields a diverse dataset of reachable targets with varying circuit complexity while ensuring that the target distribution is consistent with the underlying compilation task. For the 2-qubit setting, Algorithm 1 details this procedure for the gate set XX(±π/128),YY(±π/128),Rz(±π/128)⊗,⊗Rz(±π/128)\X(±π/128),\,Y(±π/128),\,R_z(±π/128) ,\,I R_z(±π/128)\; the 1-qubit small-rotation targets are generated analogously from their corresponding 1-qubit rotation basis. Input: Gate set ℬ=XX(±π128),YY(±π128),Rz(±π128)⊗,⊗Rz(±π128)B= \X(± π128),\;Y(± π128),\;R_z(± π128) ,\;I R_z(± π128) \ Output: Target unitary Utar∈SU(4)U_tar (4) Sample circuit length N∼Uniform(6,7,…,104−1)N (\6,7,…,10^4-1\ ); Initialize Utar←4U_tar _4; for k=1k=1 to N do Sample gate Gk∼Uniform(ℬ)G_k (B); Utar←Gk⋅UtarU_tar← G_k· U_tar; end for return UtarU_tar Algorithm 1 2-qubit target unitary generation. Appendix H Benchmarking OptCRLQAS against CRLQAS [39] To empirically validate the computational efficiency gained by amortizing the architecture update overhead, we benchmarked OptCRLQAS against the baseline CRLQAS framework across 66-, 88-, and 1010-qubit configurations. We tracked both the expectation value evaluation and the classical optimization time over multiple episodes. Figure 10 illustrates the substantial reduction in computational cost when utilizing OptCRLQAS across varying step intervals (m∈3,5,7,10m∈\3,5,7,10\). Figure 10: Comparative runtime analysis of the baseline CRLQAS versus OptCRLQAS across 66-, 88-, and 1010-qubit tasks. The left panel shows the quantum energy evaluation time (in seconds), and the right panel displays the classical optimization time (in seconds). Results for OptCRLQAS are reported for different replay buffer sizes (m∈3,5,7,10m∈\3,5,7,10\). Error bars denote the standard deviation measured over 33 to 55 independent episodes. The utilization of OptCRLQAS yields an average runtime reduction of up to 89%89\% for quantum evaluations and 85%85\% for classical optimizations at m=10m=10. Because CRLQAS triggers a full quantum-classical evaluation at every environment step, its runtime scales poorly as circuit complexity increases. By accumulating local architecture edits and restricting full evaluations to every m steps, OptCRLQAS closely matches the expected runtime reduction factor of ≈1/m≈ 1/m. Our empirical results demonstrate an aggressive decline in runtime as m increases. For quantum energy evaluations, OptCRLQAS achieves an average time reduction of 69.2%69.2\% at m=3m=3, scaling up to an impressive 89.3%89.3\% reduction at m=10m=10 across all qubit scales. Similarly, the classical optimization time is heavily mitigated, yielding an average decrease of 48.2%48.2\% at m=3m=3 and up to 84.9%84.9\% at m=10m=10. This benchmarking clearly establishes that accumulating structural modifications before parameter optimization successfully breaks the runtime bottleneck, enabling RL-based quantum architecture search to scale to larger qubit regimes without prohibitive GPU compute requirements. Table 5: Comparison between vanilla CRLQAS and OptCRLQAS (m=10m=10) on 8-qubit H2O over three random seeds. For OptCRLQAS, the total gate count is computed as ROT+CNOTROT+ CNOT. Method Seed Error (Ha) Total gates CNOT ROT 1 1.167×10−31.167× 10^-3 153 129 24 CRLQAS [39] 42 1.171×10−31.171× 10^-3 80 59 21 786 1.167×10−31.167× 10^-3 100 66 34 1 1.167×10−31.167× 10^-3 81 45 36 OptCRLQAS (Ours) 42 1.171×10−31.171× 10^-3 97 23 74 786 1.168×10−31.168× 10^-3 87 31 56 To isolate the effect of amortized evaluation beyond wall-clock speedup, we additionally compare vanilla CRLQAS and OptCRLQAS on 8-H2O H_2 O under matched training conditions, varying only the update schedule. Table 5 shows that the two methods reach essentially identical final energies: the mean error is 1.168636×10−3Ha1.168636× 10^-3\,Ha for CRLQAS and 1.168738×10−3Ha1.168738× 10^-3\,Ha for OptCRLQAS, a difference of only ∼10−7Ha 10^-7\,Ha. Despite this negligible difference in energy, OptCRLQAS produces substantially more hardware-efficient circuits, reducing the mean total gate count from 111.0111.0 to 88.388.3 (20.4%20.4\% fewer gates) and the mean CNOT count from 84.784.7 to 33.033.0 (61.0%61.0\% fewer CNOTs), at the cost of more single-qubit rotations. This is consistent with the intuition behind OptCRLQAS: by evaluating the circuit only after multiple architectural edits, the agent receives a less myopic and more informative reward signal, which can preserve final accuracy while steering learning toward circuits with substantially lower 2-qubit cost. Appendix I ReaPER ω sensitivity and variant selection In the main text, Figure 3 reports gate counts for a single fixed-ω ReaPER variant per molecular system: ω=0.4ω=0.4 for 6-qubit BEH2 BEH_2 and ω=0.6ω=0.6 for 8-qubit H2O H_2 O. This appendix documents the full sweep over ω that motivates these choices. For each system we train agents with ω∈0.2,0.4,0.6ω∈\0.2,0.4,0.6\ (6-qubit) and ω∈0.2,0.4,0.6,0.8ω∈\0.2,0.4,0.6,0.8\ (8-qubit), using three independent random seeds 1,42,786\1,42,786\ per setting. All other hyperparameters are identical to the main-text QAS experiments (Appendix K). We report mean and standard deviation over seeds for CNOT count and single-qubit rotation (ROT) count. Results are given in Table 6. Since two-qubit gates dominate both circuit noise and compilation cost on near-term hardware, we select the ω value that minimizes mean CNOT count subject to the agent achieving chemical accuracy (ΔE≤εchem E≤ _chem) across all seeds, using ROT count as a tiebreaker. Table 6: CNOT and ROT gate counts (mean ± std. dev. over 3 seeds) for ReaPER ω variants. All entries achieve chemical accuracy (ΔE≤εchem=1.6×10−3 E≤ _chem=1.6× 10^-3 Ha). Bold rows indicate the variant selected for the main text. PER, ReaPER++, and Vanilla are included for reference. 6-qubit BEH2 BEH_2 8-qubit H2O H_2 O ReaPER CNOT ROT CNOT ROT ω=0.2ω=0.2 30.7±18.830.7± 18.8 20.7±9.920.7± 9.9 84.3±75.484.3± 75.4 39.3±32.039.3± 32.0 ω=0.4ω=0.4 23.7±9.023.7± 9.0 29.3±21.529.3± 21.5 71.7±46.671.7± 46.6 38.3±14.638.3± 14.6 ω=0.6ω=0.6 30.7±7.430.7± 7.4 30.3±2.130.3± 2.1 51.0±25.151.0± 25.1 24.3±20.624.3± 20.6 ω=0.8ω=0.8 – – 78.7±46.278.7± 46.2 35.0±5.035.0± 5.0 For 6-qubit BEH2 BEH_2, ω=0.4ω=0.4 achieves the lowest mean CNOT count of 23.7±9.023.7± 9.0 gates among all ReaPER variants, compared with 30.7±18.830.7± 18.8 for ω=0.2ω=0.2 and 30.7±7.430.7± 7.4 for ω=0.6ω=0.6. For 8-qubit H2O H_2 O, ω=0.6ω=0.6 achieves the lowest mean CNOT count of 51.0±25.151.0± 25.1 gates–a 41%41\% reduction relative to PER (86.3±37.586.3± 37.5) and 29%29\% relative to ω=0.4ω=0.4 (71.7±46.671.7± 46.6). Across both systems, increasing ω shifts the replay distribution toward reliability-aware sampling more aggressively, which tends to reduce CNOT count at the cost of slightly higher energy variance. The optimal ω is therefore task-dependent: shorter-horizon problems with lower qubit counts benefit from moderate reliability weighting (ω=0.4ω=0.4), while longer-horizon, larger systems favour stronger weighting (ω=0.6ω=0.6) to suppress noisy TD targets in a deeper search space. This task-dependence is precisely what motivates the annealed ReaPER++ construction, which avoids committing to a fixed ω by scheduling the transition adaptively during training (Section 3). where Xi,Yi,ZiX_i,Y_i,Z_i denote the Pauli operators acting on site i and the first sum runs over nearest-neighbour pairs i∈1,…,n−1i∈\1,…,n-1\. The isotropic exchange interaction (equal X, Y, and Z couplings) places the ground state in a highly entangled singlet sector, making it a non-trivial testbed for variational ansatz construction: a circuit that correctly captures the ground state must generate multi-qubit entanglement across the full chain, which rewards replay strategies that retain long-horizon, high-fidelity trajectories. The target energy E0E_0 is obtained by exact diagonalization and the sum of Pauli coefficients are utilized as the fake minium energy. Appendix J Hamiltonians J.1 Chemical Hamiltonian In Table 7 we provide the detailed configuration of molecules utilized throughout the quantum architecture search. Table 7: The geometry and basis of molecules used in this research. The coordinates are in Angstrom units. Molecule Geometry Basis 6-BEH2 BEH_2 H (0,0,-1.33); Be (0,0,0); H (0,0,1.33) STO-3G 8-H2O H_2 O H (-0.02,-0,0); O (0.84,0.45,0); H (1.48,-0.27,0) STO-3G 10-H2O H_2 O H (-0.02,-0,0); O (0.84,0.45,0); H (1.48,-0.27,0) 6-31G 12-H2O H_2 O H (-0.02,-0,0); O (0.84,0.45,0); H (1.48,-0.27,0) 6-31G J.2 Heisenberg model Hamiltonian formulation In addition to molecular chemistry benchmarks, we evaluate OptCRLQAS and replay-buffer design on the n=5n=5 qubit one-dimensional isotropic Heisenberg model with a uniform longitudinal field. The Hamiltonian is defined on a chain of n spin-12 12 particles with open boundary conditions as HHeis=∑i=1n(XiXi+1+YiYi+1+ZiZi+1)+∑i=1nZi,H_Heis= _i=1^n (X_iX_i+1+Y_iY_i+1+Z_iZ_i+1 )+ _i=1^nZ_i, (33) Table 8: Environment, agent, and optimizer configuration for quantum architecture search benchmarks. Parameter 6-qubit 8-qubit 12-qubit Environment Episodes 5000 5000 1000 Qubits 6 8 12 Max layers 70 250 300 Accept error (Ha) 5.5 5.0 5.0 Shift threshold ball 0.001 Shift threshold time 2000 Success switch (Ha) 5.5 5.0 5.0 Success threshold 50 opt step interval (m) 10 10 15 energy interval (m) 10 10 15 Agent (DQN with n-step returns [37]) Network layers [1000]3[1000]^3 [1000]4[1000]^4 [1000]4[1000]^4 Batch size 1000 Replay buffer size 20 000 Learning rate 3×10−43× 10^-4 n-step return 5 6 6 Target net update Every 500 steps γ (discount) 0.005 Dropout 0.0 Exploration (ϵε-greedy) ϵstart _start 1.0 ϵmin _min 0.05 ϵdecay _decay 0.99995 Classical optimizer (COBYLA [41]) Iterations 1000 Appendix K Hyperparameters for quantum architecture search Tables 8 summarize the environment and agent configurations used across the three QAS benchmarks. All experiments share the same agent architecture (multi-step DQN with n-step returns), optimizer, and curriculum type, differing only where the problem scale demands it. Environment. The maximum circuit depth (num_layers) grows with qubit count, from 70 (6-qubit) to 250 (8-qubit) to 300 (12-qubit), to accommodate the deeper circuits required for larger Hamiltonians. The curriculum uses a MovingThreshold introduced in ref. [39] schedule in all cases, with the acceptance error initialized at 5.0-5.5 Ha and tightened by 0.0010.001 Ha every 2000 episodes. OptCRLQAS evaluates the circuit every m architectural edits (opt_step_interval == energy_interval): m=10m=10 for the 6- and 8-qubit problems and m=15m=15 for the 12-qubit problem, reflecting the longer episodes at larger scale. Agent. The DQN agent uses a 3-layer MLP ([1000,1000,1000][1000,1000,1000]) for the 6-qubit task and a 4-layer MLP ([1000,1000,1000,1000][1000,1000,1000,1000]) for the 8- and 12-qubit tasks. The n-step return horizon is n=5n=5 for 6-qubit and n=6n=6 for 8- and 12-qubit. Exploration follows an ϵε-greedy schedule decaying from ϵ=1.0ε=1.0 at rate 0.999950.99995 per step to a floor of ϵmin=0.05 _ =0.05. The replay buffer holds 20 000 transitions in all cases, with a batch size of 1000. Classical optimizer. All three tasks use COBYLA [41] with a local search size of 8. The 6- and 8-qubit tasks run 1000 global iterations per evaluation, as does the 12-qubit task; the reduced number of evaluations per episode at 12-qubit due to the larger amortization interval m=15m=15 keeps the total classical optimization cost manageable. Appendix L State encoding for quantum architecture search In both gateset configurations, the partially constructed ansatz at each step is represented as a three-dimensional state tensor ∈ℝL×R×nS ^L× R× n, where L is the maximum number of circuit layers (moments), n is the number of qubits, and R is the number of row channels encoding gate placement and, optionally, variational angle parameters. A moment-tracking vector ∈ℤn μ ^n records the next available layer index per qubit, ensuring causal gate ordering. At each time step the agent places at most one 2-qubit gate and one 1-qubit rotation, updates μ accordingly, and returns a flattened view of S as the observation. Encoding I (for gateset RX,RY,RZ,CX\ RX, RY, RZ, CX\). This a a complete binary encoding. The action is a four-tuple a=[a0,a1,a2,a3]a=[a_0,\,a_1,\,a_2,\,a_3], where a0a_0 is the CNOT control qubit (a0=na_0=n signals no CNOT), the target is t=(a0+a1)modnt=(a_0+a_1) n, a2a_2 is the rotation qubit (a2=na_2=n signals no rotation), and a3∈1,2,3a_3∈\1,2,3\ encodes the axis (X/Y/Z). Gate placement writes are strictly binary: [ℓ][t][a0] [ ][t][a_0] ←1(CNOT), ← 1 (CNOT), (34) [ℓ][n+a3−1][a2] [ ][n+a_3-1][a_2] ←1(rotation), ← 1 (rotation), (35) where ℓ is the resolved moment from μ. Because the gate set contains only one two-qubit gate type (CX), all non-zero connectivity entries are uniformly 1 and gate identity is unambiguous. Encoding I (for gateset RXX,RYY,RZZ,RX,RY,RZ\ RXX, RYY, RZZ, RX, RY, RZ\). This encoding is integer-valued type. The action is an eight-tuple a=[a0xx,a1xx,a0yy,a1yy,a0zz,a1zz,arot,aaxis]a=[a_0^x,a_1^x,a_0^y,a_1^y,a_0^z,a_1^z,a_rot,a_axis], providing a control-qubit and an offset for each of the three two-qubit gate types plus a single-qubit rotation. Targets are tg=(a0g+a1g)modnt_g=(a_0^g+a_1^g) n for g∈xx,yy,zzg∈\x,y,z\. Rather than introducing separate binary planes per gate type, distinct integer labels are assigned within a single shared connectivity plane: [ℓ][txx][a0xx] [ ][t_x][a_0^x] ←1, ← 1, (36) [ℓ][tyy][a0yy] [ ][t_y][a_0^y] ←2, ← 2, (37) [ℓ][tzz][a0zz] [ ][t_z][a_0^z] ←3, ← 3, (38) [ℓ][n+aaxis−1][arot] [ ][n+a_axis-1][a_rot] ←1(rotation). ← 1 (rotation). (39) A fully binary encoding for K distinct two-qubit gate types requires K separate n×n× n connectivity planes, giving a connectivity footprint of (Kn2)O(Kn^2). Encoding I consolidates all K types into a single plane via integer labels, reducing this to (n2)O(n^2) regardless of K, which is particularly advantageous as both the gate vocabulary and qubit count grow. The tradeoff is that integer labels introduce an implicit ordinal relationship among gate types that carries no physical meaning; a fully binary scheme provides a strictly one-hot signal per gate type that may be easier for the network to disentangle. A systematic comparison of the two encoding strategies across a controlled set of benchmarks is beyond the scope of the present work and is left as a direction for future investigation. Appendix M Hyperparameters for quantum compilation To isolate the effect of replay design, all quantum compilation experiments use a common off-policy deep Q-learning setup. The network architecture, optimization procedure, exploration schedule, and replay capacity are held fixed across agents, so that performance differences can be attributed to the replay mechanism rather than to changes in model capacity or training protocol. Table 9: Hyperparameters used for 1 and 2-qubit quantum compilation experiments. All agents share a common environment and training configuration; buffer-specific parameters are listed separately in the lower block. Parameter Value Network (all agents) Architecture Hidden layer 2 (each 128 units) Activation ReLU Input size 8 Output size 6 / 3 (small rotations / HRC) Discount factor γ 0.99 Learning rate 3×10−43× 10^-4 Batch size 200 Replay capacity 5×1055× 10^5 Target update freq. 100 episodes Gradient clipping 1.0 ε -start 1.0 ε -min 0.01 ε -decay 0.99931 HER k-future relabeling 5 Strategy future PER α 0.6 β0 _0 0.4 β anneal frames 10510^5 ReaPER α 0.4 ω 0.2 β0 _0 0.4 β anneal frames 10510^5 ReaPER++ α 0.4 ωmin _ 0.1 ωmax _ 0.7 TannT_ann 5×1055× 10^5 β0 _0 0.4 β anneal frames 10510^5 Table 9 lists the hyperparameters used in the 1- and 2-qubit compiling benchmarks. The upper block gives the shared network and training configuration, while the lower blocks specify the additional method-dependent parameters for HER, PER, ReaPER, and ReaPER++. In this way, the comparison remains controlled, and the gains of ReaPER and ReaPER++ can be interpreted as arising from replay design itself.