Paper deep dive

Learning to Modulate, Not to Cycle: Soft Actor---Critic Recovers Inverter-Style Heat-Pump Control

Faizan Ahmed, Aniket Dixit, James Brusey

Year: 2026Venue: arXiv preprintArea: cs.LGType: PreprintEmbeddings: 25

Intelligence

Status: not_run | Model: - | Prompt: - | Confidence: 0%

Entities (0)

No extracted entities yet.

Relation Signals (0)

No relation signals yet.

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:On--off cycling is the main cause of compressor wear in residential heat pumps, yet reinforcement learning (RL) controllers for buildings typically optimise only energy cost and thermal comfort, ignoring how much the learned policy cycles. We add a levelised compressor-wear term to the control reward and study how the resulting behaviour depends on the RL algorithm. Training Soft Actor---Critic (SAC) and Proximal Policy Optimisation (PPO) on an identical Markov decision process for the BOPTEST bestest hydronic heat pump case, we find that SAC learns a continuous modulation policy that keeps the compressor permanently engaged---the operating principle of an inverter-driven heat pump---achieving zero start-ups per day, whereas PPO collapses to bang-bang control that cycles more than the baseline. On the BOPTEST emulator the SAC policy cuts thermal discomfort by up to 90.7% for an 11.5% cost increase, while eliminating all baseline cycling.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

23,385 characters extracted from source content.

Expand or collapse full text

[datatype=bibtex] [fieldset=abstract, null] Faizan Ahmed 11institutetext: Centre for Computational Science and Mathematical Modelling, Coventry University, United Kingdom 11email: ahmedf84@uni.coventry.ac.uk Learning to Modulate, Not to Cycle: Soft Actor–Critic Recovers Inverter-Style Heat-Pump Control Faizan Ahmed Aniket Dixit James Brusey Abstract On–off cycling is the main cause of compressor wear in residential heat pumps, yet reinforcement learning (RL) controllers for buildings typically optimise only energy cost and thermal comfort, ignoring how much the learned policy cycles. We add a levelised compressor-wear term to the control reward and study how the resulting behaviour depends on the RL algorithm. Training Soft Actor–Critic (SAC) and Proximal Policy Optimisation (PPO) on an identical Markov decision process for the BOPTEST bestest_hydronic_heat_pump case, we find that SAC learns a continuous modulation policy that keeps the compressor permanently engaged—the operating principle of an inverter-driven heat pump—achieving zero start-ups per day, whereas PPO collapses to bang–bang control that cycles more than the baseline. On the BOPTEST emulator the SAC policy cuts thermal discomfort by up to 90.7 %90.7\, 37 for an 11.5 %11.5\, 37 cost increase, while eliminating all baseline cycling. keywords: heat pumps, compressor cycling, reinforcement learning, Soft Actor–Critic, BOPTEST 1 Introduction Heat pumps are central to the decarbonisation of residential heating, but their delivered efficiency and service life depend strongly on how they are operated, not only on how much energy they consume. Single-stage units satisfy a heating load by switching the compressor fully on and off; every start incurs a transient during which refrigerant migration, oil return and pressure equalisation depress the coefficient of performance (COP), and the accumulated number of starts is a primary driver of mechanical wear and eventual failure [9, 8, 7]. The standardised part-load degradation coefficient CdC_d used in rating procedures encodes precisely this cycling penalty [13]. Inverter-driven (variable-capacity) heat pumps were introduced specifically to suppress cycling: by modulating compressor speed they track the load continuously, run for longer at lower capacity, and thereby raise seasonal efficiency while reducing the number of damaging start-ups [9, 8, 7]. A controller for such equipment should therefore be judged on three axes simultaneously: thermal comfort, operating (electricity) cost, and control smoothness, i.e. how much avoidable cycling it commands. Reinforcement learning is a popular model-free route to building HVAC control [15, 14], with supervised benchmarking through the open BOPTEST framework [2]. Yet most RL control studies reward only a weighted sum of comfort violation and energy cost; the cycling behaviour of the learned policy is rarely measured and almost never optimised so that a policy can earn an excellent cost–comfort score while chattering the compressor in a way that real hardware could not sustain. This paper closes that gap for hydronic heat-pump control. We make three contributions. 1. A smoothing-aware reward that augments the comfort-plus-cost objective with a physically grounded, monetised compressor-wear term, levelising the compressor cost over its rated number of starts (Section 3.2)—making switching a first-class objective without an explicit minimum-runtime constraint. 2. An algorithm-class effect on smoothness: holding the environment, reward and observations fixed, the RL algorithm determines the qualitative form of the control signal. SAC learns a continuous-modulation policy that never switches off (zero starts/day, inverter-style operation), whereas PPO learns a bang–bang policy that cycles more than the uncontrolled baseline. 3. A validated comfort–cost trade-off: transferred to the BOPTEST emulator for this test case, the smooth SAC policy beats the built-in baseline on thermal discomfort by up to 90.7 %90.7\, 37 for a modest cost increase, while removing all baseline cycling. Our focus is the reward design, the resulting control behaviour, and its BOPTEST validation. 2 Related Work RL for building and HVAC control: Data-driven building control has been surveyed extensively [15, 14], with deep RL built on actor–critic foundations for continuous control [6, 12]. Model-predictive control remains the strong supervised baseline [5], and hybrid RL–MPC schemes have also been explored [1]. BOPTEST [2] standardises the comparison through a common emulator/KPI interface; we adopt its bestest_hydronic_heat_pump case and KPI definitions. Cycling and capacity control of heat pumps: The penalty of on–off operation is well established in the refrigeration literature: Madani et al. [9, 8] and Karlsson and Fahlén [7] quantifies how variable-capacity (inverter) control reduces start-ups and raises seasonal performance, and rating standards encode the part-load cycling loss through the degradation coefficient [13]. These results motivate treating the number of starts as a control objective. Finally, that unconstrained RL produces high-frequency, oscillatory actions is a known failure mode addressed by explicit smoothness regularisers [10]; we show that for heat-pump cycling the algorithm class—entropy-regularised off-policy learning—can itself supply the smoothness bias. 3 Problem Formulation We formulate single-zone heat-pump control as a discrete-time Markov decision process (MDP) with a control step of Δt=900 s t=$900\,s$ (15 min), matching the BOPTEST test case. Episodes span 14 days (13441344 steps). 3.1 Action, Observation, and Heat-Pump Operating Principle The action is the scalar heat-pump command uHP∈[0,1]u_HP∈[0,1], the normalised compressor capacity, where 0 is fully off and 11 is rated capacity. The observation is a Case-D–aligned vector (zone temperature, ambient temperature, direct-normal irradiance, internal gains, the comfort set-points, the current price, and a sine/cosine time-of-day encoding) followed by an 8-step look-ahead forecast of the exogenous signals; two duty features (previous on/off state and normalised dwell time) are appended when the wear term is active. The action semantics tie the learning problem to the hardware. A single-stage (on–off) unit admits only the endpoints uHP∈0,1u_HP∈\0,1\ and meets partial loads by duty-cycling between them—each alternation a start. An inverter unit admits the whole interval and meets a partial load by sustaining an intermediate capacity, eliminating the alternation [7, 9, 8]. Our action space exposes the full interval, so the agent is free to behave as an on–off thermostat (the endpoints) or as an inverter (the interior); the emerged behaviour is selected by the learning algorithm, and that selection is the object of study. 3.2 Reward The per-step reward is the negative weighted sum of three integrated costs over Δt t: rt=−(wdiscJtdisc+wcostJtcost+wwearJtwear).r_t=- (\,w_disc\,J^disc_t+w_cost\,J^cost_t+w_wear\,J^wear_t\, ). (1) The thermal-discomfort term integrates the violation of the time-varying comfort band, Jtdisc=(max⁡(Ttlo−Ttzone,0)+max⁡(Ttzone−Tthi,0))ΔthJ^disc_t= ( (T^lo_t-T^zone_t,0)+ (T^zone_t-T^hi_t,0) )\, t_h, in kelvin-hours; the electricity-cost term is Jtcost=PttotΔthπt/1000J^cost_t=P^tot_t\, t_h\, _t/1000 in euros, with πt _t the (highly dynamic) tariff. The novel compressor-wear term monetises switching. With a binary duty dt=[uHP,t>ε]d_t=1[u_HP,t> ] (ε=10−3 =10^-3), a start is a rising edge max⁡(dt−dt−1,0) (d_t-d_t-1,0), and each start is charged a levelised cost Jtwear=ccyc⋅max⁡(dt−dt−1,0),ccyc=0.0133 €,J^wear_t=c_cyc· (d_t-d_t-1,0), c_cyc=$0.0133\,€$, (2) obtained by amortising a representative compressor replacement (∼2000 € 2000\,€) over its rated ∼150 000 150\,000 starts. A short-cycle surcharge penalises starts separated by fewer than four steps. We deliberately express wear in the same currency (euros) as the energy term so that the weights w have a transparent economic meaning; in all experiments below wdisc=wcost=wwear=1w_disc=w_cost=w_wear=1. A key property of (2) is that a controller can drive JwearJ^wear to zero in two ways: by never running the compressor, or by running it continuously so that no rising edge ever occurs. The first is excluded by the discomfort term (the zone would freeze); the second is exactly the inverter-modulation regime. Whether an RL agent discovers the second, benign optimum turns out to depend on the algorithm. 4 Method Training environment (surrogate plant): The agents are not trained on the BOPTEST emulator directly but on a fast, data-driven surrogate of the plant identified offline using Sparse Identification of Nonlinear Dynamics (SINDy) [3]. The surrogate advances the zone thermal state one control step at a time, mapping the current state, the heat-pump command uHPu_HP and the exogenous inputs to the predicted next-step state; the agent’s observation at each step is formed directly from this predicted state, so the full 5×1065×10^6-step training budget is collected at a small fraction of the cost of stepping the high-fidelity emulator. A companion map supplies the heat-pump electrical power entering the cost and wear terms of (1). We adopt this surrogate as-is from our earlier work [4] and regard its identification as outside the present scope; the resulting policies are transferred to the genuine BOPTEST emulator for all evaluation reported below. We train two model-free agents with Stable-Baselines3 [11]. SAC [6] is an off-policy, maximum-entropy actor–critic whose stochastic policy, entropy bonus and replay buffer reward exploration of the interior of the action interval, biasing it away from the saturated rails. PPO [12] is an on-policy clipped policy-gradient method for which the cheapest way to raise the average duty is to shift probability mass onto the uHP=1u_HP=1 and 0 rails—a tendency towards bang–bang. Both agents share network sizes, discount (γ=0.99γ=0.99), learning rate (3×10−43× 10^-4), observation space and the reward of (1); only the algorithm differs, and each is trained for 5×1065×10^6 steps before out-of-sample evaluation on the genuine BOPTEST emulator. Evaluation protocol: We evaluate on the two BOPTEST scenarios—the peak and typical heating periods, each 14 days under the highly dynamic tariff—against the test case’s built-in baseline controller. We report (i) BOPTEST closed-loop KPIs; (i) cycling metrics from the binary duty signal (switches and starts per day, and the wear of (2)); and (i) statistics of the commanded signal. The baseline duty is inferred from its power trace (PHeaPum>50 W⇒P_HeaPum>$50\,W$ on). 5 Results 5.1 SAC Eliminates Cycling; PPO Worsens It Table 1 reports the cycling behaviour. SAC records zero switches and zero starts on both scenarios, eliminating the 1.07 to 1.501.071.50 daily starts of the baseline. PPO does the opposite, cycling 2 to 424 times more often than the baseline and turning the wear term negative. The same reward, optimised by two algorithms, yields opposite cycling outcomes. Table 1: Compressor cycling and monetised wear over the 14-day scenarios. Lower is better. “Starts/day” is the number of off→ transitions; wear follows Eq. (2); TV is the mean step-to-step change of the command. Scenario Controller Switch/d Start/d Wear (€/d) TV Peak Baseline 2.14 1.07 0.0143 0.021 SAC 0.00 0.00 0.0000 0.046 PPO 8.51 4.29 0.0570 0.096 Typical Baseline 3.00 1.50 0.0200 0.018 SAC 0.00 0.00 0.0000 0.025 PPO 5.29 2.64 0.0352 0.055 5.2 The Control Signal is Continuous, not Gated The commanded traces explain how SAC reaches zero starts. SAC’s uHPu_HP varies substantially (non-zero TV in Table 1) yet never drops to the off state: uHP∈[0.007,0.93]u_HP∈[0.007,0.93] with 0.0 %0.0\, 37 of steps below the duty threshold on the peak day, and [0.001,0.85][0.001,0.85] with again 0.0 %0.0\, 37 off on the typical day—the compressor turns on once and then modulates continuously, the signature of inverter operation. PPO instead spends 60.6 %60.6\, 37 (peak) and 82.3 %82.3\, 37 (typical) of steps fully off and saturates at the uHP=1u_HP=1 rail—a bang–bang controller. This is the central empirical finding of the paper, and Fig. 1 shows it directly for both the peak and typical days. (a) SAC, peak day — smooth modulation, 0 starts (b) PPO, peak day — 4.29 starts/day (c) SAC, typical day — smooth modulation, 0 starts (d) PPO, typical day — 2.64 starts/day Figure 1: Same reward, opposite controllers. Closed-loop traces on BOPTEST for the peak (top) and typical (bottom) heating days; each panel shows zone temperature against its comfort band (upper) and the commanded capacity uHPu_HP (lower). SAC (a, c) modulates smoothly and never lets the compressor restart—zero starts/day with comfort intact, learned inverter operation even on the milder day. PPO (b, d) slams uHPu_HP between off and on, firing 4.294.29 and 2.642.64 starts/day, more than the baseline. 5.3 Beating Discomfort on BOPTEST On the peak day (Table 2) SAC cuts thermal discomfort from 8.38 8.38\, to 0.78 K h0.78\,K\,h (a 90.7 %90.7\, 37 reduction) for an 11.5 %11.5\, 37 increase in operating cost, and reduces it by 33.1 %33.1\, 37 on the typical day. The smooth policy thus purchases comfort with a small, bounded cost premium while removing all cycling. PPO occupies the opposite corner: it lowers cost (it heats less) but barely improves comfort (17.9 %17.9\, 37 and 3.8 %3.8\, 37 ) and cycles heavily. Table 2: BOPTEST closed-loop KPIs (14-day scenarios). tdist_dis: thermal discomfort (K hK\,h); cost (€); energy: source energy (kW hkW\,h); emis: emissions (kgkg CO2). Lower is better for discomfort, cost and emissions. Scenario Controller tdist_dis cost energy emis Peak Baseline 8.382 0.909 3.478 0.581 SAC 0.777 1.013 3.860 0.645 PPO 6.884 0.631 2.436 0.407 Typical Baseline 9.446 0.413 1.773 0.296 SAC 6.322 0.631 2.721 0.454 PPO 9.088 0.255 1.107 0.185 The energy column of Table 2 locates SAC’s cost premium: it draws 11.0 %11.0\, 37 more source energy than the baseline, almost exactly its 11.5 %11.5\, 37 cost increase, so the premium buys capacity, not tariff mistiming—the modulating compressor simply delivers more heat. PPO’s apparently low cost and emissions are instead an artefact of under-delivery: it leaves the zone cold and cycles to do so. 6 Discussion Why SAC recovers inverter operation? The wear term (2) has two zero-cost optima—never on, or always on—and the discomfort term rules out the first. Reaching the second requires holding uHPu_HP strictly inside (ε,1]( ,1] for the whole episode while tracking a time-varying load. SAC’s maximum-entropy objective values a broad, stochastic action distribution and penalises collapse onto the deterministic rails, while its off-policy replay smooths the critic; interior, continuously-modulated actions thus become the path of least resistance, and the agent converges to the benign always-on optimum. PPO, lacking a comparable entropy floor and updating on-policy, finds it easier to raise the average duty by mixing fully-on and fully-off actions, landing in the bang–bang basin. SAC’s policy is an inverter controller, learned rather than engineered [7, 9, 8], while PPO’s is a thermostat. That the zero-start result holds across two scenarios differing by an order of magnitude in mean capacity (u¯=0.287 u=0.287 peak versus 0.0980.098 typical) indicates that continuous modulation is a robust attractor of the SAC objective, not an artefact of one operating point. Trade-off and practical implications: The algorithms map onto opposite ends of the Pareto picture: SAC is comfort-seeking (a small, bounded cost premium for near-zero discomfort and zero wear) and PPO cost-seeking (under-heating to save energy); for a heat pump SAC’s corner is the desirable one, since comfort is the primary obligation and eliminating cycling protects the most expensive component. Because the wear term is levelised in the same currency as operating cost, the two are directly comparable: over the 14-day peak scenario SAC avoids all 1515 baseline starts, worth 0.20 €0.20\,€ of levelised wear at the assumed 0.0133 €0.0133\,€ per start, which exceeds the 0.10 €0.10\,€ (11.5 %11.5\, 37 ) operating premium—so the premium is recovered roughly twofold (break-even at 0.007 €0.007\,€ per start), and the 2121 avoided starts on the typical scenario cover their premium likewise (≈1.3×≈\!1.3×, break-even 0.010 €0.010\,€ per start). Field controllers normally suppress cycling with hand-tuned timers and hysteresis; our result shows equivalent smoothness can arise endogenously by pricing each start, with wwearw_wear as the single knob along the smoothness–cost frontier. Limitations: The duty threshold ε=10−3 =10^-3 is permissive: a controller need only hold a near-zero capacity to register as “on,” lowering the bar for the always-on optimum; a threshold at a real inverter’s minimum stable modulation (10 % to 20 %10\, 37 20\, 37 ) would be a stricter test. The per-start cost is small in absolute terms and serves as a lifetime proxy rather than an operating expense. Finally, training uses an uncalibrated surrogate plant, so we rely on the out-of-sample BOPTEST KPIs (Table 2) for cost–comfort claims and on the measured duty signal (Table 1, Fig. 1) for smoothness claims. 7 Conclusion We added a monetised compressor-wear term to a heat-pump control reward and showed that the RL algorithm then determines the qualitative form of the learned control law. SAC discovered a continuous-modulation policy that eliminates cycling, reproducing the operating principle of an inverter-driven heat pump, while PPO collapsed to a bang–bang policy that cycled more than the baseline; on BOPTEST the SAC policy cut thermal discomfort by up to 90.7 %90.7\, 37 for a modest, bounded cost increase. For equipment whose lifetime is governed by switching, entropy-regularised off-policy learning is thus structurally aligned with the smoothness the hardware requires. Future work will sweep the wear weight to trace the switches-versus-(cost + discomfort) Pareto frontier and benchmark against an MPC baseline on BOPTEST. References [1] J. Arroyo, C. Manna, F. Spiessens, and L. Helsen (2022) Reinforced model predictive control (rl-mpc) for building energy management. 309, p. 118346. External Links: ISSN 0306-2619, Document, Link Cited by: §2. [2] D. Blum, J. Arroyo, S. Huang, J. Drgoňa, F. Jorissen, H. T. Walnum, Y. Chen, K. Benne, D. Vrabie, M. Wetter, and L. Helsen (2021) Building optimization testing framework (boptest) for simulation-based benchmarking of control strategies in buildings. 14 (5), p. 586–610. External Links: Document, Link, https://doi.org/10.1080/19401493.2021.1986574 Cited by: §1, §2. [3] S. L. Brunton, J. L. Proctor, and J. N. Kutz (2015) Discovering governing equations from data: sparse identification of nonlinear dynamical systems. Cited by: §4. [4] A. Dixit, M. I. Khan, F. Ahmed, and J. Brusey (2025) Learning from less: sindy surrogates in rl. External Links: 2504.18113, Link Cited by: §4. [5] J. Drgoňa, J. Arroyo, I. Cupeiro Figueroa, D. Blum, K. Arendt, D. Kim, E. P. Ollé, J. Oravec, M. Wetter, D. L. Vrabie, and L. Helsen (2020) All you need to know about model predictive control for buildings. 50, p. 190–232. External Links: ISSN 1367-5788, Document, Link Cited by: §2. [6] T. Haarnoja, A. Zhou, P. Abbeel, and S. Levine (2018-07) Soft actor-critic: off-policy maximum entropy deep reinforcement learning with a stochastic actor. In Proceedings of the 35th International Conference on Machine Learning, J. Dy and A. Krause (Eds.), Proceedings of Machine Learning Research, Vol. 80, p. 1861–1870. External Links: Link Cited by: §2, §4. [7] F. Karlsson and P. Fahlén (2007) Capacity-controlled ground source heat pumps in hydronic heating systems. International Journal of Refrigeration 30 (2), p. 221–229. External Links: ISSN 0140-7007, Document, Link Cited by: §1, §2, §3.1, §6. [8] H. Madani, J. Claesson, and P. Lundqvist (2011) Capacity control in ground source heat pump systems part i: comparative analysis between on/off controlled and variable capacity systems. International Journal of RefrigerationApplied EnergyApplied EnergyJournal of Building Performance SimulationAnnual Reviews in ControlApplied EnergyJournal of Machine Learning ResearcharXiv preprint arXiv:1509.03580 34 (8), p. 1934–1942. External Links: ISSN 0140-7007, Document, Link Cited by: §1, §2, §3.1, §6. [9] H. Madani, J. Claesson, and P. Lundqvist (2011) Capacity control in ground source heat pump systems: part i: modeling and simulation. International Journal of Refrigeration 34 (6), p. 1338–1347. External Links: ISSN 0140-7007, Document, Link Cited by: §1, §2, §3.1, §6. [10] S. Mysore, B. Mabsout, R. Mancuso, and K. Saenko (2021) Regularizing action policies for smooth control with reinforcement learning. In 2021 IEEE International Conference on Robotics and Automation (ICRA), Vol. , p. 1810–1816. External Links: Document Cited by: §2. [11] A. Raffin, A. Hill, A. Gleave, A. Kanervisto, M. Ernestus, and N. Dormann (2021) Stable-baselines3: reliable reinforcement learning implementations. 22 (268), p. 1–8. External Links: Link Cited by: §4. [12] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov (2017) Proximal policy optimization algorithms. External Links: 1707.06347, Link Cited by: §2, §4. [13] A. Standard (2008) Performance rating of unitary air-conditioning & air-source heat pump equipment. AHRI Standard 210, p. 240. Cited by: §1, §2. [14] J. R. Vázquez-Canteli and Z. Nagy (2019) Reinforcement learning for demand response: a review of algorithms and modeling techniques. 235, p. 1072–1089. External Links: ISSN 0306-2619, Document, Link Cited by: §1, §2. [15] Z. Wang and T. Hong (2020) Reinforcement learning for building controls: the opportunities and challenges. 269, p. 115036. External Links: ISSN 0306-2619, Document, Link Cited by: §1, §2.

Instant research discovery