Paper deep dive
Max-Q Selective Imitation for Human-in-the-Loop Online Robot Learning
Zihang Wang, Yishan Wang
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 96%
Last extracted: 8/18/2026, 5:26:12 AM
Summary
The paper introduces a Human-in-the-Loop (HIL) online reinforcement learning method for robots, featuring an MC Q-chunk critic for sample-average behavior evaluation and a max-Q selective imitation actor. This approach allows the robot to quickly absorb human interventions while transitioning to on-policy self-improvement, significantly reducing training time compared to baselines like HIL-SERL, EXPO, and E2HiL in both simulation and real-robot tasks.
Entities (11)
Relation Signals (9)
ACT QChunk-MCBC → testedon → Peg Insertion
confidence 99% · In simulation on Peg Insertion... ACT QChunk-MCBC attains 99% success
ACT QChunk-MCBC → testedon → USB pick-and-insertion
confidence 99% · On a real USB pick-and-insertion task... ACT QChunk-MCBC attains 99% success
ACT QChunk-MCBC → outperforms → HIL-SERL
confidence 95% · ACT QChunk-MCBC attains 99% success within 30 minutes... whereas HIL-SERL requires about 5 hours
Flow QChunk-MCBC → uses → MC Q-chunk Critic
confidence 95% · Flow QChunk-MCBC... share the MC Q-chunk critic
Flow QChunk-MCBC → uses → Max-Q Selective Imitation
confidence 95% · Flow QChunk-MCBC... and max-Q selective imitation
ACT QChunk-MCBC → uses → Max-Q Selective Imitation
confidence 95% · Our stack appears as ACT QChunk-MCBC... and max-Q selective imitation
ACT QChunk-MCBC → uses → MC Q-chunk Critic
confidence 95% · Our stack appears as ACT QChunk-MCBC... which share the MC Q-chunk critic
MC Q-chunk Critic → reduces → policy-target-sample gap
confidence 90% · credits intervention trajectories directly rather than diluted by current-policy TD backups
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Human-in-the-loop (HIL) online reinforcement learning for real robots must absorb human interventions quickly while continuing to improve beyond the human prior. We present a training method for this setting based on two components. First, an \emph{MC Q-chunk} critic regresses chunk-level action values onto Monte Carlo returns from the replay buffer, performing sample-average (behavior) policy evaluation so that intervention trajectories are credited directly rather than diluted by current-policy TD backups. Second, \emph{max-Q selective imitation} updates the actor by imitating, at each state, the higher-$Q$ action between the current policy action and a buffer sample under a hard winner-take-all rule. This rule automatically switches between learning from interventions and on-policy self-improvement: when the autonomous policy is stronger, targets align with the policy distribution, reducing the policy--target-sample gap that otherwise induces execution-time distribution shift. In practice we score candidates with a standard critic ensemble mean to reduce comparison noise, without softening targets or introducing score-gap thresholds. On a real USB pick-and-insertion task with 20 demonstrations, ACT QChunk-MCBC attains 99\% success within 30 minutes of HIL training, whereas HIL-SERL requires about 5 hours to converge. In simulation on Peg Insertion and Square, ACT/Flow Q-chunk variants similarly reach $\ge$96\% success within roughly half an hour of effective training, outperforming HIL-SERL, EXPO, and E2HiL on the success--time frontier.
Tags
Links
- Source: https://arxiv.org/abs/2608.15088v1
- Canonical: https://arxiv.org/abs/2608.15088v1
Trouble viewing inline? Open PDF directly →
Full Text
21,397 characters extracted from source content.
Expand or collapse full text
Max-Q Selective Imitation for Human-in-the-Loop Online Robot Learning with Monte Carlo Q-Chunk Critics Zihang Wang ∗ Yishan Wang ∗ Abstract Human-in-the-loop (HIL) online reinforcement learning for real robots must absorb human interventions quickly while continuing to improve beyond the human prior. We present a training method for this setting based on two components. First, an MC Q-chunk critic regresses chunk- level action values onto Monte Carlo returns from the replay buffer, performing sample-average (behavior) policy evaluation so that intervention trajectories are credited directly rather than diluted by current-policy TD backups. Second, max-Q selective imitation updates the actor by imitating, at each state, the higher-Q action between the current policy action and a buffer sample under a hard winner-take-all rule. This rule automatically switches between learning from interventions and on-policy self-improvement: when the autonomous policy is stronger, targets align with the policy distribution, reducing the policy–target-sample gap that otherwise induces execution-time distribution shift. In practice we score candidates with a standard critic ensemble mean to reduce comparison noise, without softening targets or introducing score-gap thresholds. On a real USB pick-and-insertion task with 20 demonstrations, ACT QChunk- MCBC attains 99% success within 30 minutes of HIL training, whereas HIL-SERL requires about 5 hours to converge. In simulation on Peg Insertion and Square, ACT/Flow Q-chunk variants similarly reach ≥96% success within roughly half an hour of effective training, outperforming HIL-SERL, EXPO, and E2HiL on the success–time frontier. 1 Introduction Real-world robot reinforcement learning (RL) remains constrained by interaction cost, contact- rich failure modes, and limited exploration under safety requirements. Human-in-the-loop (HIL) protocols address these constraints by combining autonomous rollouts with human interventions, yielding a replay buffer that mixes near-policy transitions with corrective actions that may lie far from the current policy π. Effective learning in this regime requires both rapid uptake of useful human corrections and continued improvement beyond demonstration- or intervention-level performance. Existing approaches only partially meet these requirements. Behavioral cloning of interventions is stable but conservative. Unconstrained maximization of Q(s,a) over continuous actions invites out-of-distribution (OOD) overestimation. Offline extractors such as Implicit Q-Learning (IQL) [2] keep updates in data support via advantage-weighted regression (AWR), yet AWR always imitates buffer actions and, under a mixed HIL buffer, repeatedly pulls π across heterogeneous sample modes. As a result, learning targets need not converge to the live policy even after π improves, leaving a ∗ Equal contribution. 1 arXiv:2608.15088v1 [cs.RO] 15 Aug 2026 action / chunk space ① Two distributions Human / past buffer p b Current policy π action / chunk space ② Q selects → p ⋆ p ⋆ Q(s, a) estimated ̂ Q(s, a) action / chunk space π 0 = π→p ⋆ ③ Toward Q-optimal p ⋆ old π new π 0 π ⋆ Q(s, a) action / chunk space π 0 →π ⋆ ④ On-policy π 0 →π ⋆ new π 0 on-policy mid π ⋆ Q(s, a) OODOOD ⟶ Max-Q Selective Imitation Max over a b , a π forms p ⋆ ; then on-policy learning from near π 0 converges to π ⋆ = greedy(Q). Figure 1: Max-Q selective imitation in action / chunk space. (1) The replay buffer induces a human/past sample distribution p b while the learner maintains a current policy π. (2) An MC Q-chunk critic provides Q(s,a); the estimated ˆ Q is reliable on support but overestimates out of distribution. Hard max over candidates from p b ,π selects winners that define a target p ⋆ . (3) The updated policy π ′ lies between π and p ⋆ (π ′ = π→ p ⋆ ). (4) On-policy learning near π ′ further concentrates mass toward π ⋆ = greedy(Q). persistent gap between the policy distribution and the target-sample distribution. Because closed- loop execution follows π, this gap exposes the robot to state–action regions underrepresented by the training targets and compounds tracking error. SAC-style HIL systems such as HIL-SERL [5] stay closer to the policy distribution, but their TD backups bootstrap under a ′ ∼ π rather than under the sample-average behavior in the buffer, which under-utilizes intervention returns. We address these issues with a two-part method. The critic learns chunk-level values by Monte Carlo regression onto logged returns (MC Q-chunk ), i.e., sample-average behavior evaluation that credits intervention paths. The actor performs max-Q selective imitation: for each state it compares a π = π(s) and a buffer action a b , and regresses toward the hard winner a ⋆ = arg max a∈a π ,a b Q(s,a).(1) When interventions have higher Q, they are imitated; when a π wins, updates become on-policy and the target-sample distribution aligns with execution under π. As an implementation detail, we evaluate Q with a standard critic-ensemble mean to reduce early ranking noise while keeping the discrete, non-soft switching rule. Figure 1 summarizes the idea in action space. Reading Figure 1. Panel (1) shows the two sources that feed learning: the mixed human/past buffer distribution p b and the live policy π. Panel (2) scores candidates with ˆ Q; because uncon- strained maximization would chase OOD spikes of ˆ Q, we only compare in-support actions and take a hard winner, so the selected samples prop up p ⋆ . Panel (3) makes the policy update explicit: π ′ is an intermediate between the current π and the Max-selected target p ⋆ , absorbing high-Q human corrections without jumping fully onto the buffer. Panel (4) continues with on-policy improvement around π ′ , moving the mass toward the true optimum π ⋆ implied by Q, closing the policy–target gap at execution time. Contributions. 2 • An MC Q-chunk critic for sample-average evaluation of mixed HIL trajectories, improving uti- lization of intervention returns relative to π-bootstrapped TD critics. • Max-Q selective imitation between policy and buffer actions, enabling label-free hard switching from intervention learning to on-policy self-improvement and reducing the policy–target-sample gap. • Simulation on Peg Insertion and Square, where ACT/Flow Q-chunk variants reach≥96% success within roughly half an hour of effective training and outperform HIL-SERL, EXPO, and E2HiL on the success–time frontier; plus real-robot USB pick-and-insertion, where ACT QChunk-MCBC reaches 99% in 30 minutes while HIL-SERL converges in about 5 hours. 2 Related Work Human-in-the-loop robot RL. Interactive and HIL methods inject corrections or interventions during training [5]. HIL-SERL combines demonstrations, interventions, and off-policy SAC-style learning on hardware. We share this protocol but replace π-conditioned TD evaluation with MC Q-chunk behavior evaluation, and replace likelihood-oriented actor updates with max-Q selective imitation. Offline and offline-to-online RL. Offline RL limits OOD queries by constraining policies to data support [2, 3]. IQL evaluates values in-sample and extracts policies with AWR. In online HIL fine-tuning, AWR continues to target buffer actions only, so improvement can cross-jump among modes rather than settle on the live policy. Our actor explicitly includes a π as a candidate, allowing on-policy targets once π dominates under Q. Action chunking and Q-chunking. Action chunking is widely used for temporally coherent imitation. Q-chunking [1] lifts TD actor–critic learning into a chunked action space. We retain chunk-level Q(s, a) but fit it with Monte Carlo returns so evaluation remains sample-average and does not bootstrap under the current π. Max-over-candidates policy improvement. BCQ and EMaQ [4] improve policies by max- imizing Q over actions proposed near behavior support. Our variant is specialized to HIL: one candidate is always the live policy action and the other a replay sample (often an intervention), with improvement realized by supervised imitation of the hard winner. 3 Method 3.1 Problem Setup We consider an MDP (S,A,P,r,γ) under a HIL online protocol. The replay buffer B is initial- ized with a small demonstration set and expanded by autonomous rollouts together with human interventions. Updates do not require intervention labels: each transition is treated as an ordinary state–action pair. The objective is high task success under a limited robot interaction budget. 3.2 MC Q-Chunk Critic Let h be the chunk horizon and a t:t+h−1 = (a t ,...,a t+h−1 ) an action chunk. The critic Q θ (s t , a t:t+h−1 ) scores the chunk from s t . Rather than TD bootstrapping under the current policy, we regress onto 3 the Monte Carlo return along the logged trajectory, G t = T t −1 X k=0 γ k r t+k ,(2) and minimize L Q =E (s t ,a t:t+h−1 ,G t )∼B h Q θ (s t , a t:t+h−1 )− G t 2 i .(3) Because G t is generated by the buffer behavior—a mixture of autonomous and intervention actions— Q θ estimates values under the sample-average behavior policy. Intervention actions on informative trajectories raise MC targets directly and remain visible to subsequent max-Q selection. Chunk- level scoring matches the temporal structure of demonstrations and interventions. Implementation detail: critic ensemble. Hard switching is sensitive to early Q noise. As a standard variance-reduction trick [6]—not a methodological contribution—we optionally instantiate the ranking score as the mean of M independently initialized critics trained on the same MC objective, ̄ Q(s,a) = 1 M M X i=1 Q θ i (s,a),(4) and write Q← ̄ Q in the actor comparison below. A single critic recovers the same learning rule. 3.3 Max-Q Selective Imitation Pure buffer imitation cannot prefer an improving policy over stale logged actions. Unconstrained arg max a Q(s,a) leaves the support where Q is reliable. We therefore compare two local candidates and imitate the hard winner. For a sampled state s, a π = π φ (s),a b ∼B(·| s),(5) a ⋆ = arg max a∈a π ,a b Q θ (s,a),(6) and L π =E s∼B ∥π φ (s)− a ⋆ ∥ 2 2 ,(7) optionally with an analogous chunk behavioral-cloning loss. Stop-gradient through a ⋆ and the Q comparison prevents the actor objective from distorting the critic. Importantly, the target remains a single discrete action: we do not mix a π and a b with soft weights, nor gate the switch by a hand-tuned score margin. Soft targets would keep pulling π toward buffer actions even after the policy is stronger, recreating the policy–target-sample gap that the hard rule is designed to close. This construction has three operational consequences. First, every scored action lies in the union of policy and data support, limiting OOD Q queries. Second, the hard max provides a directed improvement signal: the actor moves only toward the currently higher-Q candidate. Third, the same rule yields a label-free switch for HIL data. High-Q interventions are imitated when they outrank a π ; once π is stronger, a π wins more often, so L π regresses the policy onto itself and the target-sample distribution aligns with the policy (and thus with execution). This reduces closed- loop exposure to regions underrepresented by training targets. We use supervised regression onto a ⋆ as the actor implementation; likelihood-weighted alternatives are left as optional baselines in experiments. Online interaction still executes (optionally noisy) actions from π, preserving autonomous ex- ploration whenever self-generated actions attain higher Q. 4 Algorithm 1 HIL Online Learning with MC Q-Chunk and Max-Q Imitation 1: Initialize B with demonstrations; initialize Q θ (optionally an ensemble), π φ 2: for each online interaction step do 3:Execute a∼ π φ (with exploration); allow intervention 4:Store transition/chunk in B; compute G t on completed trajectories 5:Sample minibatch from B 6:Update Q θ by MC regression onto G t 7:Set a ⋆ ← arg max a∈a π ,a b Q θ (s,a) and imitate a ⋆ 8: end for 0.00.51.01.52.02.53.0 Training time since first checkpoint (hours) 0% 25% 50% 75% 100% Success rate HIL-SERL: recorded pure_train_time. Others: x = pure_train_time − time_of_first_checkpoint. Peg Insertion: success vs real time (t0 = first checkpoint) HIL-SERL MCBC ACT QChunk-MCBC Flow QChunk-MCBC EXPO E2HiL 012345 Env-derived time (hours) 0.000.250.500.751.001.251.501.752.00 Env steps 1e5 0% 25% 50% 75% 100% Success rate Env: screenshot / interaction_step if available; else β×training_steps. β=2.002. Peg Insertion: success vs env steps (to peak) HIL-SERL MCBC ACT QChunk-MCBC Flow QChunk-MCBC EXPO E2HiL Figure 2: Peg Insertion: success versus real training time (left; HIL-SERL uses recorded time, others start at the first checkpoint) and versus environment steps (right). Curves stop at each method’s peak checkpoint. 3.4 Training Procedure Algorithm 1 summarizes the loop (with the optional ensemble scoring trick). 4 Experiments We evaluate HIL online learning in simulation on two contact-rich gym hil tasks—Peg Insertion and Square—against HIL-SERL [5], plain MCBC (no Q-chunk actor), EXPO, and E2HiL. Our stack appears as ACT QChunk-MCBC and Flow QChunk-MCBC, which share the MC Q-chunk critic and max-Q selective imitation and differ only in the chunked action head. Success is measured by autonomous evaluation episodes at saved checkpoints. Curves are truncated at each method’s best checkpoint (highest success, then lowest average episode length). Time axes use logged pure training time; for all methods except HIL-SERL we zero the clock at the first checkpoint so early process warmup without interaction is excluded. HIL-SERL retains its recorded optimization- window time. When environment steps are not logged we impute them as β× training steps with β≈ 2.0 fit on EXPO/E2HiL. 4.1 Peg Insertion Figure 2 and Table 1 summarize Peg Insertion. Both Q-chunk variants reach near-ceiling success much earlier than HIL-SERL and E2HiL: ACT QChunk-MCBC attains 99% by 0.32 h after the first checkpoint (≈4.4k training steps), and Flow QChunk-MCBC reaches 96% by 0.50 h. Plain MCBC also climbs to 96% by 0.67 h, showing that selective imitation of high-Q buffer actions already helps, while action chunking further shortens the time to peak under the same MC critic. 5 HIL-SERL MCBC ACT QChunk-MCBC Flow QChunk-MCBC EXPO E2HiL 0% 25% 50% 75% 100% Success rate 100% @100000 96% @10112 99% @4386 96% @5075 30% @26000 90% @46000 Peg Insertion: best-checkpoint success (Wilson 95% CI) Figure 3: Peg Insertion: best-checkpoint success with Wilson 95% confidence intervals. Table 1: Peg Insertion: peak checkpoint summary MethodStepsEnv Hours † Success HIL-SERL100 000 200 1963.19100% MCBC10 112 20 9060.6796% ACT QChunk-MCBC4 3868 7810.3299% Flow QChunk-MCBC5 075 10 1600.5096% EXPO26 000 54 7821.5530% E2HiL46 000 93 3152.3690% † Hours after first checkpoint (HIL-SERL: recorded pure train time). HIL-SERL eventually matches 100% success but only after ≈3.2 h / 100k steps, reflecting slower uptake of interventions when TD backups remain on-policy. E2HiL peaks at 90% near 2.4 h; EXPO saturates at 30% and does not recover within the logged window (Figure 3). Overall, the MC Q- chunk + max-Q stack delivers the best success–time trade-off on this task. 4.2 Square On Square (Figure 4, Table 2), the same ranking appears. Flow QChunk-MCBC rises from mid- training success to a 96% peak at 2 200 steps / 0.60 h after the first checkpoint. ACT QChunk- MCBC again reaches the highest peak earliest (99% by 0.38 h), while plain MCBC hits 96% by 0.80 h. HIL-SERL eventually reaches 100% but only after several hours of training. EXPO and E2HiL remain near failure, peaking at 10%. Thus Square confirms that the MC Q-chunk + max-Q stack is sample- and time-efficient on a second contact-rich assembly task, whereas competing HIL baselines do not close the gap under the same protocol. 4.3 Real-robot USB insertion We evaluate on a real USB pick-and-insertion task (Figure 5): a Franka-class arm with a wrist camera and parallel gripper must grasp a USB plug and insert it into a motherboard rear I/O port under dual LED lighting. Training uses 20 human demonstrations, autonomous rollouts, and human interventions. Figure 6 and Table 3 report success against robot training time. HIL-SERL reaches ceiling success after approximately 5 hours of HIL training. Under the same protocol, ACT QChunk-MCBC attains 99% by 30 minutes, Flow QChunk-MCBC reaches 96% by roughly 6 0.00.51.01.52.02.53.03.54.0 Training time since first checkpoint (hours) 0% 25% 50% 75% 100% Success rate Square: success vs real time (t0 = first checkpoint) HIL-SERL MCBC ACT QChunk-MCBC Flow QChunk-MCBC EXPO E2HiL HIL-SERL MCBC ACT QChunk-MCBC Flow QChunk-MCBC EXPO E2HiL 0% 25% 50% 75% 100% Success rate 100% @43350 96% @4384 99% @1901 96% @2200 10% @32000 10% @46000 Square: best-checkpoint success (Wilson 95% CI) Figure 4: Square: success versus training time since the first checkpoint (left) and best-checkpoint success (right). Table 2: Square: peak checkpoint summary MethodStepsEnv Hours † Success HIL-SERL43 350 339 6263.85100% MCBC4 384 35 4660.8096% ACT QChunk-MCBC 1 901 14 8960.3899% Flow QChunk-MCBC 2 200 17 2360.6096% EXPO32 000 72 0312.0710% E2HiL46 000 93 3152.3610% † Same time convention as Table 1. 0.8 hours, and plain MCBC reaches 96% by about 1 hour—an order-of-magnitude reduction rela- tive to HIL-SERL. E2HiL peaks near 90% only after several hours, and EXPO saturates at 30%. These real-robot trends mirror the simulation success–time frontier: MC Q-chunk critics with max- Q selective imitation absorb interventions quickly and converge far sooner than TD-based HIL baselines. 4.4 Ablations To isolate the two components, we recommend (and plan to report) the following comparisons: (i) MC Q-chunk versus π-bootstrapped TD critics; (i) actor variants including full buffer imitation, SAC-style actors, unconstrained Q-maximization, online IQL/AWR, and max-Q selective imitation; (i) the fraction of updates in which a π wins over a b over training time; (iv) win rates stratified by intervention versus autonomous buffer actions (labels for analysis only); (v) sensitivity to demon- stration count and intervention frequency; (vi) as an optional check on the ensemble scoring trick, single critic versus ensemble mean. 5 Limitations The switching decision depends on critic quality; noisy early Q estimates can select suboptimal targets (partially mitigated in practice by a critic ensemble, which remains a heuristic rather than a complete fix). Monte Carlo returns typically have higher variance than TD backups and may require sufficient episode completion for stable targets. The chunk horizon h remains a task- dependent hyperparameter. Broader multi-task evaluation and a formal characterization of the two-candidate improvement operator are left for future work. 7 Figure 5: Real-robot USB pick-and-insertion setup. Left: workspace with dual LED illumination and a Franka-class arm holding a USB connector. Right: close-up of alignment into a motherboard rear I/O port. 012345 Robot training time (hours) 0% 25% 50% 75% 100% Success rate Real USB insertion: success vs training time (to peak) HIL-SERL MCBC ACT QChunk-MCBC Flow QChunk-MCBC EXPO E2HiL HIL-SERL MCBC ACT QChunk-MCBC Flow QChunk-MCBC EXPO E2HiL 0% 25% 50% 75% 100% Success rate 100% @5.0h 96% @1.0h 99% @0.5h 96% @0.8h 30% @2.4h 90% @3.7h Real USB insertion: best success (Wilson 95% CI) Figure 6: Real USB insertion: success versus robot training time (left; curves truncated at each method’s peak) and best-checkpoint success (right). 6 Conclusion We presented a HIL online robot learning method that combines an MC Q-chunk critic for sample- average behavior evaluation with hard max-Q selective imitation between policy and buffer actions. The resulting updates absorb useful interventions when they improve Q, and shift toward on-policy self-improvement as the autonomous policy strengthens, reducing the policy–target-sample gap at execution time. In simulation on Peg Insertion and Square, ACT/Flow Q-chunk variants reach ≥96% success within roughly half an hour of effective training and dominate HIL-SERL, EXPO, and E2HiL on the success–time frontier; on real USB pick-and-insertion, ACT QChunk-MCBC reaches 99% in 30 minutes while HIL-SERL requires about 5 hours to converge. 8 Table 3: Real USB insertion: peak training time and success MethodTime to peak Success HIL-SERL5.0 h100% MCBC1.0 h96% ACT QChunk-MCBC 0.5 h (30 min)99% Flow QChunk-MCBC0.8 h96% EXPO2.4 h30% E2HiL3.7 h90% References [1] Q. Li, Z. Zhou, and S. Levine, “Reinforcement Learning with Action Chunking,” in Proc. NeurIPS, 2025. [2] I. Kostrikov, A. Nair, and S. Levine, “Offline Reinforcement Learning with Implicit Q- Learning,” in Proc. ICLR, 2022. [3] S. Fujimoto, D. Meger, and D. Precup, “Off-Policy Deep Reinforcement Learning without Exploration,” in Proc. ICML, 2019. [4] S. K. S. Ghasemipour, D. Schuurmans, and S. S. Gu, “EMaQ: Expected-Max Q-Learning Operator for Simple Yet Effective Offline and Online RL,” in Proc. ICML, 2021. [5] J. Luo, C. Xu, J. Wu, and S. Levine, “Precise and Dexterous Robotic Manipulation via Human-in-the-Loop Reinforcement Learning,” arXiv:2410.21845, 2024. [6] X. Chen, C. Wang, Z. Zhou, and K. Ross, “Randomized Ensembled Double Q-Learning: Learning Fast Without a Model,” in Proc. ICLR, 2021. 9