Paper deep dive
AutoIntervene: Calibrated Intervention for Action-Chunking Imitation Learning Policies
Jinhe Tang, Weiming Zhi
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 89%
Last extracted: 8/10/2026, 4:06:19 AM
Summary
The paper introduces AutoIntervene, an online framework for improving the deployment reliability of action-chunking visuomotor policies. It addresses distribution shifts by selectively transferring control between the autonomous policy and a human operator based on visual-action support scores. The system uses phase-local support for policy-to-operator transfer and global support for operator-to-policy return, calibrated via empirical quantiles on held-out demonstrations. Retained intervention segments are used for corrective supervision in a DAgger-style loop, demonstrating improved task success and reduced operator control time on real-world bimanual manipulation tasks.
Entities (10)
Relation Signals (8)
AutoIntervene → improves → Deployment Reliability
confidence 95% · AutoIntervene is designed to improve a deployed action-chunking visuomotor policy by converting automatically triggered operator interventions into targeted supervision.
AutoIntervene → uses → Visual-Action Support Memory
confidence 92% · AutoIntervene evaluates proposed chunks against a visual-action support memory built from successful task executions
Phase-Local Support → governs → Policy-to-Operator Transfer
confidence 90% · Phase-local support governs policy-to-operator transfer within the current task phase
Global Support → governs → Operator-to-Policy Return
confidence 90% · global support governs the return to policy control after operator recovery.
Intervention Segments → usedfor → Policy Updates
confidence 90% · Intervention segments retained from successful rollouts target learner-induced states and provide corrective supervision for subsequent policy updates.
AutoIntervene → evaluateswith → Diffusion Policy
confidence 85% · We evaluate it with Action Chunking with Transformers (ACT), Diffusion Policy, and Flow Matching heads.
AutoIntervene → evaluateswith → Flow Matching
confidence 85% · We evaluate it with Action Chunking with Transformers (ACT), Diffusion Policy, and Flow Matching heads.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Action-chunking visuomotor policies learn from demonstrations and improve temporal consistency by predicting short action sequences rather than single-step commands. Yet perception errors and execution drift can move the robot outside the demonstration distribution, while the policy continues to produce smooth action chunks that are inconsistent with the observed state. We present AutoIntervene, an online framework that selectively transfers control between an action-chunking policy and an operator during deployment. AutoIntervene evaluates proposed chunks against a visual-action support memory built from successful task executions, combining visual similarity with consistency between proposed and reference actions. Phase-local support governs policy-to-operator transfer within the current task phase, whereas global support governs the return to policy control after operator recovery. We calibrate separate switching thresholds for the two directions from empirical quantiles of evaluation-level scores on held-out expert demonstrations, avoiding direct manual tuning of score cutoffs. Intervention segments retained from successful rollouts target learner-induced states and provide corrective supervision for subsequent policy updates. Experiments on real-world bimanual manipulation tasks show higher post-adaptation task success and lower operator-control time than manual intervention. Videos and additional results are available at this https URL.
Tags
Links
- Source: https://arxiv.org/abs/2608.07065v1
- Canonical: https://arxiv.org/abs/2608.07065v1
Trouble viewing inline? Open PDF directly →
Full Text
48,485 characters extracted from source content.
Expand or collapse full text
AutoIntervene: Calibrated Intervention for Action-Chunking Imitation Learning Policies Jinhe Tang 1 and Weiming Zhi 1,2,3,∗ 1 School of Computer Science and 2 Australian Center For Robotics, The University of Sydney, Australia 3 College of Connected Computing, Vanderbilt University, TN, USA ∗ Corresponding author: Weiming.Zhi@sydney.edu.au Project website: https://aus.bot/research/autointervene/ Abstract— Action-chunking visuomotor policies learn from demonstrations and improve temporal consistency by predicting short action sequences rather than single-step commands. Yet perception errors and execution drift can move the robot outside the demonstration distribution, while the policy con- tinues to produce smooth action chunks that are inconsistent with the observed state. We present AutoIntervene, an on- line framework that selectively transfers control between an action-chunking policy and an operator during deployment. AutoIntervene evaluates proposed chunks against a visual- action support memory built from successful task executions, combining visual similarity with consistency between proposed and reference actions. Phase-local support governs policy-to- operator transfer within the current task phase, whereas global support governs the return to policy control after operator recovery. We calibrate separate switching thresholds for the two directions from empirical quantiles of evaluation-level scores on held-out expert demonstrations, avoiding direct manual tuning of score cutoffs. Intervention segments retained from successful rollouts target learner-induced states and provide corrective supervision for subsequent policy updates. Experiments on real-world bimanual manipulation tasks show higher post- adaptation task success and lower operator-control time than manual intervention. Videos and additional results are available at https://aus.bot/research/autointervene/. I. INTRODUCTION Imitation learning provides a practical framework for learning robot policies from demonstrations [1], includ- ing stable motion generators that adapt to environmental changes [2], while offline human data has enabled complex multi-stage and long-horizon manipulation [3], [4]. Recent action-chunking policies improve temporal consistency by predicting short sequences of future actions rather than single-step commands, enabling visuomotor policies to exe- cute contact-rich manipulation tasks without hand-designed controllers [5]. However, these policies remain brittle un- der deployment-time distribution shift [6]. DART broadens demonstration coverage through noise injection but provides no deployment-time mechanism for detecting unsupported policy behaviour or requesting corrective control [7]. Small perception errors, missed contacts, or accumulated execu- tion error can move the robot into states that are poorly covered by the demonstration data. Once this occurs, an action-chunking policy can continue producing smooth ac- tion chunks while no longer making task progress, leading to failures such as misaligned grasps or incomplete subtask Fig. 1: AutoIntervene transfers control between the au- tonomous policy and the operator, thereby guiding the col- lection of targeted intervention data. We evaluate the system across diverse manipulation tasks. transitions [8], [9]. In this paper, we introduce AutoIntervene, an online in- tervention framework for improving the deployment relia- bility of action-chunking policies. AutoIntervene evaluates each proposed action chunk against successful visual-action support and transfers control when the proposal repeatedly becomes unsupported. It uses phase-local references to de- cide when control should pass from the policy to the operator and global references to decide when it should return to the policy, allowing autonomy to resume from any supported task phase after operator correction. At each adaptation round, separate switching thresholds are calibrated offline by recomputing proposals and support scores on held-out successful expert demonstrations under the corresponding retrieval scope. Operator-controlled segments from successful rollouts are retained for subsequent policy updates, yielding a selective DAgger-style loop [10], [11]. Unlike standard DAgger, which queries expert actions at learner-visited states throughout a rollout [6], AutoIntervene requests supervision only during automatically identified unsupported periods. This connects deployment-time monitoring with targeted data aggregation. Figure 1 illustrates representative executions, including ma- nipulation of deformable materials such as towels and fabric tote bags. arXiv:2608.07065v1 [cs.RO] 7 Aug 2026 The contributions of this work are threefold: 1) We introduce a bidirectional intervention framework for action-chunking imitation policies, using phase- local support for policy-to-operator transfer and global support for the return to policy control. 2) We develop a mode-specific visual-action calibration procedure that uses held-out successful expert demon- strations to recompute separate intervention and recov- ery thresholds at each adaptation round under their respective retrieval scopes. 3) We evaluate AutoIntervene on nine real-world tasks and show that targeted intervention trajectories support iterative policy improvement with substantially less operator-control time than collecting additional full demonstrations. I. RELATED WORK Visuomotor imitation policies. Visuomotor imitation policies differ in how they represent temporally extended actions. Action chunking predicts short action sequences di- rectly, whereas diffusion and flow-matching policies generate them through iterative or continuous-time processes [5], [12], [13]. Streaming flow trajectories can also be constrained post-training [14]. Although these approaches improve action generation, they do not by themselves determine when a de- ployed proposal has left demonstrated support or when con- trol should return after intervention. AutoIntervene addresses this deployment layer independently of the action head. We evaluate it with Action Chunking with Transformers (ACT), Diffusion Policy, and Flow Matching heads. Interactive imitation learning. Interactive policy adapta- tion traditionally relies on an operator to monitor policy exe- cution and manually decide when to take over and return con- trol. Robot-gated methods instead request intervention auto- matically from deployment-time signals. LazyDAgger uses policy–expert action discrepancy to trigger expert involve- ment, whereas RND-DAgger uses state novelty estimated by random network distillation [15], [16]. AutoIntervene follows this robot-gated direction but treats control transfer as a bidirectional, support-based process: phase-local support governs policy-to-operator transfer, global support governs operator-to-policy return, and retained recovery segments provide corrective data for subsequent policy adaptation. Runtime policy monitoring. Runtime monitors can be compared by the signal they inspect and the action taken after detection. Error-Aware Imitation Learning detected potential failures from teleoperation data [17] and Condi- tionNET learned action-conditioned preconditions and ef- fects [18]. Sentinel combined temporal action inconsistency with vision-language progress checks [8], whereas FAIL- Detect estimated failure uncertainty from successful data alone [19]. Trajectory-level out-of-distribution detection has also been formulated directly on SE(3) pose sequences using diffusion models [20]. PATCH conditions localized latent- patch innovation on the active action chunk to pause and resume policy execution under local scene disturbances [21]. Rewind-IL further coupled calibrated inter-chunk discrep- ancy with respawning at a semantically verified safe state [9]. These approaches respectively target trajectory-level out-of- distribution detection, localized transient disturbances, or checkpoint-based respawning. AutoIntervene instead uses retrieval-based visual-action support to govern both policy- to-operator and operator-to-policy transfer, then converts operator-controlled recovery into corrective training data. I. AUTOINTERVENE AutoIntervene is designed to improve a deployed action- chunking visuomotor policy by converting automatically triggered operator interventions into targeted supervision. As illustrated in Figure 2, it consists of two stages. The first stage, Intervention Loop, monitors the deployed pol- icy, transfers control to the operator when intervention is required, returns control to the policy when its proposed actions regain sufficient support, and records the result- ing intervention segments. The loop proceeds sequentially through three modules: (A) Visual-Action Query Con- struction, (B) Visual-Action Support Evaluation, and (C) Bidirectional Control Authority Selection. Together, these modules construct a visual-action query from the current visual observations and policy-proposed action chunk, map it to visual-support and action-risk scores, and select control authority using their calibrated acceptance criteria. The sec- ond stage, Policy Adaptation, uses the retained intervention segments to update the current policy. The updated policy is subsequently redeployed, and the intervention loop collects new intervention segments for the next adaptation round. Control setup. We implement the intervention loop using ALOHA-style leader–follower teleoperation with TriPilot- F-style arm-side force reflection [5], [22]. The leader arms serve as the operator input, while the follower arms in- teract with the task. Under policy control, the leader and follower arms receive the same policy-generated joint-and- gripper commands and remain aligned. When control passes to the operator, each follower arm switches to tracking the corresponding operator-manipulated leader arm, allowing correction to begin without repositioning the leader arms. A. Visual-Action Query Construction During deployment, visually similar observations may recur at task phases that require different actions [23]–[25]. Visual similarity alone is therefore insufficient to assess whether the policy’s proposed motion is appropriate. This motivates AutoIntervene to evaluate the current observation jointly with the action chunk that the policy is about to execute. At evaluation time t, the current policy π takes the current camera images as input. Its visual encoder maps these images to embeddings E t = (e (1) t ,...,e (C) t ), where C denotes the number of camera views. The policy’s action head predicts an H -step action chunk, and the monitor uses its first H r steps for evaluation, where 1 ≤ H r ≤ H . We denote this predicted prefix by A t . Fig. 2: Overview of AutoIntervene. During Stage I, the inter- vention loop sequentially (A) constructs a visual-action query from the current visual observations and policy-proposed action chunk, (B) evaluates visual-support and action-risk scores, and (C) selects control authority using calibrated acceptance criteria. The retained intervention segments are subsequently used to adapt the policy in Stage I, after which the updated policy is redeployed. For our bimanual system, we group the commands in A t by arm: one group contains the left-arm joint-and- gripper commands, and the other contains the corresponding right-arm commands. Together, the visual embeddings and predicted action prefix form the visual-action query Q t = (E t ,A t ). B. Visual-Action Support Evaluation Given the visual-action query Q t , AutoIntervene uses a visual-action memory M constructed from behaviour ref- erences and summarises their high-dimensional comparison into two scalar quantities: a visual-support score and an action-risk score. Visual-action memory and mode-specific retrieval. The visual-action memory is constructed from all trajectories used to train the current policy π. Suppose that there are N such trajectories, collected in D = τ i N i=1 , where τ i is the i-th trajectory in the collection. Each trajectory in D is used to generate the corresponding visual-action memory entries. For any trajectory τ i ∈ D, its length is |τ i | recorded timesteps, and it contributes |τ i |− H r + 1 memory entries. These entries correspond to all starting timesteps from which a complete H r -step action chunk can be extracted. For any such entry, let u ∈ 1,...,|τ i | − H r + 1 be its starting timestep. The H r recorded actions from timestep u through timestep u + H r − 1 form the action chunk A i,u = (a i,u ,...,a i,u+H r −1 ), while E i,u is the multi- view visual embedding of trajectory τ i at the same starting timestep. Together, A i,u and E i,u form the memory entry m i,u = (E i,u ,A i,u ). Collecting the entries generated from all valid starting timesteps across all trajectories yields the complete visual-action memoryM, as illustrated in Figure 3. Once the visual-action memory M has been constructed, AutoIntervene uses it at a mode-specific evaluation frequency during deployment to continually assess the current visual- action query. At each evaluation time t, either the policy or the operator controls the bimanual system, so AutoIntervene determines the memory entries used for the current assess- ment according to the active control mode. Let β ∈pol, op denote this mode, where β = pol indicates that the policy controls the bimanual system and β = op indicates that the operator controls it. The corresponding retrieval set is denoted by R β,t ⊆M. When β = op, the operator controls the bimanual system while the policy continues to generate predictions in the background. The operator’s recovery may change the object state or complete part of the task, so the current state need not remain near the task phase at which the intervention began. To allow the policy to regain support from any valid task phase reached after recovery, AutoIntervene retrieves from the complete visual-action memory. We refer to support evaluated under this complete-memory retrieval scope as global support. When β = pol, the policy controls the bimanual system and task progress is locally continuous across successive evaluations. However, memory entries from different task phases may contain similar visual embeddings but different recorded action chunks. Repeatedly searching the complete memory could therefore introduce a match from the wrong phase [24]. Whenever the policy begins controlling the bimanual system, its first evaluation compares the current vi- sual embedding E t with the visual embedding E i,u contained in every memory entry m i,u ∈M. AutoIntervene retains the best-matching memory entry on each training trajectory τ i and then selects the J trajectories whose retained entries have the strongest matches. Each selected trajectory contributes a forward window containing its matched entry and at most the next B− 1 entries. Subsequent evaluations retrieve only within these windows. After every U win executed policy actions, each window advances along its trajectory according to the newly matched entry without returning to an earlier task phase. Let W j,t denote the window from the j-th selected trajectory at evaluation time t, where j = 1,...,J . Thus, the mode-specific retrieval set is defined as below. We refer to support evaluated within these forward windows as phase-local support. R β,t = ( M,β = op, W 1,t ∪·∪W J,t , β = pol. For notational simplicity, throughout visual-neighbour re- trieval and action-risk scoring, we write a generic memory entry as m = (E,A) and suppress its trajectory and timestep indices. Visual-neighbour retrieval. For a reference entry m ∈ R β,t , we take its multi-view visual component E = (e (1) ,...,e (C) ) and compare it view by view with the current query embedding E t = (e (1) t ,...,e (C) t ). Their minimum cross-view visual similarity is s t (m) = min 1≤c≤C sim e (c) t ,e (c) ,(1) In (1), sim is cosine similarity between ℓ 2 -normalised em- beddings. Taking the minimum prevents a high similarity in one view from masking a mismatch in another, so every view Fig. 3: Memory construction and mode-specific retrieval. R op,t uses all entries, whereas R pol,t combines forward B- entry windows from J selected trajectories. must support the match. From R β,t , AutoIntervene retains up to K entries with the largest similarities as N β,t . Action-risk scoring. Given N β,t , we compare the proposed action chunk A t with the stored action component A of each entry m ∈ N β,t . The proposal is the chunk awaiting execution under policy control and the background pol- icy prediction under operator control. Following the action grouping defined above, the normalised distance between these chunks for each group g is given below, where the superscript (g) selects that group’s action dimensions: d t,g (m) = A (g) t − A (g) σ g 2 ,(2) In (2), σ g contains the per-dimension standard deviations of group g’s recorded actions in M, and division is applied element-wise. The Euclidean distance is computed over all retained steps and action dimensions. For each action group g, let the action-selected set P g contain the M entries in N β,t with the smallest distances. Aggregating these references reduces sensitivity to a acci- dental match while limiting interference from less action- relevant visual neighbours. Let g ⋆ be the action group whose selected entries have the largest mean distance, so that risk is governed by the least-supported group rather than masked by better-matched groups. Using the same references, the mode-dependent action risk and visual support are r β,t = 1 M X m∈P g ⋆ d t,g ⋆ (m), s β,t = 1 M X m∈P g ⋆ s t (m).(3) As defined in (3), both scores use the same action-selected references. Finally, ̄r β,t averages up to the W most recent action- risk values in the current control interval, using all available values before W evaluations. Visual support remains instan- taneous so that earlier matches do not mask a sudden loss of visual correspondence. The stage returns the score pair s β,t , ̄r β,t and, under policy control, the selected references P g ⋆ for retrieval-window updates. Policy-side retrieval-window update. After every U win ex- ecuted policy actions, the selected referencesP g ⋆ update the phase-local retrieval windows. For the j-th selected trajectory τ i j , let u j,t denote its current window position. We update this position to the largest value among its current position and the starting-timestep indices of the selected references from the same trajectory: u j,t+1 = max u j,t ∪u| m i j ,u ∈P g ⋆ .(4) The update in (4) is nondecreasing, so each trajectory win- dow moves forward or remains at its current position. C. Bidirectional Control Authority Selection The control-authority selector receives the score pair s β,t , ̄r β,t for the current control mode β ∈ pol, op and decides whether to retain or switch control after evaluation t. This decision requires separate visual-support and action-risk thresholds for the two control modes. Let D cal be a fixed held-out set of successful expert trajectories excluded from policy training and the visual- action memory. Before deploying the policy in each adap- tation round, we apply the same support evaluation to D cal separately under policy and operator control. This produces the visual-support collection S β cal and action-risk collection R β cal for mode β. Let α β s and α β r denote the prescribed lower- and upper- tail rates, respectively. With b Q p denoting the empirical p- quantile, the corresponding thresholds are θ β s = b Q α β s S β cal ,θ β r = b Q 1−α β r R β cal .(5) In (5), θ β s is the lower visual-support threshold and θ β r is the upper action-risk threshold. The two modes are calibrated separately because their retrieval scopes produce different score distributions. During deployment, a policy proposal is accepted when s β,t ≥ θ β s and ̄r β,t ≤ θ β r , and is rejected otherwise. Under policy control, a rejection, an empty phase-local reference set, or all phase-local retrieval windows reaching their final valid chunks increments the policy-side rejection counter c pol,t . Acceptance resets it. Under operator control, the policy is evaluated in the background and c op,t counts consecutive acceptances. A rejected proposal resets it. Given the required persistence lengths L pol and L op , the authority update β + is β + = op, β = pol and c pol,t ≥ L pol , pol, β = op and c op,t ≥ L op , β,otherwise. (6) The persistence lengths in (6) require the corresponding decision to remain consistent across successive evaluations, preventing brief score fluctuations from causing unnecessary control changes. D. Policy Adaptation In adaptation round k, the Intervention Loop deploys policy π (k) , trained on the accumulated trajectory collection D (k) . Each interval during which the operator controls the system is retained as an intervention segment and stored as a separate intervention trajectory. The segment begins when control switches from the policy to the operator and ends when control returns to the policy or the episode terminates. The intervention trajectories retained from successful rollouts form ∆D (k) . Keeping them separate prevents training exam- ples from crossing a change in control source and focuses supervision on states that required operator correction [6], [10]. At the end of round k, the new intervention data are added to the current training set. For any trajectory collectionD, let p D denote uniform sampling over its valid H -step training examples. The next training set and adaptation sampling distribution are D (k+1) =D (k) ∪ ∆D (k) , p (k+1) = λ mix p D (k) + (1− λ mix )p ∆D (k) ,0≤ λ mix ≤ 1,(7) The mixture in (7) retains prior training data while assigning the new intervention data a fixed sampling weight [26], [27]. The next policy π (k+1) is trained by standard behaviour cloning on examples sampled from p (k+1) . The visual-action memory is then rebuilt fromD (k+1) using the visual encoder of π (k+1) . Before redeployment, the mode-specific thresh- olds are recalibrated on D cal , after which π (k+1) enters the next Intervention Loop. Repeating this cycle progressively incorporates intervention data from unsupported task regions encountered during deployment. IV. EMPIRICAL EVALUATION We evaluate AutoIntervene on nine real-world bimanual manipulation tasks, comprising a seven-task main bench- mark and two longer-horizon extensions. The experiments are organized around four questions: Q1: Does targeted intervention improve policy success relative to the initial policy and additional full demonstrations? Q2: Does auto- matic switching achieve a better success–control-data trade- off than manual switching and collecting additional full demonstrations? Q3: Can AutoIntervene support iterative adaptation on long-horizon tasks and remain compatible with different action-generation heads? Q4: How reliably does AutoIntervene perform bidirectional handoff relative to prior monitors, and which components contribute to policy-to- operator and operator-to-policy switching? Experimental setup. All experiments used two AgileX PiPER-X 6-DoF arms with stock parallel grippers and three RGB cameras: one overhead and one on each wrist. Policy observations comprised a 28-dimensional bimanual state, including 14 joint-and-gripper values and their correspond- ing 14 measured-torque values, while actions were 14- dimensional joint-and-gripper commands. For the Diffusion Policy and Flow Matching variants, we retained the same TABLE I: Definitions of the evaluated tasks. TaskDescription Peg DisassemblyDisassemble a pegged object and place all parts into the box. Potato TransferTransfer a wooden potato between hands and place it into the box. Towel FoldingFold a towel from a fixed initial configuration. Towel BaggingFold a towel, pack it and the pegs into a bag, and move the bag to the target side. Lidded Box PackingOpen the box, pack the objects inside, and close the lid. Plant SortingPlace three wooden plant objects into their matching box slots. Towel Box PackingRemove pegs, fold the towel, reposition the box, and place the towel inside. Two-Towel Box Pack- ing Remove pegs, fold two towels, and place both into the box. Towels-and-Cable Bagging Fold two towels, pick up the cable, and place all three items into the bag. robot, observation, and deployment interfaces while replac- ing only the action-generation head. Our fixed-base setup omitted the pedal and mobile-base channels of TriPilot-F. We instantiated the policy’s visual encoder with DINOv3 ConvNeXt-Base [28]. We used two action groups corre- sponding to the left and right arms. Data and evaluation protocol. For each task, 30 of the 36 initial trajectories formed D (0) and the remaining six formed the fixed D cal . Subsequent intervention trajectories augmented both the training set and visual-action memory but not the fixed calibration set. For both Human and AutoIntervene, each adaptation round retained intervention trajectories from five successful deployment rollouts per task. Unless stated otherwise, every task-method success rate was measured over 25 unassisted physical rollouts. Success required completing every subtask in Table I. The main benchmark comprised seven bimanual manipulation tasks, while two longer-horizon extensions evaluated iterative adaptation. Table I defines all nine tasks. Figures 4 and 6 show representative executions. Implementation details. Policies used 256× 256 images and an action-chunk horizon of H = 100 future actions. All policy variants used the same encoder–decoder back- bone, consisting of one encoder layer and four decoder layers, while ACT, Diffusion Policy, and Flow Matching differed only in their action-generation mechanisms. For adaptation, accumulated pre-round training data and newly collected round-k intervention data were sampled in a 2:1 ratio (λ mix = 2/3). The monitor used J = K = 16, B = H r = 40, M = 3, and W = 5. After a 1 s warm-up, under policy control, policy actions were executed at 30 Hz, whereas AutoIntervene evaluation was performed at 5 Hz. The phase-local retrieval windows were therefore updated every U win = 6 executed policy actions. Under operator control, teleoperation commands were executed at 200 Hz, whereas AutoIntervene evaluation was performed at 30 Hz. Both directions required two consecutive decisions (L pol = L op = 2). Each round recomputed its thresholds with (α pol s ,α pol r ) = (0.05, 0.05) and (α op s ,α op r ) = (0.30, 0.30). TABLE I: Success (%) and recorded control-data time across seven tasks. Human and AutoIntervene use manually and automatically triggered interventions, respectively. R1 and R2 denote adaptation rounds. ∆t is cumulative additional operator- control time, while Initial and Additional Full Data report total demonstration time. Task InitialHuman R1Human R2AUTOINTERVENE R1 AUTOINTERVENE R2 Additional Full Data Succ. (%) Time (s) Succ. (%) ∆t (s) Succ. (%) ∆t (s) Succ. (%)∆t (s)Succ. (%)∆t (s)Succ. (%) Time (s) Peg Disassembly16%1173.752%123.656%286.964%49.372%127.160%1853.0 Potato Transfer44%368.752%20.780%74.248%29.676%50.264%547.3 Towel Folding56%766.260%36.4100%144.876%61.496%140.468%1100.0 Towel Bagging24%1761.132%130.236%266.544%121.660%171.040%2536.7 Lidded Box Packing8%989.620%107.592%246.752%63.1100%130.260%1440.0 Plant Sorting24%450.252%88.856%157.152%76.672%128.932%693.1 Towel Box Packing44%1332.452%36.460%83.380%61.484%112.368%1930.5 Average30.9%977.445.7%77.768.6%179.959.4%66.180.0%122.956.0%1442.9 Fig. 4: Policies adapted with AutoIntervene handle deformable objects in the Towels-and-Cable Bagging task, including cloth, cables, and a tote bag. InitialHuman R2 Auto- Intervene R2 Additional Full Data 800 1000 1200 1400 Collection time (s) 977.4s 1157.3s 1100.3s 1442.9s (a) Average data-collection time InitialHuman R2 Auto- Intervene R2 Additional Full Data 0 25 50 75 100 30.9% 68.6% 80.0% 56.0% (b) Average success rate Fig. 5: Across-task mean success and total recorded control- data time after R2, including the shared initial demonstra- tions. Baseline monitor settings. For the controlled comparison, we implemented deployment-time handoff monitors adapted from LazyDAgger [15] and RND-DAgger [16] to the ACT deployment setting, while following their original switching criteria and reported threshold-setting procedures. For RND- DAgger, we followed its reported threshold rule, set the threshold to twice the mean training-set RND score, and tested W rec ∈5, 30. Compared methods. The main benchmark compares four settings. Initial trains only on the 30 original training demonstrations. Additional Full Data adds ten full expert trajectories per task from the nominal initial-state distribu- tion. Human uses the same intervention-learning pipeline as our method, but an operator manually selects policy- to-operator and operator-to-policy switches. Its retained in- tervention segments are mixed with previous training data as separate intervention trajectories. AUTOINTERVENE uses the proposed visual-action support evaluation to trigger both switching directions automatically. For both intervention- based methods, R1 and R2 denote successive adaptation rounds. Metrics. The primary metric is task success rate. To measure data efficiency, we also report recorded control-data time. Table I reports demonstration time for Initial and Additional Full Data and additional operator-control time for Human and AUTOINTERVENE. For all methods, Figure 5 reports total time by adding the shared initial demonstration time to the intervention methods. A. Targeted Intervention Improves Policy Adaptation (Q1) Across the seven-task benchmark in Table I, targeted intervention consistently improves policy adaptation: after two rounds, both Human and AUTOINTERVENE outperform the Initial policy and Additional Full Data in aggregate. AUTOINTERVENE delivers the strongest improvement, in- creasing the across-task mean success rate by 49.1 per- centage points, with gains across all seven tasks. These results support AUTOINTERVENE as an effective approach for collecting targeted corrective supervision from learner- induced failure states. B. Automatic Switching Enables More Efficient Policy Adap- tation (Q2) As shown in Table I and Figure 5, AUTOINTERVENE achieves higher average success than both Human and Ad- ditional Full Data, while using approximately 74% less additional recorded control-data time than Additional Full Data. Relative to manual switching, AUTOINTERVENE re- turns control to the policy once the recovered state and policy action return to demonstrated support, keeping each retained intervention segment focused on the failure and recovery. A manually selected operator-to-policy switch depends on operator judgment and button timing, so the segment can extend beyond recovery into states that the policy already handles. Training on this extra tail dilutes the corrective Fig. 6: Policies adapted with AutoIntervene perform precise manipulation in the Plant Sorting task, including insertion and handovers. Fig. 6: Policies adapted with AutoIntervene perform precise manipulation in the Plant Sorting task, including insertion and handovers. 1 Policy → Operator (at 8.9s) 2 Operator → Policy (at 15.9s) 3 Policy → Operator (at 24.1s) 4 Operator → Policy (at 27.0s) Policy Rollout and Intervention Timeline 08.915.924.127.045 Runtime time (s)Intervention Policy Fig. 7: One AutoIntervene rollout on Peg Disassembly containing two policy–operator–policy intervention cycles. Top: the four control switches. Bottom: the corresponding timeline, with policy control in grey and operator control in red. centage points, with gains across all seven tasks. These results support AUTOINTERVENE as an effective approach for collecting targeted corrective supervision from learner- induced failure states. B. Automatic Switching Enables More Efficient Policy Adap- tation (Q2) As shown in Table I and Figure 5, AUTOINTERVENE achieves higher average success than both Human and Ad- ditional Full Data, while using approximately 74% less additional recorded control-data time than Additional Full Data. Relative to manual switching, AUTOINTERVENE re- turns control to the policy once the recovered state and policy action return to demonstrated support, keeping each retained intervention segment focused on the failure and recovery. A manually selected operator-to-policy switch depends on operator judgment and button timing, so the segment can extend beyond recovery into states that the policy already handles. Training on this extra tail dilutes the corrective update by adding supervision in already-supported regions. Additional Full Data is also less targeted because it spends collection time on complete nominal trajectories rather than learner-induced failure states. Figure 7 shows two policy–operator–policy cycles within a single Peg Disassembly trajectory, illustrating that AutoIn- tervene can collect multiple intervention segments from one rollout. TABLE I: Success (%) over three AutoIntervene adaptation rounds on two long-horizon tasks. Training Stage Two-Towel Box Packing Towels-and-Cable Bagging Initial Policy28%8% AutoIntervene R144%20% AutoIntervene R2 64%28% AutoIntervene R388%48% Additional Full Data52%28% TABLE IV: Success (%) over three AutoIntervene adaptation rounds on Two-Towel Box Packing with Diffusion Policy (DP), Flow Matching (FM), and ACT action heads. Training StageDPFMACT Initial Policy32%32%28% AutoIntervene R1 40%36%44% AutoIntervene R256%48%64% AutoIntervene R392%80%88% Additional Full Data44%40%52% C. Iterative Adaptation for Longer-Horizon Tasks (Q3) To evaluate AutoIntervene on longer task executions, we perform three adaptation rounds on two longer-horizon ma- nipulation tasks. As shown in Table I, success improves after each round on both tasks, and the final policies out- perform those trained with Additional Full Data. These results show that AutoIntervene is not limited to correcting a single local failure: by collecting intervention data from the problems encountered during each deployment and using it to update the policy, it continues improving performance over longer, more complex task executions. D. Adaptation across ACT, Diffusion, and Flow-Matching Heads (Q3) To test whether AutoIntervene transfers beyond ACT, we replace the ACT action head with Diffusion Policy and Flow Matching heads while keeping the shared backbone, robot interfaces, and monitor unchanged. As shown in Table IV, all three action heads improve over successive adaptation rounds, and their R3 policies outperform both the Initial policies and Additional Full Data. AutoIntervene therefore transfers across different action-generation mechanisms with- out head-specific modification. E. AutoIntervene Outperforms Prior Handoff Monitors through Complementary Components (Q4) Protocol and metrics. Handoff is evaluated during lid opening in Lidded Box Packing using the same 20,000-step policy checkpoint, physical configuration, and pre-specified monitor parameters. Each method is tested in 10 rollouts after a fixed 5 cm box translation and 10 nominal rollouts, with completion, 60 s, or five handoff cycles terminating a rollout. The perturbed condition tests whether the monitor transfers control to the operator after a policy failure and returns it to the policy after operator recovery. The nominal condition tests whether a successful autonomous execution remains uninterrupted when no intervention is needed. A valid cut-in occurs between displacement and 3 s after the resulting failed Fig. 7: One AutoIntervene rollout on Peg Disassembly containing two policy–operator–policy intervention cycles. Top: the four control switches. Bottom: the corresponding timeline, with policy control in grey and operator control in red. update by adding supervision in already-supported regions. Additional Full Data is also less targeted because it spends collection time on complete nominal trajectories rather than learner-induced failure states. Figure 7 shows two policy–operator–policy cycles within a single Peg Disassembly trajectory, illustrating that AutoIn- tervene can collect multiple intervention segments from one rollout. C. Iterative Adaptation for Longer-Horizon Tasks (Q3) To evaluate AutoIntervene on longer task executions, we perform three adaptation rounds on two longer-horizon ma- nipulation tasks. As shown in Table I, success improves after each round on both tasks, and the final policies out- perform those trained with Additional Full Data. These results show that AutoIntervene is not limited to correcting a single local failure: by collecting intervention data from the problems encountered during each deployment and using it to update the policy, it continues improving performance over longer, more complex task executions. D. Adaptation across ACT, Diffusion, and Flow-Matching Heads (Q3) To test whether AutoIntervene transfers beyond ACT, we replace the ACT action head with Diffusion Policy and Flow Matching heads while keeping the shared backbone, robot interfaces, and monitor unchanged. As shown in Table IV, all three action heads improve over successive adaptation TABLE I: Success (%) over three AutoIntervene adaptation rounds on two long-horizon tasks. Training Stage Two-Towel Box Packing Towels-and-Cable Bagging Initial Policy28%8% AutoIntervene R144%20% AutoIntervene R2 64%28% AutoIntervene R388%48% Additional Full Data52%28% TABLE IV: Success (%) over three AutoIntervene adaptation rounds on Two-Towel Box Packing with Diffusion Policy (DP), Flow Matching (FM), and ACT action heads. Training StageDPFMACT Initial Policy32%32%28% AutoIntervene R1 40%36%44% AutoIntervene R256%48%64% AutoIntervene R392%80%88% Additional Full Data44%40%52% rounds, and their R3 policies outperform both the Initial policies and Additional Full Data. AutoIntervene therefore transfers across different action-generation mechanisms with- out head-specific modification. E. AutoIntervene Outperforms Prior Handoff Monitors through Complementary Components (Q4) Protocol and metrics. Handoff is evaluated during lid opening in Lidded Box Packing using the same 20,000-step policy checkpoint, physical configuration, and pre-specified monitor parameters. Each method is tested in 10 rollouts after a fixed 5 cm box translation and 10 nominal rollouts, with completion, 60 s, or five handoff cycles terminating a rollout. The perturbed condition tests whether the monitor transfers control to the operator after a policy failure and returns it to the policy after operator recovery. The nominal condition tests whether a successful autonomous execution remains uninterrupted when no intervention is needed. A valid cut-in occurs between displacement and 3 s after the resulting failed grasp. A valid cut-out is the operator-to-policy switch after the operator restores the box and guides the gripper to the lid. For the four transition metrics on perturbed rollouts, let V d , M d , and E d denote the numbers of valid transitions, missed opportunities, and extra transitions for d∈cut-in, cut-out. We compute Recall d = V d V d + M d ,Precision d = V d V d + E d .(8) For the metrics in (8), a zero denominator is reported as N/A. A valid cut-out must follow a valid cut-in. Consequently, cut-out recall is conditioned on valid cut-ins, whereas cut-out precision considers all attempted operator-to-policy switches. The false-trigger rate is the fraction of nominal rollouts containing at least one unnecessary cut-in, with each rollout counted at most once regardless of repeated triggers. The comparison includes LazyDAgger [15] and RND- DAgger [16], with the two RND-DAgger settings differing only in recovery persistence W rec . The ablations preserve re- TABLE V: Controlled handoff comparison on Lidded Box Packing using 10 perturbed and 10 nominal rollouts per method, capped at five complete handoff cycles. Values are rates. N/A indicates an undefined denominator. PerturbedNominal Method Cut-in Recall ↑ Cut-in Prec. ↑ Cut-out Recall ↑ Cut-out Prec. ↑ False-trigger Rate ↓ AutoIntervene1.001.001.001.000.00 Prior handoff monitors LazyDAgger [15]0.401.000.00N/A0.80 RND-DAgger (W rec = 5) [16]0.800.160.000.000.90 RND-DAgger (W rec = 30) [16]0.800.160.750.120.90 Ablations of visual support and action risk w/o visual support0.00N/AN/AN/A0.00 w/o action risk0.901.000.560.560.00 trieval and neighbour selection, removing only visual support or action risk from the acceptance criteria. Comparison with prior monitors. As shown in Table V, reliable bidirectional handoff without unnecessary policy- to-operator switches during nominal execution is achieved only by AutoIntervene. LazyDAgger misses failures and retains operator control because policy–operator disagree- ment remains large during recovery. RND-DAgger uses one novelty threshold for both directions. After an operator-to- policy switch, a score increase can cross the same threshold and immediately return control to the operator. Increasing recovery persistence only delays the return to policy control, whereas AutoIntervene uses separately calibrated, mode- specific switching criteria to implement hysteretic switching and suppress immediate reversal. Ablations of visual support and action risk. As shown in Table V, without visual support, AutoIntervene misses the workspace displacement because action risk can remain low despite weak visual correspondence. Without action risk, most policy-to-operator switches remain, but the return to policy control may be less reliable when visual correspon- dence recovers before the proposal matches successful refer- ence actions. Thus, visual support detects when intervention is needed and action risk prevents a premature operator-to- policy switch. Both are required for reliable bidirectional handoff. Ablation of policy-side retrieval-window updates. Beyond the score components, we evaluate policy-side retrieval- window updates by lowering the target object while pre- serving a similar RGB appearance, causing the nominal grasp to fail. Across 10 rollouts per setting, AutoIntervene with retrieval-window updates achieves a cut-in recall of 1.00. During repeated-grasp failures, the updates advance the phase-local windows until they all reach their final valid chunks, incrementing the policy-side rejection counter until control transfers to the operator. Without these updates, the policy can repeat failed grasps without transferring control, yielding a cut-in recall of 0.30. V. CONCLUSION AutoIntervene combines phase-local visual-action moni- toring, bidirectional handoff, and intervention-data aggrega- tion. Across nine real-world bimanual tasks, it improved poli- cies over successive rounds with less additional control data than full demonstrations, achieved higher mean success than manual switching, and required less operator-control time on average. Together with separately calibrated criteria for both switching directions, these components focus operator control on unsupported periods and resume autonomy once the policy proposal returns to demonstrated support. Future work will extend online calibration across broader tasks, perturbations, and operators. REFERENCES [1] T. Osa, J. Pajarinen, G. Neumann, J. A. Bagnell, P. Abbeel, and J. Pe- ters, “An algorithmic perspective on imitation learning,” Foundations and Trends in Robotics, 2018. [2] W. Zhi, T. Lai, L. Ott, and F. Ramos, “Diffeomorphic transforms for generalised imitation learning,” in Proceedings of the 4th Annual Learning for Dynamics and Control Conference, ser. Proceedings of Machine Learning Research, vol. 168, 2022, p. 508–519. [3] A. Mandlekar et al., “What matters in learning from offline human demonstrations for robot manipulation,” in Proceedings of the 5th Conference on Robot Learning, 2022. [4] A. Mandlekar, D. Xu, R. Martín-Martín, S. Savarese, and L. Fei-Fei, “GTI: Learning to generalize across long-horizon tasks from human demonstrations,” in Robotics: Science and Systems, 2020. [5] T. Z. Zhao, V. Kumar, S. Levine, and C. Finn, “Learning fine-grained bimanual manipulation with low-cost hardware,” in Proceedings of Robotics: Science and Systems, 2023. [6] S. Ross, G. J. Gordon, and J. A. Bagnell, “A reduction of imitation learning and structured prediction to no-regret online learning,” in Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics, 2011. [7] M. Laskey, J. Lee, R. Fox, A. Dragan, and K. Goldberg, “DART: Noise injection for robust imitation learning,” in Proceedings of the 1st Annual Conference on Robot Learning, 2017. [8] C. Agia et al., “Unpacking failure modes of generative policies: Runtime monitoring of consistency and progress,” in Proceedings of the 8th Conference on Robot Learning, 2025. [9] G. Zheng, S. Seenivasan, M. Johnson-Roberson, and W. Zhi, “Rewind- IL: Online failure detection and state respawning for imitation learn- ing,” arXiv preprint arXiv:2604.16683, 2026. [10] M. Kelly, C. Sidrane, K. Driggs-Campbell, and M. J. Kochenderfer, “HG-DAgger: Interactive imitation learning with human experts,” in IEEE International Conference on Robotics and Automation, 2019. [11] Z. Hu et al., “RaC: Robot learning for long-horizon tasks by scaling recovery and correction,” arXiv preprint arXiv:2509.07953, 2025. [12] C. Chi et al., “Diffusion policy: Visuomotor policy learning via action diffusion,” in Proceedings of Robotics: Science and Systems, 2023. [13] Q. Zhang, Z. Liu, H. Fan, G. Liu, B. Zeng, and S. Liu, “FlowPolicy: Enabling fast and robust 3D flow-based policy via consistency flow matching for robot manipulation,” in AAAI Conference on Artificial Intelligence, 2025. [14] J. Long, D. Liu, W. Cai, I. Manchester, and W. Zhi, “Safe policies post-training: Constraining streaming flow models for adapting learned robot trajectory distributions,” IEEE Robotics and Automation Letters, vol. 11, no. 9, p. 10 656–10 663, 2026. [15] R. Hoque et al., “LazyDAgger: Reducing context switching in in- teractive imitation learning,” in IEEE International Conference on Automation Science and Engineering, 2021. [16] E. Biré, A. Kobanda, L. Denoyer, and R. Portelas, “Efficient active imitation learning with random network distillation,” in International Conference on Learning Representations, 2025. [17] J. Wong et al., “Error-aware imitation learning from teleoperation data for mobile manipulation,” in Proceedings of the 5th Conference on Robot Learning, 2022. [18] D. Sliwowski and D. Lee, “ConditionNET: Learning preconditions and effects for execution monitoring,” IEEE Robotics and Automation Letters, 2025. [19] C. Xu et al., “Can we detect failures without failure data? Uncertainty- Aware runtime failure detection for imitation learning policies,” in Proceedings of Robotics: Science and Systems, 2025. [20] H. Cheng, T. Zheng, T. Zhang, M. Johnson-Roberson, and W. Zhi, “DOSE3: Diffusion-based out-of-distribution detection on SE(3) tra- jectories,” arXiv preprint arXiv:2502.16725, 2025. [21] Y. Zhou, R. Qiu, Y. Chen, J. Cui, and W. Zhi, “PATCH: Action- chunk-conditioned latent patch innovation monitoring for robot ma- nipulation,” arXiv preprint arXiv:2606.16690, 2026. [22] Z. Li, Y. Zhou, R. Qiu, H. Wu, G. Ren, and W. Zhi, “TriPilot-F: Coordinated whole-body teleoperation with force feedback,” arXiv preprint arXiv:2602.09888, 2026. [23] P. Sermanet et al., “Time-contrastive networks: Self-supervised learn- ing from video,” in IEEE International Conference on Robotics and Automation, 2018. [24] Z. Ouyang, K. Wang, J. Liu, H. Lu, and W. Zhang, “SCIL: Stage- conditioned imitation learning for multi-stage manipulation,” IEEE Control Systems Letters, 2025. [25] Z. Li, R. Qiu, Y. Chen, G. Ren, and W. Zhi, “TRACE: Trajectory- routed causal memory for delayed-evidence visuomotor imitation,” arXiv preprint arXiv:2606.14551, 2026. [26] A. Mandlekar, D. Xu, R. Martín-Martín, Y. Zhu, L. Fei-Fei, and S. Savarese, “Human-in-the-loop imitation learning using remote tele- operation,” arXiv preprint arXiv:2012.06733, 2020. [27] D. Rolnick, A. Ahuja, J. Schwarz, T. P. Lillicrap, and G. Wayne, “Experience replay for continual learning,” in Advances in Neural Information Processing Systems, vol. 32, 2019. [28] O. Siméoni et al., “DINOv3,” Transactions on Machine Learning Research, 2026.