Paper deep dive
RE-0: Verified Recursive Improvement of Embodied Code-as-Policy Agents through Local On-Policy Distillation
Jiawei Zhang, Xiangrong Zhang, Rui Song, Huanbin Zhou, Chengye Song, Hongzhou Wang
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Code-as-Policy agents accomplish long-horizon embodied tasks by generating and executing code, yet continually improving them with teachers that are stronger but not globally reliable remains a key challenge. Existing distillation methods typically treat the teacher's complete behavior as the supervision target and thus misassign training credit on states where the teacher fails. We propose RE-0, a recursively verified policy improvement framework: rather than assuming that the teacher globally outperforms the student, RE-0 requests local corrections from the teacher on the student's own failure histories and checks in the environment whether each correction is genuinely beneficial; verified corrections yield immediate improvement. Building on this, we propose RE-OPD, which turns verified interventions into supervision for on-policy distillation. Only counterfactually verified teacher interventions provide distribution-level supervision, weighted by their measured local benefit, and the improvement they induce is projected back into the standalone student, so both where supervision is applied and how much credit the teacher receives co-evolve with the student policy. We further prove that the student's per-round gain is lower-bounded by its verified intervention gain up to verification and projection error terms. Experiments across multiple Code-as-Policy embodied tasks show that RE-0 improves both teacher-assisted execution and the standalone student, and generalizes to novel robots and scenes.
Tags
Links
- Source: https://arxiv.org/abs/2609.32416v1
- Canonical: https://arxiv.org/abs/2609.32416v1
Trouble viewing inline? Open PDF directly →
Full Text
66,937 characters extracted from source content.
Expand or collapse full text
RE-0: Verified Recursive Improvement of Embodied Code-as-Policy Agents through Local On-Policy Distillation Jiawei Zhang Affiliation: Jilin University, Changchun, China Xiangrong Zhang Affiliation: Jilin University, Changchun, China Rui Song Affiliation: Dalian University of Technology, Dalian, China Huanbin Zhou Affiliation: Jilin University, Changchun, China Chengye Song Affiliation: Dalian University of Technology, Dalian, China Hongzhou Wang Abstract Code-as-Policy agents accomplish long-horizon embodied tasks by generating and executing code, yet continually improving them with teachers that are stronger but not globally reliable remains a key challenge. Existing distillation methods typically treat the teacher’s complete behavior as the supervision target and thus misassign training credit on states where the teacher fails. We propose RE-0, a recursively verified policy improvement framework: rather than assuming that the teacher globally outperforms the student, RE-0 requests local corrections from the teacher on the student’s own failure histories and checks in the environment whether each correction is genuinely beneficial; verified corrections yield immediate improvement. Building on this, we propose RE-OPD, which turns verified interventions into supervision for on-policy distillation. Only counterfactually verified teacher interventions provide distribution-level supervision, weighted by their measured local benefit, and the improvement they induce is projected back into the standalone student—so both where supervision is applied and how much credit the teacher receives co-evolve with the student policy. We further prove that the student’s per-round gain is lower-bounded by its verified intervention gain up to verification and projection error terms. Experiments across multiple Code-as-Policy embodied tasks show that RE-0 improves both teacher-assisted execution and the standalone student, and generalizes to novel robots and scenes. †footnotetext: †Corresponding author: wanghongzhou@jlu.edu.cn 1 Introduction Code-as-Policy agents turn language models into policies that interact with an environment through executable programs—code as the interface for compositional control, tool calling, and long-horizon reasoning. Policy errors are then strongly history-dependent: one harmful segment not only fails the current action but alters the environment state, the program namespace, and the subsequent observable history. Improving such an agent therefore poses a question more basic than what code should be generated: where, along the execution history actually induced by the student, should corrective credit be assigned. Existing approaches improve code policies through expert-trajectory supervision, outcome-level reinforcement learning, or teacher distillation, but none answers whether one concrete teacher intervention on a real student failure produced a positive causal contribution. Even a strong teacher accumulates errors through perception ambiguity or earlier executions, while a teacher unable to finish the task alone may still decide better at a local failure boundary the student has reached; neither global success rate nor prefix-level reliability determines whether a teacher action deserves training credit. We therefore reformulate teacher supervision as local counterfactual credit on student-induced histories: on a history h genuinely visited by the current student, does a bounded teacher correction beat the student’s action, and does the advantage persist after control returns to the same student? Three coupled challenges arise: localizing the earliest plausibly causal harmful boundary; comparing correction and student from the identical environment state; and absorbing verified assistance into the student so deployment needs no teacher. We call this verifier-mediated local teacher credit assignment. Our framework, RE-0 (Figure 1), solves it recursively. Given a failed trajectory of the current student πk _k, a diagnostic agent localizes the earliest candidate harmful boundary and constructs repair contracts from the smallest scope; the teacher emits one bounded code macro-action within the contract, never taking over the remaining trajectory. RE-0 restores the environment to the snapshot saved at that boundary, executes the student baseline and the teacher correction under shared randomness, and admits the intervention only if the lower confidence bound of the paired advantage is positive; control then returns to the student, yielding a verified local assistant μk _k that intervenes only where the teacher has positive local credit. To convert this transient advantage into standalone capability, RE-OPD distills on the histories actually visited by μk _k: the current student is the retention term, the contract-constrained teacher the local target, and the distillation weight is the verified counterfactual advantage; the next student then regenerates trajectories and supervision locations are re-localized on the new occupancy, giving the recursion πk→μk→πk+1 _k\!→\! _k\!→\! _k+1 whose per-round gain is lower-bounded by the verified intervention gain up to verification and projection error, with no global teacher dominance required. Our ablations locate the load-bearing component precisely: it is the verified admission of patches (the corpus), not the specific weighting—uniform SFT on the verified corpus is within trainer-seed variance of the weighted objective, while unverified corpora fail (composition and selection analyses in §4). On six long-horizon tasks and a physical robot platform, RE-0 lifts the base from 4–68% to 62–100% with 3–67 verified rows (details in §4). Our contributions are: (1) verifier-mediated local teacher credit on student-induced histories; (2) RE-OPD, which recursively projects verified local improvement into a teacher-free student with a finite-sample improvement guarantee; and (3) empirical evidence across six long-horizon tasks and physical deployment that verified corpus selection—not weighting alone—is the load-bearing mechanism. 2 Related Work Code-as-Policy and embodied code agents. Large language models have been used to map natural-language tasks into embodied plans, callable skills, and executable robot programs: SayCan constrains planning with environment affordances, Inner Monologue folds environment feedback into closed-loop planning, and Code as Policies and ProgPrompt exploit code structure, control APIs, and program context to generate embodied policies (Ahn et al., 2022; Huang et al., 2022; Liang et al., 2023; Singh et al., 2023). Recent work repairs code policies from vision and execution traces, or searches and evolves high-return programs with rollout evaluators (Lin et al., 2025; Sygkounas et al., 2026). These methods focus on inference-time planning, program repair, or candidate search; RE-0 asks how corrections discovered during execution become verifiable teacher credit amortized into a teacher-free student. On-policy distillation and interactive imitation. Interactive imitation queries an expert on learner-induced states to mitigate covariate shift, with DAgger as the classic form (Ross et al., 2011); in autoregressive models, GKD distills on student-generated sequences and DistiLLM lowers its cost via adaptive off-policy reuse (Agarwal et al., 2024; Ko et al., 2024). For multi-turn agents, prefixes near the student occupancy can themselves make the teacher target unreliable, motivating reliability-aware replay approximations of OPD. RE-0 shares the goal of learning on the student distribution but never presumes teacher reliability on queried histories: it estimates the teacher’s local counterfactual advantage against student behavior from the same simulation snapshot and projects only environment-verified positive-credit interventions. Agentic correction, program repair, and verifiers. Self-Refine, Reflexion, and CRITIC revise generations with self-feedback, verbal memory, or external tools (Madaan et al., 2023; Shinn et al., 2023; Gou et al., 2024); in code generation, CodeRL scores programs from compilation and unit tests (Le et al., 2022). Such methods typically regenerate whole answers, select among candidates at the outcome level, or feed execution traces back as prompts. RE-0 instead localizes the earliest harmful code boundary in the student’s trajectory, compares the student baseline and the teacher correction from the identical environment snapshot, and confines teacher control to a minimal verified macro-action after which the student regenerates the suffix; the verifier thus decides not only whether a candidate succeeds but whether the teacher’s distribution earns local training credit. Selective supervision and local credit assignment. Prior work selectively queries experts or weights demonstrations by uncertainty or quality (Zhang & Cho, 2017; Kelly et al., 2019; Zhang et al., 2021). RE-0 instead defines the credit object itself: the teacher’s conditional behavior on a specific student history, evaluated by paired same-state counterfactual advantage. 3 RE-0: Recursive Verified Policy Improvement Figure 1: RE-0 overview. RE-0 localizes harmful segments in the current student’s trajectories, requests bounded teacher repairs, and verifies them by paired counterfactual rollout from the same snapshot (LCB gate). Verified interventions μn _n are distilled into the standalone student via cold-start SFT (Stage I) and RE-OPD refinement (Stage I); the student then regenerates trajectories, closing the loop. Left: simulation, real-robot, and stage-level results. Learning from a locally better but globally imperfect teacher. Standard on-policy distillation matches the teacher’s conditional distribution on histories visited by the student. RE-0 assumes no global policy dominance between student and teacher: the teacher holds positive local counterfactual advantage only on some student-induced histories, and supervision credit is assigned not to teacher trajectories as wholes but to the marginal improvement an intervention produces over the student’s own behavior (Eq. 1). The correction δ takes over only a minimal executable segment and immediately returns control to the current student πk _k. A diagnostic agent localizes failure boundaries that may carry positive credit, an auditable counterfactual executor verifies the credit is real, and RE-OPD amortizes the verified improvement into the next independent student—verifier-mediated local credit assignment. 3.1 Local Teacher Credit on Student-Induced Histories With J(π)J(π) the expected discounted return, the policy-visible history at the t-th code decision point is ht=(o0,c0,…,ot)h_t=(o_0,c_0,…,o_t), bounded code segments acting as macro-actions, dπktd_ _k^t the step-t history distribution of the generation-k student, and stsnaps_t^snap the verifier’s policy-invisible snapshot. RE-0 asks whether the teacher decides better locally on histories actually induced by the current student: for the teacher’s local correction policy τk(δ∣x,ht) _k(δ x,h_t), whose corrections return control to πk _k after executing, the expected local credit is (formal definitions in Appendix B) Δk(ht)=δ∼τk(⋅∣x,ht)[Atπk(ht,δ)]. _k(h_t)=E_δ _k(· x,h_t) [A_t _k(h_t,δ) ]. (1) Equation 1 captures the counterfactual marginal contribution of teacher intervention relative to the current student. If Δk(ht)>0 _k(h_t)>0, the teacher’s local policy is better than continuing the student at hth_t; if Δk(ht)≤0 _k(h_t)≤ 0, the teacher’s supervision on that history must not be accepted automatically, regardless of its overall strength. The ideal allocation—supervision only on ℋk+=(t,ht):Δk(ht)>0H_k^+=\(t,h_t): _k(h_t)>0\ under budget B, a fractional-knapsack LP (appendix)—is not directly observable: Δk(h) _k(h) is hidden and the history space forbids exhaustive verification. RE-0 therefore uses diagnostics to localize executable failure boundaries that may carry positive local credit, then decides by counterfactual execution from identical states. Figure 2: One recursion on a failing two-arm lift program. Locate: the earliest harmful boundary t(failure). Verify: from the same snapshot, student baseline vs. teacher patch under shared randomness; admitted iff Δ¯>0 >0. Distill: verified repairs are credit-weighted into the standalone student; deployment uses no teacher calls. 3.2 Verified Local Assistance Equation 1 defines the teacher’s true local credit, which cannot be observed directly. RE-0 approximates this ideal allocation with a two-stage mechanism: a diagnostic agent localizes the executable failure boundary that may carry positive local credit, and an environment verifier decides through same-state counterfactual execution whether the intervention yields a real gain. The former shrinks the candidate space; the latter decides whether teacher supervision enters the hybrid policy. Given a student-executed trajectory ζ, the student diagnostic agent outputs the earliest candidate failure history with structured evidence, (h^,z)( h,z): z collects runtime exceptions, unrealized expected effects, or other structured failure evidence, and the execution system saves the full simulation snapshot sh^snaps_ h^snap; if no executable, causally consistent boundary is found, no teacher call fires. A teacher diagnostic then judges whether the failure is locally repairable and emits repair contracts of monotonically increasing scope cost κ (formally, ((h^,z),gkdiag(h^))=kT(x,h^,z)( X( h,z),\,g_k^diag( h))=A_k^T(x, h,z); definitions in the appendix). Each contract constrains the repair boundary, the allowed program symbols and APIs, and the maximum segment length. RE-0 verifies contracts in increasing κ(χ)κ(χ) and selects the smallest verified repair scope—minimal necessary repair means the smallest verified scope, not the shortest candidate string (selection rule and validity predicate in the appendix). If gkdiag(h^)=0g_k^diag( h)=0 or no contract qualifies, no intervention fires; we write the selected contract teacher distribution as τ¯kχk⋆ τ_k _k . For each teacher candidate, RE-0 restores the environment to sh^snaps_ h^snap. Under shared subsequent randomness ξ1,…,ξM _1,…, _M, the teacher correction and a student resample cmS∼πk(⋅∣x,h^)c_m^S _k(· x, h) are executed and control returns to the same student πk _k after the segment completes; the paired counterfactual advantage estimate A^k(h^,δ) A_k( h,δ) is the M-average of the return difference (δ→πk)(δ\!→\! _k) minus (cmS→πk)(c_m^S\!→\! _k). RE-0 verifies the expected credit of the contract teacher distribution: averaging K candidates δj∼i.i.d.τ¯k(⋅∣x,h^) _j i.i.d. τ_k(· x, h) with a two-layer confidence radius controlling both teacher-candidate and rollout sampling error, Δ^k(h^)=1K∑jA^k(h^,δj),Δ¯k(h^)=Δ^k(h^)−bk(h^,K,M,α). _k( h)= 1K _j A_k( h, _j), _k( h)= _k( h)-b_k( h;K,M,α). (2) When verifying multiple histories we allocate αk,h _k,h with ∑k,hαk,h≤α _k,h _k,h≤α for simultaneous coverage over the whole recursion; concentration bounds appear in the appendix. The environment gate is gkenv(h^)=[Δ¯k(h^)>ϵ]⋅[Valid(χk⋆,τ¯k,sh^snap)]g_k^env( h)=1[ _k( h)>ε]·1[Valid( _k , τ_k,s_ h^snap)], with ϵ≥0ε≥ 0 the minimum advantage threshold and ValidValid the execution-safety, program-context, and snapshot-identity predicate (appendix); candidate returns never post-select a single best candidate. Combining the budget allocation qkq_k (appendix), the final intervention probability is gk(h)=qk(h)gkdiag(h)gkenv(h)g_k(h)=q_k(h)\,g_k^diag(h)\,g_k^env(h), giving the verified local hybrid policy μk(c∣x,h)=(1−gk(h))πk(c∣x,h)+gk(h)τ¯k(c∣x,h). _k(c x,h)= (1-g_k(h) )\, _k(c x,h)+g_k(h)\, τ_k(c x,h). (3) From here on, the teacher policy in Eq. 1 is taken to be the same contract distribution τ¯k τ_k. The teacher takes over only the current bounded repair segment; after it executes, the student regenerates subsequent code from the new observation rather than executing a suffix generated by the old policy. Remark (rewind semantics). The pipeline runs two passes—a live attempt that localizes candidate boundaries, then verification from the stored snapshot—but both arms of every paired comparison start from the same snapshot, so the first pass is a common data-collection cost that cancels in Eq. 4; μk _k is equivalently a one-pass policy of the augmented process whose state includes the snapshot stack and the first-pass diagnostic evidence, and the bound applies to the post-rewind continuation return of that process, not to a snapshot-free online policy (appendix). By the history–state performance-difference lemma, J(μk)−J(πk)=∑tγth∼dμkt[gk(h)Δk(h)]J( _k)-J( _k)= _tγ^tE_h d_ _k^t[g_k(h) _k(h)]. Define the verifiable intervention gain ^k=∑tγth∼dμkt[gk(h)[Δ^k(h)]+] G_k= _tγ^tE_h d_ _k^t[g_k(h)[ _k(h)]_+] and the verification error budget ℰverify,k=∑tγth∼dμkt[gk(h)bk(h)]+ℰsys,kE_verify,k= _tγ^tE_h d_ _k^t[g_k(h)b_k(h)]+E_sys,k, where ℰsys,kE_sys,k collects system approximation errors such as snapshot restoration and execution identity. Whenever returns are bounded, contract supports are valid, and the simultaneous-coverage condition of Eq. 2 holds, with probability at least 1−α1-α, J(μk)−J(πk)≥^k−ℰverify,k.J( _k)-J( _k)\;≥\; G_k-E_verify,k. (4) Thus RE-0 converts verified positive local credit into expected performance gain of the hybrid policy without requiring global teacher dominance. 3.3 RE-OPD: Amortizing Local Credit into the Student The verified local hybrid μk _k obtains higher expected return through local teacher interventions, but executing it still requires teacher calls and environment verification. RE-OPD amortizes these transient local advantages into student parameters, so that πk+1 _k+1 reproduces the improved behavior without teacher access. Recall the hybrid of Eq. 3. The ideal projection finds the independent student closest to μk _k on the histories μk _k actually visits, πk+1=argminπθ∑tγth∼dμkt[DKL(μk(⋅∣x,h)∥πθ(⋅∣x,h))] _k+1= _ _θ _tγ^tE_h d_ _k^t[D_KL( _k(· x,h)\| _θ(· x,h))]. Training histories cover both the failure states the student would visit and the new states induced by the student after teacher correction—not fixed teacher trajectories, nor mechanically stitched student suffixes. Because μk _k is itself the gkg_k-mixture of πk _k and τ¯k τ_k (Eq. 3), substituting into the KL and dropping parameter-independent entropy terms turns the ideal projection exactly into a gkg_k-weighted sum of a student-retention term and a teacher-projection term; RE-OPD re-weights teacher supervision by verified local credit: ℒRE-OPD(θ)=∑tγth∼dμkt[(1−gk(h))DKL(πk∥πθ)+λgk(h)wk(h)DKL(τ¯k∥πθ)],L_RE -OPD(θ)= _tγ^t\,E_h d_ _k^t [(1-g_k(h))\,D_KL ( _k\,\|\, _θ )+λ\,g_k(h)\,w_k(h)\,D_KL ( τ_k\,\|\, _θ ) ], (5) with all conditional distributions conditioned on (x,h)(x,h), λ controlling local distillation strength, and the local credit weight wk(h)=[Δ¯k(h)]+w_k(h)=[ _k(h)]_+. Equation 5 is a credit-weighted surrogate inspired by the exact mixture projection, not a strict upper bound of the ideal projection for arbitrary λwk(h)λ w_k(h); through gk(h)wk(h)g_k(h)w_k(h) it decides how much teacher supervision earns training credit where. Restricted estimators implement the same projection (derivation in the appendix): Top-K token probabilities give a truncated sparse estimate, and a black-box teacher degenerates to the trajectory-level hard estimator ℓhard(h,δ)=−λgk(h)wk(h)logπθ(δ∣x,h) _hard(h,δ)=-λ g_k(h)w_k(h) _θ(δ x,h) used throughout our experiments. Optimizing Eq. 5 controls the occupancy-weighted projection error εproj,k _proj,k (definition in the appendix), for which [J(μk)−J(πk+1)]+≤CT,γ,Rεproj,k[J( _k)-J( _k+1)]_+≤ C_T,γ,R _proj,k with CT,γ,RC_T,γ,R depending only on horizon, discount, and return bound (proof in the appendix); combining this with Eq. 4 yields the core result. Theorem 1 (Recursive verified policy improvement). If returns are bounded, contracts are valid, the counterfactual verifier satisfies the simultaneous-coverage condition of Eq. 2, and the occupancy-weighted projection error of RE-OPD is bounded, then with probability at least 1−α1-α, J(πk+1)−J(πk)≥^k−ℰverify,k−CT,γ,Rεproj,k.J( _k+1)-J( _k)\;≥\; G_k-E_verify,k-C_T,γ,R _proj,k. (6) Hence J(πk+1)>J(πk)J( _k+1)>J( _k) whenever ^k>ℰverify,k+CT,γ,Rεproj,k G_k>E_verify,k+C_T,γ,R _proj,k. After each projection, RE-0 regenerates trajectories with πk+1 _k+1 from the raw task input and re-allocates local teacher credit on the new occupancy. Remark 1 (Scope of the guarantee). ^k G_k and ℰverify,kE_verify,k are measured per event by the audit and the admission rule; the projection term is not estimated a priori but guarded by the outer acceptance criterion (Remark 2), which retains a generation only if its measured performance does not regress—verified at the credit level, guarded by measurement at the policy level (per-round sign accounting in Appendix D). 3.4 Cold Start and Recursive Refinement The construction assumes nothing about who occupies the policy seat: replacing dπkd_ _k by dσd_σ for any seat policy σ defines μσ=ℐ(σ)μ^σ=I(σ), and under the simultaneous coverage of Eq. 2 the guarantee of Eq. 4 transfers verbatim, with μσ=σμ^σ=σ when the gate is identically zero—no global capability relation, no training (Proposition 1, appendix). RE-OPD-cold. Taking σ=τσ=τ puts the teacher in the seat: its own failure histories are likewise diagnosed, repaired under progressive contracts, and verified by the same gate, and the verified-completed trajectories form the cold corpus 0D_0 on which full-trajectory supervision yields π1 _1. 0D_0’s quality is guaranteed by Proposition 1, not by global teacher reliability—even a teacher unable to complete the task contributes trajectories inheriting verified self-improvement credit, and π1 _1 inherits the lower bound of Theorem 1 with the teacher in the seat (Corollary 1, appendix; the cold surrogate is completion-conditioned there). RE-OPD-refine. Later generations put the student in the seat, πk+1=Θ(ℐ(πk)) _k+1=P_ (I( _k)); the projections from the teacher and student seats are RE-OPD-cold and RE-OPD-refine, sharing all machinery and differing only in the occupancy source of the corpus. Termination is structural: if Δ¯k(h)≤ϵ _k(h)≤ε for almost all h∈supp(dπk)h (d_ _k), the gate closes, Eq. 5 degenerates to the retention term, and the recursion safely abstains (Corollary 2, appendix)—triggered by exhausted headroom or by teacher effect below the verification noise floor bkb_k. Remark 2 (Deployment criterion and two conditions for useful refinement). Given threshold ρ, halt if J^(π1)≥ρ J( _1)≥ρ and otherwise refine, re-checking after each round. Refinement is useful only when headroom η (student below ρ) and verifiable teacher effect ν (admitted credit mass above the noise floor) hold jointly; either alone is insufficient, as the lift experiment in §4.2 shows empirically. 4 Experiments 4.1 Experimental Setup Tasks and environments. We evaluate on six long-horizon robosuite Code-as-Policy tasks—cube restack, spill wipe, nut assembly, two-arm lift, cube stack, and door opening—covering single/dual arms, rigid/articulated objects, and coarse/fine manipulation (details in the appendix). Base-model success spans 4% to 100% (restack 8%, lift 4%, nut 4%, spill 68%, door 96%, cubestack 100%), a full difficulty spectrum; each task keeps a held-out seed set disjoint from training collection. Models, seats, collection, and training. The student is Qwen3.8-27B (Qwen Team, 2026) fine-tuned with LoRA; the teacher and both diagnostic agents are the qwen3.8-max API. The teacher is black-box (no conditional distributions), so all projections use the trajectory-level hard estimator; Eq. 5 thus becomes a supervised loss weighted by verified credit, with the same optimizer family as SFT—the difference is not the loss form but what enters it: supervision locations, per-event weights gkwkg_kw_k, and the retention term (1−gk)(1-g_k) are decided by environment verification, not teacher sampling. Baselines (plain SFT, RFT/STaR, GRPO, OPD without verification) share the optimizer, loss family, and evaluation protocol, differing only in corpus provenance and quality assurance. Collection sends fresh-rollout failures (cycles the student already solves are skipped) through the mechanism of §3.2 with LCB admission per Eq. 2; the same pipeline on the teacher seat and the student seat yields the RE-OPD-cold and RE-OPD-refine corpora (hyperparameters in the appendix). Abstention rule: tasks already at ≥ 80% success do not enter refinement—the practical instance of Corollary 2—and are marked as abstained. The execution system saves full simulation snapshots at code decision points; diagnostic evidence z includes 1–2 perception-derived physical quantities per task (§4.4). Evaluation protocol. All methods use 50 paired held-out seeds per task (Wilson 95% CIs in Appendix D; paired per-seed outcomes for McNemar tests ship with the released artifacts) and share identical training and evaluation infrastructure apart from the variable under study. One protocol caveat: stage-entry (ρ) decisions reused the 50-seed evaluation cohort; collection seeds are always disjoint, and re-evaluation of all base and cold checkpoints on a disjoint, never-used 50-seed cohort reproduces the reported conclusions within binomial variation (Appendix D). 4.2 Main Results Table 1 reports held-out success on 50 seeds across the six tasks. Task Base Flash-API RE-OPD-cold (n rows) RE-OPD-refine restack 4/50a 2 31 (3) 40b spill 34 33 50 (67) —c nut 2 26 44 (27) — lift 2 6 38 (7) 29d door 48 — 50 (40) — cubestack 50 50 50 (0, no-train)e — Table 1: Main results (successes / 50 held-out seeds). Flash-API: same teacher, single-shot without diagnose–repair–verify. n: verified training rows. “—”: criterion abstained, not unmeasured. Trainer-seed replication (three seeds, Appendix D): restack 31/44/36, spill 50/33/33, lift 38/33/35, nut 44/40/43, door 50/50/50. Sample efficiency of verified data. RE-OPD-cold lifts the base from 4–68% to 62–100% with 3–67 verified rows (restack 62% with 3 rows, lift 76% with 7, nut 88% with 27)—improvement is not monotone in row count; the decisive factor is the verified credit each row carries (Corollary 1), not corpus size. The same teacher’s single-shot calls reach only 4% on restack while solving nut (52%): without the diagnose–repair–verify loop, teacher capability does not land on hard tasks. When no teacher is needed is decided by the framework itself: cubestack is saturated and exempted; door goes 96%→ 100% with 40 rows. Adaptive criterion. Refinement happens only where the target is unmet: restack 62%<ρ=80%<ρ=80\% refines to 80%; spill is already at 100%, its admission is empty, and the framework correctly abstains—a runtime instance of Corollary 2. Admissions decay with occupancy (6/6 at the π1 _1 round vs. 4/15 at π2 _2; Figure 3b; per-round accounting in the appendix). The lift row exercises the outer acceptance test (Remark 1): 76%<ρ<ρ but π2 _2 admits one event (w=0.069w=0.069)—thin verified credit; the projection regresses to 58% with branch collapse (Appendix D, note (d); the regression replicates at training seed 2, 76%→ 46%), and the criterion halts the recursion, disclosing the regressed checkpoint rather than propagating it: η without ν yields a harmful projection, and it is the stage-entry measurement that catches it. Failure of pooling. Naively pooling verified data across tasks causes severe interference despite larger corpus size (Table 8, appendix), consistent with the allocation failure mode in §3.1. 4.3 Verification and Credit Analysis The admission ledger measures teacher effect. The same gate, applied to different teachers, yields a monotone ladder (appendix, Table 6): 0 admissions for student self-play (34 cycles) and for narrow local experts (32 events; ∼ 10% rescue ceiling, deterministic contract-format failures), versus 4 admissions on 13 events for the strong API teacher (rescue rates 19–47%). These gradings precede any training: the verifier directly measures the teacher effect ν of Remark 2—not “who is stronger,” but “who holds verifiable local credit on the student’s failure history.” Strong separation makes the strict gate free; the gate is two-sided. Admitted events cluster at Δ^∈[0.19,0.44] ∈[0.19,0.44] while abstained ones satisfy |Δ^|≤0.04| |≤ 0.04 (Figure 3c); sweeping the threshold from 0.24 to 0.01 leaves the admitted set unchanged (Figure 3d)—the LCB gate’s statistical strictness costs no sensitivity. The verifier also rejects harm: one API patch dropped the baseline 8/16→ 1/16 (Δ^=−0.44 =-0.44) and was rejected. Credit assignment is a decision on the causal sign of an intervention, not one-directional imitation. Figure 3: Credit analysis. (a) Headroom alone does not imply credit: all tasks sit below ρ=0.80ρ=0.80 yet admissions range 0–4. (b) Admission decay across rounds. (c) Δ separation: admitted vs. abstained over 110 events. (d) Threshold sweep: the admitted set is invariant. Figure 4: Zero-shot sim-to-real transfer. Real-robot execution of the simulation-trained policies on the dual-arm AgileX PiPER platform: task keyframes (left), hardware success counts on 20 trials per task (middle), and hardware success rates against the simulation base model (right). The deployed policy code is identical to the simulation policy—no hardware-specific edits. Passing the gate ≠ policy flip. Weight-scale and normalization ablations cannot recover the gain of the verified corpus and can induce catastrophic forgetting (Appendix E), showing that verification determines corpus quality while weighting alone cannot substitute for composition-preserving training. This isolates the load-bearing variable: the gate’s product is the corpus. GRPO control. With terminal rewards in the same environments, GRPO (Shao et al., 2024) undergoes entropy collapse (0.30→ 0.002) and transfers none of it: 0–1/50 on the non-saturated tasks against base 8/68/4%, retaining only the saturated task (details and tuning notes in the appendix). Sparse terminal rewards cannot localize the harmful decision inside a long program, in mechanistic contrast to locate–verify–weight. 4.4 Ablations Selection-rule ablation: the gate is the source of credit quality. Holding the candidate journal and training recipe fixed, LCB admission outperforms unfiltered and sign-only selection on lift across three trainer seeds (Table 2; Appendix E). Selection rule (lift journal) Rows Successes / 50 Mean s1s_1 s2s_2 s3s_3 None (all candidates) 25 6 2 3 3.7 Sign (Δ^>0 >0) 7 5 4 4 4.3 LCB (paired, Eq. 2) 3 11 5 13 9.7 Outcome (teacher programs) 27 5 9 4 6.0 Table 2: Selection-rule ablation (lift; three trainer seeds; uniform weights). LCB dominates unfiltered and sign rules in every seed pairing. Corpus composition is decisive. On restack, success-filtered teacher demonstrations remain at base (3/50), whereas three verified event patches reach 46/50. Outcome filtering selects the identical corpus on this task, so the discriminating admission-rule comparison is Table 2 (Appendix E). Objective-composition quartet; teacher necessity. Replacing only the objective composition (restack; separate 50-seed cohort; armwise scores, definitions, and per-cohort variance in the appendix) confirms the decomposition of Eq. 5: stripping the student prefix (patch-only) or the retention term (weighted) collapses the gain that the full objective and uniform credit largely retain. Student self-bootstrap yields 0 admissions on four tasks, echoing §4.3. 4.5 Generalization and Real-Robot Deployment Transfer and deployment. Cross-task evaluation shows selective rather than universal transfer: restack training improves nut while spill’s wiping dynamics do not carry, and pooled multi-task training exhibits substantial interference (Appendix F). We additionally deploy the distilled, teacher-free policies zero-shot, without hardware-specific policy edits, on a dual-arm AgileX PiPER platform with SAM3-based open-vocabulary perception, obtaining 13/20 restack, 12/20 nut, 20/20 spill, 20/20 cubestack, and 2/20 lift successes (Figure 4, Appendix F). Verification is used only during simulation-side collection; hardware deployment makes no teacher calls. 5 Conclusion We recast self-improvement of embodied Code-as-Policy agents as a locate–verify–weight recursion: teacher interventions are minimal patches, verified event-wise by same-snapshot counterfactual experiments, admitted only with finite-sample confidence, and returned to the student after a single bounded macro-action. Across six tasks and physical deployment, 3–67 verified rows lift the base from 4–68% to 62–100%, while the same admission ledger doubles as a measurement instrument driving self-termination. The central empirical finding is that verified corpus selection, not weighting alone, is the load-bearing mechanism—confirmed by selection ablations across three trainer seeds (Table 2); making the ideal allocation of supervision under composition constraints explicit is the natural next step. AI use statement Roles of AI systems. All policies evaluated in this work are LLM-based code-as-policy agents; they constitute the object of study and are reported as experimental setup, not as AI assistance. Required disclosure. We used generative AI tools to: (i) design and provide feedback on research methodology and experiments (experiment prioritization and ablation design were discussed with an AI assistant; all protocol parameters were fixed in advance by the authors and never altered); (i) implement methods (operational scripts for training, evaluation, and data assembly were drafted with AI assistance and executed under author supervision; the core environment and pipeline code is the authors’ own); (i) interpret results (log analysis and attribution of negative results were AI-assisted and re-verified by the authors against raw artifacts); and (iv) assist with translation of the manuscript draft. We did not use generative AI to generate synthetic datasets, to formulate or prove mathematical claims, or to produce any experimental data: every number in this paper is programmatically extracted from logged evaluation runs of the described pipeline. Recommended disclosure. We additionally used generative AI tools to draft parts of the paper, edit text for readability, create scientific figures (all figure scripts and underlying event data will be released), suggest paper structure, and format references. Responsibility. All AI-assisted code was executed and tested by the authors, all reported numbers were cross-checked against raw logs, and all AI-assisted text was reviewed and revised by the authors. We take full responsibility for the final content of this work. Reproducibility statement We will release the complete pipeline and all artifacts: (i) environment configurations for all six robosuite tasks; (i) the full admission journals—110 audited events with paired counterfactual advantages, lower confidence bounds, and assigned training weights; (i) every verified training parquet with row-level provenance to admission events; (iv) raw evaluation JSONs underlying all reported evaluation tables and ablations, together with the admission journals underlying the analysis figures; and (v) training configurations. All numbers in the paper are traceable to these artifacts via a release manifest. References Agarwal et al. (2024) Rishabh Agarwal, Nino Vieillard, Yongchao Zhou, et al. On-policy distillation of language models: Learning from self-generated mistakes. In International Conference on Learning Representations (ICLR), 2024. Ahn et al. (2022) Michael Ahn, Anthony Brohan, Noah Brown, et al. Do as i can, not as i say: Grounding language in robotic affordances. In Conference on Robot Learning (CoRL), 2022. Gou et al. (2024) Zhibin Gou, Zhihong Shao, Yeyun Gong, Yelong Shen, Yujiu Yang, Nan Duan, and Weizhu Chen. Critic: Large language models can self-correct with tool-interactive critiquing. In International Conference on Learning Representations (ICLR), 2024. Huang et al. (2022) Wenlong Huang, Fei Xia, Ted Xiao, Harris Chan, Jacky Liang, Pete Florence, Andy Zeng, et al. Inner monologue: Embodied reasoning through planning with language models. In Conference on Robot Learning (CoRL), 2022. Kelly et al. (2019) Michael Kelly, Chelsea Sidrane, Katherine Driggs-Campbell, and Mykel J. Kochenderfer. Hg-dagger: Interactive imitation learning with human experts. In IEEE International Conference on Robotics and Automation (ICRA), 2019. Ko et al. (2024) Jongwoo Ko, Sungnyun Kim, Tianyi Chen, and Se-Young Yun. Distillm: Towards streamlined distillation for large language models. In International Conference on Machine Learning (ICML), 2024. Le et al. (2022) Hung Le, Yue Wang, Akhilesh Deepak Gotmare, Silvio Savarese, and Steven C. H. Hoi. Coderl: Mastering code generation through pretrained models and deep reinforcement learning. In Advances in Neural Information Processing Systems (NeurIPS), 2022. Liang et al. (2023) Jacky Liang, Wenlong Huang, Fei Xia, Peng Xu, Karol Hausman, Brian Ichter, Pete Florence, and Andy Zeng. Code as policies: Language model programs for embodied control. In IEEE International Conference on Robotics and Automation (ICRA), 2023. Lin et al. (2025) Zefu Lin et al. Embodiedcoder: Parameterized embodied mobile manipulation via modern coding model. arXiv preprint arXiv:2510.06207, 2025. Madaan et al. (2023) Aman Madaan, Niket Tandon, Prakhar Gupta, et al. Self-refine: Iterative refinement with self-feedback. In Advances in Neural Information Processing Systems (NeurIPS), 2023. Qwen Team (2026) Qwen Team. Qwen3.8-27b model card. https://huggingface.co/Qwen/Qwen3.8-27B, 2026. Ross et al. (2011) Stéphane Ross, Geoffrey Gordon, and Drew Bagnell. A reduction of imitation learning and structured prediction to no-regret online learning. In International Conference on Artificial Intelligence and Statistics (AISTATS), 2011. Shao et al. (2024) Zhihong Shao, Peiyi Wang, Qihao Zhu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024. Shinn et al. (2023) Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning. In Advances in Neural Information Processing Systems (NeurIPS), 2023. Singh et al. (2023) Ishika Singh, Valts Blukis, Arsalan Mousavian, Ankit Goyal, Danfei Xu, et al. Progprompt: Generating situated robot task plans using large language models. In IEEE International Conference on Robotics and Automation (ICRA), 2023. Sygkounas et al. (2026) Alkis Sygkounas, Victor Aregbede, Amy Loutfi, and Andreas Persson. Memento: Memory-guided memetic code-as-policy evolution. arXiv preprint arXiv:2607.22832, 2026. Zhang & Cho (2017) Jiakai Zhang and Kyunghyun Cho. Query-efficient imitation learning for end-to-end simulated driving. In Proceedings of the AAAI Conference on Artificial Intelligence, 2017. Zhang et al. (2021) Songyuan Zhang, Zhangjie Cao, Dorsa Sadigh, and Yanan Sui. Confidence-aware imitation learning from demonstrations with varying optimality. In Advances in Neural Information Processing Systems (NeurIPS), 2021. Appendix A Limitations (1) The allocation layer is not closed: w≡LCBw suppresses patches to 1–1.5% weight, Σw→1 w→ 1 normalization fails to flip and destroys composition, and unprotected distillation forgets; converting verified credit into policy change needs an allocation theory under composition constraints (we contribute a stress test and the η/νη/ν criteria). The teacher is black-box, so deployed distillation uses the trajectory-level hard estimator; the credit weighting did not measurably outperform uniform weighting on the same verified corpus—we claim verified data selection, not the weighting scheme, as the empirical mechanism. (2) Teacher necessity is calibrated against an API frontier model, and paired-counterfactual cost grows linearly with audited events. (3) Conclusions rest on six robosuite families and one physical platform; transfer is selective. (4) The π1.5 _1.5 row carries a collection-seat artifact; ablation pools differ from the main-table protocol; the normalization arm is a single run, and selection-ablation arms are single runs per seed (three seeds on lift). (5) The handback design (control returning to the student immediately after each verified patch, §3.2) rests on design argument, and RFT/STaR remains inferred from the gate-off ablation and the self-imitation arm. (6) Stage-entry (ρ) decisions reused the evaluation cohort as validation (validation and test are not fully separated for the reported stage-entry decisions); a disjoint never-used cohort re-evaluation reproduces the base/cold conclusions (Appendix D), and the released protocol separates them prospectively. The reuse is consequential only for thin margins: retained generations cleared +18+18 to +84+84 points and abstentions sat 1616–2020 points above ρ; the one thin entry (lift, 76% against ρ=80%ρ=80\%) is precisely the case the criterion re-measured and rejected (§4.2). Appendix B Proofs and statistical guarantees Throughout, R¯:=R1−γT1−γ R:=R\, 1-γ^T1-γ bounds any trajectory return (T-step horizon, |rt|≤R|r_t|≤ R), and Sγ:=∑t=0T−1γtS_γ:= _t=0^T-1γ^t. Snapshot-restoration and execution-identity discrepancies are booked into ℰsys,kE_sys,k. Formal definitions (§3.1). • Return. J(π)=x∼ρ,ζ∼Pπ[∑t=0T−1γtrt]J(π)=E_x ρ,ζ P^π [ _t=0^T-1γ^tr_t ]. • Histories. At the t-th code decision point, ht=(o0,c0,o1,c1,…,ct−1,ot)h_t=(o_0,c_0,o_1,c_1,…,c_t-1,o_t) with bounded segments ct∈c_t treated as macro-actions; the occupancy is dπkt(ht∣x)=Prπk(Ht=ht∣x)d_ _k^t(h_t x)= _ _k(H_t=h_t x). • Teacher policy and action values. The teacher’s local policy τk(δ∣x,ht) _k(δ x,h_t) proposes a bounded correction whose suffix is regenerated by πk _k; Qtπk(ht,δ)=[∑s≥tγs−trs|Ht=ht,Ct=δ,C>t∼πk],Q_t _k(h_t,δ)=E [ _s≥ tγ^s-tr_s\, |\,H_t=h_t,\,C_t=δ,\,C_>t _k ], (B.1) with Vtπk(ht)=c∼πk[Qtπk(ht,c)]V_t _k(h_t)=E_c _k[Q_t _k(h_t,c)], Atπk=Q−VA_t _k=Q-V, and Δk(ht)=δ∼τk[Atπk(ht,δ)] _k(h_t)=E_δ _k[A_t _k(h_t,δ)]. • Contract selection. χk⋆(h^)=argminχ∈(h^,z)κ(χ) _k ( h)= _χ∈ X( h,z)κ(χ) subject to LCB^k(h^,χ)>ϵ LCB_k( h;χ)>ε and Valid(χ,τ¯kχ,sh^snap)=1Valid(χ, τ_k^χ,s^snap_ h)=1, applied per τ¯kχ τ_k^χ during selection, where ValidValid requires that the contract-constrained policy support meets execution-safety and program-context constraints and that snapshot-restoration and execution-identity checks agree; with constrained decoding, support constraints are realized implicitly by the decoder. Two-layer confidence radius. Fix a history h h and let teacher candidates δ1,…,δK∼iidτ¯k(⋅∣x,h^) _1,…, _K iid τ_k(· x, h) with paired estimates A^k(h^,δj)=1M∑m=1M[GjmT−GjmS] A_k( h, _j)= 1M _m=1^M[G^T_jm-G^S_jm] under shared randomness ξm _m, and Δ^k(h^)=1K∑jA^k(h^,δj) _k( h)= 1K _j A_k( h, _j). Since the paired differences lie in [−R¯,R¯][- R, R] with common noise eliminated by pairing, Hoeffding applied to the inner (M-pair) and outer (K-candidate) layers gives, for any α1,α2∈(0,1) _1, _2∈(0,1), with probability at least 1−(α1+α2)1-( _1+ _2), |Δ^k(h^)−Δk(h)|≤R¯2ln(2/α1)K+R¯2ln(2/α2)M⏟=bk(h^,K,M,α). | _k( h)- _k(h) |\;≤\; R 2 (2/ _1)K+ R 2 (2/ _2)M_=\,b_k( h;K,M,α). (B.2) Simultaneous coverage and selection. Allocate levels αk,h _k,h with ∑k,hαk,h≤α _k,h _k,h≤α across all histories verified in generation k; by the union bound the event ℰcov:=∀(k,h):Δk(h)≥Δ^k(h)−bk(h,K,M,αk,h)E_cov:=\∀(k,h): _k(h)≥ _k(h)-b_k(h;K,M, _k,h)\ holds with probability at least 1−α1-α. The admission gate gkg_k is measurable in Δ^k _k (data-dependent selection), but ℰcovE_cov holds uniformly over all histories, so selection does not invalidate the per-history bound—this is why simultaneous rather than pointwise confidence bounds are used. Implementation note (analytical vs. operational bound). The Hoeffding radius above is the analytical form used in the theorem statements. The deployed verifier (anytime_valid_betting_mixture_v1) uses an anytime-valid e-process over the same paired differences (success and normalized-reward channels in parallel), which enjoys the same coverage guarantee with a tighter radius; the theorem statements are unaffected by this choice. Admission used ϵ=0ε=0 and α=0.05α=0.05; each audited event starts at M=8M=8 shared-randomness pairs and escalates (12/16/24/32/4812/16/24/32/48) when evidence is insufficient; candidates are drawn under top-k constrained decoding within the selected contract. At the base pairing, one audited event costs up to (K+1)×M(K+1)× M suffix rollouts; the full ledger contains 110 audited events. History–state performance-difference lemma. For any policies π,π′π,π , J(π′)−J(π)=∑t=0T−1γth∼dπ′t[c∼π′(⋅∣x,h)[Qtπ(h,c)]−Vtπ(h)].J(π )-J(π)= _t=0^T-1γ^t\,E_h d_π ^t [E_c π (· x,h)[Q_t^π(h,c)]-V_t^π(h) ]. (B.3) Proof. Induction on T. For T=1T=1 both sides equal h∼d0c∼π′[r(h,c)]−VπE_h d^0E_c π [r(h,c)]-V^π. For the step, decompose QTπ(h,c)=r(h,c)+γh′VT−1π(h′)Q_T^π(h,c)=r(h,c)+γ\,E_h V_T-1^π(h ), difference the first-step expectations of π′π and π, expand the continuation of π′π along dπ′td_π ^t, and apply the hypothesis. Only the Markov property of the macro-action history process is used. □ Execution-time identity. Under the mixture of Eq. 3, J(μk)−J(πk)=∑tγth∼dμkt[gk(h)Δk(h)].J( _k)-J( _k)= _tγ^tE_h d_ _k^t [g_k(h)\, _k(h) ]. (B.4) Proof. Take π′=μkπ = _k in the lemma; per history, the student component (1−gk)πk(1-g_k) _k cancels against Vtπk(h)V_t _k(h) and the teacher component contributes gk(h)Δk(h)g_k(h) _k(h). □ Execution-time bound (Eq. 4). On ℰcovE_cov, gk(h)Δk(h)≥gk(h)(Δ^k(h)−bk(h))g_k(h) _k(h)≥ g_k(h) ( _k(h)-b_k(h) ) per history; on admitted histories Δ^k(h)>bk(h)≥0 _k(h)>b_k(h)≥ 0 so [Δ^k(h)]+=Δ^k(h)[ _k(h)]_+= _k(h), and both terms vanish elsewhere. Summing yields J(μk)−J(πk)≥^k−ℰverify,kJ( _k)-J( _k)≥ G_k-E_verify,k. □ Seat invariance (Proposition 1). The proofs above use only the history distribution induced by the seat policy; replacing πk _k by any σ throughout gives J(μσ)−J(σ)≥^σ−ℰverify,σJ(μ^σ)-J(σ)≥ G^σ-E_verify,σ, with μσ=σμ^σ=σ when g≡0g≡ 0 (identical process), and strict inequality when a visited history carries g(h)>0g(h)>0 (its admitted contribution is strictly positive and the verification term is controlled by admission). □ Occupancy divergence chain. For any θ and t≥1t≥ 1, TV(dμkt,dπθt)≤∑s<th∼dμks[TV(μk(⋅∣x,h),πθ(⋅∣x,h))].TV(d_ _k^t,d_ _θ^t)≤ _s<tE_h d_ _k^s [TV ( _k(· x,h),\, _θ(· x,h) ) ]. (B.5) Proof. The two history distributions can diverge only after their first divergence point; sum over divergence times and merge same-time contributions by the recursion of the history distributions (standard simulation argument). □ Projection-error bound. Combining the chain with Pinsker (TV≤DKL/2TV≤ D_KL/2) and Cauchy–Schwarz, |J(μk)−J(πθ)|≤CT,γ,Rεproj,k,|J( _k)-J( _θ)|\;≤\;C_T,γ,R _proj,k, (B.6) with εproj,k=∑tγth∼dμkt[DKL(μk∥πθ)] _proj,k= _tγ^tE_h d_ _k^t[D_KL( _k\| _θ)] and CT,γ,R=R¯normSγ/2/(1−γ)C_T,γ,R= R_norm S_γ/2/(1-γ) for the appropriately normalized return bound. Theorem 1. On ℰcovE_cov, decompose J(πk+1)−J(πk)=[J(μk)−J(πk)]−[J(μk)−J(πk+1)]J( _k+1)-J( _k)=[J( _k)-J( _k)]-[J( _k)-J( _k+1)] and bound the first term by the execution-time bound and the second by the projection-error bound at θ of πk+1 _k+1. □ Token-level decomposition. When teacher and student share a tokenizer, DKL(τ¯k∥πθ)D_KL( τ_k\| _θ) decomposes into per-token conditional KLs by the chain rule, with the termination symbol included in δ. □ Corollary 1 (cold-start amortization). The cold corpus 0D_0 consists of verified-completed trajectories of PμτP^μ^τ, so the full-trajectory empirical risk 1|0|∑ζ−logπθ(ζ) 1|D_0| _ζ- _θ(ζ) is an unbiased surrogate of the completion-conditioned divergence DKL(Pμτ∣C∥Pπθ)D_KL(P^μ^τ\! \!C\,\|\,P _θ) (C the completion event; trajectory-level chain decomposition), whose population optimum is the projection on the success-conditioned occupancy—not on the unconditioned μτμ^τ occupancy. This conditioning is the cold-start instance of the same credit weighting: with no per-history gates yet, verification reduces to the completion indicator—the trajectory-level w≡Cw 1\C\ of Eq. 5—and the Theorem-1 decomposition carries the teacher-seat bound of Proposition 1 through with the projection error measured against this weighted target. The gap between the conditioned and unconditioned occupancies is exactly the selection cost that later rounds avoid by conditioning on local counterfactual credit instead of trajectory outcomes. □ Corollary 2 (safe abstention). If Δ¯σ(h)≤ϵ _σ(h)≤ε almost everywhere, then gσ(h)=0g_σ(h)=0, so μσ=σμ^σ=σ per history (no execution-time change) and the training weight gσ(h)wσ(h)≡0g_σ(h)w_σ(h)≡ 0 (no teacher term): neither execution nor training introduces unverified teacher credit. □ Remark (rewind semantics, full). The deployed pipeline runs two passes: the student’s live attempt realizes a failure and localizes candidate boundaries; verification then rewinds to the stored snapshot sh^snaps_ h^snap. We analyze the post-rewind process: J(μk)J( _k) measures the return of the repaired continuation from sh^snaps_ h^snap onward, and both arms of every paired comparison (teacher correction vs. student resample) start from the same snapshot under shared subsequent randomness. The first pass is therefore a fixed data-collection cost common to the comparison and cancels in Eq. 4; equivalently, μk _k is a one-pass policy over histories whose state includes the snapshot stack. Formally, the augmented process has state (s,)(s,Z) with Z the snapshot–evidence stack: the diagnostic evidence z observed after the boundary is part of the state that selects the intervention point, so whether to intervene at h h depends on information observed after h h in the original rollout. J(μk)J( _k) is the expected return of the post-rewind continuation under this two-pass process; it is not the return of a causal online policy in the original environment, and the first pass cancels only because both arms of each paired comparison share it as a fixed cost. Theorem 1 bounds the augmented-process return; a snapshot-free deployment that diagnoses from the current history alone—as our hardware deployment does, running the standalone student with no teacher calls—is outside this guarantee. Ideal supervision allocation. Given a budget B of teacher calls and verification rollouts, the ideal allocation solves, over histories, max∑tqkγth∼dπkt[qk(h)Δk(h)]s.t.∑tγth∼dπkt[qk(h)κ(h)]≤B,qk(h)∈[0,1], _q_k\; _tγ^t\,E_h d_ _k^t [q_k(h)\, _k(h) ] .t. _tγ^t\,E_h d_ _k^t [q_k(h)\,κ(h) ]≤ B,\;\;q_k(h)∈[0,1], (B.7) a fractional-knapsack linear program requiring Δk(h) _k(h) and the occupancy, neither directly observable. RE-0’s diagnostic localization (shrinking the candidate history set) and LCB admission (accepting only verifiable positive credit) form its practical proxy: in the empirical-LCB sense, the admitted set is exactly the histories with estimated positive credit. Appendix C Experimental details and reproducibility Training and evaluation configuration. All students are Qwen3.8-27B with LoRA (r=16r=16, α=32α=32, all-linear), trained with the trajectory-level hard estimator for 120 steps (batch 2, LR 10−510^-5 constant, seed 1, bf16); refinement warm-starts from the previous generation at LR 5×10−65× 10^-6. Evaluation uses 50 held-out seeds per task (temperature 0.2, top-p 0.95, non-thinking mode) reset seeds from the 210000001 series and generation seeds from the 220000001 series (stride 10007), disjoint from collection. Collection verifies each contract with K teacher candidates against M shared-randomness paired rollouts; admission requires the two-layer LCB of Eq. 2 to exceed ϵε under the α-allocated simultaneous budget. Verification cost. Across the six tasks the full ledger contains 110 audited events, of which 19 were admitted. Each event verifies its selected candidate from the boundary snapshot with M shared-randomness pairs (escalating 8/12/16/24/32/488/12/16/24/32/48 when evidence is insufficient; observed distribution 27/50/3/8/1/2127/50/3/8/1/21 events, median M=12M=12), each pair executing one student resample and one teacher continuation: the selected candidates alone consumed 4,1924,192 suffix rollouts in total. Rejected candidates and the live first pass add further environment rollouts, so verified rows are information-dense but not rollout-free. Appendix D Main-result accounting and robustness Table 1 notes. (a) Base row uses the standard 50-seed evaluation cohort, disjoint from collection. (b) The refine cell for restack comes from an intermediate teacher-seat recollection; a collection-seat artifact affecting this cell is disclosed in the Limitations. (c) Spill saturates at stage one; its counterfactual admission set is empty, so the criterion abstains—the runtime instance of Corollary 2. (d) Lift refinement consumed a single admitted event (w=0.069w=0.069) plus replay: 76%→ 58%, with collateral collapse on the branch (spill 28→028→ 0)—headroom without credit mass is harmful (Remark 2). (e) Cubestack is saturated at base; the system exempts it from training. The transfer table’s own-task cells for π1/π2′ _1/ _2 (restack) are 31/40 (Table 1). (f) Composition arms (restack, 50 seeds): plain SFT on 8 success-filtered teacher-seat demonstrations scores 3/50; SFT on outcome-selected patches (3 rows, unweighted) scores 46/50. The nine patches rejected by the outcome rule all have intervention success 0.0—zero divergence from the LCB gate on this corpus; both arms are single runs. (g) Row counts and admission counts are different objects: cold rows count verified-completed teacher-seat trajectories (the Corollary 1 corpus), while Figure 3b counts admission events in student-seat collection rounds. Per-round sign accounting. Theorem 1 bounds the standalone gain by the verified intervention gain net of verification and projection terms; the first two are measured at admission, while the projection term is guarded operationally by the outer acceptance criterion (Remark 1). Table 3 reports, for every round in which the criterion made a retain-or-halt decision, the admitted credit of the round against the realized change in standalone performance on the 50-seed protocol. The signs agree in all four rounds, including the round the criterion rejected. We read this as sign-level consistency evidence over four rounds, not an estimate of εproj,k _proj,k. Re-executing the rejected lift round with training seed 2 reproduces the regression (76%→ 46%), so the halt reflects the round’s credit rather than a seed artifact. Round Admitted credit J J change Decision Retained restack π0→π1 _0\!→\! _1 (cold) 6/6 events, Δ^≈+1.0 \!≈\!+1.0 +54+54 pt (→62%8\!→\!62\%) J<ρJ<ρ: continue π1 _1 restack π1→π2′ _1\!→\! _2 (refine) 4/15, Δ^∈[0.19,0.88] \!∈\![0.19,0.88] +18+18 pt (→80%62\!→\!80\%) J≥ρJ≥ρ: halt π2′ _2 lift π1→π2 _1\!→\! _2 (refine) 1 event, w=0.069w=0.069 −18-18 pt (→58%76\!→\!58\%) regression: halt π1 _1 spill π1→ _1\!→ (refine) 0 admitted — (no projection) abstain (Cor. 2) π1 _1 Table 3: Per-round sign accounting: admitted credit against realized standalone change. The retained generation is the measured-better one in every round. Wilson confidence intervals. Task Base RE-OPD-cold RE-OPD-refine restack [3.2, 18.8] [48.2, 74.1] [67.0, 88.8] spill [54.2, 79.2] [92.9, 100.0] — nut [1.1, 13.5] [76.2, 94.4] — lift [1.1, 13.5] [62.6, 85.7] [44.2, 70.6] door [86.5, 98.9] [92.9, 100.0] — cubestack [92.9, 100.0] [92.9, 100.0] — Table 4: Wilson 95% CIs (%) for Table 1. Trainer-seed replication. We retrained the restack and spill cold-start arms with two additional trainer seeds on the identical corpora and recipe. Restack: seed 1 =31/50=31/50, seed 2 =44/50=44/50, seed 3 =36/50=36/50 (mean 37.0)—every replicate far above the base 4/50. Spill: seed 1 =50/50=50/50, seeds 2 and 3 =33/50=33/50 each (mean 38.7)—two of three replicates sit at base level (34/50), so the spill cold-start gain is seed-sensitive. With the same protocol we additionally replicated the remaining cold arms: lift reaches 38/33/3538/33/35 (mean 35.3), nut 44/40/4344/40/43 (mean 42.3), and door 50/50/5050/50/50—every replicate sits far above its base, so the large cold-start gains on the thin-corpus tasks are robust to trainer seed, and spill remains the sole seed-sensitive task. Never-used cohort re-evaluation. Base and cold checkpoints were additionally re-evaluated on a disjoint, never-used 50-seed cohort (reset offset 230000001, generation offset 240000001): base 3/3/5/47/503/3/5/47/50 (restack, nut, lift, door, cubestack) and cold 28/45/42/50/4928/45/42/50/49 (restack, nut, lift, door, spill) match the reported cells within binomial noise, confirming that the reported numbers are not artifacts of the evaluation-seed protocol. Appendix E Mechanism and ablation details Table 2 holds each task’s collection journal fixed and varies only the admission rule. Provenance: the lift/spill/nut journals are teacher-seat (self-play) collections; all arms use identical row construction (uniform weight, full-trajectory SFT). The four lift events admitted by the sign rule (Δ^>0 >0) but rejected by the LCB have Δ^∈+0.21,+0.06,+0.04,+0.00004 ∈\+0.21,+0.06,+0.04,+0.00004\ at M∈48,48,48,12M∈\48,48,48,12\: positive in mean, statistically indistinguishable from zero. Table 5 completes the grid with direction cells and the restack self-imitation arm. Task / rule Rows Seed 1 Seed 2 Note lift: none 25 6 2 lift: sign 7 5 4 lift: LCB 3 11 5 lift: outcome 27 5 9 spill: none 6 10 — mean reward 0.63 spill: sign 2 0 — collapse; reward 0.00 spill, nut: LCB 0 — — abstain nut: none 4 0 — nut: sign 3 2 — restack: self-imitation 11 2 — RFT analogue Table 5: Full selection-ablation grid (successes/50; one run per seed). The spill sign arm collapses to 0/50 with exactly zero reward: all fifty rollouts execute the same three-block program with no wipe coverage—two near-duplicate rows reduce the policy to template repetition. The LCB admits nothing on spill or nut and abstains. True base-seat RFT is infeasible in our protocol; the nearest analogue—self-imitation on the 11 success programs of the π1 _1-seat policy—reaches 2/50 under identical row construction. Teacher-effect ladder. Table 6 details the admission-ledger grading: student self-play yields no verifiable credit; narrow local task experts (win rates 62–100% on their own tasks) fail deterministically at the contract-format layer with a ∼ 10% rescue ceiling; the strong API teacher earns 4 admissions on 13 events with rescue rates 19–47%. Table 6: The admission ledger as a teacher-effect instrument. Teacher Events Admitted Rescue rate Failure mode Student self-play 33+ 0 ∼ 0 no credit at p≈0p\!≈\!0 Local task experts 32 0 ∼ 10% ceiling contract-format failures API frontier 13 4 19–47% — +restack 11, nut 6, spill 5. Appendix F Transfer, pooling, and deployment Table 7: Cross-task transfer (successes / 50, @120). Policy \ Task nut spill lift handover Base 2 34 2 — π1 _1 (restack student) 22 24 6 0 π2′ _2 (restack refined) 16 45 3 1 Lift expert 0 28 38 0 Own-task (restack): 31/40. Table 8: Pooled joint training fails (successes / 50). Two separable failures: task interference and quota starvation. Task Expert Equal (42) Proportional (25) Full-pool (172) Credit-equal restack 31 3 2 2 1 spill 50 31 0 0 32 nut 44 9 0 0 21 lift 38 20 25 5 15 cubestack 50 50 0 48 48 door 50 50 0 49 49 Real-robot platform. The deployment platform is a dual-arm AgileX PiPER rig with SAM3-based open-vocabulary perception (Figure 5); the execution stack exposes the simulation API surface unchanged, so the deployed policy code requires no hardware-specific edits. Compute infrastructure. All simulation training, collection, and evaluation ran on an eight-GPU NVIDIA H200 server (143 GB per GPU), one GPU per run: the 27B student is served with vLLM (LoRA adapters, runtime hot-loading), training uses FSDP with gradient checkpointing, and verification rollouts run against the same engine; teacher and diagnosis calls go to an external API model. Figure 5: Real-robot platform. The dual-arm AgileX PiPER rig used for the zero-shot deployment experiments. Appendix G Baseline details GRPO configuration and fairness. The GRPO control was tuned until it trained at all in this stack: chat-template thinking mode disabled (otherwise every response clips at the token budget and all rewards are zero), max response length 3072, thread-safety fixes for the robosuite reward and IK-server liveness monitoring, and memory-feasible two-card colocate settings for the 27B policy. KL regularization (reference model) was disabled to fit GPU memory; entropy collapse (0.30→ 0.002) persisted across all of these fixes. We cannot rule out that a larger-memory configuration with KL regularization behaves differently; the comparison should be read as against a resource-matched GRPO, not a maximally tuned one. The GRPO control evaluates on the same 50-seed protocol: restack 1/50, spill 0/50, lift 0/50, cubestack 50/50 (base 4/34/2/50); the nut evaluation could not be completed due to a serving-stack incompatibility and is omitted.