Paper deep dive
Prospective Hindsight: Self-Calibrating Reinforcement Learning via Prediction-Reality Gaps
Jiaxin Zhang, Xiangyu Peng, Qinglin Chen, Yu Li, Hiroaki Hayashi, Chien-Sheng Wu
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 92%
Last extracted: 10/5/2026, 6:49:16 AM
Summary
The paper introduces Prospective Hindsight (PH), a self-calibrating reinforcement learning training principle that addresses the 'hindsight trap' where agents receive only retrospective feedback. PH augments base methods (like GRPO and On-Policy Distillation) by comparing the agent's prospective prediction of success with the actual outcome. This 'surprise' signal identifies miscalibrated rollouts (overconfident failures and underconfident successes) and upweights their gradient contribution. The method leverages a privileged-information gap framework, showing that minimizing this surprise residual reduces miscalibration as a byproduct of optimization, improving both task performance and calibration across model scales.
Entities (8)
Relation Signals (6)
Prospective Hindsight → addresses → Hindsight Trap
confidence 95% · We introduce Prospective Hindsight (PH)... We call this structural obstruction the hindsight trap... PH improves both task performance and calibration
Prospective Hindsight → augments → On-Policy Distillation
confidence 90% · PH improves both task performance and calibration... under... on-policy distillation
Prospective Hindsight → augments → GRPO
confidence 90% · PH improves both task performance and calibration... under GRPO
Prospective Hindsight → basedon → Privileged Information
confidence 90% · We connect this principle to a privileged-information gap... This places agent RL inside Learning Under Privileged Information (LUPI) framework
Prospective Hindsight → reduces → Overconfident Failure
confidence 85% · PH addresses both successfully... overconfident failures in single-turn
Prospective Hindsight → reduces → Underconfident Success
confidence 85% · PH addresses both successfully... underconfident successes in multi-turn
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Reinforcement learning for long-horizon agents relies on purely retrospective training signals: credit is assigned only after observing environmental consequences, leaving the agent's belief at action time invisible to the gradient. We introduce Prospective Hindsight (PH), a self-calibrating training principle that augments any retrospective base method with a signal derived from the gap between the agent's prospective prediction (before feedback) and the retrospective evaluation (after feedback). This per-rollout surprise identifies samples where the agent's self-model is most inaccurate and amplifies their gradient contribution through a stop-gradient surprise-weighted advantage. Since the prospective predictor shares parameters with the policy, the two co-evolve, progressively shifting focus to the agent's remaining blind spots. We connect this principle to a privileged-information gap and show that minimizing the surprise residual provides a descent pathway on the agent's miscalibration rate; calibration thus emerges as a byproduct of optimization rather than from an added objective. On single-turn verifiable tasks and a multi-turn personal-agent task (under GRPO, on-policy distillation, and their combination), PH improves both task performance and calibration, with consistent gains across model scales. Notably, the dominant miscalibration mode shifts structurally between regimes, overconfident failures in single-turn, underconfident successes in multi-turn, yet the same training principle addresses both successfully.
Tags
Links
- Source: https://arxiv.org/abs/2610.02740v1
- Canonical: https://arxiv.org/abs/2610.02740v1
Trouble viewing inline? Open PDF directly →
Full Text
137,935 characters extracted from source content.
Expand or collapse full text
Prospective Hindsight: Self-Calibrating Reinforcement Learning via Prediction–Reality Gaps Jiaxin Zhang †thanks: Correspondence to jiaxin.zhang, wu.jason@salesforce.com Xiangyu Peng Qinglin Chen Yu Li, Hiroaki Hayashi, Chien-Sheng Wu Affiliation: Salesforce AI Research Abstract Reinforcement learning for long-horizon agents relies on purely retrospective training signals: credit is assigned only after observing environmental consequences, leaving the agent’s belief at action time invisible to the gradient. We introduce Prospective Hindsight (PH), a self-calibrating training principle that augments any retrospective base method with a signal derived from the gap between the agent’s prospective prediction (before feedback) and the retrospective evaluation (after feedback). This per-rollout surprise identifies samples where the agent’s self-model is most inaccurate and amplifies their gradient contribution through a stop-gradient surprise-weighted advantage. Since the prospective predictor shares parameters with the policy, the two co-evolve, progressively shifting focus to the agent’s remaining blind spots. We connect this principle to a privileged-information gap and show that minimizing the surprise residual provides a descent pathway on the agent’s miscalibration rate; calibration thus emerges as a byproduct of optimization rather than from an added objective. On single-turn verifiable tasks and a multi-turn personal-agent task (under GRPO, on-policy distillation, and their combination), PH improves both task performance and calibration, with consistent gains across model scales. Notably, the dominant miscalibration mode shifts structurally between regimes, overconfident failures in single-turn, underconfident successes in multi-turn, yet the same training principle addresses both successfully. Figure 1: Prospective Hindsight (PH) converts hindsight feedback into a self-calibrating training signal. PH compares an agent’s action-time prediction with the realized verifier/PRM outcome, partitions rollouts into calibrated and miscalibrated cases, and upweights surprising prediction–reality mismatches during policy updating within GRPO, on-policy distillation and their combination. 1 Introduction LLMs increasingly act as agents: decomposing tasks, calling tools, writing code, conducting extended tutoring conversations, and making intermediate commitments whose consequences unfold over many steps Wang et al. (2026b); Buening et al. (2026). In these settings, the most damaging failures are often not simple mistakes, but mistakes made with confidence Zhang et al. (2025c); Zhang et al. (2025b); Zhu et al. (2025); Zhang et al. (2026d). A confidently wrong agent continues down a flawed trajectory, ignores opportunities for recovery, and triggers incorrect downstream actions; an agent that fails while expressing uncertainty Zhang et al. (2026a); Oh et al. (2026), in contrast, invites a fallback, a retry, or a request for human input. Reliable agentic learning therefore requires more than improving final accuracy — it requires aligning the agent’s prospective belief about its own success with the reality of its outcomes. The hindsight trap. Modern agentic RL has converged on increasingly rich retrospective feedback Zhang et al. (2025a); Dong et al. (2025). Process reward models score each step by inspecting the next environmental state Wang et al. (2026b); Buening et al. (2026); on-policy distillation (OPD) methods Lu and Lab (2025); Song and Zheng (2026) construct a hindsight teacher conditioned on post-action context unavailable to the agent at decision time Agarwal et al. (2024); Ye et al. (2026); Hübotter et al. (2026); Shenfeld et al. (2026). These methods differ in mechanism but share a structural property: every learning signal is computed from quantities observed after the agent has acted, and the agent’s prospective belief at action time, i.e., what it expected would happen, never enters the gradient. Two rollouts that produced the same outcome therefore receive the same update, regardless of whether one was a confident success and the other a lucky guess, or whether one was a failure the agent could have anticipated and the other a failure that took the agent by surprise. We call this structural obstruction the hindsight trap, and it is the reason why retrospective training reliably improves task accuracy without resolving overconfidence Zhang et al. (2026c); Leng et al. (2024); Damani et al. (2025); Kalai et al. (2025); Groot and Valdenegro-Toro (2024). Calibration as a training signal, not a diagnostic. A substantial literature on LLM calibration has measured this overconfidence and proposed inference-time fixes: verbalized confidence prompts, semantic uncertainty estimators, post-hoc temperature scaling Guo et al. (2017); Kadavath et al. (2022); Tian et al. (2023); Kirchhof et al. (2025). Recent work extends these ideas into RL by adding explicit confidence-target losses or rewriting confidence tokens in the teacher distribution Zhang et al. (2026c); Leng et al. (2024); Damani et al. (2025), but in doing so introduces a separate calibration objective alongside the task objective; the two are reported to trade against each other. We approach the problem from a different angle. The gap between the agent’s prospective belief and the eventual outcome is not just a diagnostic of how well-calibrated the agent is; instead, it is a training signal that points to exactly the rollouts where action-time information was insufficient. Why prediction–reality gaps are informative. The same property that makes the prospective belief invisible to retrospective losses also makes the gap between belief and outcome informative. By a law-of-total-variance argument, the residual variance in the eventual outcome given action-time information decomposes into aleatoric noise (irreducible even with privileged context) plus a non-negative privileged-information gap: the share of outcome variance that no action-time predictor can resolve. This places agent RL inside Learning Under Privileged Information (LUPI) framework Vapnik and Vashist (2009); Pechyony and Vapnik (2010); Lopez-Paz et al. (2015). The rollouts that materialize this gap, overconfident failures and underconfident successes, are exactly where action-time information falls short of post-action information. Standard hindsight-only methods see these rollouts only through their outcome label and treat them indistinguishably from calibrated ones; we ask what changes if training explicitly uses the disagreement. We introduce Prospective Hindsight (PH), a self-calibrating training principle that augments any retrospective base method with a signal derived from the prediction–reality gap. Before each rollout is scored by the verifier or PRM, we re-query the same policy with a different prompt to elicit a prospective prediction of the eventual outcome; comparing this prediction to the retrospective evaluation yields a per-rollout surprise that identifies the training samples where the agent’s self-model is most inaccurate. PH amplifies the gradient contribution of these samples through a stop-gradient reweighting of the base loss, preserving the rollout sampling distribution and the internal structure of GRPO, OPD, or their combination. Because the prospective predictor shares parameters with the policy rather than being an external head bolted onto the model, the predictor co-evolves with the policy as training progresses: the agent’s self-model improves implicitly alongside the agent’s behavior, the surprise signal progressively shifts focus to the agent’s remaining blind spots, and the reweighting diminishes as the prediction–reality gap closes. PH therefore reduces miscalibration as a byproduct of optimization rather than through an added calibration loss, sidestepping the capability–calibration trade-off Leng et al. (2024); Damani et al. (2025). Our main contributions are summarized as follows: • We identify the hindsight trap as a structural blind spot of retrospective RL and connect the resulting prediction–reality gap to a privileged-information quantity within the LUPI framework. • We propose Prospective Hindsight, a principle method that converts this gap into a self-calibrating training signal compatible with binary RL (GRPO), On-policy distillation, and their combination. • We define a calibration taxonomy by pairing the prospective belief with the verifier outcome yields four rollout cells that decompose calibration along two interpretable axes during training. • PH improves both task performance and calibration on single-turn verifiable tasks and on a multi-turn personal-agent task, with consistent gains across model scales. The dominant miscalibration mode shifts structurally between regimes and PH addresses both successfully. 2 The Hindsight Trap in Agent RL 2.1 Agentic Reinforcement Learning We consider an agent represented by a policy πθ _θ acting in environments with verifiable outcomes. At each step t∈1,…,Tt∈\1,…,T\, the agent observes a state sts_t (a query, a conversation history, or a partial task description), produces an action at∼πθ(⋅∣st)a_t _θ(· s_t), and receives a step-level outcome rt=R(st,at,ct)∈−1,0,+1,yt= 1[rt>0]∈0,1,r_t\;=\;R(s_t,a_t,c_t)\;∈\;\-1,0,+1\, y_t\;=\;1[r_t>0]\;∈\;\0,1\, (1) where ctc_t denotes privileged context that the evaluator R observes but the agent does not at action time. This covers single-step tasks (T=1T=1, with ctc_t a ground-truth reference) and multi-turn tasks (T>1T>1, with ct=st+1c_t=s_t+1 the next environmental state). A trajectory is τ=(st,at,rt)t=1Tτ=\(s_t,a_t,r_t)\_t=1^T; the base methods of §2.2 differ in how rtr_t and ctc_t shape the policy update. 2.2 Retrospective Training Methods Current methods construct the policy update from rtr_t, optionally combined with ctc_t. Scalar-advantage methods such as GRPO Shao et al. (2024) normalize the reward within a group of rollouts sharing the same query; token-level distillation methods such as OPD Agarwal et al. (2024); Ye et al. (2026); Hübotter et al. (2026) use ctc_t to construct a hindsight teacher whose action distribution is distilled into the student; combined methods sum the two signals Wang et al. (2026b). Full advantage formulations are given in App. C.1. All three families share the same hindsight-only property: the learning signal is computed entirely from post-action quantities (rtr_t, ctc_t, or both), and the agent’s prospective belief at action time is invisible to the update. We abstract any such instantiation as a base loss ℓbase ^base whose precise form does not matter for the analysis below. 2.3 The Self-Evaluation Blind Spot Hindsight-only feedback collapses two distinct questions about each step into a single label. The first is observable from the retrospective evaluator: what happened, captured by rtr_t. The second is invisible: whether the agent could have anticipated what would happen when it chose ata_t. Formally, let zt∈−1,0,+1z_t∈\-1,0,+1\ denote the agent’s prospective belief about yty_t, formed from (st,at)(s_t,a_t) alone before ctc_t is observed: zt=+1z_t=+1 encodes an expectation of success, zt=−1z_t=-1 an expectation of failure, and zt=0z_t=0 an explicit acknowledgement of uncertainty (honest uncertainty). The base loss is a function of post-action quantities alone, ℓbase=ℱ(τ,ctt=1T) ^base=F(τ,\c_t\_t=1^T), and therefore independent of any such ztz_t: [∇θℓbase|zt,yt]=[∇θℓbase|yt]∀zt∈−1,0,+1,E\! [ _θ ^base\, |\,z_t,y_t ]\;=\;E\! [ _θ ^base\, |\,y_t ] ∀\,z_t∈\-1,0,+1\, (2) i.e., the gradient distribution depends on the realized outcome yty_t alone and is invariant to the agent’s prospective belief. We call the absence of supervision on ztz_t the self-evaluation blind spot: standard updates assign the same gradient to two rollouts that produced the same outcome, regardless of whether the agent acted with calibrated belief, miscalibrated belief, or honest uncertainty. Two consequences follow on the calibrated subset zt∈−1,+1z_t∈\-1,+1\ (the zt=0z_t=0 case inherits the same invariance and is treated as a neutral class). A confident success and a lucky success receive identical reinforcement, even though the former reflects calibrated competence while the latter is an unrecognized capability the agent cannot deploy with confidence; symmetrically, an overconfident failure is penalized exactly like an aware failure, even though the former is the more dangerous mode — the agent’s own model endorses an action the verifier rejects, so the same mistake is likely to recur. Closing this blind spot requires a learning signal tied to the gap between ztz_t and yty_t, but only if that gap is structurally meaningful rather than noise that should be averaged out. 2.4 Why Prediction–Reality Gaps Are Informative To characterize the structural quantity that the discrete prospective belief ztz_t should approximate, consider its Bayes-optimal continuous idealization, f⋆(st,at):=[yt∣st,at]=Pr(yt=1∣st,at)f (s_t,a_t)\,:=\,E[y_t s_t,a_t]= (y_t=1 s_t,a_t), the minimum-MSE estimator of yty_t available without observing ctc_t. Proposition 1 (Privileged-information gap). For any (st,at)(s_t,a_t) in the support of the rollout distribution, Var(yt∣st,at)=[Var(yt∣st,at,ct)∣st,at]⏟aleatoric noise+I⋆(st,at)⏟privileged-information gap,Var(y_t s_t,a_t)\;=\; E[Var(y_t s_t,a_t,c_t) s_t,a_t]_aleatoric noise\;+\; I (s_t,a_t)_privileged-information gap, (3) where I⋆(st,at):=Var([yt∣st,at,ct]∣st,at)≥0I (s_t,a_t):=Var(E[y_t s_t,a_t,c_t] s_t,a_t)≥ 0. The first term is the aleatoric noise irreducible even with ctc_t; the second, I⋆(st,at)I (s_t,a_t), is precisely the contribution that no action-time predictor f(st,at)f(s_t,a_t) can resolve. States with large I⋆I identify exactly where the agent’s epistemic horizon is most limited and where the prediction–reality gap is informative rather than residual noise. This places our setting in the Learning Under Privileged Information framework Vapnik and Vashist (2009); Pechyony and Vapnik (2010): signals available at training time but not at decision time fundamentally constrain the achievable predictor (proof and full LUPI connection in App. B.1). The decomposition motivates a learning rule: maintain a self-evaluator that produces a prospective belief ztz_t from action-time information alone, and reweight ℓbase ^base as a function of the gap between ztz_t and yty_t. 3 The Prospective Hindsight Method Prospective Hindsight (PH) converts this gap into a usable training signal through three components: a self-evaluator that produces a per-rollout belief from action-time information; a four-cell taxonomy that partitions rollouts by belief and outcome; and a one-line reweighting of the base loss driven by a binary surprise indicator. We show that the same rule has a complementary interpretation as implicit calibration. The end-to-end procedure is summarized in Algorithm 1(App. C). 3.1 Prospective Self-Evaluator The agent πθ _θ produces both the action ata_t and a prospective self-evaluation of the same action. The self-evaluator distribution pθeval(⋅∣st,at)p^eval_θ(· s_t,a_t) is realized by prompting πθ _θ with a template that asks whether the eventual outcome of ata_t will be a success; the answer is parsed from a ⋅· field as a ternary score in −1,0,+1\-1,0,+1\ (success / failure / honest uncertainty; App. C.4). Crucially, the self-evaluator sees only (st,at)(s_t,a_t) and does not observe the privileged context ctc_t (full templates in App. F). To reduce variance, we draw M independent samples per step and aggregate by majority vote: zt=maj(zt(1),…,zt(M)),zt(m)∼i.i.d.pθeval(⋅∣st,at).z_t\;=\;maj\! (z_t^(1),…,z_t^(M) ),z_t^(m) .i.d. p^eval_θ(· s_t,a_t). The only training-time overhead is M extra forward passes per rollout step. 3.2 A Calibration Taxonomy of Rollouts As shown in Table 1, the pair (zt,yt)(z_t,y_t) partitions every rollout with zt∈−1,+1z_t∈\-1,+1\ into one of four calibration cells along two epistemic axes, the outcome axis (yty_t, observable from the verifier) and the anticipation axis (ztz_t, the agent’s prospective self-assessment). Diagonal cells (CS, AF) are calibrated; off-diagonal cells (OF, US) are miscalibrated and capture the asymmetry analyzed in §2.3: OF rollouts are the most consequential failure mode (the agent’s own model endorses an action the verifier rejects), while US rollouts identify capability the agent has but cannot yet recognize. Table 1: Calibration cells of rollouts, indexed by the agent’s prospective belief zt∈−1,+1z_t∈\-1,+1\ and the realized outcome yt∈0,1y_t∈\0,1\. Rollouts with zt=0z_t=0 are deferred to App. C.4. Cell ztz_t yty_t Epistemic interpretation CS (confident success) +1+1 11 calibrated competence OF (overconfident failure) +1+1 00 miscalibrated; agent trusts a wrong action US (underconfident success) −1-1 11 miscalibrated; agent doubts a correct action AF (aware failure) −1-1 00 calibrated incompetence 3.3 The Prospective Hindsight Update Rule PH attaches to each rollout a binary surprise scalar and amplifies the base loss in proportion. Surprise. For zt∈−1,+1z_t∈\-1,+1\ and yt∈0,1y_t∈\0,1\, the per-rollout surprise is the binary indicator of a prediction–reality gap, ξt=[(zt,yt)∈OF,US]∈0,1. _t\;=\;1\! [(z_t,y_t)∈\OF,US\ ]\;∈\;\0,1\. (4) ξt=1 _t=1 exactly when the agent’s prospective belief disagrees with the realized outcome (overconfident or underconfident); ξt=0 _t=0 when they agree (CS or AF). This is the per-rollout realization of the privileged-information gap I⋆I from Proposition 1: rollouts with ξt=1 _t=1 are exactly those whose existence makes I⋆I visible from inside the agent’s own model. Surprise-weighted update. Let ℓtbase(θ) ^base_t(θ) be the per-rollout base loss (§2.2); PH is agnostic to whether the base method is GRPO, OPD, or their combination. The PH-modified loss is ℓtPH(θ)=(1+α⋅ξt)⋅ℓtbase(θ),∇θℓtPH(θ)=(1+α⋅ξt)⋅∇θℓtbase(θ), ^PH_t(θ)\;=\; (1+α· _t )· ^base_t(θ), _θ ^PH_t(θ)\;=\; (1+α· _t )· _θ ^base_t(θ), (5) with α≥0α≥ 0 the only hyperparameter and ξt _t a stop-gradient constant. Calibrated rollouts (ξt=0 _t=0) keep the base loss unchanged; surprising rollouts (ξt=1 _t=1) are upweighted by 1+α1+α. Setting α=0α=0 recovers the base method exactly; we use α=1α=1 as the default. 3.4 PH as Calibration through Reweighting We also develop a complementary interpretation of the same rule as an implicit calibration mechanism. As the prospective evaluator pθevalp^eval_θ shares parameters with the policy πθ _θ, every PH update changes both the action distribution and the agent’s prospective belief; PH biases this joint update toward states in which the two agree. We frame the analysis below as motivation for why PH would be expected to reduce miscalibration as a side effect of optimization, rather than as a formal guarantee. Surprise-weighted decomposition. The PH expected loss decomposes additively into the base objective plus a non-negative surprise residual: ρθ[ℓtPH]=ρθ[ℓtbase]+α⋅ℛ(θ),ℛ(θ):=ρθ[ξt⋅ℓtbase]≥ 0,E_ _θ\! [ ^PH_t ]\;=\;E_ _θ\! [ ^base_t ]\;+\;α·R(θ), (θ)\;:=\;E_ _θ\! [ _t· ^base_t ]\;≥\;0, (6) where ρθ _θ is the joint rollout distribution induced by πθ _θ, pθevalp^eval_θ, and the verifier (App. B.2). The surprise residual ℛ(θ)R(θ) is the base loss restricted to the miscalibrated cells OF and US: PH minimizes a regularized objective in which surprising rollouts contribute through both their own loss and the surprise penalty. Miscalibration rate. Define the agent’s miscalibration rate against the verifier: ℳ(θ):=ρθ[ξt]=Prρθ[(zt,yt)∈OF,US],M(θ)\;:=\;E_ _θ\! [ _t ]\;=\; _ _θ\! [(z_t,y_t)∈\OF,US\ ], (7) which equals the binary Brier score of the deterministic predictor zt′=(zt+1)/2z _t=(z_t+1)/2 against yty_t (App. B.2). Let c(θ):=ρθ[ℓtbase|ξt=1]c(θ)\;:=\;E_ _θ\! [ ^base_t\, |\, _t=1 ] denote the average base loss on surprising rollouts. By the tower rule, ℛ(θ)=c(θ)⋅ℳ(θ),R(θ)\;=\;c(θ)·M(θ), (8) which is an exact identity rather than an inequality (Prop. 3 of App. B.2). Eq. 8 admits two descent pathways for the residual: (i) reducing ℳ(θ)M(θ) by aligning the prospective belief with the verifier outcome, or (i) reducing c(θ)c(θ) by lowering the base loss on the miscalibrated set. Each PH gradient step is one or the other (or both), without an explicit calibration loss ever entering the optimization. When does the residual identity bind? Eq. 8 is informative as a calibration lever in the regime where c(θ)c(θ) remains bounded away from zero. Three training phases follow naturally. Early on, both c(θ)c(θ) and ℳ(θ)M(θ) are large, and minimizing ℛ(θ)R(θ) admits both pathways simultaneously; this is the regime where PH delivers its largest empirical gains (§4). In mid-training, both factors decrease as the agent’s self-model improves. At convergence, c(θ)→0c(θ)→ 0 and ℳ(θ)→0M(θ)→ 0 together, so ℛ(θ)R(θ) vanishes and PH and the base method share the same fixed points (App. B.2). The calibration interpretation is therefore not a uniform statement about the optimization landscape but a description of how PH’s residual decomposes during the operative training window. Reported metric. Among the two cells contributing to ℳ(θ)M(θ), OF carries the dominant practical weight (§2.3). We therefore report the overconfident-failure rate OFR(θ):=Prρθ[yt=0|zt=+1]=Pr[OF]Pr[zt=+1],OFR(θ)\;:=\; _ _θ\! [\,y_t=0\, |\,z_t=+1\, ]\;=\; [OF] [z_t=+1], (9) the OF-conditional projection of ℳ(θ)M(θ), alongside task success. This connects PH to implicit calibration of language models Kadavath et al. (2022); Tian et al. (2023); Lin et al. (2022), in which post-training feedback induces well-calibrated confidence without an explicit calibration loss; PH makes such implicit calibration the explicit driver of optimization rather than an emergent side effect. We emphasize that this connection is diagnostic: PH does not minimize ℳ(θ)M(θ) as an explicit objective, and the alignment between the PH update direction and −∇θℳ(θ)- _θM(θ) is a quantitative motivation rather than a pointwise guarantee (App. B.2). Table 2: Single-turn results. Validation accuracy is the average / maximum logged mean@16 (%); OFR =OF/(CS+OF)=OF/(CS+OF) (failure rate among predicted successes) and Surprise =OF+US=OF+US (total prediction–reality mismatch percentage);Early/Late are the first / last 5 training steps and Δ=Late−Early =Late-Early. Dataset Method Val. Acc. OFR Surprise Avg. ↑ Max ↑ Early Late ↓ Δ ↓ Early Late ↓ Δ ↓ Science Q&A SDPO 63.0 73.4 75.3 34.2 -41.1 70.9 35.2 -35.7 +Random 65.4 74.3 74.1 33.3 -41.7 69.0 35.3 -33.7 +Failure-only 65.1 74.7 73.9 25.0 -48.9 69.0 24.4 -44.6 +PH (α=0.5α=0.5) 65.7 75.5 76.8 20.9 -56.0 71.7 21.8 -49.9 +PH (α=1.0α=1.0) 64.0 74.2 73.2 23.9 -49.3 68.6 21.1 -47.5 +PH (α=2.0α=2.0) 65.8 76.3 74.4 24.9 -49.5 69.9 24.8 -45.1 Tool Use SDPO 57.7 60.2 77.7 40.7 -37.0 72.8 41.6 -31.2 +Random 58.2 61.6 77.4 36.3 -41.0 72.3 36.2 -36.1 +Failure-only 58.1 60.6 76.9 39.3 -37.6 72.7 39.2 -33.5 +PH (α=0.5α=0.5) 57.3 61.0 77.9 36.2 -41.7 73.0 36.2 -36.8 +PH (α=1.0α=1.0) 58.2 61.7 76.6 33.6 -43.0 71.9 33.5 -38.4 +PH (α=2.0α=2.0) 58.2 60.9 77.6 38.2 -39.4 73.0 38.3 -34.7 Figure 2: Self-evaluation dynamics under PH on Science Q&A (α=1α=1). Left: per-step composition of the four PH cells (CS / OF / US / AF); overconfident failures shrink while confident successes grow. Right: the mean PH weight decays as the verifier pass rate rises, indicating that PH acts as an adaptive training signal that anneals as the prediction–reality gap closes. 4 Experiments 4.1 Experimental Setup We evaluate PH in two complementary regimes that stress retrospective signals of different fidelity. Single-turn uses verifiers on Science Q&A Feng et al. (2024) and Tool Use Tang et al. (2023) with SDPO Hübotter et al. (2026) as the base loss. Multi-turn uses an LLM-based PRM judge on the OpenClaw-RL Wang et al. (2026b) Personal Agent task (GPT-4.1 user simulator), where PH plugs into RL (GRPO Shao et al. (2024)), OPD Agarwal et al. (2024); Ye et al. (2026); Hübotter et al. (2026), and their Combined variant. All main runs use OLMo-3-7B-Instruct Olmo et al. (2025) as both policy and self-evaluator (distinct prompts, shared parameters). The prospective predictor uses M=1M=1 sample in single-turn (deterministic verifier) and M=3M=3 in multi-turn (stochastic PRM); cross-scale evidence on gpt-oss-20B Agarwal et al. (2025) and full hyperparameters are deferred to App. D. Baselines and ablations. The boundary case α=0α=0 recovers the base method exactly (Baseline). To separate PH’s surprise-based selection of which rollouts to upweight from the contribution of additional gradient mass, we compare against two strong reweighting controls (single-turn, App. D.3): Random Reweight, which assigns the same per-step total weight as PH but to a uniformly sampled subset of rollouts, and Failure-only Reweight, which upweights all verifier failures (yt=0y_t=0). Both use the same multiplier (1+α)(1+α) as PH, so only the selection rule differs. Metrics and Evaluations. Capability: mean@16 (single-turn) or external-judge score (multi-turn). Calibration: OFR=Pr[yt=0∣zt=+1]OFR= [y_t=0 z_t=+1], surprise rate ℳ(θ)=Pr[ξt=1]M(θ)= [ _t=1], and prospective accuracy PAcc=1−ℳ(θ)PAcc=1-M(θ). Single-turn tables report Early/Late as the average over the first/last 5 training steps; multi-turn tables report Avg over post-initial checkpoints, Best, and Δbest=Best−initial _best=Best-initial. All headline numbers are aggregated over 3 seeds. Table 3: Multi-turn personal-agent results (GPT-4.1 simulator). PH plugged into RL, OPD, and the Combined recipe; performance is the teacher-side personalization-evaluator score, calibration is computed over teacher-side rollouts within mixed training, Δbest=Best−initial _best=Best-initial. Capability Calibration Method Setting Avg. ↑ Best ↑ Δbest _best ↑ OFR ↓ Surprise ↓ Prosp. Acc. ↑ RL GRPO 68.6 77.7 +8.5 5.3 16.5 83.5 +PH (α=0.5α=0.5) 72.6 77.7 +8.7 4.9 15.0 85.0 +PH (α=1.0α=1.0) 75.8 82.9 +14.8 3.1 12.2 87.8 +PH (α=2.0α=2.0) 68.6 77.3 +9.4 2.5 16.0 84.0 OPD SDPO 54.0 65.6 +2.5 15.9 37.2 62.8 +PH (α=0.5α=0.5) 61.8 77.7 +5.6 1.5 13.7 86.3 +PH (α=1.0α=1.0) 69.3 75.2 +2.7 2.6 18.1 81.9 +PH (α=2.0α=2.0) 59.2 77.3 +17.1 5.0 17.2 82.8 Combined RL+OPD 61.2 73.5 +0.8 10.0 40.1 59.9 +PH (α=0.5α=0.5) 66.5 75.6 +8.1 4.3 28.4 71.6 +PH (α=1.0α=1.0) 65.1 78.5 +8.1 6.8 36.0 64.0 +PH (α=2.0α=2.0) 60.3 64.4 +6.0 2.5 29.5 70.5 Figure 3: Capability and calibration dynamics on Science Q&A across α. Left: OFR vs. Validation Accuracy; Middle: prospective accuracy over training. Right: Effect of α on overconfidence rate. 4.2 Single-Turn Verifiable Tasks Across both domains in Table 2, PH preserves or slightly improves SDPO while substantially shrinking calibration error. On Science Q&A, late-training OFR drops from 34.2%34.2\% (SDPO) to 20.9%20.9\% at α=0.5α=0.5, and the surprise rate tracks OFR closely (35.2%→21.8%35.2\%→21.8\%). On Tool Use, α=1.0α=1.0 jointly attains the best capability and calibration (mean@16=58.2/61.7 mean@16=58.2/61.7, OFR 33.6%33.6\%, surprise 33.5%33.5\%). Figure 4 replicates the SDPO vs. SDPO+PH comparison on gpt-oss-20B. Using the same α=1.0α=1.0, PH delivers comparable or larger improvements on all three metrics, showing that it tracks a structural property of hindsight-only training that persists across model scale. Selection, not mass, drives the gain. A natural concern is that PH simply adds gradient mass to a subset of rollouts; any reweighting with the same per-step total weight might do equally well. Random reweighting leaves calibration essentially unchanged on Science Q&A (OFR 33.3%33.3\% vs. SDPO’s 34.2%34.2\%) and improves it only modestly on Tool Use; the gain over SDPO is far smaller than PH’s. Failure-only reweighting recovers most of PH’s OFR reduction on Science Q&A (25.0%25.0\% vs. 20.9%20.9\%) but trails PH on the surprise rate (24.4%24.4\% vs. 21.8%21.8\%). The OFR gap reflects that single-turn miscalibration is OF-dominated (Figure 5), so a 1D outcome-axis baseline already captures most of it; the surprise-rate gap reflects the cells PH amplifies but failure-only misses underconfident successes. PH is a self-extinguishing adaptive curriculum. Figure 2 shows the per-step PH-cell composition and the corresponding PH-weight trajectory on Science Q&A. The overconfident-failure (OF) cell shrinks monotonically across training while the confident-success (CS) cell grows; the underconfident-success (US) and aware-failure (AF) cells stay small throughout. Crucially, the mean PH weight (right) decays as the verifier pass rate rises: PH amplifies gradient signal early when miscalibration is rampant and naturally anneals once the policy has learned to predict its own outcomes, matching the asymptotic argument in App. B.2 (wt→1w_t→1 as ξt→0 _t→0). This means PH does not require a learning-rate schedule for the surprise weight: the data itself controls the curriculum. Calibration emerges as a byproduct of surprise reweighting. Figure 3 (middle) tracks prospective accuracy PAcc(θ)=1−ℳ(θ)PAcc(θ)=1-M(θ) across training. The pre-training self-evaluator is essentially uncalibrated but PH lifts PAccPAcc from ∼30% 30\% to ∼80% 80\% while the SDPO baseline plateaus near 73%73\%. The predictor receives no direct gradient (the reweighting is stop-gradient on ξt _t); the gap is therefore a byproduct of policy optimization, evidence that PH co-trains a calibrated self-model alongside the policy at no additional parameter cost. This is the mechanism behind the simultaneous capability and calibration improvement: the same gradient steps that reduce ℓbase ^base on miscalibrated rollouts also drive ℳ(θ)→0M(θ)→0 via the parameter-sharing channel. The full α-sweep confirms the entire window improves both axes simultaneously, with α=0.5α=0.5 attaining the lowest mean OFR at matched capability. Figure 4: Capability and calibration gains from PH transfer across model scale using SDPO baseline and SDPO+PH (α=1.0α=1.0) on OLMo-3-7B-Instruct and gpt-oss-20B. 4.3 Multi-Turn Personal Agent Multi-turn shifts the dominant miscalibration mode from OF to US. Before discussing capability and calibration numbers, we characterize the kind of miscalibration the multi-turn regime produces. Figure 5 aggregates (zt,yt)(z_t,y_t) pairs across training under α=1.0α=1.0 and contrasts single-turn tasks with the three multi-turn base methods. The picture is structurally different: while single-turn rollouts are dominated by overconfident failures (OF=34.1%=34.1\%, US=2.7%=2.7\%), multi-turn rollouts shift the dominant miscalibration to underconfident successes (US), which grow monotonically across base methods, RL (13.8%13.8\%) << OPD (27.9%27.9\%) << Combined (34.5%34.5\%), while OF stays uniformly small (44–5%5\%). The two regimes therefore expose complementary failure modes: single-turn agents over-trust wrong actions, while multi-turn agents under-trust correct ones. Figure 5: PH cell composition differs structurally between single-turn and multi-turn settings. PH improves all three base methods, with the largest lift on OPD. At α=1.0α=1.0, PH improves the trajectory-averaged teacher-side score over the baseline in every block of Table 3: RL (+7.2+7.2), OPD (+15.3+15.3), and Combined (+3.9+3.9). The OPD block is the most striking: its baseline has the weakest absolute performance, and PH simultaneously stabilizes the trajectory (+15.3+15.3 Avg) and raises the peak. The cell-composition picture above is consistent with the OPD-vs-RL ordering: OPD’s privileged-context teacher leaves the student with substantially more underconfident successes than RL (US: 27.9%27.9\% vs. 13.8%13.8\%), and PH amplifies precisely those rollouts. The Combined block is a partial exception to this ordering since its US share (34.5%34.5\%) is the highest of the three, yet its PH gain is the smallest. Calibration improves uniformly at α=1.0α=1.0. At the headline α=1.0α=1.0, OFR drops in every block and prospective accuracy rises in every block, peaking at 87.8%87.8\% (RL+PH, α=1.0α=1.0) and 86.3%86.3\% (OPD+PH, α=0.5α=0.5). The surprise gate concentrates gradient mass on exactly the cells (OF, US) that contribute to ℳ(θ)M(θ), and those cells empirically shrink under PH training. A per-turn breakdown (Figure 8) shows that PH compresses overconfidence most in late-turn positions, where the agent’s self-model is least developed. Larger α trades stability for further OFR reduction: at α=2.0α=2.0, the RL block reaches the lowest OFR in the table (2.5%2.5\%) but the predictor collapses toward a near-constant zt=−1z_t=-1 output (Surprise rate 16.0%16.0\%, prospective accuracy 84.0%84.0\%), so the calibration improvement comes at the cost of a degraded self-model rather than from genuinely better belief–outcome agreement. Replacing the GPT-4.1 user simulator with GPT-5.4 preserves the qualitative pattern of Table 3 (see Figure 9). What PH is doing, across regimes. The base method’s gradient is invariant to the agent’s prospective belief; the surprise indicator ξt _t is the per-rollout realization of the privileged-information gap I⋆I that this invariance hides. Three observations across both regimes are consistent with this view: (i) calibration improves as a byproduct of optimization, so the same gradient steps that drive the base loss also reduce ℳ(θ)M(θ); (i) the dominant miscalibration cell flips between regimes yet the same update rule covers both; and (i) the surprise weight self-extinguishes as the prediction–reality gap closes, so PH asymptotically reduces to the base method. Together these suggest that what PH amplifies is not a particular failure mode but the existence of a prediction–reality discrepancy itself, which is why the same scalar α transfers across tasks, base methods, and model scales without re-tuning. 5 Related work Retrospective training signals in Agentic RL. Recent RL/agentic RL frameworks have steadily enriched the post-action signal delivered to the policy Zhang et al. (2025a); Cui et al. (2026). OpenClaw-RL Wang et al. (2026b) unifies binary RL (GRPO Shao et al. (2024), PPO Schulman et al. (2017)), on-policy distillation (OPD Agarwal et al. (2024); Ye et al. (2026); Hübotter et al. (2026); Shenfeld et al. (2026); Sang et al. (2026); Kim et al. (2026)), and their combination under a single asynchronous training stack Zhao et al. (2026); Ding (2026); Buening et al. Buening et al. (2026) further show that raw multi-turn user interactions are themselves sufficient supervision for self-distillation without an explicit reward model. These methods differ in mechanism but share a structural property: every learning signal is computed entirely from quantities observed after the agent has acted, so the agent’s prospective belief at action time never enters the gradient Agarwal et al. (2024); Ye et al. (2026); Wang et al. (2026a). Prospective Hindsight is orthogonal to this axis of progress: it leaves the base method’s rollout, advantage, and regularization structure intact, and instead converts the gap between an action-time prospective prediction and the eventual retrospective evaluation into a per-rollout reweighting that plugs into any of the above base methods. Calibration of language-model agents. Most prior work treats calibration as a property of the trained model to be measured or controlled at inference time, through diagnostic studies of confidence–accuracy alignment Guo et al. (2017); Kadavath et al. (2022); Geng et al. (2024), elicitation of verbalized confidence or sampling-based uncertainty estimates Tian et al. (2023); Lin et al. (2022); Kuhn et al. (2023), trajectory-level uncertainty quantification in agentic settings Kirchhof et al. (2025); Duan et al. (2025); Zhao et al. (2025); Liu et al. (2024); Han et al. (2024); Zhang et al. (2026a); Zhang et al. (2026d), and recent analyses linking hallucination to training procedures that reward confident guesses over abstention Kalai et al. (2025). A separate line moves calibration into RL by adding an explicit confidence-target objective: CaOPD Zhang et al. (2026c) attributes overconfidence in hindsight distillation to teacher–student information asymmetry and proposes confidence-token replacement; Leng et al. Leng et al. (2024) and Damani et al. Damani et al. (2025) report similar calibration-augmented RL recipes. These approaches consistently report a capability–calibration trade-off Yang et al. (2026): improved calibration comes at the cost of a regression in task accuracy. PH proceeds differently: rather than adding a calibration term to compete with the task objective, we use the per-rollout disagreement between the agent’s prospective belief and the verifier outcome to reweight which base-method gradients deserve more mass. Calibration improves as a byproduct of the same gradient steps that drive task success, and the trade-off observed in calibration-augmented baselines does not arise because calibration and capability share a single descent direction. 6 Conclusion Prospective Hindsight treats the gap between an agent’s prospective belief and the verifier’s retrospective outcome as a learning resource rather than a diagnostic, amplifying the gradient on exactly the rollouts where retrospective signals are uninformative. The same principle should extend beyond the binary verifiers and per-turn PRMs studied here, to longer horizons such as software engineering, terminal-use, GUI, and research agents, and to base methods beyond GRPO and on-policy distillation. We see PH as a first but critical step toward reliable agent-RL pipelines that optimize task performance and self-calibration jointly, by construction rather than through a separate calibration objective. References [1] R. Agarwal, N. Vieillard, Y. Zhou, P. Stanczyk, S. R. Garea, M. Geist, and O. Bachem (2024) On-policy distillation of language models: learning from self-generated mistakes. In The twelfth international conference on learning representations, Cited by: Appendix A, §C.1, §1, §2.2, §4.1, §5. [2] S. Agarwal, L. Ahmad, J. Ai, S. Altman, A. Applebaum, E. Arbus, R. K. Arora, Y. Bai, B. Baker, H. Bao, et al. (2025) Gpt-oss-120b & gpt-oss-20b model card. arXiv preprint arXiv:2508.10925. Cited by: §D.1, §4.1. [3] T. K. Buening, J. Hübotter, B. Pásztor, I. Shenfeld, G. Ramponi, and A. Krause (2026) Aligning language models from user interactions. arXiv preprint arXiv:2603.12273. Cited by: Appendix A, §1, §1, §5. [4] F. Cui, R. Zhu, C. Fang, S. Li, and J. Li (2026) Rethinking agentic reinforcement learning in large language models. arXiv preprint arXiv:2604.27859. Cited by: §5. [5] M. Damani, I. Puri, S. Slocum, I. Shenfeld, L. Choshen, Y. Kim, and J. Andreas (2025) Beyond binary rewards: training lms to reason about their uncertainty. arXiv preprint arXiv:2507.16806. Cited by: Appendix A, §1, §1, §1, §5. [6] K. Ding (2026) HDPO: hybrid distillation policy optimization via privileged self-distillation. arXiv preprint arXiv:2603.23871. Cited by: §5. [7] G. Dong, H. Mao, K. Ma, L. Bao, Y. Chen, Z. Wang, Z. Chen, J. Du, H. Wang, F. Zhang, et al. (2025) Agentic reinforced policy optimization. arXiv preprint arXiv:2507.19849. Cited by: §1. [8] J. Duan, J. Diffenderfer, S. Madireddy, T. Chen, B. Kailkhura, and K. Xu (2025) UProp: investigating the uncertainty propagation of llms in multi-step agentic decision-making. arXiv preprint arXiv:2506.17419. Cited by: Appendix A, §5. [9] K. Feng, X. Shen, W. Wang, X. Zhuang, Y. Tang, Q. Zhang, and K. Ding (2024) Sciknoweval: evaluating multi-level scientific knowledge of large language models. arXiv preprint arXiv:2406.09098. Cited by: §D.1, §4.1. [10] J. Geng, F. Cai, Y. Wang, H. Koeppl, P. Nakov, and I. Gurevych (2024) A survey of confidence estimation and calibration in large language models. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), p. 6577–6595. Cited by: Appendix A, §5. [11] T. Groot and M. Valdenegro-Toro (2024) Overconfidence is key: verbalized uncertainty evaluation in large language and vision-language models. arXiv preprint arXiv:2405.02917. Cited by: §1. [12] C. Guo, G. Pleiss, Y. Sun, and K. Q. Weinberger (2017) On calibration of modern neural networks. In International conference on machine learning, p. 1321–1330. Cited by: Appendix A, §1, §5. [13] J. Han, W. Buntine, and E. Shareghi (2024) Towards uncertainty-aware language agent. In Findings of the Association for Computational Linguistics ACL 2024, p. 6662–6685. Cited by: Appendix A, §5. [14] J. Hübotter, F. Lübeck, L. Behric, A. Baumann, M. Bagatella, D. Marta, I. Hakimi, I. Shenfeld, T. K. Buening, C. Guestrin, et al. (2026) Reinforcement learning via self-distillation. arXiv preprint arXiv:2601.20802. Cited by: Appendix A, §C.1, §D.1, Appendix D, §1, §2.2, §4.1, §5. [15] S. Kadavath, T. Conerly, A. Askell, T. Henighan, D. Drain, E. Perez, N. Schiefer, Z. Hatfield-Dodds, N. DasSarma, E. Tran-Johnson, et al. (2022) Language models (mostly) know what they know. arXiv preprint arXiv:2207.05221. Cited by: Appendix A, §1, §3.4, §5. [16] A. T. Kalai, O. Nachum, S. S. Vempala, and E. Zhang (2025) Why language models hallucinate. arXiv preprint arXiv:2509.04664. Cited by: Appendix A, §1, §5. [17] J. Kim, X. Luo, M. Kim, S. Lee, D. Kim, J. Jeon, D. Li, and Y. Yang (2026) Why does self-distillation (sometimes) degrade the reasoning capability of llms?. arXiv preprint arXiv:2603.24472. Cited by: §5. [18] M. Kirchhof, G. Kasneci, and E. Kasneci (2025) Position: uncertainty quantification needs reassessment for large language model agents. In Forty-second International Conference on Machine Learning Position Paper Track, Cited by: Appendix A, §1, §5. [19] L. Kuhn, Y. Gal, and S. Farquhar (2023) Semantic uncertainty: linguistic invariances for uncertainty estimation in natural language generation. In The Eleventh International Conference on Learning Representations, Cited by: Appendix A, §5. [20] W. Kwon, Z. Li, S. Zhuang, Y. Sheng, L. Zheng, C. H. Yu, J. Gonzalez, H. Zhang, and I. Stoica (2023) Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th symposium on operating systems principles, p. 611–626. Cited by: §D.1. [21] J. Leng, C. Huang, B. Zhu, and J. Huang (2024) Taming overconfidence in LLMs: reward calibration in RLHF. arXiv preprint arXiv:2410.09724. Cited by: Appendix A, §1, §1, §1, §5. [22] S. Lin, J. Hilton, and O. Evans (2022) Teaching models to express their uncertainty in words. arXiv preprint arXiv:2205.14334. Cited by: Appendix A, §3.4, §5. [23] H. Liu, Z. Dou, Y. Wang, N. Peng, and Y. Yue (2024) Uncertainty calibration for tool-using language agents. In Findings of the Association for Computational Linguistics: EMNLP 2024, p. 16781–16805. Cited by: Appendix A, §5. [24] D. Lopez-Paz, L. Bottou, B. Schölkopf, and V. Vapnik (2015) Unifying distillation and privileged information. arXiv preprint arXiv:1511.03643. Cited by: Appendix A, §B.1, §1. [25] K. Lu and T. M. Lab (2025) On-policy distillation. Thinking Machines Lab: Connectionism. Note: https://thinkingmachines.ai/blog/on-policy-distillation External Links: Document Cited by: Appendix A, Appendix A, §1. [26] A. Madaan, N. Tandon, P. Gupta, S. Hallinan, L. Gao, S. Wiegreffe, U. Alon, N. Dziri, S. Prabhumoye, Y. Yang, et al. (2023) Self-refine: iterative refinement with self-feedback. Advances in Neural Information Processing Systems 36, p. 46534–46594. Cited by: Appendix A, 4th item. [27] P. Manakul, A. Liusie, and M. Gales (2023) Selfcheckgpt: zero-resource black-box hallucination detection for generative large language models. In Proceedings of the 2023 conference on empirical methods in natural language processing, p. 9004–9017. Cited by: Appendix A. [28] C. Oh, S. Park, T. E. Kim, J. Li, W. Li, S. Yeh, X. Du, H. Hassani, P. Bogdan, D. Song, et al. (2026) Uncertainty quantification in llm agents: foundations, emerging challenges, and opportunities. arXiv preprint arXiv:2602.05073. Cited by: §1. [29] T. Olmo, A. Ettinger, A. Bertsch, B. Kuehl, D. Graham, D. Heineman, D. Groeneveld, F. Brahman, F. Timbers, H. Ivison, et al. (2025) Olmo 3. arXiv preprint arXiv:2512.13961. Cited by: §D.1, §4.1. [30] D. Pechyony and V. Vapnik (2010) On the theory of learnining with privileged information. Advances in neural information processing systems 23. Cited by: Appendix A, §B.1, §1, §2.4. [31] H. Sang, Y. Xu, Z. Zhou, R. He, Z. Wang, and J. Sun (2026) On-policy self-distillation for reasoning compression. arXiv preprint arXiv:2603.05433. Cited by: §5. [32] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov (2017) Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347. Cited by: §5. [33] Z. Shao, P. Wang, Q. Zhu, R. Xu, J. Song, X. Bi, H. Zhang, M. Zhang, Y. Li, Y. Wu, et al. (2024) Deepseekmath: pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300. Cited by: Appendix A, §C.1, §D.2, §2.2, §4.1, §5. [34] I. Shenfeld, M. Damani, J. Hübotter, and P. Agrawal (2026) Self-distillation enables continual learning. arXiv preprint arXiv:2601.19897. Cited by: Appendix A, §1, §5. [35] G. Sheng, C. Zhang, Z. Ye, X. Wu, W. Zhang, R. Zhang, Y. Peng, H. Lin, and C. Wu (2025) Hybridflow: a flexible and efficient rlhf framework. In Proceedings of the Twentieth European Conference on Computer Systems, p. 1279–1297. Cited by: §D.1. [36] N. Shinn, F. Cassano, A. Gopinath, K. Narasimhan, and S. Yao (2023) Reflexion: language agents with verbal reinforcement learning. Advances in Neural Information Processing Systems 36, p. 8634–8652. Cited by: Appendix A, 4th item. [37] M. Song and M. Zheng (2026) A survey of on-policy distillation for large language models. arXiv preprint arXiv:2604.00626. Cited by: §1. [38] Q. Tang, Z. Deng, H. Lin, X. Han, Q. Liang, B. Cao, and L. Sun (2023) Toolalpaca: generalized tool learning for language models with 3000 simulated cases. arXiv preprint arXiv:2306.05301. Cited by: §D.1, §4.1. [39] K. Tian, E. Mitchell, A. Zhou, A. Sharma, R. Rafailov, H. Yao, C. Finn, and C. D. Manning (2023) Just ask for calibration: strategies for eliciting calibrated confidence scores from language models fine-tuned with human feedback. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, p. 5433–5442. Cited by: Appendix A, §1, §3.4, §5. [40] V. Vapnik and A. Vashist (2009) A new learning paradigm: learning using privileged information. Neural networks 22 (5-6), p. 544–557. Cited by: Appendix A, §B.1, §1, §2.4. [41] J. Wang, W. Zhang, W. Shi, Y. Li, and J. Cheng (2026) TCOD: exploring temporal curriculum in on-policy distillation for multi-turn autonomous agents. arXiv preprint arXiv:2604.24005. Cited by: §5. [42] Y. Wang, X. Chen, X. Jin, M. Wang, and L. Yang (2026) OpenClaw-rl: train any agent simply by talking. arXiv preprint arXiv:2603.10165. Cited by: Appendix A, §C.1, §D.2, §D.2, §D.2, §D.2, Appendix D, 3rd item, §1, §1, §2.2, §4.1, §5. [43] S. Yang, C. Wu, C. Lin, Y. Chen, H. Lee, and S. Sun (2026) On calibration of large language models: from response to capability. arXiv preprint arXiv:2602.13540. Cited by: §5. [44] T. Ye, L. Dong, X. Wu, S. Huang, and F. Wei (2026) On-policy context distillation for language models. arXiv preprint arXiv:2602.12275. Cited by: Appendix A, §C.1, §1, §2.2, §4.1, §5. [45] G. Zhang, H. Geng, X. Yu, Z. Yin, Z. Zhang, Z. Tan, H. Zhou, Z. Li, X. Xue, Y. Li, et al. (2025) The landscape of agentic reinforcement learning for llms: a survey. arXiv preprint arXiv:2509.02547. Cited by: §1, §5. [46] G. Zhang, J. Wang, J. Chen, W. Zhou, K. Wang, and S. Yan (2025) AgenTracer: who is inducing failure in the llm agentic systems?. arXiv preprint arXiv:2509.03312. Cited by: §1. [47] J. Zhang, P. K. Choubey, K. Huang, C. Xiong, and C. Wu (2026) Agentic uncertainty quantification. arXiv preprint arXiv:2601.15703. Cited by: §1, §5. [48] J. Zhang, W. Cui, Z. Li, L. Huang, B. A. Malin, C. Xiong, and C. Wu (2026) From passive metric to active signal: the evolving role of uncertainty quantification in large language models. In Findings of the Association for Computational Linguistics: ACL 2026, p. 41525–41544. Cited by: Appendix A. [49] J. Zhang, Z. Li, K. Das, B. Malin, and S. Kumar (2023) SAC3: reliable hallucination detection in black-box language models via semantic-aware cross-check consistency. In Findings of the Association for Computational Linguistics: EMNLP 2023, p. 15445–15458. Cited by: Appendix A. [50] J. Zhang, X. Peng, Q. Chen, Q. Ye, C. Xiong, and C. Wu (2026) The illusion of certainty: decoupling capability and calibration in on-policy distillation. arXiv preprint arXiv:2604.16830. Cited by: Appendix A, §D.1, §D.1, §1, §1, §5. [51] J. Zhang, C. Xiong, and C. Wu (2026) Agentic confidence calibration. arXiv preprint arXiv:2601.15778. Cited by: §1, §5. [52] S. Zhang, M. Yin, J. Zhang, J. Liu, Z. Han, J. Zhang, B. Li, C. Wang, H. Wang, Y. Chen, et al. (2025) Which agent causes task failures and when? on automated failure attribution of llm multi-agent systems. arXiv preprint arXiv:2505.00212. Cited by: §1. [53] Q. Zhao, D. Li, Y. Liu, W. Cheng, Y. Sun, M. Oishi, T. Osaki, K. Matsuda, H. Yao, C. Zhao, et al. (2025) Uncertainty propagation on llm agent. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 6064–6073. Cited by: Appendix A, §5. [54] S. Zhao, Z. Xie, M. Liu, J. Huang, G. Pang, F. Chen, and A. Grover (2026) Self-distilled reasoner: on-policy self-distillation for large language models. arXiv preprint arXiv:2601.18734. Cited by: Appendix A, Appendix A, §5. [55] K. Zhu, Z. Liu, B. Li, M. Tian, Y. Yang, J. Zhang, P. Han, Q. Xie, F. Cui, W. Zhang, et al. (2025) Where llm agents fail and how they can learn from failures. arXiv preprint arXiv:2509.25370. Cited by: §1. Contents Contents Appendix A Extended Related Work and Discussion Agent RL: from outcome rewards to process supervision. Reinforcement learning for language-model agents has progressed from sparse outcome-only rewards (e.g., GRPO [33] applied at the trajectory level) to dense, step-level supervision delivered by process reward models and structured frameworks. OpenClaw-RL [42] provides a unified asynchronous training stack that combines binary RL, on-policy distillation, and their composition in one rollout–score–update loop, together with a per-turn PRM that grades each turn from the next environmental state. Buening et al. [3] push this further by showing that raw multi-turn user interactions are already sufficient supervision for self-distillation, eliminating the need for a separately trained reward model. Across these systems, the per-turn signal varies considerably in fidelity and modality: rule-based binary verifiers for verifiable single-turn tasks, learned scalar PRMs for soft multi-turn signals, and hindsight-teacher token distributions for OPD [25, 54]. What does not vary is the structural property that motivates this paper: every learning signal is computed from quantities observable only after the action is committed, and the agent’s prospective belief at action time is invisible to the gradient. Prospective Hindsight is orthogonal to the choice of base method, leaves the post-action signal untouched, and adds a single per-rollout reweighting on top. Hindsight distillation and the OPD lineage. On-policy distillation [1] introduced the idea of distilling a teacher’s improved actions into the student under on-policy rollouts; subsequent work has extended this in several directions. Ye et al. [44] formalize on-policy context distillation, in which the teacher conditions on a privileged context ctc_t that the student does not see at deployment; Hübotter et al. [14] and Shenfeld et al. [34] develop self-distillation variants in which the teacher and student share parameters and the privileged context is constructed on-the-fly from verified-correct rollouts. We position PH as a complementary mechanism in this lineage. Standard OPD [25] uses privileged information to construct better actions for the teacher, and the resulting per-token signal is the dominant training signal. PH instead uses the gap between privileged and non-privileged evaluations to identify which rollouts deserve more weight, and absorbs OPD’s per-token signal into a single per-rollout multiplier. The two mechanisms can be applied jointly without interaction: in our multi-turn experiments PH is plugged on top of GRPO, OPD, and the OpenClaw-RL Combined recipe with no method-specific tuning. Calibration and self-awareness in LLMs. Modern confidence calibration in LLMs draws on several lines of work. At the diagnostic level, Guo et al. [12] pioneered the study of miscalibrated softmax outputs in deep networks, and Kadavath et al. [15] systematically measured whether language models “know what they know” across knowledge tasks. Methods to elicit calibrated confidence include verbalized confidence prompts [39, 22] and sampling-based estimators of semantic uncertainty [19]; Geng et al. [10] provide a survey. At the agentic level, Kirchhof et al. [18] argue for trajectory-level calibration diagnostics, and a series of recent papers [8, 53, 23, 48] quantify uncertainty propagation through multi-step trajectories; Han et al. [13] use uncertainty as an inference-time control signal. Kalai et al. [16] take a different angle and argue that hallucination itself is incentivized by training procedures that reward confident guesses over calibrated abstention, linking training-procedure design to overconfidence at the model-output level. This literature largely treats calibration as a diagnostic property to measure or to control at inference, rather than as a training signal to optimize. PH inverts that relationship: it converts the binary self-evaluation mismatch into a per-rollout training-time weight, and the binary Brier score of the agent’s self-evaluator coincides with the miscalibration rate that the PH residual lower-bounds. Calibration emerges as a side effect of the same gradient steps that drive task success, rather than as a post-hoc check or a separate auxiliary loss. Calibration-augmented RL and the capability–calibration trade-off. A line of recent work specifically targets the overconfidence introduced by hindsight distillation. CaOPD [50] attributes this overconfidence to the information asymmetry between the privileged-context teacher and the student and proposes explicit confidence-token replacement in the teacher distribution; Leng et al. [21] and Damani et al. [5] study calibration-augmented RL recipes that add explicit confidence supervision to the loss. Across these approaches, calibration improvements typically come at the cost of a modest capability regression, and the calibration objective is a separate term added to the base RL loss. PH is complementary along two dimensions. First, the base method is left untouched: PH does not rewrite confidence targets, does not add a separate calibration loss, and does not require a calibration-specific reward model. Second, the per-rollout reweighting connects calibration and capability through the surprise residual decomposition: a gradient step that reduces the base loss on miscalibrated rollouts and a gradient step that reduces the miscalibration rate are both valid descent directions on. Empirically, this removes the capability–calibration trade-off seen in the cited baselines: PH preserves capability while substantially cutting OFR on single-turn and lifts both the trajectory-averaged and best-checkpoint scores across all three multi-turn base methods. Privileged information and prediction-error signals. Our framing of the prediction–reality gap connects directly to Vapnik’s Learning Under Privileged Information framework [40, 30]. In LUPI, training examples are accompanied by privileged information that is not available at deployment, and the goal is a predictor that uses standard inputs alone. Lopez-Paz et al. [24] unify LUPI with knowledge distillation, showing that distillation can be viewed as a transfer of privileged information from teacher to student. Our agent-RL setting instantiates LUPI with (st,at)(s_t,a_t) as the standard input, yty_t as the target, and ctc_t as the privileged signal. Standard OPD-style methods use ctc_t to construct improved actions for the teacher; PH instead uses the evaluation gap that ctc_t creates, formalized as the non-negative privileged-information gap I⋆(st,at)I (s_t,a_t) in Proposition 1. To our knowledge this is the first explicit use of I⋆I as a sample-level training signal in agent RL: the surprise indicator ξt _t is the per-rollout realization of I⋆I from inside the agent’s own model, and upweighting surprising rollouts concentrates gradient budget exactly where action-time information is insufficient. A related but distinct line is intrinsic-motivation RL, in which prediction errors between two predictors of the next state drive exploration of novel states. PH differs in three ways. First, the prediction error is computed in the outcome / evaluation space, not the state space, and is used for training-signal prioritization rather than for action selection. Second, the predictor is the same model as the policy, parameterized by the same θ and queried via a different prompt, rather than a separate frozen network. Third, the surprise scalar is stop-gradient: it modulates the magnitude of the base gradient on the existing on-policy rollouts but does not introduce a new gradient component of its own. The two perspectives are complementary — one amplifies novelty in state space, the other amplifies prediction errors in evaluation space — and we view their combination as a natural avenue for future work. Self-correction, self-refinement, and reasoning chains. A parallel line of work uses model self-evaluation to drive inference-time behavior: Reflexion [36] and Self-Refine [26] iterate on a draft response using a self-critic. PH operates at training time rather than inference time; the prospective prediction is consumed once, used to compute ξt _t, and discarded after the gradient step. This is a deliberate design choice: PH does not change deployment behavior, does not lengthen the agent’s effective context, and does not require the agent to reason about its own confidence at inference. Inference-time self-correction is therefore composable with PH (it operates on the policy distribution after PH has shaped it), but the two mechanisms target different stages of the pipeline. A second related line is hallucination characterization (e.g., [27, 49, 54]), which studies the conditions under which LLMs produce unsupported content; we view PH’s reduction of overconfident-failure rate as operating in the same regime, but at the agent-RL training-loop level rather than at the per-output classification level. Appendix B Theoretical Details and Proofs B.1 Proof of Proposition 1 Fix (st,at)(s_t,a_t) and apply the law of total variance to yty_t with conditioning random variable ctc_t: Var(yt∣st,at) (y_t s_t,a_t) =[Var(yt∣st,at,ct)|st,at] =E\! [Var(y_t s_t,a_t,c_t)\, |\,s_t,a_t ] +Var([yt∣st,at,ct]|st,at) +Var\! (E[y_t s_t,a_t,c_t]\, |\,s_t,a_t ) =[Var(yt∣st,at,ct)|st,at]+I⋆(st,at), =E\! [Var(y_t s_t,a_t,c_t)\, |\,s_t,a_t ]+I (s_t,a_t), (10) where the second equality uses the definition of I⋆I . Both terms are non-negative, and the second vanishes if and only if [yt∣st,at,ct]E[y_t s_t,a_t,c_t] is almost surely a function of (st,at)(s_t,a_t), that is, ctc_t provides no additional information about yty_t beyond (st,at)(s_t,a_t). □ Connection to Learning Under Privileged Information The decomposition in Proposition 1 places the agent RL setting of §2.1 inside the Learning Under Privileged Information (LUPI) paradigm introduced by [40]. In LUPI, training examples are accompanied by privileged information available only at training time, and the goal is a predictor that uses standard inputs alone. [30] formalize the statistical-learning theory of this setting. [24] unify LUPI with knowledge distillation, showing that distillation can be viewed as a transfer of privileged information from teacher to student. Our setting instantiates LUPI with (st,at)(s_t,a_t) as the standard input, yty_t as the target, and ctc_t as the privileged signal. The base methods of §2.2 use ctc_t in different ways: GRPO uses ctc_t only through the scalar reward rt=R(st,at,ct)r_t=R(s_t,a_t,c_t); OPD-style methods condition the hindsight teacher distribution on ctc_t and distill to a student that drops ctc_t at inference. Both retain the blind spot of §2.3: training never asks the policy to predict yty_t from (st,at)(s_t,a_t) before ctc_t arrives, so the privileged-information gap I⋆(st,at)I (s_t,a_t) is never measured per step. Our method (§3) makes this gap an explicit training signal. Equivalent information-theoretic statement. Equation (3) can equivalently be expressed in entropy form. For binary yty_t, let H(y∣⋅):=−[logp(y∣⋅)]H(y ·):=-E[ p(y ·)]. Then H(yt∣st,at)−H(yt∣st,at,ct)=I(yt;ct|st,at)≥ 0,H\! (y_t s_t,a_t )-H\! (y_t s_t,a_t,c_t )\;=\;I\! (y_t;c_t\, |\,s_t,a_t )\;≥\;0, (11) that is, the conditional mutual information between yty_t and ctc_t given (st,at)(s_t,a_t) measures the same privileged-information increment as I⋆I . The equality I(yt;ct∣st,at)=0I(y_t;c_t s_t,a_t)=0 characterizes the degenerate case in which I⋆(st,at)=0I (s_t,a_t)=0 for almost every (st,at)(s_t,a_t): no prospective predictor can be improved by access to ctc_t at training time. B.2 PH as Calibration through Reweighting: Detailed Analysis This appendix provides the formal setup, the exact residual identity and its consequences, a gradient-flow analysis of the PH update direction, and a discussion of the assumptions’ empirical validity. We frame the results below as motivation for the calibration interpretation of PH outlined in §3.4, not as formal guarantees of monotone calibration descent. Setup and notation We work in a single-step task to keep notation light; the multi-turn extension is recorded in §B.2. Let θ∈Θθ∈ parameterize a shared autoregressive model that defines: • the policy πθ(a∣s) _θ(a s), a conditional distribution over actions a∈a given state s∈s ; • the prospective self-evaluator pθeval(z∣s,a)p^eval_θ(z s,a), a conditional distribution over evaluator outputs z∈−1,0,+1z∈\-1,0,+1\. The verifier y is sampled from a parameter-independent kernel penv(y∣s,a)p_env(y s,a) with y∈0,1y∈\0,1\; the state distribution is ρ0(s) _0(s). The joint rollout distribution induced by θ is ρθ(s,a,z,y)=ρ0(s)πθ(a∣s)pθeval(z∣s,a)penv(y∣s,a). _θ(s,a,z,y)\;=\; _0(s)\, _θ(a s)\,p^eval_θ(z s,a)\,p_env(y s,a). (12) We restrict attention to z∈−1,+1z∈\-1,+1\; the residual z=0z=0 class is treated separately in App. C.4 and contributes neutrally to all expressions below since ξt=0 _t=0 there. The surprise indicator (Eq. 4 of the main text) is ξ(z,y)=[(z,y)∈OF,US],ξ(z,y)\;=\;1\! [(z,y)∈\OF,US\ ], (13) and we write ξt _t when the rollout is implicit. We assume the base loss ℓtbase(θ) ^base_t(θ) is non-negative; this is satisfied by all retrospective methods of §2.2 after additive constant shifts that do not affect the gradient. Surprise-weighted decomposition Proposition 2 (Surprise-weighted decomposition). For any α≥0α≥ 0, ρθ[ℓtPH]=ρθ[ℓtbase]+α⋅ℛ(θ),ℛ(θ):=ρθ[ξt⋅ℓtbase].E_ _θ\! [ ^PH_t ]\;=\;E_ _θ\! [ ^base_t ]+α·R(θ), (θ):=E_ _θ\! [ _t· ^base_t ]. The surprise residual ℛ(θ)R(θ) is non-negative and vanishes if and only if ξt⋅ℓtbase=0 _t· ^base_t=0 holds ρθ _θ-almost surely. Proof. By Eq. 5 of the main text, ℓtPH=(1+αξt)ℓtbase=ℓtbase+αξtℓtbase ^PH_t=(1+α _t)\, ^base_t= ^base_t+α _t ^base_t. Linearity of expectation gives the decomposition. Non-negativity of ℛR follows from ξt∈0,1 _t∈\0,1\ and ℓtbase≥0 ^base_t≥ 0. The vanishing condition is the standard a.s. zero condition for non-negative integrands. ∎ The two pathways for residual minimization—reducing Pr[ξ=1] [ξ=1] (calibration) or reducing ℓbase ^base on miscalibrated rollouts (base-objective improvement on those rollouts)—are formalized exactly in the next subsection. Exact residual identity Proposition 3 (Exact residual identity). Define the agent’s miscalibration rate ℳ(θ):=ρθ[ξt]=Prρθ[(zt,yt)∈OF,US],M(θ)\;:=\;E_ _θ\! [ _t ]\;=\; _ _θ\! [(z_t,y_t)∈\OF,US\ ], (14) and the conditional mean base loss on surprising rollouts c(θ):=ρθ[ℓtbase|ξt=1].c(θ)\;:=\;E_ _θ\! [ ^base_t\, |\, _t=1 ]. (15) Then ℛ(θ)=c(θ)⋅ℳ(θ).R(θ)\;=\;c(θ)·M(θ). (16) Proof. By the tower rule, ℛ(θ) (θ) =[ξtℓtbase]=[ℓtbase|ξt=1]⋅Pr[ξt=1]=c(θ)⋅ℳ(θ).∎ \;=\;E\! [ _t ^base_t ]\;=\;E\! [ ^base_t\, |\, _t=1 ]· [ _t=1 ]\;=\;c(θ)·M(θ). Eq. 16 is an exact identity, not an inequality: the residual factors cleanly into a base-loss-on-miscalibrated component c(θ)c(θ) and a miscalibration-rate component ℳ(θ)M(θ). As a corollary, whenever c(θ)≥c0>0c(θ)≥ c_0>0 on a contiguous training window, ℛ(θ)≥c0⋅ℳ(θ)R(θ)≥ c_0·M(θ) on that window, and minimizing ℛ(θ)R(θ) via PH must reduce ℳ(θ)M(θ) at rate at least c0c_0 relative to the rate of residual descent. When does Eq. 16 drive calibration? The identity is most informative as a calibration lever when both factors are non-negligible. Three regimes arise as training progresses: • Early training. c(θ)c(θ) is large (the base method has not yet learned to solve surprising rollouts) and ℳ(θ)M(θ) is large (many rollouts are miscalibrated). Both pathways for residual minimization are active, and PH’s surprise reweighting allocates additional gradient budget to cells where the prediction–reality gap is widest. • Mid training. c(θ)c(θ) and ℳ(θ)M(θ) decrease together as the agent’s self-model and the base policy co-evolve. The dual-pathway decomposition continues to hold, but the per-step weight of each pathway shifts. • Convergence. c(θ)→0c(θ)→ 0 and ℳ(θ)→0M(θ)→ 0 jointly, ℛ(θ)→0R(θ)→ 0, and PH and the base method share the same fixed points (App. B.2). The identity becomes uninformative because both sides are zero, but PH itself is no longer doing work in this regime. The bound ℛ(θ)≥c0⋅ℳ(θ)R(θ)≥ c_0·M(θ) is therefore not a uniform property of the optimization landscape: it characterizes the operative training window during which PH’s calibration gains accrue. Equivalence to the binary Brier score. For binary zt′=(zt+1)/2∈0,1z _t=(z_t+1)/2∈\0,1\ and yt∈0,1y_t∈\0,1\, (zt′−yt)2=(zt′≠yt)=ξt(z _t-y_t)^2=1(z _t≠ y_t)= _t, so ℳ(θ)M(θ) coincides with the standard binary Brier score [(zt′−yt)2]E[(z _t-y_t)^2] of the deterministic predictor zt′z _t against the verifier yty_t. We use the term miscalibration rate throughout because it more directly conveys the operational meaning—the fraction of rollouts on which the agent’s prospective belief disagrees with the verifier outcome—and because the experimental section reports a one-sided projection of ℳM rather than the full Brier score (§B.2). Connection to the overconfident-failure rate (OFR) The miscalibration rate ℳ(θ)M(θ) in Eq. 14 aggregates both miscalibrated cells, OF and US, symmetrically. The experimental section reports the OF-conditional projection OFR(θ):=Prρθ[yt=0|zt=+1]=Pr[OF]Pr[CS]+Pr[OF]=Pr[OF]Pr[zt=+1].OFR(θ)\;:=\; _ _θ\! [y_t=0\, |\,z_t=+1 ]\;=\; [OF] [CS]+ [OF]\;=\; [OF] [z_t=+1]. (17) This is a one-sided, conditional version of ℳM focused on the OF cell, which is the dominant miscalibration mode singled out in §2.3. The two metrics are related by Pr[OF]≤ℳ(θ),OFR(θ)=Pr[OF]/Pr[zt=+1], [OF]\;≤\;M(θ), (θ)\;=\; [OF]\,/\, [z_t=+1], (18) so reductions in ℳ(θ)M(θ) imply reductions in Pr[OF] [OF] whenever Pr[US] [US] does not increase, and reductions in Pr[OF] [OF] imply reductions in OFR(θ)OFR(θ) whenever Pr[zt=+1] [z_t=+1] does not collapse. Both monotonicity assumptions hold in our experiments: the prospective predictor’s confidence rate Pr[zt=+1] [z_t=+1] stays in a stable range across training, and the underconfident-success rate Pr[US] [US] remains a small fraction of ℳ(θ)M(θ) at all checkpoints. We therefore report OFR(θ)OFR(θ) as a faithful empirical proxy for ℳ(θ)M(θ) in §4. Gradient-flow analysis: when is the PH update calibration-aligned? We now characterize the direction in parameter space along which PH gradient flow moves, relative to the negative gradient of the miscalibration rate. The result below should be read as a motivation for why one would expect PH to reduce ℳ(θ)M(θ), not as a guarantee that it does so monotonically in every step. Let uPH:=−∇θρθ[ℓPH]u^PH:=- _θE_ _θ[ ^PH] and ubase:=−∇θρθ[ℓbase]u^base:=- _θE_ _θ[ ^base] denote the PH and base update directions. By Prop. 2, uPH=ubase−α∇θℛ(θ).u^PH\;=\;u^base-α _θR(θ). (19) Score-function decomposition of ∇θℛ _θR. Applying the score-function trick to ℛ(θ)=ρθ[ξtℓtbase]R(θ)=E_ _θ[ _t ^base_t], ∇θℛ(θ)=ρθ[ξt∇θℓtbase]⏟base-loss descent on ξ=1+ρθ[ξtℓtbase∇θlogρθ]⏟distributional shift in ξ, _θR(θ)\;=\; E_ _θ[ _t _θ ^base_t]_base-loss descent on \ξ=1\\;+\; E_ _θ[ _t ^base_t _θ _θ]_distributional shift in ξ, (20) where ∇θlogρθ=∇θlogπθ(a∣s)+∇θlogpθeval(z∣s,a) _θ _θ= _θ _θ(a s)+ _θ p^eval_θ(z s,a), since ρ0 _0 and penvp_env are parameter-free. Similarly, ∇θℳ(θ)=ρθ[ξt∇θlogρθ]. _θM(θ)\;=\;E_ _θ[ _t _θ _θ]. (21) Theorem 4 (PH gradient under uniform-loss assumptions). Let ℓ¯(θ):=ρθ[ℓtbase∣ξt=1]=c(θ) (θ):=E_ _θ[ ^base_t _t=1]=c(θ) and δ(θ):=Varρθ,ξt=1[ℓtbase]δ(θ):=Var_ _θ,\, _t=1[ ^base_t]. Suppose [ξt‖∇θlogρθ‖2]<∞E[ _t\| _θ _θ\|^2]<∞. Then uPH=ubase−α⋅ρθ[ξt∇θℓtbase]−αℓ¯(θ)⋅∇θℳ(θ)+r(θ),u^PH\;=\;u^base\;-\;α·E_ _θ[ _t _θ ^base_t]\;-\;α\, (θ)· _θM(θ)\;+\;r(θ), (22) where the remainder r(θ)r(θ) satisfies ‖r(θ)‖≤αδ(θ)⋅[ξt‖∇θlogρθ‖2].\|r(θ)\|\;≤\;α δ(θ)· E[ _t\| _θ _θ\|^2]. (23) In the limit δ(θ)→0δ(θ)→ 0 (the base loss is constant on surprising rollouts), the PH update direction differs from the base direction by exactly a −αℓ¯⋅∇θℳ(θ)-α · _θM(θ) component plus a base-loss descent component on the miscalibrated set. Proof. Decompose the second term of Eq. 20 via the identity [ξtℓtbase∇θlogρθ]=ℓ¯⋅[ξt∇θlogρθ]+[ξt(ℓtbase−ℓ¯)∇θlogρθ].E[ _t ^base_t _θ _θ]\;=\; ·E[ _t _θ _θ]\;+\;E [ _t( ^base_t- ) _θ _θ ]. The first term equals ℓ¯⋅∇θℳ(θ) · _θM(θ) by Eq. 21. The second term is bounded in norm by Cauchy–Schwarz: ‖[ξt(ℓtbase−ℓ¯)∇θlogρθ]‖≤Varξt=1[ℓtbase]⋅[ξt‖∇θlogρθ‖2]=δ(θ)⋅[ξt‖∇θlogρθ‖2], \|E[ _t( ^base_t- ) _θ _θ] \|\;≤\; Var_ _t=1[ ^base_t]· E[ _t\| _θ _θ\|^2]\;=\; δ(θ)· E[ _t\| _θ _θ\|^2], which gives Eq. 23 after multiplication by α. Substituting back into Eqs. 20 and 19 yields Eq. 22. ∎ Remark 1 (Empirical magnitude of the remainder). The remainder bound in Eq. 23 is not an assumption but a quantity measurable from training logs. We define the relative remainder ρ(θ):=δ(θ)c(θ)=Varξt=1[ℓtbase]ξt=1[ℓtbase],ρ(θ)\;:=\; δ(θ)c(θ)\;=\; Var_ _t=1[ ^base_t]E_ _t=1[ ^base_t], (24) the coefficient of variation of the base loss conditional on ξt=1 _t=1. Theorem 4 is most informative when ρ(θ)ρ(θ) is small; the limit ρ(θ)→0ρ(θ)→ 0 recovers the uniform-loss case in which the leading term −αℓ¯⋅∇θℳ-α · _θM dominates the remainder. Calibration alignment, qualified. Taking the inner product of Eq. 22 with −∇θℳ(θ)- _θM(θ) gives ⟨uPH,−∇θℳ(θ)⟩ u^PH,- _θM(θ) =⟨ubase,−∇θℳ(θ)⟩ \;=\; u^base,- _θM(θ) +αℓ¯(θ)⋅∥∇θℳ(θ)∥2 +α (θ)·\| _θM(θ)\|^2 −α⟨[ξt∇θℓtbase],−∇θℳ(θ)⟩ -α [ _t _θ ^base_t],- _θM(θ) +⟨r(θ),−∇θℳ(θ)⟩. + r(θ),- _θM(θ) . (25) Among the four terms on the right-hand side, the second is the leading calibration-aligned term and is non-negative; the fourth is bounded by αδ(θ)⋅[ξt‖∇θlogρθ‖2]⋅‖∇θℳ(θ)‖α δ(θ)· E[ _t\| _θ _θ\|^2]·\| _θM(θ)\| via Eq. 23. The third term is a base-loss descent component on the miscalibrated set; its inner product with −∇θℳ- _θM is sign-indeterminate in general, but in policy-gradient base losses ∇θℓtbase _θ ^base_t is bounded by the per-batch advantage normalization of §2.2, so its contribution scales with the magnitude of the base advantage. The PH update direction is therefore biased toward −∇θℳ(θ)- _θM(θ) rather than exactly aligned with it. Whether this bias is large enough to drive monotone descent on ℳ(θ)M(θ) depends on the relative magnitudes of the four terms in Eq. 25, which are themselves a function of the training step. We do not claim monotone descent; we claim that the leading term provides a calibration-aligned restoring force that is consistent with the empirical observation in §4 that OFR(θ)OFR(θ) decreases monotonically under PH training. Scope and limitations of the alignment claim. We make four honest qualifications. First, Theorem 4 is an exact identity, but the calibration-alignment interpretation that follows it requires the leading ℓ¯⋅∇θℳ · _θM term to dominate the remainder—i.e., ρ(θ)ρ(θ) to be small relative to ℓ¯/‖∇θℳ‖ /\| _θM\|. Second, the [ξt∇θℓtbase]E[ _t _θ ^base_t] term has indeterminate sign relative to −∇θℳ(θ)- _θM(θ), and we do not claim it contributes to calibration descent. Third, the analysis is gradient-flow, not finite-step SGD: under standard learning-rate and smoothness assumptions, the conclusions hold approximately for finite steps with higher-order corrections, but we have not derived sharp rates. Fourth, the analysis is per-step: it does not compose across the trajectory of training to yield a convergence-rate result on ℳ(θ)M(θ). We therefore frame Theorem 4 as a motivation for why PH’s update direction would be expected to reduce ℳ(θ)M(θ), not as a guarantee that it does so. The empirical observation that OFR(θ)OFR(θ) decreases monotonically under PH training across all base methods and tasks in §4 is then evidence that the motivation translates into practice, rather than a derivation of that fact. Discussion Single-step versus multi-turn. The arguments above treat each rollout as a single (s,a,z,y)(s,a,z,y) tuple. In multi-turn settings, the per-turn surprise ξt _t is computed from (zt,yt)(z_t,y_t) at turn t, and the joint distribution ρθ _θ extends to a sequence over turns. Prop. 2 extends turn-by-turn (linearity of expectation across turns), and Theorem 4 applies at each turn under the same assumptions; the per-turn alignment is then aggregated over the trajectory. The remainder bound in Eq. 23 is typically larger in multi-turn settings because the per-turn variance of the base loss is amplified by stochasticity in the LLM-based PRM judge (App. D.2). Tabular versus neural-network policies. Theorem 4 is stated as an exact gradient-flow identity modulo the explicit remainder r(θ)r(θ). For finite-step gradient descent with neural-network policies, the conclusion holds approximately under standard learning-rate, smoothness, and Lipschitz assumptions, with equalities replaced by inequalities up to higher-order terms in the learning rate. Asymptotic behavior. As θ converges, both ℓbase ^base and ξ approach zero on the support of ρθ _θ, so c(θ)→0c(θ)→ 0, ℳ(θ)→0M(θ)→ 0, and ℛ(θ)→0R(θ)→ 0. By Prop. 2, PH and the base method share the same fixed points; PH differs only in the optimization trajectory leading up to convergence, in which additional gradient budget is allocated to surprising rollouts via Eq. 16. The miscalibration rate also vanishes asymptotically, consistent with calibration being achieved at convergence. Relation to existing calibration losses. The miscalibration rate ℳ(θ)M(θ) is the binary specialization of mean-squared-error calibration (the Brier score). Expected Calibration Error (ECE), defined for predictors that emit confidence p∈[0,1]p∈[0,1], coincides with ℳ(θ)M(θ) when predictions are binarized via thresholding. Negative Log-Likelihood of the prospective evaluator is a different but related calibration objective; PH does not directly minimize NLL, but does so implicitly through the parameter-sharing channel that couples the policy and evaluator gradients. Among these losses, ℳM is the one most naturally connected to the PH update rule, since PH’s reweighting is itself an indicator-weighted scheme. The OF-conditional projection OFR(θ)OFR(θ) used in our experiments (Eq. 17) is a one-sided form of ℳM tailored to the OF-focused narrative of §2.3. Appendix C Algorithm and Implementation Details C.1 Base Method Advantage Formulations We give the explicit advantage formulations of the three retrospective base methods abstracted as ℓbase ^base in §2.2. GRPO [33]. For a group of G rollouts (st,at(g),rt(g))g=1G\(s_t,a_t^(g),r_t^(g))\_g=1^G sharing the same query, the per-rollout advantage is AtGRPO=rt−r¯σr+ϵ,A^GRPO_t\;=\; r_t- r _r+ε, (26) where r¯=1G∑grt(g) r= 1G _gr_t^(g) and σr _r is the group standard deviation; ϵ>0ε>0 is a small constant for numerical stability. OPD [1, 44, 14]. The privileged context ctc_t (in multi-turn, ct=st+1c_t=s_t+1) is used to construct a hindsight teacher conditioned on (st,ct)(s_t,c_t), whose per-token action distribution is distilled into the student: At,jOPD=logπθ(at,j∣at,<j,st,ct)−logπθ(at,j∣at,<j,st),A^OPD_t,j\;=\; _θ(a_t,j a_t,<j,s_t,c_t)- _θ(a_t,j a_t,<j,s_t), (27) where at,ja_t,j is the j-th token of ata_t and at,<ja_t,<j is its prefix. Combined [42]. The combined advantage sums the scalar and token-level signals: At,jcomb=wrlAtGRPO+wopdAt,jOPD,A^comb_t,j\;=\;w_rlA^GRPO_t+w_opdA^OPD_t,j, (28) with wrl=wopd=1w_rl=w_opd=1 in our experiments. In all three cases, the per-rollout (or per-token) base loss ℓtbase ^base_t is the policy-gradient surrogate of the corresponding advantage; the precise form does not affect any analysis in §2–§3. How PH wraps each base method. PH’s per-rollout reweighting in Eq. 5 preserves all internal structure of the base advantage. For GRPO, the group normalization σr _r is computed on the original rewards rtr_t, not the surprise-weighted loss. For OPD, the per-token signal At,jOPDA^OPD_t,j is aggregated as in the base method; PH then multiplies the resulting per-rollout (single-turn) or per-turn (multi-turn) loss by (1+αξt)(1+α _t). The surprise scalar is broadcast across action tokens within the rollout/turn, which is the operation that absorbs OPD’s per-token signal into the per-rollout reweighting. C.2 PH Algorithm Details Algorithm 1 states the end-to-end training step. Inside any base method of §2.2, PH inserts three operations into each rollout step: (i) draw M samples from the prospective self-evaluator and aggregate them by majority vote (§3.1); (i) compute the surprise indicator ξt _t from the resulting ztz_t and the verifier outcome yty_t (§3.2, §3.3); (i) multiply the base per-rollout loss by 1+αξt1+α _t (§3.3). The base optimizer then updates the parameters using the resulting PH loss; the rollout sampling distribution, the group reward normalization, the per-token aggregation in OPD, and any auxiliary regularizer in ℓbase ^base are all unchanged. Algorithm 1 Prospective Hindsight (PH) training step. 1: parameters θ, batch of prompts xbb=1B\x_b\_b=1^B, base method (GRPO, OPD, or Combined), hyperparameters α≥0α≥ 0 and M≥1M≥ 1 2: for each prompt xbx_b, b=1,…,Bb=1,…,B do 3: for each step (or turn) t of the rollout do 4: at∼πθ(⋅∣st)a_t _θ(· s_t) ⊳ base method’s action sampling 5: for m=1,…,Mm=1,…,M do 6: zt(m)∼pθeval(⋅∣st,at)z_t^(m) p^eval_θ(· s_t,a_t) ⊳ Eq. 3.1, prospective forward pass 7: end for 8: zt←maj(zt(1),…,zt(M))z_t (z_t^(1),…,z_t^(M) ) 9: Receive ctc_t and yty_t from environment and verifier 10: ξt←[(zt,yt)∈OF,US] _t 1\! [(z_t,y_t)∈\OF,US\ ] ⊳ Eq. 4 11: Compute ℓtbase(θ) ^base_t(θ) following §2.2 12: ℓtPH←(1+α⋅ξt)⋅ℓtbase(θ) ^PH_t← (1+α· _t )· ^base_t(θ) ⊳ Eq. 5 13: end for 14: end for 15: θ←optimizer_step(θ,∇θ∑b,tℓtPH)θ← optimizer\_step\! (θ,\; _θ _b,\,t ^PH_t ) ⊳ ξt _t is stop-gradient 16: return θ Cost. The only additional compute cost relative to the base method is M prospective forward passes per rollout step (line 5 of Algorithm 1). With shared parameters between the action policy and the self-evaluator, no extra trainable parameters are introduced, so PH does not change the memory footprint of the base method beyond the storage of ztz_t and ξt _t per step. With M=1M=1, the prospective overhead is one extra forward pass per step. The verifier and privileged-context computations on line 8 are inherited unchanged from the base method, since the retrospective methods of §2.2 already require them. PH thus adds a single training-time hyperparameter α and a single inference-time hyperparameter M on top of the base method. Special cases. Two boundary cases are useful for sanity checks. First, α=0α=0 collapses Algorithm 1 into the base method exactly, since 1+αξt=11+α _t=1 for all t. Second, M=1M=1 removes the majority vote and uses a single prospective sample per rollout step, which is the configuration in all our single-turn experiments. C.3 Discrete and Continuous PH Gates The PH update rule of Eq. 5 weights the base loss by wt=1+αξtw_t=1+α\, _t for a non-negative surprise scalar ξt _t. We use two concrete realizations of ξt _t, depending on whether the retrospective signal is deterministic or stochastic; both share the same shape wt∈[1,1+α]w_t∈[1,1+α] and the same asymptotic behaviour wt→1w_t→ 1 as the prospective belief and the realized outcome agree. Discrete gate (single-turn). With a rule-based binary verifier yt∈0,1y_t∈\0,1\ and a single ternary prospective sample zt∈−1,0,+1z_t∈\-1,0,+1\ (M=1M=1, App. D.1), the surprise indicator is the binary cell membership of Eq. 4, ξtdisc=[(zt,yt)∈OF,US]∈0,1,wt= 1+αξtdisc∈1, 1+α. _t^disc\;=\;1\! [(z_t,y_t)∈\OF,US\ ]\;∈\;\0,1\, w_t\;=\;1+α\, _t^disc\;∈\;\1,\,1+α\. (29) Honest-uncertainty rollouts (zt=0z_t=0) are folded into ξt=0 _t=0 to preserve the neutral-class semantics of §C.4, so wt=1w_t=1 on zt=0\z_t=0\. Continuous gate (multi-turn). With an LLM-based per-turn PRM judge that emits an aggregated turn score rt∈−1,0,+1r_t∈\-1,0,+1\ and M predictor samples averaged into z¯t∈[−1,+1] z_t∈[-1,+1] (App. D.2), we use the continuous relaxation ξtcont=12|z¯t−rt|∈[0,1],wt= 1+αξtcont∈[1, 1+α]. _t^cont\;=\; 12\, | z_t-r_t |\;∈\;[0,1], w_t\;=\;1+α\, _t^cont\;∈\;[1,\,1+α]. (30) The factor 12 12 rescales the maximum disagreement |z¯t−rt|=2| z_t-r_t|=2 (i.e., z¯t=+1 z_t=+1, rt=−1r_t=-1 or vice versa) to ξt=1 _t=1. Eq. 30 reduces to Eq. 29 on the corners of the (z¯t,rt)( z_t,r_t) grid: when both z¯t∈−1,+1 z_t∈\-1,+1\ and rt∈−1,+1r_t∈\-1,+1\, ξtcont=ξtdisc _t^cont= _t^disc. Intermediate values of z¯t z_t encode predictor uncertainty (averaged over M samples) and intermediate values of rtr_t encode PRM-judge uncertainty; the continuous gate carries this uncertainty through to wtw_t rather than collapsing it to 0,1\0,1\. Self-extinguishing curriculum. Both gates share the limiting behaviour wt→1w_t→ 1 as ξt→0 _t→ 0, i.e., as the prospective belief and the realized outcome converge. This is the mechanism behind the empirical PH-weight decay reported in Figs. 2 and 7: as training closes the prediction–reality gap, the prospective signal naturally fades and the PH loss reduces to the base loss. By Proposition 2, the surprise residual ℛ(θ)→0R(θ)→ 0 in the same limit, so PH and the base method share the same fixed points (App. B.2). Implementation flag. A single training-time flag SURPRISE_MODE selects between the two gates. Setting SURPRISE_MODE=discrete uses Eq. 29; setting SURPRISE_MODE=continuous uses Eq. 30. The single-turn experiments of §4.2 use the discrete gate; the multi-turn experiments of §4.3 use the continuous gate. All other PH-side hyperparameters (α, M, weight clip) are independent of this choice. C.4 Honest Uncertainty: The zt=0z_t=0 Class The prospective self-evaluator of §3.1 is prompted to return a ternary score zt∈−1,0,+1z_t∈\-1,0,+1\; the value zt=0z_t=0 encodes honest uncertainty: an explicit signal that the agent’s prospective belief about yty_t from (st,at)(s_t,a_t) alone is inconclusive. This appendix records our handling of this residual class. Treatment as a neutral class. Rollouts with zt=0z_t=0 inherit the gradient invariance of Eq. 2 but do not fit the four-cell calibration taxonomy of Table 1, because the agent has not committed to a directional prediction. The PH update rule (§3.3) assigns weight 1.01.0 to all such rollouts, matching the weight given to the calibrated cells (CS, AF) so that zt=0z_t=0 neither amplifies nor dampens the base loss. Tie-breaking under majority vote. When M>1M>1 samples in Eq. 3.1 produce a strict majority for one of −1,+1\-1,+1\, we use that value. Under a tie, or when all M samples are unparseable (e.g., the model fails to emit a valid ⋅· field), we default to zt=0z_t=0. This conservative rule prevents PH from amplifying ambiguous rollouts and folds parsing failures into the neutral class. Empirical frequency. Across our main experiments, zt=0z_t=0 accounts for a small minority of rollouts in both single-step and multi-turn settings, with the exact rate varying by task and model. The frequency is largely independent of the base method (GRPO, OPD, Combined) and decreases as M increases in the majority vote, since aggregating over multiple samples collapses borderline cases toward the dominant prediction. Table 4: Hyperparameters for multi-turn personal agent: Combined (Binary RL + OPD) and Combined + PH on OpenClaw-RL. Parameter Value Models, GPU layout Backbone (student / PRM / teacher) OLMo-3-7B-Instruct (shared) Total GPUs 8 (1 node) Actor 4 GPUs, TP=4, sequence-parallel Rollout (sglang) 2 GPUs, TP=1 each PRM judge (sglang) 1 GPU OPD teacher (sglang+Megatron) 1 GPU Rollout / context Rollout batch size 1616 Samples per prompt 11 (multi-turn rollout = full session) Rollout temperature 0.60.6 Max response length / turn 81928192 tokens Max context length 3276832768 tokens Max tokens per GPU 3276832768 Conversation Sessions per round 22 Training rounds 1616 Max turns per session TmaxT_ 4848 Scenario schedule alternate Student / Teacher each round Training problems (offsets) GSM8K 36:∞36:∞ Eval problems (offsets) 0:360:36 student, 0:240:24 teacher PRM judge / OPD teacher PRM majority vote MprmM_prm 11 PRM temperature 0.60.6 PRM max new tokens 20482048 EMA teacher update rate 0.050.05 OPD teacher source Megatron actor checkpoints Combined advantage / PPO surrogate Advantage estimator GRPO wrlw_rl, wopdw_opd 1.01.0, 1.01.0 PPO clip εlo/εhi _lo/ _hi 0.20.2 / 0.280.28 KL loss coefficient 0.00.0 Entropy coefficient 0.00.0 Prospective Hindsight Surprise mode continuous Predictor sample size M 33 (averaged into z¯t z_t; PRM noise) Predictor max new tokens 256256 Surprise amplification α 1.01.0 default; sweep 0.5,1.0,2.0\0.5,1.0,2.0\ Surprise weight clip off (gate already in [1,1+α][1,1+α]) Predictor gradient stop-gradient Optimization Optimizer Adam (Megatron) Learning rate 1×10−61×10^-6 LR decay style constant Weight decay 0.10.1 Adam β1,β2 _1, _2 0.9,0.980.9,0.98 Save interval (in weight-update steps) 2525 Train epochs (data passes) 11 Appendix D Additional Experimental Setup This appendix expands §4.1 with all information needed to reproduce our experiments. §D.1 covers the single-turn verifiable tasks (Science Q&A and Tool Use) built on top of the SDPO [14] backbone, and §D.2 covers the multi-turn personal agent built on top of the OpenClaw-RL [42] framework. §D.4 summarizes compute, software environment, and code release. The two regimes differ along one important axis — single-turn uses a rule-based binary verifier with no learned reward model, while multi-turn uses an OLMo-3-7B-Instruct PRM judge — which explains the configuration differences below. D.1 Single-Turn Verifiable Tasks Setting in one sentence. Each rollout is a single prompt x and a single model response a, with a rule-based verifier R(x,a)∈0,1R(x,a)∈\0,1\ that returns binary correctness and no auxiliary reward model; this is the simplest possible instantiation of PH (single prediction, single outcome, T=1T=1, deterministic retrospective signal). Datasets and verifiers. We use the Chemistry subset of SciKnowEval [9] following the released CaOPD [50] preprocessing. Each prompt asks the model to reason step-by-step and emit a single letter y^∈A,B,C,D y∈\A,B,C,D\; the verifier is rule-based, R(x,a)=1R(x,a)=1 iff the parsed letter equals the gold answer. We also use the Tool Use split derived from ToolAlpaca [38] as preprocessed in CaOPD; each prompt provides API documentation in a ReAct-style template and the verifier checks (a) the action name matches the gold tool and (b) the JSON arguments match the gold schema under canonical normalization. Both training sets contain 44–55k prompts; we evaluate on the corresponding test split. Together they probe both knowledge-intensive reasoning (Science Q&A) and structured agentic prediction (Tool Use), and were the two domains used in CaOPD, which lets our calibration numbers be directly compared against theirs. Backbone OPD method (SDPO) and privileged-context construction. We build the single-turn experiments on top of SDPO [14], using the official codebase as modified in CaOPD [50]. SDPO is an on-policy self-distillation method: the same model serves as both student πθ(⋅|x) _θ(·|x) and teacher πθ(⋅|x,z) _θ(·|x,z), where z is a privileged context not available at deployment. We use the SDPO “correct-solution” mode of CaOPD: for each prompt, after the rollout phase, one verified-correct rollout a⋆a is chosen on-the-fly and inserted as z, so the teacher is then asked to “Correctly solve the original question” conditioned on (x,z)(x,z). The training loss is per-token reverse KL between student and teacher on the teacher trajectory, with rollout importance sampling correction and SDPO defaults (top-K=100K=100 distillation, EMA teacher update rate 0.050.05, IS clip 22); thinking mode is disabled (enable_thinking=false) so that the trajectory we distill is the answer trajectory. Models, training framework, and hardware. Backbones are OLMo-3-7B-Instruct [29] (primary, both Science Q&A and Tool Use) and gpt-oss-20B [2] (scale ablation). Both are used as student and EMA-teacher (the EMA teacher is the same model with the privileged context z). We deliberately keep the verbalized-confidence template of CaOPD intact so that PH plugs in cleanly: each rollout still ends with a parseable Confidence: x token, but PH ignores this string and uses only the binary verifier outcome R(x,a)R(x,a) as the retrospective signal. The codebase forks CaOPD’s SDPO branch, implemented on top of verl [35] with PyTorch FSDP2 for sharded optimizer state and vLLM [20] for batched rollout generation; our PH plug-in adds (i) a per-prompt prospective prediction call, (i) a stop-gradient weight w(ξt)w( _t) on the per-rollout reverse-KL contribution, and (i) lightweight surprise/OFR bookkeeping in the training logger. Each run uses one node of p5en.48xlarge (8×8× NVIDIA H200), which is sufficient for both 7B and 20B-MoE training under FSDP2. Prospective Hindsight plug-in. For each rollout, after the student emits its answer a but before the verifier evaluates R(x,a)R(x,a), we issue an additional generation call to the same student model with a different prompt that strips the verbalized-confidence instruction and asks the model to predict whether its previous answer is correct, incorrect, or uncertain. The exact predictor template is given in App. F. Because the verifier here is rule-based and deterministic, a single ternary prediction zt∈−1,0,+1z_t∈\-1,0,+1\ per rollout already gives a low-variance signal, so we use M=1M=1 predictor sample (no majority vote). Predictor generation is capped at max_new_tokens=128=128 tokens at temperature 0.60.6, top-p=0.95p=0.95; un-parseable outputs are treated as zt=0z_t=0 (uncertain). The surprise indicator and weight are ξt=[(zt,yt)∈OF,US],wt=1+αξt, _t=1 [(z_t,y_t)∈\OF,US\ ], w_t=1+α _t, with α=1.0α=1.0 default and clip wt∈[0.5,2.0]w_t∈[0.5,2.0] as a safety net (the unclipped weight is in 1,2\1,2\ already, so the clip is rarely active). The weight is a stop-gradient scalar and multiplies the SDPO per-rollout reverse-KL loss; no gradient flows back through the predictor. Hyperparameters. Table 5 reports the hyperparameters for all single-turn runs. All values are taken either from the CaOPD defaults or from a one-shot tuning on a held-out validation slice; no hyperparameter is tuned per-method (the same configuration is used for SDPO, SDPO + PH, and the α-sweep ablation). Table 5: Hyperparameters for single-turn SDPO and SDPO + PH on Science Q&A and Tool Use. Parameter Value Models, data, and inference Backbone models OLMo-3-7B-Instruct; gpt-oss-20B (ablation) Thinking mode disabled (enable_thinking=false) Max prompt length 20482048 tokens Max response length 81928192 tokens Inference engine vLLM Rollout (training) Question batch size 3232 Rollouts per question KtrainK_train 88 Rollout temperature 1.01.0 Validation Rollouts per question KvalK_val 1616 (val_kwargs.n, “mean@16”) Validation temperature 0.60.6 Validation top-p 0.950.95 SDPO loss (CaOPD defaults) Distillation top-K 100100 Distillation divergence token-level reverse KL EMA teacher update rate 0.050.05 Rollout IS clip 22 Prospective Hindsight (this work) Predictor max new tokens 128128 Predictor temperature / top-p 0.60.6 / 0.950.95 Predictor sample size M 11 (deterministic verifier; no majority vote) Surprise amplification α 1.01.0 default; sweep 0.0,0.25,0.5,1.0,2.0,5.0\0.0,0.25,0.5,1.0,2.0,5.0\ Weight clip [wmin,wmax][w_ ,w_ ] [0.5,2.0][0.5,2.0] Predictor gradient stop-gradient Optimization Optimizer AdamW Learning rate 1×10−51×10^-5 LR warmup steps 1010 Weight decay 0.10.1 Gradient clip norm 1.01.0 PPO mini-batch size 3232 Total epochs 33 Hardware Hardware ×1\!×\! p5en.48xlarge (×8\!× H200) Sharding FSDP2, bf16 Evaluation protocol and metrics. Capability is reported as mean@16: for each held-out test prompt we sample Kval=16K_val=16 rollouts at temperature 0.60.6, top-p=0.95p=0.95, evaluate the verifier on each, and report the mean accuracy. From the same Kval=16K_val=16 rollouts we compute, per prompt and then averaged across the test set, (i) the surprise rate ℳ^=Pr[ξt=1] M= [ _t=1], (i) the overconfident-failure rate Pr[yt=0∣zt=+1] [y_t=0 z_t=+1], and (i) the honest-uncertainty rate Pr[zt=0] [z_t=0] (App. C.4, where we verify that the neutral cell does not collapse under PH). All headline numbers are mean± over 33 seeds, reported at the end of epoch 3; the α-sweep additionally logs per-step trajectories of mean@16 and OFR (App. E.1). D.2 Multi-Turn Personal Agent Setting in one sentence. A trained student model serves as an OpenAI-compatible chat agent inside the OpenClaw runtime; an external LLM “user” (GPT-4.1) drives multi-turn math-tutoring conversations grounded in GSM8K, an OLMo-3-7B-Instruct PRM judges each main-line turn, and PH attaches a per-turn surprise signal to the OpenClaw-RL training loop. Task and agent design. We follow the personal-agent track of OpenClaw-RL [42] verbatim. At each training round we alternate between two scenarios: (i) a Student scenario in which the simulated user is a “lazy student” who needs the agent to solve a math problem in a non-AI-style tone (no bold, no bullet points); the student writes the problem to homework/i.txt, asks the agent to solve, critiques the style, and ends with the literal sentinel DONE once satisfied; and (i) a Teacher scenario in which the simulated user is a strict math teacher who wants warm, specific, prose-only grading comments. The agent has access to the OpenClaw runtime’s standard file-I/O tools but only text generation is exercised in this benchmark. Each session is one (problem, scenario) pair with up to Tmax=48T_ =48 turns, ending whenever the simulator emits DONE; turns inside a session are tagged turn_type=main (used for training) or turn_type=side (eval-only). We split GSM8K into disjoint training and evaluation pools (offsets 36:∞36:∞ training; 0:360:36 student-eval, 0:240:24 teacher-eval), with sessions_per_round=2 sessions\_per\_round=2 and 1616 training rounds. User simulators and conversation protocol. The simulators are GPT-4.1 calls (temperature 00) with the system prompts taken verbatim from the OpenClaw-RL evaluation script; the full strings are reproduced in App. F. The salient content of the Student prompt is: “You are role-playing as a lazy student who knows nothing… If the AI’s answer has stuff like bold text, numbered lists, Final answer:, or anything too AI-like, tell it to rewrite… After the AI shows you a satisfactory version, say exactly: DONE.” The Teacher prompt asks for warm, specific, prose-only grading comments and ends similarly with DONE. The first user message in each scenario is templated from the GSM8K problem and (for the Teacher scenario) the gold answer. Base method: Combined (Binary RL + OPD). The default base method we plug PH into is the Combined recipe of OpenClaw-RL [42]; we also evaluate Binary RL only and OPD only as ablations. For each main-line turn the PRM judge produces a scalar score rt∈−1,0,+1r_t∈\-1,0,+1\ from the next-state signal (the simulator’s reply); GRPO advantages are computed within the trajectory with mean and standard deviation across the same session, with disable_rewards_normalization so that the per-turn scale of rtr_t is preserved. The OPD component uses the same model conditioned on a textual hint extracted from the next-state signal; the teacher’s per-token log-probability gap with the student forms the token-level directional advantage At,jOPDA^OPD_t,j (EMA teacher tracking rate 0.050.05). The combined per-turn advantage is Atcomb=wrlAtGRPO+wopdAtOPDA^comb_t=w_rlA^GRPO_t+w_opdA^OPD_t with wrl=wopd=1w_rl=w_opd=1; the PPO surrogate uses asymmetric clip εlo=0.2 _lo=0.2, εhi=0.28 _hi=0.28 from GRPO [33] and kl_loss_coef=0=0. PRM judge and OPD hindsight teacher. Both the PRM and the OPD teacher share weights with the student (full self-distillation): they all run OLMo-3-7B-Instruct, served by separate sglang engines (separate from the rollout engine) so they do not block training. We use prm_num_gpus=1, prm_teacher_num_gpus=1, prm_m=1 (PRM majority vote; we ablate prm_m=3), prm_temperature=0.6, prm_max_new_tokens=2048, and the OPD teacher source is megatron so that the teacher distribution is on-policy with the actor. Training framework: Slime + Megatron-LM. We use the asynchronous 4-component training loop of OpenClaw-RL [42], implemented on slime (RL orchestrator) and Megatron-LM (training backend), with sglang serving rollouts/PRM/teacher. The 8 GPUs of one p5en.48xlarge node split as: 4 GPUs for the actor (TP=4, sequence-parallel, bf16); 2 GPUs for rollout (one engine per GPU, TP=1); 1 GPU for the PRM judge; 1 GPU for the OPD teacher. PH inserts a per-turn predictor call between the actor’s response and the PRM’s score; in the asynchronous regime the predictor runs concurrently with PRM scoring (neither reads the next-state signal), so PH adds essentially zero wall-clock overhead in steady state. Each session writes a JSONL record of (session_id, turn_index, turn_type, prm_score, prospective_score, surprise, weight_version) to disk, allowing offline reconstruction of all calibration metrics. Prospective Hindsight plug-in. For each main-line turn t, after the agent emits its response ata_t but before the PRM scores it, we issue an additional generation request to the same actor with a prompt that asks the model to predict whether its own response in the current dialog state will be judged correct/incorrect/uncertain by an external judge. The predictor sees (s1:t,at)(s_1:t,a_t) but never the simulator’s reply st+1s_t+1. Because the per-turn PRM signal is itself an LLM-based score and therefore stochastic, we use M=3M=3 predictor samples, parsed to −1,0,+1\-1,0,+1\ and averaged into a continuous z¯t∈[−1,+1] z_t∈[-1,+1]. We then use the continuous PH gate (App. C.3): ξt=12|z¯t−rt|∈[0,1],wt= 1+αξt, _t\;=\; 12 | z_t-r_t |\;∈[0,1], w_t\;=\;1+α _t, where rt∈−1,0,+1r_t∈\-1,0,+1\ is the PRM-aggregated turn score. The implementation flag is SURPRISE_MODE=continuous; setting SURPRISE_MODE=discrete recovers the binary ξt∈0,1 _t∈\0,1\ used in single-turn experiments. The weight applies to the per-turn combined advantage AtcombA^comb_t before the PPO surrogate (equivalently, broadcast across all tokens of ata_t since the surprise is a turn-level scalar) and is a stop-gradient constant. PH-specific knobs: PROSPECTIVE_M=3, PROSPECTIVE_MAX_TOKENS=256, SURPRISE_ALPHA=1.0 default, no weight clip (SURPRISE_WEIGHT_MAX unset; the gate already gives wt∈[1,1+α]w_t∈[1,1+α]). Hyperparameters. Table 4 reports the hyperparameters for all multi-turn runs. The same configuration is used for the Combined baseline, Combined + PH, and the α-sweep ablation; only the PH-specific rows differ across rows of Table 3. Evaluation protocol and metrics. We use the personalization evaluator of OpenClaw-RL [42] (script personalization_evaluator.py). For each held-out evaluation prompt we collect a single first response (turn 11, no follow-up) at temperature 0.30.3, and score it with GPT-4.1 against a fixed preference template that requests a boxed score from 0,0.25,0.5,0.75,1\0,0.25,0.5,0.75,1\ at temperature 00 with Nvotes=3N_votes=3 parallel votes whose successful boxed scores are averaged. The user-side preference text is one of PREFERENCE_STUDENT (no AI-style formatting; full reasoning shown) or PREFERENCE_TEACHER (warm, specific, prose-only grading comments); the full strings are in App. F. Capability is the mean evaluator score across the eval pool, reported separately for Student (N=36N=36) and Teacher (N=24N=24) plus the macro-average. We define multi-turn OFR as the per-turn analogue Pr[rt≤0∣zt=+1] [r_t≤ 0 z_t=+1], computed from the per-turn JSONL bookkeeping over the held-out eval pool, and additionally report surprise rate ℳ^=Pr[ξt>0] M= [ _t>0] and prospective accuracy. Headline numbers are reported at the final checkpoint of round 16; dynamics plots evaluate every 4 weight updates. D.3 Reweighting-Baseline Protocols To isolate the contribution of PH’s surprise-based selection from the contribution of additional gradient mass, we compare PH against two reweighting controls in the single-turn setting (Table 2). Both use the same multiplier (1+α)(1+α) as PH and are detached from the computation graph (stop-gradient); only the rule for which rollouts receive the multiplier differs. Random reweighting. At each training step, we compute the empirical fraction ptp_t of surprising rollouts the PH protocol produces on the same data. We then sample uniformly at random a subset of size ⌊pt⋅N⌋ p_t· N from the current batch and assign these rollouts weight (1+α)(1+α); the remaining rollouts retain weight 11. By construction this matches PH’s per-step total weight ∑i(1+α⋅ξt(i)) _i(1+α· _t^(i)), so the only difference between Random and PH is the location of the upweighted mass: uniform vs. surprise-targeted. Selection seeds are derived from hash(rollout_id, step) so that within an epoch each rollout receives a deterministic weight; we use α=1.0α=1.0 throughout, matching the headline PH configuration. Failure-only reweighting. For every rollout with verifier outcome yt=0y_t=0, we set the weight to (1+α)(1+α); rollouts with yt=1y_t=1 retain weight 11. This corresponds to upweighting cells OF∪AFOF in the calibration taxonomy (Table 1), as opposed to PH’s OF∪USOF . We again use α=1.0α=1.0. Failure-only therefore differs from PH in exactly two cells: it includes AF (calibrated incompetence, which PH leaves unweighted) and excludes US (underconfident successes, which PH amplifies). What each baseline controls for. Random isolates the effect of selection: any gain of PH over Random must come from which rollouts are upweighted, not from how much weight is added. Failure-only matches PH on the dominant single-turn cell (OF; see Figure 5) but differs on the smaller cells, isolating whether the prospective signal ztz_t contributes information beyond the verifier outcome yty_t alone. The empirical comparison is reported in Table 2 and discussed in §4.2; the PH-cell composition over training is shown in Figure 2. D.4 Compute and Reproducibility Hardware and software. Both single-turn and multi-turn experiments run on a single node of p5en.48xlarge (8×8× NVIDIA H200). Multi-node is supported by the underlying frameworks (verl / Slime) but not used in our reported runs. Software: CUDA 12.x, PyTorch 2.5+, vLLM for single-turn rollouts, sglang for multi-turn rollouts/PRM/teacher, Megatron-LM (multi-turn actor backend), slime (multi-turn async orchestrator), verl (single-turn orchestrator). Wall-clock and overhead from PH. Single-turn (3 epochs, 3232 prompt batch, K=8K=8 rollouts): ∼8 8 hours per run on Science Q&A, ∼6 6 hours on Tool Use; the PH plug-in adds ∼6% 6\% wall-clock overhead from the predictor calls (amortized into the same vLLM batch). Multi-turn (1616 rounds, 22 sessions/round, Tmax=48T_ =48): ∼4 4 hours per run; the PH plug-in adds essentially zero wall-clock overhead because the predictor runs concurrently with the PRM in the asynchronous loop. Table 6: PH weight strength on Science Q&A: numerical summary across the full α-sweep. OFR =OF/(CS+OF)=OF/(CS+OF); Early/Late are first/last 5 training steps; Mean Weight and Verifier Pass Rate are reported as early → late. Bold: best per column. α Early OFR Late OFR OFR Drop Rel. Drop Mean Weight Verifier Pass Rate 0.1 0.741 0.253 0.487 65.8% 1.07→1.021.07→ 1.02 24.8%→71.4%24.8\%→ 71.4\% 0.25 0.739 0.250 0.489 66.2% 1.17→1.061.17→ 1.06 25.6%→71.2%25.6\%→ 71.2\% 0.5 0.768 0.209 0.560 72.9% 1.35→1.091.35→ 1.09 22.9%→73.2%22.9\%→ 73.2\% 1.0 0.732 0.239 0.423 57.8% 1.67→1.291.67→ 1.29 26.0%→67.5%26.0\%→ 67.5\% 2.0 0.744 0.249 0.495 66.5% 1.68→1.221.68→ 1.22 24.9%→71.7%24.9\%→ 71.7\% Appendix E Additional Experiment Results and Analysis This appendix expands the experiments of §4.2 (single-turn SDPO + PH on Science Q&A and Tool Use) and §4.3 (multi-turn RL / OPD / Combined + PH) with the full α-sweeps that the main tables abridge, finer-grained training-time dynamics, and per-turn breakdowns. All configurations match App. D unless stated otherwise. E.1 Single-Turn α-Sweep: SDPO + PH on Science Q&A PH cell composition is qualitatively robust across α. Figure 6 repeats the per-step cell-composition plot of Figure 2 (which fixes α=1.0α=1.0) for the additional weights α∈0.1,0.25,0.5,2.0α∈\0.1,0.25,0.5,2.0\ on Science Q&A. The four panels share a common pattern: the overconfident-failure (OF) cell shrinks throughout training while the confident-success (CS) cell grows, and the underconfident-success (US) and aware-failure (AF) cells stay small. The fact that this dynamics holds essentially unchanged across two orders of magnitude of α tells us that PH is not a delicate optimization-noise instrument: it is a structural reweighting whose direction (compress OF, grow CS) is set by the data, with α controlling only the speed of that compression. OFR decreases monotonically and the PH weight self-extinguishes. Figure 7 (left) plots OFR over training for each α on Science Q&A. All five trajectories descend monotonically; α=0.5α=0.5 reaches one of the lowest late-stage OFR values, consistent with the main-text observation in §4.2. Figure 7 (right) plots the corresponding mean PH weight [wt]E[w_t]: it starts above 11 (because ξt>0 _t>0 on many rollouts initially), decays toward 11 as ξt→0 _t\!→\!0 on the bulk of rollouts, and the larger the α the higher the initial weight but also the steeper the relaxation. The weight curriculum is therefore self-extinguishing — exactly the behaviour predicted by the continuous-relaxation analysis of App. C.3. Why α=0.5α=0.5 is the Science Q&A sweet spot. Table 6 numerically compares the five α values along three axes: late-training OFR, the early-to-late OFR drop, and the verifier pass rate at the final checkpoint. α=0.5α=0.5 is best on all three: largest absolute OFR drop (0.5600.560), largest relative drop (72.9%72.9\%), and highest final verifier pass rate (73.2%73.2\%). The weight column shows why: α=0.5α=0.5’s mean weight starts at 1.351.35 (large enough to materially reweight miscalibrated samples) and anneals to 1.091.09 (small enough to not perturb late-stage learning), giving the cleanest curriculum. α=1.0α=1.0 and α=2.0α=2.0 start the curriculum from much higher initial weights (1.67,1.681.67,1.68) and do not anneal as cleanly, trading a fraction of the calibration improvement for slightly noisier capability. Conversely, α∈0.1,0.25α∈\0.1,0.25\ are nearly identity reweightings throughout training (initial weight 1.07,1.171.07,1.17) and so leave the OFR drop close to the small-α floor of ≈0.49≈ 0.49. Figure 6: PH cell composition across α on Chemistry. Per-step decomposition of training rollouts into CS / OF / US / AF for α∈0.1,0.25,0.5,2.0α∈\0.1,0.25,0.5,2.0\ (the α=1.0α=1.0 panel is in Figure 2). The compress-OF / grow-CS pattern is qualitatively unchanged across two orders of magnitude of α. Figure 7: OFR and PH-weight dynamics across α on Science Q&A. Left: OFR over training for each α; all five trajectories descend monotonically. Right: the mean PH weight starts above 11 and anneals toward 11 as ξt→0 _t\!→\!0 (App. C.3). Faint lines: 5-step bins; solid lines: EMAs. E.2 Multi-Turn α-Sweep: Per-Round Capability and Per-Turn Calibration α sweep across capability and per-turn calibration. Figure 8 plots the multi-turn α-sweep on the Combined base method under the GPT-4.1 simulator: teacher-side score across training rounds (left), per-turn surprise rate across conversation turns (middle), and per-turn OFR across conversation turns (right). The capability trajectory (left) reproduces the headline of Table 3: α=1.0α=1.0 rises fastest and plateaus highest, α=0.5α=0.5 tracks closely, and α=2.0α=2.0 exhibits the highest round-to-round variance. The per-turn surprise (middle) and per-turn OFR (right) reveal a structural pattern that the trajectory-averaged metrics in the main text do not surface: later turns have higher initial overconfidence, consistent with the long-horizon credit-assignment intuition that the agent’s self-model is least developed for late, context-saturated turns; and PH compresses exactly this region most. At α=1.0α=1.0, late-turn OFR drops to levels comparable to the early-turn OFR of the baseline, effectively flattening the per-turn calibration profile. When α=2.0α=2.0 overshoots in long horizons. Figure 8 also concretizes the α=2.0α=2.0 caveat from §4.3: the right panel shows that α=2.0α=2.0 achieves the lowest per-turn OFR overall, but the left panel shows that its capability oscillates round-to-round and ends below α=1.0α=1.0. The mechanism is the variance-amplification effect quantified in App. B.2: very large α multiplies the per-rollout combined-advantage by up to 1+α=31+α=3 on miscalibrated rollouts, and in long-horizon training those rollouts also tend to be the longest token sequences (overconfident failures are correlated with more aggressive generation), so the per-batch gradient norm spikes relative to the baseline distribution. We did not observe this regime in single-turn (Table 2) because the per-rollout token count there is bounded and similar across all rollouts; the multi-turn regime, with its variable-length sessions of up to Tmax=48T_ =48 turns (App. D.2), exposes it. Practitioner recommendation. For both regimes, a single α=1.0α=1.0 default works well across methods, datasets, and simulators. The per-domain optimum may shift to α=0.5α=0.5 (Science Q&A, Combined) or remain at α=1.0α=1.0 (Tool Use, RL), but no α outside the 0.5,1.0\0.5,1.0\ band ever beats the in-band defaults on both capability and calibration simultaneously. We therefore recommend α=1.0α=1.0 as the practitioner default and α=0.5α=0.5 as a slightly more conservative alternative whenever capability stability is paramount. Figure 8: Multi-turn α-sweep (Combined, GPT-4.1 simulator). Left: teacher-side evaluation score across training rounds. Middle: teacher-side surprise rate across conversation turns. Right: teacher-side OFR across conversation turns. Curves: α∈0.1,0.5,1.0,2.0α∈\0.1,0.5,1.0,2.0\. Later turns start more overconfident than early turns, and PH compresses this region most. Figure 9: Capability and calibration across user simulators. OLMo-3-7B-Instruct trained on the multi-turn personal-agent task with 8k8k response-length budget and mixed Student/Teacher rounds, under two simulators (GPT-4.1 and GPT-5.4). Left: teacher-side personalization-evaluator score (higher is better). Right: teacher-side OFR (lower is better). Appendix F Prompt Templates This appendix lists the prompt templates used by PH’s prospective self-evaluator (§3.1) and, in multi-turn settings, the retrospective process reward model (PRM) used as the verifier R in Eq. 1. All prompts elicit a ternary score parsed from a ⋅· field as ztz_t or rt∈−1,0,+1r_t∈\-1,0,+1\; ties under majority voting are broken toward 00. F.1 Single-Step Tasks For single-step tasks (Science Q&A, Tool-Use, etc.), the prospective self-evaluator uses a task-type-specific template that wraps a common ternary scoring instruction. The task type is routed automatically from the dataset identifier: math (GSM8K, MATH, AIME, etc.) → math; multiple-choice (SciKnowEval, GPQA, MMLU) → mcq; tool-use → tool_use; code (LiveCodeBench, HumanEval+, etc.) → code; otherwise → default. Placeholders problem and solution are filled with the task instance and the agent’s just-emitted action ata_t. The common ternary scoring instruction below is appended verbatim to every single-step template (Listing ); each task type then wraps it with its own header (Listings –). ⬇ 1 Return exactly one score and no explanation: 2 +1 if you are confident it is correct 3 -1 if you believe it is wrong 4 0 if you are unsure 5 Your entire response must be exactly one of: +1, 0, -1 Listing 1: Common ternary scoring instruction (single-step). ⬇ 1 Review your solution to the following problem. 2 3 Problem: 4 problem 5 6 Your solution: 7 solution 8 9 Do you believe your final answer is correct? 10 [common scoring instruction] Listing 2: Single-step prospective self-evaluator: Math template. ⬇ 1 Review the question and your chosen answer below. 2 3 Question: 4 problem 5 6 Your response: 7 solution 8 9 Do you believe you selected the correct answer? 10 [common scoring instruction] Listing 3: Single-step prospective self-evaluator: Multiple-choice template. ⬇ 1 Review the task and your proposed solution below. 2 3 Task: 4 problem 5 6 Your solution: 7 solution 8 9 Do you believe you selected the correct actions and inputs to 10 complete this task? 11 [common scoring instruction] Listing 4: Single-step prospective self-evaluator: Tool-use template. ⬇ 1 Review the problem and your code solution below. 2 3 Problem: 4 problem 5 6 Your solution: 7 solution 8 9 Do you believe your code produces the correct output for all 10 test cases? 11 [common scoring instruction] Listing 5: Single-step prospective self-evaluator: Code template. ⬇ 1 Review your solution to the following problem. 2 3 Problem: 4 problem 5 6 Your solution: 7 solution 8 9 Do you believe your solution is correct? 10 [common scoring instruction] Listing 6: Single-step prospective self-evaluator: Default template (fallback). F.2 Multi-Turn Tasks In multi-turn settings, two distinct prompts are used: a prospective prompt that the agent’s policy answers immediately after emitting ata_t (without observing the next state ct=st+1c_t=s_t+1), and a retrospective PRM prompt that scores the same response after st+1s_t+1 is revealed. The retrospective PRM serves as the verifier R in Eq. 1; the prospective prompt produces ztz_t as in Eq. 3.1. Placeholders are filled at runtime: response_text is the agent’s action ata_t, last_user_message is the most recent user turn from the conversation history, next_state_text is st+1s_t+1, and next_state_role indicates whether the next state is a user reply or a tool return. Prospective self-evaluator (general multi-turn). This is the default prospective prompt used by PH at every turn t. It receives only (st,at)(s_t,a_t) via last_user_message and response_text; the next state st+1s_t+1 is not exposed. ⬇ 1 User request: 2 last_user_message 3 4 Assistant response: 5 response_text 6 7 Does this response correctly and completely address the user’s 8 request? 9 Answer 1 for yes, -1 for no, 0 for unsure. 10 Be brief. End your answer with SCORE. Listing 7: Multi-turn prospective self-evaluator (general). Prospective self-evaluator (Personal Agent, style-aware). This variant is used in the Personal Agent setting where user satisfaction depends on stylistic conformance (natural prose, no AI-style formatting) in addition to factual correctness. The prompt is split into a system message that defines the user’s style preferences and a user message that asks for the prediction. ⬇ 1 [SYSTEM] 2 You are predicting whether a user will be SATISFIED or 3 DISSATISFIED with an AI assistant’s response, based on the 4 user’s strong style preferences. 5 6 The user HATES AI-style formatting. Specifically, the user 7 will ask for a rewrite if the response contains ANY of these: 8 - Markdown bold (**text**), headers (## or ###), or horizontal 9 rules 10 - Numbered lists (1. 2. 3.) or bullet points (- or *) 11 - Overly structured step-by-step layouts 12 - Phrases like "Here is", "Sure!", "Great question!", "Let me", 13 "Certainly" 14 - Emojis or excessive exclamation marks 15 16 The user WANTS natural, human-like prose, as if a real person 17 (student or teacher) wrote it casually. Flowing sentences, no 18 formatting artifacts. 19 20 Scoring: 21 - -1: response has formatting issues or sounds AI- 22 generated; user will request a rewrite 23 - 1: response is natural, human-like prose with no 24 formatting artifacts; user will be satisfied 25 - 0: borderline / unsure 26 27 Be strict. Even one instance of bold text or a numbered list 28 means -1. 29 30 [USER] 31 User message: 32 last_user_message 33 34 Assistant response: 35 response_text 36 37 Check the response for AI-style formatting (bold, headers, 38 numbered lists, bullet points, AI phrases). Does it read like 39 natural human writing? 40 Answer with SCORE. Listing 8: Multi-turn prospective self-evaluator (Personal Agent, style-aware). Retrospective PRM (verifier). The PRM observes the next state st+1s_t+1 as evidence and assigns the outcome rt∈−1,0,+1r_t∈\-1,0,+1\. ⬇ 1 [SYSTEM] 2 You are a process reward model (PRM) evaluating an AI 3 assistant. You will see the assistant’s output and the 4 subsequent next state. Your task: decide whether the 5 assistant’s output successfully fulfilled the user’s intent 6 at that step, using the next state as evidence. 7 8 Understanding the next state’s role: 9 - role=’user’: a reply from the user. 10 - role=’tool’: the return value of a tool the assistant 11 invoked. This content was NOT available before the 12 assistant’s action; it exists BECAUSE the assistant called 13 the tool. A successful, non-error tool output means the 14 assistant’s action worked correctly and should be scored 15 positively. 16 17 Scoring rules: 18 - 1 (good): next state shows the task progressed 19 as expected (user moves on, says 20 thanks, environment confirms success, 21 tool returns successful non-error 22 result). 23 - -1 (bad): next state signals the assistant’s 24 output was wrong, incomplete, or 25 unwanted. Key negative signals: 26 * user asks the assistant to redo, 27 retry, or repeat the same action 28 * user requests a correction or 29 modification 30 * user rephrases or restates the same 31 request 32 * environment returns an error, 33 failure, or unexpected result. 34 - 0 (neutral): next state is ambiguous. 35 36 A change request is negative feedback: it means the previous 37 output did not meet the user’s need; do not treat it as a 38 neutral new instruction. Think step-by-step, then give your 39 final score inside . 40 41 [USER] 42 Assistant output: 43 response_text 44 45 Next state [role: next_state_role]: 46 next_state_text 47 48 First, classify the next state: is it (a) positive 49 progression, (b) a correction / redo / change request, or 50 (c) ambiguous? Then assign 1, -1, or 0. Listing 9: Multi-turn retrospective PRM verifier (next-state grounded). Appendix G Limitations and Future Work While Prospective Hindsight (PH) provides a robust and simple mechanism for self-calibration in agent reinforcement learning, we identify several limitations that offer avenues for future work: • Variance-Amplification at High α: High values of the surprise amplification hyperparameter (α≥2.0α≥ 2.0) can lead to gradient norm spikes, particularly in multi-turn, long-horizon tasks where overconfident failures are correlated with longer token sequences . While α=1.0α=1.0 is a stable default, extreme upweighting may destabilize training. • Theoretical Assumptions: Our gradient-flow analysis, which shows the PH update is strictly calibration-aligned, relies on a “uniform-base-loss” assumption—specifically, that the base loss is approximately constant on surprising rollouts. In practice, this alignment is quantitative rather than absolute, as the per-rollout loss varies with specific log-probabilities. • Training-Time Compute Overhead: PH requires M additional forward passes per rollout step to elicit the prospective belief. While this adds negligible wall-clock time in asynchronous multi-turn settings (approx. 0%), it introduces a small overhead (approx. 6%) in single-turn synchronous training. • Dependence on Self-Model Quality: The effectiveness of the surprise signal depends on the policy’s ability to produce meaningful ternary predictions. Although PH facilitates the co-evolution of the self-model and policy, the initial training phase may provide noisier signals if the base model is extremely poorly calibrated. • Scope of Empirical Evaluation: Our experiments focused on verifiable tasks (Science Q&A, Tool Use) and a math-tutoring personal agent. While we demonstrated cross-scale transfer from 7B to 20B models, the performance of PH in purely creative or subjective generation tasks where retrospective rewards are inherently ambiguous remains to be fully explored. The limitations above suggest several directions where the Prospective Hindsight principle can be extended or its current evidence base strengthened. • Independent characterization of the prospective evaluator. Decoupling the evaluator’s quality from the training trajectory is important both for methodological cleanliness and for understanding when PH should be expected to work. Promising directions include: (i) measuring evaluator accuracy on held-out prompts at fixed checkpoints, (i) comparing evaluator outputs against a stronger external judge, (i) calibrating the evaluator’s token-level confidence against its boxed score, and (iv) studying the regime in which evaluator and policy share systematic blind spots and how PH behaves under such failure. • Scaling to longer-horizon agentic tasks. Our multi-turn experiments cap conversations at Tmax=48T_ =48 turns. Genuinely long-horizon agents—software engineering trajectories spanning hundreds of tool calls, multi-stage research assistants, or extended pair-programming sessions—raise two questions PH does not yet answer: how does the surprise signal interact with sparse, delayed verifier feedback at long horizons, and does the per-turn calibration cell composition (Fig. 5) shift further as horizons grow? Combining PH with hierarchical credit-assignment schemes that already address sparse-reward long-horizon RL is a natural follow-up. • Continuous-valued PRM scores and richer surprise signals. The current surprise indicator is binary (discrete gate) or a half-distance in [0,1][0,1] (continuous gate). Recent process reward models emit graded scores, confidence intervals, or per-criterion breakdowns [42]. Extending PH to exploit these richer retrospective signals—for example, by defining ξt _t as a function of the PRM’s confidence interval on rtr_t rather than its point estimate—may further reduce the variance amplification at large α that we observe in long horizons. • Composition with inference-time self-correction. PH operates entirely at training time and leaves the agent’s deployment behavior unchanged. Inference-time self-correction methods such as Reflexion [36] and Self-Refine [26] act on the policy distribution after PH has shaped it. Whether the two mechanisms compose constructively—PH producing a better-calibrated self-model, then inference-time self-correction exploiting that calibration to drive re-tries—is an open question with practical implications for deployed agentic systems. Appendix H Broader Impact The development of Prospective Hindsight (PH) aims to bridge the gap between capability and calibration in language model agents. As LLMs are increasingly deployed in autonomous agentic roles—such as personal assistants, tool-use interfaces, and tutoring systems—the societal implications of their reliability become paramount. Positive Societal Impacts: • Enhanced Safety and Reliability: By explicitly penalizing overconfident failures, PH encourages agents to recognize their own limitations. This reduction in "confidently wrong" behavior is critical in high-stakes domains where an agent should ideally invite human intervention rather than proceeding with a flawed trajectory. • Improved Human-AI Trust: Calibration is a cornerstone of effective human-AI collaboration. Agents that can accurately communicate their uncertainty (through honest uncertainty markers) allow users to set appropriate expectations and intervene when necessary. • Resource Efficiency: PH is a self-extinguishing curriculum that reduces the need for manually designed reward schedules or expensive, separate calibration-specific training phases. This can lower the computational and human-labor barriers to developing reliable AI systems. Potential Negative Impacts and Mitigations: • Risk of Deceptive Calibration: While PH aligns the agent’s prospective belief with verifier outcomes, there is a theoretical risk that an agent could learn to "appear" calibrated by avoiding difficult but necessary tasks to minimize surprise. We mitigate this by using PH as a plug-in on top of task-success objectives, ensuring capability is not sacrificed for calibration. • Over-reliance on Automated Feedback: The method relies on the quality of retrospective verifiers or PRMs. If these evaluators contain biases or incorrect logic, PH may amplify these errors by aggressively upweighting rollouts that "surprise" the model based on flawed ground truth. Developers should ensure that the underlying verifiers are robust and subject to human auditing. • Dual-Use Concerns: Improved agentic autonomy and tool-use capabilities, while beneficial for productivity, could be misused to automate malicious activities. However, PH’s primary contribution is the alignment of internal belief with reality, which is a fundamentally defensive property aimed at reducing unintended failures rather than increasing raw harmful capabilities.