Paper deep dive
CLIFT: Conformal Self-Verification for Web Agent Training and Test-Time Scaling
Yifan Zhang, Yutong Dai, Viraj Prabhu, Zhiyuan Hu, Ran Xu, Zeyuan Chen
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Open-source web agents are now strong enough to execute realistic browser tasks, but training them with reinforcement learning still depends on weak supervision: binary task success is too sparse for credit assignment, while frontier-language-model judges are too expensive to call at every step and cannot be assumed available at deployment. We introduce CLIFT, a training and test-time scaling method built around conformal self-verification. During training, the agent answers natural-language verification questions about its own rollouts; a Compositional Conformal Certifier keeps only question signals whose URL-conditional evidence agrees with a training-time judge, assigns signed trust weights through polarity-aware lift, and blends the resulting verifier score into per-step rewards in a way that never subtracts from the judge baseline. At test time, the same certified bank is frozen and reused as structured evidence for Conformal Trajectory Selection (CTS): the agent samples a greedy rollout and one or more diverse retries, the self-verifier summarises each URL trace, and a conservative majority-vote rule chooses whether to swap away from the current incumbent without calling any external judge. This single mechanism supports three settings. On WebArena Infinity, CLIFT achieves state-of-the-art performance among open-source web agents. On VisualWebArena, a bank trained with the open model transfers to GPT-5.5 at test time and reaches state-of-the-art performance under the canonical harness. On Online Mind2Web, without training an agent on the benchmark, translating the certified question bank improves a live-web agent in zero-shot evaluation. Together these results position conformal self-verification as a way to turn costly judge feedback into a reusable training signal and a judge-free test-time scaling signal.
Tags
Links
- Source: https://arxiv.org/abs/2610.06829v1
- Canonical: https://arxiv.org/abs/2610.06829v1
Trouble viewing inline? Open PDF directly →
Full Text
67,758 characters extracted from source content.
Expand or collapse full text
CLIFT: Conformal Self-Verification for Web Agent Training and Test-Time Scaling Yifan Zhang Yutong Dai Viraj Prabhu Zhiyuan Hu Ran Xu Zeyuan Chen Affiliation: Salesforce AI Research Affiliation: yifan.zhang, yutong.dai, viraj.prabhu@salesforce.com Affiliation: zhiyuan.hu, ran.xu, zeyuan.chen@salesforce.com Abstract Open-source web agents are now strong enough to execute realistic browser tasks, but training them with reinforcement learning still depends on weak supervision: binary task success is too sparse for credit assignment, while frontier-language-model judges are too expensive to call at every step and cannot be assumed available at deployment. We introduce CLIFT, a training and test-time scaling method built around conformal self-verification. During training, the agent answers natural-language verification questions about its own rollouts; a Compositional Conformal Certifier keeps only question signals whose URL-conditional evidence agrees with a training-time judge, assigns signed trust weights through polarity-aware lift, and blends the resulting verifier score into per-step rewards in a way that never subtracts from the judge baseline. At test time, the same certified bank is frozen and reused as structured evidence for Conformal Trajectory Selection (CTS): the agent samples a greedy rollout and one or more diverse retries, the self-verifier summarises each URL trace, and a conservative majority-vote rule chooses whether to swap away from the current incumbent without calling any external judge. This single mechanism supports three settings. On WebArena Infinity, CLIFT achieves state-of-the-art performance among open-source web agents. On VisualWebArena, a bank trained with the open model transfers to GPT-5.5 at test time and reaches state-of-the-art performance under the canonical harness. On Online Mind2Web, without training an agent on the benchmark, translating the certified question bank improves a live-web agent in zero-shot evaluation. Together these results position conformal self-verification as a way to turn costly judge feedback into a reusable training signal and a judge-free test-time scaling signal. 1 Introduction Open-source web agents have reached the point where they can operate realistic browser interfaces, but the verifier remains the bottleneck for improving them with reinforcement learning. A binary success signal is easy to deploy but too sparse for credit assignment: when all rollouts in a GRPO group (Shao et al., 2024) receive the same outcome, the advantage estimate collapses. A frontier language model judge (Bai et al., 2022; Liu et al., 2023) provides richer feedback, but it is costly during training and cannot be assumed available at test time. This tension is especially visible in web-agent benchmarks: WAI (Zhou, 2026) uses programmatic per-app verifiers, VWA (Koh et al., 2024) uses outcome-based URL/DOM checks, and live-web suites such as OM2W (Xue et al., 2025) depend on noisy LLM judges. The policy is often not the only limiting object; the way we score and select its rollouts can dominate the observed success rate. Table 1: Feedback mechanisms for web-agent RL and test-time scaling. Dense = step-level feedback beyond binary outcome; URL = URL-state grounding; Cert. = certified trust; J-free = judge-free test time; TTS = test-time scaling; Xfer = reusable bank transfer. Approach Dense URL Cert. J-free TTS Xfer Outcome RLVR / GRPO (trajectory outcome) (Shao et al., 2024) × × × ✓ × × Stratified GRPO (structural strata) (Zhu et al., 2025) ✓ × × ✓ × × Rich-feedback self-distillation (teacher feedback) (Hübotter et al., 2026) ✓ × × ✓ ✓ × Process / verifier reward models (learned PRM) (Lightman et al., 2024) ✓ × × ✓ × × Post-hoc search / BoN (sample then select) (Snell et al., 2025) × × × ✓ ✓ × Conformal routing / judging (uncertainty gate) (Badshah et al., 2026) × × ✓ × × × CLIFT (URL-certified bank for training and CTS) ✓ ✓ ✓ ✓ ✓ ✓ CLIFT asks whether the judge’s feedback can be turned into a reusable verifier that the agent carries with itself, rather than a separate process reward model (Lightman et al., 2024; Wang et al., 2024) or a post-hoc best-of-N scorer (Cobbe et al., 2021; Snell et al., 2025). CLIFT maintains a bank of natural-language verification questions about web-agent behaviour and certifies which questions are reliable at which URL states. The same bank is used in two places (Figure 2). During training, certified bank evidence is blended with the comparative judge to produce a denser per-step reward and URL-stratified advantages. At test time, the trained agent runs a greedy trajectory plus diverse retries; the frozen certified bank summarizes each URL trace; and a conservative trajectory selector keeps one emitted trajectory per task. The common object is not a new model but a calibrated verifier: a set of questions, URL scopes, polarity signs, and conformal trust weights that can be reused after training. Figure 1: CLIFT concept. Vanilla GRPO broadcasts one trajectory-scalar reward; CLIFT scores each visited URL with verification questions and inherits the URL score to actions taken there. Full pipeline: Figure 2. The certifier is the part that makes this more than prompt engineering. Each verification question carries a URL scope and a polarity sign indicating whether YES is evidence of progress or failure. A Mondrian adaptive conformal tracker (Gibbs and Candès, 2021; Vovk, 2012) estimates whether that question separates high- and low-judge rollouts at the relevant URL granularity. Certified questions receive signed trust weights. During training this gives a verifier reward that can only add evidence on top of the judge baseline, preventing the reward-collapse failure of earlier linear blends. During test-time scaling, the same signed evidence is frozen and presented to a comparative selector over two completed trajectories. In both cases, the question bank is a compact interface between expensive judge feedback and the agent’s own rollouts. We evaluate this idea in three complementary ways. On WAI, where programmatic verifiers provide dense training supervision, CLIFT improves a trainable open Gemma-4 agent and reaches the strongest public 9-app score we are aware of; the point is both training and CTS on the same open backbone. On VWA, where the evaluation is more outcome-based, we use the bank created during Gemma-4 training as a test-time verifier for GPT-5.5, showing that the learned verification knowledge transfers to a stronger closed backbone and reaches state-of-the-art performance under the canonical harness. On OM2W, we push transfer further: no agent is trained on OM2W; instead, the certified question bank is translated and recalibrated, then used by CTS to improve live-web test-time scaling. These three stories support the same claim: conformal self-verification is useful both as a training signal and as a reusable, judge-free test-time scaling signal. Related work is discussed after the empirical section (§6). 2 CLIFT: Conformal Lift-calibrated Introspective Filtering of Trajectories Figure 2: CLIFT overview. Training: rollouts are scored by a comparative judge and by a self-verifier that queries a polarity-tagged question bank. The C certifies question weights by URL tier, yielding a signed verifier score that is asymmetrically added to the judge reward and then converted into URL-stratified token advantages. Test-time scaling: the trained agent emits a greedy rollout and diverse retries; the frozen certified bank produces per-URL evidence for each pairwise comparison, and a conservative repeated-vote rule decides whether to keep the incumbent or swap. CLIFT is a GRPO-family policy-gradient method (Shao et al., 2024) that changes how rollouts are scored and reused. It turns natural-language verification questions into a conformally certified URL-aware verifier, uses that verifier to add non-negative per-step evidence during training, and freezes the same bank for CTS at test time. Thus both halves of Figure 2 share one object: questions with URL scopes, polarity signs, and certified trust weights. 2.1 The Conformal Verifier A verification question (VQ) is a YES/NO check on the agent’s behaviour. Each question q has a URL pattern πq _q and polarity σq∈−1,0,+1 _q∈\-1,0,+1\ indicating progress, failure, or ambiguity. At training and test time, the agent’s base model answers applicable questions with the task adapter detached. Answers at URL state u are aggregated into RiVQ(u)=∑q∈Qcert(u)wτ⋆(u)q⋅[yiq=YES]∈[−1,+1],R^VQ_i(u)=\!\!\! _q∈ Q^cert(u)\!\!w^q_τ (u)·1\! [y^q_i=YES ]\;∈\;[-1,+1], (1) where Qcert(u)Q^cert(u) contains certified in-scope questions and wτ⋆(u)qw^q_τ (u) is the most-specific certified trust weight. Positive weights indicate progress, negative weights indicate failure, and an empty set gives RiVQ(u)=0R^VQ_i(u)=0. Certification. The same question can be reliable on one app and useless on another, so the C keeps a Mondrian-ACI tracker (Vovk, 2012; Gibbs and Candès, 2021) for each question and URL tier τ∈global,host,URL-pathτ∈\global,host,URL-path\. The non-conformity score is polarity-aware: stq,τ s_t^q,τ =[predq(yqt)≠zt],zt=[R^judget>θj], =1\! [\,pred_q(y^q_t)≠ z_t\, ], z_t=1[ R^judge_t> _j], (2) predq(YES) _q(YES) =[σq=+1],predq(NO)=[σq=−1]. =1[ _q=+1], _q(NO)=1[ _q=-1]. Positive questions conform when YES agrees with a high judge score; negative questions conform when YES agrees with a low score. Neutral questions do not certify. A tier τ of question q is certified when nt,τq≥nmin∧αt,τq<α⋆∧|liftt,τq|>liftmin∧σq≠0,n^q_t,τ≥ n_ \;\, \;\,α^q_t,τ<α \;\, \;\, |lift^q_t,τ |>lift_ \;\, \;\, _q≠ 0, (3) where liftt,τqlift^q_t,τ is the mean judge-score difference between YES and NO examples at tier τ; nminn_ , α⋆α , and liftminlift_ are calibration hyperparameters reported with the experimental setup. Certified questions receive wτq=σq⋅|liftt,τq|⋅max(0, 1−αt,τqα⋆)∈[−1,+1],w^q_τ= _q· |lift^q_t,τ |· \! (0,\,1- α^q_t,τα )\;∈\;[-1,+1], (4) with wτq=0w^q_τ=0 for uncertified tiers. At URL u we query the most specific certified tier, falling back URL-path → host → global. Bank growth. The bank starts from a judge-grounded seed set of VQs, each tagged with a polarity. At periodic training intervals, a CEGAR-style loop (Clarke et al., 2000) proposes a small batch of new URL-scoped questions separating top and bottom rollouts. New questions enter uncertified. A sanity veto demotes polarity to neutral when observed lift contradicts the tag: σq←0ifnq≥nveto∧sgn(liftq)≠σqLLM∧|liftq|>δ. _q← 0 n_q≥ n_veto\; \;sgn\! (lift^q )≠ _q^LLM\; \; |lift^q |>δ. (5) The same lift estimate therefore certifies questions and catches polarity mistakes. 2.2 Training: Per-step Rewards and URL-stratified Advantages For each task, one comparative judge call scores a GRPO group of G rollouts and returns trajectory scores RijudgeR^judge_i and per-step scores rti,judger^i,judge_t. In parallel, the self-verifier answers bank questions for each rollout and URL state. The training reward is r~ti=rti,judge+β⋅max(0,RiVQ(uti)), r^i_t\;=\;r^i,judge_t\;+\;β· \! (0,\,R^VQ_i(u^i_t) ), (6) where β controls the verifier contribution. Since the verifier term is clipped at zero, r~ti≥rti,judge r^i_t≥ r^i,judge_t: certified evidence can strengthen the judge signal, but cannot drag it down. URL-stratified per-token advantage. We discount step rewards into returns Gti=∑ℓ=tTi−1γℓ−tr~ℓiG^i_t= _ =t^T_i-1γ -t\, r^i_ with discount γ, then group-normalise within each (rollout-group, step-position, URL-state) cell: A^ti=Gti−G¯utiσG,uti+ε. A^i_t= G^i_t- G_u^i_t _G,u^i_t+ . (7) Advantages are broadcast to tokens over tool-call spans; tokens outside those spans receive zero gradient. The GRPO update is unchanged, but credit assignment becomes URL-local. Training-time judge. The judge sees compact traces with human-readable actions; opaque browser-use object strings collapse judge variance. One prompt sees all G rollouts and returns trajectory scores, step scores, and rationales for CEGAR. The judge is training-only and is not called by CTS. 2.3 Test Time: CTS (Conformal Trajectory Selection) At deployment the external training judge is dropped. The policy emits candidate rollouts under a small budget K: a greedy incumbent ρA _A and one or more diverse retries ρB _B. CLIFT then uses the frozen bank and base self-verifier in a pairwise selector to decide whether to keep the incumbent or swap to the retry, avoiding brittle step-wise selection. For K>2K>2, the same selector is applied as a tournament chain that maintains one incumbent and still emits a single trajectory. For each rollout ρ∈ρA,ρBρ∈\ _A, _B\, let U(ρ)=(u0,u1,…,uTρ)U(ρ)=(u_0,u_1,…,u_T_ρ) be its sequence of distinct URL states (consecutive duplicates collapsed). The verifier produces a per-state evidence record at every ut∈U(ρ)u_t∈ U(ρ): Et(ρ)=(q,wτ⋆(ut)q,liftτ⋆(ut)q,yq(ρ,ut)):q∈Qcert(ut),πq∼ut,E_t(ρ)= \\, (q,\;w^q_τ (u_t),\;lift^q_τ (u_t),\;y^q(ρ,u_t) )\;:\;q∈ Q^cert(u_t),\; _q u_t\, \, (8) where yq(ρ,ut)y^q(ρ,u_t) is the verifier’s YES probability on the partial trajectory. We keep this as structured evidence rather than a single scalar. The evidence sheets, action traces, and final states are passed to the self-verifier in a comparative prompt, which outputs A (keep greedy), B (swap), or TIE. This uses the bank as structured evidence instead of a hand-designed URL-trace scalar. To prevent verifier variance from causing harmful swaps, each pairwise comparison uses a small odd number m of independent self-verifier calls and applies a conservative majority rule: CTS(ρA,ρB)=ρBif at least ⌈(m+1)/2⌉ calls vote B,ρAotherwise (including ties and majority-A votes).CTS( _A, _B)= cases _B&if at least (m+1)/2 calls vote B,\\ _A&otherwise (including ties and majority-A votes). cases (9) The asymmetric default is the safety valve: ties and uncertainty keep ρA _A. If the bank-conditioned comparison is above chance, then [SR(CTS)]≥[SR(ρA)]−ϵ(m,δ),E[SR(CTS)]\;≥\;E[SR( _A)]-ε(m,δ), (10) with ϵε decreasing as the majority vote becomes more reliable. Here m is the number of verifier votes in a pairwise comparison, whereas K denotes the rollout budget reported in the result tables. CTS swaps only when evidence repeats; otherwise it defaults to greedy. Training and test time therefore share one abstraction: per-URL-state C-certified bank evidence. Training aggregates it into rewards and advantages; CTS aggregates it through a comparative selector. The bank is frozen after training but remains a reusable representation of what the judge taught the agent to notice. 3 Experimental Design We evaluate CLIFT as both a training method and a test-time scaling method. Table 2: Experimental roles. OM2W supplies no reported policy-training data. Benchmark Training role Eval role WAI train Gemma-4 in-domain + board VWA train bank transfer to GPT-5.5 OM2W none zero-shot transfer WAI is the primary training setting: its self-hosted SaaS-style apps expose programmatic verifiers that can support online RL and public-board comparison. VWA is the controlled transfer setting: its Reddit, Shopping, and Classifieds environments use outcome-based URL/DOM checks, so we test whether a bank learned while training an open model remains useful when applied to GPT-5.5 at inference. OM2W is the live-web transfer setting: it contains 300 public-domain tasks judged by an LLM rubric, and no reported policy is trained on OM2W data. 3.1 Task synthesis and bank construction Training tasks are synthesized without reading evaluation tasks. WAI seeds come from app descriptions; VWA seeds come from SynthAgent (Wang et al., 2026) site exploration. Two frontier LLMs expand the seeds, and a shared filter removes malformed, duplicate, hardcoded, pure-lookup, and trivial-navigation tasks. The reported checkpoint trains on 1,8861,886 WAI tasks and 1,2721,272 VWA tasks across 1616 LoRA shards. OM2W is excluded from policy training and used only for bank transfer and CTS. Appendix A.2 gives CLIFT training details. 3.2 Per-benchmark protocols The three benchmarks probe complementary properties of the same verifier bank: WAI tests whether certified self-verification can train a stronger open agent, VWA tests whether the trained-on-open bank transfers to a frozen closed backbone, and OM2W tests whether the same bank recalibrates for live-web zero-shot selection. WAI. We train Gemma 4 31B + LoRA with the full URL-path / host / global C hierarchy. The comparison isolates three effects: the base browser-use backbone, vanilla GRPO with the same comparative judge but no certified bank, and CLIFT with certified step-level feedback. We then apply CTS to the trained checkpoint to measure whether the frozen bank gives an additional no-regression test-time selection gain on the same in-domain tasks. VWA. We use the bank learned during open-model training as structured evidence for a stronger closed backbone. GPT-5.5 is not updated and sees the same completed-task interface; the only added component is CTS over finished trajectories. This setting isolates whether the bank stores transferable web-interaction knowledge rather than merely improving the Gemma-4 policy that produced it. OM2W. We translate and recalibrate the WAI/VWA bank for live-web domains, then run CTS directly at inference. Reported rows use upstream Gemma-4 or GPT-5.5 weights with no OM2W-specific LoRA. Question translation rewrites benchmark-specific URL patterns into the live-web equivalents, drops predicates that no longer apply, and adds OM2W-specific failure-mode questions for bot-block pages, hallucinated extraction, partial-filter application, and wrong-final-answer stops. The recalibration pass scores each translated or new question on a held-out OM2W rollout sample using the same lift-based certifier as training, so any reported gain comes from cross-benchmark transfer of the certified question taxonomy plus the calibration recipe, not from policy training on OM2W. 4 Result Analysis We present results in the same order as the experimental design, from the setting with the most direct supervision to the setting with no policy training at all. WAI asks whether certified self-verification improves an open trainable web agent; VWA asks whether the bank learned during open-model training remains useful for a stronger closed model; OM2W asks whether the same bank can be translated and recalibrated for live-web zero-shot selection. 4.1 Training with certified feedback Table 3: WAI leaderboard slice. Success rates on the public 9-app comparison set; values are percentages without percent signs. The CLIFT +CTS column reports the trained Gemma-4 backbone with K=3K=3 Conformal Trajectory Selection (one selected trajectory per task). Overall is the task-weighted success rate over the 9 apps and matches the CLIFT +CTS row of Table 6. Gemini 3 Qwen 3.5+ Kimi Gemma-4 Gemma-4 + CLIFT +CTS App (n) Flash + BU K2.5 (base, ours) (K=3K=3, ours) Elation Clinical (120) 82.0 54.0 50.0 69.2 91.7 Elation Prescription (120) 81.0 42.0 23.0 70.0 91.7 GitLab Plan & Track (140) 64.0 37.0 39.0 59.3 69.3 Gmail (60) 75.0 57.0 70.0 60.8 78.3 Handshake (200) 50.0 50.0 50.0 47.0 56.0 Linear Acct Mgmt (120) 73.0 66.0 54.0 53.3 64.2 PayPal Wallet (140) 89.0 71.0 71.0 82.1 90.7 Super-human (120) 50.0 26.0 15.0 45.8 52.5 Xero Invoicing (120) 81.0 56.0 52.0 75.0 79.2 Overall (9 apps) 70.1 50.7 46.4 61.8 74.6 Figure 3: WAI training reward. CLIFT separates from vanilla GRPO under the same judge and browser-use stack. Figure 4: WAI progression and gains. Operating points and deltas. The public-board comparison in Table 3 is the main WAI headline: CLIFT-trained Gemma-4 with K=3K=3 CTS reaches the highest 9-app success rate among the published systems we compare against (Claude Opus 4.6 is the training-time judge; CTS itself calls no external judge). The gain is not concentrated in one easy app: CLIFT wins 7 of 9 apps, is competitive on the rest, and improves the same Gemma-4 browser-use backbone by 12.8 percentage points overall. The two apps led by Gemini 3 Flash are within the leaderboard’s integer-rounding noise, and the larger 110-task Gmail variant is reported separately in Figure 4. This is the in-domain training claim: the verifier bank is not merely a post-hoc reranker; it changes the online RL trajectory enough for a 31B open model to exceed the public frontier reference on the matched WAI slice. The training trace in Figure 4 explains why the gap is not just a larger rollout budget at evaluation time. Once the certified bank begins contributing signed per-step evidence, training separates from vanilla GRPO before selection is ever used. This lets the WAI story read from training dynamics to aggregate success, rather than from a single final leaderboard number alone. The reward curves also make the role of CTS easier to interpret: the selector is operating on an already-improved policy, so a small additional WAI lift is expected rather than a failure of test-time scaling. For the aggregate comparison, Figure 4 keeps the operating points in order: base Gemma-4, vanilla GRPO, corrected Opus 4.6, CLIFT, and final CLIFT +CTS. The same-denominator claim is no per-app regression, so the CTS bar marks the final conservative selection point rather than a pure reranking lift. Training supplies the large WAI jump, CTS preserves it and adds a small safe lift, and the larger test-time-scaling effects appear below. 4.2 Bank transfer to a closed backbone Table 4: VWA success rates. C/S/R are Classifieds, Shopping, and Reddit; Avg is weighted by env size. For CTS rows, K denotes rollout budget and one selected trajectory is emitted per task. Method C S R Avg. SGV (Andrade et al., 2026) 52.052.0 57.057.0 33.033.0 50.250.2 WALT (Prabhu et al., 2025) 64.164.1 53.453.4 39.039.0 52.952.9 Gemma-4 base 41.541.5 28.528.5 24.324.3 30.930.9 Gemma-4 + CLIFT 44.944.9 30.730.7 31.031.0 34.434.4 Gemma-4 + CLIFT+CTS (K=2) 45.345.3 32.632.6 31.431.4 35.635.6 GPT-5.5 + CLIFT 60.760.7 43.143.1 47.147.1 48.548.5 GPT-5.5 + CLIFT+CTS (K=4) 62.462.4 51.051.0 50.050.0 53.753.7 Human 91.791.7 88.488.4 87.187.1 88.788.7 VWA is the controlled transfer benchmark between the two training settings. Its three Docker environments have deterministic URL/DOM outcome verifiers, but their visual grounding, page layouts, and navigation routines differ from WAI’s SaaS apps. We therefore use VWA to ask whether a bank learned while training an open Gemma-4 agent can still provide useful evidence when the evaluated policy is GPT-5.5, the executor weights are frozen, and only the certified question bank is reused at inference. The published baselines in Table 4 follow the VisualWebArena comparison protocol (Koh et al., 2024); lower published baselines are omitted from the main table for space. Table 4 follows the same accounting convention as the OM2W table: each selected-output row emits exactly one trajectory per task, so it is directly comparable to the base rollout on the same task set. The Gemma rows isolate the training effect on an open backbone: CLIFT improves the base model before any closed-model transfer is introduced, and CTS adds a smaller additional lift. This mirrors the WAI pattern, but at lower absolute success rates because VWA’s visual and navigation demands are less aligned with the training harness. The GPT-5.5 rows test the transfer claim directly. The policy weights are unchanged; only the certified bank evidence and CTS selection are added at inference. Adding CTS with rollout budget K=4K=4 raises the weighted average to 53.753.7, narrowly above WALT’s 52.952.9 reference, with gains on all three environments. The strongest message is not that VWA is solved (the human row remains far above every agent) but that a bank constructed while training an open model remains useful when the underlying executor is a closed model. 4.3 Cross-benchmark transfer Table 5: Transferred bank on OM2W. Questions are reused from WAI/VWA or expanded, then recalibrated on OM2W rollouts; the final certified bank has 7575 positive-lift and 2525 negative-lift questions with mean lift 0.340.34. Origin Construction |||Q| Certified WAI bank keep/rewrite 3636 3333 (91.7%91.7\%) VWA bank keep/rewrite 4848 3131 (64.6%64.6\%) OM2W expansion live-web checks 4848 3636 (75.0%75.0\%) Final bank 132 qs 132 100 (75.8%) OM2W is our hardest setting: 300 live-web tasks across roughly 30 public domains, judged end-to-end by an LLM rubric rather than DOM matching. We use it as a pure transfer benchmark: no reported policy is fine-tuned on OM2W data. The only adaptation is at the verifier layer: questions mined from the WAI/VWA banks are filtered, rewritten, expanded for live-web failure modes, and recalibrated on OM2W rollouts. Table 5 compresses the main bank evidence. The transferred taxonomy does not collapse under the noisier live-web judge; instead, 100100 of 132132 questions certify, including both positive progress signals and negative failure-mode signals. This is the strongest evidence that the bank is a reusable artifact rather than a benchmark-local prompt list. Figure 5 reports realised success: every “+ ours” row emits one selected trajectory per task, not an oracle union over samples. On the open backbone, the transferred bank and CTS raise Gemma-4 from 40.040.0 to 45.045.0, enough to exceed the GPT-5.5 low-effort baseline on the same judge (43.743.7). On the stronger GPT-5.5 high-effort backbone, the same test-time mechanism raises success from 49.749.7 to 61.061.0 at rollout budget K=4K=4, a +11.3+11.3p gain. This is where test-time scaling is most visible: unlike WAI, where training already moves the greedy policy close to its ceiling, OM2W leaves substantial room for the selector to choose among distinct completed trajectories. The same numbers also show that the open Gemma-4 backbone with transferred selection reaches 45.045.0, slightly above the GPT-5.5 low-effort baseline, even though no OM2W policy weights are trained. Figure 5: OM2W transfer gains. Realised SR; K is rollout budget. The cross-benchmark result also clarifies what transfers. The policy does not: the Gemma rows use upstream weights and the GPT-5.5 rows keep the closed model fixed. What transfers is the certified question taxonomy plus the calibration recipe, which turns prior judge feedback into structured evidence for a new live-web environment. Because the bank is just a list of URL-scoped, polarity-tagged questions with calibrated trust weights, recalibration on a new domain only requires a sample of judge-scored rollouts; no policy update is needed to begin selecting trajectories. Detailed bank construction and transfer tables are in Appendix A.4. 4.4 Ablation study Table 6: WAI ablation summary. Success rates are shown without percent signs. All training rows use Claude Opus 4.6 as the comparative judge with the same browser-use backbone; only the bank/CTS components vary. Steps are averaged over successful tasks in available logs; CTS improves/ties/regresses 5/4/0 apps with 17.8% average swap rate. Variant Isolated change SR Δ vs base Avg solve steps Gemma-4 base no RL, greedy rollout 61.8 – 15.4 Vanilla GRPO scalar comparative-judge reward 64.7 +2.9 18.5 CLIFT certified step reward + URL advantages 72.6 +10.8 16.0 CLIFT +CTS add conservative trajectory selection 74.6 +12.8 15.5 Opus 4.6 closed-model reference, corrected common task IDs 69.8 +8.0 – Table 6 consolidates the ablations. Base Gemma-4 is the no-RL reference; vanilla GRPO isolates scalar judge-reward training; CLIFT adds certified step feedback; and CLIFT +CTS adds only test-time selection. Training accounts for most of the WAI gain, while CTS improves five apps, ties four, and never lowers the single-rollout CLIFT anchor. The decomposition further shows that the gain is not explained simply by additional interaction or inference-time search. Vanilla GRPO improves the base model by only 2.9 points while increasing the average number of steps on successful trajectories from 15.4 to 18.5. In contrast, CLIFT reaches a 10.8-point improvement with an average of 16.0 steps, suggesting that the certified URL-local feedback primarily improves credit assignment rather than encouraging longer trajectories. CTS then adds a further 2.0 points while reducing the average to 15.5 steps. Together with its 17.8% swap rate and zero per-app regressions, this supports the intended division of labor: CLIFT produces the stronger policy during training, while CTS intervenes selectively at test time when the certified bank provides sufficient evidence to prefer an alternative rollout. 5 Discussion and future work CLIFT’s failures are intended to be visible. On WAI and VWA, many URL states accumulate enough observations for signed trust weights; on OM2W, the noisier live-web judge leaves more questions uncertified. This refusal is a diagnostic rather than a silent collapse: it says the current judge, state partition, or question bank does not support reliable distillation, and keeps noisy verifier evidence out of both the training reward and the CTS selector. The study is still bounded. We train one open-weight backbone and rely on a frontier judge during training and bank construction; if that judge is systematically biased, calibration can detect disagreement only through observed data. The transfer results show that the bank can help stronger closed-source agents, but not that one bank is universal across all web domains. Stronger open web agents also carry misuse risk, so deployment should retain rate limits and domain-specific safeguards. A natural next step is to replace natural-language questions with executable browser-state assertions written by a coding agent. The agent would learn not only to act on the web, but also to write compact checks that can be executed during rollouts and reused for test-time selection, reducing dependence on a separate self-verification bank while preserving auditability. 6 Related work CLIFT sits at the intersection of web-agent benchmarks, RL credit assignment, test-time search, and reliable verifier-based selection. The closest systems usually improve one component of this loop: stronger web interfaces for agents, denser reward signals for training, or calibrated decision rules for deployment. CLIFT instead studies how one certified verification bank can connect these components, carrying judge feedback from online RL into test-time trajectory selection. Web agents and test-time search. Language agents combine reasoning, browser actions, and tools (Nakano et al., 2021; Yao et al., 2023b; Shinn et al., 2023; Schick et al., 2023; Qin et al., 2024); search methods add deliberation or tree expansion at inference time (Wei et al., 2022; Yao et al., 2023a; Zhou et al., 2024a; Snell et al., 2025; Song et al., 2024); and modern web, software, desktop, and embodied benchmarks stress realistic interfaces (Zhou et al., 2024b; Koh et al., 2024; Deng et al., 2023; Yao et al., 2022; Shridhar et al., 2021; Yang et al., 2024; Xie et al., 2024). CLIFT differs from prompt-level search because its selector reuses a certified per-URL bank learned during RL training, not just more samples. RL credit assignment and process rewards. GRPO and related RLVR methods improve LLMs from scalar outcomes (Schulman et al., 2017; Shao et al., 2024; DeepSeek-AI, 2025; Qi et al., 2025), while agent variants refine credit assignment through stratification, anchor states, success rates, or Monte-Carlo values (Zhu et al., 2025; Feng et al., 2025; Chen et al., 2025; Kazemnejad et al., 2025). Process-verifier and rich-feedback methods provide denser supervision (Cobbe et al., 2021; Lightman et al., 2024; Wang et al., 2024; Hübotter et al., 2026; Madaan et al., 2023; Agarwal et al., 2024), and LLM judges provide it for open-ended generations (Bai et al., 2022; Liu et al., 2023; Zheng et al., 2023; Rafailov et al., 2023). CLIFT keeps GRPO but changes the reward object: judge feedback is augmented by a certified self-verifier and normalised by URL state. Conformal reliability and specification mining. Conformal prediction gives finite-sample reliability guarantees (Vovk et al., 2005; Angelopoulos and Bates, 2021; Gibbs and Candès, 2021; Vovk, 2012), with recent adaptations to language modeling, decision making, off-policy prediction, LLM-agent routing, and LLM judging (Quach et al., 2024; Lekeufack et al., 2024; Kuipers et al., 2024; Si et al., 2026; Badshah et al., 2026). Those works calibrate outputs, policies, or judgments; CLIFT calibrates the verification questions themselves, using URL scope, polarity, and a Mondrian-ACI trust weight. A CEGAR-style loop (Clarke et al., 2000) turns judge rationales into reusable specifications for both training and test-time selection. 7 Conclusion CLIFT turns expensive judge feedback into a reusable verifier for web agents: conformally certified URL-state evidence densifies training reward, then the frozen bank supports judge-free Conformal Trajectory Selection. Across WAI, VWA, and OM2W, the same bank helps train an open model, transfers to a stronger closed model, and recalibrates for live-web selection without policy fine-tuning. A trajectory score is consumed once; a certified bank can be inspected, recalibrated, and reused. The bank is the durable artefact behind all three results. Each certified question carries a URL scope, a polarity sign, and a trust weight derived from observed lift, with provenance back to the judge rationale that introduced it. This makes the bank small enough to ship with the agent, auditable when a result looks off, and recoverable: when a question’s lift collapses on a new distribution, the certifier withholds trust rather than amplifying noise. Together these properties give one knob, namely what the judge taught the agent to notice, that persists from training reward to test-time selection, from open backbone to closed backbone, and from sandboxed verifiers to live-web judges, and that stronger future judges can grow without discarding what earlier rounds taught. References Agarwal et al. [2024] Rishabh Agarwal, Nino Vieillard, Yongchao Zhou, Piotr Stanczyk, Sabela Ramos, Matthieu Geist, and Olivier Bachem. On-policy distillation of language models: Learning from self-generated mistakes. In International Conference on Learning Representations, 2024. Andrade et al. [2026] Moises Andrade, Joonhyuk Cha, Brandon Ho, Vriksha Srihari, Karmesh Yadav, and Zsolt Kira. Let’s think in two steps: Mitigating agreement bias in MLLMs with self-grounded verification. In International Conference on Learning Representations, 2026. Angelopoulos and Bates [2021] Anastasios N. Angelopoulos and Stephen Bates. A gentle introduction to conformal prediction and distribution-free uncertainty quantification. arXiv preprint arXiv:2107.07511, 2021. Badshah et al. [2026] Sher Badshah, Ali Emami, and Hassan Sajjad. SCOPE: Selective conformal optimized pairwise LLM judging. In International Conference on Machine Learning, 2026. arXiv:2602.13110. Bai et al. [2022] Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. Constitutional AI: Harmlessness from AI feedback. arXiv preprint arXiv:2212.08073, 2022. Chen et al. [2025] Yuhan Chen, Yuxuan Liu, Long Zhang, Pengzhi Gao, Jian Luan, and Wei Liu. STEP: Success-rate-aware trajectory-efficient policy optimization. arXiv:2511.13091, 2025. Clarke et al. [2000] Edmund Clarke, Orna Grumberg, Somesh Jha, Yuan Lu, and Helmut Veith. Counterexample-guided abstraction refinement. In Computer Aided Verification, 2000. Cobbe et al. [2021] Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021. DeepSeek-AI [2025] DeepSeek-AI. DeepSeek-R1: Incentivizing reasoning capability in LLMs via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025. Deng et al. [2023] Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Samuel Stevens, Boshi Wang, Huan Sun, and Yu Su. Mind2Web: Towards a generalist agent for the web. In NeurIPS Datasets and Benchmarks, 2023. Feng et al. [2025] Lang Feng, Zhenghai Xue, Tingcong Liu, and Bo An. Group-in-group policy optimization for LLM agent training. In Advances in Neural Information Processing Systems, 2025. Gibbs and Candès [2021] Isaac Gibbs and Emmanuel Candès. Adaptive conformal inference under distribution shift. Advances in Neural Information Processing Systems, 2021. Hübotter et al. [2026] Jonas Hübotter, Frederike Lübeck, Lejs Behric, Anton Baumann, Marco Bagatella, Daniel Marta, Ido Hakimi, Idan Shenfeld, Thomas Kleine Buening, Carlos Guestrin, and Andreas Krause. Reinforcement learning via self-distillation. arXiv:2601.20802, 2026. Kazemnejad et al. [2025] Amirhossein Kazemnejad, Milad Aghajohari, Eva Portelance, Alessandro Sordoni, Siva Reddy, Aaron Courville, and Nicolas Le Roux. VinePPO: Refining credit assignment in RL training of LLMs. In International Conference on Machine Learning, 2025. Koh et al. [2024] Jing Yu Koh, Robert Lo, Lawrence Jang, Vikram Duvvur, Ming Chong Lim, Po-Yu Huang, Graham Neubig, Shuyan Zhou, Ruslan Salakhutdinov, and Daniel Fried. VisualWebArena: Evaluating multimodal agents on realistic visual web tasks. In ACL, 2024. Kuipers et al. [2024] Tom Kuipers, Renukanandan Tumu, Shuo Yang, Milad Kazemi, Rahul Mangharam, and Nicola Paoletti. Conformal off-policy prediction for multi-agent systems. In IEEE Conference on Decision and Control, 2024. Lekeufack et al. [2024] Jordan Lekeufack, Anastasios N. Angelopoulos, Andrea Bajcsy, Michael I. Jordan, and Jitendra Malik. Conformal decision theory: Safe autonomous decisions from imperfect predictions. In IEEE International Conference on Robotics and Automation (ICRA), 2024. Lightman et al. [2024] Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let’s verify step by step. In International Conference on Learning Representations, 2024. Liu et al. [2023] Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. G-Eval: NLG evaluation using GPT-4 with better human alignment. In Conference on Empirical Methods in Natural Language Processing, 2023. Madaan et al. [2023] Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al. Self-Refine: Iterative refinement with self-feedback. In Advances in Neural Information Processing Systems, 2023. Nakano et al. [2021] Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, Xu Jiang, Karl Cobbe, Tyna Eloundou, Gretchen Krueger, Kevin Button, Matthew Knight, Benjamin Chess, and John Schulman. WebGPT: Browser-assisted question-answering with human feedback. arXiv:2112.09332, 2021. Prabhu et al. [2025] Viraj Prabhu, Yutong Dai, Matthew Fernandez, Jing Gu, Krithika Ramakrishnan, Yanqi Luo, Silvio Savarese, Caiming Xiong, Junnan Li, Zeyuan Chen, and Ran Xu. WALT: Web agents that learn tools. arXiv:2510.01524, 2025. Qi et al. [2025] Zehan Qi, Xiao Liu, Iat Long Iong, Hanyu Lai, Xueqiao Sun, Wenyi Zhao, Yu Yang, Xinyue Yang, Jiadai Sun, Shuntian Yao, Tianjie Zhang, Wei Xu, Jie Tang, and Yuxiao Dong. WebRL: Training LLM web agents via self-evolving online curriculum reinforcement learning. In International Conference on Learning Representations, 2025. Qin et al. [2024] Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, et al. ToolLLM: Facilitating large language models to master 16000+ real-world APIs. In International Conference on Learning Representations, 2024. Quach et al. [2024] Victor Quach, Adam Fisch, Tal Schuster, Adam Yala, Jae Ho Sohn, Tommi S. Jaakkola, and Regina Barzilay. Conformal language modeling. In ICLR, 2024. Rafailov et al. [2023] Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. In Advances in Neural Information Processing Systems, 2023. Schick et al. [2023] Timo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves to use tools. In Advances in Neural Information Processing Systems, 2023. Schulman et al. [2017] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017. Shao et al. [2024] Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Yu Wu, et al. DeepSeekMath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024. Shinn et al. [2023] Noah Shinn, Federico Cassano, Edward Berman, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning. In NeurIPS, 2023. Shridhar et al. [2021] Mohit Shridhar, Xingdi Yuan, Marc-Alexandre Côté, Yonatan Bisk, Adam Trischler, and Matthew Hausknecht. ALFWorld: Aligning text and embodied environments for interactive learning. In ICLR, 2021. Si et al. [2026] Wenwen Si, Sooyong Jang, Insup Lee, and Osbert Bastani. Conformal constrained policy optimization for cost-effective LLM agents. In AAAI Conference on Artificial Intelligence, 2026. Snell et al. [2025] Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. Scaling LLM test-time compute optimally can be more effective than scaling parameters for reasoning. In International Conference on Learning Representations, 2025. Song et al. [2024] Yifan Song, Da Yin, Xiang Yue, Jie Huang, Sujian Li, and Bill Yuchen Lin. Trial and error: Exploration-based trajectory optimization of LLM agents. In Annual Meeting of the Association for Computational Linguistics, pages 7584–7600, 2024. Vovk [2012] Vladimir Vovk. Conditional validity of inductive conformal predictors. In Asian Conference on Machine Learning, 2012. Vovk et al. [2005] Vladimir Vovk, Alexander Gammerman, and Glenn Shafer. Algorithmic Learning in a Random World. Springer, 2005. Wang et al. [2024] Peiyi Wang, Lei Li, Zhihong Shao, Runxin Xu, Damai Dai, Yifei Li, Deli Chen, Yu Wu, and Zhifang Sui. Math-Shepherd: Verify and reinforce LLMs step-by-step without human annotations. In Annual Meeting of the Association for Computational Linguistics, pages 9426–9439, 2024. Wang et al. [2026] Zhaoyang Wang, Yiming Liang, Xuchao Zhang, Qianhui Wu, Siwei Han, Anson Bastos, Rujia Wang, Chetan Bansal, Baolin Peng, Jianfeng Gao, Saravan Rajmohan, and Huaxiu Yao. SynthAgent: Adapting web agents with synthetic supervision. In Annual Meeting of the Association for Computational Linguistics (ACL), 2026. arXiv:2511.06101. Wei et al. [2022] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models. In NeurIPS, 2022. Xie et al. [2024] Tianbao Xie, Danyang Zhang, Jixuan Chen, Xiaochuan Li, Siheng Zhao, Ruisheng Cao, Jing Hua Toh, Zhoujun Cheng, Dongchan Shin, Fangyu Lei, Yitao Liu, Yiheng Xu, Shuyan Zhou, Silvio Savarese, Caiming Xiong, Victor Zhong, and Tao Yu. OSWorld: Benchmarking multimodal agents for open-ended tasks in real computer environments. In NeurIPS, 2024. Xue et al. [2025] Tianci Xue, Weijian Qi, Tianneng Shi, Chan Hee Song, Boyu Gou, Dawn Song, Huan Sun, and Yu Su. An illusion of progress? assessing the current state of web agents. In Conference on Language Modeling (COLM), 2025. arXiv:2504.01382. Yang et al. [2024] John Yang, Carlos E. Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. SWE-agent: Agent-computer interfaces enable automated software engineering. In NeurIPS, 2024. Yao et al. [2022] Shunyu Yao, Howard Chen, John Yang, and Karthik Narasimhan. WebShop: Towards scalable real-world web interaction with grounded language agents. In NeurIPS, 2022. Yao et al. [2023a] Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L. Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. In NeurIPS, 2023a. Yao et al. [2023b] Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. ReAct: Synergizing reasoning and acting in language models. In ICLR, 2023b. Zheng et al. [2023] Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, et al. Judging LLM-as-a-judge with MT-Bench and chatbot arena. In NeurIPS Datasets and Benchmarks, 2023. Zhou et al. [2024a] Andy Zhou, Kai Yan, Michal Shlapentokh-Rothman, Haohan Wang, and Yu-Xiong Wang. Language agent tree search unifies reasoning, acting, and planning in language models. In International Conference on Machine Learning, 2024a. Zhou [2026] Shuyan Zhou. WebArena-Infinity: Generating browser environments with verifiable tasks at scale. https://webarena.dev/webarena-infinity/, March 2026. Zhou et al. [2024b] Shuyan Zhou, Frank F. Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, Uri Alon, and Graham Neubig. WebArena: A realistic web environment for building autonomous agents. In ICLR, 2024b. Zhu et al. [2025] Mingkang Zhu, Xi Chen, Bei Yu, Hengshuang Zhao, and Jiaya Jia. Stratified GRPO: Handling structural heterogeneity in reinforcement learning of LLM search agents. arXiv:2510.06214, 2025. Appendix A Appendix This appendix contains evaluation prompts, the CLIFT-only training setup, detailed bank-transfer accounting for OM2W, and case studies drawn from evaluation logs. A.1 Evaluation Prompts WAI browser-use agent prompt. The WAI browser-use rollout code uses the following system and user templates (abridged only by replacing runtime fields with braces). You are a browsing agent completing a task on a web app. Each turn you see a simplified accessibility snapshot of the current page listing interactable elements with numeric ids. Call exactly ONE tool per turn: click(element_id), type_text(element_id, text, press_enter=True), scroll(direction), goto(url), go_back(), press(key), hover(element_id), stop(answer). Do not narrate. Emit a single tool call. Task: intent Current URL: url Page title: title Accessibility snapshot: snapshot Recent action history: history VWA Gemma-4 agent prompt. The VWA evaluation harness uses the Gemma-4 browser prompt below. You are an autonomous intelligent agent tasked with navigating a web browser. You will be given web-based tasks. The user’s objective: This is the task you’re trying to complete. The current web page screenshot has numbered interactable elements. The observation lists interactable elements as [id] [tagType] [text]. Use browser control tools to navigate and interact with pages. Rules: issue only valid actions; issue one tool call at a time; reason step by step before acting; use element ids; verify before stop; do not stop prematurely; do not construct internal URLs with goto. OBJECTIVE: objective URL: url OBSERVATION: observation PREVIOUS ACTION: previous_action OM2W agent and WebJudge prompts. The OM2W rollout prompt uses the same browser-agent pattern, but for live websites and final-answer tasks: You are a web browsing agent. You will be given a task objective and must interact with a real website to complete it. Each turn you receive a screenshot with numbered element IDs, an accessibility tree, the current URL, and your previous action. Call exactly one tool per turn. Do not construct URLs; click links on the page. Never stop on the first step. For find/show/look-up tasks, navigate to the relevant page first. Verify before calling stop. OBSERVATION: observation URL: url OBJECTIVE: objective PREVIOUS ACTION: previous_action The canonical OM2W judge (WebJudge; Xue et al., 2025) first extracts key points, scores snapshots, and then applies the final judging prompt: You are an expert in evaluating the performance of a web navigation agent. Given the user’s task, the agent’s action history, key points for task completion, and important web pages in the trajectory, determine whether the agent completed the task and achieved all requirements. Important criteria: filters must be applied correctly; exact ranges must match; submission/display actions must be completed; invalid or empty retrieved information may still be success if the required action was correctly performed. Respond in two lines: Thoughts: reasoning Status: "success" or "failure" A.2 CLIFT Training Setup This subsection documents the training setup for the reported CLIFT checkpoint, organised into the four moving parts of the system: the backbone and the task pool that drives RL rollouts; the compute budget and shard layout; the per-rollout reward and verification loop; and the certification rule that turns the running bank into the deployment object loaded by CTS at test time. The concrete hyperparameter values used by all four stages are listed separately in §A.3. Backbone and shard layout. The reported CLIFT checkpoint trains Gemma-4-31B-it with a rank-8 LoRA adapter. We use 1616 single-node shards, each with its own browser workers, reward pipeline, and LoRA adapter. After training, the per-shard adapters are averaged element-wise and merged into the base model for evaluation. OM2W is excluded from policy training entirely; we use it only for bank transfer and CTS at evaluation. Seed pool. The training pool is synthesised from scratch without reading any evaluation task. WAI seeds (186186 tasks) are drawn from each application’s schema and user-facing documentation; VWA seeds (206206 tasks) come from a separate SynthAgent crawl that probes the three Docker environments via header-only navigation. Both seed sets describe what the apps can do, not what the public benchmarks ask the agent to do. Frontier-LLM expansion. Two frontier LLMs (Claude Opus 4.6 and GPT-5.5) expand each seed into a draft pool of roughly 20×20× candidates per seed. Each expansion call sees only the seed’s schema or crawl trace and a small list of formatting constraints; it never sees the public benchmark task list, evaluator code, or any reference answer. Four-stage filter. A shared filter then removes (i) malformed tasks that fail to parse or execute against a sandbox replay, (i) near duplicates under embedding-based dedup at cosine 0.920.92, (i) hardcoded tasks whose answers are already present verbatim in the seed schema, and (iv) trivial-navigation tasks that resolve in a single URL change without any UI interaction or form input. What remains is the 1,8861,886 WAI and 1,2721,272 VWA training tasks used for all reported runs. Train/eval isolation. At no point during synthesis, filtering, sharding, or RL training did the pipeline read WAI, VWA, or OM2W evaluation task descriptions, evaluation HTML/DOM snapshots, or evaluation reward signals. The eval suites are first loaded only after all training runs and adapter averaging are complete, and only by the read-only evaluator. Compute resources. Training ran as shard-parallel single-node jobs on H200 machines; at peak we used eight nodes, each with 8×8×H200 GPUs, and completed 1616 shard jobs across the WAI and VWA training pools. Within each node, resources were split between vLLM serving, LoRA optimization, and browser-rollout processes. Closed-backbone VWA and OM2W evaluations used GPT-5.5 API inference for rollouts and CTS candidates, but did not fine-tune the closed model. Rollout and verification loop. Each prompt produces a GRPO group of G=8G=8 browser rollouts. One comparative batch judge call scores the group at step level and supplies reasoning for CEGAR. In parallel, the base Gemma-4 model answers verification questions with the LoRA adapter detached, so verifier answers come from the same backbone without using the trained task adapter. Certification, reward, and deployment object. For each question, C maintains global, host, and URL-path Mondrian-ACI trackers with target α⋆=0.10α =0.10 and lift-based trust weights; certification uses nmin=50n_ =50 and liftmin=0.02lift_ =0.02. Certified verifier evidence is added through the asymmetric per-step blend, so it can support the judge reward but cannot subtract from it. The resulting URL-stratified returns are written to token advantages over tool-call spans, and the frozen certified bank is the deployment object reused by CTS at test time. A.3 Reported CLIFT Hyperparameters The method section uses symbolic quantities; the concrete values used for the reported runs are: judge threshold θj=0.5 _j=0.5; GRPO group size G=8G=8; certification target α⋆=0.10α =0.10; minimum certification count nmin=50n_ =50; minimum lift liftmin=0.02lift_ =0.02; CEGAR interval M=60M=60 training steps; CEGAR batch size 3 to 5 questions; reward blend β=0.15β=0.15; return discount γ=0.95γ=0.95; CTS vote count m=3m=3 with self-verifier temperature 0.70.7; greedy incumbent temperature 00; diverse retry temperature 1.01.0; and seed-bank sizes 281281 WAI questions and 8989 VWA questions before filtering and certification. A.4 Bank Details and OM2W Transfer This appendix tracks the construction, certification, and transfer of the verification banks that drive both training and test-time selection. For each benchmark we report the seed bank size, the size of the candidate bank after CEGAR-driven growth across training shards, the certified subset retained by the conformal certifier, and (for OM2W only) the subset that survives filtering and rewriting for live-web reuse. The same bank object is read by the training reward (§2.2) and by CTS at test time (§2.3); the numbers below describe how that object grows and shrinks at each stage. WAI and VWA training banks. The WAI training bank starts with 281281 seed verification questions derived from app schemas and user documentation; the VWA training bank starts with 8989 seed questions from SynthAgent site exploration of the three Docker environments. Both banks grow through the CEGAR refinement loop, in which the judge proposes 33 to 55 new questions every M=60M=60 training steps targeting rollouts that the existing bank fails to discriminate, and shrink through certification: questions whose lift falls below 0.020.02 at all tiers, or whose polarity is contradicted by observed lift, are demoted to neutral or dropped. After cross-shard merging, the WAI candidate bank totals 992992 unique questions, of which 4444 retain non-zero trust weights at one or more URL tiers and form the certified WAI bank used by the training reward and by CTS at test time. The VWA candidate bank totals 1,5651,565 unique questions with 2121 certified at the end of training. The 3636 WAI and 4848 VWA questions reported in Table 7 are the portion of each bank that the filter/rewrite stage judged general enough to reuse on live-web OM2W: certified questions kept as-is (77 WAI, 1010 VWA) plus rewritten variants of certified or promising uncertified questions (2929 WAI, 3838 VWA). The remainder stay in their original training banks because their URL patterns or DOM references are app- or environment-specific and do not transfer. Representative certified questions. To make the bank concrete, we walk through six representative certified questions drawn from the merged WAI and VWA banks, using the same prompt-style format as the agent and judge templates above. Each block shows the question identifier, the URL pattern at which it applies (* = global; the bank also contains per-host and per-URL-path refinements such as *google.com* and *reddit.com/r/* that share the same text but track separately under Mondrian-ACI), the observed trust weight w (signed by polarity, magnitude scaled by lift), and the number of judge-scored observations n that supported certification, followed by the YES/NO question that is asked of every rollout at runtime. Q.1: Reaching content-bearing pages (WAI, w=+0.54w=+0.54). The single strongest signal in the bank. Rollouts that visit a page containing task-relevant content are far more likely to receive a high judge score than rollouts that loiter on app homepages or navigation menus, and this pattern holds across all 9 WAI apps. Identifier: made_meaningful_progress_toward_goal URL pattern: * Trust weight: w=+0.54w=+0.54, observations: n=82n=82. Did the agent reach at least one page or state that contains information directly relevant to answering or completing the task (not just the app homepage or navigation menus)? Q.2: Constructive non-destructive behaviour (WAI, w=+0.54w=+0.54). A two-part check that distinguishes "did nothing harmful" from "did something productive". Rollouts that take destructive shortcuts (delete, cancel, revoke) without a constructive counterpart fail this question even when the final URL looks plausible. Identifier: avoided_destructive_unintended_actions URL pattern: * Trust weight: w=+0.54w=+0.54, observations: n=82n=82. Did the agent both (a) avoid performing destructive actions (delete, cancel, revoke) not required by the task AND (b) take at least one constructive action toward the task goal? Q.3: Source-grounded final answer (WAI, w=+0.27w=+0.27). Targets a common failure mode in information-retrieval tasks: emitting an answer without ever opening the page that actually contains the requested fact. Certified at high n, so the verifier has stable evidence at the global tier. Identifier: extracted_info_from_correct_page URL pattern: * Trust weight: w=+0.27w=+0.27, observations: n=520n=520. For information-retrieval tasks, did the agent extract its final answer from the page that actually contains the requested information? Q.4: End-state coherence (WAI, w=+0.26w=+0.26). A simple URL coherence check that catches "wandered off" rollouts in which the agent ends on an unrelated section of the app. Cheap to answer for the base verifier (it only needs the final URL and the task description) and stable across apps. Identifier: final_page_matches_task_context URL pattern: * Trust weight: w=+0.26w=+0.26, observations: n=238n=238. Is the final URL or page the agent ended on relevant to the task objective rather than an unrelated section of the app? Q.5: Active search behaviour (VWA, w=+0.20w=+0.20). A VWA-specific predicate. Distinguishes agents that submit a query and then act on the results from agents that issue a search and freeze on the results page without clicking through. Especially discriminative on the Classifieds and Shopping environments. Identifier: used_search_and_acted_on_results URL pattern: * Trust weight: w=+0.20w=+0.20, observations: n=66n=66. Did the agent both submit a search query AND subsequently take a meaningful action on the results (e.g., clicking a result link rather than just observing the search page)? Q.6: Navigation depth (VWA, w=+0.13w=+0.13). Many VWA tasks require going past the search-results page to a specific listing or product detail page to extract a price, image, or description. This question rewards depth of navigation rather than breadth and is the highest-trust VWA question whose URL pattern is already general enough to translate to live-web OM2W without rewriting. Identifier: visited_target_detail_page URL pattern: * Trust weight: w=+0.13w=+0.13, observations: n=200n=200. Did the agent navigate to a specific item detail page (product page, listing page, or submission page) relevant to the task, beyond the search results page? These six illustrate three properties the certifier rewards: the question must discriminate between high- and low-judge rollouts (positive lift), it must be answerable from the trajectory by the base self-verifier (not too low-level for the LoRA-detached Gemma-4 to read off), and its YES/NO must align with the judge’s separation under the chosen polarity. Questions that fail these tests, e.g., ones that ask about UI elements specific to a single page or that the base model cannot reliably answer, are demoted to neutral or dropped by the polarity-veto rule of Eq. 5. OM2W transfer setting. The main text reports the compact bank-transfer table. Here we keep the more detailed accounting for how the OM2W bank was produced and why it is a strict transfer setting. No reported policy is trained on OM2W; only the question bank is filtered, rewritten, expanded, and recalibrated. Table 7: OM2W bank construction and recalibration. Questions come from WAI/VWA banks plus an OM2W-specific failure-mode expansion. Origin Stage Questions Certified Mean lift WAI bank kept as-is 7 7 0.46 WAI bank rewritten 29 26 0.46 VWA bank kept as-is 10 4 0.31 VWA bank rewritten 38 27 0.28 OM2W expansion live-web checks 48 36 0.30 Total – 132 100 0.34 Transferred question taxonomy. The filter/rewrite stage keeps general web-interaction predicates where possible and removes benchmark-specific app names or URL patterns when they do not transfer. The expansion stage adds questions for live-web failure modes observed on OM2W, including bot-block pages, timeout or wandering behaviour, hallucinated extraction, partial-filter application, search-vs-detail-page confusion, and wrong-final-answer stops. The resulting 132-question bank uses the same schema as the training-time bank, so CTS loads it without code changes. Recalibration. Each question is scored on a stratified pass/fail rollout sample using the base Gemma-4 verifier with logprob-extracted YES probabilities. Lift is computed as the pass/fail separation of the YES probability. Questions whose absolute lift is below the discrimination floor remain in the bank with zero weight; they can be re-certified later on a different rollout sample. Among the 100 certified OM2W questions, 75 have positive lift and 25 have negative lift, so the selector receives both progress evidence and failure-mode evidence. A.5 OM2W Transfer Performance Details Table 8: OM2W transfer performance details. K is the rollout budget and every row emits one selected trajectory. Canonical judge is WebJudge [Xue et al., 2025] + o4-mini at threshold 4; Sonnet is a cross-check. System K Canonical SR Gain Sonnet SR Gemma-4 base 1 40.0 – 46.8 Gemma-4 + ours 2 45.0 +5.0 partial positive GPT-5.5 low base 1 43.7 – 47.3 GPT-5.5 high base 1 49.7 – 52.0 GPT-5.5 high + ours 2 57.3 +7.7 55.7 GPT-5.5 high + ours 4 61.0 +11.3 61.7 Table 8 reports the detailed cross-judge performance numbers behind the compact main-text OM2W figure. It is intentionally separate from Table 7: the bank table explains how the question set was transferred and recalibrated, while this table measures how much realised success that transferred evidence adds after CTS selects one trajectory to emit. The canonical column is the number used in the main paper, since it matches the OM2W judging protocol. The Sonnet column is included only as an external consistency check. It follows the same ordering at the important operating points: the transferred selector improves the Gemma-4 backbone, and the GPT-5.5 high-effort setting gains further as the rollout budget increases from K=2K=2 to K=4K=4. Rows with K=1K=1 are ordinary single rollouts; rows with larger K are CTS-selected outputs, not oracle unions over multiple attempts. This table closes the quantitative OM2W appendix. The next subsection switches from aggregate accounting to qualitative trajectory evidence. A.6 Case Studies Case A.1: CTS save on WAI Gmail. The selector replaces a long failing rollout with a shorter successful one on the same task. Task: Change profile name to "Alex J. Johnson"; create label "Priority" with color #EA4335; apply it to all contacts in the Family label. Greedy trajectory: 87 browser steps; final verifier label = fail. Diverse trajectory: 25 browser steps; changed profile name, created the Priority label, applied it to Laura Johnson-Martinez, Leo Martinez, Margaret Johnson, and Richard Johnson; final verifier label = pass. CTS decision: swap=True, greedy score=-0.5, diverse score=+0.5, chosen_success=True. Case A.2: CTS save on a long WAI Handshake chain. The selected rollout completes a multi-hop message, event, and job chain that the greedy rollout leaves unfinished. Task: Read the latest unread message, RSVP to that sender’s upcoming event, and save all active jobs from that sender. Greedy trajectory: failed to complete the full message/event/job chain. Diverse trajectory: navigated through messages, identified the sender’s event, RSVP’d, and saved the relevant active jobs. CTS decision: swap=True, greedy score=-0.5, diverse score=+0.5, chosen_success=True. Case A.3: No-regression behaviour on WAI Gmail. CTS can also decline a replacement when the greedy rollout is already the better candidate. Task: Disable two-factor authentication. Greedy trajectory: completed the account-security change successfully. Diverse trajectory: failed the task. CTS decision: swap=False; the selector kept the greedy rollout rather than replacing a working trajectory with a worse sample. Case A.4: Live-web recovery on OM2W Indeed. On a live site, the successful trajectory recovers from blank results and an additional-verification page by returning to the native search flow. Task: Find support-services jobs in Bentonville, Arkansas. Trajectory snippet: navigate to Indeed; enter "support services" and "Bentonville, AR"; direct results URL first renders blank; alternate URL hits an additional-verification page; agent returns home, uses the native search form, waits for results, and stops with visible listings. Final answer: includes Healthcare Driver in Bentonville and other support-services listings; WebJudge label = success.